2024-04-21T08:26:08.974 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-21T08:26:08.974 DEBUG:teuthology.run:Teuthology command: teuthology --name leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518 --verbose --description fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} -- /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518/orig.config.yaml 2024-04-21T08:26:09.036 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T08:26:09.127 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518 branch: wip-lusov-quiescer description: fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} email: null first_in_suite: false job_id: '7666518' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 4f6bc5273643aaf5660909e85400647a8951c10b owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7599 sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e sleep_before_teardown: 0 subset: 111/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa suite_relpath: qa suite_repo: https://github.com/leonid-s-usov/ceph-ceph suite_sha1: 4f6bc5273643aaf5660909e85400647a8951c10b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBKjjGJRTuaUxct5Q5Wcl9QsCi7UWUAOc3L0FyKS+058JLFaZuhEyHCRCGbMZByxCoCP/DZLb4KtLwXvi7Y1nt4= smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCPA5gnhRYpDDSIONHCWDTz5JjIYaaFC7IENuniK6XodNcSPGlnzQ4npkgK5yMsq8C7UxDuUYX2ePEQCIqFyf/o= smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAD+Od2KpE3QA4lzWnpiQXsIuqKwIoe2HhQedEiSXETzNJtyvIj8o5FNSDEYOygCckGi3aHGnCyIq4VNj2cxaBc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - kernel_untar_build.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_08:12:55 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-21T08:26:09.128 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa; will attempt to use it 2024-04-21T08:26:09.128 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks 2024-04-21T08:26:09.129 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-21T08:26:09.130 INFO:teuthology.task.internal:Checking packages... 2024-04-21T08:26:09.153 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a630765c1f93c6e61d9b40c4b76437611d5ef29e' 2024-04-21T08:26:09.153 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-21T08:26:09.153 INFO:teuthology.packaging:ref: None 2024-04-21T08:26:09.153 INFO:teuthology.packaging:tag: None 2024-04-21T08:26:09.153 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-21T08:26:09.153 INFO:teuthology.packaging:sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:26:09.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-21T08:26:09.307 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3154.ga630765c 2024-04-21T08:26:09.308 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-21T08:26:09.316 INFO:teuthology.task.internal:no buildpackages task found 2024-04-21T08:26:09.316 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-21T08:26:09.355 INFO:teuthology.task.internal:Saving configuration 2024-04-21T08:26:09.375 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-21T08:26:09.412 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T08:26:09.437 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:06.310808', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBKjjGJRTuaUxct5Q5Wcl9QsCi7UWUAOc3L0FyKS+058JLFaZuhEyHCRCGbMZByxCoCP/DZLb4KtLwXvi7Y1nt4='} 2024-04-21T08:26:09.461 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:06.308956', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCPA5gnhRYpDDSIONHCWDTz5JjIYaaFC7IENuniK6XodNcSPGlnzQ4npkgK5yMsq8C7UxDuUYX2ePEQCIqFyf/o='} 2024-04-21T08:26:09.482 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:06.312132', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAD+Od2KpE3QA4lzWnpiQXsIuqKwIoe2HhQedEiSXETzNJtyvIj8o5FNSDEYOygCckGi3aHGnCyIq4VNj2cxaBc='} 2024-04-21T08:26:09.482 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-21T08:26:09.489 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-21T08:26:09.489 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-21T08:26:09.489 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-21T08:26:09.489 INFO:teuthology.run_tasks:Running task console_log... 2024-04-21T08:26:09.605 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f734caf50d0>, signals=[15]) 2024-04-21T08:26:09.606 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-21T08:26:09.612 INFO:teuthology.task.internal:Opening connections... 2024-04-21T08:26:09.612 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-04-21T08:26:09.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:26:09.695 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-04-21T08:26:09.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:26:09.774 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-04-21T08:26:09.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:26:09.850 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-21T08:26:09.856 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-04-21T08:26:09.875 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-04-21T08:26:09.875 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-04-21T08:26:09.933 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-04-21T08:26:09.933 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2024-04-21T08:26:09.933 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-04-21T08:26:09.934 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:26:09.934 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2024-04-21T08:26:09.934 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:26:09.934 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:26:09.934 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-04-21T08:26:09.934 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:26:09.934 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:26:09.934 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:26:09.934 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:26:09.934 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:26:09.934 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:26:09.935 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-04-21T08:26:09.953 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-04-21T08:26:09.971 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-04-21T08:26:09.971 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-04-21T08:26:10.029 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-04-21T08:26:10.029 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-04-21T08:26:10.029 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-04-21T08:26:10.029 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:26:10.029 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-04-21T08:26:10.030 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:26:10.030 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:26:10.030 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-04-21T08:26:10.030 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:26:10.030 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:26:10.030 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:26:10.030 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:26:10.030 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:26:10.030 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:26:10.030 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-04-21T08:26:10.053 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-04-21T08:26:10.070 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-04-21T08:26:10.070 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-04-21T08:26:10.128 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-04-21T08:26:10.128 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="9" 2024-04-21T08:26:10.128 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-04-21T08:26:10.128 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:26:10.128 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="9" 2024-04-21T08:26:10.128 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:26:10.128 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:26:10.128 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-04-21T08:26:10.128 INFO:teuthology.orchestra.run.smithi077.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:26:10.128 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:26:10.128 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:26:10.129 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:26:10.129 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:26:10.129 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:26:10.129 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-04-21T08:26:10.152 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-21T08:26:10.163 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-21T08:26:10.197 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-21T08:26:10.197 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:26:10.200 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:26:10.202 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:26:10.219 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-21T08:26:10.225 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-21T08:26:10.225 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:26:10.257 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:26:10.259 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:26:10.339 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-21T08:26:10.418 INFO:teuthology.run_tasks:Running task kernel... 2024-04-21T08:26:10.433 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-21T08:26:10.433 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-21T08:26:10.434 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-21T08:26:10.434 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:26:10.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:26:10.451 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-04-21T08:26:10.509 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-437.el9.x86_64 2024-04-21T08:26:10.510 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-437.el9.x86_64 2024-04-21T08:26:10.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2024-04-21T08:26:14.539 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.0 MB 00:02 2024-04-21T08:26:16.365 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-04-21T08:26:28.065 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 795 kB/s | 6.2 MB 00:07 2024-04-21T08:26:29.574 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-04-21T08:26:30.936 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-21T08:26:36.225 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-21T08:26:38.223 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-21T08:26:38.223 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-21T08:26:38.314 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-04-21T08:26:38.315 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-04-21T08:26:38.315 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-04-21T08:26:38.427 DEBUG:teuthology.orchestra.run.smithi007:> echo no | sudo yum reinstall kernel || true 2024-04-21T08:26:38.958 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:02 ago on Sun 21 Apr 2024 08:26:36 AM UTC. 2024-04-21T08:26:39.082 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-04-21T08:26:39.095 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-21T08:26:39.096 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-04-21T08:26:39.096 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-21T08:26:39.096 INFO:teuthology.orchestra.run.smithi007.stdout:Reinstalling: 2024-04-21T08:26:39.096 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T08:26:39.096 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:26:39.096 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-04-21T08:26:39.096 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-21T08:26:39.096 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:26:39.096 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 255 k 2024-04-21T08:26:39.096 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 0 2024-04-21T08:26:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:Operation aborted. 2024-04-21T08:26:39.150 INFO:teuthology.orchestra.run.smithi007.stdout:Is this ok [y/N]: 2024-04-21T08:26:39.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum reinstall -y kernel || true 2024-04-21T08:26:39.664 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:03 ago on Sun 21 Apr 2024 08:26:36 AM UTC. 2024-04-21T08:26:39.785 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-04-21T08:26:39.785 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-21T08:26:39.786 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-04-21T08:26:39.786 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-21T08:26:39.786 INFO:teuthology.orchestra.run.smithi007.stdout:Reinstalling: 2024-04-21T08:26:39.786 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T08:26:39.786 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:26:39.786 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-04-21T08:26:39.786 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-21T08:26:39.786 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:26:39.786 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 255 k 2024-04-21T08:26:39.786 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 0 2024-04-21T08:26:39.786 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-04-21T08:26:41.036 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-437.el9.x86_64.rpm 259 kB/s | 255 kB 00:00 2024-04-21T08:26:41.036 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:26:41.036 INFO:teuthology.orchestra.run.smithi007.stdout:Total 204 kB/s | 255 kB 00:01 2024-04-21T08:26:41.060 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-04-21T08:26:41.270 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-04-21T08:26:41.270 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-04-21T08:26:41.276 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-04-21T08:26:41.276 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-04-21T08:26:41.447 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-04-21T08:26:41.511 INFO:teuthology.orchestra.run.smithi007.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T08:26:41.511 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:26:42.154 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:26:42.154 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T08:26:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:26:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:26:42.825 INFO:teuthology.orchestra.run.smithi007.stdout:Reinstalled: 2024-04-21T08:26:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-21T08:26:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:26:42.825 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-04-21T08:26:42.995 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-04-21T08:26:43.043 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-21T08:26:43.043 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-21T08:26:43.044 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-21T08:26:43.044 DEBUG:teuthology.parallel:result is None 2024-04-21T08:26:43.044 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-21T08:26:43.051 INFO:teuthology.task.internal:Creating test directory... 2024-04-21T08:26:43.051 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:26:43.054 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:26:43.056 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:26:43.074 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-21T08:26:43.091 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-21T08:26:43.124 INFO:teuthology.task.internal:Creating archive directory... 2024-04-21T08:26:43.125 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:26:43.127 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:26:43.129 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:26:43.162 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-21T08:26:43.169 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-21T08:26:43.169 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:26:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:26:43.213 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:26:43.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:26:43.229 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:26:43.232 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:26:43.256 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:26:43.272 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:26:43.287 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:26:43.288 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:26:43.299 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:26:43.303 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:26:43.336 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:26:43.338 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-21T08:26:43.345 INFO:teuthology.task.internal:Configuring sudo... 2024-04-21T08:26:43.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:26:43.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:26:43.350 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:26:43.403 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-21T08:26:43.412 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-21T08:26:43.412 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:26:43.414 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:26:43.417 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:26:43.458 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:26:43.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:26:43.554 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:26:43.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:26:43.675 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:26:43.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:26:43.739 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:26:43.779 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:26:43.838 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:26:43.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:26:43.968 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-21T08:26:43.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:26:44.036 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:26:44.077 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:26:44.127 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:26:44.205 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:26:44.252 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-21T08:26:44.252 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:26:44.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-04-21T08:26:44.319 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-04-21T08:26:44.322 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-04-21T08:26:44.356 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:26:44.358 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:26:44.402 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:26:44.896 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-21T08:26:44.905 INFO:teuthology.task.internal:Starting timer... 2024-04-21T08:26:44.905 INFO:teuthology.run_tasks:Running task pcp... 2024-04-21T08:26:44.953 INFO:teuthology.run_tasks:Running task selinux... 2024-04-21T08:26:44.985 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-21T08:26:44.985 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-04-21T08:26:45.028 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-04-21T08:26:45.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-04-21T08:26:45.091 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-04-21T08:26:45.093 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-04-21T08:26:45.152 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-04-21T08:26:45.154 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-21T08:26:45.154 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-04-21T08:26:45.176 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-04-21T08:26:45.177 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-04-21T08:26:45.231 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-04-21T08:26:45.232 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-04-21T08:26:45.259 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-04-21T08:26:45.259 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi077.front.sepia.ceph.com': 'permissive'} 2024-04-21T08:26:45.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:26:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:26:45.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:26:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:26:45.314 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:26:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:26:45.341 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-21T08:26:45.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:26:45.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:26:45.397 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:26:45.428 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-21T08:26:45.438 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-21T08:26:45.454 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-21T08:26:45.455 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi016.front.sepia.ceph.com,smithi077.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-21T08:30:46.344 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi077.front.sepia.ceph.com')] 2024-04-21T08:30:46.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-04-21T08:30:46.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:30:46.428 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-04-21T08:30:46.511 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-04-21T08:30:46.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-21T08:30:46.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:30:46.589 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-04-21T08:30:46.675 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-21T08:30:46.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-21T08:30:46.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:30:46.793 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-04-21T08:30:46.831 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-21T08:30:46.832 INFO:teuthology.run_tasks:Running task clock... 2024-04-21T08:30:46.843 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-21T08:30:46.843 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:30:46.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:30:46.847 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:30:46.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:30:46.849 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:30:46.849 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:30:46.879 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:30:46.879 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:30:46.893 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:30:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:30:46.900 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:30:46.914 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:30:46.918 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-04-21T08:30:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-04-21T08:30:46.930 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-04-21T08:30:46.931 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-04-21T08:30:46.936 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-04-21T08:30:46.944 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:30:46.946 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:30:46.947 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-04-21T08:30:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:30:46.960 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:30:46.960 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:30:46.974 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:30:47.011 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:30:47.015 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:30:47.015 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-21T08:30:47.015 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.015 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.015 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.015 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.025 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:30:47.029 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:30:47.029 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-04-21T08:30:47.029 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.029 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.029 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.029 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.031 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:30:47.035 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:30:47.037 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-04-21T08:30:47.037 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.037 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.037 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.037 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:30:47.038 INFO:teuthology.run_tasks:Running task install... 2024-04-21T08:30:47.046 DEBUG:teuthology.task.install:project ceph 2024-04-21T08:30:47.046 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e'}} 2024-04-21T08:30:47.046 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e'} 2024-04-21T08:30:47.046 INFO:teuthology.task.install:Using flavor: default 2024-04-21T08:30:47.052 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-21T08:30:47.052 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-21T08:30:47.052 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:30:47.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:30:47.055 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:30:47.055 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:30:47.056 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:30:47.056 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:30:47.279 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:30:47.279 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:30:47.289 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:30:47.289 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:30:47.298 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:30:47.299 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:30:47.453 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:30:47.453 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:30:47.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:30:47.471 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:30:47.472 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-21T08:30:47.472 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:30:47.474 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:30:47.474 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-21T08:30:47.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:30:47.484 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:30:47.484 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:30:47.498 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:30:47.498 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:30:47.500 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:30:47.500 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:30:47.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:30:47.562 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:30:47.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:30:47.630 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:30:47.634 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:30:47.641 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:30:47.696 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2024-04-21T08:30:47.696 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-04-21T08:30:47.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2024-04-21T08:30:47.699 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-04-21T08:30:47.706 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-04-21T08:30:47.708 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-04-21T08:30:47.922 INFO:teuthology.orchestra.run.smithi077.stdout:48 files removed 2024-04-21T08:30:47.934 INFO:teuthology.orchestra.run.smithi007.stdout:48 files removed 2024-04-21T08:30:47.942 INFO:teuthology.orchestra.run.smithi016.stdout:48 files removed 2024-04-21T08:30:47.948 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:30:47.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:30:47.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:30:48.607 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 207 kB/s | 83 kB 00:00 2024-04-21T08:30:48.644 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 203 kB/s | 83 kB 00:00 2024-04-21T08:30:48.708 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 180 kB/s | 83 kB 00:00 2024-04-21T08:30:49.058 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-21T08:30:49.069 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-21T08:30:49.134 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-21T08:30:49.429 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-21T08:30:49.475 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-04-21T08:30:49.494 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-21T08:30:50.097 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-21T08:30:51.192 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.0 MB 00:01 2024-04-21T08:30:51.890 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-04-21T08:30:53.893 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-21T08:30:56.308 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.2 MB 00:00 2024-04-21T08:30:57.975 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-21T08:30:58.567 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 911 kB/s | 8.0 MB 00:09 2024-04-21T08:30:59.502 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.2 MB 00:02 2024-04-21T08:30:59.804 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-21T08:31:01.086 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-21T08:31:01.435 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-21T08:31:02.904 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-21T08:31:04.951 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-21T08:31:05.719 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-21T08:31:07.195 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:31:07.195 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:31:07.207 INFO:teuthology.orchestra.run.smithi007.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:31:07.207 INFO:teuthology.orchestra.run.smithi007.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:31:07.208 INFO:teuthology.orchestra.run.smithi007.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:31:07.209 INFO:teuthology.orchestra.run.smithi007.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:31:07.211 INFO:teuthology.orchestra.run.smithi007.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:31:07.213 INFO:teuthology.orchestra.run.smithi007.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:31:07.214 INFO:teuthology.orchestra.run.smithi007.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:31:07.280 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-04-21T08:31:07.289 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-04-21T08:31:07.289 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-04-21T08:31:07.289 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-04-21T08:31:07.289 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:31:07.290 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:31:07.291 INFO:teuthology.orchestra.run.smithi007.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:31:07.292 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:31:07.293 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:31:07.294 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:31:07.295 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:31:07.295 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:31:07.295 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:31:07.295 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:31:07.295 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:31:07.295 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:31:07.295 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:31:07.295 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:31:07.295 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:31:07.295 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:31:07.296 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:31:07.297 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:31:07.298 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:31:07.299 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:31:07.300 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:31:07.301 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:31:07.302 INFO:teuthology.orchestra.run.smithi007.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:31:07.303 INFO:teuthology.orchestra.run.smithi007.stdout:Installing weak dependencies: 2024-04-21T08:31:07.303 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:31:07.303 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:31:07.303 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-04-21T08:31:07.303 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================= 2024-04-21T08:31:07.303 INFO:teuthology.orchestra.run.smithi007.stdout:Install 148 Packages 2024-04-21T08:31:07.303 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 2 Packages 2024-04-21T08:31:07.303 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:31:07.304 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 218 M 2024-04-21T08:31:07.304 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-04-21T08:31:07.361 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-21T08:31:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-21T08:31:08.170 INFO:teuthology.orchestra.run.smithi007.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-04-21T08:31:08.595 INFO:teuthology.orchestra.run.smithi007.stdout:(2/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 2.0 MB/s | 890 kB 00:00 2024-04-21T08:31:08.670 INFO:teuthology.orchestra.run.smithi007.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 1.9 MB/s | 145 kB 00:00 2024-04-21T08:31:09.087 INFO:teuthology.orchestra.run.smithi007.stdout:(4/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 4.7 MB/s | 5.4 MB 00:01 2024-04-21T08:31:09.246 INFO:teuthology.orchestra.run.smithi007.stdout:(5/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 4.1 MB/s | 2.4 MB 00:00 2024-04-21T08:31:09.321 INFO:teuthology.orchestra.run.smithi007.stdout:(6/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 6.4 MB/s | 1.5 MB 00:00 2024-04-21T08:31:09.438 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-21T08:31:10.047 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:31:10.047 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:31:10.058 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:31:10.059 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:31:10.060 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:31:10.061 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:31:10.062 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:31:10.064 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:31:10.065 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:31:10.124 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-04-21T08:31:10.141 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-04-21T08:31:10.141 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-04-21T08:31:10.141 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-04-21T08:31:10.141 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-04-21T08:31:10.141 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:31:10.141 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:31:10.141 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:31:10.141 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:31:10.142 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:31:10.143 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:31:10.144 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:31:10.145 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:31:10.146 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:31:10.147 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:31:10.148 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:31:10.149 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:31:10.150 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:31:10.151 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-04-21T08:31:10.152 INFO:teuthology.orchestra.run.smithi077.stdout:Install 148 Packages 2024-04-21T08:31:10.153 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 2 Packages 2024-04-21T08:31:10.153 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:31:10.153 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 218 M 2024-04-21T08:31:10.153 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-04-21T08:31:10.171 INFO:teuthology.orchestra.run.smithi007.stdout:(7/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 5.1 MB/s | 4.7 MB 00:00 2024-04-21T08:31:10.763 INFO:teuthology.orchestra.run.smithi007.stdout:(8/150): ceph-common-19.0.0-3154.ga630765c.el9. 7.5 MB/s | 21 MB 00:02 2024-04-21T08:31:10.830 INFO:teuthology.orchestra.run.smithi007.stdout:(9/150): ceph-selinux-19.0.0-3154.ga630765c.el9 379 kB/s | 25 kB 00:00 2024-04-21T08:31:11.089 INFO:teuthology.orchestra.run.smithi077.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 32 kB/s | 6.6 kB 00:00 2024-04-21T08:31:11.155 INFO:teuthology.orchestra.run.smithi007.stdout:(10/150): ceph-osd-19.0.0-3154.ga630765c.el9.x8 9.2 MB/s | 17 MB 00:01 2024-04-21T08:31:11.256 INFO:teuthology.orchestra.run.smithi007.stdout:(11/150): cephfs-mirror-19.0.0-3154.ga630765c.e 2.2 MB/s | 225 kB 00:00 2024-04-21T08:31:11.331 INFO:teuthology.orchestra.run.smithi007.stdout:(12/150): cephfs-shell-19.0.0-3154.ga630765c.el 361 kB/s | 27 kB 00:00 2024-04-21T08:31:11.431 INFO:teuthology.orchestra.run.smithi007.stdout:(13/150): libcephfs-devel-19.0.0-3154.ga630765c 317 kB/s | 32 kB 00:00 2024-04-21T08:31:11.556 INFO:teuthology.orchestra.run.smithi007.stdout:(14/150): libcephfs2-19.0.0-3154.ga630765c.el9. 5.8 MB/s | 746 kB 00:00 2024-04-21T08:31:11.657 INFO:teuthology.orchestra.run.smithi007.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 1.6 MB/s | 163 kB 00:00 2024-04-21T08:31:11.665 INFO:teuthology.orchestra.run.smithi077.stdout:(2/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 1.5 MB/s | 890 kB 00:00 2024-04-21T08:31:11.757 INFO:teuthology.orchestra.run.smithi007.stdout:(16/150): librados-devel-19.0.0-3154.ga630765c. 1.2 MB/s | 127 kB 00:00 2024-04-21T08:31:11.807 INFO:teuthology.orchestra.run.smithi077.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 1.0 MB/s | 145 kB 00:00 2024-04-21T08:31:11.874 INFO:teuthology.orchestra.run.smithi007.stdout:(17/150): libradosstriper1-19.0.0-3154.ga630765 4.2 MB/s | 501 kB 00:00 2024-04-21T08:31:12.466 INFO:teuthology.orchestra.run.smithi007.stdout:(18/150): ceph-radosgw-19.0.0-3154.ga630765c.el 4.6 MB/s | 11 MB 00:02 2024-04-21T08:31:12.558 INFO:teuthology.orchestra.run.smithi007.stdout:(19/150): python3-ceph-argparse-19.0.0-3154.ga6 493 kB/s | 45 kB 00:00 2024-04-21T08:31:12.641 INFO:teuthology.orchestra.run.smithi007.stdout:(20/150): librgw2-19.0.0-3154.ga630765c.el9.x86 7.0 MB/s | 5.4 MB 00:00 2024-04-21T08:31:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:(21/150): python3-ceph-common-19.0.0-3154.ga630 1.2 MB/s | 135 kB 00:00 2024-04-21T08:31:12.733 INFO:teuthology.orchestra.run.smithi007.stdout:(22/150): python3-cephfs-19.0.0-3154.ga630765c. 1.7 MB/s | 162 kB 00:00 2024-04-21T08:31:12.759 INFO:teuthology.orchestra.run.smithi007.stdout:(23/150): python3-rados-19.0.0-3154.ga630765c.e 3.4 MB/s | 321 kB 00:00 2024-04-21T08:31:12.818 INFO:teuthology.orchestra.run.smithi007.stdout:(24/150): python3-rbd-19.0.0-3154.ga630765c.el9 3.5 MB/s | 298 kB 00:00 2024-04-21T08:31:12.852 INFO:teuthology.orchestra.run.smithi007.stdout:(25/150): python3-rgw-19.0.0-3154.ga630765c.el9 1.1 MB/s | 100 kB 00:00 2024-04-21T08:31:12.882 INFO:teuthology.orchestra.run.smithi077.stdout:(4/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 2.2 MB/s | 2.4 MB 00:01 2024-04-21T08:31:12.894 INFO:teuthology.orchestra.run.smithi007.stdout:(26/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-21T08:31:12.986 INFO:teuthology.orchestra.run.smithi007.stdout:(27/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 1.8 MB/s | 171 kB 00:00 2024-04-21T08:31:13.070 INFO:teuthology.orchestra.run.smithi007.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3154.g 410 kB/s | 34 kB 00:00 2024-04-21T08:31:13.170 INFO:teuthology.orchestra.run.smithi007.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 1.4 MB/s | 142 kB 00:00 2024-04-21T08:31:13.603 INFO:teuthology.orchestra.run.smithi007.stdout:(30/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 4.1 MB/s | 3.1 MB 00:00 2024-04-21T08:31:13.640 INFO:teuthology.orchestra.run.smithi077.stdout:(5/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 2.0 MB/s | 1.5 MB 00:00 2024-04-21T08:31:13.762 INFO:teuthology.orchestra.run.smithi007.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 6.0 MB/s | 3.5 MB 00:00 2024-04-21T08:31:13.870 INFO:teuthology.orchestra.run.smithi007.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3154.ga6 2.2 MB/s | 249 kB 00:00 2024-04-21T08:31:13.962 INFO:teuthology.orchestra.run.smithi007.stdout:(33/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 537 kB/s | 49 kB 00:00 2024-04-21T08:31:14.063 INFO:teuthology.orchestra.run.smithi007.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3154.ga 165 kB/s | 16 kB 00:00 2024-04-21T08:31:14.163 INFO:teuthology.orchestra.run.smithi007.stdout:(35/150): ceph-volume-19.0.0-3154.ga630765c.el9 2.7 MB/s | 273 kB 00:00 2024-04-21T08:31:14.165 INFO:teuthology.orchestra.run.smithi077.stdout:(6/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 1.6 MB/s | 5.4 MB 00:03 2024-04-21T08:31:14.338 INFO:teuthology.orchestra.run.smithi007.stdout:(36/150): cephadm-19.0.0-3154.ga630765c.el9.noa 4.3 MB/s | 762 kB 00:00 2024-04-21T08:31:14.430 INFO:teuthology.orchestra.run.smithi007.stdout:(37/150): cephfs-top-19.0.0-3154.ga630765c.el9. 199 kB/s | 18 kB 00:00 2024-04-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi007.stdout:(38/150): fuse-2.9.9-15.el9.x86_64.rpm 689 kB/s | 80 kB 00:00 2024-04-21T08:31:14.647 INFO:teuthology.orchestra.run.smithi007.stdout:(39/150): libgfortran-11.4.1-3.el9.x86_64.rpm 7.8 MB/s | 803 kB 00:00 2024-04-21T08:31:14.689 INFO:teuthology.orchestra.run.smithi007.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-04-21T08:31:14.706 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-04-21T08:31:14.731 INFO:teuthology.orchestra.run.smithi007.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 6.0 MB/s | 253 kB 00:00 2024-04-21T08:31:14.807 INFO:teuthology.orchestra.run.smithi007.stdout:(42/150): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-04-21T08:31:14.848 INFO:teuthology.orchestra.run.smithi007.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-21T08:31:14.890 INFO:teuthology.orchestra.run.smithi007.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-21T08:31:14.933 INFO:teuthology.orchestra.run.smithi007.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-21T08:31:14.974 INFO:teuthology.orchestra.run.smithi007.stdout:(46/150): python3-urllib3-1.26.5-5.el9.noarch.r 5.1 MB/s | 215 kB 00:00 2024-04-21T08:31:15.066 INFO:teuthology.orchestra.run.smithi007.stdout:(47/150): automake-1.16.2-8.el9.noarch.rpm 7.3 MB/s | 689 kB 00:00 2024-04-21T08:31:15.217 INFO:teuthology.orchestra.run.smithi007.stdout:(48/150): ceph-mgr-diskprediction-local-19.0.0- 4.6 MB/s | 7.4 MB 00:01 2024-04-21T08:31:15.259 INFO:teuthology.orchestra.run.smithi007.stdout:(49/150): bison-3.7.4-5.el9.x86_64.rpm 4.8 MB/s | 943 kB 00:00 2024-04-21T08:31:15.309 INFO:teuthology.orchestra.run.smithi007.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 431 kB/s | 21 kB 00:00 2024-04-21T08:31:15.334 INFO:teuthology.orchestra.run.smithi007.stdout:(51/150): boost-program-options-1.75.0-8.el9.x8 914 kB/s | 107 kB 00:00 2024-04-21T08:31:15.385 INFO:teuthology.orchestra.run.smithi007.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 651 kB/s | 33 kB 00:00 2024-04-21T08:31:15.427 INFO:teuthology.orchestra.run.smithi007.stdout:(53/150): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-21T08:31:15.441 INFO:teuthology.orchestra.run.smithi077.stdout:(7/150): ceph-common-19.0.0-3154.ga630765c.el9. 4.6 MB/s | 21 MB 00:04 2024-04-21T08:31:15.477 INFO:teuthology.orchestra.run.smithi007.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-04-21T08:31:15.507 INFO:teuthology.orchestra.run.smithi077.stdout:(8/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 2.5 MB/s | 4.7 MB 00:01 2024-04-21T08:31:15.558 INFO:teuthology.orchestra.run.smithi077.stdout:(9/150): ceph-selinux-19.0.0-3154.ga630765c.el9 505 kB/s | 25 kB 00:00 2024-04-21T08:31:15.928 INFO:teuthology.orchestra.run.smithi007.stdout:(55/150): ceph-test-19.0.0-3154.ga630765c.el9.x 9.6 MB/s | 49 MB 00:05 2024-04-21T08:31:15.961 INFO:teuthology.orchestra.run.smithi007.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 23 kB/s | 11 kB 00:00 2024-04-21T08:31:16.011 INFO:teuthology.orchestra.run.smithi007.stdout:(57/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.8 MB/s | 3.0 MB 00:00 2024-04-21T08:31:16.045 INFO:teuthology.orchestra.run.smithi007.stdout:(58/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-21T08:31:16.070 INFO:teuthology.orchestra.run.smithi007.stdout:(59/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 416 kB/s | 45 kB 00:00 2024-04-21T08:31:16.121 INFO:teuthology.orchestra.run.smithi007.stdout:(60/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 509 kB/s | 25 kB 00:00 2024-04-21T08:31:16.146 INFO:teuthology.orchestra.run.smithi007.stdout:(61/150): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-04-21T08:31:16.180 INFO:teuthology.orchestra.run.smithi007.stdout:(62/150): libtool-2.4.6-45.el9.x86_64.rpm 4.2 MB/s | 582 kB 00:00 2024-04-21T08:31:16.205 INFO:teuthology.orchestra.run.smithi007.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-04-21T08:31:16.255 INFO:teuthology.orchestra.run.smithi007.stdout:(64/150): openblas-0.3.21-2.el9.x86_64.rpm 707 kB/s | 35 kB 00:00 2024-04-21T08:31:16.280 INFO:teuthology.orchestra.run.smithi007.stdout:(65/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-04-21T08:31:16.305 INFO:teuthology.orchestra.run.smithi007.stdout:(66/150): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-04-21T08:31:16.441 INFO:teuthology.orchestra.run.smithi077.stdout:(10/150): ceph-radosgw-19.0.0-3154.ga630765c.el 11 MB/s | 11 MB 00:01 2024-04-21T08:31:16.473 INFO:teuthology.orchestra.run.smithi007.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-04-21T08:31:16.492 INFO:teuthology.orchestra.run.smithi077.stdout:(11/150): cephfs-mirror-19.0.0-3154.ga630765c.e 4.4 MB/s | 225 kB 00:00 2024-04-21T08:31:16.531 INFO:teuthology.orchestra.run.smithi007.stdout:(68/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 17 MB/s | 4.1 MB 00:00 2024-04-21T08:31:16.542 INFO:teuthology.orchestra.run.smithi077.stdout:(12/150): cephfs-shell-19.0.0-3154.ga630765c.el 543 kB/s | 27 kB 00:00 2024-04-21T08:31:16.584 INFO:teuthology.orchestra.run.smithi077.stdout:(13/150): libcephfs-devel-19.0.0-3154.ga630765c 763 kB/s | 32 kB 00:00 2024-04-21T08:31:16.589 INFO:teuthology.orchestra.run.smithi007.stdout:(69/150): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-04-21T08:31:16.648 INFO:teuthology.orchestra.run.smithi007.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 133 kB 00:00 2024-04-21T08:31:16.682 INFO:teuthology.orchestra.run.smithi007.stdout:(71/150): postgresql-contrib-13.14-1.el9.x86_64 4.3 MB/s | 911 kB 00:00 2024-04-21T08:31:16.743 INFO:teuthology.orchestra.run.smithi077.stdout:(14/150): libcephfs2-19.0.0-3154.ga630765c.el9. 4.6 MB/s | 746 kB 00:00 2024-04-21T08:31:16.749 INFO:teuthology.orchestra.run.smithi007.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-21T08:31:16.791 INFO:teuthology.orchestra.run.smithi007.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-04-21T08:31:16.809 INFO:teuthology.orchestra.run.smithi077.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 2.4 MB/s | 163 kB 00:00 2024-04-21T08:31:16.877 INFO:teuthology.orchestra.run.smithi077.stdout:(16/150): librados-devel-19.0.0-3154.ga630765c. 1.9 MB/s | 127 kB 00:00 2024-04-21T08:31:16.908 INFO:teuthology.orchestra.run.smithi007.stdout:(74/150): postgresql-server-13.14-1.el9.x86_64. 18 MB/s | 5.8 MB 00:00 2024-04-21T08:31:16.941 INFO:teuthology.orchestra.run.smithi007.stdout:(75/150): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-04-21T08:31:16.968 INFO:teuthology.orchestra.run.smithi077.stdout:(17/150): libradosstriper1-19.0.0-3154.ga630765 5.3 MB/s | 501 kB 00:00 2024-04-21T08:31:16.982 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:31:16.983 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:31:16.983 INFO:teuthology.orchestra.run.smithi007.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 618 kB/s | 26 kB 00:00 2024-04-21T08:31:16.994 INFO:teuthology.orchestra.run.smithi016.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:31:16.995 INFO:teuthology.orchestra.run.smithi016.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:31:16.996 INFO:teuthology.orchestra.run.smithi016.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:31:16.997 INFO:teuthology.orchestra.run.smithi016.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:31:16.998 INFO:teuthology.orchestra.run.smithi016.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:31:17.001 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:31:17.002 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:31:17.008 INFO:teuthology.orchestra.run.smithi007.stdout:(77/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-21T08:31:17.042 INFO:teuthology.orchestra.run.smithi007.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 326 kB/s | 19 kB 00:00 2024-04-21T08:31:17.068 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-21T08:31:17.084 INFO:teuthology.orchestra.run.smithi007.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-21T08:31:17.085 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-04-21T08:31:17.085 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-21T08:31:17.086 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-04-21T08:31:17.086 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-21T08:31:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:31:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:31:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:31:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:31:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:31:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:31:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:31:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:31:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:31:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:31:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:31:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:31:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:31:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:31:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:31:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:31:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:31:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:31:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:31:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:31:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:31:17.090 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:31:17.090 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:31:17.091 INFO:teuthology.orchestra.run.smithi016.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:31:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:31:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:31:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:31:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:31:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:31:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:31:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:31:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:31:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout:Install 148 Packages 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 2 Packages 2024-04-21T08:31:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:31:17.102 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 218 M 2024-04-21T08:31:17.102 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-21T08:31:17.109 INFO:teuthology.orchestra.run.smithi007.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 519 kB/s | 35 kB 00:00 2024-04-21T08:31:17.227 INFO:teuthology.orchestra.run.smithi007.stdout:(81/150): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-04-21T08:31:17.277 INFO:teuthology.orchestra.run.smithi007.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 250 kB 00:00 2024-04-21T08:31:17.327 INFO:teuthology.orchestra.run.smithi007.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-04-21T08:31:17.353 INFO:teuthology.orchestra.run.smithi007.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 77 kB 00:00 2024-04-21T08:31:17.402 INFO:teuthology.orchestra.run.smithi007.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 562 kB/s | 42 kB 00:00 2024-04-21T08:31:17.470 INFO:teuthology.orchestra.run.smithi007.stdout:(86/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-04-21T08:31:17.495 INFO:teuthology.orchestra.run.smithi007.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 1.9 MB/s | 269 kB 00:00 2024-04-21T08:31:17.527 INFO:teuthology.orchestra.run.smithi077.stdout:(18/150): librgw2-19.0.0-3154.ga630765c.el9.x86 9.6 MB/s | 5.4 MB 00:00 2024-04-21T08:31:17.537 INFO:teuthology.orchestra.run.smithi007.stdout:(88/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-21T08:31:17.562 INFO:teuthology.orchestra.run.smithi007.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 761 kB/s | 51 kB 00:00 2024-04-21T08:31:17.569 INFO:teuthology.orchestra.run.smithi077.stdout:(19/150): python3-ceph-argparse-19.0.0-3154.ga6 1.1 MB/s | 45 kB 00:00 2024-04-21T08:31:17.596 INFO:teuthology.orchestra.run.smithi007.stdout:(90/150): python3-pyasn1-modules-0.4.8-6.el9.no 2.2 MB/s | 279 kB 00:00 2024-04-21T08:31:17.619 INFO:teuthology.orchestra.run.smithi077.stdout:(20/150): python3-ceph-common-19.0.0-3154.ga630 2.6 MB/s | 135 kB 00:00 2024-04-21T08:31:17.621 INFO:teuthology.orchestra.run.smithi007.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 640 kB/s | 54 kB 00:00 2024-04-21T08:31:17.670 INFO:teuthology.orchestra.run.smithi077.stdout:(21/150): python3-cephfs-19.0.0-3154.ga630765c. 3.2 MB/s | 162 kB 00:00 2024-04-21T08:31:17.671 INFO:teuthology.orchestra.run.smithi007.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-04-21T08:31:17.697 INFO:teuthology.orchestra.run.smithi007.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 586 kB/s | 44 kB 00:00 2024-04-21T08:31:17.745 INFO:teuthology.orchestra.run.smithi077.stdout:(22/150): python3-rados-19.0.0-3154.ga630765c.e 4.2 MB/s | 321 kB 00:00 2024-04-21T08:31:17.747 INFO:teuthology.orchestra.run.smithi007.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 721 kB/s | 36 kB 00:00 2024-04-21T08:31:17.772 INFO:teuthology.orchestra.run.smithi007.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-04-21T08:31:17.798 INFO:teuthology.orchestra.run.smithi016.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 33 kB/s | 6.6 kB 00:00 2024-04-21T08:31:17.803 INFO:teuthology.orchestra.run.smithi077.stdout:(23/150): python3-rbd-19.0.0-3154.ga630765c.el9 5.0 MB/s | 298 kB 00:00 2024-04-21T08:31:17.831 INFO:teuthology.orchestra.run.smithi007.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 567 kB/s | 47 kB 00:00 2024-04-21T08:31:17.862 INFO:teuthology.orchestra.run.smithi077.stdout:(24/150): python3-rgw-19.0.0-3154.ga630765c.el9 1.7 MB/s | 100 kB 00:00 2024-04-21T08:31:17.881 INFO:teuthology.orchestra.run.smithi007.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 305 kB 00:00 2024-04-21T08:31:17.906 INFO:teuthology.orchestra.run.smithi007.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 768 kB/s | 58 kB 00:00 2024-04-21T08:31:17.921 INFO:teuthology.orchestra.run.smithi077.stdout:(25/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-21T08:31:17.982 INFO:teuthology.orchestra.run.smithi007.stdout:(99/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 848 kB/s | 64 kB 00:00 2024-04-21T08:31:18.007 INFO:teuthology.orchestra.run.smithi007.stdout:(100/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 747 kB/s | 94 kB 00:00 2024-04-21T08:31:18.066 INFO:teuthology.orchestra.run.smithi007.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 535 kB/s | 45 kB 00:00 2024-04-21T08:31:18.223 INFO:teuthology.orchestra.run.smithi016.stdout:(2/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 2.0 MB/s | 890 kB 00:00 2024-04-21T08:31:18.258 INFO:teuthology.orchestra.run.smithi007.stdout:(102/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-04-21T08:31:18.271 INFO:teuthology.orchestra.run.smithi077.stdout:(26/150): ceph-osd-19.0.0-3154.ga630765c.el9.x8 4.1 MB/s | 17 MB 00:04 2024-04-21T08:31:18.283 INFO:teuthology.orchestra.run.smithi007.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 81 kB/s | 22 kB 00:00 2024-04-21T08:31:18.322 INFO:teuthology.orchestra.run.smithi077.stdout:(27/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 3.3 MB/s | 171 kB 00:00 2024-04-21T08:31:18.323 INFO:teuthology.orchestra.run.smithi016.stdout:(3/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 7.4 MB/s | 5.4 MB 00:00 2024-04-21T08:31:18.349 INFO:teuthology.orchestra.run.smithi016.stdout:(4/150): ceph-immutable-object-cache-19.0.0-315 1.1 MB/s | 145 kB 00:00 2024-04-21T08:31:18.389 INFO:teuthology.orchestra.run.smithi077.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3154.g 514 kB/s | 34 kB 00:00 2024-04-21T08:31:18.481 INFO:teuthology.orchestra.run.smithi077.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 1.5 MB/s | 142 kB 00:00 2024-04-21T08:31:18.548 INFO:teuthology.orchestra.run.smithi077.stdout:(30/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 4.9 MB/s | 3.1 MB 00:00 2024-04-21T08:31:18.591 INFO:teuthology.orchestra.run.smithi016.stdout:(5/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 8.8 MB/s | 2.4 MB 00:00 2024-04-21T08:31:18.874 INFO:teuthology.orchestra.run.smithi016.stdout:(6/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-04-21T08:31:18.925 INFO:teuthology.orchestra.run.smithi007.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 1.0 MB/s | 863 kB 00:00 2024-04-21T08:31:19.141 INFO:teuthology.orchestra.run.smithi016.stdout:(7/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 8.5 MB/s | 4.7 MB 00:00 2024-04-21T08:31:19.158 INFO:teuthology.orchestra.run.smithi007.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 353 kB/s | 309 kB 00:00 2024-04-21T08:31:19.375 INFO:teuthology.orchestra.run.smithi007.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 43 kB/s | 19 kB 00:00 2024-04-21T08:31:19.459 INFO:teuthology.orchestra.run.smithi007.stdout:(107/150): abseil-cpp-20211102.0-3.el9.x86_64.r 456 kB/s | 548 kB 00:01 2024-04-21T08:31:19.484 INFO:teuthology.orchestra.run.smithi007.stdout:(108/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-04-21T08:31:19.560 INFO:teuthology.orchestra.run.smithi007.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 483 kB/s | 49 kB 00:00 2024-04-21T08:31:19.632 INFO:teuthology.orchestra.run.smithi077.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 3.1 MB/s | 3.5 MB 00:01 2024-04-21T08:31:19.752 INFO:teuthology.orchestra.run.smithi007.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 252 kB/s | 67 kB 00:00 2024-04-21T08:31:19.790 INFO:teuthology.orchestra.run.smithi077.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3154.ga6 1.5 MB/s | 249 kB 00:00 2024-04-21T08:31:19.819 INFO:teuthology.orchestra.run.smithi007.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 584 kB/s | 151 kB 00:00 2024-04-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi077.stdout:(33/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 539 kB/s | 49 kB 00:00 2024-04-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi077.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3154.ga 179 kB/s | 16 kB 00:00 2024-04-21T08:31:20.157 INFO:teuthology.orchestra.run.smithi077.stdout:(35/150): ceph-volume-19.0.0-3154.ga630765c.el9 1.5 MB/s | 273 kB 00:00 2024-04-21T08:31:20.500 INFO:teuthology.orchestra.run.smithi077.stdout:(36/150): cephadm-19.0.0-3154.ga630765c.el9.noa 2.2 MB/s | 762 kB 00:00 2024-04-21T08:31:20.569 INFO:teuthology.orchestra.run.smithi007.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 731 kB/s | 548 kB 00:00 2024-04-21T08:31:20.583 INFO:teuthology.orchestra.run.smithi077.stdout:(37/150): ceph-mgr-diskprediction-local-19.0.0- 3.6 MB/s | 7.4 MB 00:02 2024-04-21T08:31:20.608 INFO:teuthology.orchestra.run.smithi077.stdout:(38/150): cephfs-top-19.0.0-3154.ga630765c.el9. 168 kB/s | 18 kB 00:00 2024-04-21T08:31:20.641 INFO:teuthology.orchestra.run.smithi016.stdout:(8/150): ceph-radosgw-19.0.0-3154.ga630765c.el9 7.1 MB/s | 11 MB 00:01 2024-04-21T08:31:20.669 INFO:teuthology.orchestra.run.smithi007.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 435 kB/s | 43 kB 00:00 2024-04-21T08:31:20.700 INFO:teuthology.orchestra.run.smithi016.stdout:(9/150): ceph-selinux-19.0.0-3154.ga630765c.el9 434 kB/s | 25 kB 00:00 2024-04-21T08:31:20.751 INFO:teuthology.orchestra.run.smithi077.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 482 kB/s | 80 kB 00:00 2024-04-21T08:31:20.769 INFO:teuthology.orchestra.run.smithi007.stdout:(114/150): python3-cachetools-4.2.4-1.el9.noarc 322 kB/s | 32 kB 00:00 2024-04-21T08:31:20.842 INFO:teuthology.orchestra.run.smithi077.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-21T08:31:20.844 INFO:teuthology.orchestra.run.smithi007.stdout:(115/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 767 kB/s | 838 kB 00:01 2024-04-21T08:31:20.870 INFO:teuthology.orchestra.run.smithi007.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-21T08:31:20.926 INFO:teuthology.orchestra.run.smithi077.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-04-21T08:31:20.985 INFO:teuthology.orchestra.run.smithi077.stdout:(42/150): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-04-21T08:31:21.043 INFO:teuthology.orchestra.run.smithi077.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-21T08:31:21.104 INFO:teuthology.orchestra.run.smithi007.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 664 kB/s | 172 kB 00:00 2024-04-21T08:31:21.127 INFO:teuthology.orchestra.run.smithi077.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-21T08:31:21.210 INFO:teuthology.orchestra.run.smithi077.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-21T08:31:21.242 INFO:teuthology.orchestra.run.smithi016.stdout:(10/150): ceph-common-19.0.0-3154.ga630765c.el9 5.8 MB/s | 21 MB 00:03 2024-04-21T08:31:21.286 INFO:teuthology.orchestra.run.smithi077.stdout:(46/150): python3-cryptography-36.0.1-4.el9.x86 3.4 MB/s | 1.2 MB 00:00 2024-04-21T08:31:21.311 INFO:teuthology.orchestra.run.smithi077.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 2.1 MB/s | 215 kB 00:00 2024-04-21T08:31:21.371 INFO:teuthology.orchestra.run.smithi007.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 716 kB/s | 358 kB 00:00 2024-04-21T08:31:21.409 INFO:teuthology.orchestra.run.smithi016.stdout:(11/150): ceph-osd-19.0.0-3154.ga630765c.el9.x8 6.6 MB/s | 17 MB 00:02 2024-04-21T08:31:21.435 INFO:teuthology.orchestra.run.smithi016.stdout:(12/150): cephfs-mirror-19.0.0-3154.ga630765c.e 1.1 MB/s | 225 kB 00:00 2024-04-21T08:31:21.460 INFO:teuthology.orchestra.run.smithi016.stdout:(13/150): cephfs-shell-19.0.0-3154.ga630765c.el 537 kB/s | 27 kB 00:00 2024-04-21T08:31:21.471 INFO:teuthology.orchestra.run.smithi007.stdout:(119/150): python3-colorama-0.4.4-12.el9.noarch 354 kB/s | 35 kB 00:00 2024-04-21T08:31:21.485 INFO:teuthology.orchestra.run.smithi016.stdout:(14/150): libcephfs-devel-19.0.0-3154.ga630765c 631 kB/s | 32 kB 00:00 2024-04-21T08:31:21.544 INFO:teuthology.orchestra.run.smithi016.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 2.7 MB/s | 163 kB 00:00 2024-04-21T08:31:21.563 INFO:teuthology.orchestra.run.smithi007.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 660 kB/s | 303 kB 00:00 2024-04-21T08:31:21.569 INFO:teuthology.orchestra.run.smithi016.stdout:(16/150): libcephfs2-19.0.0-3154.ga630765c.el9. 6.7 MB/s | 746 kB 00:00 2024-04-21T08:31:21.594 INFO:teuthology.orchestra.run.smithi016.stdout:(17/150): librados-devel-19.0.0-3154.ga630765c. 2.5 MB/s | 127 kB 00:00 2024-04-21T08:31:21.636 INFO:teuthology.orchestra.run.smithi016.stdout:(18/150): libradosstriper1-19.0.0-3154.ga630765 7.3 MB/s | 501 kB 00:00 2024-04-21T08:31:21.695 INFO:teuthology.orchestra.run.smithi016.stdout:(19/150): python3-ceph-argparse-19.0.0-3154.ga6 774 kB/s | 45 kB 00:00 2024-04-21T08:31:21.771 INFO:teuthology.orchestra.run.smithi016.stdout:(20/150): python3-ceph-common-19.0.0-3154.ga630 1.7 MB/s | 135 kB 00:00 2024-04-21T08:31:21.821 INFO:teuthology.orchestra.run.smithi007.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 578 kB/s | 203 kB 00:00 2024-04-21T08:31:21.838 INFO:teuthology.orchestra.run.smithi016.stdout:(21/150): python3-cephfs-19.0.0-3154.ga630765c. 2.4 MB/s | 162 kB 00:00 2024-04-21T08:31:21.913 INFO:teuthology.orchestra.run.smithi016.stdout:(22/150): python3-rados-19.0.0-3154.ga630765c.e 4.2 MB/s | 321 kB 00:00 2024-04-21T08:31:21.988 INFO:teuthology.orchestra.run.smithi016.stdout:(23/150): python3-rbd-19.0.0-3154.ga630765c.el9 3.9 MB/s | 298 kB 00:00 2024-04-21T08:31:22.053 INFO:teuthology.orchestra.run.smithi077.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 899 kB/s | 689 kB 00:00 2024-04-21T08:31:22.055 INFO:teuthology.orchestra.run.smithi016.stdout:(24/150): python3-rgw-19.0.0-3154.ga630765c.el9 1.5 MB/s | 100 kB 00:00 2024-04-21T08:31:22.080 INFO:teuthology.orchestra.run.smithi007.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 560 kB/s | 144 kB 00:00 2024-04-21T08:31:22.095 INFO:teuthology.orchestra.run.smithi077.stdout:(49/150): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-04-21T08:31:22.122 INFO:teuthology.orchestra.run.smithi016.stdout:(25/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 1.2 MB/s | 85 kB 00:00 2024-04-21T08:31:22.154 INFO:teuthology.orchestra.run.smithi077.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 1.0 MB/s | 107 kB 00:00 2024-04-21T08:31:22.180 INFO:teuthology.orchestra.run.smithi007.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-21T08:31:22.187 INFO:teuthology.orchestra.run.smithi077.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 234 kB/s | 21 kB 00:00 2024-04-21T08:31:22.189 INFO:teuthology.orchestra.run.smithi016.stdout:(26/150): librgw2-19.0.0-3154.ga630765c.el9.x86 9.0 MB/s | 5.4 MB 00:00 2024-04-21T08:31:22.263 INFO:teuthology.orchestra.run.smithi077.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-21T08:31:22.272 INFO:teuthology.orchestra.run.smithi016.stdout:(27/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-21T08:31:22.288 INFO:teuthology.orchestra.run.smithi077.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 326 kB/s | 33 kB 00:00 2024-04-21T08:31:22.289 INFO:teuthology.orchestra.run.smithi007.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-21T08:31:22.348 INFO:teuthology.orchestra.run.smithi016.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3154.g 455 kB/s | 34 kB 00:00 2024-04-21T08:31:22.389 INFO:teuthology.orchestra.run.smithi077.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 182 kB/s | 18 kB 00:00 2024-04-21T08:31:22.389 INFO:teuthology.orchestra.run.smithi007.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-21T08:31:22.415 INFO:teuthology.orchestra.run.smithi016.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 2.1 MB/s | 142 kB 00:00 2024-04-21T08:31:22.474 INFO:teuthology.orchestra.run.smithi016.stdout:(30/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 8.8 MB/s | 3.1 MB 00:00 2024-04-21T08:31:22.489 INFO:teuthology.orchestra.run.smithi077.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-04-21T08:31:22.489 INFO:teuthology.orchestra.run.smithi007.stdout:(126/150): python3-jaraco-functools-3.5.0-2.el9 195 kB/s | 19 kB 00:00 2024-04-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi007.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 197 kB/s | 20 kB 00:00 2024-04-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi077.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-04-21T08:31:22.706 INFO:teuthology.orchestra.run.smithi077.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-04-21T08:31:22.790 INFO:teuthology.orchestra.run.smithi077.stdout:(58/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.7 MB/s | 3.0 MB 00:00 2024-04-21T08:31:22.823 INFO:teuthology.orchestra.run.smithi077.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-04-21T08:31:22.891 INFO:teuthology.orchestra.run.smithi016.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 7.4 MB/s | 3.5 MB 00:00 2024-04-21T08:31:22.907 INFO:teuthology.orchestra.run.smithi077.stdout:(60/150): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-04-21T08:31:22.924 INFO:teuthology.orchestra.run.smithi077.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 253 kB/s | 25 kB 00:00 2024-04-21T08:31:22.966 INFO:teuthology.orchestra.run.smithi016.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3154.ga6 3.3 MB/s | 249 kB 00:00 2024-04-21T08:31:22.999 INFO:teuthology.orchestra.run.smithi077.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 516 kB/s | 47 kB 00:00 2024-04-21T08:31:23.033 INFO:teuthology.orchestra.run.smithi077.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-21T08:31:23.033 INFO:teuthology.orchestra.run.smithi016.stdout:(33/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 739 kB/s | 49 kB 00:00 2024-04-21T08:31:23.100 INFO:teuthology.orchestra.run.smithi016.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3154.ga 247 kB/s | 16 kB 00:00 2024-04-21T08:31:23.108 INFO:teuthology.orchestra.run.smithi077.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-21T08:31:23.125 INFO:teuthology.orchestra.run.smithi077.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 383 kB/s | 35 kB 00:00 2024-04-21T08:31:23.225 INFO:teuthology.orchestra.run.smithi016.stdout:(35/150): ceph-mgr-diskprediction-local-19.0.0- 9.9 MB/s | 7.4 MB 00:00 2024-04-21T08:31:23.250 INFO:teuthology.orchestra.run.smithi016.stdout:(36/150): ceph-volume-19.0.0-3154.ga630765c.el9 1.8 MB/s | 273 kB 00:00 2024-04-21T08:31:23.317 INFO:teuthology.orchestra.run.smithi016.stdout:(37/150): cephfs-top-19.0.0-3154.ga630765c.el9. 273 kB/s | 18 kB 00:00 2024-04-21T08:31:23.351 INFO:teuthology.orchestra.run.smithi016.stdout:(38/150): cephadm-19.0.0-3154.ga630765c.el9.noa 5.9 MB/s | 762 kB 00:00 2024-04-21T08:31:23.543 INFO:teuthology.orchestra.run.smithi016.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 357 kB/s | 80 kB 00:00 2024-04-21T08:31:23.627 INFO:teuthology.orchestra.run.smithi016.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-21T08:31:23.669 INFO:teuthology.orchestra.run.smithi016.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-04-21T08:31:23.711 INFO:teuthology.orchestra.run.smithi016.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-04-21T08:31:23.761 INFO:teuthology.orchestra.run.smithi016.stdout:(43/150): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.2 MB 00:00 2024-04-21T08:31:23.786 INFO:teuthology.orchestra.run.smithi016.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-21T08:31:23.828 INFO:teuthology.orchestra.run.smithi016.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-21T08:31:23.854 INFO:teuthology.orchestra.run.smithi016.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-21T08:31:23.895 INFO:teuthology.orchestra.run.smithi016.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-04-21T08:31:23.915 INFO:teuthology.orchestra.run.smithi007.stdout:(128/150): python3-kubernetes-26.1.0-2.el9.noar 789 kB/s | 1.0 MB 00:01 2024-04-21T08:31:23.999 INFO:teuthology.orchestra.run.smithi007.stdout:(129/150): python3-grpcio-1.46.7-10.el9.x86_64. 857 kB/s | 2.0 MB 00:02 2024-04-21T08:31:24.024 INFO:teuthology.orchestra.run.smithi007.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-04-21T08:31:24.174 INFO:teuthology.orchestra.run.smithi007.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 450 kB/s | 79 kB 00:00 2024-04-21T08:31:24.208 INFO:teuthology.orchestra.run.smithi007.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 314 kB/s | 58 kB 00:00 2024-04-21T08:31:24.308 INFO:teuthology.orchestra.run.smithi007.stdout:(133/150): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-21T08:31:24.492 INFO:teuthology.orchestra.run.smithi007.stdout:(134/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 490 kB/s | 90 kB 00:00 2024-04-21T08:31:24.559 INFO:teuthology.orchestra.run.smithi007.stdout:(135/150): libarrow-9.0.0-10.el9.x86_64.rpm 838 kB/s | 4.4 MB 00:05 2024-04-21T08:31:24.601 INFO:teuthology.orchestra.run.smithi007.stdout:(136/150): python3-pecan-1.4.2-3.el9.noarch.rpm 638 kB/s | 272 kB 00:00 2024-04-21T08:31:24.626 INFO:teuthology.orchestra.run.smithi007.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 210 kB/s | 28 kB 00:00 2024-04-21T08:31:24.660 INFO:teuthology.orchestra.run.smithi007.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 305 kB/s | 31 kB 00:00 2024-04-21T08:31:24.760 INFO:teuthology.orchestra.run.smithi007.stdout:(139/150): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-04-21T08:31:24.792 INFO:teuthology.orchestra.run.smithi077.stdout:(66/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 2.5 MB/s | 4.1 MB 00:01 2024-04-21T08:31:24.810 INFO:teuthology.orchestra.run.smithi007.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-04-21T08:31:24.861 INFO:teuthology.orchestra.run.smithi007.stdout:(141/150): python3-routes-2.5.1-5.el9.noarch.rp 727 kB/s | 188 kB 00:00 2024-04-21T08:31:24.886 INFO:teuthology.orchestra.run.smithi007.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 406 kB/s | 51 kB 00:00 2024-04-21T08:31:25.004 INFO:teuthology.orchestra.run.smithi016.stdout:(48/150): ceph-test-19.0.0-3154.ga630765c.el9.x 11 MB/s | 49 MB 00:04 2024-04-21T08:31:25.044 INFO:teuthology.orchestra.run.smithi007.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 488 kB/s | 90 kB 00:00 2024-04-21T08:31:25.153 INFO:teuthology.orchestra.run.smithi007.stdout:(144/150): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-04-21T08:31:25.195 INFO:teuthology.orchestra.run.smithi007.stdout:(145/150): python3-webob-1.8.7-6.el9.noarch.rpm 597 kB/s | 230 kB 00:00 2024-04-21T08:31:25.412 INFO:teuthology.orchestra.run.smithi007.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 741 kB/s | 192 kB 00:00 2024-04-21T08:31:25.421 INFO:teuthology.orchestra.run.smithi016.stdout:(49/150): automake-1.16.2-8.el9.noarch.rpm 440 kB/s | 689 kB 00:01 2024-04-21T08:31:25.442 INFO:teuthology.orchestra.run.smithi077.stdout:(67/150): ceph-test-19.0.0-3154.ga630765c.el9.x 5.0 MB/s | 49 MB 00:09 2024-04-21T08:31:25.521 INFO:teuthology.orchestra.run.smithi007.stdout:(147/150): python3-werkzeug-2.0.3-3.el9.1.noarc 673 kB/s | 427 kB 00:00 2024-04-21T08:31:25.780 INFO:teuthology.orchestra.run.smithi016.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 138 kB/s | 107 kB 00:00 2024-04-21T08:31:25.937 INFO:teuthology.orchestra.run.smithi007.stdout:(148/150): librados2-19.0.0-3154.ga630765c.el9. 6.5 MB/s | 3.4 MB 00:00 2024-04-21T08:31:25.971 INFO:teuthology.orchestra.run.smithi016.stdout:(51/150): bison-3.7.4-5.el9.x86_64.rpm 454 kB/s | 943 kB 00:02 2024-04-21T08:31:25.996 INFO:teuthology.orchestra.run.smithi007.stdout:(149/150): librbd1-19.0.0-3154.ga630765c.el9.x8 6.6 MB/s | 3.1 MB 00:00 2024-04-21T08:31:26.055 INFO:teuthology.orchestra.run.smithi016.stdout:(52/150): elfutils-libelf-devel-0.190-2.el9.x86 34 kB/s | 21 kB 00:00 2024-04-21T08:31:26.184 INFO:teuthology.orchestra.run.smithi077.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 1.2 MB/s | 911 kB 00:00 2024-04-21T08:31:26.293 INFO:teuthology.orchestra.run.smithi077.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-21T08:31:26.614 INFO:teuthology.orchestra.run.smithi016.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 51 kB/s | 33 kB 00:00 2024-04-21T08:31:26.814 INFO:teuthology.orchestra.run.smithi016.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 303 kB/s | 313 kB 00:01 2024-04-21T08:31:27.043 INFO:teuthology.orchestra.run.smithi077.stdout:(70/150): postgresql-13.14-1.el9.x86_64.rpm 722 kB/s | 1.6 MB 00:02 2024-04-21T08:31:27.139 INFO:teuthology.orchestra.run.smithi016.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 35 kB/s | 18 kB 00:00 2024-04-21T08:31:27.151 INFO:teuthology.orchestra.run.smithi077.stdout:(71/150): openblas-openmp-0.3.21-2.el9.x86_64.r 1.2 MB/s | 4.7 MB 00:04 2024-04-21T08:31:27.171 INFO:teuthology.orchestra.run.smithi007.stdout:(150/150): thrift-0.15.0-2.el9.x86_64.rpm 820 kB/s | 1.6 MB 00:01 2024-04-21T08:31:27.178 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:31:27.178 INFO:teuthology.orchestra.run.smithi007.stdout:Total 11 MB/s | 218 MB 00:19 2024-04-21T08:31:27.323 INFO:teuthology.orchestra.run.smithi016.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 22 kB/s | 11 kB 00:00 2024-04-21T08:31:27.335 INFO:teuthology.orchestra.run.smithi077.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 477 kB/s | 87 kB 00:00 2024-04-21T08:31:27.460 INFO:teuthology.orchestra.run.smithi077.stdout:(73/150): postgresql-server-13.14-1.el9.x86_64. 5.0 MB/s | 5.8 MB 00:01 2024-04-21T08:31:27.802 INFO:teuthology.orchestra.run.smithi077.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 715 kB/s | 244 kB 00:00 2024-04-21T08:31:27.939 INFO:teuthology.orchestra.run.smithi016.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 73 kB/s | 45 kB 00:00 2024-04-21T08:31:27.965 INFO:teuthology.orchestra.run.smithi016.stdout:(58/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 194 kB/s | 160 kB 00:00 2024-04-21T08:31:28.144 INFO:teuthology.orchestra.run.smithi077.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 725 kB/s | 248 kB 00:00 2024-04-21T08:31:28.244 INFO:teuthology.orchestra.run.smithi077.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-04-21T08:31:28.257 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-04-21T08:31:28.345 INFO:teuthology.orchestra.run.smithi077.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 191 kB/s | 19 kB 00:00 2024-04-21T08:31:28.399 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-04-21T08:31:28.399 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-04-21T08:31:28.503 INFO:teuthology.orchestra.run.smithi077.stdout:(78/150): protobuf-3.14.0-13.el9.x86_64.rpm 707 kB/s | 1.0 MB 00:01 2024-04-21T08:31:28.612 INFO:teuthology.orchestra.run.smithi077.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 645 kB/s | 172 kB 00:00 2024-04-21T08:31:28.637 INFO:teuthology.orchestra.run.smithi077.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 260 kB/s | 35 kB 00:00 2024-04-21T08:31:28.996 INFO:teuthology.orchestra.run.smithi077.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 699 kB/s | 250 kB 00:00 2024-04-21T08:31:29.315 INFO:teuthology.orchestra.run.smithi016.stdout:(59/150): libtool-2.4.6-45.el9.x86_64.rpm 431 kB/s | 582 kB 00:01 2024-04-21T08:31:29.338 INFO:teuthology.orchestra.run.smithi077.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 650 kB/s | 222 kB 00:00 2024-04-21T08:31:29.415 INFO:teuthology.orchestra.run.smithi016.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 449 kB/s | 662 kB 00:01 2024-04-21T08:31:29.522 INFO:teuthology.orchestra.run.smithi077.stdout:(83/150): python3-packaging-20.9-5.el9.noarch.r 421 kB/s | 77 kB 00:00 2024-04-21T08:31:29.630 INFO:teuthology.orchestra.run.smithi077.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 390 kB/s | 42 kB 00:00 2024-04-21T08:31:29.941 INFO:teuthology.orchestra.run.smithi016.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 41 kB/s | 25 kB 00:00 2024-04-21T08:31:30.032 INFO:teuthology.orchestra.run.smithi016.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 77 kB/s | 47 kB 00:00 2024-04-21T08:31:30.047 INFO:teuthology.orchestra.run.smithi077.stdout:(85/150): python3-protobuf-3.14.0-13.el9.noarch 646 kB/s | 269 kB 00:00 2024-04-21T08:31:30.229 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-04-21T08:31:30.230 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-04-21T08:31:30.322 INFO:teuthology.orchestra.run.smithi077.stdout:(86/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 577 kB/s | 159 kB 00:00 2024-04-21T08:31:30.641 INFO:teuthology.orchestra.run.smithi016.stdout:(63/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 672 kB/s | 3.0 MB 00:04 2024-04-21T08:31:30.756 INFO:teuthology.orchestra.run.smithi077.stdout:(87/150): python3-pyasn1-modules-0.4.8-6.el9.no 643 kB/s | 279 kB 00:00 2024-04-21T08:31:30.865 INFO:teuthology.orchestra.run.smithi077.stdout:(88/150): python3-pytz-2021.1-5.el9.noarch.rpm 471 kB/s | 51 kB 00:00 2024-04-21T08:31:30.891 INFO:teuthology.orchestra.run.smithi016.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 219 kB/s | 188 kB 00:00 2024-04-21T08:31:30.958 INFO:teuthology.orchestra.run.smithi016.stdout:(65/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 287 kB/s | 292 kB 00:01 2024-04-21T08:31:30.982 INFO:teuthology.orchestra.run.smithi077.stdout:(89/150): python3-requests-oauthlib-1.3.0-12.el 461 kB/s | 54 kB 00:00 2024-04-21T08:31:31.326 INFO:teuthology.orchestra.run.smithi016.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 51 kB/s | 35 kB 00:00 2024-04-21T08:31:34.103 INFO:teuthology.orchestra.run.smithi016.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 586 kB/s | 1.6 MB 00:02 2024-04-21T08:31:35.532 INFO:teuthology.orchestra.run.smithi077.stdout:(90/150): python3-babel-2.9.1-2.el9.noarch.rpm 745 kB/s | 6.0 MB 00:08 2024-04-21T08:31:35.598 INFO:teuthology.orchestra.run.smithi077.stdout:(91/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 747 kB/s | 5.1 MB 00:06 2024-04-21T08:31:35.632 INFO:teuthology.orchestra.run.smithi077.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 417 kB/s | 42 kB 00:00 2024-04-21T08:31:35.699 INFO:teuthology.orchestra.run.smithi077.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 441 kB/s | 44 kB 00:00 2024-04-21T08:31:35.791 INFO:teuthology.orchestra.run.smithi077.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 391 kB/s | 36 kB 00:00 2024-04-21T08:31:35.853 INFO:teuthology.orchestra.run.smithi016.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 521 kB/s | 911 kB 00:01 2024-04-21T08:31:35.883 INFO:teuthology.orchestra.run.smithi077.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 517 kB/s | 47 kB 00:00 2024-04-21T08:31:35.975 INFO:teuthology.orchestra.run.smithi077.stdout:(96/150): qatlib-24.02.0-1.el9.x86_64.rpm 644 kB/s | 221 kB 00:00 2024-04-21T08:31:36.067 INFO:teuthology.orchestra.run.smithi077.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 632 kB/s | 58 kB 00:00 2024-04-21T08:31:36.259 INFO:teuthology.orchestra.run.smithi077.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 488 kB/s | 94 kB 00:00 2024-04-21T08:31:36.317 INFO:teuthology.orchestra.run.smithi077.stdout:(99/150): socat-1.7.4.1-5.el9.x86_64.rpm 702 kB/s | 305 kB 00:00 2024-04-21T08:31:36.347 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-04-21T08:31:36.418 INFO:teuthology.orchestra.run.smithi077.stdout:(100/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 448 kB/s | 45 kB 00:00 2024-04-21T08:31:36.424 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:31:36.443 INFO:teuthology.orchestra.run.smithi077.stdout:(101/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 346 kB/s | 64 kB 00:00 2024-04-21T08:31:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:31:36.502 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:31:36.568 INFO:teuthology.orchestra.run.smithi077.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 148 kB/s | 22 kB 00:00 2024-04-21T08:31:36.644 INFO:teuthology.orchestra.run.smithi016.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 168 kB/s | 133 kB 00:00 2024-04-21T08:31:36.735 INFO:teuthology.orchestra.run.smithi077.stdout:(103/150): protobuf-compiler-3.14.0-13.el9.x86_ 2.9 MB/s | 863 kB 00:00 2024-04-21T08:31:36.836 INFO:teuthology.orchestra.run.smithi077.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 2.0 MB/s | 548 kB 00:00 2024-04-21T08:31:36.847 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:31:36.895 INFO:teuthology.orchestra.run.smithi077.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 334 kB/s | 19 kB 00:00 2024-04-21T08:31:36.928 INFO:teuthology.orchestra.run.smithi077.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-04-21T08:31:36.958 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:31:36.970 INFO:teuthology.orchestra.run.smithi077.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-04-21T08:31:37.012 INFO:teuthology.orchestra.run.smithi077.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-21T08:31:37.045 INFO:teuthology.orchestra.run.smithi016.stdout:(70/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 695 kB/s | 4.1 MB 00:06 2024-04-21T08:31:37.054 INFO:teuthology.orchestra.run.smithi077.stdout:(109/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-21T08:31:37.083 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:31:37.109 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:31:37.112 INFO:teuthology.orchestra.run.smithi077.stdout:(110/150): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-21T08:31:37.202 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:31:37.229 INFO:teuthology.orchestra.run.smithi077.stdout:(111/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-04-21T08:31:37.244 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:31:37.272 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:31:37.298 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:31:37.305 INFO:teuthology.orchestra.run.smithi077.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-04-21T08:31:37.346 INFO:teuthology.orchestra.run.smithi077.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-04-21T08:31:37.389 INFO:teuthology.orchestra.run.smithi077.stdout:(114/150): python3-cachetools-4.2.4-1.el9.noarc 772 kB/s | 32 kB 00:00 2024-04-21T08:31:37.431 INFO:teuthology.orchestra.run.smithi077.stdout:(115/150): python3-certifi-2021.10.8-2.el9.noar 360 kB/s | 15 kB 00:00 2024-04-21T08:31:37.461 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:31:37.512 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:31:37.514 INFO:teuthology.orchestra.run.smithi077.stdout:(116/150): libarrow-9.0.0-10.el9.x86_64.rpm 7.1 MB/s | 4.4 MB 00:00 2024-04-21T08:31:37.539 INFO:teuthology.orchestra.run.smithi077.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-21T08:31:37.540 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:31:37.581 INFO:teuthology.orchestra.run.smithi077.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-04-21T08:31:37.624 INFO:teuthology.orchestra.run.smithi077.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-04-21T08:31:37.645 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:31:37.648 INFO:teuthology.orchestra.run.smithi077.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 528 kB/s | 35 kB 00:00 2024-04-21T08:31:37.674 INFO:teuthology.orchestra.run.smithi077.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 4.0 MB/s | 203 kB 00:00 2024-04-21T08:31:37.674 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:31:37.716 INFO:teuthology.orchestra.run.smithi077.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-04-21T08:31:37.728 INFO:teuthology.orchestra.run.smithi016.stdout:(71/150): openblas-openmp-0.3.21-2.el9.x86_64.r 713 kB/s | 4.7 MB 00:06 2024-04-21T08:31:37.737 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:31:37.758 INFO:teuthology.orchestra.run.smithi077.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 258 kB/s | 11 kB 00:00 2024-04-21T08:31:37.759 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:31:37.781 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:31:37.816 INFO:teuthology.orchestra.run.smithi077.stdout:(124/150): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-04-21T08:31:37.842 INFO:teuthology.orchestra.run.smithi077.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-04-21T08:31:37.867 INFO:teuthology.orchestra.run.smithi077.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-21T08:31:37.893 INFO:teuthology.orchestra.run.smithi077.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-04-21T08:31:37.918 INFO:teuthology.orchestra.run.smithi077.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-04-21T08:31:37.960 INFO:teuthology.orchestra.run.smithi077.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-04-21T08:31:37.968 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:31:37.993 INFO:teuthology.orchestra.run.smithi077.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-04-21T08:31:38.000 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:31:38.046 INFO:teuthology.orchestra.run.smithi077.stdout:(131/150): python3-kubernetes-26.1.0-2.el9.noar 6.7 MB/s | 1.0 MB 00:00 2024-04-21T08:31:38.071 INFO:teuthology.orchestra.run.smithi077.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 747 kB/s | 58 kB 00:00 2024-04-21T08:31:38.096 INFO:teuthology.orchestra.run.smithi077.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-21T08:31:38.112 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:31:38.121 INFO:teuthology.orchestra.run.smithi077.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-04-21T08:31:38.147 INFO:teuthology.orchestra.run.smithi077.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-04-21T08:31:38.172 INFO:teuthology.orchestra.run.smithi077.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-04-21T08:31:38.197 INFO:teuthology.orchestra.run.smithi077.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-04-21T08:31:38.209 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:31:38.231 INFO:teuthology.orchestra.run.smithi077.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-04-21T08:31:38.249 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:31:38.256 INFO:teuthology.orchestra.run.smithi077.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-21T08:31:38.282 INFO:teuthology.orchestra.run.smithi077.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-04-21T08:31:38.285 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:31:38.307 INFO:teuthology.orchestra.run.smithi077.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-21T08:31:38.332 INFO:teuthology.orchestra.run.smithi077.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-21T08:31:38.362 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:31:38.366 INFO:teuthology.orchestra.run.smithi077.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-21T08:31:38.391 INFO:teuthology.orchestra.run.smithi077.stdout:(144/150): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-04-21T08:31:38.401 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:31:38.416 INFO:teuthology.orchestra.run.smithi077.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-04-21T08:31:38.437 INFO:teuthology.orchestra.run.smithi016.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 124 kB/s | 87 kB 00:00 2024-04-21T08:31:38.441 INFO:teuthology.orchestra.run.smithi077.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-21T08:31:38.445 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:31:38.486 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:31:38.513 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:31:38.567 INFO:teuthology.orchestra.run.smithi077.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-21T08:31:38.586 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:31:38.646 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:31:38.675 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:31:38.719 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:31:38.798 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:31:38.829 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:31:38.881 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:31:38.911 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:31:38.984 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:31:39.004 INFO:teuthology.orchestra.run.smithi016.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 528 kB/s | 1.0 MB 00:01 2024-04-21T08:31:39.034 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:31:39.111 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:31:39.260 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:31:39.313 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:31:39.834 INFO:teuthology.orchestra.run.smithi077.stdout:(148/150): librados2-19.0.0-3154.ga630765c.el9. 2.4 MB/s | 3.4 MB 00:01 2024-04-21T08:31:39.880 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:31:39.946 INFO:teuthology.orchestra.run.smithi016.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 259 kB/s | 244 kB 00:00 2024-04-21T08:31:39.967 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:31:40.002 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:31:40.045 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:31:40.083 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:31:40.116 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:31:40.182 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:31:40.184 INFO:teuthology.orchestra.run.smithi077.stdout:(149/150): librbd1-19.0.0-3154.ga630765c.el9.x8 1.9 MB/s | 3.1 MB 00:01 2024-04-21T08:31:40.256 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:31:40.301 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:31:40.337 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:31:40.385 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:31:40.420 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:31:40.464 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:31:40.506 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:31:40.652 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:31:40.694 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:31:40.724 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:31:40.766 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:31:40.829 INFO:teuthology.orchestra.run.smithi016.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 281 kB/s | 248 kB 00:00 2024-04-21T08:31:40.867 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:31:40.899 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:31:40.940 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:31:40.965 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:31:40.989 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:31:41.069 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:31:41.199 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:31:41.239 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:31:41.267 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:31:41.318 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:31:41.363 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:31:41.401 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:31:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 43 kB/s | 26 kB 00:00 2024-04-21T08:31:41.440 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:31:41.470 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:31:41.504 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:31:41.537 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:31:41.579 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:31:41.613 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:31:41.706 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:31:41.762 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:31:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:31:41.963 INFO:teuthology.orchestra.run.smithi016.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 35 kB/s | 19 kB 00:00 2024-04-21T08:31:42.027 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:31:42.371 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:31:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:31:42.830 INFO:teuthology.orchestra.run.smithi016.stdout:(78/150): python3-mako-1.1.4-6.el9.noarch.rpm 199 kB/s | 172 kB 00:00 2024-04-21T08:31:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:31:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:31:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:31:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:31:43.438 INFO:teuthology.orchestra.run.smithi016.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 57 kB/s | 35 kB 00:00 2024-04-21T08:31:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:31:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:31:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:31:43.770 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:31:43.799 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:31:43.880 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:31:44.580 INFO:teuthology.orchestra.run.smithi016.stdout:(80/150): postgresql-server-13.14-1.el9.x86_64. 746 kB/s | 5.8 MB 00:07 2024-04-21T08:31:44.582 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:31:44.704 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:31:45.531 INFO:teuthology.orchestra.run.smithi016.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 264 kB/s | 250 kB 00:00 2024-04-21T08:31:46.044 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:31:46.155 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:31:46.183 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:31:46.216 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi016.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 252 kB/s | 222 kB 00:00 2024-04-21T08:31:46.546 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:31:46.595 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:31:46.689 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:31:46.717 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:31:46.753 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:31:46.881 INFO:teuthology.orchestra.run.smithi016.stdout:(83/150): python3-babel-2.9.1-2.el9.noarch.rpm 723 kB/s | 6.0 MB 00:08 2024-04-21T08:31:47.165 INFO:teuthology.orchestra.run.smithi016.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 103 kB/s | 77 kB 00:00 2024-04-21T08:31:47.260 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:31:47.290 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:31:47.362 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:31:47.391 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:31:47.515 INFO:teuthology.orchestra.run.smithi016.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 67 kB/s | 42 kB 00:00 2024-04-21T08:31:48.117 INFO:teuthology.orchestra.run.smithi016.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 283 kB/s | 269 kB 00:00 2024-04-21T08:31:48.390 INFO:teuthology.orchestra.run.smithi016.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 182 kB/s | 159 kB 00:00 2024-04-21T08:31:49.132 INFO:teuthology.orchestra.run.smithi016.stdout:(88/150): python3-pytz-2021.1-5.el9.noarch.rpm 69 kB/s | 51 kB 00:00 2024-04-21T08:31:49.157 INFO:teuthology.orchestra.run.smithi016.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 268 kB/s | 279 kB 00:01 2024-04-21T08:31:49.774 INFO:teuthology.orchestra.run.smithi016.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 84 kB/s | 54 kB 00:00 2024-04-21T08:31:49.787 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:31:49.822 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:31:50.391 INFO:teuthology.orchestra.run.smithi016.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 68 kB/s | 42 kB 00:00 2024-04-21T08:31:50.392 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:31:50.442 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:31:50.516 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:31:50.516 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:31:50.516 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:31:50.516 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:31:50.543 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:31:50.583 INFO:teuthology.orchestra.run.smithi016.stdout:(92/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 731 kB/s | 5.1 MB 00:07 2024-04-21T08:31:51.183 INFO:teuthology.orchestra.run.smithi016.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 56 kB/s | 44 kB 00:00 2024-04-21T08:31:51.676 INFO:teuthology.orchestra.run.smithi016.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 202 kB/s | 221 kB 00:01 2024-04-21T08:31:51.801 INFO:teuthology.orchestra.run.smithi016.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 58 kB/s | 36 kB 00:00 2024-04-21T08:31:52.326 INFO:teuthology.orchestra.run.smithi016.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 73 kB/s | 47 kB 00:00 2024-04-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi077.stdout:(150/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 764 kB/s | 16 MB 00:21 2024-04-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi077.stdout:Total 5.1 MB/s | 218 MB 00:42 2024-04-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi016.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 313 kB/s | 305 kB 00:00 2024-04-21T08:31:52.985 INFO:teuthology.orchestra.run.smithi016.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 88 kB/s | 58 kB 00:00 2024-04-21T08:31:53.527 INFO:teuthology.orchestra.run.smithi016.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 125 kB/s | 94 kB 00:00 2024-04-21T08:31:53.677 INFO:teuthology.orchestra.run.smithi016.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 92 kB/s | 64 kB 00:00 2024-04-21T08:31:53.838 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-04-21T08:31:53.854 INFO:teuthology.orchestra.run.smithi016.stdout:(101/150): lua-devel-5.4.4-4.el9.x86_64.rpm 128 kB/s | 22 kB 00:00 2024-04-21T08:31:53.964 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-04-21T08:31:53.964 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-04-21T08:31:53.994 INFO:teuthology.orchestra.run.smithi016.stdout:(102/150): protobuf-compiler-3.14.0-13.el9.x86_ 5.9 MB/s | 863 kB 00:00 2024-04-21T08:31:54.186 INFO:teuthology.orchestra.run.smithi016.stdout:(103/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 68 kB/s | 45 kB 00:00 2024-04-21T08:31:54.303 INFO:teuthology.orchestra.run.smithi016.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-04-21T08:31:54.328 INFO:teuthology.orchestra.run.smithi016.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 2.1 MB/s | 309 kB 00:00 2024-04-21T08:31:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-04-21T08:31:54.462 INFO:teuthology.orchestra.run.smithi016.stdout:(107/150): libarrow-9.0.0-10.el9.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2024-04-21T08:31:54.488 INFO:teuthology.orchestra.run.smithi016.stdout:(108/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-04-21T08:31:54.513 INFO:teuthology.orchestra.run.smithi016.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-04-21T08:31:54.538 INFO:teuthology.orchestra.run.smithi016.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-21T08:31:54.580 INFO:teuthology.orchestra.run.smithi016.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-21T08:31:54.622 INFO:teuthology.orchestra.run.smithi016.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-04-21T08:31:54.656 INFO:teuthology.orchestra.run.smithi016.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 7.3 MB/s | 548 kB 00:00 2024-04-21T08:31:54.681 INFO:teuthology.orchestra.run.smithi016.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 757 kB/s | 43 kB 00:00 2024-04-21T08:31:54.731 INFO:teuthology.orchestra.run.smithi016.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-04-21T08:31:54.748 INFO:teuthology.orchestra.run.smithi016.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-04-21T08:31:54.799 INFO:teuthology.orchestra.run.smithi016.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-21T08:31:54.840 INFO:teuthology.orchestra.run.smithi016.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-04-21T08:31:54.933 INFO:teuthology.orchestra.run.smithi016.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-04-21T08:31:54.974 INFO:teuthology.orchestra.run.smithi016.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 264 kB/s | 35 kB 00:00 2024-04-21T08:31:55.000 INFO:teuthology.orchestra.run.smithi016.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-04-21T08:31:55.067 INFO:teuthology.orchestra.run.smithi016.stdout:(122/150): python3-grpcio-1.46.7-10.el9.x86_64. 22 MB/s | 2.0 MB 00:00 2024-04-21T08:31:55.092 INFO:teuthology.orchestra.run.smithi016.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-21T08:31:55.117 INFO:teuthology.orchestra.run.smithi016.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 215 kB/s | 11 kB 00:00 2024-04-21T08:31:55.176 INFO:teuthology.orchestra.run.smithi016.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-04-21T08:31:55.201 INFO:teuthology.orchestra.run.smithi016.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 276 kB/s | 23 kB 00:00 2024-04-21T08:31:55.227 INFO:teuthology.orchestra.run.smithi016.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-04-21T08:31:55.261 INFO:teuthology.orchestra.run.smithi016.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-04-21T08:31:55.344 INFO:teuthology.orchestra.run.smithi016.stdout:(129/150): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-04-21T08:31:55.369 INFO:teuthology.orchestra.run.smithi016.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-04-21T08:31:55.395 INFO:teuthology.orchestra.run.smithi016.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-21T08:31:55.420 INFO:teuthology.orchestra.run.smithi016.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-21T08:31:55.445 INFO:teuthology.orchestra.run.smithi016.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-21T08:31:55.470 INFO:teuthology.orchestra.run.smithi016.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-04-21T08:31:55.513 INFO:teuthology.orchestra.run.smithi016.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-21T08:31:55.538 INFO:teuthology.orchestra.run.smithi016.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 421 kB/s | 28 kB 00:00 2024-04-21T08:31:55.563 INFO:teuthology.orchestra.run.smithi016.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-04-21T08:31:55.588 INFO:teuthology.orchestra.run.smithi016.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-21T08:31:55.614 INFO:teuthology.orchestra.run.smithi016.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-21T08:31:55.639 INFO:teuthology.orchestra.run.smithi016.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-04-21T08:31:55.664 INFO:teuthology.orchestra.run.smithi016.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-21T08:31:55.671 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-04-21T08:31:55.671 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-04-21T08:31:55.689 INFO:teuthology.orchestra.run.smithi016.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-21T08:31:55.732 INFO:teuthology.orchestra.run.smithi016.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-21T08:31:55.757 INFO:teuthology.orchestra.run.smithi016.stdout:(144/150): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-04-21T08:31:55.799 INFO:teuthology.orchestra.run.smithi016.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 303 kB/s | 20 kB 00:00 2024-04-21T08:31:55.841 INFO:teuthology.orchestra.run.smithi016.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-21T08:31:55.883 INFO:teuthology.orchestra.run.smithi016.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-04-21T08:31:56.341 INFO:teuthology.orchestra.run.smithi016.stdout:(148/150): librados2-19.0.0-3154.ga630765c.el9. 6.8 MB/s | 3.4 MB 00:00 2024-04-21T08:31:56.442 INFO:teuthology.orchestra.run.smithi016.stdout:(149/150): librbd1-19.0.0-3154.ga630765c.el9.x8 5.6 MB/s | 3.1 MB 00:00 2024-04-21T08:32:01.410 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-04-21T08:32:01.478 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:32:01.503 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:32:01.546 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:32:01.951 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:32:01.996 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:32:02.117 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:32:02.147 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:02.222 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:02.264 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:32:02.292 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:32:02.318 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:32:02.415 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:32:02.415 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2024-04-21T08:32:02.415 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2024-04-21T08:32:02.415 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2024-04-21T08:32:02.415 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2024-04-21T08:32:02.415 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2024-04-21T08:32:02.415 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /root 2024-04-21T08:32:02.415 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2024-04-21T08:32:02.415 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:02.471 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:02.610 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:32:02.627 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:32:02.627 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:02.627 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:32:02.628 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:02.648 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:02.670 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:32:02.733 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:32:02.754 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:32:02.779 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:32:02.979 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:32:03.012 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:32:03.146 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:32:03.240 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:32:03.278 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:32:03.313 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:32:03.386 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:32:03.429 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:32:03.480 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:32:03.513 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:32:03.541 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:32:03.611 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:32:03.681 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:32:03.711 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:32:03.755 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:32:03.831 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:32:03.874 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:32:03.917 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:32:03.945 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:32:04.018 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:32:04.060 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:32:04.137 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:32:04.169 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:32:04.196 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:32:04.196 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:04.196 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:32:04.197 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:32:04.197 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:32:04.197 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:04.280 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:32:04.330 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:32:04.436 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:32:04.460 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:32:04.460 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:04.460 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:32:04.460 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:32:04.460 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:32:04.461 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:04.875 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:32:04.929 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:32:04.953 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:32:04.953 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:04.953 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:32:04.953 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:32:04.953 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:32:04.953 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:04.963 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:32:04.996 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:32:05.039 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:32:05.060 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:32:05.077 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:32:05.119 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:32:05.119 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:32:05.175 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:32:05.182 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:32:05.248 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:32:05.278 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:32:05.315 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:32:05.362 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:32:05.385 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:32:05.397 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:32:05.439 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:32:05.441 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:32:05.483 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:32:05.520 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:32:05.611 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:32:05.620 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:32:05.664 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:32:05.694 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:32:05.735 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:32:05.762 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:32:05.806 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:32:05.832 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:32:05.837 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:32:05.865 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:32:05.868 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:32:05.909 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:32:05.934 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:32:05.959 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:32:06.026 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:32:06.155 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:32:06.191 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:32:06.219 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:32:06.270 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:32:06.275 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:32:06.298 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:32:06.323 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:32:06.330 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:32:06.361 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:32:06.394 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:32:06.401 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:32:06.420 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:32:06.430 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:32:06.465 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:32:06.498 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:32:06.540 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:32:06.566 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:32:06.646 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:32:06.697 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:32:06.728 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:32:06.776 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:32:06.884 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:32:06.938 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:32:07.517 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:32:07.556 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:32:07.675 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:32:07.719 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:32:07.752 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:32:07.854 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:32:07.902 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:32:07.992 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:32:08.040 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:32:08.223 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:32:08.260 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:32:08.338 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:32:08.571 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:32:08.598 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:32:08.705 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:32:08.877 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:32:08.909 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:32:08.933 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:32:08.933 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:08.933 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:32:08.933 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:32:08.933 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:32:08.934 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:08.984 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:32:08.995 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:32:08.996 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:32:09.097 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:32:09.120 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:32:09.150 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:32:10.125 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:32:10.154 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:32:10.178 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:32:10.178 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:10.178 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:32:10.178 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:32:10.178 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:32:10.178 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:10.221 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:32:10.244 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:32:10.244 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:10.244 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:32:10.244 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:10.288 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:32:10.312 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:32:10.312 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:10.312 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:32:10.312 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:32:10.312 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:32:10.312 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:10.340 INFO:teuthology.orchestra.run.smithi016.stdout:(150/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 786 kB/s | 16 MB 00:21 2024-04-21T08:32:10.347 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:32:10.348 INFO:teuthology.orchestra.run.smithi016.stdout:Total 4.1 MB/s | 218 MB 00:53 2024-04-21T08:32:10.368 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:32:10.449 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:32:10.477 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:32:10.510 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:32:10.647 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:32:10.671 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:32:10.671 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:10.672 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:32:10.672 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:32:10.672 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:32:10.672 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:10.824 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:32:10.865 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:32:10.939 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:32:10.969 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:32:11.006 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:32:11.450 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-21T08:32:11.489 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:32:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:32:11.589 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-21T08:32:11.589 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-21T08:32:11.589 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:32:11.619 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:32:13.415 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-21T08:32:13.416 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-21T08:32:13.871 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:13.908 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:14.449 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:14.486 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:32:14.576 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:32:14.576 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:32:14.576 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:32:14.576 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:14.604 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:32:15.518 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:32:15.560 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:32:15.626 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:32:15.657 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:32:15.685 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:32:15.881 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:32:15.931 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:32:15.961 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:32:16.597 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:32:16.634 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:32:16.654 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:32:17.516 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:32:17.605 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:32:17.647 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:32:17.732 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:32:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:32:17.770 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:32:17.854 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:32:17.854 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:32:19.457 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-21T08:32:19.536 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:32:19.560 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:32:19.604 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:32:19.948 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:32:20.069 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:32:20.194 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:32:20.228 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:20.321 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:32:20.391 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:32:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:32:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:32:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:32:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:20.818 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:32:20.881 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:32:20.919 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:32:20.939 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:32:21.112 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:32:21.144 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:32:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:32:21.380 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:32:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:32:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:32:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:32:21.570 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:32:21.614 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:32:21.654 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:32:21.690 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:32:21.764 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:32:21.847 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:32:21.876 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:32:21.920 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:32:22.001 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:32:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:32:22.188 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:32:22.243 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:32:22.323 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:32:22.478 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:32:22.533 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:32:23.132 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:32:23.239 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:32:23.287 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:32:23.330 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:32:23.369 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:32:23.401 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:32:23.460 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:32:23.535 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:32:23.577 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:32:23.623 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:32:23.672 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:32:23.706 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:32:23.749 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:32:23.800 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:32:23.880 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:32:23.880 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:23.881 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:32:23.882 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:32:23.883 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:32:23.883 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:32:23.894 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:32:23.894 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:32:23.894 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:32:23.894 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:32:23.894 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:32:23.894 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:32:23.894 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:32:23.894 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:32:23.895 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:32:23.896 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:32:23.897 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:32:23.898 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:32:23.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:32:23.900 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:32:23.901 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:32:23.902 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:32:23.903 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:32:23.903 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:32:23.903 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:32:23.903 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:32:23.948 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:32:23.988 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:32:24.018 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:32:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:32:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:32:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:32:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:32:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:32:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:32:24.389 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:32:24.519 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:32:24.565 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:32:24.593 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:32:24.628 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:32:24.674 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:32:24.711 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:32:24.750 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:32:24.779 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:32:24.814 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:32:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:32:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:32:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:32:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:32:25.072 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:32:25.102 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:32:25.151 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:32:25.264 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:32:25.322 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:32:25.916 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:32:25.916 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-04-21T08:32:25.916 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-04-21T08:32:25.916 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-04-21T08:32:25.916 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-04-21T08:32:25.916 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-04-21T08:32:25.917 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /root 2024-04-21T08:32:25.917 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-04-21T08:32:25.917 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:26.092 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:32:26.097 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.098 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.099 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:32:26.100 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:32:26.101 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:32:26.102 INFO:teuthology.orchestra.run.smithi007.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:32:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:32:26.104 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:32:26.105 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:32:26.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:32:26.107 INFO:teuthology.orchestra.run.smithi007.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:32:26.108 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:32:26.109 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:32:26.109 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:26.109 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:32:26.109 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:32:26.109 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:32:26.109 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-04-21T08:32:26.128 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:32:26.176 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:32:26.210 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:32:26.320 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:32:26.378 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:32:26.475 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:32:26.486 DEBUG:teuthology.parallel:result is None 2024-04-21T08:32:26.520 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:32:26.715 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:32:26.751 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:32:26.831 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:32:27.543 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:32:27.612 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:32:27.637 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:32:27.638 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:27.638 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:32:27.638 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:32:27.638 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:32:27.638 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:27.673 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:32:27.864 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:32:27.887 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:32:27.888 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:27.888 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:32:27.888 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:32:27.888 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:32:27.888 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:28.340 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:32:28.363 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:32:28.363 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:28.363 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:32:28.363 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:32:28.363 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:32:28.364 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:28.470 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:32:28.527 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:32:28.582 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:32:28.766 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:32:28.824 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:32:28.893 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:32:28.982 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:32:29.032 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:32:29.111 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:32:29.122 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:32:29.150 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:32:29.157 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:32:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:32:29.190 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:32:29.217 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:32:29.512 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:32:29.554 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:32:29.637 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:32:29.645 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:32:29.674 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:32:29.675 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:32:29.707 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:32:29.712 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:32:29.768 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:32:29.798 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:32:30.217 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:32:30.249 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:32:30.320 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:32:30.359 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:32:30.826 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:32:30.866 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:32:31.823 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:32:31.850 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:32:31.952 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:32:32.139 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:32:32.185 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:32:32.208 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:32:32.208 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:32.208 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:32:32.208 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:32:32.209 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:32:32.209 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:32.262 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:32:32.273 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:32:32.379 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:32:32.435 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:32:32.693 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:32.730 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:33.304 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:33.350 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:32:33.356 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:32:33.389 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:32:33.412 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:32:33.412 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:33.412 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:32:33.412 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:32:33.412 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:32:33.412 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:33.446 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:32:33.446 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:32:33.446 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:32:33.447 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:32:33.456 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:32:33.478 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:32:33.478 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:33.478 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:32:33.478 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:33.493 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:32:33.524 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:32:33.546 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:32:33.546 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:33.546 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:32:33.546 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:32:33.547 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:32:33.547 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:33.858 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:32:33.880 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:32:33.880 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:33.881 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:32:33.881 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:32:33.881 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:32:33.881 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:38.470 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:32:38.511 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:32:38.552 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:32:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:32:38.612 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:32:38.795 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:32:38.841 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:32:38.871 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:32:39.464 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:32:39.502 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:32:39.522 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:32:40.296 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:32:40.391 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:32:40.432 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:32:40.521 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:32:40.553 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:32:40.553 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:32:40.645 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:32:40.646 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:32:45.543 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:32:45.544 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-04-21T08:32:45.544 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-04-21T08:32:45.544 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-04-21T08:32:45.544 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-04-21T08:32:45.544 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-04-21T08:32:45.544 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-04-21T08:32:45.544 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-04-21T08:32:45.544 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:32:45.720 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:32:45.736 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:32:45.737 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:45.737 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:32:45.737 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:32:46.678 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:32:46.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:32:46.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:32:46.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:32:46.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:32:46.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:32:46.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:46.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:32:46.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:32:46.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:32:46.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:32:46.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:32:46.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:32:46.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:32:46.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:32:46.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:32:46.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:32:46.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:32:46.692 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:32:46.692 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:32:46.692 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:32:46.692 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:32:46.693 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:32:46.694 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:32:46.695 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:32:46.696 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:32:46.697 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:32:46.698 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:32:46.699 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:32:46.700 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:32:46.701 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:32:47.291 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:32:47.318 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:32:47.318 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:47.319 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:32:47.319 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:32:47.319 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:32:47.319 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:32:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:32:47.574 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:32:47.574 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:47.574 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:32:47.574 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:32:47.574 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:32:47.574 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:32:48.043 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:32:48.066 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:32:48.066 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:48.067 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:32:48.067 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:32:48.067 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:32:48.067 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:32:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:32:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:32:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:32:48.499 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:32:48.552 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:32:48.642 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:32:48.740 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:32:48.746 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:32:48.746 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:48.746 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-04-21T08:32:48.746 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.746 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.746 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-04-21T08:32:48.746 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.747 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:48.748 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:48.748 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:48.748 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:48.748 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:48.748 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.748 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.748 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:48.748 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.748 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.748 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.748 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:48.749 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:48.749 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.749 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.749 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:32:48.749 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:32:48.749 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:32:48.749 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:32:48.749 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:32:48.749 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:32:48.750 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:32:48.750 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:32:48.750 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:32:48.750 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:32:48.750 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:32:48.750 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:32:48.750 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.750 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.750 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.750 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:32:48.750 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:32:48.751 INFO:teuthology.orchestra.run.smithi077.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:32:48.752 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:32:48.753 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:32:48.754 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:32:48.754 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:32:48.754 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:32:48.754 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:32:48.754 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:32:48.754 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:32:48.754 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:32:48.754 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:32:48.754 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:32:48.754 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:32:48.754 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:32:48.755 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.756 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:32:48.757 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:32:48.758 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:32:48.759 INFO:teuthology.orchestra.run.smithi077.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:32:48.759 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:32:48.759 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:32:48.759 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:32:48.759 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:32:48.759 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-04-21T08:32:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:32:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:32:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:32:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:32:49.061 DEBUG:teuthology.parallel:result is None 2024-04-21T08:32:49.364 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:32:49.394 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:32:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:32:49.490 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:32:49.525 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:32:50.628 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:32:50.669 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:32:51.686 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:32:51.719 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:32:51.827 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:32:51.997 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:32:52.029 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:32:52.054 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:32:52.054 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:52.054 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:32:52.054 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:32:52.054 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:32:52.055 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:32:52.105 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:32:52.117 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:32:52.218 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:32:52.272 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:32:53.249 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:32:53.292 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:32:53.316 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:32:53.316 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:53.316 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:32:53.316 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:32:53.316 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:32:53.316 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:32:53.359 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:32:53.382 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:32:53.383 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:53.383 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:32:53.383 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:32:53.434 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:32:53.458 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:32:53.459 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:53.459 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:32:53.459 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:32:53.459 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:32:53.459 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:32:53.777 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:32:53.800 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:32:53.801 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:53.801 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:32:53.801 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:32:53.801 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:32:53.801 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:32:58.665 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:32:58.707 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:32:58.748 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:32:58.779 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:32:58.806 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:32:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:32:59.060 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:32:59.090 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:32:59.732 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:32:59.771 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:32:59.792 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:33:00.656 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:33:00.776 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:33:00.818 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:33:00.912 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:33:00.947 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:33:00.947 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:33:01.039 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:33:01.039 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:33:07.275 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:33:07.275 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:33:07.275 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:33:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:33:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:33:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:33:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:33:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:33:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:33:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:33:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:33:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:33:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:33:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:33:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:33:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:33:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:33:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:33:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:33:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:33:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:33:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:33:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:33:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:33:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:33:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:33:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:33:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:33:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:33:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:33:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:33:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:33:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:33:07.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:33:07.295 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:33:07.296 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:33:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:33:07.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:33:07.299 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:33:07.299 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:33:07.299 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:33:07.299 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:33:07.299 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:33:07.299 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:33:07.299 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:33:07.299 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:33:09.496 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:33:09.496 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:33:09.496 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-04-21T08:33:09.496 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.496 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.496 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-21T08:33:09.496 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:33:09.496 INFO:teuthology.orchestra.run.smithi016.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:09.497 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:09.498 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:09.498 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:09.498 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.498 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.498 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:09.498 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.498 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.498 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.498 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:09.498 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:09.498 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:33:09.499 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:33:09.500 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:33:09.501 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:33:09.502 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:33:09.503 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:33:09.504 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:33:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:33:09.507 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:33:09.508 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:33:09.508 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-21T08:33:09.508 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-21T08:33:09.839 DEBUG:teuthology.parallel:result is None 2024-04-21T08:33:09.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:09.980 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:33:10.007 INFO:teuthology.orchestra.run.smithi007.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:33:10.008 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:33:10.008 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:33:10.010 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:10.225 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:33:10.247 INFO:teuthology.orchestra.run.smithi016.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:33:10.247 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:33:10.247 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:33:10.249 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:10.395 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:33:10.420 INFO:teuthology.orchestra.run.smithi077.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:33:10.421 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:33:10.421 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:33:10.423 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-21T08:33:10.423 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:33:10.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:33:10.456 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-21T08:33:10.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:33:10.482 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-21T08:33:10.482 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:33:10.513 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-21T08:33:10.513 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:33:10.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:33:10.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:33:10.607 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-21T08:33:10.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:33:10.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:33:10.696 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-21T08:33:10.696 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:33:10.722 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:33:10.786 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-21T08:33:10.786 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:33:10.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:33:10.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:33:10.882 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-21T08:33:10.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:33:10.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:33:10.971 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-21T08:33:10.972 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:33:10.996 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:33:11.064 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-21T08:33:11.064 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:33:11.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:33:11.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:33:11.153 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-21T08:33:11.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:33:11.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:33:11.243 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-21T08:33:11.243 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:33:11.267 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:33:11.331 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-21T08:33:11.428 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-21T08:33:11.428 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:11.429 INFO:tasks.cephadm:Cluster fsid is c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:33:11.429 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-21T08:33:11.429 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.7', 'mon.b': '172.21.15.16', 'mon.c': '172.21.15.77'} 2024-04-21T08:33:11.429 INFO:tasks.cephadm:First mon is mon.a on smithi007 2024-04-21T08:33:11.430 INFO:tasks.cephadm:First mgr is x 2024-04-21T08:33:11.430 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-21T08:33:11.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo hostname $(hostname -s) 2024-04-21T08:33:11.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-04-21T08:33:11.480 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2024-04-21T08:33:11.504 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-21T08:33:11.504 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:11.718 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78740/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3154-ga630765c', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3154.ga630765c'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-19 11:13:53.331234', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-21T08:33:11.928 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 a630765c1f93c6e61d9b40c4b76437611d5ef29e from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:11.929 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm 2024-04-21T08:33:11.930 DEBUG:teuthology.orchestra.run.smithi007:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:12.382 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:33 /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:12.382 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:12.856 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:33 /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:12.856 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:13.356 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:33 /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:13.356 DEBUG:teuthology.orchestra.run.smithi007:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:13.376 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:13.396 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:13.430 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e on all hosts... 2024-04-21T08:33:13.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:33:13.434 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:33:13.439 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:33:15.006 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:33:15.022 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:33:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:33:44.289 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-21T08:33:44.290 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:33:44.290 INFO:teuthology.orchestra.run.smithi077.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:33:44.290 INFO:teuthology.orchestra.run.smithi077.stdout: "repo_digests": [ 2024-04-21T08:33:44.290 INFO:teuthology.orchestra.run.smithi077.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:33:44.290 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-04-21T08:33:44.290 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-21T08:33:44.926 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:33:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:33:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:33:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "repo_digests": [ 2024-04-21T08:33:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:33:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:33:44.927 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:33:54.444 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-21T08:33:54.445 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:33:54.445 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:33:54.445 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-04-21T08:33:54.445 INFO:teuthology.orchestra.run.smithi016.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:33:54.445 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-21T08:33:54.445 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-21T08:33:54.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2024-04-21T08:33:54.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-04-21T08:33:54.528 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2024-04-21T08:33:54.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /etc/ceph 2024-04-21T08:33:54.589 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-04-21T08:33:54.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2024-04-21T08:33:54.641 INFO:tasks.cephadm:Writing seed config... 2024-04-21T08:33:54.642 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-21T08:33:54.642 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-21T08:33:54.642 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-21T08:33:54.642 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-21T08:33:54.642 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-21T08:33:54.642 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-21T08:33:54.642 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-21T08:33:54.643 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-21T08:33:54.644 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-21T08:33:54.645 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-21T08:33:54.645 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-21T08:33:54.645 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-21T08:33:54.645 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-21T08:33:54.645 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-21T08:33:54.645 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-21T08:33:54.645 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-21T08:33:54.645 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-21T08:33:54.645 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-21T08:33:54.645 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-21T08:33:54.646 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:33:54.646 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-21T08:33:54.664 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c9c60282-ffb9-11ee-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-21T08:33:54.665 DEBUG:teuthology.orchestra.run.smithi007:mon.a> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.a.service 2024-04-21T08:33:54.708 DEBUG:teuthology.orchestra.run.smithi007:mgr.x> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.x.service 2024-04-21T08:33:54.751 INFO:tasks.cephadm:Bootstrapping... 2024-04-21T08:33:54.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e -v bootstrap --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:33:54.951 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:33:54.951 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e', '-v', 'bootstrap', '--fsid', 'c9c60282-ffb9-11ee-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.7', '--skip-admin-label'] 2024-04-21T08:33:54.951 INFO:teuthology.orchestra.run.smithi007.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-21T08:33:54.952 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying podman|docker is present... 2024-04-21T08:33:54.978 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T08:33:54.978 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying lvm2 is present... 2024-04-21T08:33:54.978 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying time synchronization is in place... 2024-04-21T08:33:54.986 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T08:33:54.986 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T08:33:54.992 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T08:33:54.992 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-04-21T08:33:54.998 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2024-04-21T08:33:55.005 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2024-04-21T08:33:55.005 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2024-04-21T08:33:55.005 INFO:teuthology.orchestra.run.smithi007.stdout:Repeating the final host check... 2024-04-21T08:33:55.029 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T08:33:55.029 INFO:teuthology.orchestra.run.smithi007.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-21T08:33:55.030 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl is present 2024-04-21T08:33:55.030 INFO:teuthology.orchestra.run.smithi007.stdout:lvcreate is present 2024-04-21T08:33:55.036 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T08:33:55.036 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T08:33:55.042 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T08:33:55.042 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-04-21T08:33:55.048 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2024-04-21T08:33:55.055 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2024-04-21T08:33:55.055 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2024-04-21T08:33:55.055 INFO:teuthology.orchestra.run.smithi007.stdout:Host looks OK 2024-04-21T08:33:55.055 INFO:teuthology.orchestra.run.smithi007.stdout:Cluster fsid: c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:33:55.055 INFO:teuthology.orchestra.run.smithi007.stdout:Acquiring lock 139877102828704 on /run/cephadm/c9c60282-ffb9-11ee-bc93-c7b262605968.lock 2024-04-21T08:33:55.055 INFO:teuthology.orchestra.run.smithi007.stdout:Lock 139877102828704 acquired on /run/cephadm/c9c60282-ffb9-11ee-bc93-c7b262605968.lock 2024-04-21T08:33:55.056 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 3300 ... 2024-04-21T08:33:55.056 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 6789 ... 2024-04-21T08:33:55.056 INFO:teuthology.orchestra.run.smithi007.stdout:Base mon IP(s) is [172.21.15.7:3300, 172.21.15.7:6789], mon addrv is [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-04-21T08:33:55.059 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.7 metric 100 2024-04-21T08:33:55.059 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.7 metric 100 2024-04-21T08:33:55.062 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-21T08:33:55.062 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-21T08:33:55.062 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-21T08:33:55.064 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-21T08:33:55.064 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-21T08:33:55.064 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T08:33:55.065 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 2: ens1f0: mtu 1500 state UP qlen 1000 2024-04-21T08:33:55.065 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15ea/64 scope link noprefixroute 2024-04-21T08:33:55.065 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T08:33:55.065 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-04-21T08:33:55.065 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-04-21T08:33:55.065 INFO:teuthology.orchestra.run.smithi007.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-21T08:33:55.066 INFO:teuthology.orchestra.run.smithi007.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-21T08:33:55.067 INFO:teuthology.orchestra.run.smithi007.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:33:55.552 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 2024-04-21T08:33:55.553 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:33:55.553 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Getting image source signatures 2024-04-21T08:33:55.553 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:7eab8a2da10d06ac88aea1fd8998a2559b4de9d07c585e3b1e6ee6c4d35fc119 2024-04-21T08:33:55.553 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-21T08:33:55.553 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying config sha256:b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 2024-04-21T08:33:55.553 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-21T08:33:56.383 INFO:teuthology.orchestra.run.smithi007.stdout:ceph: stdout ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev) 2024-04-21T08:33:56.383 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph version: ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev) 2024-04-21T08:33:56.383 INFO:teuthology.orchestra.run.smithi007.stdout:Extracting ceph user uid/gid from container image... 2024-04-21T08:33:57.339 INFO:teuthology.orchestra.run.smithi007.stdout:stat: stdout 167 167 2024-04-21T08:33:57.339 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial keys... 2024-04-21T08:33:58.328 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQB1zyRmybA6LBAAoxnEvzlEYAjuKJ25j+jelw== 2024-04-21T08:34:06.615 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQB3zyRmsGVtLRAA7Nn0UqxSYZgBNG0wNqXC2Q== 2024-04-21T08:34:11.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQCBzyRmQOlAHRAAMfbQeWMYo7hCcxpn91VVdA== 2024-04-21T08:34:11.174 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial monmap... 2024-04-21T08:34:12.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T08:34:12.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-21T08:34:12.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:12.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T08:34:12.582 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool for a [v2:172.21.15.7:3300,v1:172.21.15.7:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T08:34:12.582 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = quincy 2024-04-21T08:34:12.582 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: set fsid to c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:12.583 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T08:34:12.583 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:34:12.583 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mon... 2024-04-21T08:34:13.479 INFO:teuthology.orchestra.run.smithi007.stdout:create mon.a on 2024-04-21T08:34:13.697 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-21T08:34:13.911 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-21T08:34:14.089 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c9c60282-ffb9-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-c9c60282-ffb9-11ee-bc93-c7b262605968.target. 2024-04-21T08:34:14.089 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c9c60282-ffb9-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-c9c60282-ffb9-11ee-bc93-c7b262605968.target. 2024-04-21T08:34:14.330 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.a 2024-04-21T08:34:14.331 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.a.service: Unit ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.a.service not loaded. 2024-04-21T08:34:14.503 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c9c60282-ffb9-11ee-bc93-c7b262605968.target.wants/ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@.service. 2024-04-21T08:34:15.028 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:34:15.028 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-04-21T08:34:15.035 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:34:15.035 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-04-21T08:34:15.035 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-04-21T08:34:15.035 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T08:34:15.035 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon to start... 2024-04-21T08:34:15.036 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon... 2024-04-21T08:34:16.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:16 smithi007 ceph-mon[28104]: from='client.? 172.21.15.7:0/286404582' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-21T08:34:17.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout cluster: 2024-04-21T08:34:17.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout id: c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout services: 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.564098s) 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout data: 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pgs: 2024-04-21T08:34:17.175 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.468+0000 7f451f98c700 1 Processor -- start 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.468+0000 7f451f98c700 1 -- start start 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.468+0000 7f451f98c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451810b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.468+0000 7f451f98c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f451807be60 con 0x7f4518108df0 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.469+0000 7f451d728700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451810b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.469+0000 7f451d728700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451810b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35986/0 (socket says 172.21.15.7:35986) 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.469+0000 7f451d728700 1 -- 172.21.15.7:0/2634632841 learned_addr learned my addr 172.21.15.7:0/2634632841 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.469+0000 7f451d728700 1 -- 172.21.15.7:0/2634632841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f451810b720 con 0x7f4518108df0 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.469+0000 7f451d728700 1 --2- 172.21.15.7:0/2634632841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451810b1e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f45080098a0 tx=0x7f4508009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e700756a059df4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.469+0000 7f450ffff700 1 -- 172.21.15.7:0/2634632841 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4508004160 con 0x7f4518108df0 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.470+0000 7f450ffff700 1 -- 172.21.15.7:0/2634632841 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4508004300 con 0x7f4518108df0 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.470+0000 7f450ffff700 1 -- 172.21.15.7:0/2634632841 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45080036a0 con 0x7f4518108df0 2024-04-21T08:34:17.176 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.470+0000 7f451f98c700 1 -- 172.21.15.7:0/2634632841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 msgr2=0x7f451810b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.470+0000 7f451f98c700 1 --2- 172.21.15.7:0/2634632841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451810b1e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f45080098a0 tx=0x7f4508009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.470+0000 7f451f98c700 1 -- 172.21.15.7:0/2634632841 shutdown_connections 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.470+0000 7f451f98c700 1 --2- 172.21.15.7:0/2634632841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451810b1e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.470+0000 7f451f98c700 1 -- 172.21.15.7:0/2634632841 >> 172.21.15.7:0/2634632841 conn(0x7f4518102820 msgr2=0x7f4518104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.470+0000 7f451f98c700 1 -- 172.21.15.7:0/2634632841 shutdown_connections 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.470+0000 7f451f98c700 1 -- 172.21.15.7:0/2634632841 wait complete. 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.471+0000 7f451f98c700 1 Processor -- start 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.471+0000 7f451f98c700 1 -- start start 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.471+0000 7f451f98c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451819db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.471+0000 7f451f98c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f451810bb90 con 0x7f4518108df0 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.471+0000 7f451d728700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451819db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.471+0000 7f451d728700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451819db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35990/0 (socket says 172.21.15.7:35990) 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.471+0000 7f451d728700 1 -- 172.21.15.7:0/286404582 learned_addr learned my addr 172.21.15.7:0/286404582 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:17.177 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.471+0000 7f451d728700 1 -- 172.21.15.7:0/286404582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f451819e0c0 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.472+0000 7f451d728700 1 --2- 172.21.15.7:0/286404582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451819db80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4508000c00 tx=0x7f4508004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.472+0000 7f450e7fc700 1 -- 172.21.15.7:0/286404582 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4508003db0 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.472+0000 7f450e7fc700 1 -- 172.21.15.7:0/286404582 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4508011630 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.472+0000 7f451f98c700 1 -- 172.21.15.7:0/286404582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f451819e350 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.472+0000 7f450e7fc700 1 -- 172.21.15.7:0/286404582 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4508011950 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.472+0000 7f451f98c700 1 -- 172.21.15.7:0/286404582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45181a1040 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.472+0000 7f450e7fc700 1 -- 172.21.15.7:0/286404582 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4508011af0 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.472+0000 7f450e7fc700 1 -- 172.21.15.7:0/286404582 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f450801c450 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.473+0000 7f451f98c700 1 -- 172.21.15.7:0/286404582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45180640a0 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.475+0000 7f450e7fc700 1 -- 172.21.15.7:0/286404582 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4508004260 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.548+0000 7f451f98c700 1 -- 172.21.15.7:0/286404582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f45181a1530 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.548+0000 7f450e7fc700 1 -- 172.21.15.7:0/286404582 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f450801b450 con 0x7f4518108df0 2024-04-21T08:34:17.178 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.549+0000 7f451f98c700 1 -- 172.21.15.7:0/286404582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 msgr2=0x7f451819db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:17.179 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.549+0000 7f451f98c700 1 --2- 172.21.15.7:0/286404582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451819db80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4508000c00 tx=0x7f4508004750 comp rx=0 tx=0).stop 2024-04-21T08:34:17.179 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.549+0000 7f451f98c700 1 -- 172.21.15.7:0/286404582 shutdown_connections 2024-04-21T08:34:17.179 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.549+0000 7f451f98c700 1 --2- 172.21.15.7:0/286404582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4518108df0 0x7f451819db80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:17.179 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.549+0000 7f451f98c700 1 -- 172.21.15.7:0/286404582 >> 172.21.15.7:0/286404582 conn(0x7f4518102820 msgr2=0x7f4518103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:17.179 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.549+0000 7f451f98c700 1 -- 172.21.15.7:0/286404582 shutdown_connections 2024-04-21T08:34:17.179 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:15.549+0000 7f451f98c700 1 -- 172.21.15.7:0/286404582 wait complete. 2024-04-21T08:34:17.179 INFO:teuthology.orchestra.run.smithi007.stdout:mon is available 2024-04-21T08:34:17.179 INFO:teuthology.orchestra.run.smithi007.stdout:Assimilating anything we can from ceph.conf... 2024-04-21T08:34:17.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:17 smithi007 ceph-mon[28104]: from='client.? 172.21.15.7:0/2006860101' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T08:34:17.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:17 smithi007 ceph-mon[28104]: from='client.? 172.21.15.7:0/2006860101' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-21T08:34:18.273 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T08:34:18.274 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.584+0000 7f117e25d700 1 Processor -- start 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.584+0000 7f117e25d700 1 -- start start 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.584+0000 7f117e25d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178108770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.584+0000 7f117e25d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1178108d40 con 0x7f1178108350 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.585+0000 7f11777fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.585+0000 7f11777fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36006/0 (socket says 172.21.15.7:36006) 2024-04-21T08:34:18.275 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.585+0000 7f11777fe700 1 -- 172.21.15.7:0/901113717 learned_addr learned my addr 172.21.15.7:0/901113717 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.585+0000 7f11777fe700 1 -- 172.21.15.7:0/901113717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1178109550 con 0x7f1178108350 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.585+0000 7f11777fe700 1 --2- 172.21.15.7:0/901113717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178108770 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1168009e80 tx=0x7f116800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace49d1b767d3afc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.585+0000 7f11767fc700 1 -- 172.21.15.7:0/901113717 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1168009200 con 0x7f1178108350 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.585+0000 7f11767fc700 1 -- 172.21.15.7:0/901113717 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1168003a20 con 0x7f1178108350 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.585+0000 7f11767fc700 1 -- 172.21.15.7:0/901113717 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1168003d40 con 0x7f1178108350 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.586+0000 7f117e25d700 1 -- 172.21.15.7:0/901113717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 msgr2=0x7f1178108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.586+0000 7f117e25d700 1 --2- 172.21.15.7:0/901113717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178108770 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1168009e80 tx=0x7f116800f740 comp rx=0 tx=0).stop 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.586+0000 7f117e25d700 1 -- 172.21.15.7:0/901113717 shutdown_connections 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.586+0000 7f117e25d700 1 --2- 172.21.15.7:0/901113717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178108770 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.586+0000 7f117e25d700 1 -- 172.21.15.7:0/901113717 >> 172.21.15.7:0/901113717 conn(0x7f1178103930 msgr2=0x7f1178105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.586+0000 7f117e25d700 1 -- 172.21.15.7:0/901113717 shutdown_connections 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.586+0000 7f117e25d700 1 -- 172.21.15.7:0/901113717 wait complete. 2024-04-21T08:34:18.276 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.587+0000 7f117e25d700 1 Processor -- start 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.587+0000 7f117e25d700 1 -- start start 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.587+0000 7f117e25d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.587+0000 7f117e25d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1178109960 con 0x7f1178108350 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.587+0000 7f11777fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.587+0000 7f11777fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36020/0 (socket says 172.21.15.7:36020) 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.587+0000 7f11777fe700 1 -- 172.21.15.7:0/2006860101 learned_addr learned my addr 172.21.15.7:0/2006860101 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.588+0000 7f11777fe700 1 -- 172.21.15.7:0/2006860101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1178199c80 con 0x7f1178108350 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.588+0000 7f11777fe700 1 --2- 172.21.15.7:0/2006860101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178199740 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1168019040 tx=0x7f1168004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.588+0000 7f1174ff9700 1 -- 172.21.15.7:0/2006860101 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1168027070 con 0x7f1178108350 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.588+0000 7f1174ff9700 1 -- 172.21.15.7:0/2006860101 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1168009200 con 0x7f1178108350 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.588+0000 7f117e25d700 1 -- 172.21.15.7:0/2006860101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1178199f10 con 0x7f1178108350 2024-04-21T08:34:18.277 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.588+0000 7f1174ff9700 1 -- 172.21.15.7:0/2006860101 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1168021580 con 0x7f1178108350 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.588+0000 7f117e25d700 1 -- 172.21.15.7:0/2006860101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f117819cc00 con 0x7f1178108350 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.588+0000 7f1174ff9700 1 -- 172.21.15.7:0/2006860101 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f116802a070 con 0x7f1178108350 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.588+0000 7f1174ff9700 1 -- 172.21.15.7:0/2006860101 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1168014070 con 0x7f1178108350 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.589+0000 7f117e25d700 1 -- 172.21.15.7:0/2006860101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11780640a0 con 0x7f1178108350 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.591+0000 7f1174ff9700 1 -- 172.21.15.7:0/2006860101 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1168031030 con 0x7f1178108350 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.662+0000 7f117e25d700 1 -- 172.21.15.7:0/2006860101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f117819cdd0 con 0x7f1178108350 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.665+0000 7f1174ff9700 1 -- 172.21.15.7:0/2006860101 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+463 (secure 0 0 0) 0x7f116802c7b0 con 0x7f1178108350 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.666+0000 7f1174ff9700 1 -- 172.21.15.7:0/2006860101 <== mon.0 v2:172.21.15.7:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1168021c00 con 0x7f1178108350 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.667+0000 7f117e25d700 1 -- 172.21.15.7:0/2006860101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 msgr2=0x7f1178199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.667+0000 7f117e25d700 1 --2- 172.21.15.7:0/2006860101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178199740 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1168019040 tx=0x7f1168004ad0 comp rx=0 tx=0).stop 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.667+0000 7f117e25d700 1 -- 172.21.15.7:0/2006860101 shutdown_connections 2024-04-21T08:34:18.278 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.667+0000 7f117e25d700 1 --2- 172.21.15.7:0/2006860101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1178108350 0x7f1178199740 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:18.279 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.667+0000 7f117e25d700 1 -- 172.21.15.7:0/2006860101 >> 172.21.15.7:0/2006860101 conn(0x7f1178103930 msgr2=0x7f1178105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:18.279 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.667+0000 7f117e25d700 1 -- 172.21.15.7:0/2006860101 shutdown_connections 2024-04-21T08:34:18.279 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:17.667+0000 7f117e25d700 1 -- 172.21.15.7:0/2006860101 wait complete. 2024-04-21T08:34:18.279 INFO:teuthology.orchestra.run.smithi007.stdout:Generating new minimal ceph.conf... 2024-04-21T08:34:18.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:18 smithi007 ceph-mon[28104]: from='client.? 172.21.15.7:0/3267995175' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:19.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.681+0000 7f7af5cd0700 1 Processor -- start 2024-04-21T08:34:19.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.681+0000 7f7af5cd0700 1 -- start start 2024-04-21T08:34:19.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.681+0000 7f7af5cd0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af01087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:19.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.681+0000 7f7af5cd0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7af0108da0 con 0x7f7af01083b0 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.681+0000 7f7aef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af01087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.681+0000 7f7aef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af01087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36030/0 (socket says 172.21.15.7:36030) 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.681+0000 7f7aef7fe700 1 -- 172.21.15.7:0/1496264940 learned_addr learned my addr 172.21.15.7:0/1496264940 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.682+0000 7f7aef7fe700 1 -- 172.21.15.7:0/1496264940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af0109560 con 0x7f7af01083b0 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.682+0000 7f7aef7fe700 1 --2- 172.21.15.7:0/1496264940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af01087d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00098a0 tx=0x7f7ae0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96e8e6aacd8b754 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.682+0000 7f7aee7fc700 1 -- 172.21.15.7:0/1496264940 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ae0004160 con 0x7f7af01083b0 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.682+0000 7f7aee7fc700 1 -- 172.21.15.7:0/1496264940 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7ae0004300 con 0x7f7af01083b0 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.682+0000 7f7aee7fc700 1 -- 172.21.15.7:0/1496264940 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ae0011af0 con 0x7f7af01083b0 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.682+0000 7f7af5cd0700 1 -- 172.21.15.7:0/1496264940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 msgr2=0x7f7af01087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.682+0000 7f7af5cd0700 1 --2- 172.21.15.7:0/1496264940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af01087d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00098a0 tx=0x7f7ae0009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.683+0000 7f7af5cd0700 1 -- 172.21.15.7:0/1496264940 shutdown_connections 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.683+0000 7f7af5cd0700 1 --2- 172.21.15.7:0/1496264940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af01087d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.683+0000 7f7af5cd0700 1 -- 172.21.15.7:0/1496264940 >> 172.21.15.7:0/1496264940 conn(0x7f7af01039b0 msgr2=0x7f7af0105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.683+0000 7f7af5cd0700 1 -- 172.21.15.7:0/1496264940 shutdown_connections 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.683+0000 7f7af5cd0700 1 -- 172.21.15.7:0/1496264940 wait complete. 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.683+0000 7f7af5cd0700 1 Processor -- start 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.683+0000 7f7af5cd0700 1 -- start start 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7af5cd0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7af5cd0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7af010c910 con 0x7f7af01083b0 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7aef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7aef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af019e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36042/0 (socket says 172.21.15.7:36042) 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7aef7fe700 1 -- 172.21.15.7:0/3267995175 learned_addr learned my addr 172.21.15.7:0/3267995175 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7aef7fe700 1 -- 172.21.15.7:0/3267995175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af019e940 con 0x7f7af01083b0 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7aef7fe700 1 --2- 172.21.15.7:0/3267995175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af019e400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0003f40 tx=0x7f7ae0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7aecff9700 1 -- 172.21.15.7:0/3267995175 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ae00038b0 con 0x7f7af01083b0 2024-04-21T08:34:19.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7aecff9700 1 -- 172.21.15.7:0/3267995175 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7ae0003a50 con 0x7f7af01083b0 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7af5cd0700 1 -- 172.21.15.7:0/3267995175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af019ebd0 con 0x7f7af01083b0 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7aecff9700 1 -- 172.21.15.7:0/3267995175 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ae001a5c0 con 0x7f7af01083b0 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.684+0000 7f7af5cd0700 1 -- 172.21.15.7:0/3267995175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af01a18c0 con 0x7f7af01083b0 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.685+0000 7f7aecff9700 1 -- 172.21.15.7:0/3267995175 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7ae001ac50 con 0x7f7af01083b0 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.685+0000 7f7aecff9700 1 -- 172.21.15.7:0/3267995175 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7ae0019a40 con 0x7f7af01083b0 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.685+0000 7f7af5cd0700 1 -- 172.21.15.7:0/3267995175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af00640a0 con 0x7f7af01083b0 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.688+0000 7f7aecff9700 1 -- 172.21.15.7:0/3267995175 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7ae0019ce0 con 0x7f7af01083b0 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.758+0000 7f7af5cd0700 1 -- 172.21.15.7:0/3267995175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f7af01a1e00 con 0x7f7af01083b0 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.759+0000 7f7aecff9700 1 -- 172.21.15.7:0/3267995175 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+173 (secure 0 0 0) 0x7f7ae0024430 con 0x7f7af01083b0 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.759+0000 7f7af5cd0700 1 -- 172.21.15.7:0/3267995175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 msgr2=0x7f7af019e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.759+0000 7f7af5cd0700 1 --2- 172.21.15.7:0/3267995175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af019e400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0003f40 tx=0x7f7ae0011890 comp rx=0 tx=0).stop 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.760+0000 7f7af5cd0700 1 -- 172.21.15.7:0/3267995175 shutdown_connections 2024-04-21T08:34:19.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.760+0000 7f7af5cd0700 1 --2- 172.21.15.7:0/3267995175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af01083b0 0x7f7af019e400 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:19.318 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.760+0000 7f7af5cd0700 1 -- 172.21.15.7:0/3267995175 >> 172.21.15.7:0/3267995175 conn(0x7f7af01039b0 msgr2=0x7f7af01043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:19.318 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.760+0000 7f7af5cd0700 1 -- 172.21.15.7:0/3267995175 shutdown_connections 2024-04-21T08:34:19.318 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:18.760+0000 7f7af5cd0700 1 -- 172.21.15.7:0/3267995175 wait complete. 2024-04-21T08:34:19.318 INFO:teuthology.orchestra.run.smithi007.stdout:Restarting the monitor... 2024-04-21T08:34:19.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:19 smithi007 systemd[1]: Stopping Ceph mon.a for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:34:19.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:19 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a[28100]: 2024-04-21T08:34:19.586+0000 7fa92d7e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:34:19.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:19 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a[28100]: 2024-04-21T08:34:19.586+0000 7fa92d7e4700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-21T08:34:19.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:19 smithi007 podman[28354]: 2024-04-21 08:34:19.791136986 +0000 UTC m=+0.311700952 container died 2c5fd0de04ce09df55030c7147617772229e75151bbcf5b52adf3a683bf2ea12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux ) 2024-04-21T08:34:20.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 podman[28354]: 2024-04-21 08:34:20.119304206 +0000 UTC m=+0.639868211 container cleanup 2c5fd0de04ce09df55030c7147617772229e75151bbcf5b52adf3a683bf2ea12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-21T08:34:20.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 bash[28354]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a 2024-04-21T08:34:20.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 podman[28368]: 2024-04-21 08:34:20.278350241 +0000 UTC m=+0.485817607 container remove 2c5fd0de04ce09df55030c7147617772229e75151bbcf5b52adf3a683bf2ea12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-21T08:34:20.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-21T08:34:20.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 systemd[1]: Stopped Ceph mon.a for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:34:20.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 systemd[1]: Starting Ceph mon.a for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:34:20.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 podman[28458]: 2024-04-21 08:34:20.630068201 +0000 UTC m=+0.018409174 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 podman[28458]: 2024-04-21 08:34:20.729354944 +0000 UTC m=+0.117695916 container create 1e1e16479767a4409873e9eeadf797279f7340849ee2935be063ebb7e9f567e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 podman[28458]: 2024-04-21 08:34:20.9071178 +0000 UTC m=+0.295458771 container init 1e1e16479767a4409873e9eeadf797279f7340849ee2935be063ebb7e9f567e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 podman[28458]: 2024-04-21 08:34:20.910007661 +0000 UTC m=+0.298348632 container start 1e1e16479767a4409873e9eeadf797279f7340849ee2935be063ebb7e9f567e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: set uid:gid to 167:167 (ceph:ceph) 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev), process ceph-mon, pid 2 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: pidfile_write: ignore empty --pid-file 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: load: jerasure load: lrc 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: RocksDB version: 7.9.2 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Git sha 0 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Compile date 2024-04-19 14:19:22 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: DB SUMMARY 2024-04-21T08:34:20.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: DB Session ID: M5WR4QFDSOV7CQ0TYMQZ 2024-04-21T08:34:20.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: CURRENT file: CURRENT 2024-04-21T08:34:20.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: IDENTITY file: IDENTITY 2024-04-21T08:34:20.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-21T08:34:20.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-21T08:34:20.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104502 ; 2024-04-21T08:34:20.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.error_if_exists: 0 2024-04-21T08:34:20.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.create_if_missing: 0 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.paranoid_checks: 1 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.env: 0x55687b698cc0 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.fs: PosixFileSystem 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.info_log: 0x55687d7fbbe0 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_file_opening_threads: 16 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.statistics: (nil) 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.use_fsync: 0 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_log_file_size: 0 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-21T08:34:20.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.keep_log_file_num: 1000 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.recycle_log_file_num: 0 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.allow_fallocate: 1 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.allow_mmap_reads: 0 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.allow_mmap_writes: 0 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.use_direct_reads: 0 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.create_missing_column_families: 0 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.db_log_dir: 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.wal_dir: 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-21T08:34:20.985 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.advise_random_on_open: 1 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.db_write_buffer_size: 0 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.write_buffer_manager: 0x55687d78f0e0 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.rate_limiter: (nil) 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.wal_recovery_mode: 2 2024-04-21T08:34:20.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.enable_thread_tracking: 0 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.enable_pipelined_write: 0 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.unordered_write: 0 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.row_cache: None 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.wal_filter: None 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.allow_ingest_behind: 0 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.two_write_queues: 0 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.manual_wal_flush: 0 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.wal_compression: 0 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.atomic_flush: 0 2024-04-21T08:34:20.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.log_readahead_size: 0 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.best_efforts_recovery: 0 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.allow_data_in_errors: 0 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.db_host_id: __hostname__ 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_background_jobs: 2 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_background_compactions: -1 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_subcompactions: 1 2024-04-21T08:34:20.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_total_wal_size: 0 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_open_files: -1 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bytes_per_sync: 0 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_readahead_size: 0 2024-04-21T08:34:20.989 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_background_flushes: -1 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Compression algorithms supported: 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: kZSTD supported: 0 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: kXpressCompression supported: 0 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: kLZ4HCCompression supported: 1 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: kZlibCompression supported: 1 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: kSnappyCompression supported: 1 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: kLZ4Compression supported: 1 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: kBZip2Compression supported: 0 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-21T08:34:20.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.merge_operator: 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_filter: None 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_filter_factory: None 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.sst_partitioner_factory: None 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55687d7fbdc0) 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-04-21T08:34:20.991 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_type: 0 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_index_type: 0 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_shortening: 1 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: checksum: 4 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: no_block_cache: 0 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache: 0x55687d7c11f0 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_options: 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: capacity : 536870912 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_shard_bits : 4 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: strict_capacity_limit : 0 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_compressed: (nil) 2024-04-21T08:34:20.992 INFO:journalctl@ceph.mon.a.smithi007.stdout: persistent_cache: (nil) 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size: 4096 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size_deviation: 10 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_restart_interval: 16 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_block_restart_interval: 1 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: metadata_block_size: 4096 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: partition_filters: 0 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: use_delta_encoding: 1 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: filter_policy: bloomfilter 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: whole_key_filtering: 1 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: verify_compression: 0 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: format_version: 5 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_index_compression: 1 2024-04-21T08:34:20.993 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_align: 0 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout: max_auto_readahead_size: 262144 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout: prepopulate_block_cache: 0 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout: initial_auto_readahead_size: 8192 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_file_reads_for_auto_readahead: 2 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.write_buffer_size: 33554432 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_write_buffer_number: 2 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compression: NoCompression 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bottommost_compression: Disabled 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.prefix_extractor: nullptr 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.num_levels: 7 2024-04-21T08:34:20.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compression_opts.level: 32767 2024-04-21T08:34:20.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compression_opts.strategy: 0 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compression_opts.enabled: false 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.target_file_size_base: 67108864 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-21T08:34:20.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-21T08:34:20.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-21T08:34:20.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-21T08:34:20.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-21T08:34:20.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-21T08:34:20.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-21T08:34:20.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-21T08:34:20.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-21T08:34:20.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-21T08:34:20.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-21T08:34:20.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-21T08:34:20.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.arena_block_size: 1048576 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.disable_auto_compactions: 0 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-21T08:34:20.998 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.inplace_update_support: 0 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.bloom_locality: 0 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.max_successive_merges: 0 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.paranoid_file_checks: 0 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.force_consistency_checks: 1 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.report_bg_io_stats: 0 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.ttl: 2592000 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-21T08:34:20.999 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.enable_blob_files: false 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.min_blob_size: 0 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.blob_file_size: 268435456 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.blob_file_starting_level: 0 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-21T08:34:21.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c39775ed-8e12-42e0-9c61-e1823faeadfc 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688460939219, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688460940913, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99729, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97611, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14028, "raw_average_key_size": 49, "raw_value_size": 89969, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713688460, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c39775ed-8e12-42e0-9c61-e1823faeadfc", "db_session_id": "M5WR4QFDSOV7CQ0TYMQZ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688460941026, "job": 1, "event": "recovery_finished"} 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55687d8d6000 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: DB pointer 0x55687d8c0000 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** DB Stats ** 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:34:21.001 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: L0 2/0 99.25 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 91.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: Sum 2/0 99.25 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 91.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:34:21.002 INFO:journalctl@ceph.mon.a.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 91.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 91.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T08:34:21.003 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative compaction: 0.00 GB write, 20.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval compaction: 0.00 GB write, 20.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache BinnedLRUCache@0x55687d7c11f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: starting mon.a rank 0 at public addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] at bind addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: mon.a@-1(???) e1 preinit fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: mon.a@-1(???).mds e1 new map 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: mon.a@-1(???).mds e1 print_map 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout: e1 2024-04-21T08:34:21.004 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:34:21.005 INFO:journalctl@ceph.mon.a.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:34:21.005 INFO:journalctl@ceph.mon.a.smithi007.stdout: legacy client fscid: -1 2024-04-21T08:34:21.005 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-21T08:34:21.005 INFO:journalctl@ceph.mon.a.smithi007.stdout: No filesystems configured 2024-04-21T08:34:21.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T08:34:21.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:34:21.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:34:21.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:34:21.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 ceph-mon[28477]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-21T08:34:21.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 bash[28458]: 1e1e16479767a4409873e9eeadf797279f7340849ee2935be063ebb7e9f567e6 2024-04-21T08:34:21.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:20 smithi007 systemd[1]: Started Ceph mon.a for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:34:21.006 INFO:teuthology.orchestra.run.smithi007.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-21T08:34:21.283 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:21 smithi007 ceph-mon[28477]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T08:34:21.284 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:21 smithi007 ceph-mon[28477]: monmap epoch 1 2024-04-21T08:34:21.284 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:21 smithi007 ceph-mon[28477]: fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:21.284 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:21 smithi007 ceph-mon[28477]: last_changed 2024-04-21T08:34:11.499844+0000 2024-04-21T08:34:21.284 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:21 smithi007 ceph-mon[28477]: created 2024-04-21T08:34:11.499844+0000 2024-04-21T08:34:21.284 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:21 smithi007 ceph-mon[28477]: min_mon_release 19 (squid) 2024-04-21T08:34:21.284 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:21 smithi007 ceph-mon[28477]: election_strategy: 1 2024-04-21T08:34:21.284 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:21 smithi007 ceph-mon[28477]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-21T08:34:21.284 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:21 smithi007 ceph-mon[28477]: fsmap 2024-04-21T08:34:21.284 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:21 smithi007 ceph-mon[28477]: osdmap e1: 0 total, 0 up, 0 in 2024-04-21T08:34:21.284 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:21 smithi007 ceph-mon[28477]: mgrmap e1: no daemons active 2024-04-21T08:34:22.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.389+0000 7f1378114700 1 Processor -- start 2024-04-21T08:34:22.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.390+0000 7f1378114700 1 -- start start 2024-04-21T08:34:22.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.390+0000 7f1378114700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f13701087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:22.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.390+0000 7f1378114700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1370108da0 con 0x7f13701083b0 2024-04-21T08:34:22.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.390+0000 7f1375eb0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f13701087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.390+0000 7f1375eb0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f13701087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36046/0 (socket says 172.21.15.7:36046) 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.390+0000 7f1375eb0700 1 -- 172.21.15.7:0/3526060858 learned_addr learned my addr 172.21.15.7:0/3526060858 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.391+0000 7f1375eb0700 1 -- 172.21.15.7:0/3526060858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1370109560 con 0x7f13701083b0 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.391+0000 7f1375eb0700 1 --2- 172.21.15.7:0/3526060858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f13701087d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f136c0098a0 tx=0x7f136c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4b65dd2eed2dbf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.391+0000 7f1374eae700 1 -- 172.21.15.7:0/3526060858 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f136c004160 con 0x7f13701083b0 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.391+0000 7f1374eae700 1 -- 172.21.15.7:0/3526060858 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f136c004300 con 0x7f13701083b0 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.391+0000 7f1374eae700 1 -- 172.21.15.7:0/3526060858 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f136c011af0 con 0x7f13701083b0 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.391+0000 7f1378114700 1 -- 172.21.15.7:0/3526060858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 msgr2=0x7f13701087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.391+0000 7f1378114700 1 --2- 172.21.15.7:0/3526060858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f13701087d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f136c0098a0 tx=0x7f136c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.391+0000 7f1378114700 1 -- 172.21.15.7:0/3526060858 shutdown_connections 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.391+0000 7f1378114700 1 --2- 172.21.15.7:0/3526060858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f13701087d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.391+0000 7f1378114700 1 -- 172.21.15.7:0/3526060858 >> 172.21.15.7:0/3526060858 conn(0x7f13701039b0 msgr2=0x7f1370105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:22.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.392+0000 7f1378114700 1 -- 172.21.15.7:0/3526060858 shutdown_connections 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.392+0000 7f1378114700 1 -- 172.21.15.7:0/3526060858 wait complete. 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.392+0000 7f1378114700 1 Processor -- start 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.392+0000 7f1378114700 1 -- start start 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.392+0000 7f1378114700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f137019c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.392+0000 7f1378114700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f137010c910 con 0x7f13701083b0 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.393+0000 7f1375eb0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f137019c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.393+0000 7f1375eb0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f137019c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36052/0 (socket says 172.21.15.7:36052) 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.393+0000 7f1375eb0700 1 -- 172.21.15.7:0/905313274 learned_addr learned my addr 172.21.15.7:0/905313274 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.393+0000 7f1375eb0700 1 -- 172.21.15.7:0/905313274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137019c790 con 0x7f13701083b0 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.393+0000 7f1375eb0700 1 --2- 172.21.15.7:0/905313274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f137019c250 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f136c003f40 tx=0x7f136c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.393+0000 7f1366ffd700 1 -- 172.21.15.7:0/905313274 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f136c0036a0 con 0x7f13701083b0 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.393+0000 7f1366ffd700 1 -- 172.21.15.7:0/905313274 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f136c024430 con 0x7f13701083b0 2024-04-21T08:34:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.393+0000 7f1366ffd700 1 -- 172.21.15.7:0/905313274 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f136c01a5c0 con 0x7f13701083b0 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.393+0000 7f1378114700 1 -- 172.21.15.7:0/905313274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f137019ca20 con 0x7f13701083b0 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.393+0000 7f1378114700 1 -- 172.21.15.7:0/905313274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f137019f710 con 0x7f13701083b0 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.393+0000 7f1366ffd700 1 -- 172.21.15.7:0/905313274 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f136c01ac50 con 0x7f13701083b0 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.394+0000 7f1366ffd700 1 -- 172.21.15.7:0/905313274 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f136c019700 con 0x7f13701083b0 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.394+0000 7f1378114700 1 -- 172.21.15.7:0/905313274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13700640a0 con 0x7f13701083b0 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.397+0000 7f1366ffd700 1 -- 172.21.15.7:0/905313274 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f136c042e50 con 0x7f13701083b0 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.469+0000 7f1378114700 1 -- 172.21.15.7:0/905313274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f137019f8e0 con 0x7f13701083b0 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.470+0000 7f1366ffd700 1 -- 172.21.15.7:0/905313274 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f136c031030 con 0x7f13701083b0 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.471+0000 7f1378114700 1 -- 172.21.15.7:0/905313274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 msgr2=0x7f137019c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.471+0000 7f1378114700 1 --2- 172.21.15.7:0/905313274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f137019c250 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f136c003f40 tx=0x7f136c004790 comp rx=0 tx=0).stop 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.471+0000 7f1378114700 1 -- 172.21.15.7:0/905313274 shutdown_connections 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.471+0000 7f1378114700 1 --2- 172.21.15.7:0/905313274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13701083b0 0x7f137019c250 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.471+0000 7f1378114700 1 -- 172.21.15.7:0/905313274 >> 172.21.15.7:0/905313274 conn(0x7f13701039b0 msgr2=0x7f13701044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:22.005 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.471+0000 7f1378114700 1 -- 172.21.15.7:0/905313274 shutdown_connections 2024-04-21T08:34:22.006 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:21.471+0000 7f1378114700 1 -- 172.21.15.7:0/905313274 wait complete. 2024-04-21T08:34:22.006 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-21T08:34:22.028 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:34:22.028 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mgr... 2024-04-21T08:34:22.029 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-21T08:34:22.029 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-21T08:34:22.671 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.x 2024-04-21T08:34:22.671 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.x.service: Unit ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-21T08:34:22.832 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c9c60282-ffb9-11ee-bc93-c7b262605968.target.wants/ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@.service. 2024-04-21T08:34:22.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:22 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/905313274' entity='client.admin' 2024-04-21T08:34:22.848 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:22 smithi007 systemd[1]: Starting Ceph mgr.x for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:34:23.152 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:23 smithi007 podman[28696]: 2024-04-21 08:34:22.97710997 +0000 UTC m=+0.018113580 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:34:23.152 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:23 smithi007 podman[28696]: 2024-04-21 08:34:23.078056183 +0000 UTC m=+0.119059793 container create d8b0a57a71a30ac1cf5a83a9b4d7677cc81b235cb0544d029b9121988eb46591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:34:23.380 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:34:23.381 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-04-21T08:34:23.388 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:34:23.388 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-04-21T08:34:23.388 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-04-21T08:34:23.388 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T08:34:23.397 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:34:23.397 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-04-21T08:34:23.404 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:34:23.404 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-04-21T08:34:23.404 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-04-21T08:34:23.404 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-21T08:34:23.404 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr to start... 2024-04-21T08:34:23.404 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr... 2024-04-21T08:34:23.448 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:23 smithi007 podman[28696]: 2024-04-21 08:34:23.284850364 +0000 UTC m=+0.325853975 container init d8b0a57a71a30ac1cf5a83a9b4d7677cc81b235cb0544d029b9121988eb46591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-21T08:34:23.449 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:23 smithi007 podman[28696]: 2024-04-21 08:34:23.288384334 +0000 UTC m=+0.329387948 container start d8b0a57a71a30ac1cf5a83a9b4d7677cc81b235cb0544d029b9121988eb46591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-04-21T08:34:23.449 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:23 smithi007 bash[28696]: d8b0a57a71a30ac1cf5a83a9b4d7677cc81b235cb0544d029b9121988eb46591 2024-04-21T08:34:23.449 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:23 smithi007 systemd[1]: Started Ceph mgr.x for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:34:23.749 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:23 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:23.448+0000 7f3c98f7e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:34:23.750 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:23 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:23.642+0000 7f3c98f7e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:34:23.750 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:23 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:23.750+0000 7f3c98f7e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:34:24.307 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:24 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1949377336' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:34:24.308 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:24 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:24.085+0000 7f3c98f7e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:34:24.308 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:24 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:24.173+0000 7f3c98f7e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:34:24.563 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:24.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:34:24.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "c9c60282-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:34:24.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:34:24.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:34:24.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:34:24.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:34:24.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:24.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:34:24.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:34:24.564 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:34:24.565 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:34:24.566 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:34:24.574 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:34:24.574 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:34:24.574 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:34:24.574 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:24.575 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:34:24.576 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:34:14.986720+0000", 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.873+0000 7f8d20171700 1 Processor -- start 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.873+0000 7f8d20171700 1 -- start start 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.873+0000 7f8d20171700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d18108780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.873+0000 7f8d20171700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8d18108d50 con 0x7f8d18108360 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.874+0000 7f8d1df0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d18108780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.874+0000 7f8d1df0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d18108780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36074/0 (socket says 172.21.15.7:36074) 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.874+0000 7f8d1df0d700 1 -- 172.21.15.7:0/645531878 learned_addr learned my addr 172.21.15.7:0/645531878 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:24.577 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.874+0000 7f8d1df0d700 1 -- 172.21.15.7:0/645531878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d18109560 con 0x7f8d18108360 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.874+0000 7f8d1df0d700 1 --2- 172.21.15.7:0/645531878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d18108780 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009e80 tx=0x7f8d0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19011df2ccd9e7f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.874+0000 7f8d1cf0b700 1 -- 172.21.15.7:0/645531878 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d08017070 con 0x7f8d18108360 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.874+0000 7f8d1cf0b700 1 -- 172.21.15.7:0/645531878 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8d0800b8c0 con 0x7f8d18108360 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.874+0000 7f8d1cf0b700 1 -- 172.21.15.7:0/645531878 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d08004490 con 0x7f8d18108360 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.875+0000 7f8d20171700 1 -- 172.21.15.7:0/645531878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 msgr2=0x7f8d18108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.875+0000 7f8d20171700 1 --2- 172.21.15.7:0/645531878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d18108780 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009e80 tx=0x7f8d0800b330 comp rx=0 tx=0).stop 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.875+0000 7f8d20171700 1 -- 172.21.15.7:0/645531878 shutdown_connections 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.875+0000 7f8d20171700 1 --2- 172.21.15.7:0/645531878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d18108780 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.875+0000 7f8d20171700 1 -- 172.21.15.7:0/645531878 >> 172.21.15.7:0/645531878 conn(0x7f8d181038e0 msgr2=0x7f8d18105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.875+0000 7f8d20171700 1 -- 172.21.15.7:0/645531878 shutdown_connections 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.875+0000 7f8d20171700 1 -- 172.21.15.7:0/645531878 wait complete. 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.876+0000 7f8d20171700 1 Processor -- start 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.876+0000 7f8d20171700 1 -- start start 2024-04-21T08:34:24.578 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.876+0000 7f8d20171700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d1807db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:24.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.876+0000 7f8d20171700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1810c910 con 0x7f8d18108360 2024-04-21T08:34:24.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.876+0000 7f8d1df0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d1807db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:24.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.876+0000 7f8d1df0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d1807db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36076/0 (socket says 172.21.15.7:36076) 2024-04-21T08:34:24.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.876+0000 7f8d1df0d700 1 -- 172.21.15.7:0/1949377336 learned_addr learned my addr 172.21.15.7:0/1949377336 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:24.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.877+0000 7f8d1df0d700 1 -- 172.21.15.7:0/1949377336 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1807e0a0 con 0x7f8d18108360 2024-04-21T08:34:24.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.877+0000 7f8d1df0d700 1 --2- 172.21.15.7:0/1949377336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d1807db60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8d08004a60 tx=0x7f8d0800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:24.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.877+0000 7f8d16ffd700 1 -- 172.21.15.7:0/1949377336 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d08020020 con 0x7f8d18108360 2024-04-21T08:34:24.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.877+0000 7f8d16ffd700 1 -- 172.21.15.7:0/1949377336 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8d08003c60 con 0x7f8d18108360 2024-04-21T08:34:24.579 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.877+0000 7f8d20171700 1 -- 172.21.15.7:0/1949377336 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1807c290 con 0x7f8d18108360 2024-04-21T08:34:24.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.877+0000 7f8d16ffd700 1 -- 172.21.15.7:0/1949377336 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d08017040 con 0x7f8d18108360 2024-04-21T08:34:24.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.877+0000 7f8d20171700 1 -- 172.21.15.7:0/1949377336 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1807c7d0 con 0x7f8d18108360 2024-04-21T08:34:24.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.878+0000 7f8d16ffd700 1 -- 172.21.15.7:0/1949377336 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8d08023020 con 0x7f8d18108360 2024-04-21T08:34:24.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.878+0000 7f8d16ffd700 1 -- 172.21.15.7:0/1949377336 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8d0800f050 con 0x7f8d18108360 2024-04-21T08:34:24.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.878+0000 7f8d20171700 1 -- 172.21.15.7:0/1949377336 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d180640a0 con 0x7f8d18108360 2024-04-21T08:34:24.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.880+0000 7f8d16ffd700 1 -- 172.21.15.7:0/1949377336 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8d0802a020 con 0x7f8d18108360 2024-04-21T08:34:24.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.955+0000 7f8d20171700 1 -- 172.21.15.7:0/1949377336 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8d1807cb10 con 0x7f8d18108360 2024-04-21T08:34:24.580 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.955+0000 7f8d16ffd700 1 -- 172.21.15.7:0/1949377336 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8d08003e10 con 0x7f8d18108360 2024-04-21T08:34:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.956+0000 7f8d20171700 1 -- 172.21.15.7:0/1949377336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 msgr2=0x7f8d1807db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.956+0000 7f8d20171700 1 --2- 172.21.15.7:0/1949377336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d1807db60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8d08004a60 tx=0x7f8d0800bd50 comp rx=0 tx=0).stop 2024-04-21T08:34:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.956+0000 7f8d20171700 1 -- 172.21.15.7:0/1949377336 shutdown_connections 2024-04-21T08:34:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.956+0000 7f8d20171700 1 --2- 172.21.15.7:0/1949377336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d18108360 0x7f8d1807db60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.956+0000 7f8d20171700 1 -- 172.21.15.7:0/1949377336 >> 172.21.15.7:0/1949377336 conn(0x7f8d181038e0 msgr2=0x7f8d18104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.956+0000 7f8d20171700 1 -- 172.21.15.7:0/1949377336 shutdown_connections 2024-04-21T08:34:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:23.956+0000 7f8d20171700 1 -- 172.21.15.7:0/1949377336 wait complete. 2024-04-21T08:34:24.581 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (1/15)... 2024-04-21T08:34:25.787 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:25 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:25.436+0000 7f3c98f7e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:34:25.787 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:25 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:25.612+0000 7f3c98f7e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:34:25.787 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:25 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:25.787+0000 7f3c98f7e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:34:26.390 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:26 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:26.086+0000 7f3c98f7e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:34:26.390 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:26 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:26.181+0000 7f3c98f7e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:34:26.390 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:26 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:26.390+0000 7f3c98f7e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:34:26.651 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:26 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:26.591+0000 7f3c98f7e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:34:27.237 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:27 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/320431922' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:34:27.639 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:27 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:27.238+0000 7f3c98f7e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:34:27.651 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:27.651 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:34:27.651 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "c9c60282-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-21T08:34:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:27.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:34:27.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:34:14.986720+0000", 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.987+0000 7f1852934700 1 Processor -- start 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.987+0000 7f1852934700 1 -- start start 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.988+0000 7f1852934700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c07c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.988+0000 7f1852934700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f184c07c8c0 con 0x7f184c07de90 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.988+0000 7f184bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c07c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.988+0000 7f184bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c07c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38520/0 (socket says 172.21.15.7:38520) 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.988+0000 7f184bfff700 1 -- 172.21.15.7:0/1533646173 learned_addr learned my addr 172.21.15.7:0/1533646173 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.988+0000 7f184bfff700 1 -- 172.21.15.7:0/1533646173 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184c07ca40 con 0x7f184c07de90 2024-04-21T08:34:27.656 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.988+0000 7f184bfff700 1 --2- 172.21.15.7:0/1533646173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c07c2f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f183c0098a0 tx=0x7f183c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9827f3e0713dc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.988+0000 7f184affd700 1 -- 172.21.15.7:0/1533646173 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f183c004160 con 0x7f184c07de90 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.989+0000 7f184affd700 1 -- 172.21.15.7:0/1533646173 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f183c004300 con 0x7f184c07de90 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.989+0000 7f184affd700 1 -- 172.21.15.7:0/1533646173 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f183c011af0 con 0x7f184c07de90 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.989+0000 7f1852934700 1 -- 172.21.15.7:0/1533646173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 msgr2=0x7f184c07c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.989+0000 7f1852934700 1 --2- 172.21.15.7:0/1533646173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c07c2f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f183c0098a0 tx=0x7f183c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.989+0000 7f1852934700 1 -- 172.21.15.7:0/1533646173 shutdown_connections 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.989+0000 7f1852934700 1 --2- 172.21.15.7:0/1533646173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c07c2f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.989+0000 7f1852934700 1 -- 172.21.15.7:0/1533646173 >> 172.21.15.7:0/1533646173 conn(0x7f184c1038c0 msgr2=0x7f184c105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.989+0000 7f1852934700 1 -- 172.21.15.7:0/1533646173 shutdown_connections 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.989+0000 7f1852934700 1 -- 172.21.15.7:0/1533646173 wait complete. 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.990+0000 7f1852934700 1 Processor -- start 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.990+0000 7f1852934700 1 -- start start 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.990+0000 7f1852934700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:27.657 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.991+0000 7f1852934700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f184c10e8f0 con 0x7f184c07de90 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.991+0000 7f184bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.991+0000 7f184bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c19e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38524/0 (socket says 172.21.15.7:38524) 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.991+0000 7f184bfff700 1 -- 172.21.15.7:0/320431922 learned_addr learned my addr 172.21.15.7:0/320431922 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.991+0000 7f184bfff700 1 -- 172.21.15.7:0/320431922 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184c19e940 con 0x7f184c07de90 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.991+0000 7f184bfff700 1 --2- 172.21.15.7:0/320431922 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c19e400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f183c003f40 tx=0x7f183c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.991+0000 7f18497fa700 1 -- 172.21.15.7:0/320431922 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f183c0038b0 con 0x7f184c07de90 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.991+0000 7f18497fa700 1 -- 172.21.15.7:0/320431922 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f183c003a50 con 0x7f184c07de90 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.991+0000 7f1852934700 1 -- 172.21.15.7:0/320431922 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f184c19ebd0 con 0x7f184c07de90 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.991+0000 7f18497fa700 1 -- 172.21.15.7:0/320431922 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f183c01a5c0 con 0x7f184c07de90 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.991+0000 7f1852934700 1 -- 172.21.15.7:0/320431922 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f184c1a18c0 con 0x7f184c07de90 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.992+0000 7f18497fa700 1 -- 172.21.15.7:0/320431922 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f183c01ac50 con 0x7f184c07de90 2024-04-21T08:34:27.658 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.992+0000 7f18497fa700 1 -- 172.21.15.7:0/320431922 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f183c019700 con 0x7f184c07de90 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.992+0000 7f1852934700 1 -- 172.21.15.7:0/320431922 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f184c0640a0 con 0x7f184c07de90 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:26.996+0000 7f18497fa700 1 -- 172.21.15.7:0/320431922 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f183c0199a0 con 0x7f184c07de90 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:27.070+0000 7f1852934700 1 -- 172.21.15.7:0/320431922 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f184c1a1e00 con 0x7f184c07de90 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:27.071+0000 7f18497fa700 1 -- 172.21.15.7:0/320431922 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f183c019b80 con 0x7f184c07de90 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:27.072+0000 7f1852934700 1 -- 172.21.15.7:0/320431922 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 msgr2=0x7f184c19e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:27.072+0000 7f1852934700 1 --2- 172.21.15.7:0/320431922 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c19e400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f183c003f40 tx=0x7f183c011890 comp rx=0 tx=0).stop 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:27.072+0000 7f1852934700 1 -- 172.21.15.7:0/320431922 shutdown_connections 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:27.072+0000 7f1852934700 1 --2- 172.21.15.7:0/320431922 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c07de90 0x7f184c19e400 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:27.072+0000 7f1852934700 1 -- 172.21.15.7:0/320431922 >> 172.21.15.7:0/320431922 conn(0x7f184c1038c0 msgr2=0x7f184c104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:27.072+0000 7f1852934700 1 -- 172.21.15.7:0/320431922 shutdown_connections 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:27.072+0000 7f1852934700 1 -- 172.21.15.7:0/320431922 wait complete. 2024-04-21T08:34:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (2/15)... 2024-04-21T08:34:27.901 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:27 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:27.680+0000 7f3c98f7e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:34:27.901 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:27 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:27.780+0000 7f3c98f7e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:34:28.401 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:27 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:27.989+0000 7f3c98f7e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:34:28.894 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:28 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:28.542+0000 7f3c98f7e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:34:28.895 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:28 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:28.639+0000 7f3c98f7e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:34:29.151 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:28 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:28.894+0000 7f3c98f7e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:34:29.151 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:29 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:29.034+0000 7f3c98f7e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:34:29.151 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:29 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:29.133+0000 7f3c98f7e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:34:29.595 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:29 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:29.255+0000 7f3c98f7e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:34:29.596 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:29 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:29.595+0000 7f3c98f7e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:34:30.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:30 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1732737288' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:34:30.401 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:30 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:30.243+0000 7f3c98f7e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:34:30.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:30.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:34:30.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "c9c60282-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:34:30.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:34:30.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:30.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:34:30.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:30.689 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:34:14.986720+0000", 2024-04-21T08:34:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.050+0000 7f0130120700 1 Processor -- start 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.050+0000 7f0130120700 1 -- start start 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.050+0000 7f0130120700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f0128107670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.050+0000 7f0130120700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0128107c40 con 0x7f0128107250 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.050+0000 7f012debc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f0128107670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.050+0000 7f012debc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f0128107670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38534/0 (socket says 172.21.15.7:38534) 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.050+0000 7f012debc700 1 -- 172.21.15.7:0/1035326459 learned_addr learned my addr 172.21.15.7:0/1035326459 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.051+0000 7f012debc700 1 -- 172.21.15.7:0/1035326459 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0128108450 con 0x7f0128107250 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.051+0000 7f012debc700 1 --2- 172.21.15.7:0/1035326459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f0128107670 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f01240098a0 tx=0x7f0124009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66bf4cb5178e7cf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.051+0000 7f012ceba700 1 -- 172.21.15.7:0/1035326459 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0124004160 con 0x7f0128107250 2024-04-21T08:34:30.691 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.051+0000 7f012ceba700 1 -- 172.21.15.7:0/1035326459 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0124004300 con 0x7f0128107250 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.051+0000 7f012ceba700 1 -- 172.21.15.7:0/1035326459 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0124011af0 con 0x7f0128107250 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.052+0000 7f0130120700 1 -- 172.21.15.7:0/1035326459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 msgr2=0x7f0128107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.052+0000 7f0130120700 1 --2- 172.21.15.7:0/1035326459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f0128107670 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f01240098a0 tx=0x7f0124009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.052+0000 7f0130120700 1 -- 172.21.15.7:0/1035326459 shutdown_connections 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.052+0000 7f0130120700 1 --2- 172.21.15.7:0/1035326459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f0128107670 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.052+0000 7f0130120700 1 -- 172.21.15.7:0/1035326459 >> 172.21.15.7:0/1035326459 conn(0x7f0128102890 msgr2=0x7f0128104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.052+0000 7f0130120700 1 -- 172.21.15.7:0/1035326459 shutdown_connections 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.052+0000 7f0130120700 1 -- 172.21.15.7:0/1035326459 wait complete. 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.052+0000 7f0130120700 1 Processor -- start 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.052+0000 7f0130120700 1 -- start start 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f0130120700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f012819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f0130120700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f012810b800 con 0x7f0128107250 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f012debc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f012819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f012debc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f012819c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38548/0 (socket says 172.21.15.7:38548) 2024-04-21T08:34:30.692 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f012debc700 1 -- 172.21.15.7:0/1732737288 learned_addr learned my addr 172.21.15.7:0/1732737288 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f012debc700 1 -- 172.21.15.7:0/1732737288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012819c790 con 0x7f0128107250 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f012debc700 1 --2- 172.21.15.7:0/1732737288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f012819c250 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0124000c00 tx=0x7f0124011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f011effd700 1 -- 172.21.15.7:0/1732737288 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0124003b20 con 0x7f0128107250 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f0130120700 1 -- 172.21.15.7:0/1732737288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f012819ca20 con 0x7f0128107250 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f011effd700 1 -- 172.21.15.7:0/1732737288 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f012401a8c0 con 0x7f0128107250 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f011effd700 1 -- 172.21.15.7:0/1732737288 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0124024940 con 0x7f0128107250 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.053+0000 7f0130120700 1 -- 172.21.15.7:0/1732737288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f012819f710 con 0x7f0128107250 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.054+0000 7f011effd700 1 -- 172.21.15.7:0/1732737288 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0124022020 con 0x7f0128107250 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.054+0000 7f011effd700 1 -- 172.21.15.7:0/1732737288 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f012400b640 con 0x7f0128107250 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.054+0000 7f0130120700 1 -- 172.21.15.7:0/1732737288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01280640a0 con 0x7f0128107250 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.058+0000 7f011effd700 1 -- 172.21.15.7:0/1732737288 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f01240042a0 con 0x7f0128107250 2024-04-21T08:34:30.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.132+0000 7f0130120700 1 -- 172.21.15.7:0/1732737288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f012802e250 con 0x7f0128107250 2024-04-21T08:34:30.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.133+0000 7f011effd700 1 -- 172.21.15.7:0/1732737288 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f012401aa70 con 0x7f0128107250 2024-04-21T08:34:30.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.134+0000 7f0130120700 1 -- 172.21.15.7:0/1732737288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 msgr2=0x7f012819c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:30.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.134+0000 7f0130120700 1 --2- 172.21.15.7:0/1732737288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f012819c250 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0124000c00 tx=0x7f0124011b70 comp rx=0 tx=0).stop 2024-04-21T08:34:30.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.134+0000 7f0130120700 1 -- 172.21.15.7:0/1732737288 shutdown_connections 2024-04-21T08:34:30.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.134+0000 7f0130120700 1 --2- 172.21.15.7:0/1732737288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0128107250 0x7f012819c250 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:30.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.134+0000 7f0130120700 1 -- 172.21.15.7:0/1732737288 >> 172.21.15.7:0/1732737288 conn(0x7f0128102890 msgr2=0x7f0128103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:30.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.134+0000 7f0130120700 1 -- 172.21.15.7:0/1732737288 shutdown_connections 2024-04-21T08:34:30.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:30.134+0000 7f0130120700 1 -- 172.21.15.7:0/1732737288 wait complete. 2024-04-21T08:34:30.694 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (3/15)... 2024-04-21T08:34:30.901 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:30 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:30.471+0000 7f3c98f7e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:34:31.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: Activating manager daemon x 2024-04-21T08:34:31.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: mgrmap e2: x(active, starting, since 0.00331961s) 2024-04-21T08:34:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: from='mgr.14100 172.21.15.7:0/3644731658' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:34:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: from='mgr.14100 172.21.15.7:0/3644731658' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:34:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: from='mgr.14100 172.21.15.7:0/3644731658' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:34:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: from='mgr.14100 172.21.15.7:0/3644731658' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:34:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: from='mgr.14100 172.21.15.7:0/3644731658' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:34:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: Manager daemon x is now available 2024-04-21T08:34:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: from='mgr.14100 172.21.15.7:0/3644731658' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:34:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: from='mgr.14100 172.21.15.7:0/3644731658' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:34:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: from='mgr.14100 172.21.15.7:0/3644731658' entity='mgr.x' 2024-04-21T08:34:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: from='mgr.14100 172.21.15.7:0/3644731658' entity='mgr.x' 2024-04-21T08:34:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:31 smithi007 ceph-mon[28477]: from='mgr.14100 172.21.15.7:0/3644731658' entity='mgr.x' 2024-04-21T08:34:32.796 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:32 smithi007 ceph-mon[28477]: mgrmap e3: x(active, since 1.00696s) 2024-04-21T08:34:33.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:33 smithi007 ceph-mon[28477]: mgrmap e4: x(active, since 2s) 2024-04-21T08:34:33.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:33 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1162961782' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:34:33.907 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:33.907 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:34:33.907 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "c9c60282-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:34:33.907 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:34:33.907 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:34:33.907 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:34:33.907 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:33.908 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:34:33.909 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:34:33.910 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:34:33.910 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:34:33.910 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:34:33.910 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:34:33.910 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:34:33.910 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:34:33.910 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:34:33.910 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:33.910 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:34:33.910 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:33.911 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:34:33.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:33.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:34:33.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:33.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:34:14.986720+0000", 2024-04-21T08:34:33.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:34:33.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:33.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:34:33.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:34:33.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.088+0000 7f6085cb5700 1 Processor -- start 2024-04-21T08:34:33.912 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.088+0000 7f6085cb5700 1 -- start start 2024-04-21T08:34:33.913 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.088+0000 7f6085cb5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f60801087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:33.913 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.088+0000 7f6085cb5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6080108da0 con 0x7f60801083b0 2024-04-21T08:34:33.913 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.088+0000 7f607f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f60801087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:33.913 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.088+0000 7f607f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f60801087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38634/0 (socket says 172.21.15.7:38634) 2024-04-21T08:34:33.913 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.088+0000 7f607f7fe700 1 -- 172.21.15.7:0/778476075 learned_addr learned my addr 172.21.15.7:0/778476075 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:33.913 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.089+0000 7f607f7fe700 1 -- 172.21.15.7:0/778476075 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6080109560 con 0x7f60801083b0 2024-04-21T08:34:33.913 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.089+0000 7f607f7fe700 1 --2- 172.21.15.7:0/778476075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f60801087d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6070009e80 tx=0x7f607000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ebbc28cb6e1b83 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:33.913 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.089+0000 7f607e7fc700 1 -- 172.21.15.7:0/778476075 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6070017070 con 0x7f60801083b0 2024-04-21T08:34:33.913 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.089+0000 7f607e7fc700 1 -- 172.21.15.7:0/778476075 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f607000b8c0 con 0x7f60801083b0 2024-04-21T08:34:33.913 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.089+0000 7f607e7fc700 1 -- 172.21.15.7:0/778476075 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6070004490 con 0x7f60801083b0 2024-04-21T08:34:33.913 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.090+0000 7f6085cb5700 1 -- 172.21.15.7:0/778476075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 msgr2=0x7f60801087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.090+0000 7f6085cb5700 1 --2- 172.21.15.7:0/778476075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f60801087d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6070009e80 tx=0x7f607000b330 comp rx=0 tx=0).stop 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.090+0000 7f6085cb5700 1 -- 172.21.15.7:0/778476075 shutdown_connections 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.090+0000 7f6085cb5700 1 --2- 172.21.15.7:0/778476075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f60801087d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.090+0000 7f6085cb5700 1 -- 172.21.15.7:0/778476075 >> 172.21.15.7:0/778476075 conn(0x7f60801039b0 msgr2=0x7f6080105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.090+0000 7f6085cb5700 1 -- 172.21.15.7:0/778476075 shutdown_connections 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.090+0000 7f6085cb5700 1 -- 172.21.15.7:0/778476075 wait complete. 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.090+0000 7f6085cb5700 1 Processor -- start 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.091+0000 7f6085cb5700 1 -- start start 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.091+0000 7f6085cb5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f608019c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.091+0000 7f6085cb5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f608010c910 con 0x7f60801083b0 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.091+0000 7f607f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f608019c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:33.914 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.091+0000 7f607f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f608019c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38644/0 (socket says 172.21.15.7:38644) 2024-04-21T08:34:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.091+0000 7f607f7fe700 1 -- 172.21.15.7:0/1162961782 learned_addr learned my addr 172.21.15.7:0/1162961782 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.091+0000 7f607f7fe700 1 -- 172.21.15.7:0/1162961782 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608019c6f0 con 0x7f60801083b0 2024-04-21T08:34:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.091+0000 7f607f7fe700 1 --2- 172.21.15.7:0/1162961782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f608019c1b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6070004a60 tx=0x7f607000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.092+0000 7f607cff9700 1 -- 172.21.15.7:0/1162961782 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6070020020 con 0x7f60801083b0 2024-04-21T08:34:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.092+0000 7f607cff9700 1 -- 172.21.15.7:0/1162961782 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6070003c60 con 0x7f60801083b0 2024-04-21T08:34:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.092+0000 7f6085cb5700 1 -- 172.21.15.7:0/1162961782 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f608019c980 con 0x7f60801083b0 2024-04-21T08:34:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.092+0000 7f607cff9700 1 -- 172.21.15.7:0/1162961782 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6070017040 con 0x7f60801083b0 2024-04-21T08:34:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.092+0000 7f6085cb5700 1 -- 172.21.15.7:0/1162961782 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f608019f670 con 0x7f60801083b0 2024-04-21T08:34:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.093+0000 7f607cff9700 1 -- 172.21.15.7:0/1162961782 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f6070023020 con 0x7f60801083b0 2024-04-21T08:34:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.093+0000 7f6085cb5700 1 -- 172.21.15.7:0/1162961782 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60800640a0 con 0x7f60801083b0 2024-04-21T08:34:33.915 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.093+0000 7f607cff9700 1 --2- 172.21.15.7:0/1162961782 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f606c039670 0x7f606c03bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:33.916 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.093+0000 7f607cff9700 1 -- 172.21.15.7:0/1162961782 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f607000f050 con 0x7f60801083b0 2024-04-21T08:34:33.916 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.093+0000 7f607effd700 1 --2- 172.21.15.7:0/1162961782 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f606c039670 0x7f606c03bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:33.916 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.094+0000 7f607effd700 1 --2- 172.21.15.7:0/1162961782 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f606c039670 0x7f606c03bb30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6068006fd0 tx=0x7f6068006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:33.916 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.099+0000 7f607cff9700 1 -- 172.21.15.7:0/1162961782 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f607002a020 con 0x7f60801083b0 2024-04-21T08:34:33.916 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.393+0000 7f6085cb5700 1 -- 172.21.15.7:0/1162961782 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f608019f840 con 0x7f60801083b0 2024-04-21T08:34:33.916 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.393+0000 7f607cff9700 1 -- 172.21.15.7:0/1162961782 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f608019f840 con 0x7f60801083b0 2024-04-21T08:34:33.916 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.395+0000 7f6085cb5700 1 -- 172.21.15.7:0/1162961782 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f606c039670 msgr2=0x7f606c03bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:33.916 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.395+0000 7f6085cb5700 1 --2- 172.21.15.7:0/1162961782 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f606c039670 0x7f606c03bb30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6068006fd0 tx=0x7f6068006e40 comp rx=0 tx=0).stop 2024-04-21T08:34:33.916 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.395+0000 7f6085cb5700 1 -- 172.21.15.7:0/1162961782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 msgr2=0x7f608019c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:33.916 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.395+0000 7f6085cb5700 1 --2- 172.21.15.7:0/1162961782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f608019c1b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6070004a60 tx=0x7f607000bd50 comp rx=0 tx=0).stop 2024-04-21T08:34:33.916 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.396+0000 7f6085cb5700 1 -- 172.21.15.7:0/1162961782 shutdown_connections 2024-04-21T08:34:33.917 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.396+0000 7f6085cb5700 1 --2- 172.21.15.7:0/1162961782 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f606c039670 0x7f606c03bb30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:33.917 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.396+0000 7f6085cb5700 1 --2- 172.21.15.7:0/1162961782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60801083b0 0x7f608019c1b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:33.917 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.396+0000 7f6085cb5700 1 -- 172.21.15.7:0/1162961782 >> 172.21.15.7:0/1162961782 conn(0x7f60801039b0 msgr2=0x7f6080104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:33.917 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.396+0000 7f6085cb5700 1 -- 172.21.15.7:0/1162961782 shutdown_connections 2024-04-21T08:34:33.917 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:33.396+0000 7f6085cb5700 1 -- 172.21.15.7:0/1162961782 wait complete. 2024-04-21T08:34:33.917 INFO:teuthology.orchestra.run.smithi007.stdout:mgr is available 2024-04-21T08:34:34.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:34 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/168585792' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T08:34:35.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:35.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T08:34:35.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:35.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-21T08:34:35.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T08:34:35.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T08:34:35.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.283+0000 7f85a526b700 1 Processor -- start 2024-04-21T08:34:35.026 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.283+0000 7f85a526b700 1 -- start start 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.283+0000 7f85a526b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a0108790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.283+0000 7f85a526b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f85a0108d60 con 0x7f85a0108370 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.283+0000 7f859effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a0108790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.283+0000 7f859effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a0108790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38656/0 (socket says 172.21.15.7:38656) 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.283+0000 7f859effd700 1 -- 172.21.15.7:0/994409815 learned_addr learned my addr 172.21.15.7:0/994409815 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.284+0000 7f859effd700 1 -- 172.21.15.7:0/994409815 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a0109570 con 0x7f85a0108370 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.284+0000 7f859effd700 1 --2- 172.21.15.7:0/994409815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a0108790 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8598009e80 tx=0x7f859800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4adef0b2399dfcd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.284+0000 7f859dffb700 1 -- 172.21.15.7:0/994409815 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8598017070 con 0x7f85a0108370 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.284+0000 7f859dffb700 1 -- 172.21.15.7:0/994409815 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f859800b8c0 con 0x7f85a0108370 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.284+0000 7f859dffb700 1 -- 172.21.15.7:0/994409815 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8598004490 con 0x7f85a0108370 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.285+0000 7f85a526b700 1 -- 172.21.15.7:0/994409815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 msgr2=0x7f85a0108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.285+0000 7f85a526b700 1 --2- 172.21.15.7:0/994409815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a0108790 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8598009e80 tx=0x7f859800b330 comp rx=0 tx=0).stop 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.285+0000 7f85a526b700 1 -- 172.21.15.7:0/994409815 shutdown_connections 2024-04-21T08:34:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.285+0000 7f85a526b700 1 --2- 172.21.15.7:0/994409815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a0108790 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.285+0000 7f85a526b700 1 -- 172.21.15.7:0/994409815 >> 172.21.15.7:0/994409815 conn(0x7f85a0103950 msgr2=0x7f85a0105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.285+0000 7f85a526b700 1 -- 172.21.15.7:0/994409815 shutdown_connections 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.285+0000 7f85a526b700 1 -- 172.21.15.7:0/994409815 wait complete. 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.285+0000 7f85a526b700 1 Processor -- start 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.286+0000 7f85a526b700 1 -- start start 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.286+0000 7f85a526b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a019c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.286+0000 7f85a526b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f85a010c920 con 0x7f85a0108370 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.286+0000 7f859effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a019c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.286+0000 7f859effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a019c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38662/0 (socket says 172.21.15.7:38662) 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.286+0000 7f859effd700 1 -- 172.21.15.7:0/168585792 learned_addr learned my addr 172.21.15.7:0/168585792 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.286+0000 7f859effd700 1 -- 172.21.15.7:0/168585792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a019c6f0 con 0x7f85a0108370 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.286+0000 7f859effd700 1 --2- 172.21.15.7:0/168585792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a019c1b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8598004a60 tx=0x7f859800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.287+0000 7f8597fff700 1 -- 172.21.15.7:0/168585792 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8598020020 con 0x7f85a0108370 2024-04-21T08:34:35.028 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.287+0000 7f8597fff700 1 -- 172.21.15.7:0/168585792 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8598003c60 con 0x7f85a0108370 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.287+0000 7f85a526b700 1 -- 172.21.15.7:0/168585792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a019c980 con 0x7f85a0108370 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.287+0000 7f8597fff700 1 -- 172.21.15.7:0/168585792 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8598017040 con 0x7f85a0108370 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.287+0000 7f85a526b700 1 -- 172.21.15.7:0/168585792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a019f670 con 0x7f85a0108370 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.288+0000 7f8597fff700 1 -- 172.21.15.7:0/168585792 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f8598023020 con 0x7f85a0108370 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.288+0000 7f85a526b700 1 -- 172.21.15.7:0/168585792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a00640a0 con 0x7f85a0108370 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.288+0000 7f8597fff700 1 --2- 172.21.15.7:0/168585792 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f858c039620 0x7f858c03bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.288+0000 7f8597fff700 1 -- 172.21.15.7:0/168585792 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f859800f050 con 0x7f85a0108370 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.288+0000 7f859e7fc700 1 --2- 172.21.15.7:0/168585792 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f858c039620 0x7f858c03bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.288+0000 7f859e7fc700 1 --2- 172.21.15.7:0/168585792 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f858c039620 0x7f858c03bae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8588006fd0 tx=0x7f8588006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.293+0000 7f8597fff700 1 -- 172.21.15.7:0/168585792 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f859802a020 con 0x7f85a0108370 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.500+0000 7f85a526b700 1 -- 172.21.15.7:0/168585792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f85a019f840 con 0x7f85a0108370 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.501+0000 7f8597fff700 1 -- 172.21.15.7:0/168585792 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f85a019f840 con 0x7f85a0108370 2024-04-21T08:34:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.503+0000 7f85a526b700 1 -- 172.21.15.7:0/168585792 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f858c039620 msgr2=0x7f858c03bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:35.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.503+0000 7f85a526b700 1 --2- 172.21.15.7:0/168585792 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f858c039620 0x7f858c03bae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8588006fd0 tx=0x7f8588006e40 comp rx=0 tx=0).stop 2024-04-21T08:34:35.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.503+0000 7f85a526b700 1 -- 172.21.15.7:0/168585792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 msgr2=0x7f85a019c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:35.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.503+0000 7f85a526b700 1 --2- 172.21.15.7:0/168585792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a019c1b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8598004a60 tx=0x7f859800bd50 comp rx=0 tx=0).stop 2024-04-21T08:34:35.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.503+0000 7f85a526b700 1 -- 172.21.15.7:0/168585792 shutdown_connections 2024-04-21T08:34:35.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.503+0000 7f85a526b700 1 --2- 172.21.15.7:0/168585792 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f858c039620 0x7f858c03bae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:35.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.503+0000 7f85a526b700 1 --2- 172.21.15.7:0/168585792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85a0108370 0x7f85a019c1b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:35.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.503+0000 7f85a526b700 1 -- 172.21.15.7:0/168585792 >> 172.21.15.7:0/168585792 conn(0x7f85a0103950 msgr2=0x7f85a01043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:35.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.503+0000 7f85a526b700 1 -- 172.21.15.7:0/168585792 shutdown_connections 2024-04-21T08:34:35.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:34.503+0000 7f85a526b700 1 -- 172.21.15.7:0/168585792 wait complete. 2024-04-21T08:34:35.030 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling cephadm module... 2024-04-21T08:34:36.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:35 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/855266243' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-21T08:34:36.031 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:35 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: ignoring --setuser ceph since I am not root 2024-04-21T08:34:36.031 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:35 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: ignoring --setgroup ceph since I am not root 2024-04-21T08:34:36.031 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:35 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:35.937+0000 7f521d1cb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:34:36.255 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.434+0000 7f0d9cd8f700 1 Processor -- start 2024-04-21T08:34:36.255 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.434+0000 7f0d9cd8f700 1 -- start start 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.434+0000 7f0d9cd8f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9810b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.434+0000 7f0d9cd8f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9807be40 con 0x7f0d98108e20 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.434+0000 7f0d9659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9810b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.434+0000 7f0d9659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9810b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35144/0 (socket says 172.21.15.7:35144) 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.434+0000 7f0d9659c700 1 -- 172.21.15.7:0/1701507487 learned_addr learned my addr 172.21.15.7:0/1701507487 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.435+0000 7f0d9659c700 1 -- 172.21.15.7:0/1701507487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9810b750 con 0x7f0d98108e20 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.435+0000 7f0d9659c700 1 --2- 172.21.15.7:0/1701507487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9810b210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0d880098a0 tx=0x7f0d88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33ebf206bf3a5f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.435+0000 7f0d9559a700 1 -- 172.21.15.7:0/1701507487 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d88004160 con 0x7f0d98108e20 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.435+0000 7f0d9559a700 1 -- 172.21.15.7:0/1701507487 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0d88004300 con 0x7f0d98108e20 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.435+0000 7f0d9559a700 1 -- 172.21.15.7:0/1701507487 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d88011af0 con 0x7f0d98108e20 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.435+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/1701507487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 msgr2=0x7f0d9810b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:36.256 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.435+0000 7f0d9cd8f700 1 --2- 172.21.15.7:0/1701507487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9810b210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0d880098a0 tx=0x7f0d88009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.436+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/1701507487 shutdown_connections 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.436+0000 7f0d9cd8f700 1 --2- 172.21.15.7:0/1701507487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9810b210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.436+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/1701507487 >> 172.21.15.7:0/1701507487 conn(0x7f0d98102890 msgr2=0x7f0d98104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.436+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/1701507487 shutdown_connections 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.436+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/1701507487 wait complete. 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.436+0000 7f0d9cd8f700 1 Processor -- start 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.436+0000 7f0d9cd8f700 1 -- start start 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.436+0000 7f0d9cd8f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9819c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.436+0000 7f0d9cd8f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9810c6c0 con 0x7f0d98108e20 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.437+0000 7f0d9659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9819c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.437+0000 7f0d9659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9819c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35156/0 (socket says 172.21.15.7:35156) 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.437+0000 7f0d9659c700 1 -- 172.21.15.7:0/855266243 learned_addr learned my addr 172.21.15.7:0/855266243 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:36.257 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.437+0000 7f0d9659c700 1 -- 172.21.15.7:0/855266243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9819c6f0 con 0x7f0d98108e20 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.437+0000 7f0d9659c700 1 --2- 172.21.15.7:0/855266243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9819c1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0d88011ac0 tx=0x7f0d88011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.437+0000 7f0d877fe700 1 -- 172.21.15.7:0/855266243 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d88003b20 con 0x7f0d98108e20 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.437+0000 7f0d877fe700 1 -- 172.21.15.7:0/855266243 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0d8801a8c0 con 0x7f0d98108e20 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.437+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/855266243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9819c980 con 0x7f0d98108e20 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.437+0000 7f0d877fe700 1 -- 172.21.15.7:0/855266243 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d880248d0 con 0x7f0d98108e20 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.437+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/855266243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9819f670 con 0x7f0d98108e20 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.438+0000 7f0d877fe700 1 -- 172.21.15.7:0/855266243 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f0d88022020 con 0x7f0d98108e20 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.438+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/855266243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d980640a0 con 0x7f0d98108e20 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.439+0000 7f0d877fe700 1 --2- 172.21.15.7:0/855266243 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f0d80039620 0x7f0d8003bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.439+0000 7f0d877fe700 1 -- 172.21.15.7:0/855266243 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0d8804d660 con 0x7f0d98108e20 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.439+0000 7f0d95d9b700 1 --2- 172.21.15.7:0/855266243 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f0d80039620 0x7f0d8003bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.439+0000 7f0d95d9b700 1 --2- 172.21.15.7:0/855266243 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f0d80039620 0x7f0d8003bae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c006fd0 tx=0x7f0d8c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:36.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.444+0000 7f0d877fe700 1 -- 172.21.15.7:0/855266243 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d8804d3c0 con 0x7f0d98108e20 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.713+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/855266243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f0d9819f840 con 0x7f0d98108e20 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.765+0000 7f0d877fe700 1 -- 172.21.15.7:0/855266243 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f0d9819f840 con 0x7f0d98108e20 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.766+0000 7f0d877fe700 1 -- 172.21.15.7:0/855266243 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f0d8802d910 con 0x7f0d98108e20 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.767+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/855266243 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f0d80039620 msgr2=0x7f0d8003bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.767+0000 7f0d9cd8f700 1 --2- 172.21.15.7:0/855266243 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f0d80039620 0x7f0d8003bae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c006fd0 tx=0x7f0d8c006e40 comp rx=0 tx=0).stop 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.767+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/855266243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 msgr2=0x7f0d9819c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.767+0000 7f0d9cd8f700 1 --2- 172.21.15.7:0/855266243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9819c1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0d88011ac0 tx=0x7f0d88011ba0 comp rx=0 tx=0).stop 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.768+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/855266243 shutdown_connections 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.768+0000 7f0d9cd8f700 1 --2- 172.21.15.7:0/855266243 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f0d80039620 0x7f0d8003bae0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.768+0000 7f0d9cd8f700 1 --2- 172.21.15.7:0/855266243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d98108e20 0x7f0d9819c1b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.768+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/855266243 >> 172.21.15.7:0/855266243 conn(0x7f0d98102890 msgr2=0x7f0d98103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.768+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/855266243 shutdown_connections 2024-04-21T08:34:36.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:35.768+0000 7f0d9cd8f700 1 -- 172.21.15.7:0/855266243 wait complete. 2024-04-21T08:34:36.348 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:36.128+0000 7f521d1cb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:34:36.349 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:36.234+0000 7f521d1cb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:34:36.651 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:36.565+0000 7f521d1cb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:34:36.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:36 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/855266243' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-21T08:34:36.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:36 smithi007 ceph-mon[28477]: mgrmap e5: x(active, since 5s) 2024-04-21T08:34:36.975 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:36.655+0000 7f521d1cb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:34:37.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:34:37.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.694+0000 7fc8c6258700 1 Processor -- start 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.694+0000 7fc8c6258700 1 -- start start 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.694+0000 7fc8c6258700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c01087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.694+0000 7fc8c6258700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c0108d70 con 0x7fc8c0108380 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.694+0000 7fc8bf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c01087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.694+0000 7fc8bf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c01087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35182/0 (socket says 172.21.15.7:35182) 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.694+0000 7fc8bf7fe700 1 -- 172.21.15.7:0/3708087271 learned_addr learned my addr 172.21.15.7:0/3708087271 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.694+0000 7fc8bf7fe700 1 -- 172.21.15.7:0/3708087271 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c0109530 con 0x7fc8c0108380 2024-04-21T08:34:37.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.695+0000 7fc8bf7fe700 1 --2- 172.21.15.7:0/3708087271 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c01087a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00098a0 tx=0x7fc8b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2a3a570c0f91809 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.695+0000 7fc8be7fc700 1 -- 172.21.15.7:0/3708087271 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8b0004160 con 0x7fc8c0108380 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.695+0000 7fc8be7fc700 1 -- 172.21.15.7:0/3708087271 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc8b0004300 con 0x7fc8c0108380 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.695+0000 7fc8be7fc700 1 -- 172.21.15.7:0/3708087271 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8b0011af0 con 0x7fc8c0108380 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.695+0000 7fc8c6258700 1 -- 172.21.15.7:0/3708087271 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 msgr2=0x7fc8c01087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.695+0000 7fc8c6258700 1 --2- 172.21.15.7:0/3708087271 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c01087a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00098a0 tx=0x7fc8b0009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.695+0000 7fc8c6258700 1 -- 172.21.15.7:0/3708087271 shutdown_connections 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.695+0000 7fc8c6258700 1 --2- 172.21.15.7:0/3708087271 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c01087a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.695+0000 7fc8c6258700 1 -- 172.21.15.7:0/3708087271 >> 172.21.15.7:0/3708087271 conn(0x7fc8c0103920 msgr2=0x7fc8c0105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.696+0000 7fc8c6258700 1 -- 172.21.15.7:0/3708087271 shutdown_connections 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.696+0000 7fc8c6258700 1 -- 172.21.15.7:0/3708087271 wait complete. 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.696+0000 7fc8c6258700 1 Processor -- start 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.696+0000 7fc8c6258700 1 -- start start 2024-04-21T08:34:37.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.696+0000 7fc8c6258700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.696+0000 7fc8c6258700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c010c8e0 con 0x7fc8c0108380 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.697+0000 7fc8bf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.697+0000 7fc8bf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c019e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35196/0 (socket says 172.21.15.7:35196) 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.697+0000 7fc8bf7fe700 1 -- 172.21.15.7:0/1808222546 learned_addr learned my addr 172.21.15.7:0/1808222546 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.697+0000 7fc8bf7fe700 1 -- 172.21.15.7:0/1808222546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c019e940 con 0x7fc8c0108380 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.697+0000 7fc8bf7fe700 1 --2- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c019e400 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0003f40 tx=0x7fc8b0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.697+0000 7fc8bcff9700 1 -- 172.21.15.7:0/1808222546 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8b00038b0 con 0x7fc8c0108380 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.697+0000 7fc8bcff9700 1 -- 172.21.15.7:0/1808222546 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc8b0003a50 con 0x7fc8c0108380 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.697+0000 7fc8bcff9700 1 -- 172.21.15.7:0/1808222546 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8b001a5c0 con 0x7fc8c0108380 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.697+0000 7fc8c6258700 1 -- 172.21.15.7:0/1808222546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8c019ebd0 con 0x7fc8c0108380 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.697+0000 7fc8c6258700 1 -- 172.21.15.7:0/1808222546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8c01a18c0 con 0x7fc8c0108380 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.698+0000 7fc8bcff9700 1 -- 172.21.15.7:0/1808222546 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fc8b001a870 con 0x7fc8c0108380 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.698+0000 7fc8c6258700 1 -- 172.21.15.7:0/1808222546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8c00640a0 con 0x7fc8c0108380 2024-04-21T08:34:37.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.699+0000 7fc8bcff9700 1 --2- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7fc8a8039670 0x7fc8a803bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.699+0000 7fc8bcff9700 1 -- 172.21.15.7:0/1808222546 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc8b004da60 con 0x7fc8c0108380 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.699+0000 7fc8beffd700 1 -- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7fc8a8039670 msgr2=0x7fc8a803bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2031045761 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.699+0000 7fc8beffd700 1 --2- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7fc8a8039670 0x7fc8a803bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.704+0000 7fc8bcff9700 1 -- 172.21.15.7:0/1808222546 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc8b0019c70 con 0x7fc8c0108380 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.900+0000 7fc8beffd700 1 -- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7fc8a8039670 msgr2=0x7fc8a803bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2031045761 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.900+0000 7fc8beffd700 1 --2- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7fc8a8039670 0x7fc8a803bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.974+0000 7fc8c6258700 1 -- 172.21.15.7:0/1808222546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc8c01a1a90 con 0x7fc8c0108380 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.975+0000 7fc8bcff9700 1 -- 172.21.15.7:0/1808222546 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fc8c01a1a90 con 0x7fc8c0108380 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.977+0000 7fc8c6258700 1 -- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7fc8a8039670 msgr2=0x7fc8a803bb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.977+0000 7fc8c6258700 1 --2- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7fc8a8039670 0x7fc8a803bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.977+0000 7fc8c6258700 1 -- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 msgr2=0x7fc8c019e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.977+0000 7fc8c6258700 1 --2- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c019e400 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0003f40 tx=0x7fc8b0011890 comp rx=0 tx=0).stop 2024-04-21T08:34:37.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.977+0000 7fc8c6258700 1 -- 172.21.15.7:0/1808222546 shutdown_connections 2024-04-21T08:34:37.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.977+0000 7fc8c6258700 1 --2- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7fc8a8039670 0x7fc8a803bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:37.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.977+0000 7fc8c6258700 1 --2- 172.21.15.7:0/1808222546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c0108380 0x7fc8c019e400 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:37.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.977+0000 7fc8c6258700 1 -- 172.21.15.7:0/1808222546 >> 172.21.15.7:0/1808222546 conn(0x7fc8c0103920 msgr2=0x7fc8c0104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:37.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.977+0000 7fc8c6258700 1 -- 172.21.15.7:0/1808222546 shutdown_connections 2024-04-21T08:34:37.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:36.977+0000 7fc8c6258700 1 -- 172.21.15.7:0/1808222546 wait complete. 2024-04-21T08:34:37.535 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2024-04-21T08:34:37.535 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 5... 2024-04-21T08:34:37.857 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:37 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1808222546' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T08:34:38.151 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:37 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:37.918+0000 7f521d1cb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:34:38.151 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:38.098+0000 7f521d1cb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:34:38.567 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:38.274+0000 7f521d1cb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:34:38.879 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:38.573+0000 7f521d1cb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:34:38.879 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:38.669+0000 7f521d1cb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:34:38.879 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:38.879+0000 7f521d1cb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:34:39.151 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:39.078+0000 7f521d1cb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:34:40.138 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:39.700+0000 7f521d1cb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:34:40.401 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:40 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:40.137+0000 7f521d1cb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:34:40.401 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:40 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:40.237+0000 7f521d1cb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:34:40.901 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:40 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:40.446+0000 7f521d1cb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:34:41.341 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:40 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:40.992+0000 7f521d1cb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:34:41.341 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:41.088+0000 7f521d1cb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:34:41.651 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:41.341+0000 7f521d1cb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:34:41.651 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:41.479+0000 7f521d1cb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:34:41.651 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:41.577+0000 7f521d1cb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:34:42.010 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:41.697+0000 7f521d1cb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:34:42.011 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:42.011+0000 7f521d1cb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:34:42.901 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:42.637+0000 7f521d1cb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:34:42.901 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:34:42.861+0000 7f521d1cb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:34:43.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: Active manager daemon x restarted 2024-04-21T08:34:43.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: Activating manager daemon x 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: osdmap e2: 0 total, 0 up, 0 in 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: mgrmap e6: x(active, starting, since 0.003615s) 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: Manager daemon x is now available 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:43.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:42 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:44.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:43 smithi007 ceph-mon[28477]: Found migration_current of "None". Setting to last migration. 2024-04-21T08:34:44.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:43 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:34:44.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:43 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:34:44.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:43 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:44.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:43 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:44.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:43 smithi007 ceph-mon[28477]: [21/Apr/2024:08:34:43] ENGINE Bus STARTING 2024-04-21T08:34:44.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:43 smithi007 ceph-mon[28477]: mgrmap e7: x(active, since 1.00567s) 2024-04-21T08:34:44.438 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.960+0000 7f44bab5e700 1 Processor -- start 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.960+0000 7f44bab5e700 1 -- start start 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.960+0000 7f44bab5e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b4108830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.960+0000 7f44bab5e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f44b4108e00 con 0x7f44b4108410 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.960+0000 7f44b88fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b4108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.960+0000 7f44b88fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b4108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35204/0 (socket says 172.21.15.7:35204) 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.960+0000 7f44b88fa700 1 -- 172.21.15.7:0/2204613531 learned_addr learned my addr 172.21.15.7:0/2204613531 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.960+0000 7f44b88fa700 1 -- 172.21.15.7:0/2204613531 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b4109610 con 0x7f44b4108410 2024-04-21T08:34:44.439 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.961+0000 7f44b88fa700 1 --2- 172.21.15.7:0/2204613531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b4108830 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f44a80098a0 tx=0x7f44a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df29ed67769e0bd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.961+0000 7f44b37fe700 1 -- 172.21.15.7:0/2204613531 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44a8004160 con 0x7f44b4108410 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.961+0000 7f44b37fe700 1 -- 172.21.15.7:0/2204613531 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f44a8004300 con 0x7f44b4108410 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.961+0000 7f44b37fe700 1 -- 172.21.15.7:0/2204613531 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44a8011af0 con 0x7f44b4108410 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.961+0000 7f44bab5e700 1 -- 172.21.15.7:0/2204613531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 msgr2=0x7f44b4108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.961+0000 7f44bab5e700 1 --2- 172.21.15.7:0/2204613531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b4108830 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f44a80098a0 tx=0x7f44a8009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.961+0000 7f44bab5e700 1 -- 172.21.15.7:0/2204613531 shutdown_connections 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.961+0000 7f44bab5e700 1 --2- 172.21.15.7:0/2204613531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b4108830 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.961+0000 7f44bab5e700 1 -- 172.21.15.7:0/2204613531 >> 172.21.15.7:0/2204613531 conn(0x7f44b4103990 msgr2=0x7f44b4105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.962+0000 7f44bab5e700 1 -- 172.21.15.7:0/2204613531 shutdown_connections 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.962+0000 7f44bab5e700 1 -- 172.21.15.7:0/2204613531 wait complete. 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.962+0000 7f44bab5e700 1 Processor -- start 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.962+0000 7f44bab5e700 1 -- start start 2024-04-21T08:34:44.440 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.962+0000 7f44bab5e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b407db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.962+0000 7f44bab5e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f44b410c9c0 con 0x7f44b4108410 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.962+0000 7f44b88fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b407db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.962+0000 7f44b88fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b407db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35220/0 (socket says 172.21.15.7:35220) 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.962+0000 7f44b88fa700 1 -- 172.21.15.7:0/1778782970 learned_addr learned my addr 172.21.15.7:0/1778782970 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.963+0000 7f44b88fa700 1 -- 172.21.15.7:0/1778782970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b407e0a0 con 0x7f44b4108410 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.963+0000 7f44b88fa700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b407db60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f44a8003f40 tx=0x7f44a8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.963+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44a80038b0 con 0x7f44b4108410 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.963+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f44a8003a50 con 0x7f44b4108410 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.963+0000 7f44bab5e700 1 -- 172.21.15.7:0/1778782970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b407c290 con 0x7f44b4108410 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.963+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44a801a5c0 con 0x7f44b4108410 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.963+0000 7f44bab5e700 1 -- 172.21.15.7:0/1778782970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b407c7d0 con 0x7f44b4108410 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.964+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f44a801a870 con 0x7f44b4108410 2024-04-21T08:34:44.441 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.964+0000 7f44b1ffb700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 0x7f44a403bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.964+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 --> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f44a403c240 con 0x7f44a4039670 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.964+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f44a804dbc0 con 0x7f44b4108410 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.964+0000 7f44b3fff700 1 -- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 msgr2=0x7f44a403bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2031045761 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:37.965+0000 7f44b3fff700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 0x7f44a403bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:38.166+0000 7f44b3fff700 1 -- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 msgr2=0x7f44a403bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2031045761 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:38.166+0000 7f44b3fff700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 0x7f44a403bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:38.567+0000 7f44b3fff700 1 -- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 msgr2=0x7f44a403bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2031045761 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:38.567+0000 7f44b3fff700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 0x7f44a403bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:39.368+0000 7f44b3fff700 1 -- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 msgr2=0x7f44a403bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2031045761 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:39.368+0000 7f44b3fff700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 0x7f44a403bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:40.969+0000 7f44b3fff700 1 -- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 msgr2=0x7f44a403bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2031045761 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:40.969+0000 7f44b3fff700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 0x7f44a403bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T08:34:44.442 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:42.865+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f44a801abb0 con 0x7f44b4108410 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:42.865+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 msgr2=0x7f44a403bb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:42.865+0000 7f44b1ffb700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 0x7f44a403bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.866+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f44a804f760 con 0x7f44b4108410 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.866+0000 7f44b1ffb700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f44a403cf60 0x7f44a403f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.866+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 --> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f44a8008ea0 con 0x7f44a403cf60 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.867+0000 7f44b3fff700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f44a403cf60 0x7f44a403f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.867+0000 7f44b3fff700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f44a403cf60 0x7f44a403f350 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f44a0003a10 tx=0x7f44a00092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.868+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 <== mgr.14120 v2:172.21.15.7:6800/102274072 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f44a8008ea0 con 0x7f44a403cf60 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 -- 172.21.15.7:0/1778782970 --> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f44b4069b50 con 0x7f44a403cf60 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44b1ffb700 1 -- 172.21.15.7:0/1778782970 <== mgr.14120 v2:172.21.15.7:6800/102274072 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f44b4069b50 con 0x7f44a403cf60 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 -- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f44a403cf60 msgr2=0x7f44a403f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f44a403cf60 0x7f44a403f350 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f44a0003a10 tx=0x7f44a00092b0 comp rx=0 tx=0).stop 2024-04-21T08:34:44.443 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 -- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 msgr2=0x7f44b407db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:44.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b407db60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f44a8003f40 tx=0x7f44a8011890 comp rx=0 tx=0).stop 2024-04-21T08:34:44.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 -- 172.21.15.7:0/1778782970 shutdown_connections 2024-04-21T08:34:44.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f44a403cf60 0x7f44a403f350 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:44.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44b4108410 0x7f44b407db60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:44.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 --2- 172.21.15.7:0/1778782970 >> [v2:172.21.15.7:6800/2031045761,v1:172.21.15.7:6801/2031045761] conn(0x7f44a4039670 0x7f44a403bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:44.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 -- 172.21.15.7:0/1778782970 >> 172.21.15.7:0/1778782970 conn(0x7f44b4103990 msgr2=0x7f44b4104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:44.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 -- 172.21.15.7:0/1778782970 shutdown_connections 2024-04-21T08:34:44.444 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:43.875+0000 7f44bab5e700 1 -- 172.21.15.7:0/1778782970 wait complete. 2024-04-21T08:34:44.444 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 5 is available 2024-04-21T08:34:44.444 INFO:teuthology.orchestra.run.smithi007.stdout:Setting orchestrator backend to cephadm... 2024-04-21T08:34:45.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:44 smithi007 ceph-mon[28477]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T08:34:45.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:44 smithi007 ceph-mon[28477]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T08:34:45.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:44 smithi007 ceph-mon[28477]: [21/Apr/2024:08:34:43] ENGINE Serving on https://172.21.15.7:7150 2024-04-21T08:34:45.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:44 smithi007 ceph-mon[28477]: [21/Apr/2024:08:34:44] ENGINE Serving on http://172.21.15.7:8765 2024-04-21T08:34:45.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:44 smithi007 ceph-mon[28477]: [21/Apr/2024:08:34:44] ENGINE Bus STARTED 2024-04-21T08:34:45.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:44 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:45.653 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.869+0000 7f248878f700 1 Processor -- start 2024-04-21T08:34:45.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.869+0000 7f248878f700 1 -- start start 2024-04-21T08:34:45.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.869+0000 7f248878f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f248010b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:45.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.869+0000 7f248878f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f248007bea0 con 0x7f2480108db0 2024-04-21T08:34:45.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.869+0000 7f248652b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f248010b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:45.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.869+0000 7f248652b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f248010b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39738/0 (socket says 172.21.15.7:39738) 2024-04-21T08:34:45.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.869+0000 7f248652b700 1 -- 172.21.15.7:0/2789990051 learned_addr learned my addr 172.21.15.7:0/2789990051 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:45.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.870+0000 7f248652b700 1 -- 172.21.15.7:0/2789990051 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248010b6e0 con 0x7f2480108db0 2024-04-21T08:34:45.659 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.870+0000 7f248652b700 1 --2- 172.21.15.7:0/2789990051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f248010b1a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f247c0098a0 tx=0x7f247c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f9e16f3098a3d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.870+0000 7f2485529700 1 -- 172.21.15.7:0/2789990051 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f247c004160 con 0x7f2480108db0 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.870+0000 7f2485529700 1 -- 172.21.15.7:0/2789990051 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f247c004300 con 0x7f2480108db0 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.870+0000 7f2485529700 1 -- 172.21.15.7:0/2789990051 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f247c011af0 con 0x7f2480108db0 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.871+0000 7f248878f700 1 -- 172.21.15.7:0/2789990051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 msgr2=0x7f248010b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.871+0000 7f248878f700 1 --2- 172.21.15.7:0/2789990051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f248010b1a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f247c0098a0 tx=0x7f247c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.871+0000 7f248878f700 1 -- 172.21.15.7:0/2789990051 shutdown_connections 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.871+0000 7f248878f700 1 --2- 172.21.15.7:0/2789990051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f248010b1a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.871+0000 7f248878f700 1 -- 172.21.15.7:0/2789990051 >> 172.21.15.7:0/2789990051 conn(0x7f2480102820 msgr2=0x7f2480104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.871+0000 7f248878f700 1 -- 172.21.15.7:0/2789990051 shutdown_connections 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.871+0000 7f248878f700 1 -- 172.21.15.7:0/2789990051 wait complete. 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.872+0000 7f248878f700 1 Processor -- start 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.872+0000 7f248878f700 1 -- start start 2024-04-21T08:34:45.660 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.872+0000 7f248878f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f2480108630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.872+0000 7f248878f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f248010c6b0 con 0x7f2480108db0 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.872+0000 7f248652b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f2480108630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.872+0000 7f248652b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f2480108630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39744/0 (socket says 172.21.15.7:39744) 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.872+0000 7f248652b700 1 -- 172.21.15.7:0/4022581018 learned_addr learned my addr 172.21.15.7:0/4022581018 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.872+0000 7f248652b700 1 -- 172.21.15.7:0/4022581018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2480108b70 con 0x7f2480108db0 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.872+0000 7f248652b700 1 --2- 172.21.15.7:0/4022581018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f2480108630 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f247c003f40 tx=0x7f247c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.873+0000 7f24777fe700 1 -- 172.21.15.7:0/4022581018 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f247c0038b0 con 0x7f2480108db0 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.873+0000 7f24777fe700 1 -- 172.21.15.7:0/4022581018 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f247c003a50 con 0x7f2480108db0 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.873+0000 7f248878f700 1 -- 172.21.15.7:0/4022581018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2480106d60 con 0x7f2480108db0 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.873+0000 7f24777fe700 1 -- 172.21.15.7:0/4022581018 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f247c01a5c0 con 0x7f2480108db0 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.873+0000 7f248878f700 1 -- 172.21.15.7:0/4022581018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24801072a0 con 0x7f2480108db0 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.874+0000 7f24777fe700 1 -- 172.21.15.7:0/4022581018 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f247c024430 con 0x7f2480108db0 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.874+0000 7f248878f700 1 -- 172.21.15.7:0/4022581018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24800640a0 con 0x7f2480108db0 2024-04-21T08:34:45.661 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.874+0000 7f24777fe700 1 --2- 172.21.15.7:0/4022581018 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f246c039560 0x7f246c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.874+0000 7f24777fe700 1 -- 172.21.15.7:0/4022581018 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f247c04d8b0 con 0x7f2480108db0 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.874+0000 7f2485d2a700 1 --2- 172.21.15.7:0/4022581018 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f246c039560 0x7f246c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.874+0000 7f2485d2a700 1 --2- 172.21.15.7:0/4022581018 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f246c039560 0x7f246c03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2470006fd0 tx=0x7f2470006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.879+0000 7f24777fe700 1 -- 172.21.15.7:0/4022581018 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f247c02d910 con 0x7f2480108db0 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:44.917+0000 7f24777fe700 1 -- 172.21.15.7:0/4022581018 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f247c02d3f0 con 0x7f2480108db0 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.105+0000 7f248878f700 1 -- 172.21.15.7:0/4022581018 --> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f2480103a50 con 0x7f246c039560 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.110+0000 7f24777fe700 1 -- 172.21.15.7:0/4022581018 <== mgr.14120 v2:172.21.15.7:6800/102274072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2480103a50 con 0x7f246c039560 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.112+0000 7f248878f700 1 -- 172.21.15.7:0/4022581018 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f246c039560 msgr2=0x7f246c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.112+0000 7f248878f700 1 --2- 172.21.15.7:0/4022581018 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f246c039560 0x7f246c03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2470006fd0 tx=0x7f2470006e40 comp rx=0 tx=0).stop 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.112+0000 7f248878f700 1 -- 172.21.15.7:0/4022581018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 msgr2=0x7f2480108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.112+0000 7f248878f700 1 --2- 172.21.15.7:0/4022581018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f2480108630 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f247c003f40 tx=0x7f247c011890 comp rx=0 tx=0).stop 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.112+0000 7f248878f700 1 -- 172.21.15.7:0/4022581018 shutdown_connections 2024-04-21T08:34:45.662 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.112+0000 7f248878f700 1 --2- 172.21.15.7:0/4022581018 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f246c039560 0x7f246c03ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:45.663 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.112+0000 7f248878f700 1 --2- 172.21.15.7:0/4022581018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2480108db0 0x7f2480108630 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:45.663 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.112+0000 7f248878f700 1 -- 172.21.15.7:0/4022581018 >> 172.21.15.7:0/4022581018 conn(0x7f2480102820 msgr2=0x7f2480103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:45.663 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.112+0000 7f248878f700 1 -- 172.21.15.7:0/4022581018 shutdown_connections 2024-04-21T08:34:45.663 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:45.112+0000 7f248878f700 1 -- 172.21.15.7:0/4022581018 wait complete. 2024-04-21T08:34:46.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:45 smithi007 ceph-mon[28477]: mgrmap e8: x(active, since 2s) 2024-04-21T08:34:46.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:45 smithi007 ceph-mon[28477]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:46.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:45 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:46.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:45 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:46.740 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-21T08:34:46.740 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.053+0000 7f204bc59700 1 Processor -- start 2024-04-21T08:34:46.740 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.053+0000 7f204bc59700 1 -- start start 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.053+0000 7f204bc59700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f204410b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.053+0000 7f204bc59700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f204407be40 con 0x7f2044108e50 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.054+0000 7f20499f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f204410b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.054+0000 7f20499f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f204410b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39748/0 (socket says 172.21.15.7:39748) 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.054+0000 7f20499f5700 1 -- 172.21.15.7:0/2984182160 learned_addr learned my addr 172.21.15.7:0/2984182160 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.054+0000 7f20499f5700 1 -- 172.21.15.7:0/2984182160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204410b780 con 0x7f2044108e50 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.054+0000 7f20499f5700 1 --2- 172.21.15.7:0/2984182160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f204410b240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f20400098a0 tx=0x7f2040009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36b07da01f787d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.054+0000 7f20489f3700 1 -- 172.21.15.7:0/2984182160 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2040004160 con 0x7f2044108e50 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.054+0000 7f20489f3700 1 -- 172.21.15.7:0/2984182160 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2040004300 con 0x7f2044108e50 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.054+0000 7f20489f3700 1 -- 172.21.15.7:0/2984182160 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2040011af0 con 0x7f2044108e50 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.055+0000 7f204bc59700 1 -- 172.21.15.7:0/2984182160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 msgr2=0x7f204410b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.055+0000 7f204bc59700 1 --2- 172.21.15.7:0/2984182160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f204410b240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f20400098a0 tx=0x7f2040009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:46.741 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.055+0000 7f204bc59700 1 -- 172.21.15.7:0/2984182160 shutdown_connections 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.055+0000 7f204bc59700 1 --2- 172.21.15.7:0/2984182160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f204410b240 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.055+0000 7f204bc59700 1 -- 172.21.15.7:0/2984182160 >> 172.21.15.7:0/2984182160 conn(0x7f2044102860 msgr2=0x7f2044104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.055+0000 7f204bc59700 1 -- 172.21.15.7:0/2984182160 shutdown_connections 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.055+0000 7f204bc59700 1 -- 172.21.15.7:0/2984182160 wait complete. 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.056+0000 7f204bc59700 1 Processor -- start 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.056+0000 7f204bc59700 1 -- start start 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.056+0000 7f204bc59700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f20441086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.056+0000 7f204bc59700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f204410c6f0 con 0x7f2044108e50 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.056+0000 7f20499f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f20441086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.056+0000 7f20499f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f20441086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39752/0 (socket says 172.21.15.7:39752) 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.056+0000 7f20499f5700 1 -- 172.21.15.7:0/919566107 learned_addr learned my addr 172.21.15.7:0/919566107 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.056+0000 7f20499f5700 1 -- 172.21.15.7:0/919566107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2044108c10 con 0x7f2044108e50 2024-04-21T08:34:46.742 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.057+0000 7f20499f5700 1 --2- 172.21.15.7:0/919566107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f20441086d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2040011ac0 tx=0x7f2040011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.057+0000 7f203affd700 1 -- 172.21.15.7:0/919566107 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2040003b20 con 0x7f2044108e50 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.057+0000 7f203affd700 1 -- 172.21.15.7:0/919566107 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f204001a8c0 con 0x7f2044108e50 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.057+0000 7f204bc59700 1 -- 172.21.15.7:0/919566107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2044106e00 con 0x7f2044108e50 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.057+0000 7f203affd700 1 -- 172.21.15.7:0/919566107 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20400248d0 con 0x7f2044108e50 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.057+0000 7f204bc59700 1 -- 172.21.15.7:0/919566107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2044107340 con 0x7f2044108e50 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.058+0000 7f203affd700 1 -- 172.21.15.7:0/919566107 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f2040022020 con 0x7f2044108e50 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.058+0000 7f204bc59700 1 -- 172.21.15.7:0/919566107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2044069b50 con 0x7f2044108e50 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.058+0000 7f203affd700 1 --2- 172.21.15.7:0/919566107 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2030039670 0x7f203003bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.058+0000 7f203affd700 1 -- 172.21.15.7:0/919566107 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f204004da80 con 0x7f2044108e50 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.058+0000 7f20491f4700 1 --2- 172.21.15.7:0/919566107 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2030039670 0x7f203003bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.058+0000 7f20491f4700 1 --2- 172.21.15.7:0/919566107 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2030039670 0x7f203003bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2034006fd0 tx=0x7f2034006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.063+0000 7f203affd700 1 -- 172.21.15.7:0/919566107 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f204000b5a0 con 0x7f2044108e50 2024-04-21T08:34:46.743 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.271+0000 7f204bc59700 1 -- 172.21.15.7:0/919566107 --> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f2044107ba0 con 0x7f2030039670 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.271+0000 7f203affd700 1 -- 172.21.15.7:0/919566107 <== mgr.14120 v2:172.21.15.7:6800/102274072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f2044107ba0 con 0x7f2030039670 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.273+0000 7f204bc59700 1 -- 172.21.15.7:0/919566107 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2030039670 msgr2=0x7f203003bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.273+0000 7f204bc59700 1 --2- 172.21.15.7:0/919566107 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2030039670 0x7f203003bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2034006fd0 tx=0x7f2034006e40 comp rx=0 tx=0).stop 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.273+0000 7f204bc59700 1 -- 172.21.15.7:0/919566107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 msgr2=0x7f20441086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.273+0000 7f204bc59700 1 --2- 172.21.15.7:0/919566107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f20441086d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2040011ac0 tx=0x7f2040011ba0 comp rx=0 tx=0).stop 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.273+0000 7f204bc59700 1 -- 172.21.15.7:0/919566107 shutdown_connections 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.273+0000 7f204bc59700 1 --2- 172.21.15.7:0/919566107 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2030039670 0x7f203003bb30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.273+0000 7f204bc59700 1 --2- 172.21.15.7:0/919566107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2044108e50 0x7f20441086d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.273+0000 7f204bc59700 1 -- 172.21.15.7:0/919566107 >> 172.21.15.7:0/919566107 conn(0x7f2044102860 msgr2=0x7f20441032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.273+0000 7f204bc59700 1 -- 172.21.15.7:0/919566107 shutdown_connections 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:46.273+0000 7f204bc59700 1 -- 172.21.15.7:0/919566107 wait complete. 2024-04-21T08:34:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:Generating ssh key... 2024-04-21T08:34:47.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:46 smithi007 ceph-mon[28477]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:47.651 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: Generating public/private rsa key pair. 2024-04-21T08:34:47.651 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: Your identification has been saved in /tmp/tmp53_mvh6f/key. 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: Your public key has been saved in /tmp/tmp53_mvh6f/key.pub. 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: The key fingerprint is: 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: SHA256:uXrTa4u30EscF34qzFFPfCzsXUIle/DyEbDT2sqqk6I ceph-c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: The key's randomart image is: 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: +---[RSA 3072]----+ 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: | .=o.| 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: | +o*.| 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: | ooOo*| 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: | . o =+Oo| 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: | S o o.+.o| 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: | * +.o. | 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: | o.O .o | 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: | oo*+o. | 2024-04-21T08:34:47.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: | Eo.o=B= | 2024-04-21T08:34:47.653 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:34:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: +----[SHA256]-----+ 2024-04-21T08:34:48.198 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.154+0000 7fcb62c08700 1 Processor -- start 2024-04-21T08:34:48.198 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.154+0000 7fcb62c08700 1 -- start start 2024-04-21T08:34:48.198 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.154+0000 7fcb62c08700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c1087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:48.198 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.154+0000 7fcb62c08700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c108da0 con 0x7fcb5c1083b0 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.154+0000 7fcb609a4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c1087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.154+0000 7fcb609a4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c1087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39762/0 (socket says 172.21.15.7:39762) 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.154+0000 7fcb609a4700 1 -- 172.21.15.7:0/1547263046 learned_addr learned my addr 172.21.15.7:0/1547263046 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.155+0000 7fcb609a4700 1 -- 172.21.15.7:0/1547263046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5c109560 con 0x7fcb5c1083b0 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.155+0000 7fcb609a4700 1 --2- 172.21.15.7:0/1547263046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c1087d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb500098a0 tx=0x7fcb50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68ac4f666ea6ee6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.155+0000 7fcb5b7fe700 1 -- 172.21.15.7:0/1547263046 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb50004160 con 0x7fcb5c1083b0 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.155+0000 7fcb5b7fe700 1 -- 172.21.15.7:0/1547263046 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcb50004300 con 0x7fcb5c1083b0 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.155+0000 7fcb5b7fe700 1 -- 172.21.15.7:0/1547263046 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb50011af0 con 0x7fcb5c1083b0 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.155+0000 7fcb62c08700 1 -- 172.21.15.7:0/1547263046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 msgr2=0x7fcb5c1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.155+0000 7fcb62c08700 1 --2- 172.21.15.7:0/1547263046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c1087d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb500098a0 tx=0x7fcb50009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.156+0000 7fcb62c08700 1 -- 172.21.15.7:0/1547263046 shutdown_connections 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.156+0000 7fcb62c08700 1 --2- 172.21.15.7:0/1547263046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c1087d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:48.199 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.156+0000 7fcb62c08700 1 -- 172.21.15.7:0/1547263046 >> 172.21.15.7:0/1547263046 conn(0x7fcb5c1039b0 msgr2=0x7fcb5c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.156+0000 7fcb62c08700 1 -- 172.21.15.7:0/1547263046 shutdown_connections 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.156+0000 7fcb62c08700 1 -- 172.21.15.7:0/1547263046 wait complete. 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.156+0000 7fcb62c08700 1 Processor -- start 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.156+0000 7fcb62c08700 1 -- start start 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.156+0000 7fcb62c08700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.156+0000 7fcb62c08700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c10c910 con 0x7fcb5c1083b0 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.157+0000 7fcb609a4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.157+0000 7fcb609a4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c19e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39772/0 (socket says 172.21.15.7:39772) 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.157+0000 7fcb609a4700 1 -- 172.21.15.7:0/4041752936 learned_addr learned my addr 172.21.15.7:0/4041752936 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.157+0000 7fcb609a4700 1 -- 172.21.15.7:0/4041752936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5c19e940 con 0x7fcb5c1083b0 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.157+0000 7fcb609a4700 1 --2- 172.21.15.7:0/4041752936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c19e400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcb50000c00 tx=0x7fcb50011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.157+0000 7fcb59ffb700 1 -- 172.21.15.7:0/4041752936 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb50003b20 con 0x7fcb5c1083b0 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.157+0000 7fcb62c08700 1 -- 172.21.15.7:0/4041752936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb5c19ebd0 con 0x7fcb5c1083b0 2024-04-21T08:34:48.200 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.157+0000 7fcb59ffb700 1 -- 172.21.15.7:0/4041752936 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcb5001a8c0 con 0x7fcb5c1083b0 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.157+0000 7fcb59ffb700 1 -- 172.21.15.7:0/4041752936 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb50024940 con 0x7fcb5c1083b0 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.157+0000 7fcb62c08700 1 -- 172.21.15.7:0/4041752936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb5c1a18c0 con 0x7fcb5c1083b0 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.158+0000 7fcb59ffb700 1 -- 172.21.15.7:0/4041752936 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fcb50022020 con 0x7fcb5c1083b0 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.158+0000 7fcb62c08700 1 -- 172.21.15.7:0/4041752936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb5c0640a0 con 0x7fcb5c1083b0 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.159+0000 7fcb59ffb700 1 --2- 172.21.15.7:0/4041752936 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fcb4c039670 0x7fcb4c03bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.159+0000 7fcb59ffb700 1 -- 172.21.15.7:0/4041752936 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcb5004d9a0 con 0x7fcb5c1083b0 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.159+0000 7fcb5bfff700 1 --2- 172.21.15.7:0/4041752936 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fcb4c039670 0x7fcb4c03bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.159+0000 7fcb5bfff700 1 --2- 172.21.15.7:0/4041752936 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fcb4c039670 0x7fcb4c03bb30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcb48006fd0 tx=0x7fcb48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.164+0000 7fcb59ffb700 1 -- 172.21.15.7:0/4041752936 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb5001a610 con 0x7fcb5c1083b0 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.371+0000 7fcb62c08700 1 -- 172.21.15.7:0/4041752936 --> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fcb5c104b20 con 0x7fcb4c039670 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.606+0000 7fcb59ffb700 1 -- 172.21.15.7:0/4041752936 <== mgr.14120 v2:172.21.15.7:6800/102274072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcb5c104b20 con 0x7fcb4c039670 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.608+0000 7fcb62c08700 1 -- 172.21.15.7:0/4041752936 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fcb4c039670 msgr2=0x7fcb4c03bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.608+0000 7fcb62c08700 1 --2- 172.21.15.7:0/4041752936 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fcb4c039670 0x7fcb4c03bb30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcb48006fd0 tx=0x7fcb48006e40 comp rx=0 tx=0).stop 2024-04-21T08:34:48.202 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.608+0000 7fcb62c08700 1 -- 172.21.15.7:0/4041752936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 msgr2=0x7fcb5c19e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:48.202 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.608+0000 7fcb62c08700 1 --2- 172.21.15.7:0/4041752936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c19e400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcb50000c00 tx=0x7fcb50011b70 comp rx=0 tx=0).stop 2024-04-21T08:34:48.202 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.608+0000 7fcb62c08700 1 -- 172.21.15.7:0/4041752936 shutdown_connections 2024-04-21T08:34:48.202 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.608+0000 7fcb62c08700 1 --2- 172.21.15.7:0/4041752936 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fcb4c039670 0x7fcb4c03bb30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:48.202 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.608+0000 7fcb62c08700 1 --2- 172.21.15.7:0/4041752936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb5c1083b0 0x7fcb5c19e400 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:48.202 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.608+0000 7fcb62c08700 1 -- 172.21.15.7:0/4041752936 >> 172.21.15.7:0/4041752936 conn(0x7fcb5c1039b0 msgr2=0x7fcb5c104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:48.202 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.608+0000 7fcb62c08700 1 -- 172.21.15.7:0/4041752936 shutdown_connections 2024-04-21T08:34:48.202 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:47.608+0000 7fcb62c08700 1 -- 172.21.15.7:0/4041752936 wait complete. 2024-04-21T08:34:48.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:48 smithi007 ceph-mon[28477]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:48.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:48 smithi007 ceph-mon[28477]: Generating ssh key... 2024-04-21T08:34:48.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:48 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:48.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:48 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:49.363 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjzqzXzxtIZ99IN3sOmnj7DnU0JE0BpkzRL9ohpiOaPyh+gi2Cb/KrV1SizIKCVft1cfHcMEKrF/C+Bk6ez+lOnudgbkxrmIOtHAqmqunPmiH/ixL6k+To1Aoeb67MuaaKZRiPPdWhuBR5okYRDB14u+yIcmW97agV5ah5KVlFwbMSKEomPLgo6RERXRgbckqG0I9ZUwM9QClMCFEKFbie9aebQSc8sggzQsKVP8AKXSGlPA8QUtoILuGgNkW5MrmNEXFS2F8tcRvd0bXV6yLPyfeqfoEqaDcUoxSCYVPv7FzXlv/FN7p+6ArnVs13YeSlHVlW1lqb5Qb7JPE54mWugtCygGmMeNi8nCiY9H//eKwllj0OPlZU/MCgvkbk8DLP96zZPIweCFbLNVFNOb6I0pffbKSUylPCG8ydT0OCifE6VfCbXKmgPzHTHdZDeF8CpyGP63E+Hpt+mi+gkAzAW3RckqBRRlrX/DKFyfVY5SIycOAtjwr1Q+9SOUERDQk= ceph-c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:49.363 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.588+0000 7f371a25a700 1 Processor -- start 2024-04-21T08:34:49.363 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.588+0000 7f371a25a700 1 -- start start 2024-04-21T08:34:49.363 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.589+0000 7f371a25a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371407c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:49.363 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.589+0000 7f371a25a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f371407c8a0 con 0x7f371407de70 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.589+0000 7f37137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371407c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.589+0000 7f37137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371407c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39778/0 (socket says 172.21.15.7:39778) 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.589+0000 7f37137fe700 1 -- 172.21.15.7:0/2601748439 learned_addr learned my addr 172.21.15.7:0/2601748439 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.589+0000 7f37137fe700 1 -- 172.21.15.7:0/2601748439 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371407ca20 con 0x7f371407de70 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.589+0000 7f37137fe700 1 --2- 172.21.15.7:0/2601748439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371407c2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f37040098a0 tx=0x7f3704009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7df38d1e09db318 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.590+0000 7f37127fc700 1 -- 172.21.15.7:0/2601748439 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3704004160 con 0x7f371407de70 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.590+0000 7f37127fc700 1 -- 172.21.15.7:0/2601748439 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3704004300 con 0x7f371407de70 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.590+0000 7f37127fc700 1 -- 172.21.15.7:0/2601748439 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3704011af0 con 0x7f371407de70 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.590+0000 7f371a25a700 1 -- 172.21.15.7:0/2601748439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 msgr2=0x7f371407c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.590+0000 7f371a25a700 1 --2- 172.21.15.7:0/2601748439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371407c2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f37040098a0 tx=0x7f3704009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.590+0000 7f371a25a700 1 -- 172.21.15.7:0/2601748439 shutdown_connections 2024-04-21T08:34:49.364 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.590+0000 7f371a25a700 1 --2- 172.21.15.7:0/2601748439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371407c2d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.590+0000 7f371a25a700 1 -- 172.21.15.7:0/2601748439 >> 172.21.15.7:0/2601748439 conn(0x7f37141038f0 msgr2=0x7f3714105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.591+0000 7f371a25a700 1 -- 172.21.15.7:0/2601748439 shutdown_connections 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.591+0000 7f371a25a700 1 -- 172.21.15.7:0/2601748439 wait complete. 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.591+0000 7f371a25a700 1 Processor -- start 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.591+0000 7f371a25a700 1 -- start start 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f371a25a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371419e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f371a25a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f371410e920 con 0x7f371407de70 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f37137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371419e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f37137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371419e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39794/0 (socket says 172.21.15.7:39794) 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f37137fe700 1 -- 172.21.15.7:0/4171584148 learned_addr learned my addr 172.21.15.7:0/4171584148 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f37137fe700 1 -- 172.21.15.7:0/4171584148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371419e8d0 con 0x7f371407de70 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f37137fe700 1 --2- 172.21.15.7:0/4171584148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371419e390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3704011ac0 tx=0x7f3704011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:49.365 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f3710ff9700 1 -- 172.21.15.7:0/4171584148 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3704003b20 con 0x7f371407de70 2024-04-21T08:34:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f3710ff9700 1 -- 172.21.15.7:0/4171584148 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f370401a8c0 con 0x7f371407de70 2024-04-21T08:34:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f371a25a700 1 -- 172.21.15.7:0/4171584148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f371419eb60 con 0x7f371407de70 2024-04-21T08:34:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f3710ff9700 1 -- 172.21.15.7:0/4171584148 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37040248d0 con 0x7f371407de70 2024-04-21T08:34:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.592+0000 7f371a25a700 1 -- 172.21.15.7:0/4171584148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37141a1850 con 0x7f371407de70 2024-04-21T08:34:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.593+0000 7f3710ff9700 1 -- 172.21.15.7:0/4171584148 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3704022020 con 0x7f371407de70 2024-04-21T08:34:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.593+0000 7f371a25a700 1 -- 172.21.15.7:0/4171584148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37140640a0 con 0x7f371407de70 2024-04-21T08:34:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.593+0000 7f3710ff9700 1 --2- 172.21.15.7:0/4171584148 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f36fc039670 0x7f36fc03bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.593+0000 7f3710ff9700 1 -- 172.21.15.7:0/4171584148 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f370404d8a0 con 0x7f371407de70 2024-04-21T08:34:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.594+0000 7f3712ffd700 1 --2- 172.21.15.7:0/4171584148 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f36fc039670 0x7f36fc03bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.594+0000 7f3712ffd700 1 --2- 172.21.15.7:0/4171584148 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f36fc039670 0x7f36fc03bb30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3708006fd0 tx=0x7f3708006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.599+0000 7f3710ff9700 1 -- 172.21.15.7:0/4171584148 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f370400b5a0 con 0x7f371407de70 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.808+0000 7f371a25a700 1 -- 172.21.15.7:0/4171584148 --> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f3714104a40 con 0x7f36fc039670 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.809+0000 7f3710ff9700 1 -- 172.21.15.7:0/4171584148 <== mgr.14120 v2:172.21.15.7:6800/102274072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f3714104a40 con 0x7f36fc039670 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.811+0000 7f371a25a700 1 -- 172.21.15.7:0/4171584148 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f36fc039670 msgr2=0x7f36fc03bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.811+0000 7f371a25a700 1 --2- 172.21.15.7:0/4171584148 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f36fc039670 0x7f36fc03bb30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3708006fd0 tx=0x7f3708006e40 comp rx=0 tx=0).stop 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.811+0000 7f371a25a700 1 -- 172.21.15.7:0/4171584148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 msgr2=0x7f371419e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.811+0000 7f371a25a700 1 --2- 172.21.15.7:0/4171584148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371419e390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3704011ac0 tx=0x7f3704011ba0 comp rx=0 tx=0).stop 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.811+0000 7f371a25a700 1 -- 172.21.15.7:0/4171584148 shutdown_connections 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.811+0000 7f371a25a700 1 --2- 172.21.15.7:0/4171584148 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f36fc039670 0x7f36fc03bb30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.811+0000 7f371a25a700 1 --2- 172.21.15.7:0/4171584148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f371407de70 0x7f371419e390 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.811+0000 7f371a25a700 1 -- 172.21.15.7:0/4171584148 >> 172.21.15.7:0/4171584148 conn(0x7f37141038f0 msgr2=0x7f3714104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.811+0000 7f371a25a700 1 -- 172.21.15.7:0/4171584148 shutdown_connections 2024-04-21T08:34:49.367 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:48.811+0000 7f371a25a700 1 -- 172.21.15.7:0/4171584148 wait complete. 2024-04-21T08:34:49.368 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:34:49.368 INFO:teuthology.orchestra.run.smithi007.stdout:Adding key to root@localhost authorized_keys... 2024-04-21T08:34:49.368 INFO:teuthology.orchestra.run.smithi007.stdout:Adding host smithi007... 2024-04-21T08:34:49.765 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:49 smithi007 ceph-mon[28477]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:50.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:50 smithi007 ceph-mon[28477]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:50.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:50 smithi007 ceph-mon[28477]: Deploying cephadm binary to smithi007 2024-04-21T08:34:54.803 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Added host 'smithi007' with addr '172.21.15.7' 2024-04-21T08:34:54.804 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.803+0000 7fe938168700 1 Processor -- start 2024-04-21T08:34:54.804 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.804+0000 7fe938168700 1 -- start start 2024-04-21T08:34:54.804 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.804+0000 7fe938168700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93007c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:54.804 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.804+0000 7fe938168700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe93007c8e0 con 0x7fe93007de60 2024-04-21T08:34:54.804 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.804+0000 7fe935f04700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93007c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:54.804 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.804+0000 7fe935f04700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93007c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39798/0 (socket says 172.21.15.7:39798) 2024-04-21T08:34:54.804 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.804+0000 7fe935f04700 1 -- 172.21.15.7:0/4080894062 learned_addr learned my addr 172.21.15.7:0/4080894062 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:54.804 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.804+0000 7fe935f04700 1 -- 172.21.15.7:0/4080894062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93007ca60 con 0x7fe93007de60 2024-04-21T08:34:54.804 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.805+0000 7fe935f04700 1 --2- 172.21.15.7:0/4080894062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93007c310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0098a0 tx=0x7fe92c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c9338a2f8bda21 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:54.804 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.805+0000 7fe934f02700 1 -- 172.21.15.7:0/4080894062 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe92c004160 con 0x7fe93007de60 2024-04-21T08:34:54.804 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.805+0000 7fe934f02700 1 -- 172.21.15.7:0/4080894062 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe92c004300 con 0x7fe93007de60 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.805+0000 7fe934f02700 1 -- 172.21.15.7:0/4080894062 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe92c011af0 con 0x7fe93007de60 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.805+0000 7fe938168700 1 -- 172.21.15.7:0/4080894062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 msgr2=0x7fe93007c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.805+0000 7fe938168700 1 --2- 172.21.15.7:0/4080894062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93007c310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0098a0 tx=0x7fe92c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.805+0000 7fe938168700 1 -- 172.21.15.7:0/4080894062 shutdown_connections 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.805+0000 7fe938168700 1 --2- 172.21.15.7:0/4080894062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93007c310 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.805+0000 7fe938168700 1 -- 172.21.15.7:0/4080894062 >> 172.21.15.7:0/4080894062 conn(0x7fe9301039b0 msgr2=0x7fe930105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.806+0000 7fe938168700 1 -- 172.21.15.7:0/4080894062 shutdown_connections 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.806+0000 7fe938168700 1 -- 172.21.15.7:0/4080894062 wait complete. 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.806+0000 7fe938168700 1 Processor -- start 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.806+0000 7fe938168700 1 -- start start 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.806+0000 7fe938168700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93019e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.806+0000 7fe938168700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe93010e8e0 con 0x7fe93007de60 2024-04-21T08:34:54.805 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.806+0000 7fe935f04700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93019e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.807+0000 7fe935f04700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93019e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39804/0 (socket says 172.21.15.7:39804) 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.807+0000 7fe935f04700 1 -- 172.21.15.7:0/2635452227 learned_addr learned my addr 172.21.15.7:0/2635452227 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.807+0000 7fe935f04700 1 -- 172.21.15.7:0/2635452227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93019e8f0 con 0x7fe93007de60 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.807+0000 7fe935f04700 1 --2- 172.21.15.7:0/2635452227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93019e3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe92c003f40 tx=0x7fe92c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.807+0000 7fe926ffd700 1 -- 172.21.15.7:0/2635452227 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe92c0038b0 con 0x7fe93007de60 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.807+0000 7fe938168700 1 -- 172.21.15.7:0/2635452227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe93019eb80 con 0x7fe93007de60 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.807+0000 7fe938168700 1 -- 172.21.15.7:0/2635452227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9301a1870 con 0x7fe93007de60 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.807+0000 7fe926ffd700 1 -- 172.21.15.7:0/2635452227 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe92c003a50 con 0x7fe93007de60 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.807+0000 7fe926ffd700 1 -- 172.21.15.7:0/2635452227 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe92c024c80 con 0x7fe93007de60 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.808+0000 7fe926ffd700 1 -- 172.21.15.7:0/2635452227 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe92c01a430 con 0x7fe93007de60 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.808+0000 7fe938168700 1 -- 172.21.15.7:0/2635452227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe914005360 con 0x7fe93007de60 2024-04-21T08:34:54.806 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.808+0000 7fe926ffd700 1 --2- 172.21.15.7:0/2635452227 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fe91c039620 0x7fe91c03bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.808+0000 7fe926ffd700 1 -- 172.21.15.7:0/2635452227 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe92c04d9f0 con 0x7fe93007de60 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.808+0000 7fe935703700 1 --2- 172.21.15.7:0/2635452227 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fe91c039620 0x7fe91c03bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.809+0000 7fe935703700 1 --2- 172.21.15.7:0/2635452227 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fe91c039620 0x7fe91c03bae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe920006fd0 tx=0x7fe920006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:49.813+0000 7fe926ffd700 1 -- 172.21.15.7:0/2635452227 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe92c016df0 con 0x7fe93007de60 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.031+0000 7fe938168700 1 -- 172.21.15.7:0/2635452227 --> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}) -- 0x7fe914000bf0 con 0x7fe91c039620 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:54.222+0000 7fe926ffd700 1 -- 172.21.15.7:0/2635452227 <== mgr.14120 v2:172.21.15.7:6800/102274072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7fe914000bf0 con 0x7fe91c039620 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:54.225+0000 7fe938168700 1 -- 172.21.15.7:0/2635452227 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fe91c039620 msgr2=0x7fe91c03bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:54.225+0000 7fe938168700 1 --2- 172.21.15.7:0/2635452227 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fe91c039620 0x7fe91c03bae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe920006fd0 tx=0x7fe920006e40 comp rx=0 tx=0).stop 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:54.225+0000 7fe938168700 1 -- 172.21.15.7:0/2635452227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 msgr2=0x7fe93019e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:54.225+0000 7fe938168700 1 --2- 172.21.15.7:0/2635452227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93019e3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe92c003f40 tx=0x7fe92c011890 comp rx=0 tx=0).stop 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:54.225+0000 7fe938168700 1 -- 172.21.15.7:0/2635452227 shutdown_connections 2024-04-21T08:34:54.807 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:54.225+0000 7fe938168700 1 --2- 172.21.15.7:0/2635452227 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fe91c039620 0x7fe91c03bae0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:54.808 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:54.225+0000 7fe938168700 1 --2- 172.21.15.7:0/2635452227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93007de60 0x7fe93019e3b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:54.808 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:54.225+0000 7fe938168700 1 -- 172.21.15.7:0/2635452227 >> 172.21.15.7:0/2635452227 conn(0x7fe9301039b0 msgr2=0x7fe930104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:54.808 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:54.225+0000 7fe938168700 1 -- 172.21.15.7:0/2635452227 shutdown_connections 2024-04-21T08:34:54.808 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:54.225+0000 7fe938168700 1 -- 172.21.15.7:0/2635452227 wait complete. 2024-04-21T08:34:54.808 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mon service... 2024-04-21T08:34:55.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:55 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:55.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:55 smithi007 ceph-mon[28477]: Added host smithi007 2024-04-21T08:34:55.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:55 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.261+0000 7f4de9aed700 1 Processor -- start 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.261+0000 7f4de9aed700 1 -- start start 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.261+0000 7f4de9aed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de4106a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.261+0000 7f4de9aed700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4de4107030 con 0x7f4de4106640 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.261+0000 7f4de37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de4106a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.261+0000 7f4de37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de4106a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46638/0 (socket says 172.21.15.7:46638) 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.261+0000 7f4de37fe700 1 -- 172.21.15.7:0/2667581593 learned_addr learned my addr 172.21.15.7:0/2667581593 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.261+0000 7f4de37fe700 1 -- 172.21.15.7:0/2667581593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de4109670 con 0x7f4de4106640 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.261+0000 7f4de37fe700 1 --2- 172.21.15.7:0/2667581593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de4106a60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4dd40098a0 tx=0x7f4dd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5cef154f3530940 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.262+0000 7f4de27fc700 1 -- 172.21.15.7:0/2667581593 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4dd4004160 con 0x7f4de4106640 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.262+0000 7f4de27fc700 1 -- 172.21.15.7:0/2667581593 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4dd4004300 con 0x7f4de4106640 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.262+0000 7f4de27fc700 1 -- 172.21.15.7:0/2667581593 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4dd4011af0 con 0x7f4de4106640 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.262+0000 7f4de9aed700 1 -- 172.21.15.7:0/2667581593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 msgr2=0x7f4de4106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.262+0000 7f4de9aed700 1 --2- 172.21.15.7:0/2667581593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de4106a60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4dd40098a0 tx=0x7f4dd4009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.262+0000 7f4de9aed700 1 -- 172.21.15.7:0/2667581593 shutdown_connections 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.262+0000 7f4de9aed700 1 --2- 172.21.15.7:0/2667581593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de4106a60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.262+0000 7f4de9aed700 1 -- 172.21.15.7:0/2667581593 >> 172.21.15.7:0/2667581593 conn(0x7f4de4101fb0 msgr2=0x7f4de41043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.262+0000 7f4de9aed700 1 -- 172.21.15.7:0/2667581593 shutdown_connections 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.262+0000 7f4de9aed700 1 -- 172.21.15.7:0/2667581593 wait complete. 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.263+0000 7f4de9aed700 1 Processor -- start 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.263+0000 7f4de9aed700 1 -- start start 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.263+0000 7f4de9aed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de419c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.263+0000 7f4de9aed700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4de410c8a0 con 0x7f4de4106640 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.263+0000 7f4de37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de419c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.263+0000 7f4de37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de419c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46640/0 (socket says 172.21.15.7:46640) 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.263+0000 7f4de37fe700 1 -- 172.21.15.7:0/3755189663 learned_addr learned my addr 172.21.15.7:0/3755189663 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.264+0000 7f4de37fe700 1 -- 172.21.15.7:0/3755189663 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de419c750 con 0x7f4de4106640 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.264+0000 7f4de37fe700 1 --2- 172.21.15.7:0/3755189663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de419c210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4000c00 tx=0x7f4dd4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.264+0000 7f4de0ff9700 1 -- 172.21.15.7:0/3755189663 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4dd4003b20 con 0x7f4de4106640 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.264+0000 7f4de0ff9700 1 -- 172.21.15.7:0/3755189663 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4dd401a8c0 con 0x7f4de4106640 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.264+0000 7f4de9aed700 1 -- 172.21.15.7:0/3755189663 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4de419c9e0 con 0x7f4de4106640 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.264+0000 7f4de0ff9700 1 -- 172.21.15.7:0/3755189663 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4dd4024940 con 0x7f4de4106640 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.264+0000 7f4de9aed700 1 -- 172.21.15.7:0/3755189663 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4de419f6d0 con 0x7f4de4106640 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.265+0000 7f4de0ff9700 1 -- 172.21.15.7:0/3755189663 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4dd4022020 con 0x7f4de4106640 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.265+0000 7f4de9aed700 1 -- 172.21.15.7:0/3755189663 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de40640a0 con 0x7f4de4106640 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.265+0000 7f4de0ff9700 1 --2- 172.21.15.7:0/3755189663 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4dcc039670 0x7f4dcc03bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.265+0000 7f4de0ff9700 1 -- 172.21.15.7:0/3755189663 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4dd404d990 con 0x7f4de4106640 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.265+0000 7f4de2ffd700 1 --2- 172.21.15.7:0/3755189663 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4dcc039670 0x7f4dcc03bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.266+0000 7f4de2ffd700 1 --2- 172.21.15.7:0/3755189663 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4dcc039670 0x7f4dcc03bb30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8006fd0 tx=0x7f4dd8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.271+0000 7f4de0ff9700 1 -- 172.21.15.7:0/3755189663 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4dd401aae0 con 0x7f4de4106640 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.509+0000 7f4de9aed700 1 -- 172.21.15.7:0/3755189663 --> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4de4103120 con 0x7f4dcc039670 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.514+0000 7f4de0ff9700 1 -- 172.21.15.7:0/3755189663 <== mgr.14120 v2:172.21.15.7:6800/102274072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4de4103120 con 0x7f4dcc039670 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.516+0000 7f4de9aed700 1 -- 172.21.15.7:0/3755189663 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4dcc039670 msgr2=0x7f4dcc03bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.516+0000 7f4de9aed700 1 --2- 172.21.15.7:0/3755189663 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4dcc039670 0x7f4dcc03bb30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8006fd0 tx=0x7f4dd8006e40 comp rx=0 tx=0).stop 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.516+0000 7f4de9aed700 1 -- 172.21.15.7:0/3755189663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 msgr2=0x7f4de419c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.516+0000 7f4de9aed700 1 --2- 172.21.15.7:0/3755189663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de419c210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4000c00 tx=0x7f4dd4011b70 comp rx=0 tx=0).stop 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.516+0000 7f4de9aed700 1 -- 172.21.15.7:0/3755189663 shutdown_connections 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.516+0000 7f4de9aed700 1 --2- 172.21.15.7:0/3755189663 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4dcc039670 0x7f4dcc03bb30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.516+0000 7f4de9aed700 1 --2- 172.21.15.7:0/3755189663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4de4106640 0x7f4de419c210 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.516+0000 7f4de9aed700 1 -- 172.21.15.7:0/3755189663 >> 172.21.15.7:0/3755189663 conn(0x7f4de4101fb0 msgr2=0x7f4de4102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.516+0000 7f4de9aed700 1 -- 172.21.15.7:0/3755189663 shutdown_connections 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.516+0000 7f4de9aed700 1 -- 172.21.15.7:0/3755189663 wait complete. 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mgr service... 2024-04-21T08:34:56.785 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:56 smithi007 ceph-mon[28477]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:56.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:56 smithi007 ceph-mon[28477]: Saving service mon spec with placement count:5 2024-04-21T08:34:56.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:56 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:58.167 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-21T08:34:58.168 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.173+0000 7efd5ee42700 1 Processor -- start 2024-04-21T08:34:58.168 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.173+0000 7efd5ee42700 1 -- start start 2024-04-21T08:34:58.168 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.173+0000 7efd5ee42700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd58107670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:58.168 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.173+0000 7efd5ee42700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efd58107c40 con 0x7efd58107250 2024-04-21T08:34:58.168 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.173+0000 7efd5cbde700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd58107670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:58.168 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.173+0000 7efd5cbde700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd58107670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46652/0 (socket says 172.21.15.7:46652) 2024-04-21T08:34:58.168 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.173+0000 7efd5cbde700 1 -- 172.21.15.7:0/4015850590 learned_addr learned my addr 172.21.15.7:0/4015850590 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:58.168 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.173+0000 7efd5cbde700 1 -- 172.21.15.7:0/4015850590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd58108450 con 0x7efd58107250 2024-04-21T08:34:58.168 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.174+0000 7efd5cbde700 1 --2- 172.21.15.7:0/4015850590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd58107670 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efd480098a0 tx=0x7efd48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f21b1d3ddcf8a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:58.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.174+0000 7efd577fe700 1 -- 172.21.15.7:0/4015850590 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd48004160 con 0x7efd58107250 2024-04-21T08:34:58.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.174+0000 7efd577fe700 1 -- 172.21.15.7:0/4015850590 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efd48004300 con 0x7efd58107250 2024-04-21T08:34:58.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.174+0000 7efd577fe700 1 -- 172.21.15.7:0/4015850590 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd48011af0 con 0x7efd58107250 2024-04-21T08:34:58.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.174+0000 7efd5ee42700 1 -- 172.21.15.7:0/4015850590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 msgr2=0x7efd58107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:58.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.174+0000 7efd5ee42700 1 --2- 172.21.15.7:0/4015850590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd58107670 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efd480098a0 tx=0x7efd48009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:58.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.174+0000 7efd5ee42700 1 -- 172.21.15.7:0/4015850590 shutdown_connections 2024-04-21T08:34:58.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.174+0000 7efd5ee42700 1 --2- 172.21.15.7:0/4015850590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd58107670 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:58.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.174+0000 7efd5ee42700 1 -- 172.21.15.7:0/4015850590 >> 172.21.15.7:0/4015850590 conn(0x7efd58102890 msgr2=0x7efd58104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:58.169 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.174+0000 7efd5ee42700 1 -- 172.21.15.7:0/4015850590 shutdown_connections 2024-04-21T08:34:58.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.174+0000 7efd5ee42700 1 -- 172.21.15.7:0/4015850590 wait complete. 2024-04-21T08:34:58.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.175+0000 7efd5ee42700 1 Processor -- start 2024-04-21T08:34:58.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.175+0000 7efd5ee42700 1 -- start start 2024-04-21T08:34:58.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.175+0000 7efd5ee42700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd5819c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:58.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.175+0000 7efd5ee42700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efd5810b800 con 0x7efd58107250 2024-04-21T08:34:58.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.175+0000 7efd5cbde700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd5819c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:58.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.175+0000 7efd5cbde700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd5819c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46654/0 (socket says 172.21.15.7:46654) 2024-04-21T08:34:58.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.175+0000 7efd5cbde700 1 -- 172.21.15.7:0/3982248862 learned_addr learned my addr 172.21.15.7:0/3982248862 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:58.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.176+0000 7efd5cbde700 1 -- 172.21.15.7:0/3982248862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5819c780 con 0x7efd58107250 2024-04-21T08:34:58.170 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.176+0000 7efd5cbde700 1 --2- 172.21.15.7:0/3982248862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd5819c240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efd48003f40 tx=0x7efd48011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:58.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.176+0000 7efd55ffb700 1 -- 172.21.15.7:0/3982248862 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd480038b0 con 0x7efd58107250 2024-04-21T08:34:58.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.176+0000 7efd55ffb700 1 -- 172.21.15.7:0/3982248862 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efd48003a50 con 0x7efd58107250 2024-04-21T08:34:58.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.176+0000 7efd5ee42700 1 -- 172.21.15.7:0/3982248862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd5819ca10 con 0x7efd58107250 2024-04-21T08:34:58.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.176+0000 7efd55ffb700 1 -- 172.21.15.7:0/3982248862 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd4801a5c0 con 0x7efd58107250 2024-04-21T08:34:58.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.176+0000 7efd5ee42700 1 -- 172.21.15.7:0/3982248862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd5819f700 con 0x7efd58107250 2024-04-21T08:34:58.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.177+0000 7efd55ffb700 1 -- 172.21.15.7:0/3982248862 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7efd48024430 con 0x7efd58107250 2024-04-21T08:34:58.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.177+0000 7efd55ffb700 1 --2- 172.21.15.7:0/3982248862 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7efd40039670 0x7efd4003bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:58.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.177+0000 7efd5ee42700 1 -- 172.21.15.7:0/3982248862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd580640a0 con 0x7efd58107250 2024-04-21T08:34:58.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.177+0000 7efd57fff700 1 --2- 172.21.15.7:0/3982248862 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7efd40039670 0x7efd4003bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:58.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.177+0000 7efd55ffb700 1 -- 172.21.15.7:0/3982248862 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efd4804dae0 con 0x7efd58107250 2024-04-21T08:34:58.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.178+0000 7efd57fff700 1 --2- 172.21.15.7:0/3982248862 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7efd40039670 0x7efd4003bb30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efd4c006fd0 tx=0x7efd4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:58.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.183+0000 7efd55ffb700 1 -- 172.21.15.7:0/3982248862 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd4801f1a0 con 0x7efd58107250 2024-04-21T08:34:58.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.403+0000 7efd5ee42700 1 -- 172.21.15.7:0/3982248862 --> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7efd5802e2e0 con 0x7efd40039670 2024-04-21T08:34:58.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.407+0000 7efd55ffb700 1 -- 172.21.15.7:0/3982248862 <== mgr.14120 v2:172.21.15.7:6800/102274072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7efd5802e2e0 con 0x7efd40039670 2024-04-21T08:34:58.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.409+0000 7efd5ee42700 1 -- 172.21.15.7:0/3982248862 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7efd40039670 msgr2=0x7efd4003bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:58.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.409+0000 7efd5ee42700 1 --2- 172.21.15.7:0/3982248862 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7efd40039670 0x7efd4003bb30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efd4c006fd0 tx=0x7efd4c006e40 comp rx=0 tx=0).stop 2024-04-21T08:34:58.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.409+0000 7efd5ee42700 1 -- 172.21.15.7:0/3982248862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 msgr2=0x7efd5819c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:58.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.409+0000 7efd5ee42700 1 --2- 172.21.15.7:0/3982248862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd5819c240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efd48003f40 tx=0x7efd48011890 comp rx=0 tx=0).stop 2024-04-21T08:34:58.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.409+0000 7efd5ee42700 1 -- 172.21.15.7:0/3982248862 shutdown_connections 2024-04-21T08:34:58.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.409+0000 7efd5ee42700 1 --2- 172.21.15.7:0/3982248862 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7efd40039670 0x7efd4003bb30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:58.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.409+0000 7efd5ee42700 1 --2- 172.21.15.7:0/3982248862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd58107250 0x7efd5819c240 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:58.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.409+0000 7efd5ee42700 1 -- 172.21.15.7:0/3982248862 >> 172.21.15.7:0/3982248862 conn(0x7efd58102890 msgr2=0x7efd58103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:58.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.409+0000 7efd5ee42700 1 -- 172.21.15.7:0/3982248862 shutdown_connections 2024-04-21T08:34:58.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.409+0000 7efd5ee42700 1 -- 172.21.15.7:0/3982248862 wait complete. 2024-04-21T08:34:58.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:58 smithi007 ceph-mon[28477]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:34:58.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:58 smithi007 ceph-mon[28477]: Saving service mgr spec with placement count:2 2024-04-21T08:34:58.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:58 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:58.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:58 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:58.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:58 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:34:59.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.620+0000 7f2f80e75700 1 Processor -- start 2024-04-21T08:34:59.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.620+0000 7f2f80e75700 1 -- start start 2024-04-21T08:34:59.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.620+0000 7f2f80e75700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c10b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:59.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.620+0000 7f2f80e75700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c07bea0 con 0x7f2f7c108db0 2024-04-21T08:34:59.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.621+0000 7f2f7a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c10b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:59.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.621+0000 7f2f7a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c10b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46656/0 (socket says 172.21.15.7:46656) 2024-04-21T08:34:59.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.621+0000 7f2f7a59c700 1 -- 172.21.15.7:0/3144512235 learned_addr learned my addr 172.21.15.7:0/3144512235 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:59.693 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.621+0000 7f2f7a59c700 1 -- 172.21.15.7:0/3144512235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7c10b6e0 con 0x7f2f7c108db0 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.621+0000 7f2f7a59c700 1 --2- 172.21.15.7:0/3144512235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c10b1a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2f700098a0 tx=0x7f2f70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea548e7f0193fee server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.621+0000 7f2f7959a700 1 -- 172.21.15.7:0/3144512235 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f70004160 con 0x7f2f7c108db0 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.621+0000 7f2f7959a700 1 -- 172.21.15.7:0/3144512235 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f70004300 con 0x7f2f7c108db0 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.621+0000 7f2f7959a700 1 -- 172.21.15.7:0/3144512235 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f70011a20 con 0x7f2f7c108db0 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.622+0000 7f2f80e75700 1 -- 172.21.15.7:0/3144512235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 msgr2=0x7f2f7c10b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.622+0000 7f2f80e75700 1 --2- 172.21.15.7:0/3144512235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c10b1a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2f700098a0 tx=0x7f2f70009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.622+0000 7f2f80e75700 1 -- 172.21.15.7:0/3144512235 shutdown_connections 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.622+0000 7f2f80e75700 1 --2- 172.21.15.7:0/3144512235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c10b1a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.622+0000 7f2f80e75700 1 -- 172.21.15.7:0/3144512235 >> 172.21.15.7:0/3144512235 conn(0x7f2f7c102820 msgr2=0x7f2f7c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.622+0000 7f2f80e75700 1 -- 172.21.15.7:0/3144512235 shutdown_connections 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.622+0000 7f2f80e75700 1 -- 172.21.15.7:0/3144512235 wait complete. 2024-04-21T08:34:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.623+0000 7f2f80e75700 1 Processor -- start 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.623+0000 7f2f80e75700 1 -- start start 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.623+0000 7f2f80e75700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c19c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.623+0000 7f2f80e75700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c10c6f0 con 0x7f2f7c108db0 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.623+0000 7f2f7a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c19c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.623+0000 7f2f7a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c19c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46666/0 (socket says 172.21.15.7:46666) 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.623+0000 7f2f7a59c700 1 -- 172.21.15.7:0/872168655 learned_addr learned my addr 172.21.15.7:0/872168655 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.624+0000 7f2f7a59c700 1 -- 172.21.15.7:0/872168655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7c19c6c0 con 0x7f2f7c108db0 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.624+0000 7f2f7a59c700 1 --2- 172.21.15.7:0/872168655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c19c180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f700116e0 tx=0x7f2f70011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.624+0000 7f2f6b7fe700 1 -- 172.21.15.7:0/872168655 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f70003a60 con 0x7f2f7c108db0 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.624+0000 7f2f6b7fe700 1 -- 172.21.15.7:0/872168655 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f7001b8f0 con 0x7f2f7c108db0 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.624+0000 7f2f80e75700 1 -- 172.21.15.7:0/872168655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7c19c950 con 0x7f2f7c108db0 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.625+0000 7f2f6b7fe700 1 -- 172.21.15.7:0/872168655 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f70019860 con 0x7f2f7c108db0 2024-04-21T08:34:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.625+0000 7f2f80e75700 1 -- 172.21.15.7:0/872168655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f7c19f640 con 0x7f2f7c108db0 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.625+0000 7f2f6b7fe700 1 -- 172.21.15.7:0/872168655 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f2f70019b30 con 0x7f2f7c108db0 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.625+0000 7f2f6b7fe700 1 --2- 172.21.15.7:0/872168655 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2f64039670 0x7f2f6403bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.625+0000 7f2f6b7fe700 1 -- 172.21.15.7:0/872168655 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2f7001bda0 con 0x7f2f7c108db0 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.625+0000 7f2f79d9b700 1 --2- 172.21.15.7:0/872168655 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2f64039670 0x7f2f6403bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.626+0000 7f2f80e75700 1 -- 172.21.15.7:0/872168655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f7c0640a0 con 0x7f2f7c108db0 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.626+0000 7f2f79d9b700 1 --2- 172.21.15.7:0/872168655 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2f64039670 0x7f2f6403bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c006fd0 tx=0x7f2f6c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.631+0000 7f2f6b7fe700 1 -- 172.21.15.7:0/872168655 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f7001f030 con 0x7f2f7c108db0 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.951+0000 7f2f80e75700 1 -- 172.21.15.7:0/872168655 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f2f7c19fb80 con 0x7f2f7c108db0 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.953+0000 7f2f6b7fe700 1 -- 172.21.15.7:0/872168655 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f2f70022e00 con 0x7f2f7c108db0 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.955+0000 7f2f80e75700 1 -- 172.21.15.7:0/872168655 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2f64039670 msgr2=0x7f2f6403bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.955+0000 7f2f80e75700 1 --2- 172.21.15.7:0/872168655 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2f64039670 0x7f2f6403bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c006fd0 tx=0x7f2f6c006e40 comp rx=0 tx=0).stop 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.955+0000 7f2f80e75700 1 -- 172.21.15.7:0/872168655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 msgr2=0x7f2f7c19c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:59.696 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.955+0000 7f2f80e75700 1 --2- 172.21.15.7:0/872168655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c19c180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f700116e0 tx=0x7f2f70011aa0 comp rx=0 tx=0).stop 2024-04-21T08:34:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.955+0000 7f2f80e75700 1 -- 172.21.15.7:0/872168655 shutdown_connections 2024-04-21T08:34:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.955+0000 7f2f80e75700 1 --2- 172.21.15.7:0/872168655 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f2f64039670 0x7f2f6403bb30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.955+0000 7f2f80e75700 1 --2- 172.21.15.7:0/872168655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f7c108db0 0x7f2f7c19c180 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.955+0000 7f2f80e75700 1 -- 172.21.15.7:0/872168655 >> 172.21.15.7:0/872168655 conn(0x7f2f7c102820 msgr2=0x7f2f7c103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.956+0000 7f2f80e75700 1 -- 172.21.15.7:0/872168655 shutdown_connections 2024-04-21T08:34:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:58.956+0000 7f2f80e75700 1 -- 172.21.15.7:0/872168655 wait complete. 2024-04-21T08:35:00.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:34:59 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/872168655' entity='client.admin' 2024-04-21T08:35:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.131+0000 7fbe255d9700 1 Processor -- start 2024-04-21T08:35:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.131+0000 7fbe255d9700 1 -- start start 2024-04-21T08:35:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.132+0000 7fbe255d9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:00.895 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.132+0000 7fbe255d9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2007c810 con 0x7fbe2007de20 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.132+0000 7fbe1effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.132+0000 7fbe1effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46670/0 (socket says 172.21.15.7:46670) 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.132+0000 7fbe1effd700 1 -- 172.21.15.7:0/3118198197 learned_addr learned my addr 172.21.15.7:0/3118198197 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.132+0000 7fbe1effd700 1 -- 172.21.15.7:0/3118198197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2007c990 con 0x7fbe2007de20 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.132+0000 7fbe1effd700 1 --2- 172.21.15.7:0/3118198197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2007c2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe100098a0 tx=0x7fbe10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa953a722c0df71e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.132+0000 7fbe1dffb700 1 -- 172.21.15.7:0/3118198197 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe10004160 con 0x7fbe2007de20 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.132+0000 7fbe1dffb700 1 -- 172.21.15.7:0/3118198197 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe10004300 con 0x7fbe2007de20 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.132+0000 7fbe1dffb700 1 -- 172.21.15.7:0/3118198197 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe10011a20 con 0x7fbe2007de20 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.133+0000 7fbe255d9700 1 -- 172.21.15.7:0/3118198197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 msgr2=0x7fbe2007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.133+0000 7fbe255d9700 1 --2- 172.21.15.7:0/3118198197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2007c2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe100098a0 tx=0x7fbe10009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.133+0000 7fbe255d9700 1 -- 172.21.15.7:0/3118198197 shutdown_connections 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.133+0000 7fbe255d9700 1 --2- 172.21.15.7:0/3118198197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2007c2d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:00.896 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.133+0000 7fbe255d9700 1 -- 172.21.15.7:0/3118198197 >> 172.21.15.7:0/3118198197 conn(0x7fbe201039b0 msgr2=0x7fbe20105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.133+0000 7fbe255d9700 1 -- 172.21.15.7:0/3118198197 shutdown_connections 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.133+0000 7fbe255d9700 1 -- 172.21.15.7:0/3118198197 wait complete. 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.134+0000 7fbe255d9700 1 Processor -- start 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.134+0000 7fbe255d9700 1 -- start start 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.134+0000 7fbe255d9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2019e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.134+0000 7fbe255d9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2010e910 con 0x7fbe2007de20 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.134+0000 7fbe1effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2019e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.134+0000 7fbe1effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2019e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46686/0 (socket says 172.21.15.7:46686) 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.134+0000 7fbe1effd700 1 -- 172.21.15.7:0/1069765 learned_addr learned my addr 172.21.15.7:0/1069765 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.134+0000 7fbe1effd700 1 -- 172.21.15.7:0/1069765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2019e8d0 con 0x7fbe2007de20 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.134+0000 7fbe1effd700 1 --2- 172.21.15.7:0/1069765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2019e390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe10000c00 tx=0x7fbe10011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.135+0000 7fbe07fff700 1 -- 172.21.15.7:0/1069765 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe10003af0 con 0x7fbe2007de20 2024-04-21T08:35:00.897 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.135+0000 7fbe07fff700 1 -- 172.21.15.7:0/1069765 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe1001b8f0 con 0x7fbe2007de20 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.135+0000 7fbe255d9700 1 -- 172.21.15.7:0/1069765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2019eb60 con 0x7fbe2007de20 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.135+0000 7fbe07fff700 1 -- 172.21.15.7:0/1069765 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe100238e0 con 0x7fbe2007de20 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.135+0000 7fbe255d9700 1 -- 172.21.15.7:0/1069765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe201a1850 con 0x7fbe2007de20 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.136+0000 7fbe07fff700 1 -- 172.21.15.7:0/1069765 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fbe10023be0 con 0x7fbe2007de20 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.136+0000 7fbe255d9700 1 -- 172.21.15.7:0/1069765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe200640a0 con 0x7fbe2007de20 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.136+0000 7fbe07fff700 1 --2- 172.21.15.7:0/1069765 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fbe08039670 0x7fbe0803bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.136+0000 7fbe07fff700 1 -- 172.21.15.7:0/1069765 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbe1004df60 con 0x7fbe2007de20 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.136+0000 7fbe1e7fc700 1 --2- 172.21.15.7:0/1069765 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fbe08039670 0x7fbe0803bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.136+0000 7fbe1e7fc700 1 --2- 172.21.15.7:0/1069765 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fbe08039670 0x7fbe0803bb30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbe14006fd0 tx=0x7fbe14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.141+0000 7fbe07fff700 1 -- 172.21.15.7:0/1069765 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe1001f030 con 0x7fbe2007de20 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.350+0000 7fbe255d9700 1 -- 172.21.15.7:0/1069765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fbe201a1eb0 con 0x7fbe2007de20 2024-04-21T08:35:00.898 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.353+0000 7fbe07fff700 1 -- 172.21.15.7:0/1069765 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fbe100292a0 con 0x7fbe2007de20 2024-04-21T08:35:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.355+0000 7fbe255d9700 1 -- 172.21.15.7:0/1069765 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fbe08039670 msgr2=0x7fbe0803bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.355+0000 7fbe255d9700 1 --2- 172.21.15.7:0/1069765 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fbe08039670 0x7fbe0803bb30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbe14006fd0 tx=0x7fbe14006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.355+0000 7fbe255d9700 1 -- 172.21.15.7:0/1069765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 msgr2=0x7fbe2019e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.355+0000 7fbe255d9700 1 --2- 172.21.15.7:0/1069765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2019e390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe10000c00 tx=0x7fbe10011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.355+0000 7fbe255d9700 1 -- 172.21.15.7:0/1069765 shutdown_connections 2024-04-21T08:35:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.356+0000 7fbe255d9700 1 --2- 172.21.15.7:0/1069765 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fbe08039670 0x7fbe0803bb30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.356+0000 7fbe255d9700 1 --2- 172.21.15.7:0/1069765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe2007de20 0x7fbe2019e390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.356+0000 7fbe255d9700 1 -- 172.21.15.7:0/1069765 >> 172.21.15.7:0/1069765 conn(0x7fbe201039b0 msgr2=0x7fbe201043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.356+0000 7fbe255d9700 1 -- 172.21.15.7:0/1069765 shutdown_connections 2024-04-21T08:35:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:00.356+0000 7fbe255d9700 1 -- 172.21.15.7:0/1069765 wait complete. 2024-04-21T08:35:00.899 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling the dashboard module... 2024-04-21T08:35:01.581 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:01 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1069765' entity='client.admin' 2024-04-21T08:35:01.581 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:01 smithi007 ceph-mon[28477]: from='mgr.14120 172.21.15.7:0/3328134329' entity='mgr.x' 2024-04-21T08:35:02.620 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:02 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/184659447' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-21T08:35:02.620 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:02 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: ignoring --setuser ceph since I am not root 2024-04-21T08:35:02.620 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:02 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: ignoring --setgroup ceph since I am not root 2024-04-21T08:35:02.621 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:02 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:02.543+0000 7f7497009200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:35:02.901 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:02 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:02.740+0000 7f7497009200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:35:02.901 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:02 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:02.846+0000 7f7497009200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:35:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.300+0000 7f4b9b464700 1 Processor -- start 2024-04-21T08:35:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.300+0000 7f4b9b464700 1 -- start start 2024-04-21T08:35:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.301+0000 7f4b9b464700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b941070a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.301+0000 7f4b9b464700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9407bdc0 con 0x7f4b94106c80 2024-04-21T08:35:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.301+0000 7f4b99200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b941070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.301+0000 7f4b99200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b941070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46700/0 (socket says 172.21.15.7:46700) 2024-04-21T08:35:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.301+0000 7f4b99200700 1 -- 172.21.15.7:0/308646025 learned_addr learned my addr 172.21.15.7:0/308646025 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:02.958 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.301+0000 7f4b99200700 1 -- 172.21.15.7:0/308646025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b941075e0 con 0x7f4b94106c80 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.301+0000 7f4b99200700 1 --2- 172.21.15.7:0/308646025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b941070a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4b900098a0 tx=0x7f4b90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e830f6802c55f12 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.301+0000 7f4b8bfff700 1 -- 172.21.15.7:0/308646025 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b90004160 con 0x7f4b94106c80 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.302+0000 7f4b8bfff700 1 -- 172.21.15.7:0/308646025 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b90004300 con 0x7f4b94106c80 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.302+0000 7f4b8bfff700 1 -- 172.21.15.7:0/308646025 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b90011a20 con 0x7f4b94106c80 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.302+0000 7f4b9b464700 1 -- 172.21.15.7:0/308646025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 msgr2=0x7f4b941070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.302+0000 7f4b9b464700 1 --2- 172.21.15.7:0/308646025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b941070a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4b900098a0 tx=0x7f4b90009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.302+0000 7f4b9b464700 1 -- 172.21.15.7:0/308646025 shutdown_connections 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.302+0000 7f4b9b464700 1 --2- 172.21.15.7:0/308646025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b941070a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.302+0000 7f4b9b464700 1 -- 172.21.15.7:0/308646025 >> 172.21.15.7:0/308646025 conn(0x7f4b94102820 msgr2=0x7f4b94104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.303+0000 7f4b9b464700 1 -- 172.21.15.7:0/308646025 shutdown_connections 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.303+0000 7f4b9b464700 1 -- 172.21.15.7:0/308646025 wait complete. 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.303+0000 7f4b9b464700 1 Processor -- start 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.303+0000 7f4b9b464700 1 -- start start 2024-04-21T08:35:02.959 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.303+0000 7f4b9b464700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b9419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.303+0000 7f4b9b464700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4b94108650 con 0x7f4b94106c80 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.304+0000 7f4b99200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b9419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.304+0000 7f4b99200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b9419e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46702/0 (socket says 172.21.15.7:46702) 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.304+0000 7f4b99200700 1 -- 172.21.15.7:0/184659447 learned_addr learned my addr 172.21.15.7:0/184659447 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.304+0000 7f4b99200700 1 -- 172.21.15.7:0/184659447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9419e8c0 con 0x7f4b94106c80 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.304+0000 7f4b99200700 1 --2- 172.21.15.7:0/184659447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b9419e380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4b90000c00 tx=0x7f4b90011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.304+0000 7f4b8a7fc700 1 -- 172.21.15.7:0/184659447 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b90003aa0 con 0x7f4b94106c80 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.304+0000 7f4b8a7fc700 1 -- 172.21.15.7:0/184659447 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b9001b8f0 con 0x7f4b94106c80 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.304+0000 7f4b9b464700 1 -- 172.21.15.7:0/184659447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9419eb50 con 0x7f4b94106c80 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.304+0000 7f4b8a7fc700 1 -- 172.21.15.7:0/184659447 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b90023870 con 0x7f4b94106c80 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.304+0000 7f4b9b464700 1 -- 172.21.15.7:0/184659447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b941a1840 con 0x7f4b94106c80 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.305+0000 7f4b8a7fc700 1 -- 172.21.15.7:0/184659447 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4b90023b70 con 0x7f4b94106c80 2024-04-21T08:35:02.960 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.305+0000 7f4b8a7fc700 1 --2- 172.21.15.7:0/184659447 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4b80039670 0x7f4b8003bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.305+0000 7f4b8a7fc700 1 -- 172.21.15.7:0/184659447 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4b9004edf0 con 0x7f4b94106c80 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.306+0000 7f4b989ff700 1 --2- 172.21.15.7:0/184659447 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4b80039670 0x7f4b8003bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.306+0000 7f4b9b464700 1 -- 172.21.15.7:0/184659447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b940640a0 con 0x7f4b94106c80 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.306+0000 7f4b989ff700 1 --2- 172.21.15.7:0/184659447 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4b80039670 0x7f4b8003bb30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4b84006fd0 tx=0x7f4b84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.311+0000 7f4b8a7fc700 1 -- 172.21.15.7:0/184659447 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b9001f030 con 0x7f4b94106c80 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.581+0000 7f4b9b464700 1 -- 172.21.15.7:0/184659447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f4b9402e100 con 0x7f4b94106c80 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.357+0000 7f4b8a7fc700 1 -- 172.21.15.7:0/184659447 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f4b9001baa0 con 0x7f4b94106c80 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.358+0000 7f4b8a7fc700 1 -- 172.21.15.7:0/184659447 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f4b9002c770 con 0x7f4b94106c80 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.360+0000 7f4b9b464700 1 -- 172.21.15.7:0/184659447 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4b80039670 msgr2=0x7f4b8003bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.360+0000 7f4b9b464700 1 --2- 172.21.15.7:0/184659447 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4b80039670 0x7f4b8003bb30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4b84006fd0 tx=0x7f4b84006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.360+0000 7f4b9b464700 1 -- 172.21.15.7:0/184659447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 msgr2=0x7f4b9419e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:02.961 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.360+0000 7f4b9b464700 1 --2- 172.21.15.7:0/184659447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b9419e380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4b90000c00 tx=0x7f4b90011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.360+0000 7f4b9b464700 1 -- 172.21.15.7:0/184659447 shutdown_connections 2024-04-21T08:35:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.360+0000 7f4b9b464700 1 --2- 172.21.15.7:0/184659447 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f4b80039670 0x7f4b8003bb30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.360+0000 7f4b9b464700 1 --2- 172.21.15.7:0/184659447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b94106c80 0x7f4b9419e380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.360+0000 7f4b9b464700 1 -- 172.21.15.7:0/184659447 >> 172.21.15.7:0/184659447 conn(0x7f4b94102820 msgr2=0x7f4b94195040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.360+0000 7f4b9b464700 1 -- 172.21.15.7:0/184659447 shutdown_connections 2024-04-21T08:35:02.962 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:02.360+0000 7f4b9b464700 1 -- 172.21.15.7:0/184659447 wait complete. 2024-04-21T08:35:03.263 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:03 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:03.173+0000 7f7497009200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:35:03.263 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:03 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:03.263+0000 7f7497009200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:35:03.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:03 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/184659447' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-21T08:35:03.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:03 smithi007 ceph-mon[28477]: mgrmap e9: x(active, since 19s) 2024-04-21T08:35:04.233 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:35:04.233 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-21T08:35:04.233 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:35:04.233 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T08:35:04.233 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T08:35:04.233 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.385+0000 7f535c69c700 1 Processor -- start 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.385+0000 7f535c69c700 1 -- start start 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.385+0000 7f535c69c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f5354108790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.385+0000 7f535c69c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5354108d60 con 0x7f5354108370 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.386+0000 7f535a438700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f5354108790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.386+0000 7f535a438700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f5354108790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46730/0 (socket says 172.21.15.7:46730) 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.386+0000 7f535a438700 1 -- 172.21.15.7:0/430609827 learned_addr learned my addr 172.21.15.7:0/430609827 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.386+0000 7f535a438700 1 -- 172.21.15.7:0/430609827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5354109520 con 0x7f5354108370 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.386+0000 7f535a438700 1 --2- 172.21.15.7:0/430609827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f5354108790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f53440098a0 tx=0x7f5344009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b68cde68da9594 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.386+0000 7f5359436700 1 -- 172.21.15.7:0/430609827 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5344004160 con 0x7f5354108370 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.386+0000 7f5359436700 1 -- 172.21.15.7:0/430609827 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5344004300 con 0x7f5354108370 2024-04-21T08:35:04.234 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.386+0000 7f5359436700 1 -- 172.21.15.7:0/430609827 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5344011a20 con 0x7f5354108370 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.387+0000 7f535c69c700 1 -- 172.21.15.7:0/430609827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 msgr2=0x7f5354108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.387+0000 7f535c69c700 1 --2- 172.21.15.7:0/430609827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f5354108790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f53440098a0 tx=0x7f5344009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.387+0000 7f535c69c700 1 -- 172.21.15.7:0/430609827 shutdown_connections 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.387+0000 7f535c69c700 1 --2- 172.21.15.7:0/430609827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f5354108790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.387+0000 7f535c69c700 1 -- 172.21.15.7:0/430609827 >> 172.21.15.7:0/430609827 conn(0x7f53541038f0 msgr2=0x7f5354105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.387+0000 7f535c69c700 1 -- 172.21.15.7:0/430609827 shutdown_connections 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.387+0000 7f535c69c700 1 -- 172.21.15.7:0/430609827 wait complete. 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.388+0000 7f535c69c700 1 Processor -- start 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.388+0000 7f535c69c700 1 -- start start 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.388+0000 7f535c69c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f535407db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.388+0000 7f535c69c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f535410c8d0 con 0x7f5354108370 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.388+0000 7f535a438700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f535407db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:04.235 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.388+0000 7f535a438700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f535407db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46738/0 (socket says 172.21.15.7:46738) 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.388+0000 7f535a438700 1 -- 172.21.15.7:0/1300622923 learned_addr learned my addr 172.21.15.7:0/1300622923 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.388+0000 7f535a438700 1 -- 172.21.15.7:0/1300622923 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f535407e0a0 con 0x7f5354108370 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.388+0000 7f535a438700 1 --2- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f535407db60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5344009520 tx=0x7f5344011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.389+0000 7f534b7fe700 1 -- 172.21.15.7:0/1300622923 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53440039e0 con 0x7f5354108370 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.389+0000 7f534b7fe700 1 -- 172.21.15.7:0/1300622923 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f534401b8f0 con 0x7f5354108370 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.389+0000 7f534b7fe700 1 -- 172.21.15.7:0/1300622923 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5344019770 con 0x7f5354108370 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.389+0000 7f535c69c700 1 -- 172.21.15.7:0/1300622923 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f535407c290 con 0x7f5354108370 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.389+0000 7f535c69c700 1 -- 172.21.15.7:0/1300622923 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f535407c7d0 con 0x7f5354108370 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.390+0000 7f535c69c700 1 -- 172.21.15.7:0/1300622923 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53540640a0 con 0x7f5354108370 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.390+0000 7f534b7fe700 1 -- 172.21.15.7:0/1300622923 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f534401b440 con 0x7f5354108370 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.390+0000 7f534b7fe700 1 --2- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f53400396c0 0x7f534003bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.390+0000 7f534b7fe700 1 -- 172.21.15.7:0/1300622923 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f534404def0 con 0x7f5354108370 2024-04-21T08:35:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.390+0000 7f5359c37700 1 -- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f53400396c0 msgr2=0x7f534003bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/102274072 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.390+0000 7f5359c37700 1 --2- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f53400396c0 0x7f534003bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.395+0000 7f534b7fe700 1 -- 172.21.15.7:0/1300622923 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f534401f030 con 0x7f5354108370 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.591+0000 7f5359c37700 1 -- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f53400396c0 msgr2=0x7f534003bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/102274072 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.591+0000 7f5359c37700 1 --2- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f53400396c0 0x7f534003bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.680+0000 7f535c69c700 1 -- 172.21.15.7:0/1300622923 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f535407ca50 con 0x7f5354108370 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.680+0000 7f534b7fe700 1 -- 172.21.15.7:0/1300622923 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f5344010380 con 0x7f5354108370 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.684+0000 7f535c69c700 1 -- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f53400396c0 msgr2=0x7f534003bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.684+0000 7f535c69c700 1 --2- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f53400396c0 0x7f534003bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.684+0000 7f535c69c700 1 -- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 msgr2=0x7f535407db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.684+0000 7f535c69c700 1 --2- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f535407db60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5344009520 tx=0x7f5344011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.685+0000 7f535c69c700 1 -- 172.21.15.7:0/1300622923 shutdown_connections 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.685+0000 7f535c69c700 1 --2- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7f53400396c0 0x7f534003bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:04.237 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.685+0000 7f535c69c700 1 --2- 172.21.15.7:0/1300622923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5354108370 0x7f535407db60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:04.238 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.685+0000 7f535c69c700 1 -- 172.21.15.7:0/1300622923 >> 172.21.15.7:0/1300622923 conn(0x7f53541038f0 msgr2=0x7f5354104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:04.238 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.685+0000 7f535c69c700 1 -- 172.21.15.7:0/1300622923 shutdown_connections 2024-04-21T08:35:04.238 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:03.685+0000 7f535c69c700 1 -- 172.21.15.7:0/1300622923 wait complete. 2024-04-21T08:35:04.238 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2024-04-21T08:35:04.238 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 9... 2024-04-21T08:35:04.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:04 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1300622923' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T08:35:04.651 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:04 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:04.536+0000 7f7497009200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:35:05.151 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:04 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:04.718+0000 7f7497009200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:35:05.151 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:04 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:04.895+0000 7f7497009200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:35:05.503 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:05 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:05.195+0000 7f7497009200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:35:05.503 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:05 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:05.292+0000 7f7497009200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:35:05.503 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:05 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:05.503+0000 7f7497009200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:35:05.901 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:05 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:05.704+0000 7f7497009200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:35:06.401 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:06 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:06.330+0000 7f7497009200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:35:07.085 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:06 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:06.771+0000 7f7497009200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:35:07.085 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:06 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:06.872+0000 7f7497009200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:35:07.085 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:07 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:07.085+0000 7f7497009200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:35:07.901 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:07 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:07.633+0000 7f7497009200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:35:07.901 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:07 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:07.729+0000 7f7497009200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:35:08.346 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:07 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:07.983+0000 7f7497009200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:35:08.347 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:08 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:08.123+0000 7f7497009200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:35:08.347 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:08 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:08.223+0000 7f7497009200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:35:08.347 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:08 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:08.347+0000 7f7497009200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:35:09.151 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:08 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:08.681+0000 7f7497009200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:35:09.592 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:09.310+0000 7f7497009200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:35:09.592 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:35:09.539+0000 7f7497009200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:35:09.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-mon[28477]: Active manager daemon x restarted 2024-04-21T08:35:09.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-mon[28477]: Activating manager daemon x 2024-04-21T08:35:09.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-mon[28477]: osdmap e3: 0 total, 0 up, 0 in 2024-04-21T08:35:09.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-mon[28477]: mgrmap e10: x(active, starting, since 0.00353986s) 2024-04-21T08:35:09.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:35:09.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:35:09.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:35:09.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:35:09.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:35:09.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-mon[28477]: Manager daemon x is now available 2024-04-21T08:35:09.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:10.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:35:10.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:35:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:10 smithi007 ceph-mon[28477]: mgrmap e11: x(active, since 1.00784s) 2024-04-21T08:35:11.392 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:35:11.392 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.687+0000 7fa56d7c4700 1 Processor -- start 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.687+0000 7fa56d7c4700 1 -- start start 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.687+0000 7fa56d7c4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa568107050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.687+0000 7fa56d7c4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa56807be20 con 0x7fa568106c30 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.687+0000 7fa566ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa568107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.687+0000 7fa566ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa568107050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54114/0 (socket says 172.21.15.7:54114) 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.687+0000 7fa566ffd700 1 -- 172.21.15.7:0/4144210245 learned_addr learned my addr 172.21.15.7:0/4144210245 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.688+0000 7fa566ffd700 1 -- 172.21.15.7:0/4144210245 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa568107590 con 0x7fa568106c30 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.688+0000 7fa566ffd700 1 --2- 172.21.15.7:0/4144210245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa568107050 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa5580098a0 tx=0x7fa558009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a27453a47d0176 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:11.393 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.688+0000 7fa565ffb700 1 -- 172.21.15.7:0/4144210245 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa558004160 con 0x7fa568106c30 2024-04-21T08:35:11.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.688+0000 7fa565ffb700 1 -- 172.21.15.7:0/4144210245 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa558004300 con 0x7fa568106c30 2024-04-21T08:35:11.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.688+0000 7fa565ffb700 1 -- 172.21.15.7:0/4144210245 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa558011a20 con 0x7fa568106c30 2024-04-21T08:35:11.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.689+0000 7fa56d7c4700 1 -- 172.21.15.7:0/4144210245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 msgr2=0x7fa568107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:11.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.689+0000 7fa56d7c4700 1 --2- 172.21.15.7:0/4144210245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa568107050 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa5580098a0 tx=0x7fa558009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:11.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.689+0000 7fa56d7c4700 1 -- 172.21.15.7:0/4144210245 shutdown_connections 2024-04-21T08:35:11.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.689+0000 7fa56d7c4700 1 --2- 172.21.15.7:0/4144210245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa568107050 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:11.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.689+0000 7fa56d7c4700 1 -- 172.21.15.7:0/4144210245 >> 172.21.15.7:0/4144210245 conn(0x7fa568102890 msgr2=0x7fa568104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:11.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.689+0000 7fa56d7c4700 1 -- 172.21.15.7:0/4144210245 shutdown_connections 2024-04-21T08:35:11.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.689+0000 7fa56d7c4700 1 -- 172.21.15.7:0/4144210245 wait complete. 2024-04-21T08:35:11.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.689+0000 7fa56d7c4700 1 Processor -- start 2024-04-21T08:35:11.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa56d7c4700 1 -- start start 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa56d7c4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa56819e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa56d7c4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa568108600 con 0x7fa568106c30 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa566ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa56819e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa566ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa56819e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54124/0 (socket says 172.21.15.7:54124) 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa566ffd700 1 -- 172.21.15.7:0/322909863 learned_addr learned my addr 172.21.15.7:0/322909863 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa566ffd700 1 -- 172.21.15.7:0/322909863 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56819e8e0 con 0x7fa568106c30 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa566ffd700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa56819e3a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa558000c00 tx=0x7fa558011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa55ffff700 1 -- 172.21.15.7:0/322909863 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa558003aa0 con 0x7fa568106c30 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa56d7c4700 1 -- 172.21.15.7:0/322909863 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa56819eb70 con 0x7fa568106c30 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa55ffff700 1 -- 172.21.15.7:0/322909863 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa55801b8f0 con 0x7fa568106c30 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.690+0000 7fa55ffff700 1 -- 172.21.15.7:0/322909863 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa558023870 con 0x7fa568106c30 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.691+0000 7fa56d7c4700 1 -- 172.21.15.7:0/322909863 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5681a1860 con 0x7fa568106c30 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.691+0000 7fa55ffff700 1 -- 172.21.15.7:0/322909863 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fa558023b20 con 0x7fa568106c30 2024-04-21T08:35:11.395 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.692+0000 7fa55ffff700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 0x7fa55403bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:11.396 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.692+0000 7fa56d7c4700 1 -- 172.21.15.7:0/322909863 --> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa568069b50 con 0x7fa554039720 2024-04-21T08:35:11.396 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.692+0000 7fa55ffff700 1 -- 172.21.15.7:0/322909863 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa55804eda0 con 0x7fa568106c30 2024-04-21T08:35:11.396 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.692+0000 7fa5667fc700 1 -- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 msgr2=0x7fa55403bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/102274072 2024-04-21T08:35:11.396 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.692+0000 7fa5667fc700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 0x7fa55403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:35:11.396 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.893+0000 7fa5667fc700 1 -- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 msgr2=0x7fa55403bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/102274072 2024-04-21T08:35:11.396 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.893+0000 7fa5667fc700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 0x7fa55403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:35:11.396 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:05.294+0000 7fa5667fc700 1 -- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 msgr2=0x7fa55403bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/102274072 2024-04-21T08:35:11.396 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:05.294+0000 7fa5667fc700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 0x7fa55403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T08:35:11.396 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:06.095+0000 7fa5667fc700 1 -- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 msgr2=0x7fa55403bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/102274072 2024-04-21T08:35:11.396 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:06.095+0000 7fa5667fc700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 0x7fa55403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T08:35:11.396 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.696+0000 7fa5667fc700 1 -- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 msgr2=0x7fa55403bbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/102274072 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.696+0000 7fa5667fc700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 0x7fa55403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.543+0000 7fa55ffff700 1 -- 172.21.15.7:0/322909863 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7fa55802c720 con 0x7fa568106c30 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.543+0000 7fa55ffff700 1 -- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 msgr2=0x7fa55403bbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.543+0000 7fa55ffff700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 0x7fa55403bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.545+0000 7fa55ffff700 1 -- 172.21.15.7:0/322909863 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7fa55802b350 con 0x7fa568106c30 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.545+0000 7fa55ffff700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa55403cfb0 0x7fa55403f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.546+0000 7fa55ffff700 1 -- 172.21.15.7:0/322909863 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa568069b50 con 0x7fa55403cfb0 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.546+0000 7fa5667fc700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa55403cfb0 0x7fa55403f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.547+0000 7fa5667fc700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa55403cfb0 0x7fa55403f3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa550003a10 tx=0x7fa550009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.547+0000 7fa55ffff700 1 -- 172.21.15.7:0/322909863 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fa568069b50 con 0x7fa55403cfb0 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.554+0000 7fa56d7c4700 1 -- 172.21.15.7:0/322909863 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fa568062e80 con 0x7fa55403cfb0 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.554+0000 7fa55ffff700 1 -- 172.21.15.7:0/322909863 <== mgr.14152 v2:172.21.15.7:6800/2555078032 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fa568062e80 con 0x7fa55403cfb0 2024-04-21T08:35:11.397 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.555+0000 7fa56d7c4700 1 -- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa55403cfb0 msgr2=0x7fa55403f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.555+0000 7fa56d7c4700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa55403cfb0 0x7fa55403f3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa550003a10 tx=0x7fa550009270 comp rx=0 tx=0).stop 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.555+0000 7fa56d7c4700 1 -- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 msgr2=0x7fa56819e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.555+0000 7fa56d7c4700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa56819e3a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa558000c00 tx=0x7fa558011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.555+0000 7fa56d7c4700 1 -- 172.21.15.7:0/322909863 shutdown_connections 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.555+0000 7fa56d7c4700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa55403cfb0 0x7fa55403f3a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.555+0000 7fa56d7c4700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa568106c30 0x7fa56819e3a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.555+0000 7fa56d7c4700 1 --2- 172.21.15.7:0/322909863 >> [v2:172.21.15.7:6800/102274072,v1:172.21.15.7:6801/102274072] conn(0x7fa554039720 0x7fa55403bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.555+0000 7fa56d7c4700 1 -- 172.21.15.7:0/322909863 >> 172.21.15.7:0/322909863 conn(0x7fa568102890 msgr2=0x7fa568194fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.555+0000 7fa56d7c4700 1 -- 172.21.15.7:0/322909863 shutdown_connections 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.555+0000 7fa56d7c4700 1 -- 172.21.15.7:0/322909863 wait complete. 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 9 is available 2024-04-21T08:35:11.398 INFO:teuthology.orchestra.run.smithi007.stdout:Generating a dashboard self-signed certificate... 2024-04-21T08:35:12.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:12.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:12 smithi007 ceph-mon[28477]: [21/Apr/2024:08:35:11] ENGINE Bus STARTING 2024-04-21T08:35:12.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:12 smithi007 ceph-mon[28477]: [21/Apr/2024:08:35:11] ENGINE Serving on http://172.21.15.7:8765 2024-04-21T08:35:12.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:12 smithi007 ceph-mon[28477]: [21/Apr/2024:08:35:11] ENGINE Serving on https://172.21.15.7:7150 2024-04-21T08:35:12.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:12 smithi007 ceph-mon[28477]: [21/Apr/2024:08:35:11] ENGINE Bus STARTED 2024-04-21T08:35:12.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:12.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:12.665 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-21T08:35:12.665 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.816+0000 7fbaa239b700 1 Processor -- start 2024-04-21T08:35:12.665 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.816+0000 7fbaa239b700 1 -- start start 2024-04-21T08:35:12.665 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.817+0000 7fbaa239b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c10b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:12.665 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.817+0000 7fbaa239b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c07bea0 con 0x7fba9c108db0 2024-04-21T08:35:12.665 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.817+0000 7fba9bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c10b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:12.665 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.817+0000 7fba9bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c10b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54186/0 (socket says 172.21.15.7:54186) 2024-04-21T08:35:12.665 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.817+0000 7fba9bfff700 1 -- 172.21.15.7:0/4102829651 learned_addr learned my addr 172.21.15.7:0/4102829651 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:12.665 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.817+0000 7fba9bfff700 1 -- 172.21.15.7:0/4102829651 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba9c10b6e0 con 0x7fba9c108db0 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.817+0000 7fba9bfff700 1 --2- 172.21.15.7:0/4102829651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c10b1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0098a0 tx=0x7fba8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f48077b86af032 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.818+0000 7fba9affd700 1 -- 172.21.15.7:0/4102829651 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba8c004160 con 0x7fba9c108db0 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.818+0000 7fba9affd700 1 -- 172.21.15.7:0/4102829651 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba8c004300 con 0x7fba9c108db0 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.818+0000 7fba9affd700 1 -- 172.21.15.7:0/4102829651 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba8c011a20 con 0x7fba9c108db0 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.818+0000 7fbaa239b700 1 -- 172.21.15.7:0/4102829651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 msgr2=0x7fba9c10b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.818+0000 7fbaa239b700 1 --2- 172.21.15.7:0/4102829651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c10b1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0098a0 tx=0x7fba8c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.818+0000 7fbaa239b700 1 -- 172.21.15.7:0/4102829651 shutdown_connections 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.818+0000 7fbaa239b700 1 --2- 172.21.15.7:0/4102829651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c10b1a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.818+0000 7fbaa239b700 1 -- 172.21.15.7:0/4102829651 >> 172.21.15.7:0/4102829651 conn(0x7fba9c102820 msgr2=0x7fba9c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.818+0000 7fbaa239b700 1 -- 172.21.15.7:0/4102829651 shutdown_connections 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.818+0000 7fbaa239b700 1 -- 172.21.15.7:0/4102829651 wait complete. 2024-04-21T08:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.819+0000 7fbaa239b700 1 Processor -- start 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.819+0000 7fbaa239b700 1 -- start start 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.819+0000 7fbaa239b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c19c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.819+0000 7fbaa239b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c10c6f0 con 0x7fba9c108db0 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.820+0000 7fba9bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c19c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.820+0000 7fba9bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c19c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54188/0 (socket says 172.21.15.7:54188) 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.820+0000 7fba9bfff700 1 -- 172.21.15.7:0/2240968309 learned_addr learned my addr 172.21.15.7:0/2240968309 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.820+0000 7fba9bfff700 1 -- 172.21.15.7:0/2240968309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba9c19c6d0 con 0x7fba9c108db0 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.820+0000 7fba9bfff700 1 --2- 172.21.15.7:0/2240968309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c19c190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0116e0 tx=0x7fba8c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.820+0000 7fba997fa700 1 -- 172.21.15.7:0/2240968309 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba8c003a60 con 0x7fba9c108db0 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.820+0000 7fba997fa700 1 -- 172.21.15.7:0/2240968309 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba8c01b8f0 con 0x7fba9c108db0 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.820+0000 7fbaa239b700 1 -- 172.21.15.7:0/2240968309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba9c19c960 con 0x7fba9c108db0 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.820+0000 7fba997fa700 1 -- 172.21.15.7:0/2240968309 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba8c019860 con 0x7fba9c108db0 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.820+0000 7fbaa239b700 1 -- 172.21.15.7:0/2240968309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba9c19f650 con 0x7fba9c108db0 2024-04-21T08:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.821+0000 7fbaa239b700 1 -- 172.21.15.7:0/2240968309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba9c0640a0 con 0x7fba9c108db0 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.821+0000 7fba997fa700 1 -- 172.21.15.7:0/2240968309 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7fba8c019b10 con 0x7fba9c108db0 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.822+0000 7fba997fa700 1 --2- 172.21.15.7:0/2240968309 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba840395b0 0x7fba8403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.822+0000 7fba997fa700 1 -- 172.21.15.7:0/2240968309 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fba8c04de90 con 0x7fba9c108db0 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.822+0000 7fba9b7fe700 1 --2- 172.21.15.7:0/2240968309 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba840395b0 0x7fba8403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.822+0000 7fba9b7fe700 1 --2- 172.21.15.7:0/2240968309 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba840395b0 0x7fba8403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba90006fd0 tx=0x7fba90006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.827+0000 7fba997fa700 1 -- 172.21.15.7:0/2240968309 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba8c01f030 con 0x7fba9c108db0 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.040+0000 7fbaa239b700 1 -- 172.21.15.7:0/2240968309 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fba9c103930 con 0x7fba840395b0 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.099+0000 7fba997fa700 1 -- 172.21.15.7:0/2240968309 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fba9c103930 con 0x7fba840395b0 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.102+0000 7fbaa239b700 1 -- 172.21.15.7:0/2240968309 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba840395b0 msgr2=0x7fba8403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.102+0000 7fbaa239b700 1 --2- 172.21.15.7:0/2240968309 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba840395b0 0x7fba8403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba90006fd0 tx=0x7fba90006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.102+0000 7fbaa239b700 1 -- 172.21.15.7:0/2240968309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 msgr2=0x7fba9c19c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.102+0000 7fbaa239b700 1 --2- 172.21.15.7:0/2240968309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c19c190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0116e0 tx=0x7fba8c011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.102+0000 7fbaa239b700 1 -- 172.21.15.7:0/2240968309 shutdown_connections 2024-04-21T08:35:12.669 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.102+0000 7fbaa239b700 1 --2- 172.21.15.7:0/2240968309 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba840395b0 0x7fba8403ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:12.669 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.102+0000 7fbaa239b700 1 --2- 172.21.15.7:0/2240968309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba9c108db0 0x7fba9c19c190 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:12.669 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.102+0000 7fbaa239b700 1 -- 172.21.15.7:0/2240968309 >> 172.21.15.7:0/2240968309 conn(0x7fba9c102820 msgr2=0x7fba9c103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:12.669 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.102+0000 7fbaa239b700 1 -- 172.21.15.7:0/2240968309 shutdown_connections 2024-04-21T08:35:12.669 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.102+0000 7fbaa239b700 1 -- 172.21.15.7:0/2240968309 wait complete. 2024-04-21T08:35:12.669 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial admin user... 2024-04-21T08:35:13.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:13 smithi007 ceph-mon[28477]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:13.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:13 smithi007 ceph-mon[28477]: mgrmap e12: x(active, since 2s) 2024-04-21T08:35:13.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:13 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:14.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:14 smithi007 ceph-mon[28477]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:14.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:14.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$h9uQczNjL3FicG6e9l6l2uwIQNfYKUMorJHuWO8svzB1yD82aFdJi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713688513, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-21T08:35:14.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.050+0000 7ff4921da700 1 Processor -- start 2024-04-21T08:35:14.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.050+0000 7ff4921da700 1 -- start start 2024-04-21T08:35:14.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.050+0000 7ff4921da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c107670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:14.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.050+0000 7ff4921da700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c107c40 con 0x7ff48c107250 2024-04-21T08:35:14.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.050+0000 7ff48b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c107670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:14.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.050+0000 7ff48b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c107670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54194/0 (socket says 172.21.15.7:54194) 2024-04-21T08:35:14.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.050+0000 7ff48b7fe700 1 -- 172.21.15.7:0/600138241 learned_addr learned my addr 172.21.15.7:0/600138241 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.051+0000 7ff48b7fe700 1 -- 172.21.15.7:0/600138241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c108450 con 0x7ff48c107250 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.051+0000 7ff48b7fe700 1 --2- 172.21.15.7:0/600138241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c107670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00d180 tx=0x7ff47c00d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76ecdc459530fed2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.051+0000 7ff48a7fc700 1 -- 172.21.15.7:0/600138241 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff47c019070 con 0x7ff48c107250 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.051+0000 7ff48a7fc700 1 -- 172.21.15.7:0/600138241 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff47c00de20 con 0x7ff48c107250 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.051+0000 7ff48a7fc700 1 -- 172.21.15.7:0/600138241 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff47c009a70 con 0x7ff48c107250 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.052+0000 7ff4921da700 1 -- 172.21.15.7:0/600138241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 msgr2=0x7ff48c107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.052+0000 7ff4921da700 1 --2- 172.21.15.7:0/600138241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c107670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00d180 tx=0x7ff47c00d490 comp rx=0 tx=0).stop 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.052+0000 7ff4921da700 1 -- 172.21.15.7:0/600138241 shutdown_connections 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.052+0000 7ff4921da700 1 --2- 172.21.15.7:0/600138241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c107670 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.052+0000 7ff4921da700 1 -- 172.21.15.7:0/600138241 >> 172.21.15.7:0/600138241 conn(0x7ff48c1027f0 msgr2=0x7ff48c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.052+0000 7ff4921da700 1 -- 172.21.15.7:0/600138241 shutdown_connections 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.052+0000 7ff4921da700 1 -- 172.21.15.7:0/600138241 wait complete. 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.052+0000 7ff4921da700 1 Processor -- start 2024-04-21T08:35:14.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.053+0000 7ff4921da700 1 -- start start 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.053+0000 7ff4921da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.053+0000 7ff4921da700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c07afc0 con 0x7ff48c107250 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.053+0000 7ff48b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.053+0000 7ff48b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54202/0 (socket says 172.21.15.7:54202) 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.053+0000 7ff48b7fe700 1 -- 172.21.15.7:0/3094856488 learned_addr learned my addr 172.21.15.7:0/3094856488 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.053+0000 7ff48b7fe700 1 -- 172.21.15.7:0/3094856488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c19a530 con 0x7ff48c107250 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.053+0000 7ff48b7fe700 1 --2- 172.21.15.7:0/3094856488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c199ff0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009640 tx=0x7ff47c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.053+0000 7ff488ff9700 1 -- 172.21.15.7:0/3094856488 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff47c019050 con 0x7ff48c107250 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.053+0000 7ff488ff9700 1 -- 172.21.15.7:0/3094856488 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff47c025070 con 0x7ff48c107250 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.053+0000 7ff4921da700 1 -- 172.21.15.7:0/3094856488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff48c19a7c0 con 0x7ff48c107250 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.054+0000 7ff488ff9700 1 -- 172.21.15.7:0/3094856488 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff47c0209d0 con 0x7ff48c107250 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.054+0000 7ff4921da700 1 -- 172.21.15.7:0/3094856488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff48c19d4b0 con 0x7ff48c107250 2024-04-21T08:35:14.529 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.054+0000 7ff488ff9700 1 -- 172.21.15.7:0/3094856488 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7ff47c020c10 con 0x7ff48c107250 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.054+0000 7ff4921da700 1 -- 172.21.15.7:0/3094856488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff48c0640a0 con 0x7ff48c107250 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.055+0000 7ff488ff9700 1 --2- 172.21.15.7:0/3094856488 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff478039670 0x7ff47803bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.055+0000 7ff488ff9700 1 -- 172.21.15.7:0/3094856488 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff47c011050 con 0x7ff48c107250 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.055+0000 7ff48affd700 1 --2- 172.21.15.7:0/3094856488 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff478039670 0x7ff47803bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.055+0000 7ff48affd700 1 --2- 172.21.15.7:0/3094856488 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff478039670 0x7ff47803bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff474006fd0 tx=0x7ff474006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.061+0000 7ff488ff9700 1 -- 172.21.15.7:0/3094856488 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff47c0293c0 con 0x7ff48c107250 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.293+0000 7ff4921da700 1 -- 172.21.15.7:0/3094856488 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7ff48c19d740 con 0x7ff478039670 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.540+0000 7ff488ff9700 1 -- 172.21.15.7:0/3094856488 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7ff48c19d740 con 0x7ff478039670 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.542+0000 7ff4921da700 1 -- 172.21.15.7:0/3094856488 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff478039670 msgr2=0x7ff47803bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.542+0000 7ff4921da700 1 --2- 172.21.15.7:0/3094856488 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff478039670 0x7ff47803bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff474006fd0 tx=0x7ff474006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.542+0000 7ff4921da700 1 -- 172.21.15.7:0/3094856488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 msgr2=0x7ff48c199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.542+0000 7ff4921da700 1 --2- 172.21.15.7:0/3094856488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c199ff0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009640 tx=0x7ff47c003d00 comp rx=0 tx=0).stop 2024-04-21T08:35:14.530 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.542+0000 7ff4921da700 1 -- 172.21.15.7:0/3094856488 shutdown_connections 2024-04-21T08:35:14.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.542+0000 7ff4921da700 1 --2- 172.21.15.7:0/3094856488 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff478039670 0x7ff47803bb30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:14.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.543+0000 7ff4921da700 1 --2- 172.21.15.7:0/3094856488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff48c107250 0x7ff48c199ff0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:14.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.543+0000 7ff4921da700 1 -- 172.21.15.7:0/3094856488 >> 172.21.15.7:0/3094856488 conn(0x7ff48c1027f0 msgr2=0x7ff48c190610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:14.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.543+0000 7ff4921da700 1 -- 172.21.15.7:0/3094856488 shutdown_connections 2024-04-21T08:35:14.531 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.543+0000 7ff4921da700 1 -- 172.21.15.7:0/3094856488 wait complete. 2024-04-21T08:35:14.531 INFO:teuthology.orchestra.run.smithi007.stdout:Fetching dashboard port number... 2024-04-21T08:35:15.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:15 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2795680141' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-21T08:35:15.792 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 8443 2024-04-21T08:35:15.792 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.997+0000 7f1698a36700 1 Processor -- start 2024-04-21T08:35:15.792 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.997+0000 7f1698a36700 1 -- start start 2024-04-21T08:35:15.792 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.997+0000 7f1698a36700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169410b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:15.792 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.997+0000 7f1698a36700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f169407be40 con 0x7f1694108e10 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.998+0000 7f169259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169410b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.998+0000 7f169259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169410b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49246/0 (socket says 172.21.15.7:49246) 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.998+0000 7f169259c700 1 -- 172.21.15.7:0/4157800712 learned_addr learned my addr 172.21.15.7:0/4157800712 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.998+0000 7f169259c700 1 -- 172.21.15.7:0/4157800712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169410b740 con 0x7f1694108e10 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.998+0000 7f169259c700 1 --2- 172.21.15.7:0/4157800712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169410b200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f16840098a0 tx=0x7f1684009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332a9a6b97565d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.998+0000 7f169159a700 1 -- 172.21.15.7:0/4157800712 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1684004160 con 0x7f1694108e10 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.998+0000 7f169159a700 1 -- 172.21.15.7:0/4157800712 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1684004300 con 0x7f1694108e10 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.998+0000 7f169159a700 1 -- 172.21.15.7:0/4157800712 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1684011a20 con 0x7f1694108e10 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.999+0000 7f1698a36700 1 -- 172.21.15.7:0/4157800712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 msgr2=0x7f169410b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.999+0000 7f1698a36700 1 --2- 172.21.15.7:0/4157800712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169410b200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f16840098a0 tx=0x7f1684009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.999+0000 7f1698a36700 1 -- 172.21.15.7:0/4157800712 shutdown_connections 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.999+0000 7f1698a36700 1 --2- 172.21.15.7:0/4157800712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169410b200 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.999+0000 7f1698a36700 1 -- 172.21.15.7:0/4157800712 >> 172.21.15.7:0/4157800712 conn(0x7f1694102820 msgr2=0x7f1694104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.999+0000 7f1698a36700 1 -- 172.21.15.7:0/4157800712 shutdown_connections 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.999+0000 7f1698a36700 1 -- 172.21.15.7:0/4157800712 wait complete. 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.000+0000 7f1698a36700 1 Processor -- start 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.000+0000 7f1698a36700 1 -- start start 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.000+0000 7f1698a36700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169419c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.000+0000 7f1698a36700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f169410c6f0 con 0x7f1694108e10 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.000+0000 7f169259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169419c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.000+0000 7f169259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169419c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49258/0 (socket says 172.21.15.7:49258) 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.000+0000 7f169259c700 1 -- 172.21.15.7:0/2795680141 learned_addr learned my addr 172.21.15.7:0/2795680141 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.001+0000 7f169259c700 1 -- 172.21.15.7:0/2795680141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169419c6a0 con 0x7f1694108e10 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.001+0000 7f169259c700 1 --2- 172.21.15.7:0/2795680141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169419c160 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1684000c00 tx=0x7f1684011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.001+0000 7f16837fe700 1 -- 172.21.15.7:0/2795680141 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1684003aa0 con 0x7f1694108e10 2024-04-21T08:35:15.794 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.001+0000 7f16837fe700 1 -- 172.21.15.7:0/2795680141 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f168401b8f0 con 0x7f1694108e10 2024-04-21T08:35:15.795 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.001+0000 7f16837fe700 1 -- 172.21.15.7:0/2795680141 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1684023870 con 0x7f1694108e10 2024-04-21T08:35:15.795 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.001+0000 7f1698a36700 1 -- 172.21.15.7:0/2795680141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169419c930 con 0x7f1694108e10 2024-04-21T08:35:15.795 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.001+0000 7f1698a36700 1 -- 172.21.15.7:0/2795680141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f169419f620 con 0x7f1694108e10 2024-04-21T08:35:15.795 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.002+0000 7f16837fe700 1 -- 172.21.15.7:0/2795680141 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f1684023b70 con 0x7f1694108e10 2024-04-21T08:35:15.795 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.002+0000 7f16837fe700 1 --2- 172.21.15.7:0/2795680141 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f167c0396c0 0x7f167c03bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:15.795 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.002+0000 7f16837fe700 1 -- 172.21.15.7:0/2795680141 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f168404ef60 con 0x7f1694108e10 2024-04-21T08:35:15.795 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.002+0000 7f1691d9b700 1 --2- 172.21.15.7:0/2795680141 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f167c0396c0 0x7f167c03bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:15.795 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.002+0000 7f1698a36700 1 -- 172.21.15.7:0/2795680141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16940640a0 con 0x7f1694108e10 2024-04-21T08:35:15.795 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.003+0000 7f1691d9b700 1 --2- 172.21.15.7:0/2795680141 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f167c0396c0 0x7f167c03bb80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1688006fd0 tx=0x7f1688006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.008+0000 7f16837fe700 1 -- 172.21.15.7:0/2795680141 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f168401f030 con 0x7f1694108e10 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.225+0000 7f1698a36700 1 -- 172.21.15.7:0/2795680141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f169419f7f0 con 0x7f1694108e10 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.225+0000 7f16837fe700 1 -- 172.21.15.7:0/2795680141 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f168401a9c0 con 0x7f1694108e10 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.227+0000 7f1698a36700 1 -- 172.21.15.7:0/2795680141 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f167c0396c0 msgr2=0x7f167c03bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.227+0000 7f1698a36700 1 --2- 172.21.15.7:0/2795680141 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f167c0396c0 0x7f167c03bb80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1688006fd0 tx=0x7f1688006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.227+0000 7f1698a36700 1 -- 172.21.15.7:0/2795680141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 msgr2=0x7f169419c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.227+0000 7f1698a36700 1 --2- 172.21.15.7:0/2795680141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169419c160 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1684000c00 tx=0x7f1684011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.228+0000 7f1698a36700 1 -- 172.21.15.7:0/2795680141 shutdown_connections 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.228+0000 7f1698a36700 1 --2- 172.21.15.7:0/2795680141 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f167c0396c0 0x7f167c03bb80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.228+0000 7f1698a36700 1 --2- 172.21.15.7:0/2795680141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694108e10 0x7f169419c160 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.228+0000 7f1698a36700 1 -- 172.21.15.7:0/2795680141 >> 172.21.15.7:0/2795680141 conn(0x7f1694102820 msgr2=0x7f1694103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.228+0000 7f1698a36700 1 -- 172.21.15.7:0/2795680141 shutdown_connections 2024-04-21T08:35:15.796 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.228+0000 7f1698a36700 1 -- 172.21.15.7:0/2795680141 wait complete. 2024-04-21T08:35:15.799 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:35:15.800 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-04-21T08:35:15.807 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:35:15.807 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-04-21T08:35:15.807 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-04-21T08:35:15.807 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-21T08:35:15.808 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph Dashboard is now available at: 2024-04-21T08:35:15.808 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:15.809 INFO:teuthology.orchestra.run.smithi007.stdout: URL: https://smithi007:8443/ 2024-04-21T08:35:15.809 INFO:teuthology.orchestra.run.smithi007.stdout: User: admin 2024-04-21T08:35:15.809 INFO:teuthology.orchestra.run.smithi007.stdout: Password: qscdperxiu 2024-04-21T08:35:15.809 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:15.810 INFO:teuthology.orchestra.run.smithi007.stdout:Saving cluster configuration to /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config directory 2024-04-21T08:35:15.810 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling autotune for osd_memory_target 2024-04-21T08:35:16.998 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.218+0000 7f6731fe9700 1 Processor -- start 2024-04-21T08:35:16.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.218+0000 7f6731fe9700 1 -- start start 2024-04-21T08:35:16.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.218+0000 7f6731fe9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c1087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:16.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.218+0000 7f6731fe9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f672c108da0 con 0x7f672c1083b0 2024-04-21T08:35:16.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.219+0000 7f672b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c1087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:16.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.219+0000 7f672b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c1087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49274/0 (socket says 172.21.15.7:49274) 2024-04-21T08:35:16.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.219+0000 7f672b7fe700 1 -- 172.21.15.7:0/2709550572 learned_addr learned my addr 172.21.15.7:0/2709550572 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:16.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.219+0000 7f672b7fe700 1 -- 172.21.15.7:0/2709550572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672c109560 con 0x7f672c1083b0 2024-04-21T08:35:16.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.219+0000 7f672b7fe700 1 --2- 172.21.15.7:0/2709550572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c1087d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f671c009e80 tx=0x7f671c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b7bb8ada16f8e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.219+0000 7f672a7fc700 1 -- 172.21.15.7:0/2709550572 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f671c017070 con 0x7f672c1083b0 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.219+0000 7f672a7fc700 1 -- 172.21.15.7:0/2709550572 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f671c00b8c0 con 0x7f672c1083b0 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.219+0000 7f672a7fc700 1 -- 172.21.15.7:0/2709550572 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f671c004490 con 0x7f672c1083b0 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.220+0000 7f6731fe9700 1 -- 172.21.15.7:0/2709550572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 msgr2=0x7f672c1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.220+0000 7f6731fe9700 1 --2- 172.21.15.7:0/2709550572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c1087d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f671c009e80 tx=0x7f671c00b330 comp rx=0 tx=0).stop 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.220+0000 7f6731fe9700 1 -- 172.21.15.7:0/2709550572 shutdown_connections 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.220+0000 7f6731fe9700 1 --2- 172.21.15.7:0/2709550572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c1087d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.220+0000 7f6731fe9700 1 -- 172.21.15.7:0/2709550572 >> 172.21.15.7:0/2709550572 conn(0x7f672c1039b0 msgr2=0x7f672c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.220+0000 7f6731fe9700 1 -- 172.21.15.7:0/2709550572 shutdown_connections 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.220+0000 7f6731fe9700 1 -- 172.21.15.7:0/2709550572 wait complete. 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.221+0000 7f6731fe9700 1 Processor -- start 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.221+0000 7f6731fe9700 1 -- start start 2024-04-21T08:35:17.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.221+0000 7f6731fe9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c07db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.221+0000 7f6731fe9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f672c10c910 con 0x7f672c1083b0 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.221+0000 7f672b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c07db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.221+0000 7f672b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c07db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49290/0 (socket says 172.21.15.7:49290) 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.221+0000 7f672b7fe700 1 -- 172.21.15.7:0/748885785 learned_addr learned my addr 172.21.15.7:0/748885785 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.221+0000 7f672b7fe700 1 -- 172.21.15.7:0/748885785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672c07e0a0 con 0x7f672c1083b0 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.221+0000 7f672b7fe700 1 --2- 172.21.15.7:0/748885785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c07db60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f671c004220 tx=0x7f671c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.222+0000 7f6728ff9700 1 -- 172.21.15.7:0/748885785 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f671c020020 con 0x7f672c1083b0 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.222+0000 7f6728ff9700 1 -- 172.21.15.7:0/748885785 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f671c003b50 con 0x7f672c1083b0 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.222+0000 7f6731fe9700 1 -- 172.21.15.7:0/748885785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672c07c290 con 0x7f672c1083b0 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.222+0000 7f6728ff9700 1 -- 172.21.15.7:0/748885785 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f671c017040 con 0x7f672c1083b0 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.222+0000 7f6731fe9700 1 -- 172.21.15.7:0/748885785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f672c07c7d0 con 0x7f672c1083b0 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.222+0000 7f6728ff9700 1 -- 172.21.15.7:0/748885785 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f671c023020 con 0x7f672c1083b0 2024-04-21T08:35:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.223+0000 7f6728ff9700 1 --2- 172.21.15.7:0/748885785 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f67180396d0 0x7f671803bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.223+0000 7f6731fe9700 1 -- 172.21.15.7:0/748885785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f672c0640a0 con 0x7f672c1083b0 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.223+0000 7f6728ff9700 1 -- 172.21.15.7:0/748885785 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f671c00f050 con 0x7f672c1083b0 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.223+0000 7f672affd700 1 --2- 172.21.15.7:0/748885785 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f67180396d0 0x7f671803bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.223+0000 7f672affd700 1 --2- 172.21.15.7:0/748885785 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f67180396d0 0x7f671803bb90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6714006fd0 tx=0x7f6714006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.228+0000 7f6728ff9700 1 -- 172.21.15.7:0/748885785 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f671c02a020 con 0x7f672c1083b0 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.278+0000 7f6728ff9700 1 -- 172.21.15.7:0/748885785 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f671c015d10 con 0x7f672c1083b0 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.435+0000 7f6731fe9700 1 -- 172.21.15.7:0/748885785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f672c07cb10 con 0x7f672c1083b0 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.435+0000 7f6728ff9700 1 -- 172.21.15.7:0/748885785 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f672c07cb10 con 0x7f672c1083b0 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.437+0000 7f6731fe9700 1 -- 172.21.15.7:0/748885785 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f67180396d0 msgr2=0x7f671803bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.437+0000 7f6731fe9700 1 --2- 172.21.15.7:0/748885785 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f67180396d0 0x7f671803bb90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6714006fd0 tx=0x7f6714006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.437+0000 7f6731fe9700 1 -- 172.21.15.7:0/748885785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 msgr2=0x7f672c07db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:17.002 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.437+0000 7f6731fe9700 1 --2- 172.21.15.7:0/748885785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c07db60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f671c004220 tx=0x7f671c00bd30 comp rx=0 tx=0).stop 2024-04-21T08:35:17.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.437+0000 7f6731fe9700 1 -- 172.21.15.7:0/748885785 shutdown_connections 2024-04-21T08:35:17.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.437+0000 7f6731fe9700 1 --2- 172.21.15.7:0/748885785 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f67180396d0 0x7f671803bb90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:17.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.437+0000 7f6731fe9700 1 --2- 172.21.15.7:0/748885785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f672c1083b0 0x7f672c07db60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:17.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.437+0000 7f6731fe9700 1 -- 172.21.15.7:0/748885785 >> 172.21.15.7:0/748885785 conn(0x7f672c1039b0 msgr2=0x7f672c104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:17.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.438+0000 7f6731fe9700 1 -- 172.21.15.7:0/748885785 shutdown_connections 2024-04-21T08:35:17.003 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:16.438+0000 7f6731fe9700 1 -- 172.21.15.7:0/748885785 wait complete. 2024-04-21T08:35:17.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:17 smithi007 ceph-mon[28477]: mgrmap e13: x(active, since 6s) 2024-04-21T08:35:18.313 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.404+0000 7f591ae44700 1 Processor -- start 2024-04-21T08:35:18.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.404+0000 7f591ae44700 1 -- start start 2024-04-21T08:35:18.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.404+0000 7f591ae44700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f59141087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:18.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.404+0000 7f591ae44700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5914108da0 con 0x7f59141083b0 2024-04-21T08:35:18.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.404+0000 7f5918be0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f59141087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:18.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.404+0000 7f5918be0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f59141087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49300/0 (socket says 172.21.15.7:49300) 2024-04-21T08:35:18.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.404+0000 7f5918be0700 1 -- 172.21.15.7:0/1036255733 learned_addr learned my addr 172.21.15.7:0/1036255733 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:18.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.404+0000 7f5918be0700 1 -- 172.21.15.7:0/1036255733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5914109560 con 0x7f59141083b0 2024-04-21T08:35:18.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.405+0000 7f5918be0700 1 --2- 172.21.15.7:0/1036255733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f59141087d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f59080098a0 tx=0x7f5908009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e6f321792e2c57 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:18.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.405+0000 7f59137fe700 1 -- 172.21.15.7:0/1036255733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5908004160 con 0x7f59141083b0 2024-04-21T08:35:18.314 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.405+0000 7f59137fe700 1 -- 172.21.15.7:0/1036255733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5908004300 con 0x7f59141083b0 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.405+0000 7f59137fe700 1 -- 172.21.15.7:0/1036255733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5908011a20 con 0x7f59141083b0 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.405+0000 7f591ae44700 1 -- 172.21.15.7:0/1036255733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 msgr2=0x7f59141087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.405+0000 7f591ae44700 1 --2- 172.21.15.7:0/1036255733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f59141087d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f59080098a0 tx=0x7f5908009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.405+0000 7f591ae44700 1 -- 172.21.15.7:0/1036255733 shutdown_connections 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.405+0000 7f591ae44700 1 --2- 172.21.15.7:0/1036255733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f59141087d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.405+0000 7f591ae44700 1 -- 172.21.15.7:0/1036255733 >> 172.21.15.7:0/1036255733 conn(0x7f59141039b0 msgr2=0x7f5914105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.406+0000 7f591ae44700 1 -- 172.21.15.7:0/1036255733 shutdown_connections 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.406+0000 7f591ae44700 1 -- 172.21.15.7:0/1036255733 wait complete. 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.406+0000 7f591ae44700 1 Processor -- start 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.406+0000 7f591ae44700 1 -- start start 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.406+0000 7f591ae44700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f591407db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.406+0000 7f591ae44700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f591410c910 con 0x7f59141083b0 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.406+0000 7f5918be0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f591407db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.406+0000 7f5918be0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f591407db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49302/0 (socket says 172.21.15.7:49302) 2024-04-21T08:35:18.315 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.406+0000 7f5918be0700 1 -- 172.21.15.7:0/1547644380 learned_addr learned my addr 172.21.15.7:0/1547644380 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.407+0000 7f5918be0700 1 -- 172.21.15.7:0/1547644380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591407e0a0 con 0x7f59141083b0 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.407+0000 7f5918be0700 1 --2- 172.21.15.7:0/1547644380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f591407db60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f59080116e0 tx=0x7f5908011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.407+0000 7f5911ffb700 1 -- 172.21.15.7:0/1547644380 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59080039e0 con 0x7f59141083b0 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.407+0000 7f5911ffb700 1 -- 172.21.15.7:0/1547644380 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f590801b8f0 con 0x7f59141083b0 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.407+0000 7f591ae44700 1 -- 172.21.15.7:0/1547644380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591407c290 con 0x7f59141083b0 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.407+0000 7f5911ffb700 1 -- 172.21.15.7:0/1547644380 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5908023760 con 0x7f59141083b0 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.407+0000 7f591ae44700 1 -- 172.21.15.7:0/1547644380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f591407c7d0 con 0x7f59141083b0 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.408+0000 7f5911ffb700 1 -- 172.21.15.7:0/1547644380 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f59080239f0 con 0x7f59141083b0 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.408+0000 7f591ae44700 1 -- 172.21.15.7:0/1547644380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59140640a0 con 0x7f59141083b0 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.408+0000 7f5911ffb700 1 --2- 172.21.15.7:0/1547644380 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5904039710 0x7f590403bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.408+0000 7f5911ffb700 1 -- 172.21.15.7:0/1547644380 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f590804e120 con 0x7f59141083b0 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.409+0000 7f5913fff700 1 --2- 172.21.15.7:0/1547644380 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5904039710 0x7f590403bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:18.316 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.409+0000 7f5913fff700 1 --2- 172.21.15.7:0/1547644380 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5904039710 0x7f590403bbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5900006fd0 tx=0x7f5900006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:18.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.413+0000 7f5911ffb700 1 -- 172.21.15.7:0/1547644380 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f590801f030 con 0x7f59141083b0 2024-04-21T08:35:18.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.738+0000 7f591ae44700 1 -- 172.21.15.7:0/1547644380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f591407ca50 con 0x7f59141083b0 2024-04-21T08:35:18.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.741+0000 7f5911ffb700 1 -- 172.21.15.7:0/1547644380 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f5908018e30 con 0x7f59141083b0 2024-04-21T08:35:18.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.743+0000 7f591ae44700 1 -- 172.21.15.7:0/1547644380 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5904039710 msgr2=0x7f590403bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:18.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.743+0000 7f591ae44700 1 --2- 172.21.15.7:0/1547644380 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5904039710 0x7f590403bbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5900006fd0 tx=0x7f5900006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:18.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.743+0000 7f591ae44700 1 -- 172.21.15.7:0/1547644380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 msgr2=0x7f591407db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:18.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.743+0000 7f591ae44700 1 --2- 172.21.15.7:0/1547644380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f591407db60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f59080116e0 tx=0x7f5908011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:18.317 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.743+0000 7f591ae44700 1 -- 172.21.15.7:0/1547644380 shutdown_connections 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.743+0000 7f591ae44700 1 --2- 172.21.15.7:0/1547644380 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5904039710 0x7f590403bbd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.743+0000 7f591ae44700 1 --2- 172.21.15.7:0/1547644380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59141083b0 0x7f591407db60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.743+0000 7f591ae44700 1 -- 172.21.15.7:0/1547644380 >> 172.21.15.7:0/1547644380 conn(0x7f59141039b0 msgr2=0x7f59141043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.743+0000 7f591ae44700 1 -- 172.21.15.7:0/1547644380 shutdown_connections 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.743+0000 7f591ae44700 1 -- 172.21.15.7:0/1547644380 wait complete. 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout:Or, if you are only running a single cluster on this host: 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-21T08:35:18.318 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:18.319 INFO:teuthology.orchestra.run.smithi007.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-21T08:35:18.319 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:18.319 INFO:teuthology.orchestra.run.smithi007.stdout: ceph telemetry on 2024-04-21T08:35:18.319 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:18.319 INFO:teuthology.orchestra.run.smithi007.stdout:For more information see: 2024-04-21T08:35:18.319 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:18.319 INFO:teuthology.orchestra.run.smithi007.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-21T08:35:18.319 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:18.319 INFO:teuthology.orchestra.run.smithi007.stdout:Bootstrap complete. 2024-04-21T08:35:18.358 INFO:tasks.cephadm:Fetching config... 2024-04-21T08:35:18.359 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:35:18.359 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-21T08:35:18.377 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-21T08:35:18.377 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:35:18.377 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-21T08:35:18.433 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-21T08:35:18.433 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:35:18.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-21T08:35:18.500 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-21T08:35:18.501 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:35:18.501 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-21T08:35:18.556 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-21T08:35:18.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjzqzXzxtIZ99IN3sOmnj7DnU0JE0BpkzRL9ohpiOaPyh+gi2Cb/KrV1SizIKCVft1cfHcMEKrF/C+Bk6ez+lOnudgbkxrmIOtHAqmqunPmiH/ixL6k+To1Aoeb67MuaaKZRiPPdWhuBR5okYRDB14u+yIcmW97agV5ah5KVlFwbMSKEomPLgo6RERXRgbckqG0I9ZUwM9QClMCFEKFbie9aebQSc8sggzQsKVP8AKXSGlPA8QUtoILuGgNkW5MrmNEXFS2F8tcRvd0bXV6yLPyfeqfoEqaDcUoxSCYVPv7FzXlv/FN7p+6ArnVs13YeSlHVlW1lqb5Qb7JPE54mWugtCygGmMeNi8nCiY9H//eKwllj0OPlZU/MCgvkbk8DLP96zZPIweCFbLNVFNOb6I0pffbKSUylPCG8ydT0OCifE6VfCbXKmgPzHTHdZDeF8CpyGP63E+Hpt+mi+gkAzAW3RckqBRRlrX/DKFyfVY5SIycOAtjwr1Q+9SOUERDQk= ceph-c9c60282-ffb9-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:35:18.632 INFO:teuthology.orchestra.run.smithi007.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjzqzXzxtIZ99IN3sOmnj7DnU0JE0BpkzRL9ohpiOaPyh+gi2Cb/KrV1SizIKCVft1cfHcMEKrF/C+Bk6ez+lOnudgbkxrmIOtHAqmqunPmiH/ixL6k+To1Aoeb67MuaaKZRiPPdWhuBR5okYRDB14u+yIcmW97agV5ah5KVlFwbMSKEomPLgo6RERXRgbckqG0I9ZUwM9QClMCFEKFbie9aebQSc8sggzQsKVP8AKXSGlPA8QUtoILuGgNkW5MrmNEXFS2F8tcRvd0bXV6yLPyfeqfoEqaDcUoxSCYVPv7FzXlv/FN7p+6ArnVs13YeSlHVlW1lqb5Qb7JPE54mWugtCygGmMeNi8nCiY9H//eKwllj0OPlZU/MCgvkbk8DLP96zZPIweCFbLNVFNOb6I0pffbKSUylPCG8ydT0OCifE6VfCbXKmgPzHTHdZDeF8CpyGP63E+Hpt+mi+gkAzAW3RckqBRRlrX/DKFyfVY5SIycOAtjwr1Q+9SOUERDQk= ceph-c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:18.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjzqzXzxtIZ99IN3sOmnj7DnU0JE0BpkzRL9ohpiOaPyh+gi2Cb/KrV1SizIKCVft1cfHcMEKrF/C+Bk6ez+lOnudgbkxrmIOtHAqmqunPmiH/ixL6k+To1Aoeb67MuaaKZRiPPdWhuBR5okYRDB14u+yIcmW97agV5ah5KVlFwbMSKEomPLgo6RERXRgbckqG0I9ZUwM9QClMCFEKFbie9aebQSc8sggzQsKVP8AKXSGlPA8QUtoILuGgNkW5MrmNEXFS2F8tcRvd0bXV6yLPyfeqfoEqaDcUoxSCYVPv7FzXlv/FN7p+6ArnVs13YeSlHVlW1lqb5Qb7JPE54mWugtCygGmMeNi8nCiY9H//eKwllj0OPlZU/MCgvkbk8DLP96zZPIweCFbLNVFNOb6I0pffbKSUylPCG8ydT0OCifE6VfCbXKmgPzHTHdZDeF8CpyGP63E+Hpt+mi+gkAzAW3RckqBRRlrX/DKFyfVY5SIycOAtjwr1Q+9SOUERDQk= ceph-c9c60282-ffb9-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:35:18.683 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjzqzXzxtIZ99IN3sOmnj7DnU0JE0BpkzRL9ohpiOaPyh+gi2Cb/KrV1SizIKCVft1cfHcMEKrF/C+Bk6ez+lOnudgbkxrmIOtHAqmqunPmiH/ixL6k+To1Aoeb67MuaaKZRiPPdWhuBR5okYRDB14u+yIcmW97agV5ah5KVlFwbMSKEomPLgo6RERXRgbckqG0I9ZUwM9QClMCFEKFbie9aebQSc8sggzQsKVP8AKXSGlPA8QUtoILuGgNkW5MrmNEXFS2F8tcRvd0bXV6yLPyfeqfoEqaDcUoxSCYVPv7FzXlv/FN7p+6ArnVs13YeSlHVlW1lqb5Qb7JPE54mWugtCygGmMeNi8nCiY9H//eKwllj0OPlZU/MCgvkbk8DLP96zZPIweCFbLNVFNOb6I0pffbKSUylPCG8ydT0OCifE6VfCbXKmgPzHTHdZDeF8CpyGP63E+Hpt+mi+gkAzAW3RckqBRRlrX/DKFyfVY5SIycOAtjwr1Q+9SOUERDQk= ceph-c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:18.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjzqzXzxtIZ99IN3sOmnj7DnU0JE0BpkzRL9ohpiOaPyh+gi2Cb/KrV1SizIKCVft1cfHcMEKrF/C+Bk6ez+lOnudgbkxrmIOtHAqmqunPmiH/ixL6k+To1Aoeb67MuaaKZRiPPdWhuBR5okYRDB14u+yIcmW97agV5ah5KVlFwbMSKEomPLgo6RERXRgbckqG0I9ZUwM9QClMCFEKFbie9aebQSc8sggzQsKVP8AKXSGlPA8QUtoILuGgNkW5MrmNEXFS2F8tcRvd0bXV6yLPyfeqfoEqaDcUoxSCYVPv7FzXlv/FN7p+6ArnVs13YeSlHVlW1lqb5Qb7JPE54mWugtCygGmMeNi8nCiY9H//eKwllj0OPlZU/MCgvkbk8DLP96zZPIweCFbLNVFNOb6I0pffbKSUylPCG8ydT0OCifE6VfCbXKmgPzHTHdZDeF8CpyGP63E+Hpt+mi+gkAzAW3RckqBRRlrX/DKFyfVY5SIycOAtjwr1Q+9SOUERDQk= ceph-c9c60282-ffb9-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:35:18.733 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjzqzXzxtIZ99IN3sOmnj7DnU0JE0BpkzRL9ohpiOaPyh+gi2Cb/KrV1SizIKCVft1cfHcMEKrF/C+Bk6ez+lOnudgbkxrmIOtHAqmqunPmiH/ixL6k+To1Aoeb67MuaaKZRiPPdWhuBR5okYRDB14u+yIcmW97agV5ah5KVlFwbMSKEomPLgo6RERXRgbckqG0I9ZUwM9QClMCFEKFbie9aebQSc8sggzQsKVP8AKXSGlPA8QUtoILuGgNkW5MrmNEXFS2F8tcRvd0bXV6yLPyfeqfoEqaDcUoxSCYVPv7FzXlv/FN7p+6ArnVs13YeSlHVlW1lqb5Qb7JPE54mWugtCygGmMeNi8nCiY9H//eKwllj0OPlZU/MCgvkbk8DLP96zZPIweCFbLNVFNOb6I0pffbKSUylPCG8ydT0OCifE6VfCbXKmgPzHTHdZDeF8CpyGP63E+Hpt+mi+gkAzAW3RckqBRRlrX/DKFyfVY5SIycOAtjwr1Q+9SOUERDQk= ceph-c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:18.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-21T08:35:18.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:18 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1547644380' entity='client.admin' 2024-04-21T08:35:18.999 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:35:21.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.105+0000 7f9f08a43700 1 -- 172.21.15.7:0/1542626246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f041013b0 msgr2=0x7f9f041017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:21.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.105+0000 7f9f08a43700 1 --2- 172.21.15.7:0/1542626246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f041013b0 0x7f9f041017d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009910 tx=0x7f9ef4009c20 comp rx=0 tx=0).stop 2024-04-21T08:35:21.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.105+0000 7f9f08a43700 1 -- 172.21.15.7:0/1542626246 shutdown_connections 2024-04-21T08:35:21.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.105+0000 7f9f08a43700 1 --2- 172.21.15.7:0/1542626246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f041013b0 0x7f9f041017d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:21.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.105+0000 7f9f08a43700 1 -- 172.21.15.7:0/1542626246 >> 172.21.15.7:0/1542626246 conn(0x7f9f040fc950 msgr2=0x7f9f040fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:21.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.105+0000 7f9f08a43700 1 -- 172.21.15.7:0/1542626246 shutdown_connections 2024-04-21T08:35:21.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.105+0000 7f9f08a43700 1 -- 172.21.15.7:0/1542626246 wait complete. 2024-04-21T08:35:21.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.106+0000 7f9f08a43700 1 Processor -- start 2024-04-21T08:35:21.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.106+0000 7f9f08a43700 1 -- start start 2024-04-21T08:35:21.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.106+0000 7f9f08a43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f041013b0 0x7f9f04197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:21.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.106+0000 7f9f08a43700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9f041072f0 con 0x7f9f041013b0 2024-04-21T08:35:21.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.106+0000 7f9f0259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f041013b0 0x7f9f04197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:21.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.106+0000 7f9f0259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f041013b0 0x7f9f04197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49324/0 (socket says 172.21.15.7:49324) 2024-04-21T08:35:21.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.106+0000 7f9f0259c700 1 -- 172.21.15.7:0/218426275 learned_addr learned my addr 172.21.15.7:0/218426275 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:21.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.107+0000 7f9f0259c700 1 -- 172.21.15.7:0/218426275 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f04198010 con 0x7f9f041013b0 2024-04-21T08:35:21.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.107+0000 7f9f0259c700 1 --2- 172.21.15.7:0/218426275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f041013b0 0x7f9f04197ad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4005800 tx=0x7f9ef4009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:21.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.107+0000 7f9ef37fe700 1 -- 172.21.15.7:0/218426275 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ef4027070 con 0x7f9f041013b0 2024-04-21T08:35:21.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.107+0000 7f9ef37fe700 1 -- 172.21.15.7:0/218426275 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ef400be10 con 0x7f9f041013b0 2024-04-21T08:35:21.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.107+0000 7f9ef37fe700 1 -- 172.21.15.7:0/218426275 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ef400f460 con 0x7f9f041013b0 2024-04-21T08:35:21.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.107+0000 7f9f08a43700 1 -- 172.21.15.7:0/218426275 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f041982a0 con 0x7f9f041013b0 2024-04-21T08:35:21.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.107+0000 7f9f08a43700 1 -- 172.21.15.7:0/218426275 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0419af90 con 0x7f9f041013b0 2024-04-21T08:35:21.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.108+0000 7f9ef37fe700 1 -- 172.21.15.7:0/218426275 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f9ef402a070 con 0x7f9f041013b0 2024-04-21T08:35:21.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.108+0000 7f9ef37fe700 1 --2- 172.21.15.7:0/218426275 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9eec0396b0 0x7f9eec03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:21.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.108+0000 7f9ef37fe700 1 -- 172.21.15.7:0/218426275 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9ef401e030 con 0x7f9f041013b0 2024-04-21T08:35:21.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.108+0000 7f9f08a43700 1 -- 172.21.15.7:0/218426275 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f04107a70 con 0x7f9f041013b0 2024-04-21T08:35:21.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.108+0000 7f9f01d9b700 1 --2- 172.21.15.7:0/218426275 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9eec0396b0 0x7f9eec03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:21.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.109+0000 7f9f01d9b700 1 --2- 172.21.15.7:0/218426275 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9eec0396b0 0x7f9eec03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8006fd0 tx=0x7f9ef8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:21.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.114+0000 7f9ef37fe700 1 -- 172.21.15.7:0/218426275 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ef4031070 con 0x7f9f041013b0 2024-04-21T08:35:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.323+0000 7f9f08a43700 1 -- 172.21.15.7:0/218426275 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f9f0402e020 con 0x7f9f041013b0 2024-04-21T08:35:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.325+0000 7f9ef37fe700 1 -- 172.21.15.7:0/218426275 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f9ef4013070 con 0x7f9f041013b0 2024-04-21T08:35:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.328+0000 7f9f08a43700 1 -- 172.21.15.7:0/218426275 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9eec0396b0 msgr2=0x7f9eec03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.328+0000 7f9f08a43700 1 --2- 172.21.15.7:0/218426275 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9eec0396b0 0x7f9eec03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8006fd0 tx=0x7f9ef8006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.328+0000 7f9f08a43700 1 -- 172.21.15.7:0/218426275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f041013b0 msgr2=0x7f9f04197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.328+0000 7f9f08a43700 1 --2- 172.21.15.7:0/218426275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f041013b0 0x7f9f04197ad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4005800 tx=0x7f9ef4009f60 comp rx=0 tx=0).stop 2024-04-21T08:35:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.328+0000 7f9f08a43700 1 -- 172.21.15.7:0/218426275 shutdown_connections 2024-04-21T08:35:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.328+0000 7f9f08a43700 1 --2- 172.21.15.7:0/218426275 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9eec0396b0 0x7f9eec03bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.328+0000 7f9f08a43700 1 --2- 172.21.15.7:0/218426275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f041013b0 0x7f9f04197ad0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.328+0000 7f9f08a43700 1 -- 172.21.15.7:0/218426275 >> 172.21.15.7:0/218426275 conn(0x7f9f040fc950 msgr2=0x7f9f0406adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.328+0000 7f9f08a43700 1 -- 172.21.15.7:0/218426275 shutdown_connections 2024-04-21T08:35:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:21.328+0000 7f9f08a43700 1 -- 172.21.15.7:0/218426275 wait complete. 2024-04-21T08:35:21.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:21 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:21.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:21 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:21.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:21 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:21.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:21 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:21.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:21 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:21.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:21 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:21.865 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-21T08:35:21.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-21T08:35:22.112 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:35:22.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:22 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/218426275' entity='client.admin' 2024-04-21T08:35:22.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:22.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:23.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.504+0000 7fc9e463d700 1 -- 172.21.15.7:0/1004006049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9dc1013c0 msgr2=0x7fc9dc1017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.504+0000 7fc9e463d700 1 --2- 172.21.15.7:0/1004006049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9dc1013c0 0x7fc9dc1017e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc009910 tx=0x7fc9cc009c20 comp rx=0 tx=0).stop 2024-04-21T08:35:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e463d700 1 -- 172.21.15.7:0/1004006049 shutdown_connections 2024-04-21T08:35:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e463d700 1 --2- 172.21.15.7:0/1004006049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9dc1013c0 0x7fc9dc1017e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e463d700 1 -- 172.21.15.7:0/1004006049 >> 172.21.15.7:0/1004006049 conn(0x7fc9dc0fc980 msgr2=0x7fc9dc0feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e463d700 1 -- 172.21.15.7:0/1004006049 shutdown_connections 2024-04-21T08:35:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e463d700 1 -- 172.21.15.7:0/1004006049 wait complete. 2024-04-21T08:35:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e463d700 1 Processor -- start 2024-04-21T08:35:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e463d700 1 -- start start 2024-04-21T08:35:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e463d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9dc1013c0 0x7fc9dc0fe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e463d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc107300 con 0x7fc9dc1013c0 2024-04-21T08:35:23.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e23d9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9dc1013c0 0x7fc9dc0fe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:23.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e23d9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9dc1013c0 0x7fc9dc0fe7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49342/0 (socket says 172.21.15.7:49342) 2024-04-21T08:35:23.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.505+0000 7fc9e23d9700 1 -- 172.21.15.7:0/3927208092 learned_addr learned my addr 172.21.15.7:0/3927208092 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:23.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.506+0000 7fc9e23d9700 1 -- 172.21.15.7:0/3927208092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9dc0fed30 con 0x7fc9dc1013c0 2024-04-21T08:35:23.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.506+0000 7fc9e23d9700 1 --2- 172.21.15.7:0/3927208092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9dc1013c0 0x7fc9dc0fe7f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc005950 tx=0x7fc9cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:23.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.506+0000 7fc9d37fe700 1 -- 172.21.15.7:0/3927208092 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9cc026070 con 0x7fc9dc1013c0 2024-04-21T08:35:23.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.506+0000 7fc9d37fe700 1 -- 172.21.15.7:0/3927208092 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9cc00bc00 con 0x7fc9dc1013c0 2024-04-21T08:35:23.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.506+0000 7fc9e463d700 1 -- 172.21.15.7:0/3927208092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9dc0fcf20 con 0x7fc9dc1013c0 2024-04-21T08:35:23.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.506+0000 7fc9d37fe700 1 -- 172.21.15.7:0/3927208092 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9cc003a80 con 0x7fc9dc1013c0 2024-04-21T08:35:23.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.506+0000 7fc9e463d700 1 -- 172.21.15.7:0/3927208092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9dc0fd460 con 0x7fc9dc1013c0 2024-04-21T08:35:23.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.507+0000 7fc9d37fe700 1 -- 172.21.15.7:0/3927208092 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fc9cc029070 con 0x7fc9dc1013c0 2024-04-21T08:35:23.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.507+0000 7fc9d37fe700 1 --2- 172.21.15.7:0/3927208092 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc9c8039650 0x7fc9c803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:23.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.507+0000 7fc9d37fe700 1 -- 172.21.15.7:0/3927208092 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc9cc01e070 con 0x7fc9dc1013c0 2024-04-21T08:35:23.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.507+0000 7fc9e1bd8700 1 --2- 172.21.15.7:0/3927208092 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc9c8039650 0x7fc9c803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:23.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.507+0000 7fc9e463d700 1 -- 172.21.15.7:0/3927208092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9dc107a80 con 0x7fc9dc1013c0 2024-04-21T08:35:23.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.508+0000 7fc9e1bd8700 1 --2- 172.21.15.7:0/3927208092 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc9c8039650 0x7fc9c803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8006fd0 tx=0x7fc9d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:23.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.513+0000 7fc9d37fe700 1 -- 172.21.15.7:0/3927208092 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9cc030070 con 0x7fc9dc1013c0 2024-04-21T08:35:23.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.726+0000 7fc9e463d700 1 -- 172.21.15.7:0/3927208092 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fc9dc190fe0 con 0x7fc9c8039650 2024-04-21T08:35:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.729+0000 7fc9d37fe700 1 -- 172.21.15.7:0/3927208092 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc9dc190fe0 con 0x7fc9c8039650 2024-04-21T08:35:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.732+0000 7fc9e463d700 1 -- 172.21.15.7:0/3927208092 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc9c8039650 msgr2=0x7fc9c803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.732+0000 7fc9e463d700 1 --2- 172.21.15.7:0/3927208092 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc9c8039650 0x7fc9c803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8006fd0 tx=0x7fc9d8006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.732+0000 7fc9e463d700 1 -- 172.21.15.7:0/3927208092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9dc1013c0 msgr2=0x7fc9dc0fe7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.732+0000 7fc9e463d700 1 --2- 172.21.15.7:0/3927208092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9dc1013c0 0x7fc9dc0fe7f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc005950 tx=0x7fc9cc0057f0 comp rx=0 tx=0).stop 2024-04-21T08:35:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.732+0000 7fc9e463d700 1 -- 172.21.15.7:0/3927208092 shutdown_connections 2024-04-21T08:35:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.732+0000 7fc9e463d700 1 --2- 172.21.15.7:0/3927208092 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc9c8039650 0x7fc9c803bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.732+0000 7fc9e463d700 1 --2- 172.21.15.7:0/3927208092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9dc1013c0 0x7fc9dc0fe7f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.732+0000 7fc9e463d700 1 -- 172.21.15.7:0/3927208092 >> 172.21.15.7:0/3927208092 conn(0x7fc9dc0fc980 msgr2=0x7fc9dc06ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.732+0000 7fc9e463d700 1 -- 172.21.15.7:0/3927208092 shutdown_connections 2024-04-21T08:35:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:23.732+0000 7fc9e463d700 1 -- 172.21.15.7:0/3927208092 wait complete. 2024-04-21T08:35:24.443 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi016 2024-04-21T08:35:24.443 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-21T08:35:24.443 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/etc/ceph/ceph.conf 2024-04-21T08:35:24.463 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-21T08:35:24.463 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:35:24.521 INFO:tasks.cephadm:Adding host smithi016 to orchestrator... 2024-04-21T08:35:24.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch host add smithi016 2024-04-21T08:35:24.783 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:35:24.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:24.881 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:24.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:24.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:24.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:24.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-21T08:35:24.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: Updating smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:35:24.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:35:24.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: Updating smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:35:24.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:24.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:24.882 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:26.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.318+0000 7f4307540700 1 -- 172.21.15.7:0/3962898197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4300108550 msgr2=0x7f4300108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:26.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.318+0000 7f4307540700 1 --2- 172.21.15.7:0/3962898197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4300108550 0x7f4300108930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009910 tx=0x7f42fc009c20 comp rx=0 tx=0).stop 2024-04-21T08:35:26.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.318+0000 7f4307540700 1 -- 172.21.15.7:0/3962898197 shutdown_connections 2024-04-21T08:35:26.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.318+0000 7f4307540700 1 --2- 172.21.15.7:0/3962898197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4300108550 0x7f4300108930 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:26.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.318+0000 7f4307540700 1 -- 172.21.15.7:0/3962898197 >> 172.21.15.7:0/3962898197 conn(0x7f43000fdf70 msgr2=0x7f4300100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:26.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.318+0000 7f4307540700 1 -- 172.21.15.7:0/3962898197 shutdown_connections 2024-04-21T08:35:26.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.318+0000 7f4307540700 1 -- 172.21.15.7:0/3962898197 wait complete. 2024-04-21T08:35:26.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.319+0000 7f4307540700 1 Processor -- start 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.319+0000 7f4307540700 1 -- start start 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.319+0000 7f4307540700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4300108550 0x7f4300197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.319+0000 7f4307540700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f430010c9c0 con 0x7f4300108550 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.319+0000 7f43052dc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4300108550 0x7f4300197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.319+0000 7f43052dc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4300108550 0x7f4300197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59880/0 (socket says 172.21.15.7:59880) 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.319+0000 7f43052dc700 1 -- 172.21.15.7:0/1939258046 learned_addr learned my addr 172.21.15.7:0/1939258046 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.319+0000 7f43052dc700 1 -- 172.21.15.7:0/1939258046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43001983a0 con 0x7f4300108550 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.320+0000 7f43052dc700 1 --2- 172.21.15.7:0/1939258046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4300108550 0x7f4300197e60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f42fc006010 tx=0x7f42fc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.320+0000 7f42f67fc700 1 -- 172.21.15.7:0/1939258046 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42fc026070 con 0x7f4300108550 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.320+0000 7f42f67fc700 1 -- 172.21.15.7:0/1939258046 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42fc00bc00 con 0x7f4300108550 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.320+0000 7f4307540700 1 -- 172.21.15.7:0/1939258046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4300198630 con 0x7f4300108550 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.320+0000 7f42f67fc700 1 -- 172.21.15.7:0/1939258046 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42fc003910 con 0x7f4300108550 2024-04-21T08:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.320+0000 7f4307540700 1 -- 172.21.15.7:0/1939258046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f430019c730 con 0x7f4300108550 2024-04-21T08:35:26.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.321+0000 7f42f67fc700 1 -- 172.21.15.7:0/1939258046 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f42fc029070 con 0x7f4300108550 2024-04-21T08:35:26.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.321+0000 7f42f67fc700 1 --2- 172.21.15.7:0/1939258046 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42ec0396b0 0x7f42ec03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:26.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.321+0000 7f42f67fc700 1 -- 172.21.15.7:0/1939258046 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f42fc01e070 con 0x7f4300108550 2024-04-21T08:35:26.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.321+0000 7f4307540700 1 -- 172.21.15.7:0/1939258046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4300103bc0 con 0x7f4300108550 2024-04-21T08:35:26.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.321+0000 7f4304adb700 1 --2- 172.21.15.7:0/1939258046 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42ec0396b0 0x7f42ec03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:26.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.321+0000 7f4304adb700 1 --2- 172.21.15.7:0/1939258046 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42ec0396b0 0x7f42ec03bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f42f0006fd0 tx=0x7f42f0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:26.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.326+0000 7f42f67fc700 1 -- 172.21.15.7:0/1939258046 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f42fc030020 con 0x7f4300108550 2024-04-21T08:35:26.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:26.537+0000 7f4307540700 1 -- 172.21.15.7:0/1939258046 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi016", "target": ["mon-mgr", ""]}) -- 0x7f43001991c0 con 0x7f42ec0396b0 2024-04-21T08:35:27.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:27 smithi007 ceph-mon[28477]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:27.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:27 smithi007 ceph-mon[28477]: Deploying cephadm binary to smithi016 2024-04-21T08:35:30.817 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi016' with addr '172.21.15.16' 2024-04-21T08:35:30.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:30.817+0000 7f42f67fc700 1 -- 172.21.15.7:0/1939258046 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f43001991c0 con 0x7f42ec0396b0 2024-04-21T08:35:30.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:30 smithi007 ceph-mon[28477]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:30.820+0000 7f4307540700 1 -- 172.21.15.7:0/1939258046 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42ec0396b0 msgr2=0x7f42ec03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:30.820+0000 7f4307540700 1 --2- 172.21.15.7:0/1939258046 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42ec0396b0 0x7f42ec03bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f42f0006fd0 tx=0x7f42f0006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:30.820+0000 7f4307540700 1 -- 172.21.15.7:0/1939258046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4300108550 msgr2=0x7f4300197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:30.820+0000 7f4307540700 1 --2- 172.21.15.7:0/1939258046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4300108550 0x7f4300197e60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f42fc006010 tx=0x7f42fc005780 comp rx=0 tx=0).stop 2024-04-21T08:35:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:30.820+0000 7f4307540700 1 -- 172.21.15.7:0/1939258046 shutdown_connections 2024-04-21T08:35:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:30.820+0000 7f4307540700 1 --2- 172.21.15.7:0/1939258046 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42ec0396b0 0x7f42ec03bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:30.820+0000 7f4307540700 1 --2- 172.21.15.7:0/1939258046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4300108550 0x7f4300197e60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:30.820+0000 7f4307540700 1 -- 172.21.15.7:0/1939258046 >> 172.21.15.7:0/1939258046 conn(0x7f43000fdf70 msgr2=0x7f43000fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:30.821+0000 7f4307540700 1 -- 172.21.15.7:0/1939258046 shutdown_connections 2024-04-21T08:35:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:30.821+0000 7f4307540700 1 -- 172.21.15.7:0/1939258046 wait complete. 2024-04-21T08:35:31.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T08:35:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:35:32.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:31 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:32.129 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:31 smithi007 ceph-mon[28477]: Added host smithi016 2024-04-21T08:35:32.129 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:31 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:32.129 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:31 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:33.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:32 smithi007 ceph-mon[28477]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:33.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:32 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:33.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.131+0000 7f80c631d700 1 -- 172.21.15.7:0/226437490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c0106360 msgr2=0x7f80c0106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:33.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.131+0000 7f80c631d700 1 --2- 172.21.15.7:0/226437490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c0106360 0x7f80c0106740 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009910 tx=0x7f80b0009c20 comp rx=0 tx=0).stop 2024-04-21T08:35:33.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.131+0000 7f80c631d700 1 -- 172.21.15.7:0/226437490 shutdown_connections 2024-04-21T08:35:33.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.131+0000 7f80c631d700 1 --2- 172.21.15.7:0/226437490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c0106360 0x7f80c0106740 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:33.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.131+0000 7f80c631d700 1 -- 172.21.15.7:0/226437490 >> 172.21.15.7:0/226437490 conn(0x7f80c0077af0 msgr2=0x7f80c0077f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:33.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.131+0000 7f80c631d700 1 -- 172.21.15.7:0/226437490 shutdown_connections 2024-04-21T08:35:33.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.131+0000 7f80c631d700 1 -- 172.21.15.7:0/226437490 wait complete. 2024-04-21T08:35:33.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.132+0000 7f80c631d700 1 Processor -- start 2024-04-21T08:35:33.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.132+0000 7f80c631d700 1 -- start start 2024-04-21T08:35:33.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.132+0000 7f80c631d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c0106360 0x7f80c01071a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:33.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.132+0000 7f80c631d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f80c010b990 con 0x7f80c0106360 2024-04-21T08:35:33.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.132+0000 7f80bffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c0106360 0x7f80c01071a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:33.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.132+0000 7f80bffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c0106360 0x7f80c01071a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59898/0 (socket says 172.21.15.7:59898) 2024-04-21T08:35:33.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.132+0000 7f80bffff700 1 -- 172.21.15.7:0/3936078323 learned_addr learned my addr 172.21.15.7:0/3936078323 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:33.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.132+0000 7f80bffff700 1 -- 172.21.15.7:0/3936078323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c0109090 con 0x7f80c0106360 2024-04-21T08:35:33.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.132+0000 7f80bffff700 1 --2- 172.21.15.7:0/3936078323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c0106360 0x7f80c01071a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f80b00056d0 tx=0x7f80b00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:33.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.133+0000 7f80bdffb700 1 -- 172.21.15.7:0/3936078323 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80b0026070 con 0x7f80c0106360 2024-04-21T08:35:33.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.133+0000 7f80bdffb700 1 -- 172.21.15.7:0/3936078323 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80b000bc00 con 0x7f80c0106360 2024-04-21T08:35:33.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.133+0000 7f80bdffb700 1 -- 172.21.15.7:0/3936078323 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80b0003920 con 0x7f80c0106360 2024-04-21T08:35:33.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.133+0000 7f80c631d700 1 -- 172.21.15.7:0/3936078323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c01076e0 con 0x7f80c0106360 2024-04-21T08:35:33.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.133+0000 7f80c631d700 1 -- 172.21.15.7:0/3936078323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c0107ba0 con 0x7f80c0106360 2024-04-21T08:35:33.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.134+0000 7f80bdffb700 1 -- 172.21.15.7:0/3936078323 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f80b0029070 con 0x7f80c0106360 2024-04-21T08:35:33.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.134+0000 7f80c631d700 1 -- 172.21.15.7:0/3936078323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c0102b20 con 0x7f80c0106360 2024-04-21T08:35:33.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.134+0000 7f80bdffb700 1 --2- 172.21.15.7:0/3936078323 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f80ac039650 0x7f80ac03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:33.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.134+0000 7f80bdffb700 1 -- 172.21.15.7:0/3936078323 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f80b001e070 con 0x7f80c0106360 2024-04-21T08:35:33.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.134+0000 7f80b7fff700 1 --2- 172.21.15.7:0/3936078323 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f80ac039650 0x7f80ac03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:33.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.134+0000 7f80b7fff700 1 --2- 172.21.15.7:0/3936078323 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f80ac039650 0x7f80ac03bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f80a8006fd0 tx=0x7f80a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:33.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.139+0000 7f80bdffb700 1 -- 172.21.15.7:0/3936078323 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80b0013070 con 0x7f80c0106360 2024-04-21T08:35:33.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.352+0000 7f80c631d700 1 -- 172.21.15.7:0/3936078323 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f80c0107dd0 con 0x7f80ac039650 2024-04-21T08:35:33.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.353+0000 7f80bdffb700 1 -- 172.21.15.7:0/3936078323 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+158 (secure 0 0 0) 0x7f80c0107dd0 con 0x7f80ac039650 2024-04-21T08:35:33.352 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:33.352 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}] 2024-04-21T08:35:33.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.355+0000 7f80c631d700 1 -- 172.21.15.7:0/3936078323 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f80ac039650 msgr2=0x7f80ac03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:33.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.355+0000 7f80c631d700 1 --2- 172.21.15.7:0/3936078323 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f80ac039650 0x7f80ac03bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f80a8006fd0 tx=0x7f80a8006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:33.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.355+0000 7f80c631d700 1 -- 172.21.15.7:0/3936078323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c0106360 msgr2=0x7f80c01071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:33.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.355+0000 7f80c631d700 1 --2- 172.21.15.7:0/3936078323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c0106360 0x7f80c01071a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f80b00056d0 tx=0x7f80b00057b0 comp rx=0 tx=0).stop 2024-04-21T08:35:33.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.355+0000 7f80c631d700 1 -- 172.21.15.7:0/3936078323 shutdown_connections 2024-04-21T08:35:33.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.355+0000 7f80c631d700 1 --2- 172.21.15.7:0/3936078323 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f80ac039650 0x7f80ac03bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:33.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.355+0000 7f80c631d700 1 --2- 172.21.15.7:0/3936078323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80c0106360 0x7f80c01071a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:33.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.355+0000 7f80c631d700 1 -- 172.21.15.7:0/3936078323 >> 172.21.15.7:0/3936078323 conn(0x7f80c0077af0 msgr2=0x7f80c0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:33.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.355+0000 7f80c631d700 1 -- 172.21.15.7:0/3936078323 shutdown_connections 2024-04-21T08:35:33.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:33.355+0000 7f80c631d700 1 -- 172.21.15.7:0/3936078323 wait complete. 2024-04-21T08:35:33.925 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi077 2024-04-21T08:35:33.925 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-21T08:35:33.925 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.conf 2024-04-21T08:35:33.943 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-21T08:35:33.943 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:35:34.000 INFO:tasks.cephadm:Adding host smithi077 to orchestrator... 2024-04-21T08:35:34.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch host add smithi077 2024-04-21T08:35:34.241 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:35:34.355 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:33 smithi007 ceph-mon[28477]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:35:34.355 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:35.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:34 smithi007 ceph-mon[28477]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:35.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.629+0000 7faa54912700 1 -- 172.21.15.7:0/2760128673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa501015f0 msgr2=0x7faa501019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:35.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.629+0000 7faa54912700 1 --2- 172.21.15.7:0/2760128673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa501015f0 0x7faa501019d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faa44009910 tx=0x7faa44009c20 comp rx=0 tx=0).stop 2024-04-21T08:35:35.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.629+0000 7faa54912700 1 -- 172.21.15.7:0/2760128673 shutdown_connections 2024-04-21T08:35:35.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.629+0000 7faa54912700 1 --2- 172.21.15.7:0/2760128673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa501015f0 0x7faa501019d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:35.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.629+0000 7faa54912700 1 -- 172.21.15.7:0/2760128673 >> 172.21.15.7:0/2760128673 conn(0x7faa500fcea0 msgr2=0x7faa500ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:35.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.629+0000 7faa54912700 1 -- 172.21.15.7:0/2760128673 shutdown_connections 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.629+0000 7faa54912700 1 -- 172.21.15.7:0/2760128673 wait complete. 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.630+0000 7faa54912700 1 Processor -- start 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.630+0000 7faa54912700 1 -- start start 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.630+0000 7faa54912700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa501015f0 0x7faa50198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.630+0000 7faa54912700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7faa5010b870 con 0x7faa501015f0 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.630+0000 7faa4e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa501015f0 0x7faa50198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.630+0000 7faa4e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa501015f0 0x7faa50198d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45540/0 (socket says 172.21.15.7:45540) 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.630+0000 7faa4e59c700 1 -- 172.21.15.7:0/2040659151 learned_addr learned my addr 172.21.15.7:0/2040659151 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.630+0000 7faa4e59c700 1 -- 172.21.15.7:0/2040659151 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa50199270 con 0x7faa501015f0 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.631+0000 7faa4e59c700 1 --2- 172.21.15.7:0/2040659151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa501015f0 0x7faa50198d30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faa440056d0 tx=0x7faa440057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.631+0000 7faa3f7fe700 1 -- 172.21.15.7:0/2040659151 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa44026070 con 0x7faa501015f0 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.631+0000 7faa3f7fe700 1 -- 172.21.15.7:0/2040659151 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa4400bba0 con 0x7faa501015f0 2024-04-21T08:35:35.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.631+0000 7faa54912700 1 -- 172.21.15.7:0/2040659151 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa50199560 con 0x7faa501015f0 2024-04-21T08:35:35.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.631+0000 7faa3f7fe700 1 -- 172.21.15.7:0/2040659151 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa44003920 con 0x7faa501015f0 2024-04-21T08:35:35.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.631+0000 7faa54912700 1 -- 172.21.15.7:0/2040659151 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa50199a20 con 0x7faa501015f0 2024-04-21T08:35:35.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.632+0000 7faa3f7fe700 1 -- 172.21.15.7:0/2040659151 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7faa44029070 con 0x7faa501015f0 2024-04-21T08:35:35.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.632+0000 7faa54912700 1 -- 172.21.15.7:0/2040659151 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa50102ad0 con 0x7faa501015f0 2024-04-21T08:35:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.632+0000 7faa3f7fe700 1 --2- 172.21.15.7:0/2040659151 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7faa38039650 0x7faa3803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.632+0000 7faa3f7fe700 1 -- 172.21.15.7:0/2040659151 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faa4401e070 con 0x7faa501015f0 2024-04-21T08:35:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.632+0000 7faa4dd9b700 1 --2- 172.21.15.7:0/2040659151 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7faa38039650 0x7faa3803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.633+0000 7faa4dd9b700 1 --2- 172.21.15.7:0/2040659151 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7faa38039650 0x7faa3803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faa40006fd0 tx=0x7faa40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:35.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.637+0000 7faa3f7fe700 1 -- 172.21.15.7:0/2040659151 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa44013070 con 0x7faa501015f0 2024-04-21T08:35:35.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:35.848+0000 7faa54912700 1 -- 172.21.15.7:0/2040659151 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi077", "target": ["mon-mgr", ""]}) -- 0x7faa5006aec0 con 0x7faa38039650 2024-04-21T08:35:37.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:36 smithi007 ceph-mon[28477]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:37.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:36 smithi007 ceph-mon[28477]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:37.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:37.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:37.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:37.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:37.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:37.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:37.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:38.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:37 smithi007 ceph-mon[28477]: Deploying cephadm binary to smithi077 2024-04-21T08:35:38.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:37 smithi007 ceph-mon[28477]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-21T08:35:38.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:37 smithi007 ceph-mon[28477]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:35:39.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:38 smithi007 ceph-mon[28477]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:40.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:40.146+0000 7faa3f7fe700 1 -- 172.21.15.7:0/2040659151 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7faa5006aec0 con 0x7faa38039650 2024-04-21T08:35:40.146 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi077' with addr '172.21.15.77' 2024-04-21T08:35:40.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:40.148+0000 7faa54912700 1 -- 172.21.15.7:0/2040659151 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7faa38039650 msgr2=0x7faa3803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:40.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:40.148+0000 7faa54912700 1 --2- 172.21.15.7:0/2040659151 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7faa38039650 0x7faa3803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faa40006fd0 tx=0x7faa40006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:40.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:40.148+0000 7faa54912700 1 -- 172.21.15.7:0/2040659151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa501015f0 msgr2=0x7faa50198d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:40.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:40.148+0000 7faa54912700 1 --2- 172.21.15.7:0/2040659151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa501015f0 0x7faa50198d30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faa440056d0 tx=0x7faa440057b0 comp rx=0 tx=0).stop 2024-04-21T08:35:40.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:40.148+0000 7faa54912700 1 -- 172.21.15.7:0/2040659151 shutdown_connections 2024-04-21T08:35:40.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:40.148+0000 7faa54912700 1 --2- 172.21.15.7:0/2040659151 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7faa38039650 0x7faa3803bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:40.148+0000 7faa54912700 1 --2- 172.21.15.7:0/2040659151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa501015f0 0x7faa50198d30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:40.148+0000 7faa54912700 1 -- 172.21.15.7:0/2040659151 >> 172.21.15.7:0/2040659151 conn(0x7faa500fcea0 msgr2=0x7faa50102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:40.148+0000 7faa54912700 1 -- 172.21.15.7:0/2040659151 shutdown_connections 2024-04-21T08:35:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:40.148+0000 7faa54912700 1 -- 172.21.15.7:0/2040659151 wait complete. 2024-04-21T08:35:40.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T08:35:41.094 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:35:41.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:40 smithi007 ceph-mon[28477]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:35:41.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:40 smithi007 ceph-mon[28477]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:40 smithi007 ceph-mon[28477]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:35:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:40 smithi007 ceph-mon[28477]: Added host smithi077 2024-04-21T08:35:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:42.525 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:42 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:42.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.554+0000 7ff2de6b0700 1 -- 172.21.15.7:0/305609740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2d81015b0 msgr2=0x7ff2d8101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:42.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.554+0000 7ff2de6b0700 1 --2- 172.21.15.7:0/305609740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2d81015b0 0x7ff2d8101990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009910 tx=0x7ff2c8009c20 comp rx=0 tx=0).stop 2024-04-21T08:35:42.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.554+0000 7ff2de6b0700 1 -- 172.21.15.7:0/305609740 shutdown_connections 2024-04-21T08:35:42.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.554+0000 7ff2de6b0700 1 --2- 172.21.15.7:0/305609740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2d81015b0 0x7ff2d8101990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:42.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.554+0000 7ff2de6b0700 1 -- 172.21.15.7:0/305609740 >> 172.21.15.7:0/305609740 conn(0x7ff2d80fce40 msgr2=0x7ff2d80ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:42.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.554+0000 7ff2de6b0700 1 -- 172.21.15.7:0/305609740 shutdown_connections 2024-04-21T08:35:42.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.554+0000 7ff2de6b0700 1 -- 172.21.15.7:0/305609740 wait complete. 2024-04-21T08:35:42.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.555+0000 7ff2de6b0700 1 Processor -- start 2024-04-21T08:35:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.555+0000 7ff2de6b0700 1 -- start start 2024-04-21T08:35:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.555+0000 7ff2de6b0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2d81015b0 0x7ff2d8198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.555+0000 7ff2de6b0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d810b830 con 0x7ff2d81015b0 2024-04-21T08:35:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.555+0000 7ff2d7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2d81015b0 0x7ff2d8198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.555+0000 7ff2d7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2d81015b0 0x7ff2d8198d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45552/0 (socket says 172.21.15.7:45552) 2024-04-21T08:35:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.555+0000 7ff2d7fff700 1 -- 172.21.15.7:0/3671011379 learned_addr learned my addr 172.21.15.7:0/3671011379 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.556+0000 7ff2d7fff700 1 -- 172.21.15.7:0/3671011379 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d81992b0 con 0x7ff2d81015b0 2024-04-21T08:35:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.556+0000 7ff2d7fff700 1 --2- 172.21.15.7:0/3671011379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2d81015b0 0x7ff2d8198d70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff2c80056d0 tx=0x7ff2c80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.556+0000 7ff2d57fa700 1 -- 172.21.15.7:0/3671011379 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2c8026070 con 0x7ff2d81015b0 2024-04-21T08:35:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.556+0000 7ff2d57fa700 1 -- 172.21.15.7:0/3671011379 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2c800bba0 con 0x7ff2d81015b0 2024-04-21T08:35:42.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.556+0000 7ff2de6b0700 1 -- 172.21.15.7:0/3671011379 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d81995a0 con 0x7ff2d81015b0 2024-04-21T08:35:42.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.556+0000 7ff2d57fa700 1 -- 172.21.15.7:0/3671011379 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2c8003920 con 0x7ff2d81015b0 2024-04-21T08:35:42.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.556+0000 7ff2de6b0700 1 -- 172.21.15.7:0/3671011379 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d8199a60 con 0x7ff2d81015b0 2024-04-21T08:35:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.557+0000 7ff2d57fa700 1 -- 172.21.15.7:0/3671011379 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7ff2c8029070 con 0x7ff2d81015b0 2024-04-21T08:35:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.557+0000 7ff2d57fa700 1 --2- 172.21.15.7:0/3671011379 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff2c4039650 0x7ff2c403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.557+0000 7ff2d57fa700 1 -- 172.21.15.7:0/3671011379 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff2c801e070 con 0x7ff2d81015b0 2024-04-21T08:35:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.557+0000 7ff2d77fe700 1 --2- 172.21.15.7:0/3671011379 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff2c4039650 0x7ff2c403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.557+0000 7ff2de6b0700 1 -- 172.21.15.7:0/3671011379 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d8102a90 con 0x7ff2d81015b0 2024-04-21T08:35:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.558+0000 7ff2d77fe700 1 --2- 172.21.15.7:0/3671011379 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff2c4039650 0x7ff2c403bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0006fd0 tx=0x7ff2c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:42.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.563+0000 7ff2d57fa700 1 -- 172.21.15.7:0/3671011379 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff2c8030070 con 0x7ff2d81015b0 2024-04-21T08:35:42.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.776+0000 7ff2de6b0700 1 -- 172.21.15.7:0/3671011379 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff2d80640a0 con 0x7ff2c4039650 2024-04-21T08:35:42.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.777+0000 7ff2d57fa700 1 -- 172.21.15.7:0/3671011379 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+237 (secure 0 0 0) 0x7ff2d80640a0 con 0x7ff2c4039650 2024-04-21T08:35:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:35:42.778 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}] 2024-04-21T08:35:42.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.780+0000 7ff2de6b0700 1 -- 172.21.15.7:0/3671011379 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff2c4039650 msgr2=0x7ff2c403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:42.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.780+0000 7ff2de6b0700 1 --2- 172.21.15.7:0/3671011379 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff2c4039650 0x7ff2c403bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0006fd0 tx=0x7ff2c0006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:42.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.780+0000 7ff2de6b0700 1 -- 172.21.15.7:0/3671011379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2d81015b0 msgr2=0x7ff2d8198d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.780+0000 7ff2de6b0700 1 --2- 172.21.15.7:0/3671011379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2d81015b0 0x7ff2d8198d70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff2c80056d0 tx=0x7ff2c80057b0 comp rx=0 tx=0).stop 2024-04-21T08:35:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.780+0000 7ff2de6b0700 1 -- 172.21.15.7:0/3671011379 shutdown_connections 2024-04-21T08:35:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.780+0000 7ff2de6b0700 1 --2- 172.21.15.7:0/3671011379 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff2c4039650 0x7ff2c403bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.780+0000 7ff2de6b0700 1 --2- 172.21.15.7:0/3671011379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2d81015b0 0x7ff2d8198d70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.780+0000 7ff2de6b0700 1 -- 172.21.15.7:0/3671011379 >> 172.21.15.7:0/3671011379 conn(0x7ff2d80fce40 msgr2=0x7ff2d8109120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.780+0000 7ff2de6b0700 1 -- 172.21.15.7:0/3671011379 shutdown_connections 2024-04-21T08:35:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:42.780+0000 7ff2de6b0700 1 -- 172.21.15.7:0/3671011379 wait complete. 2024-04-21T08:35:43.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:43 smithi007 ceph-mon[28477]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:43.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:43 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:43.369 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-21T08:35:43.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-21T08:35:43.629 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:35:44.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:44 smithi007 ceph-mon[28477]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:35:45.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.105+0000 7f6315e5a700 1 -- 172.21.15.7:0/2823872750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63101015f0 msgr2=0x7f63101019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:45.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.105+0000 7f6315e5a700 1 --2- 172.21.15.7:0/2823872750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63101015f0 0x7f63101019d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6300009910 tx=0x7f6300009c20 comp rx=0 tx=0).stop 2024-04-21T08:35:45.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.105+0000 7f6315e5a700 1 -- 172.21.15.7:0/2823872750 shutdown_connections 2024-04-21T08:35:45.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.105+0000 7f6315e5a700 1 --2- 172.21.15.7:0/2823872750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63101015f0 0x7f63101019d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:45.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.105+0000 7f6315e5a700 1 -- 172.21.15.7:0/2823872750 >> 172.21.15.7:0/2823872750 conn(0x7f6310076ea0 msgr2=0x7f63100772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:45.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.105+0000 7f6315e5a700 1 -- 172.21.15.7:0/2823872750 shutdown_connections 2024-04-21T08:35:45.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.105+0000 7f6315e5a700 1 -- 172.21.15.7:0/2823872750 wait complete. 2024-04-21T08:35:45.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.105+0000 7f6315e5a700 1 Processor -- start 2024-04-21T08:35:45.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.106+0000 7f6315e5a700 1 -- start start 2024-04-21T08:35:45.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.106+0000 7f6315e5a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63101015f0 0x7f6310196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.106+0000 7f6315e5a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f631010b820 con 0x7f63101015f0 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.106+0000 7f630f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63101015f0 0x7f6310196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.106+0000 7f630f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63101015f0 0x7f6310196d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53606/0 (socket says 172.21.15.7:53606) 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.106+0000 7f630f7fe700 1 -- 172.21.15.7:0/3843707914 learned_addr learned my addr 172.21.15.7:0/3843707914 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.106+0000 7f630f7fe700 1 -- 172.21.15.7:0/3843707914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63101972c0 con 0x7f63101015f0 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.107+0000 7f630f7fe700 1 --2- 172.21.15.7:0/3843707914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63101015f0 0x7f6310196d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f63000056d0 tx=0x7f63000057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.107+0000 7f630d7fa700 1 -- 172.21.15.7:0/3843707914 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6300026070 con 0x7f63101015f0 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.107+0000 7f6315e5a700 1 -- 172.21.15.7:0/3843707914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63101975b0 con 0x7f63101015f0 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.107+0000 7f630d7fa700 1 -- 172.21.15.7:0/3843707914 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f630000bba0 con 0x7f63101015f0 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.107+0000 7f630d7fa700 1 -- 172.21.15.7:0/3843707914 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6300003920 con 0x7f63101015f0 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.107+0000 7f6315e5a700 1 -- 172.21.15.7:0/3843707914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f631019b650 con 0x7f63101015f0 2024-04-21T08:35:45.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.108+0000 7f630d7fa700 1 -- 172.21.15.7:0/3843707914 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f6300029070 con 0x7f63101015f0 2024-04-21T08:35:45.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.108+0000 7f630d7fa700 1 --2- 172.21.15.7:0/3843707914 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62fc0396b0 0x7f62fc03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:45.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.108+0000 7f630d7fa700 1 -- 172.21.15.7:0/3843707914 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f630001e070 con 0x7f63101015f0 2024-04-21T08:35:45.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.108+0000 7f6306ffd700 1 --2- 172.21.15.7:0/3843707914 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62fc0396b0 0x7f62fc03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:45.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.108+0000 7f6315e5a700 1 -- 172.21.15.7:0/3843707914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6310102a80 con 0x7f63101015f0 2024-04-21T08:35:45.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.108+0000 7f6306ffd700 1 --2- 172.21.15.7:0/3843707914 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62fc0396b0 0x7f62fc03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f62f8006fd0 tx=0x7f62f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:45.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.113+0000 7f630d7fa700 1 -- 172.21.15.7:0/3843707914 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6300030070 con 0x7f63101015f0 2024-04-21T08:35:45.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:45.331+0000 7f6315e5a700 1 -- 172.21.15.7:0/3843707914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f63101980e0 con 0x7f63101015f0 2024-04-21T08:35:45.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:45 smithi007 ceph-mon[28477]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:46.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:46.233+0000 7f630d7fa700 1 -- 172.21.15.7:0/3843707914 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f6300013070 con 0x7f63101015f0 2024-04-21T08:35:46.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:46.235+0000 7f6315e5a700 1 -- 172.21.15.7:0/3843707914 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62fc0396b0 msgr2=0x7f62fc03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:46.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:46.235+0000 7f6315e5a700 1 --2- 172.21.15.7:0/3843707914 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62fc0396b0 0x7f62fc03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f62f8006fd0 tx=0x7f62f8006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:46.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:46.235+0000 7f6315e5a700 1 -- 172.21.15.7:0/3843707914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63101015f0 msgr2=0x7f6310196d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:46.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:46.235+0000 7f6315e5a700 1 --2- 172.21.15.7:0/3843707914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63101015f0 0x7f6310196d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f63000056d0 tx=0x7f63000057b0 comp rx=0 tx=0).stop 2024-04-21T08:35:46.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:46.236+0000 7f6315e5a700 1 -- 172.21.15.7:0/3843707914 shutdown_connections 2024-04-21T08:35:46.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:46.236+0000 7f6315e5a700 1 --2- 172.21.15.7:0/3843707914 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62fc0396b0 0x7f62fc03bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:46.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:46.236+0000 7f6315e5a700 1 --2- 172.21.15.7:0/3843707914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63101015f0 0x7f6310196d80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:46.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:46.236+0000 7f6315e5a700 1 -- 172.21.15.7:0/3843707914 >> 172.21.15.7:0/3843707914 conn(0x7f6310076ea0 msgr2=0x7f6310109110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:46.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:46.236+0000 7f6315e5a700 1 -- 172.21.15.7:0/3843707914 shutdown_connections 2024-04-21T08:35:46.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:35:46.236+0000 7f6315e5a700 1 -- 172.21.15.7:0/3843707914 wait complete. 2024-04-21T08:35:46.237 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-04-21T08:35:46.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:46 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3843707914' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T08:35:46.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:46.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:46.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:46.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:46.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:46.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:46.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:46.901 INFO:tasks.cephadm:Adding mon.a on smithi007 2024-04-21T08:35:46.901 INFO:tasks.cephadm:Adding mon.b on smithi016 2024-04-21T08:35:46.901 INFO:tasks.cephadm:Adding mon.c on smithi077 2024-04-21T08:35:46.901 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch apply mon '3;smithi007:172.21.15.7=a;smithi016:172.21.15.16=b;smithi077:172.21.15.77=c' 2024-04-21T08:35:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:35:47.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:47 smithi007 ceph-mon[28477]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:47.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:47 smithi007 ceph-mon[28477]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-21T08:35:47.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:47 smithi007 ceph-mon[28477]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:35:47.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:47 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3843707914' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T08:35:47.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:47 smithi007 ceph-mon[28477]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:35:47.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:47 smithi007 ceph-mon[28477]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:35:47.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:47 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:47.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:47 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:47.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:47 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:48.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.591+0000 7ff5c0d74700 1 -- 172.21.15.77:0/1249984190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5bc1084e0 msgr2=0x7ff5bc1088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:48.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.591+0000 7ff5c0d74700 1 --2- 172.21.15.77:0/1249984190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5bc1084e0 0x7ff5bc1088c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009910 tx=0x7ff5b0009c20 comp rx=0 tx=0).stop 2024-04-21T08:35:48.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.591+0000 7ff5c0d74700 1 -- 172.21.15.77:0/1249984190 shutdown_connections 2024-04-21T08:35:48.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.591+0000 7ff5c0d74700 1 --2- 172.21.15.77:0/1249984190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5bc1084e0 0x7ff5bc1088c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:48.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.591+0000 7ff5c0d74700 1 -- 172.21.15.77:0/1249984190 >> 172.21.15.77:0/1249984190 conn(0x7ff5bc0fdf80 msgr2=0x7ff5bc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:48.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.591+0000 7ff5c0d74700 1 -- 172.21.15.77:0/1249984190 shutdown_connections 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.591+0000 7ff5c0d74700 1 -- 172.21.15.77:0/1249984190 wait complete. 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.591+0000 7ff5c0d74700 1 Processor -- start 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.592+0000 7ff5c0d74700 1 -- start start 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.592+0000 7ff5c0d74700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5bc1084e0 0x7ff5bc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.592+0000 7ff5c0d74700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc10c940 con 0x7ff5bc1084e0 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.592+0000 7ff5ba59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5bc1084e0 0x7ff5bc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.592+0000 7ff5ba59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5bc1084e0 0x7ff5bc074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.77:42756/0 (socket says 172.21.15.77:42756) 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.592+0000 7ff5ba59c700 1 -- 172.21.15.77:0/193307243 learned_addr learned my addr 172.21.15.77:0/193307243 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.592+0000 7ff5ba59c700 1 -- 172.21.15.77:0/193307243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5bc076c30 con 0x7ff5bc1084e0 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.592+0000 7ff5ba59c700 1 --2- 172.21.15.77:0/193307243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5bc1084e0 0x7ff5bc074d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00056d0 tx=0x7ff5b00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.593+0000 7ff5ab7fe700 1 -- 172.21.15.77:0/193307243 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5b0026070 con 0x7ff5bc1084e0 2024-04-21T08:35:48.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.593+0000 7ff5c0d74700 1 -- 172.21.15.77:0/193307243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5bc0752e0 con 0x7ff5bc1084e0 2024-04-21T08:35:48.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.593+0000 7ff5ab7fe700 1 -- 172.21.15.77:0/193307243 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5b000bc00 con 0x7ff5bc1084e0 2024-04-21T08:35:48.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.593+0000 7ff5ab7fe700 1 -- 172.21.15.77:0/193307243 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5b0003920 con 0x7ff5bc1084e0 2024-04-21T08:35:48.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.593+0000 7ff5c0d74700 1 -- 172.21.15.77:0/193307243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5bc0757a0 con 0x7ff5bc1084e0 2024-04-21T08:35:48.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.594+0000 7ff5ab7fe700 1 -- 172.21.15.77:0/193307243 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7ff5b0029070 con 0x7ff5bc1084e0 2024-04-21T08:35:48.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.594+0000 7ff5c0d74700 1 -- 172.21.15.77:0/193307243 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5bc103b90 con 0x7ff5bc1084e0 2024-04-21T08:35:48.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.594+0000 7ff5ab7fe700 1 --2- 172.21.15.77:0/193307243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5a4039650 0x7ff5a403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:48.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.594+0000 7ff5ab7fe700 1 -- 172.21.15.77:0/193307243 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff5b001e070 con 0x7ff5bc1084e0 2024-04-21T08:35:48.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.594+0000 7ff5b9d9b700 1 --2- 172.21.15.77:0/193307243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5a4039650 0x7ff5a403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:48.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.595+0000 7ff5b9d9b700 1 --2- 172.21.15.77:0/193307243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5a4039650 0x7ff5a403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac006fd0 tx=0x7ff5ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:48.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.600+0000 7ff5ab7fe700 1 -- 172.21.15.77:0/193307243 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff5b0013070 con 0x7ff5bc1084e0 2024-04-21T08:35:48.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:48 smithi007 ceph-mon[28477]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:35:48.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.811+0000 7ff5c0d74700 1 -- 172.21.15.77:0/193307243 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi016:172.21.15.16=b;smithi077:172.21.15.77=c", "target": ["mon-mgr", ""]}) -- 0x7ff5bc0762e0 con 0x7ff5a4039650 2024-04-21T08:35:48.817 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.818+0000 7ff5ab7fe700 1 -- 172.21.15.77:0/193307243 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff5bc0762e0 con 0x7ff5a4039650 2024-04-21T08:35:48.818 INFO:teuthology.orchestra.run.smithi077.stdout:Scheduled mon update... 2024-04-21T08:35:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.820+0000 7ff5c0d74700 1 -- 172.21.15.77:0/193307243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5a4039650 msgr2=0x7ff5a403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.820+0000 7ff5c0d74700 1 --2- 172.21.15.77:0/193307243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5a4039650 0x7ff5a403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac006fd0 tx=0x7ff5ac006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.820+0000 7ff5c0d74700 1 -- 172.21.15.77:0/193307243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5bc1084e0 msgr2=0x7ff5bc074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.820+0000 7ff5c0d74700 1 --2- 172.21.15.77:0/193307243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5bc1084e0 0x7ff5bc074d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff5b00056d0 tx=0x7ff5b00057b0 comp rx=0 tx=0).stop 2024-04-21T08:35:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.820+0000 7ff5c0d74700 1 -- 172.21.15.77:0/193307243 shutdown_connections 2024-04-21T08:35:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.820+0000 7ff5c0d74700 1 --2- 172.21.15.77:0/193307243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5a4039650 0x7ff5a403bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.820+0000 7ff5c0d74700 1 --2- 172.21.15.77:0/193307243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5bc1084e0 0x7ff5bc074d40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.820+0000 7ff5c0d74700 1 -- 172.21.15.77:0/193307243 >> 172.21.15.77:0/193307243 conn(0x7ff5bc0fdf80 msgr2=0x7ff5bc10a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.820+0000 7ff5c0d74700 1 -- 172.21.15.77:0/193307243 shutdown_connections 2024-04-21T08:35:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:48.820+0000 7ff5c0d74700 1 -- 172.21.15.77:0/193307243 wait complete. 2024-04-21T08:35:49.438 DEBUG:teuthology.orchestra.run.smithi016:mon.b> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.b.service 2024-04-21T08:35:49.440 DEBUG:teuthology.orchestra.run.smithi077:mon.c> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.c.service 2024-04-21T08:35:49.442 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:35:49.443 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:35:49.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:49 smithi007 ceph-mon[28477]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:49.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:49 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:49.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:49 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:49.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:49 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:49.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:49 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:49.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:49 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:49.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:49 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:35:49.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:49 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:50.085 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:35:50.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:50 smithi007 ceph-mon[28477]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi016:172.21.15.16=b;smithi077:172.21.15.77=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:50.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:50 smithi007 ceph-mon[28477]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi016:172.21.15.16=b;smithi077:172.21.15.77=c;count:3 2024-04-21T08:35:50.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:50 smithi007 ceph-mon[28477]: Deploying daemon mon.c on smithi077 2024-04-21T08:35:51.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:51 smithi007 ceph-mon[28477]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:52.801 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:52 smithi077 systemd[1]: Starting Ceph mon.c for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:35:53.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.315+0000 7f551f73a700 1 -- 172.21.15.77:0/291228468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5518103670 msgr2=0x7f5518103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:53.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.315+0000 7f551f73a700 1 --2- 172.21.15.77:0/291228468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5518103670 0x7f5518103a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f550c009910 tx=0x7f550c009c20 comp rx=0 tx=0).stop 2024-04-21T08:35:53.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.315+0000 7f551f73a700 1 -- 172.21.15.77:0/291228468 shutdown_connections 2024-04-21T08:35:53.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.315+0000 7f551f73a700 1 --2- 172.21.15.77:0/291228468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5518103670 0x7f5518103a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.315+0000 7f551f73a700 1 -- 172.21.15.77:0/291228468 >> 172.21.15.77:0/291228468 conn(0x7f55180fce60 msgr2=0x7f55180ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:53.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.315+0000 7f551f73a700 1 -- 172.21.15.77:0/291228468 shutdown_connections 2024-04-21T08:35:53.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.315+0000 7f551f73a700 1 -- 172.21.15.77:0/291228468 wait complete. 2024-04-21T08:35:53.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.316+0000 7f551f73a700 1 Processor -- start 2024-04-21T08:35:53.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.316+0000 7f551f73a700 1 -- start start 2024-04-21T08:35:53.316 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 podman[31566]: 2024-04-21 08:35:53.097204046 +0000 UTC m=+0.214544743 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.316+0000 7f551f73a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5518103670 0x7f5518197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.316+0000 7f551f73a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f551810c9a0 con 0x7f5518103670 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.316+0000 7f551d4d6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5518103670 0x7f5518197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.316+0000 7f551d4d6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5518103670 0x7f5518197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.77:42772/0 (socket says 172.21.15.77:42772) 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.316+0000 7f551d4d6700 1 -- 172.21.15.77:0/615477091 learned_addr learned my addr 172.21.15.77:0/615477091 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.317+0000 7f551d4d6700 1 -- 172.21.15.77:0/615477091 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5518198450 con 0x7f5518103670 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.317+0000 7f551d4d6700 1 --2- 172.21.15.77:0/615477091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5518103670 0x7f5518197f10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f550c006010 tx=0x7f550c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.317+0000 7f550a7fc700 1 -- 172.21.15.77:0/615477091 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f550c026070 con 0x7f5518103670 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.317+0000 7f551f73a700 1 -- 172.21.15.77:0/615477091 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55181986e0 con 0x7f5518103670 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.317+0000 7f550a7fc700 1 -- 172.21.15.77:0/615477091 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f550c00bc00 con 0x7f5518103670 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.317+0000 7f550a7fc700 1 -- 172.21.15.77:0/615477091 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f550c003910 con 0x7f5518103670 2024-04-21T08:35:53.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.317+0000 7f551f73a700 1 -- 172.21.15.77:0/615477091 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5518198b40 con 0x7f5518103670 2024-04-21T08:35:53.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.318+0000 7f550a7fc700 1 -- 172.21.15.77:0/615477091 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f550c029070 con 0x7f5518103670 2024-04-21T08:35:53.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.318+0000 7f551f73a700 1 -- 172.21.15.77:0/615477091 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f551806b170 con 0x7f5518103670 2024-04-21T08:35:53.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.318+0000 7f550a7fc700 1 --2- 172.21.15.77:0/615477091 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5504039650 0x7f550403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:53.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.318+0000 7f550a7fc700 1 -- 172.21.15.77:0/615477091 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f550c01e070 con 0x7f5518103670 2024-04-21T08:35:53.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.318+0000 7f551ccd5700 1 --2- 172.21.15.77:0/615477091 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5504039650 0x7f550403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:53.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.319+0000 7f551ccd5700 1 --2- 172.21.15.77:0/615477091 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5504039650 0x7f550403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5514006fd0 tx=0x7f5514006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:53.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.324+0000 7f550a7fc700 1 -- 172.21.15.77:0/615477091 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f550c013070 con 0x7f5518103670 2024-04-21T08:35:53.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 podman[31566]: 2024-04-21 08:35:53.423167302 +0000 UTC m=+0.540507958 container create b616f89a2b7fca97f9a0ed63592588b789428ec56f0d5acbe040f4696509bbe2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-c, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-21T08:35:53.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.612+0000 7f551f73a700 1 -- 172.21.15.77:0/615477091 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5518199270 con 0x7f5518103670 2024-04-21T08:35:53.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.612+0000 7f550a7fc700 1 -- 172.21.15.77:0/615477091 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+741 (secure 0 0 0) 0x7f550c030090 con 0x7f5518103670 2024-04-21T08:35:53.612 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:35:53.612 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","modified":"2024-04-21T08:34:11.499844Z","created":"2024-04-21T08:34:11.499844Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-21T08:35:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.614+0000 7f551f73a700 1 -- 172.21.15.77:0/615477091 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5504039650 msgr2=0x7f550403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.614+0000 7f551f73a700 1 --2- 172.21.15.77:0/615477091 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5504039650 0x7f550403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5514006fd0 tx=0x7f5514006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.614+0000 7f551f73a700 1 -- 172.21.15.77:0/615477091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5518103670 msgr2=0x7f5518197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.614+0000 7f551f73a700 1 --2- 172.21.15.77:0/615477091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5518103670 0x7f5518197f10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f550c006010 tx=0x7f550c005780 comp rx=0 tx=0).stop 2024-04-21T08:35:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.614+0000 7f551f73a700 1 -- 172.21.15.77:0/615477091 shutdown_connections 2024-04-21T08:35:53.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.614+0000 7f551f73a700 1 --2- 172.21.15.77:0/615477091 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5504039650 0x7f550403bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.615+0000 7f551f73a700 1 --2- 172.21.15.77:0/615477091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5518103670 0x7f5518197f10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.615+0000 7f551f73a700 1 -- 172.21.15.77:0/615477091 >> 172.21.15.77:0/615477091 conn(0x7f55180fce60 msgr2=0x7f5518104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.615+0000 7f551f73a700 1 -- 172.21.15.77:0/615477091 shutdown_connections 2024-04-21T08:35:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:53.615+0000 7f551f73a700 1 -- 172.21.15.77:0/615477091 wait complete. 2024-04-21T08:35:53.615 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-21T08:35:53.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:53 smithi007 ceph-mon[28477]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:54.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 podman[31566]: 2024-04-21 08:35:53.628014599 +0000 UTC m=+0.745355275 container init b616f89a2b7fca97f9a0ed63592588b789428ec56f0d5acbe040f4696509bbe2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-04-21T08:35:54.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 podman[31566]: 2024-04-21 08:35:53.630997077 +0000 UTC m=+0.748337745 container start b616f89a2b7fca97f9a0ed63592588b789428ec56f0d5acbe040f4696509bbe2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-c, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:35:54.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: set uid:gid to 167:167 (ceph:ceph) 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev), process ceph-mon, pid 2 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: pidfile_write: ignore empty --pid-file 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: load: jerasure load: lrc 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: RocksDB version: 7.9.2 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Git sha 0 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Compile date 2024-04-19 14:19:22 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: DB SUMMARY 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: DB Session ID: VY8PAQ4R11GH6L38QNAR 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: CURRENT file: CURRENT 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: IDENTITY file: IDENTITY 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-21T08:35:54.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.error_if_exists: 0 2024-04-21T08:35:54.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.create_if_missing: 0 2024-04-21T08:35:54.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.paranoid_checks: 1 2024-04-21T08:35:54.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-21T08:35:54.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-21T08:35:54.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-21T08:35:54.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.env: 0x556940832cc0 2024-04-21T08:35:54.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.fs: PosixFileSystem 2024-04-21T08:35:54.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.info_log: 0x5569422ddbe0 2024-04-21T08:35:54.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_file_opening_threads: 16 2024-04-21T08:35:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.statistics: (nil) 2024-04-21T08:35:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.use_fsync: 0 2024-04-21T08:35:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_log_file_size: 0 2024-04-21T08:35:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-21T08:35:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-21T08:35:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.keep_log_file_num: 1000 2024-04-21T08:35:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.recycle_log_file_num: 0 2024-04-21T08:35:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.allow_fallocate: 1 2024-04-21T08:35:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.allow_mmap_reads: 0 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.allow_mmap_writes: 0 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.use_direct_reads: 0 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.create_missing_column_families: 0 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.db_log_dir: 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.wal_dir: 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.advise_random_on_open: 1 2024-04-21T08:35:54.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.db_write_buffer_size: 0 2024-04-21T08:35:54.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.write_buffer_manager: 0x5569422710e0 2024-04-21T08:35:54.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-21T08:35:54.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-21T08:35:54.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-21T08:35:54.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.rate_limiter: (nil) 2024-04-21T08:35:54.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-21T08:35:54.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.wal_recovery_mode: 2 2024-04-21T08:35:54.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.enable_thread_tracking: 0 2024-04-21T08:35:54.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.enable_pipelined_write: 0 2024-04-21T08:35:54.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.unordered_write: 0 2024-04-21T08:35:54.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.row_cache: None 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.wal_filter: None 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.allow_ingest_behind: 0 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.two_write_queues: 0 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.manual_wal_flush: 0 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.wal_compression: 0 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.atomic_flush: 0 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-21T08:35:54.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.log_readahead_size: 0 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.best_efforts_recovery: 0 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.allow_data_in_errors: 0 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.db_host_id: __hostname__ 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_background_jobs: 2 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_background_compactions: -1 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_subcompactions: 1 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-21T08:35:54.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_total_wal_size: 0 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_open_files: -1 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bytes_per_sync: 0 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_readahead_size: 0 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_background_flushes: -1 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Compression algorithms supported: 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: kZSTD supported: 0 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: kXpressCompression supported: 0 2024-04-21T08:35:54.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: kLZ4HCCompression supported: 1 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: kZlibCompression supported: 1 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: kSnappyCompression supported: 1 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: kLZ4Compression supported: 1 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: kBZip2Compression supported: 0 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.merge_operator: 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_filter: None 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_filter_factory: None 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.sst_partitioner_factory: None 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-21T08:35:54.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5569422dddc0) 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: cache_index_and_filter_blocks: 1 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: pin_top_level_index_and_filter: 1 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: index_type: 0 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: data_block_index_type: 0 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: index_shortening: 1 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: checksum: 4 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: no_block_cache: 0 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache: 0x5569422a31f0 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache_name: BinnedLRUCache 2024-04-21T08:35:54.114 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache_options: 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: capacity : 536870912 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: num_shard_bits : 4 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: strict_capacity_limit : 0 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: high_pri_pool_ratio: 0.000 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache_compressed: (nil) 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: persistent_cache: (nil) 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_size: 4096 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_size_deviation: 10 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_restart_interval: 16 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: index_block_restart_interval: 1 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: metadata_block_size: 4096 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: partition_filters: 0 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_delta_encoding: 1 2024-04-21T08:35:54.115 INFO:journalctl@ceph.mon.c.smithi077.stdout: filter_policy: bloomfilter 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout: whole_key_filtering: 1 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout: verify_compression: 0 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout: read_amp_bytes_per_bit: 0 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout: format_version: 5 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout: enable_index_compression: 1 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_align: 0 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout: max_auto_readahead_size: 262144 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout: prepopulate_block_cache: 0 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout: initial_auto_readahead_size: 8192 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout: num_file_reads_for_auto_readahead: 2 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.write_buffer_size: 33554432 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_write_buffer_number: 2 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compression: NoCompression 2024-04-21T08:35:54.116 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bottommost_compression: Disabled 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.prefix_extractor: nullptr 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.num_levels: 7 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-21T08:35:54.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:35:54.118 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:35:54.118 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-21T08:35:54.118 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compression_opts.level: 32767 2024-04-21T08:35:54.118 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compression_opts.strategy: 0 2024-04-21T08:35:54.118 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-21T08:35:54.118 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:35:54.118 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:35:54.118 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-21T08:35:54.118 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compression_opts.enabled: false 2024-04-21T08:35:54.118 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:35:54.118 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.target_file_size_base: 67108864 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-21T08:35:54.119 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.arena_block_size: 1048576 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.disable_auto_compactions: 0 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-21T08:35:54.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.inplace_update_support: 0 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.bloom_locality: 0 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.max_successive_merges: 0 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-21T08:35:54.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.paranoid_file_checks: 0 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.force_consistency_checks: 1 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.report_bg_io_stats: 0 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.ttl: 2592000 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.enable_blob_files: false 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.min_blob_size: 0 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.blob_file_size: 268435456 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-21T08:35:54.122 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.blob_file_starting_level: 0 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 10a5705c-be8e-48cf-856d-a05cb88b9d9c 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688553672930, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688553673333, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713688553, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "10a5705c-be8e-48cf-856d-a05cb88b9d9c", "db_session_id": "VY8PAQ4R11GH6L38QNAR", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688553673411, "job": 1, "event": "recovery_finished"} 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5569423b8000 2024-04-21T08:35:54.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: DB pointer 0x5569423a2000 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** DB Stats ** 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** Compaction Stats [default] ** 2024-04-21T08:35:54.124 INFO:journalctl@ceph.mon.c.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** Compaction Stats [default] ** 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:35:54.125 INFO:journalctl@ceph.mon.c.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout: Block cache BinnedLRUCache@0x5569422a31f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: using public_addr v2:172.21.15.77:0/0 -> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] 2024-04-21T08:35:54.126 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: starting mon.c rank -1 at public addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] at bind addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(???) e0 preinit fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).mds e1 new map 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout: e1 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout: legacy client fscid: -1 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout: No filesystems configured 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-21T08:35:54.127 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Deploying cephadm binary to smithi077 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Added host smithi077 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3843707914' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.129 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3843707914' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.130 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi016:172.21.15.16=b;smithi077:172.21.15.77=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi016:172.21.15.16=b;smithi077:172.21.15.77=c;count:3 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: Deploying daemon mon.c on smithi077 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:54.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:53 smithi077 ceph-mon[31615]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-21T08:35:54.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:54 smithi077 bash[31566]: b616f89a2b7fca97f9a0ed63592588b789428ec56f0d5acbe040f4696509bbe2 2024-04-21T08:35:54.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:54 smithi077 systemd[1]: Started Ceph mon.c for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:35:55.214 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:35:55.215 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:35:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:35:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:35:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: mon.a calling monitor election 2024-04-21T08:35:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: mon.c calling monitor election 2024-04-21T08:35:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: monmap epoch 2 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: last_changed 2024-04-21T08:35:53.697251+0000 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: created 2024-04-21T08:34:11.499844+0000 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: min_mon_release 19 (squid) 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: election_strategy: 1 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: fsmap 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:35:59.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: mgrmap e13: x(active, since 49s) 2024-04-21T08:35:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: overall HEALTH_OK 2024-04-21T08:35:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:35:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:35:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: mon.a calling monitor election 2024-04-21T08:35:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: mon.c calling monitor election 2024-04-21T08:35:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:35:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: monmap epoch 2 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: last_changed 2024-04-21T08:35:53.697251+0000 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: created 2024-04-21T08:34:11.499844+0000 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: min_mon_release 19 (squid) 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: election_strategy: 1 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: fsmap 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:35:59.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: mgrmap e13: x(active, since 49s) 2024-04-21T08:35:59.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: overall HEALTH_OK 2024-04-21T08:35:59.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:59.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:59.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:35:59.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:35:59.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:35:59.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.957+0000 7fac70129700 1 -- 172.21.15.77:0/1979838126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac540035c0 msgr2=0x7fac54005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:59.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.957+0000 7fac70129700 1 --2- 172.21.15.77:0/1979838126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac540035c0 0x7fac54005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fac68102490 tx=0x7fac6400feb0 comp rx=0 tx=0).stop 2024-04-21T08:35:59.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.957+0000 7fac70129700 1 -- 172.21.15.77:0/1979838126 shutdown_connections 2024-04-21T08:35:59.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.957+0000 7fac70129700 1 --2- 172.21.15.77:0/1979838126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac540035c0 0x7fac54005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:59.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.957+0000 7fac70129700 1 -- 172.21.15.77:0/1979838126 >> 172.21.15.77:0/1979838126 conn(0x7fac680fe030 msgr2=0x7fac68100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:59.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.957+0000 7fac70129700 1 -- 172.21.15.77:0/1979838126 shutdown_connections 2024-04-21T08:35:59.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.957+0000 7fac70129700 1 -- 172.21.15.77:0/1979838126 wait complete. 2024-04-21T08:35:59.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.958+0000 7fac70129700 1 Processor -- start 2024-04-21T08:35:59.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.958+0000 7fac70129700 1 -- start start 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.958+0000 7fac70129700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac540035c0 0x7fac68197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.958+0000 7fac70129700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac68198480 0x7fac6819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.958+0000 7fac70129700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fac6810a4b0 con 0x7fac540035c0 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.958+0000 7fac70129700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fac6810a330 con 0x7fac68198480 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6d6c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac68198480 0x7fac6819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6d6c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac68198480 0x7fac6819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40274/0 (socket says 172.21.15.77:40274) 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6d6c4700 1 -- 172.21.15.77:0/3246493367 learned_addr learned my addr 172.21.15.77:0/3246493367 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6dec5700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac540035c0 0x7fac68197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6d6c4700 1 -- 172.21.15.77:0/3246493367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac68198480 msgr2=0x7fac6819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6d6c4700 1 -- 172.21.15.77:0/3246493367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac68198480 msgr2=0x7fac6819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6d6c4700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac68198480 0x7fac6819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6d6c4700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac68198480 0x7fac6819c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:35:59.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6dec5700 1 -- 172.21.15.77:0/3246493367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac68198480 msgr2=0x7fac6819c810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:35:59.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6dec5700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac68198480 0x7fac6819c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:59.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6dec5700 1 -- 172.21.15.77:0/3246493367 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac68198ac0 con 0x7fac540035c0 2024-04-21T08:35:59.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac6dec5700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac540035c0 0x7fac68197f40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fac64000c00 tx=0x7fac640050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:59.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.959+0000 7fac5effd700 1 -- 172.21.15.77:0/3246493367 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fac64004aa0 con 0x7fac540035c0 2024-04-21T08:35:59.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.960+0000 7fac5effd700 1 -- 172.21.15.77:0/3246493367 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac64004030 con 0x7fac540035c0 2024-04-21T08:35:59.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.960+0000 7fac5effd700 1 -- 172.21.15.77:0/3246493367 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fac64011bb0 con 0x7fac540035c0 2024-04-21T08:35:59.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.960+0000 7fac70129700 1 -- 172.21.15.77:0/3246493367 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac6819cd50 con 0x7fac540035c0 2024-04-21T08:35:59.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.960+0000 7fac70129700 1 -- 172.21.15.77:0/3246493367 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac6819d330 con 0x7fac540035c0 2024-04-21T08:35:59.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.961+0000 7fac5effd700 1 -- 172.21.15.77:0/3246493367 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fac64023070 con 0x7fac540035c0 2024-04-21T08:35:59.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.961+0000 7fac70129700 1 -- 172.21.15.77:0/3246493367 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac68103ca0 con 0x7fac540035c0 2024-04-21T08:35:59.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.961+0000 7fac5effd700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac4c039440 0x7fac4c03b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:59.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.961+0000 7fac5effd700 1 -- 172.21.15.77:0/3246493367 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fac64020050 con 0x7fac540035c0 2024-04-21T08:35:59.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.961+0000 7fac6d6c4700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac4c039440 0x7fac4c03b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:59.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.962+0000 7fac6d6c4700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac4c039440 0x7fac4c03b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fac58009180 tx=0x7fac58008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:59.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:35:59.967+0000 7fac5effd700 1 -- 172.21.15.77:0/3246493367 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac64028020 con 0x7fac540035c0 2024-04-21T08:36:00.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:59 smithi077 ceph-mon[31615]: Deploying daemon mon.b on smithi016 2024-04-21T08:36:00.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:35:59 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:00.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:59 smithi007 ceph-mon[28477]: Deploying daemon mon.b on smithi016 2024-04-21T08:36:00.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:35:59 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:00.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.251+0000 7fac70129700 1 -- 172.21.15.77:0/3246493367 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fac6819d5e0 con 0x7fac540035c0 2024-04-21T08:36:00.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.252+0000 7fac5effd700 1 -- 172.21.15.77:0/3246493367 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1007 (secure 0 0 0) 0x7fac6401a390 con 0x7fac540035c0 2024-04-21T08:36:00.251 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:36:00.251 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","modified":"2024-04-21T08:35:53.697251Z","created":"2024-04-21T08:34:11.499844Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T08:36:00.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.254+0000 7fac70129700 1 -- 172.21.15.77:0/3246493367 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac4c039440 msgr2=0x7fac4c03b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:00.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.254+0000 7fac70129700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac4c039440 0x7fac4c03b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fac58009180 tx=0x7fac58008040 comp rx=0 tx=0).stop 2024-04-21T08:36:00.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.254+0000 7fac70129700 1 -- 172.21.15.77:0/3246493367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac540035c0 msgr2=0x7fac68197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:00.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.254+0000 7fac70129700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac540035c0 0x7fac68197f40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fac64000c00 tx=0x7fac640050d0 comp rx=0 tx=0).stop 2024-04-21T08:36:00.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.254+0000 7fac70129700 1 -- 172.21.15.77:0/3246493367 shutdown_connections 2024-04-21T08:36:00.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.254+0000 7fac70129700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac4c039440 0x7fac4c03b900 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:00.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.254+0000 7fac70129700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac540035c0 0x7fac68197f40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:00.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.254+0000 7fac70129700 1 --2- 172.21.15.77:0/3246493367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac68198480 0x7fac6819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:00.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.254+0000 7fac70129700 1 -- 172.21.15.77:0/3246493367 >> 172.21.15.77:0/3246493367 conn(0x7fac680fe030 msgr2=0x7fac680ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:00.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.254+0000 7fac70129700 1 -- 172.21.15.77:0/3246493367 shutdown_connections 2024-04-21T08:36:00.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:00.254+0000 7fac70129700 1 -- 172.21.15.77:0/3246493367 wait complete. 2024-04-21T08:36:00.255 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 2 2024-04-21T08:36:01.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:00 smithi077 ceph-mon[31615]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:01.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:00 smithi077 ceph-mon[31615]: from='client.? 172.21.15.77:0/3246493367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:01.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:00 smithi007 ceph-mon[28477]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:01.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:00 smithi007 ceph-mon[28477]: from='client.? 172.21.15.77:0/3246493367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:01.777 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:36:01.777 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:36:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:36:02.508 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 podman[31240]: 2024-04-21 08:36:02.395139685 +0000 UTC m=+0.018361148 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:36:02.508 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 podman[31240]: 2024-04-21 08:36:02.509363049 +0000 UTC m=+0.132584480 container create b174a6eca6abb23896602b5db296c97e45ae9f31083876c0391fbccb47ebd97d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-21T08:36:02.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 podman[31240]: 2024-04-21 08:36:02.678767795 +0000 UTC m=+0.301989239 container init b174a6eca6abb23896602b5db296c97e45ae9f31083876c0391fbccb47ebd97d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-b, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-21T08:36:02.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 podman[31240]: 2024-04-21 08:36:02.681897324 +0000 UTC m=+0.305118757 container start b174a6eca6abb23896602b5db296c97e45ae9f31083876c0391fbccb47ebd97d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-b, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-21T08:36:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: set uid:gid to 167:167 (ceph:ceph) 2024-04-21T08:36:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev), process ceph-mon, pid 2 2024-04-21T08:36:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pidfile_write: ignore empty --pid-file 2024-04-21T08:36:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: load: jerasure load: lrc 2024-04-21T08:36:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: RocksDB version: 7.9.2 2024-04-21T08:36:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Git sha 0 2024-04-21T08:36:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Compile date 2024-04-19 14:19:22 2024-04-21T08:36:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: DB SUMMARY 2024-04-21T08:36:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: DB Session ID: VCF2ALBUFTUATHD3JMYF 2024-04-21T08:36:02.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: CURRENT file: CURRENT 2024-04-21T08:36:02.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: IDENTITY file: IDENTITY 2024-04-21T08:36:02.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-21T08:36:02.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-21T08:36:02.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-21T08:36:02.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.error_if_exists: 0 2024-04-21T08:36:02.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.create_if_missing: 0 2024-04-21T08:36:02.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.paranoid_checks: 1 2024-04-21T08:36:02.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-21T08:36:02.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-21T08:36:02.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-21T08:36:02.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.env: 0x557387cb2cc0 2024-04-21T08:36:02.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.fs: PosixFileSystem 2024-04-21T08:36:02.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.info_log: 0x5573897d5be0 2024-04-21T08:36:02.807 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_file_opening_threads: 16 2024-04-21T08:36:02.809 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.statistics: (nil) 2024-04-21T08:36:02.809 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.use_fsync: 0 2024-04-21T08:36:02.809 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_log_file_size: 0 2024-04-21T08:36:02.809 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-21T08:36:02.809 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-21T08:36:02.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.keep_log_file_num: 1000 2024-04-21T08:36:02.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.recycle_log_file_num: 0 2024-04-21T08:36:02.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.allow_fallocate: 1 2024-04-21T08:36:02.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.allow_mmap_reads: 0 2024-04-21T08:36:02.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.allow_mmap_writes: 0 2024-04-21T08:36:02.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.use_direct_reads: 0 2024-04-21T08:36:02.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-21T08:36:02.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.create_missing_column_families: 0 2024-04-21T08:36:02.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.db_log_dir: 2024-04-21T08:36:02.810 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.wal_dir: 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.advise_random_on_open: 1 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.db_write_buffer_size: 0 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.write_buffer_manager: 0x5573897690e0 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-21T08:36:02.811 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.rate_limiter: (nil) 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.wal_recovery_mode: 2 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.enable_thread_tracking: 0 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.enable_pipelined_write: 0 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.unordered_write: 0 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.row_cache: None 2024-04-21T08:36:02.812 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.wal_filter: None 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.allow_ingest_behind: 0 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.two_write_queues: 0 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.manual_wal_flush: 0 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.wal_compression: 0 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.atomic_flush: 0 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.log_readahead_size: 0 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-21T08:36:02.813 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.best_efforts_recovery: 0 2024-04-21T08:36:02.814 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-21T08:36:02.814 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-21T08:36:02.814 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.allow_data_in_errors: 0 2024-04-21T08:36:02.814 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.db_host_id: __hostname__ 2024-04-21T08:36:02.814 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-21T08:36:02.814 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_background_jobs: 2 2024-04-21T08:36:02.814 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_background_compactions: -1 2024-04-21T08:36:02.814 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_subcompactions: 1 2024-04-21T08:36:02.814 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-21T08:36:02.814 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_total_wal_size: 0 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_open_files: -1 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bytes_per_sync: 0 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_readahead_size: 0 2024-04-21T08:36:02.815 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_background_flushes: -1 2024-04-21T08:36:02.816 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Compression algorithms supported: 2024-04-21T08:36:02.816 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-21T08:36:02.816 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: kZSTD supported: 0 2024-04-21T08:36:02.816 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: kXpressCompression supported: 0 2024-04-21T08:36:02.816 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: kLZ4HCCompression supported: 1 2024-04-21T08:36:02.816 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: kZlibCompression supported: 1 2024-04-21T08:36:02.816 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: kSnappyCompression supported: 1 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: kLZ4Compression supported: 1 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: kBZip2Compression supported: 0 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.merge_operator: 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_filter: None 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_filter_factory: None 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.sst_partitioner_factory: None 2024-04-21T08:36:02.817 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-21T08:36:02.818 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-21T08:36:02.818 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5573897d5dc0) 2024-04-21T08:36:02.818 INFO:journalctl@ceph.mon.b.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-04-21T08:36:02.818 INFO:journalctl@ceph.mon.b.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-21T08:36:02.818 INFO:journalctl@ceph.mon.b.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-21T08:36:02.818 INFO:journalctl@ceph.mon.b.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-04-21T08:36:02.818 INFO:journalctl@ceph.mon.b.smithi016.stdout: index_type: 0 2024-04-21T08:36:02.818 INFO:journalctl@ceph.mon.b.smithi016.stdout: data_block_index_type: 0 2024-04-21T08:36:02.818 INFO:journalctl@ceph.mon.b.smithi016.stdout: index_shortening: 1 2024-04-21T08:36:02.818 INFO:journalctl@ceph.mon.b.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-21T08:36:02.818 INFO:journalctl@ceph.mon.b.smithi016.stdout: checksum: 4 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: no_block_cache: 0 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_cache: 0x55738979b1f0 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_cache_options: 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: capacity : 536870912 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: num_shard_bits : 4 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: strict_capacity_limit : 0 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_cache_compressed: (nil) 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: persistent_cache: (nil) 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_size: 4096 2024-04-21T08:36:02.819 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_size_deviation: 10 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_restart_interval: 16 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: index_block_restart_interval: 1 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: metadata_block_size: 4096 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: partition_filters: 0 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: use_delta_encoding: 1 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: filter_policy: bloomfilter 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: whole_key_filtering: 1 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: verify_compression: 0 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: format_version: 5 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: enable_index_compression: 1 2024-04-21T08:36:02.820 INFO:journalctl@ceph.mon.b.smithi016.stdout: block_align: 0 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout: max_auto_readahead_size: 262144 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout: prepopulate_block_cache: 0 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout: initial_auto_readahead_size: 8192 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.write_buffer_size: 33554432 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_write_buffer_number: 2 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compression: NoCompression 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bottommost_compression: Disabled 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.prefix_extractor: nullptr 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.num_levels: 7 2024-04-21T08:36:02.821 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-21T08:36:02.822 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-21T08:36:02.822 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-21T08:36:02.822 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-21T08:36:02.822 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-21T08:36:02.822 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-21T08:36:02.822 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-21T08:36:02.822 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:36:02.822 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-21T08:36:02.822 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-21T08:36:02.822 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:36:02.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:36:02.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-21T08:36:02.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compression_opts.level: 32767 2024-04-21T08:36:02.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compression_opts.strategy: 0 2024-04-21T08:36:02.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-21T08:36:02.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:36:02.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:36:02.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-21T08:36:02.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compression_opts.enabled: false 2024-04-21T08:36:02.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:36:02.823 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.target_file_size_base: 67108864 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-21T08:36:02.824 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.arena_block_size: 1048576 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.disable_auto_compactions: 0 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-21T08:36:02.825 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-21T08:36:02.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-21T08:36:02.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-21T08:36:02.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-21T08:36:02.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-21T08:36:02.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-21T08:36:02.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-21T08:36:02.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-21T08:36:02.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-21T08:36:02.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.inplace_update_support: 0 2024-04-21T08:36:02.826 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-21T08:36:02.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-21T08:36:02.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-21T08:36:02.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-21T08:36:02.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.bloom_locality: 0 2024-04-21T08:36:02.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.max_successive_merges: 0 2024-04-21T08:36:02.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-21T08:36:02.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.paranoid_file_checks: 0 2024-04-21T08:36:02.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.force_consistency_checks: 1 2024-04-21T08:36:02.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.report_bg_io_stats: 0 2024-04-21T08:36:02.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.ttl: 2592000 2024-04-21T08:36:02.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-21T08:36:02.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-21T08:36:02.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-21T08:36:02.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.enable_blob_files: false 2024-04-21T08:36:02.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.min_blob_size: 0 2024-04-21T08:36:02.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.blob_file_size: 268435456 2024-04-21T08:36:02.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-21T08:36:02.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-21T08:36:02.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.blob_file_starting_level: 0 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6f94dee6-4fb5-4f4e-b366-a2602c1ae529 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688562712031, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688562712449, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713688562, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6f94dee6-4fb5-4f4e-b366-a2602c1ae529", "db_session_id": "VCF2ALBUFTUATHD3JMYF", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688562712532, "job": 1, "event": "recovery_finished"} 2024-04-21T08:36:02.829 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-21T08:36:02.830 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-21T08:36:02.830 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5573898b0000 2024-04-21T08:36:02.830 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: DB pointer 0x55738989a000 2024-04-21T08:36:02.830 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T08:36:02.830 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T08:36:02.830 INFO:journalctl@ceph.mon.b.smithi016.stdout: ** DB Stats ** 2024-04-21T08:36:02.830 INFO:journalctl@ceph.mon.b.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:36:02.830 INFO:journalctl@ceph.mon.b.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T08:36:02.830 INFO:journalctl@ceph.mon.b.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:36:02.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:36:02.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T08:36:02.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:36:02.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:36:02.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2024-04-21T08:36:02.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-21T08:36:02.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:36:02.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T08:36:02.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:36:02.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:36:02.831 INFO:journalctl@ceph.mon.b.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:36:02.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2024-04-21T08:36:02.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-21T08:36:02.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:36:02.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T08:36:02.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:36:02.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2024-04-21T08:36:02.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T08:36:02.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2024-04-21T08:36:02.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:36:02.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T08:36:02.832 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: Block cache BinnedLRUCache@0x55738979b1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-21T08:36:02.833 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: using public_addr v2:172.21.15.16:0/0 -> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] 2024-04-21T08:36:02.834 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: starting mon.b rank -1 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:02.834 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(???) e0 preinit fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:02.834 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).mds e1 new map 2024-04-21T08:36:02.834 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-21T08:36:02.834 INFO:journalctl@ceph.mon.b.smithi016.stdout: e1 2024-04-21T08:36:02.834 INFO:journalctl@ceph.mon.b.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:36:02.834 INFO:journalctl@ceph.mon.b.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:36:02.834 INFO:journalctl@ceph.mon.b.smithi016.stdout: legacy client fscid: -1 2024-04-21T08:36:02.834 INFO:journalctl@ceph.mon.b.smithi016.stdout: 2024-04-21T08:36:02.834 INFO:journalctl@ceph.mon.b.smithi016.stdout: No filesystems configured 2024-04-21T08:36:02.834 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Deploying cephadm binary to smithi077 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-21T08:36:02.835 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Added host smithi077 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.836 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.837 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.837 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.837 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:36:02.837 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.837 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3843707914' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T08:36:02.837 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.837 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.837 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.837 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:02.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:02.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:02.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-21T08:36:02.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:02.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3843707914' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T08:36:02.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:02.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:02.838 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi016:172.21.15.16=b;smithi077:172.21.15.77=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:02.839 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi016:172.21.15.16=b;smithi077:172.21.15.77=c;count:3 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Deploying daemon mon.c on smithi077 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.a calling monitor election 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.c calling monitor election 2024-04-21T08:36:02.840 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: monmap epoch 2 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: last_changed 2024-04-21T08:35:53.697251+0000 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: created 2024-04-21T08:34:11.499844+0000 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: min_mon_release 19 (squid) 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: election_strategy: 1 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-04-21T08:36:02.841 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: fsmap 2024-04-21T08:36:02.842 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:02.842 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mgrmap e13: x(active, since 49s) 2024-04-21T08:36:02.842 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: overall HEALTH_OK 2024-04-21T08:36:02.842 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.842 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.842 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:02.842 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:02.842 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:02.842 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: Deploying daemon mon.b on smithi016 2024-04-21T08:36:02.843 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:02.843 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:02.843 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: from='client.? 172.21.15.77:0/3246493367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:02.843 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 ceph-mon[31259]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-21T08:36:02.843 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 bash[31240]: b174a6eca6abb23896602b5db296c97e45ae9f31083876c0391fbccb47ebd97d 2024-04-21T08:36:02.843 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:02 smithi016 systemd[1]: Started Ceph mon.b for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:36:07.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.950+0000 7fa7635da700 1 -- 172.21.15.77:0/2692292954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7480035c0 msgr2=0x7fa748005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:07.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.950+0000 7fa7635da700 1 --2- 172.21.15.77:0/2692292954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7480035c0 0x7fa748005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa75c06bea0 tx=0x7fa754009e50 comp rx=0 tx=0).stop 2024-04-21T08:36:07.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.950+0000 7fa7635da700 1 -- 172.21.15.77:0/2692292954 shutdown_connections 2024-04-21T08:36:07.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.950+0000 7fa7635da700 1 --2- 172.21.15.77:0/2692292954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7480035c0 0x7fa748005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:07.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.950+0000 7fa7635da700 1 -- 172.21.15.77:0/2692292954 >> 172.21.15.77:0/2692292954 conn(0x7fa75c077af0 msgr2=0x7fa75c077f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:07.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.950+0000 7fa7635da700 1 -- 172.21.15.77:0/2692292954 shutdown_connections 2024-04-21T08:36:07.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.950+0000 7fa7635da700 1 -- 172.21.15.77:0/2692292954 wait complete. 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa7635da700 1 Processor -- start 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa7635da700 1 -- start start 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa7635da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7480035c0 0x7fa75c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa7635da700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1985c0 0x7fa75c19c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa7635da700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa75c19ce90 0x7fa75c19d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa7635da700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1093e0 con 0x7fa7480035c0 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa7635da700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c109260 con 0x7fa75c19ce90 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa7635da700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c109560 con 0x7fa75c1985c0 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa760b75700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1985c0 0x7fa75c19c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa760b75700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1985c0 0x7fa75c19c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44592/0 (socket says 172.21.15.77:44592) 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa760b75700 1 -- 172.21.15.77:0/1468388780 learned_addr learned my addr 172.21.15.77:0/1468388780 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa761376700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7480035c0 0x7fa75c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:07.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa760b75700 1 -- 172.21.15.77:0/1468388780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1985c0 msgr2=0x7fa75c19c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-21T08:36:07.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa760b75700 1 -- 172.21.15.77:0/1468388780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1985c0 msgr2=0x7fa75c19c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T08:36:07.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa760b75700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1985c0 0x7fa75c19c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T08:36:07.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa760b75700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1985c0 0x7fa75c19c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:36:07.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa761376700 1 -- 172.21.15.77:0/1468388780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1985c0 msgr2=0x7fa75c19c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:36:07.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa761376700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1985c0 0x7fa75c19c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:07.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa761376700 1 -- 172.21.15.77:0/1468388780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa75c19ce90 msgr2=0x7fa75c19d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:07.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.951+0000 7fa761376700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa75c19ce90 0x7fa75c19d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:07.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.952+0000 7fa761376700 1 -- 172.21.15.77:0/1468388780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75c1a4c00 con 0x7fa7480035c0 2024-04-21T08:36:07.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.952+0000 7fa761376700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7480035c0 0x7fa75c198080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa754009fd0 tx=0x7fa75401bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.952+0000 7fa7527fc700 1 -- 172.21.15.77:0/1468388780 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754004de0 con 0x7fa7480035c0 2024-04-21T08:36:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.952+0000 7fa7635da700 1 -- 172.21.15.77:0/1468388780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75c1a4dd0 con 0x7fa7480035c0 2024-04-21T08:36:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.952+0000 7fa7527fc700 1 -- 172.21.15.77:0/1468388780 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa75401a850 con 0x7fa7480035c0 2024-04-21T08:36:07.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.952+0000 7fa7527fc700 1 -- 172.21.15.77:0/1468388780 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7540050f0 con 0x7fa7480035c0 2024-04-21T08:36:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.952+0000 7fa7635da700 1 -- 172.21.15.77:0/1468388780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa75c1a5280 con 0x7fa7480035c0 2024-04-21T08:36:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.953+0000 7fa7527fc700 1 -- 172.21.15.77:0/1468388780 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fa754015040 con 0x7fa7480035c0 2024-04-21T08:36:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.953+0000 7fa7635da700 1 -- 172.21.15.77:0/1468388780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa75c0640a0 con 0x7fa7480035c0 2024-04-21T08:36:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.953+0000 7fa7527fc700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa740039ae0 0x7fa74003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.953+0000 7fa7527fc700 1 -- 172.21.15.77:0/1468388780 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa754003cd0 con 0x7fa7480035c0 2024-04-21T08:36:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.954+0000 7fa760b75700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa740039ae0 0x7fa74003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:07.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.954+0000 7fa760b75700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa740039ae0 0x7fa74003bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009180 tx=0x7fa74c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:07.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:07.960+0000 7fa7527fc700 1 -- 172.21.15.77:0/1468388780 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa754027070 con 0x7fa7480035c0 2024-04-21T08:36:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:36:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: mon.c calling monitor election 2024-04-21T08:36:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: mon.a calling monitor election 2024-04-21T08:36:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: monmap epoch 3 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: last_changed 2024-04-21T08:36:02.735656+0000 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: created 2024-04-21T08:34:11.499844+0000 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: min_mon_release 19 (squid) 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: election_strategy: 1 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-21T08:36:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-04-21T08:36:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: 2: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.b 2024-04-21T08:36:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: fsmap 2024-04-21T08:36:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: mgrmap e13: x(active, since 58s) 2024-04-21T08:36:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: overall HEALTH_OK 2024-04-21T08:36:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:08.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: mon.c calling monitor election 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: mon.a calling monitor election 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:36:08.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: monmap epoch 3 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: last_changed 2024-04-21T08:36:02.735656+0000 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: created 2024-04-21T08:34:11.499844+0000 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: min_mon_release 19 (squid) 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: election_strategy: 1 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: 2: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.b 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: fsmap 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: mgrmap e13: x(active, since 58s) 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: overall HEALTH_OK 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:08.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:08.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.252+0000 7fa7635da700 1 -- 172.21.15.77:0/1468388780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fa75c1a5700 con 0x7fa7480035c0 2024-04-21T08:36:08.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.253+0000 7fa7527fc700 1 -- 172.21.15.77:0/1468388780 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1271 (secure 0 0 0) 0x7fa754015330 con 0x7fa7480035c0 2024-04-21T08:36:08.254 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-21T08:36:08.254 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":3,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","modified":"2024-04-21T08:36:02.735656Z","created":"2024-04-21T08:34:11.499844Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T08:36:08.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.255+0000 7fa7635da700 1 -- 172.21.15.77:0/1468388780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa740039ae0 msgr2=0x7fa74003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:08.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.255+0000 7fa7635da700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa740039ae0 0x7fa74003bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009180 tx=0x7fa74c008040 comp rx=0 tx=0).stop 2024-04-21T08:36:08.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.255+0000 7fa7635da700 1 -- 172.21.15.77:0/1468388780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7480035c0 msgr2=0x7fa75c198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:08.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.255+0000 7fa7635da700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7480035c0 0x7fa75c198080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa754009fd0 tx=0x7fa75401bdd0 comp rx=0 tx=0).stop 2024-04-21T08:36:08.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.255+0000 7fa7635da700 1 -- 172.21.15.77:0/1468388780 shutdown_connections 2024-04-21T08:36:08.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.255+0000 7fa7635da700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa740039ae0 0x7fa74003bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:08.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.255+0000 7fa7635da700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa75c19ce90 0x7fa75c19d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:08.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.255+0000 7fa7635da700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7480035c0 0x7fa75c198080 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:08.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.255+0000 7fa7635da700 1 --2- 172.21.15.77:0/1468388780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa75c1985c0 0x7fa75c19c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:08.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.255+0000 7fa7635da700 1 -- 172.21.15.77:0/1468388780 >> 172.21.15.77:0/1468388780 conn(0x7fa75c077af0 msgr2=0x7fa75c105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:08.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.255+0000 7fa7635da700 1 -- 172.21.15.77:0/1468388780 shutdown_connections 2024-04-21T08:36:08.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:08.256+0000 7fa7635da700 1 -- 172.21.15.77:0/1468388780 wait complete. 2024-04-21T08:36:08.257 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 3 2024-04-21T08:36:08.809 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-21T08:36:08.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-21T08:36:09.070 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:09.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:08 smithi077 ceph-mon[31615]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:09.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:08 smithi077 ceph-mon[31615]: from='client.? 172.21.15.77:0/1468388780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:09.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:08 smithi007 ceph-mon[28477]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:09.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:09.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:08 smithi007 ceph-mon[28477]: from='client.? 172.21.15.77:0/1468388780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:09.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:10.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:36:10.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: mon.c calling monitor election 2024-04-21T08:36:10.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: mon.a calling monitor election 2024-04-21T08:36:10.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:10.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:10.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:10.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: monmap epoch 3 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: last_changed 2024-04-21T08:36:02.735656+0000 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: created 2024-04-21T08:34:11.499844+0000 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: min_mon_release 19 (squid) 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: election_strategy: 1 2024-04-21T08:36:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: 2: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.b 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: fsmap 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: mgrmap e13: x(active, since 58s) 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: overall HEALTH_OK 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:10.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:10.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='client.? 172.21.15.77:0/1468388780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:10.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:10.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.716+0000 7f2c69c85700 1 -- 172.21.15.7:0/2490108244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c641084e0 msgr2=0x7f2c641088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:10.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.716+0000 7f2c69c85700 1 --2- 172.21.15.7:0/2490108244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c641084e0 0x7f2c641088c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2c54009910 tx=0x7f2c54009c20 comp rx=0 tx=0).stop 2024-04-21T08:36:10.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.716+0000 7f2c69c85700 1 -- 172.21.15.7:0/2490108244 shutdown_connections 2024-04-21T08:36:10.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.716+0000 7f2c69c85700 1 --2- 172.21.15.7:0/2490108244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c641084e0 0x7f2c641088c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:10.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.716+0000 7f2c69c85700 1 -- 172.21.15.7:0/2490108244 >> 172.21.15.7:0/2490108244 conn(0x7f2c640fdf60 msgr2=0x7f2c64100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:10.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.716+0000 7f2c69c85700 1 -- 172.21.15.7:0/2490108244 shutdown_connections 2024-04-21T08:36:10.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.716+0000 7f2c69c85700 1 -- 172.21.15.7:0/2490108244 wait complete. 2024-04-21T08:36:10.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.717+0000 7f2c69c85700 1 Processor -- start 2024-04-21T08:36:10.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.717+0000 7f2c69c85700 1 -- start start 2024-04-21T08:36:10.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.717+0000 7f2c69c85700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c641084e0 0x7f2c641980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:10.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.717+0000 7f2c69c85700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c64198620 0x7f2c6419c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:10.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.717+0000 7f2c69c85700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6419cef0 0x7f2c6419d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:10.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.717+0000 7f2c69c85700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6410c940 con 0x7f2c64198620 2024-04-21T08:36:10.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.717+0000 7f2c69c85700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6410c7c0 con 0x7f2c6419cef0 2024-04-21T08:36:10.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.717+0000 7f2c69c85700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6410cac0 con 0x7f2c641084e0 2024-04-21T08:36:10.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c62ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c64198620 0x7f2c6419c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:10.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c62ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c64198620 0x7f2c6419c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40582/0 (socket says 172.21.15.7:40582) 2024-04-21T08:36:10.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c62ffd700 1 -- 172.21.15.7:0/922674744 learned_addr learned my addr 172.21.15.7:0/922674744 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:36:10.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c63fff700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6419cef0 0x7f2c6419d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:10.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c637fe700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c641084e0 0x7f2c641980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:10.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c62ffd700 1 -- 172.21.15.7:0/922674744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c641084e0 msgr2=0x7f2c641980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:10.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c62ffd700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c641084e0 0x7f2c641980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:10.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c62ffd700 1 -- 172.21.15.7:0/922674744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6419cef0 msgr2=0x7f2c6419d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:10.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c62ffd700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6419cef0 0x7f2c6419d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c62ffd700 1 -- 172.21.15.7:0/922674744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c641a4c60 con 0x7f2c64198620 2024-04-21T08:36:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c637fe700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c641084e0 0x7f2c641980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:36:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c62ffd700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c64198620 0x7f2c6419c9b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009e20 tx=0x7f2c5800ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c60ff9700 1 -- 172.21.15.7:0/922674744 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58004230 con 0x7f2c64198620 2024-04-21T08:36:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c60ff9700 1 -- 172.21.15.7:0/922674744 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c58010d40 con 0x7f2c64198620 2024-04-21T08:36:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c69c85700 1 -- 172.21.15.7:0/922674744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c64074cd0 con 0x7f2c64198620 2024-04-21T08:36:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.718+0000 7f2c60ff9700 1 -- 172.21.15.7:0/922674744 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58003e10 con 0x7f2c64198620 2024-04-21T08:36:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.719+0000 7f2c69c85700 1 -- 172.21.15.7:0/922674744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c64075260 con 0x7f2c64198620 2024-04-21T08:36:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.719+0000 7f2c69c85700 1 -- 172.21.15.7:0/922674744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c640640a0 con 0x7f2c64198620 2024-04-21T08:36:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.720+0000 7f2c60ff9700 1 -- 172.21.15.7:0/922674744 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f2c58018430 con 0x7f2c64198620 2024-04-21T08:36:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.720+0000 7f2c60ff9700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2c4c039ae0 0x7f2c4c03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.720+0000 7f2c60ff9700 1 -- 172.21.15.7:0/922674744 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2c5804fcc0 con 0x7f2c64198620 2024-04-21T08:36:10.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.720+0000 7f2c637fe700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2c4c039ae0 0x7f2c4c03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:10.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.720+0000 7f2c637fe700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2c4c039ae0 0x7f2c4c03bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2c54009750 tx=0x7f2c540058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:10.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.725+0000 7f2c60ff9700 1 -- 172.21.15.7:0/922674744 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c580152a0 con 0x7f2c64198620 2024-04-21T08:36:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: mon.b calling monitor election 2024-04-21T08:36:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-21T08:36:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-21T08:36:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-21T08:36:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: mon.b calling monitor election 2024-04-21T08:36:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: mon.c calling monitor election 2024-04-21T08:36:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: mon.a calling monitor election 2024-04-21T08:36:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:36:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: monmap epoch 3 2024-04-21T08:36:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: last_changed 2024-04-21T08:36:02.735656+0000 2024-04-21T08:36:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: created 2024-04-21T08:34:11.499844+0000 2024-04-21T08:36:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: min_mon_release 19 (squid) 2024-04-21T08:36:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: election_strategy: 1 2024-04-21T08:36:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-21T08:36:10.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-04-21T08:36:10.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: 2: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.b 2024-04-21T08:36:10.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: fsmap 2024-04-21T08:36:10.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:10.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: mgrmap e13: x(active, since 60s) 2024-04-21T08:36:10.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: overall HEALTH_OK 2024-04-21T08:36:10.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:10.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:10.904 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: Updating smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:10.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:36:10.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:10.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:10.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:10.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:10 smithi007 ceph-mon[28477]: Reconfiguring daemon mon.a on smithi007 2024-04-21T08:36:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.937+0000 7f2c69c85700 1 -- 172.21.15.7:0/922674744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f2c64075550 con 0x7f2c64198620 2024-04-21T08:36:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.937+0000 7f2c60ff9700 1 -- 172.21.15.7:0/922674744 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+269 (secure 0 0 0) 0x7f2c5801f710 con 0x7f2c64198620 2024-04-21T08:36:10.937 INFO:teuthology.orchestra.run.smithi007.stdout:# minimal ceph.conf for c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:10.937 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-04-21T08:36:10.937 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:10.937 INFO:teuthology.orchestra.run.smithi007.stdout: mon_host = [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] 2024-04-21T08:36:10.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 -- 172.21.15.7:0/922674744 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2c4c039ae0 msgr2=0x7f2c4c03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2c4c039ae0 0x7f2c4c03bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2c54009750 tx=0x7f2c540058e0 comp rx=0 tx=0).stop 2024-04-21T08:36:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 -- 172.21.15.7:0/922674744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c64198620 msgr2=0x7f2c6419c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c64198620 0x7f2c6419c9b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009e20 tx=0x7f2c5800ba60 comp rx=0 tx=0).stop 2024-04-21T08:36:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 -- 172.21.15.7:0/922674744 shutdown_connections 2024-04-21T08:36:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2c4c039ae0 0x7f2c4c03bfa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c641084e0 0x7f2c641980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c6419cef0 0x7f2c6419d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 --2- 172.21.15.7:0/922674744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c64198620 0x7f2c6419c9b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 -- 172.21.15.7:0/922674744 >> 172.21.15.7:0/922674744 conn(0x7f2c640fdf60 msgr2=0x7f2c64103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 -- 172.21.15.7:0/922674744 shutdown_connections 2024-04-21T08:36:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:10.939+0000 7f2c69c85700 1 -- 172.21.15.7:0/922674744 wait complete. 2024-04-21T08:36:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: mon.b calling monitor election 2024-04-21T08:36:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-21T08:36:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-21T08:36:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-21T08:36:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: mon.b calling monitor election 2024-04-21T08:36:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: mon.c calling monitor election 2024-04-21T08:36:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: mon.a calling monitor election 2024-04-21T08:36:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:36:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: monmap epoch 3 2024-04-21T08:36:11.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:11.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: last_changed 2024-04-21T08:36:02.735656+0000 2024-04-21T08:36:11.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: created 2024-04-21T08:34:11.499844+0000 2024-04-21T08:36:11.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: min_mon_release 19 (squid) 2024-04-21T08:36:11.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: election_strategy: 1 2024-04-21T08:36:11.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: 2: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.b 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: fsmap 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: mgrmap e13: x(active, since 60s) 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: overall HEALTH_OK 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: Updating smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:36:11.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:11.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:11.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:11.059 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:10 smithi016 ceph-mon[31259]: Reconfiguring daemon mon.a on smithi007 2024-04-21T08:36:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: mon.b calling monitor election 2024-04-21T08:36:11.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:11.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-21T08:36:11.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-21T08:36:11.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-21T08:36:11.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: mon.b calling monitor election 2024-04-21T08:36:11.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: mon.c calling monitor election 2024-04-21T08:36:11.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: mon.a calling monitor election 2024-04-21T08:36:11.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:36:11.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: monmap epoch 3 2024-04-21T08:36:11.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:11.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: last_changed 2024-04-21T08:36:02.735656+0000 2024-04-21T08:36:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: created 2024-04-21T08:34:11.499844+0000 2024-04-21T08:36:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: min_mon_release 19 (squid) 2024-04-21T08:36:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: election_strategy: 1 2024-04-21T08:36:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-21T08:36:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-04-21T08:36:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: 2: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.b 2024-04-21T08:36:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: fsmap 2024-04-21T08:36:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: mgrmap e13: x(active, since 60s) 2024-04-21T08:36:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: overall HEALTH_OK 2024-04-21T08:36:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: Updating smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:11.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:36:11.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:11.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:11.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:11.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:10 smithi077 ceph-mon[31615]: Reconfiguring daemon mon.a on smithi007 2024-04-21T08:36:11.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:11 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:11.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:11 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/922674744' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:11.967 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-21T08:36:11.967 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:36:11.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:36:11.994 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:36:11.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:12.053 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:11 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:11 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/922674744' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:12.059 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-21T08:36:12.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:36:12.092 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-21T08:36:12.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:12.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:11 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:12.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:11 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/922674744' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:12.161 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-21T08:36:12.161 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:36:12.192 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-21T08:36:12.192 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:12.259 INFO:tasks.cephadm:Adding mgr.x on smithi007 2024-04-21T08:36:12.260 INFO:tasks.cephadm:Adding mgr.y on smithi016 2024-04-21T08:36:12.260 INFO:tasks.cephadm:Adding mgr.z on smithi077 2024-04-21T08:36:12.260 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi007=x;smithi016=y;smithi077=z' 2024-04-21T08:36:12.533 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:36:13.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:13 smithi016 ceph-mon[31259]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:13.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:13 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:13.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:13 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:13.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:13 smithi016 ceph-mon[31259]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:36:13.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:13 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:13.556 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:13 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:13.556 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:13 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:13.556 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:13 smithi016 ceph-mon[31259]: Reconfiguring daemon mon.b on smithi016 2024-04-21T08:36:13.640 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:13 smithi077 ceph-mon[31615]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:13.640 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:13 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:13.640 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:13 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:13.641 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:13 smithi077 ceph-mon[31615]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:36:13.641 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:13 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:13.641 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:13 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:13.641 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:13 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:13.641 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:13 smithi077 ceph-mon[31615]: Reconfiguring daemon mon.b on smithi016 2024-04-21T08:36:13.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:13 smithi007 ceph-mon[28477]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:13.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:13 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:13.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:13 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:13.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:13 smithi007 ceph-mon[28477]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:36:13.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:13 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:13.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:13 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:13.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:13 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:13.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:13 smithi007 ceph-mon[28477]: Reconfiguring daemon mon.b on smithi016 2024-04-21T08:36:14.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.035+0000 7fca65551700 1 -- 172.21.15.77:0/1555137684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca60103b90 msgr2=0x7fca60103fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.035+0000 7fca65551700 1 --2- 172.21.15.77:0/1555137684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca60103b90 0x7fca60103fb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fca50009910 tx=0x7fca50009c20 comp rx=0 tx=0).stop 2024-04-21T08:36:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.035+0000 7fca65551700 1 -- 172.21.15.77:0/1555137684 shutdown_connections 2024-04-21T08:36:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.035+0000 7fca65551700 1 --2- 172.21.15.77:0/1555137684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca60103b90 0x7fca60103fb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.035+0000 7fca65551700 1 -- 172.21.15.77:0/1555137684 >> 172.21.15.77:0/1555137684 conn(0x7fca600fc1f0 msgr2=0x7fca600fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.035+0000 7fca65551700 1 -- 172.21.15.77:0/1555137684 shutdown_connections 2024-04-21T08:36:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.035+0000 7fca65551700 1 -- 172.21.15.77:0/1555137684 wait complete. 2024-04-21T08:36:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca65551700 1 Processor -- start 2024-04-21T08:36:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca65551700 1 -- start start 2024-04-21T08:36:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca65551700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca60103b90 0x7fca60197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca65551700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca60198230 0x7fca6019d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:14.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca65551700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca601986b0 0x7fca6019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca65551700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fca60108490 con 0x7fca60103b90 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca65551700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fca60108310 con 0x7fca601986b0 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca65551700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fca60108610 con 0x7fca60198230 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca5e7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca60198230 0x7fca6019d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca5e7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca60198230 0x7fca6019d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55168/0 (socket says 172.21.15.77:55168) 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca5e7fc700 1 -- 172.21.15.77:0/998148746 learned_addr learned my addr 172.21.15.77:0/998148746 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca5effd700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca60103b90 0x7fca60197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.036+0000 7fca5f7fe700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca601986b0 0x7fca6019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca5e7fc700 1 -- 172.21.15.77:0/998148746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca601986b0 msgr2=0x7fca6019f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca5e7fc700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca601986b0 0x7fca6019f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca5e7fc700 1 -- 172.21.15.77:0/998148746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca60103b90 msgr2=0x7fca60197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:14.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca5e7fc700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca60103b90 0x7fca60197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca5e7fc700 1 -- 172.21.15.77:0/998148746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca601a0110 con 0x7fca60198230 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca5f7fe700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca601986b0 0x7fca6019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca5effd700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca60103b90 0x7fca60197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca5e7fc700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca60198230 0x7fca6019d260 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fca5400b780 tx=0x7fca5400ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca47fff700 1 -- 172.21.15.77:0/998148746 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca54004230 con 0x7fca60198230 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca47fff700 1 -- 172.21.15.77:0/998148746 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca540043d0 con 0x7fca60198230 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca65551700 1 -- 172.21.15.77:0/998148746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca601a0400 con 0x7fca60198230 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca47fff700 1 -- 172.21.15.77:0/998148746 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca54011b20 con 0x7fca60198230 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.037+0000 7fca65551700 1 -- 172.21.15.77:0/998148746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca601a09c0 con 0x7fca60198230 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.038+0000 7fca47fff700 1 -- 172.21.15.77:0/998148746 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fca54011cc0 con 0x7fca60198230 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.038+0000 7fca47fff700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fca48039ae0 0x7fca4803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.038+0000 7fca47fff700 1 -- 172.21.15.77:0/998148746 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fca5404c3f0 con 0x7fca60198230 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.038+0000 7fca5effd700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fca48039ae0 0x7fca4803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:14.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.038+0000 7fca65551700 1 -- 172.21.15.77:0/998148746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca6002df80 con 0x7fca60198230 2024-04-21T08:36:14.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.039+0000 7fca5effd700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fca48039ae0 0x7fca4803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca601991d0 tx=0x7fca5001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:14.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.044+0000 7fca47fff700 1 -- 172.21.15.77:0/998148746 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca54020500 con 0x7fca60198230 2024-04-21T08:36:14.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.263+0000 7fca65551700 1 -- 172.21.15.77:0/998148746 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi016=y;smithi077=z", "target": ["mon-mgr", ""]}) -- 0x7fca60062e80 con 0x7fca48039ae0 2024-04-21T08:36:14.269 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.270+0000 7fca47fff700 1 -- 172.21.15.77:0/998148746 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fca60062e80 con 0x7fca48039ae0 2024-04-21T08:36:14.269 INFO:teuthology.orchestra.run.smithi077.stdout:Scheduled mgr update... 2024-04-21T08:36:14.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 -- 172.21.15.77:0/998148746 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fca48039ae0 msgr2=0x7fca4803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:14.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fca48039ae0 0x7fca4803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca601991d0 tx=0x7fca5001a040 comp rx=0 tx=0).stop 2024-04-21T08:36:14.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 -- 172.21.15.77:0/998148746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca60198230 msgr2=0x7fca6019d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:14.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca60198230 0x7fca6019d260 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fca5400b780 tx=0x7fca5400ba90 comp rx=0 tx=0).stop 2024-04-21T08:36:14.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 -- 172.21.15.77:0/998148746 shutdown_connections 2024-04-21T08:36:14.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fca48039ae0 0x7fca4803bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:14.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca601986b0 0x7fca6019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:14.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca60103b90 0x7fca60197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:14.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 --2- 172.21.15.77:0/998148746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca60198230 0x7fca6019d260 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:14.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 -- 172.21.15.77:0/998148746 >> 172.21.15.77:0/998148746 conn(0x7fca600fc1f0 msgr2=0x7fca600fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:14.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 -- 172.21.15.77:0/998148746 shutdown_connections 2024-04-21T08:36:14.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:36:14.272+0000 7fca65551700 1 -- 172.21.15.77:0/998148746 wait complete. 2024-04-21T08:36:14.894 DEBUG:teuthology.orchestra.run.smithi016:mgr.y> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.y.service 2024-04-21T08:36:14.897 DEBUG:teuthology.orchestra.run.smithi077:mgr.z> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.z.service 2024-04-21T08:36:14.899 INFO:tasks.cephadm:Deploying OSDs... 2024-04-21T08:36:14.899 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:36:14.899 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:36:14.920 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:36:14.920 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-04-21T08:36:14.979 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:36:14.979 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:14.979 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-21T08:36:14.980 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:14.980 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:14.980 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-21 08:35:19.257235631 +0000 2024-04-21T08:36:14.980 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-21 08:30:00.523881400 +0000 2024-04-21T08:36:14.980 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-21 08:30:00.523881400 +0000 2024-04-21T08:36:14.980 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-21 08:30:00.523881400 +0000 2024-04-21T08:36:14.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:36:15.049 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-21T08:36:15.049 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-21T08:36:15.049 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.00013206 s, 3.9 MB/s 2024-04-21T08:36:15.050 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:36:15.109 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-04-21T08:36:15.167 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:36:15.167 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:15.168 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-21T08:36:15.168 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:15.168 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:15.168 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-21 08:35:19.294235079 +0000 2024-04-21T08:36:15.168 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-21 08:30:00.863876142 +0000 2024-04-21T08:36:15.168 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-21 08:30:00.863876142 +0000 2024-04-21T08:36:15.168 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-21 08:30:00.863876142 +0000 2024-04-21T08:36:15.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:36:15.218 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:14 smithi077 ceph-mon[31615]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:15.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:15.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:15.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:14 smithi077 ceph-mon[31615]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:36:15.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:15.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:15.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:15.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:14 smithi077 ceph-mon[31615]: Reconfiguring daemon mon.c on smithi077 2024-04-21T08:36:15.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:14 smithi077 ceph-mon[31615]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi016=y;smithi077=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:15.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:14 smithi077 ceph-mon[31615]: Saving service mgr spec with placement smithi007=x;smithi016=y;smithi077=z;count:3 2024-04-21T08:36:15.220 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:15.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:14 smithi007 ceph-mon[28477]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:15.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:15.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:15.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:14 smithi007 ceph-mon[28477]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:36:15.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:15.232 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:15.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:15.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:14 smithi007 ceph-mon[28477]: Reconfiguring daemon mon.c on smithi077 2024-04-21T08:36:15.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:14 smithi007 ceph-mon[28477]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi016=y;smithi077=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:15.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:14 smithi007 ceph-mon[28477]: Saving service mgr spec with placement smithi007=x;smithi016=y;smithi077=z;count:3 2024-04-21T08:36:15.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:15.234 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-21T08:36:15.235 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-21T08:36:15.235 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000117488 s, 4.4 MB/s 2024-04-21T08:36:15.236 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:36:15.294 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-04-21T08:36:15.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:14 smithi016 ceph-mon[31259]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:14 smithi016 ceph-mon[31259]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:36:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:14 smithi016 ceph-mon[31259]: Reconfiguring daemon mon.c on smithi077 2024-04-21T08:36:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:14 smithi016 ceph-mon[31259]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi016=y;smithi077=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:14 smithi016 ceph-mon[31259]: Saving service mgr spec with placement smithi007=x;smithi016=y;smithi077=z;count:3 2024-04-21T08:36:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:36:15.351 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:15.351 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-21T08:36:15.351 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:15.351 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:15.351 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-21 08:35:19.328234572 +0000 2024-04-21T08:36:15.351 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-21 08:30:01.211870760 +0000 2024-04-21T08:36:15.352 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-21 08:30:01.211870760 +0000 2024-04-21T08:36:15.352 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-21 08:30:01.211870760 +0000 2024-04-21T08:36:15.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:36:15.416 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-21T08:36:15.416 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-21T08:36:15.416 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000123302 s, 4.2 MB/s 2024-04-21T08:36:15.417 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:36:15.475 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-04-21T08:36:15.533 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:36:15.533 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:15.533 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-21T08:36:15.533 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:15.533 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:15.533 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-21 08:35:19.366234006 +0000 2024-04-21T08:36:15.533 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-21 08:30:01.549865533 +0000 2024-04-21T08:36:15.533 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-21 08:30:01.549865533 +0000 2024-04-21T08:36:15.533 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-21 08:30:01.549865533 +0000 2024-04-21T08:36:15.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:36:15.598 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-21T08:36:15.598 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-21T08:36:15.598 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000115574 s, 4.4 MB/s 2024-04-21T08:36:15.599 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:36:15.656 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-21T08:36:15.656 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:36:15.676 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:36:15.676 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-04-21T08:36:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:36:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:15.735 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-21T08:36:15.736 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:15.736 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:15.736 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-21 08:36:02.187684796 +0000 2024-04-21T08:36:15.736 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-21 08:30:27.638060611 +0000 2024-04-21T08:36:15.736 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-21 08:30:27.638060611 +0000 2024-04-21T08:36:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-21 08:30:27.638060611 +0000 2024-04-21T08:36:15.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:36:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-21T08:36:15.802 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-21T08:36:15.803 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000114819 s, 4.5 MB/s 2024-04-21T08:36:15.804 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:36:15.864 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-04-21T08:36:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:36:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:15.922 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-21T08:36:15.922 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:15.922 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:15.922 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-21 08:36:02.188684780 +0000 2024-04-21T08:36:15.922 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-21 08:30:27.979055077 +0000 2024-04-21T08:36:15.922 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-21 08:30:27.979055077 +0000 2024-04-21T08:36:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-21 08:30:27.979055077 +0000 2024-04-21T08:36:15.923 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:36:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-21T08:36:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-21T08:36:15.988 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000107419 s, 4.8 MB/s 2024-04-21T08:36:15.989 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:36:16.048 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-04-21T08:36:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:36:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:16.105 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-21T08:36:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-21 08:36:02.188684780 +0000 2024-04-21T08:36:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-21 08:30:28.318049574 +0000 2024-04-21T08:36:16.106 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-21 08:30:28.318049574 +0000 2024-04-21T08:36:16.106 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-21 08:30:28.318049574 +0000 2024-04-21T08:36:16.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:36:16.170 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-21T08:36:16.170 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-21T08:36:16.170 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.00011621 s, 4.4 MB/s 2024-04-21T08:36:16.171 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:36:16.228 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-04-21T08:36:16.284 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:36:16.285 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:16.285 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-21T08:36:16.285 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:16.285 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:16.285 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-21 08:36:02.188684780 +0000 2024-04-21T08:36:16.285 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-21 08:30:28.664043958 +0000 2024-04-21T08:36:16.285 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-21 08:30:28.664043958 +0000 2024-04-21T08:36:16.285 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-21 08:30:28.664043958 +0000 2024-04-21T08:36:16.286 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:36:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-21T08:36:16.350 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-21T08:36:16.352 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000117022 s, 4.4 MB/s 2024-04-21T08:36:16.353 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:36:16.410 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-21T08:36:16.410 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:36:16.430 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:36:16.430 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-04-21T08:36:16.488 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:36:16.488 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:16.488 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-21T08:36:16.488 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:16.488 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:16.488 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-21 08:35:52.570236276 +0000 2024-04-21T08:36:16.488 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-21 08:30:16.854045191 +0000 2024-04-21T08:36:16.488 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-21 08:30:16.854045191 +0000 2024-04-21T08:36:16.488 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-21 08:30:16.854045191 +0000 2024-04-21T08:36:16.489 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:36:16.553 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-21T08:36:16.553 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-21T08:36:16.553 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000282262 s, 1.8 MB/s 2024-04-21T08:36:16.554 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:36:16.611 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-04-21T08:36:16.667 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:36:16.667 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:16.667 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-21T08:36:16.667 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:16.667 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:16.667 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-21 08:35:52.570236276 +0000 2024-04-21T08:36:16.667 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-21 08:30:17.181038478 +0000 2024-04-21T08:36:16.668 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-21 08:30:17.181038478 +0000 2024-04-21T08:36:16.668 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-21 08:30:17.181038478 +0000 2024-04-21T08:36:16.668 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:36:16.729 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-21T08:36:16.729 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-21T08:36:16.729 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000215392 s, 2.4 MB/s 2024-04-21T08:36:16.730 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:36:16.786 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-04-21T08:36:16.841 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:36:16.841 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:16.841 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-21T08:36:16.841 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:16.841 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:16.842 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-21 08:35:52.570236276 +0000 2024-04-21T08:36:16.842 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-21 08:30:17.519031540 +0000 2024-04-21T08:36:16.842 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-21 08:30:17.519031540 +0000 2024-04-21T08:36:16.842 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-21 08:30:17.519031540 +0000 2024-04-21T08:36:16.842 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:36:16.904 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-21T08:36:16.904 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-21T08:36:16.904 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000202378 s, 2.5 MB/s 2024-04-21T08:36:16.905 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:36:16.961 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-04-21T08:36:17.017 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:36:17.017 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:17.017 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-21T08:36:17.017 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:17.017 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:17.017 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-21 08:35:52.571236255 +0000 2024-04-21T08:36:17.017 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-21 08:30:17.862024499 +0000 2024-04-21T08:36:17.017 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-21 08:30:17.862024499 +0000 2024-04-21T08:36:17.017 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-21 08:30:17.862024499 +0000 2024-04-21T08:36:17.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:36:17.079 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-21T08:36:17.079 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-21T08:36:17.079 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000211871 s, 2.4 MB/s 2024-04-21T08:36:17.080 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:36:17.114 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:16 smithi077 ceph-mon[31615]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:17.114 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:17.114 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:17.114 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:17.114 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:17.114 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:17.114 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:17.114 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:17.114 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:17.115 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:17.115 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:17.130 INFO:tasks.cephadm:Deploying osd.0 on smithi007 with /dev/vg_nvme/lv_4... 2024-04-21T08:36:17.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:36:17.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:16 smithi007 ceph-mon[28477]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:17.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:17.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:17.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:17.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:17.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:17.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:17.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:17.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:17.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:17.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:16 smithi016 ceph-mon[31259]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:17.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:17.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:17.382 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:18.117 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:17 smithi077 ceph-mon[31615]: Deploying daemon mgr.z on smithi077 2024-04-21T08:36:18.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:17 smithi007 ceph-mon[28477]: Deploying daemon mgr.z on smithi077 2024-04-21T08:36:18.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:17 smithi016 ceph-mon[31259]: Deploying daemon mgr.z on smithi077 2024-04-21T08:36:18.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:18 smithi007 ceph-mon[28477]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:19.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:18 smithi077 ceph-mon[31615]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:19.104 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:19.024+0000 7f10d14c1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:36:19.204 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:18 smithi016 ceph-mon[31259]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:19.286 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:36:19.286 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:36:19.286 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-21T08:36:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-21T08:36:19.287 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00825975 s, 1.3 GB/s 2024-04-21T08:36:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-21T08:36:19.603 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:19.219+0000 7f10d14c1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:36:19.603 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:19.319+0000 7f10d14c1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:36:19.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_4 2024-04-21T08:36:20.077 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:19.629+0000 7f10d14c1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:36:20.077 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:19.715+0000 7f10d14c1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:36:20.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:20.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:20.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:20.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:20.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:20.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:20.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:20.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:20 smithi007 ceph-mon[28477]: Deploying daemon mgr.y on smithi016 2024-04-21T08:36:20.204 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:20.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:20.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:20.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:20.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:20.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:20.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:20.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:20.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:20 smithi016 ceph-mon[31259]: Deploying daemon mgr.y on smithi016 2024-04-21T08:36:20.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:20.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:20.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:20.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:20.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:20.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:20.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:20.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:20 smithi077 ceph-mon[31615]: Deploying daemon mgr.y on smithi016 2024-04-21T08:36:21.116 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:21 smithi016 systemd[1]: Starting Ceph mgr.y for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:36:21.285 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:21 smithi077 ceph-mon[31615]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:21.285 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:20 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:20.940+0000 7f10d14c1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:36:21.285 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:21 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:21.112+0000 7f10d14c1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:36:21.286 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:21 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:21.286+0000 7f10d14c1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:36:21.361 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:21 smithi007 ceph-mon[28477]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:21.383 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:21 smithi016 ceph-mon[31259]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:21.383 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:21 smithi016 podman[32628]: 2024-04-21 08:36:21.287917835 +0000 UTC m=+0.104400261 container create 615c421659eb15eb8198f9c591e9510cc524bdb6867eb197be6ba3e0f9290764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-21T08:36:21.383 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:21 smithi016 podman[32628]: 2024-04-21 08:36:21.20172494 +0000 UTC m=+0.018207378 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:36:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.722+0000 7fdbb1da0700 1 -- 172.21.15.7:0/1638776263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac1019a0 msgr2=0x7fdbac101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.722+0000 7fdbb1da0700 1 --2- 172.21.15.7:0/1638776263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac1019a0 0x7fdbac101dc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c0099c0 tx=0x7fdb9c009cd0 comp rx=0 tx=0).stop 2024-04-21T08:36:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.722+0000 7fdbb1da0700 1 -- 172.21.15.7:0/1638776263 shutdown_connections 2024-04-21T08:36:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.722+0000 7fdbb1da0700 1 --2- 172.21.15.7:0/1638776263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac103540 0x7fdbac109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.722+0000 7fdbb1da0700 1 --2- 172.21.15.7:0/1638776263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac1019a0 0x7fdbac101dc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.722+0000 7fdbb1da0700 1 --2- 172.21.15.7:0/1638776263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbac102ba0 0x7fdbac103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.722+0000 7fdbb1da0700 1 -- 172.21.15.7:0/1638776263 >> 172.21.15.7:0/1638776263 conn(0x7fdbac0fcf40 msgr2=0x7fdbac0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.722+0000 7fdbb1da0700 1 -- 172.21.15.7:0/1638776263 shutdown_connections 2024-04-21T08:36:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.722+0000 7fdbb1da0700 1 -- 172.21.15.7:0/1638776263 wait complete. 2024-04-21T08:36:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.723+0000 7fdbb1da0700 1 Processor -- start 2024-04-21T08:36:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.723+0000 7fdbb1da0700 1 -- start start 2024-04-21T08:36:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.723+0000 7fdbb1da0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbac1019a0 0x7fdbac105a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.723+0000 7fdbb1da0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac102ba0 0x7fdbac104110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.723+0000 7fdbb1da0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac103540 0x7fdbac104650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.723+0000 7fdbb1da0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac10d430 con 0x7fdbac102ba0 2024-04-21T08:36:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.723+0000 7fdbb1da0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac10d2b0 con 0x7fdbac1019a0 2024-04-21T08:36:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.723+0000 7fdbb1da0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac10d5b0 con 0x7fdbac103540 2024-04-21T08:36:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbaaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac102ba0 0x7fdbac104110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbac1019a0 0x7fdbac105a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbab7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbac1019a0 0x7fdbac105a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:57508/0 (socket says 172.21.15.7:57508) 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbab7fe700 1 -- 172.21.15.7:0/975134082 learned_addr learned my addr 172.21.15.7:0/975134082 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbabfff700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac103540 0x7fdbac104650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbab7fe700 1 -- 172.21.15.7:0/975134082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac103540 msgr2=0x7fdbac104650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbab7fe700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac103540 0x7fdbac104650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbab7fe700 1 -- 172.21.15.7:0/975134082 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac102ba0 msgr2=0x7fdbac104110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbab7fe700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac102ba0 0x7fdbac104110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbab7fe700 1 -- 172.21.15.7:0/975134082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac104f20 con 0x7fdbac1019a0 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbab7fe700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbac1019a0 0x7fdbac105a70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c005b40 tx=0x7fdb9c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdba8ff9700 1 -- 172.21.15.7:0/975134082 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c027070 con 0x7fdbac1019a0 2024-04-21T08:36:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdba8ff9700 1 -- 172.21.15.7:0/975134082 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb9c022470 con 0x7fdbac1019a0 2024-04-21T08:36:21.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbb1da0700 1 -- 172.21.15.7:0/975134082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbac1a2d30 con 0x7fdbac1019a0 2024-04-21T08:36:21.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdba8ff9700 1 -- 172.21.15.7:0/975134082 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c009e60 con 0x7fdbac1019a0 2024-04-21T08:36:21.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.724+0000 7fdbb1da0700 1 -- 172.21.15.7:0/975134082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbac1a3190 con 0x7fdbac1019a0 2024-04-21T08:36:21.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.725+0000 7fdbb1da0700 1 -- 172.21.15.7:0/975134082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbac069730 con 0x7fdbac1019a0 2024-04-21T08:36:21.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.725+0000 7fdba8ff9700 1 -- 172.21.15.7:0/975134082 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fdb9c02a070 con 0x7fdbac1019a0 2024-04-21T08:36:21.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.726+0000 7fdba8ff9700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdb94039890 0x7fdb9403bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:21.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.726+0000 7fdba8ff9700 1 -- 172.21.15.7:0/975134082 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdb9c04d590 con 0x7fdbac1019a0 2024-04-21T08:36:21.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.726+0000 7fdbaaffd700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdb94039890 0x7fdb9403bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:21.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.726+0000 7fdbaaffd700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdb94039890 0x7fdb9403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdba0009c60 tx=0x7fdba0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:21.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.731+0000 7fdba8ff9700 1 -- 172.21.15.7:0/975134082 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb9c013070 con 0x7fdbac1019a0 2024-04-21T08:36:21.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:21 smithi016 podman[32628]: 2024-04-21 08:36:21.482112464 +0000 UTC m=+0.298594893 container init 615c421659eb15eb8198f9c591e9510cc524bdb6867eb197be6ba3e0f9290764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:36:21.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:21 smithi016 podman[32628]: 2024-04-21 08:36:21.484575032 +0000 UTC m=+0.301057468 container start 615c421659eb15eb8198f9c591e9510cc524bdb6867eb197be6ba3e0f9290764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:36:21.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:21 smithi016 ceph-mgr[32647]: -- 172.21.15.16:0/1286408121 <== mon.2 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f47cc125a0 con 0x55f47cbe7000 2024-04-21T08:36:21.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:21 smithi016 bash[32628]: 615c421659eb15eb8198f9c591e9510cc524bdb6867eb197be6ba3e0f9290764 2024-04-21T08:36:21.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:21 smithi016 systemd[1]: Started Ceph mgr.y for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:36:21.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:21 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:21.639+0000 7f66d60b4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:36:21.853 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:21 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:21.575+0000 7f10d14c1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:36:21.853 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:21 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:21.668+0000 7f10d14c1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:36:21.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:21.951+0000 7fdbb1da0700 1 -- 172.21.15.7:0/975134082 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fdbac06ae30 con 0x7fdb94039890 2024-04-21T08:36:22.274 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:21 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:21.835+0000 7f66d60b4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:36:22.275 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:21 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:21.942+0000 7f66d60b4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:36:22.275 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:22.274+0000 7f66d60b4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:36:22.353 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:21 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:21.871+0000 7f10d14c1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:36:22.354 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:22.072+0000 7f10d14c1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:36:22.554 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:22.373+0000 7f66d60b4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:36:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:22 smithi007 ceph-mon[28477]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:22 smithi007 ceph-mon[28477]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:22.917 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-mon[31615]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:22.917 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:22.917 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:22.917 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:22.917 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:22.917 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:22.917 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-mon[31615]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:22.917 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:22.917 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:22.918 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:22.918 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:22 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:22.704+0000 7f10d14c1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:36:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-mon[31259]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:23.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-mon[31259]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:23.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:23.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:23.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:23.243 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:23 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:23.142+0000 7f10d14c1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:36:23.243 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:23 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:23.243+0000 7f10d14c1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:36:23.603 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:23 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:23.447+0000 7f10d14c1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:36:24.034 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:23 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:23.684+0000 7f66d60b4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:36:24.034 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:23 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:23.859+0000 7f66d60b4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:36:24.034 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:24.035+0000 7f66d60b4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:36:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:36:24.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:24.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:24.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:24.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-mon[31259]: Reconfiguring daemon mgr.x on smithi007 2024-04-21T08:36:24.307 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:23 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:23.974+0000 7f10d14c1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:36:24.307 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:24.065+0000 7f10d14c1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:36:24.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:24.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:24.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.308 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:36:24.308 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:24.308 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:24.308 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:24.308 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-mon[31615]: Reconfiguring daemon mgr.x on smithi007 2024-04-21T08:36:24.603 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:24.307+0000 7f10d14c1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:36:24.603 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:24.439+0000 7f10d14c1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:36:24.604 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:24.533+0000 7f10d14c1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:36:24.636 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:24.332+0000 7f66d60b4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:36:24.636 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:24.426+0000 7f66d60b4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:36:24.636 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:24.636+0000 7f66d60b4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:36:24.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:24.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:24.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:24.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:36:24.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:24.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:24.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:24.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:24 smithi007 ceph-mon[28477]: Reconfiguring daemon mgr.x on smithi007 2024-04-21T08:36:24.962 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:24.648+0000 7f10d14c1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:36:25.054 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:24 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:24.834+0000 7f66d60b4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:36:25.218 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:24 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:24.962+0000 7f10d14c1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:36:25.554 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:25 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:25.458+0000 7f66d60b4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:36:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:25 smithi016 ceph-mon[31259]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:25 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2390146987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c55caf13-53cc-4a4d-9324-fc3ae9c6f182"}]: dispatch 2024-04-21T08:36:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:25 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2390146987' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c55caf13-53cc-4a4d-9324-fc3ae9c6f182"}]': finished 2024-04-21T08:36:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:25 smithi016 ceph-mon[31259]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:36:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:25 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:25 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:25 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3562863673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:36:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:25 smithi077 ceph-mon[31615]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:25 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2390146987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c55caf13-53cc-4a4d-9324-fc3ae9c6f182"}]: dispatch 2024-04-21T08:36:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:25 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2390146987' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c55caf13-53cc-4a4d-9324-fc3ae9c6f182"}]': finished 2024-04-21T08:36:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:25 smithi077 ceph-mon[31615]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:36:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:25 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:25 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:25 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3562863673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:36:25.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:25 smithi007 ceph-mon[28477]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:25.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:25 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2390146987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c55caf13-53cc-4a4d-9324-fc3ae9c6f182"}]: dispatch 2024-04-21T08:36:25.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:25 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2390146987' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c55caf13-53cc-4a4d-9324-fc3ae9c6f182"}]': finished 2024-04-21T08:36:25.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:25 smithi007 ceph-mon[28477]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:36:25.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:25 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:25.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:25 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:25.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:25 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3562863673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:36:25.853 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:25 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:25.559+0000 7f10d14c1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:36:25.854 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:36:25 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:36:25.774+0000 7f10d14c1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:36:26.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:25 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:25.892+0000 7f66d60b4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:36:26.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:25 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:25.990+0000 7f66d60b4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:36:26.198 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:26 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:26.199+0000 7f66d60b4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:36:26.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:26.227+0000 7fdba8ff9700 1 -- 172.21.15.7:0/975134082 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 14) ==== 92601+0+0 (secure 0 0 0) 0x7fdb9c022620 con 0x7fdbac1019a0 2024-04-21T08:36:26.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:26 smithi016 ceph-mon[31259]: Standby manager daemon z started 2024-04-21T08:36:26.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:26 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:36:26.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:26 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:36:26.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:26 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:36:26.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:26 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:36:26.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:26 smithi077 ceph-mon[31615]: Standby manager daemon z started 2024-04-21T08:36:26.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:26 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:36:26.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:26 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:36:26.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:26 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:36:26.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:26 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:36:26.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:26 smithi007 ceph-mon[28477]: Standby manager daemon z started 2024-04-21T08:36:26.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:26 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:36:26.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:26 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:36:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:26 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:36:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:26 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.77:0/698765053' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:36:27.054 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:26 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:26.758+0000 7f66d60b4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:36:27.054 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:26 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:26.853+0000 7f66d60b4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:36:27.457 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:27 smithi016 ceph-mon[31259]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:27.457 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:27 smithi016 ceph-mon[31259]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-21T08:36:27.457 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:27 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:36:27.458 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:27 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:27.103+0000 7f66d60b4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:36:27.458 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:27 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:27.241+0000 7f66d60b4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:36:27.458 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:27 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:27.339+0000 7f66d60b4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:36:27.458 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:27 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:27.458+0000 7f66d60b4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:36:27.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:27 smithi077 ceph-mon[31615]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:27.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:27 smithi077 ceph-mon[31615]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-21T08:36:27.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:27 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:36:27.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:27 smithi007 ceph-mon[28477]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:27.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:27 smithi007 ceph-mon[28477]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-21T08:36:27.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:27 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:36:28.053 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:27 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:27.782+0000 7f66d60b4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:36:28.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:28 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:28.407+0000 7f66d60b4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:36:28.804 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:36:28 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:36:28.629+0000 7f66d60b4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:36:29.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:29.235+0000 7fdba8ff9700 1 -- 172.21.15.7:0/975134082 <== mon.2 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7fdb9c020370 con 0x7fdbac1019a0 2024-04-21T08:36:29.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:29 smithi016 ceph-mon[31259]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:29 smithi016 ceph-mon[31259]: Standby manager daemon y started 2024-04-21T08:36:29.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:29 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:36:29.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:29 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:36:29.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:29 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:36:29.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:29 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:36:29.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:29 smithi077 ceph-mon[31615]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:29 smithi077 ceph-mon[31615]: Standby manager daemon y started 2024-04-21T08:36:29.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:29 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:36:29.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:29 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:36:29.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:29 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:36:29.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:29 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:36:29.618 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:29 smithi007 ceph-mon[28477]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.618 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:29 smithi007 ceph-mon[28477]: Standby manager daemon y started 2024-04-21T08:36:29.618 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:29 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:36:29.618 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:29 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:36:29.618 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:29 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:36:29.618 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:29 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.16:0/3120884372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:36:30.355 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:30 smithi007 ceph-mon[28477]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-21T08:36:30.355 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:30 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:36:30.355 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:30 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:30.355 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:30 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:30.355 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:30 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:30.355 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:30 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:30.355 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:30 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:30.355 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:30 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:30.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:30 smithi016 ceph-mon[31259]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-21T08:36:30.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:30 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:36:30.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:30 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:30.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:30 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:30.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:30 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:30.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:30 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:30.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:30 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:30.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:30 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:30.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:30 smithi077 ceph-mon[31615]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-21T08:36:30.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:30 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:36:30.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:30 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:30.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:30 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:30.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:30 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:30.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:30 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:30.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:30 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:30.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:30 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:31.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:31 smithi016 ceph-mon[31259]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:31.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:31 smithi077 ceph-mon[31615]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:31.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:31 smithi007 ceph-mon[28477]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:33.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:33 smithi016 ceph-mon[31259]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:33.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:36:33.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:33.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:33 smithi016 ceph-mon[31259]: Deploying daemon osd.0 on smithi007 2024-04-21T08:36:33.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:33 smithi077 ceph-mon[31615]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:33.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:36:33.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:33.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:33 smithi077 ceph-mon[31615]: Deploying daemon osd.0 on smithi007 2024-04-21T08:36:33.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:33 smithi007 ceph-mon[28477]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:33.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:36:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:33 smithi007 ceph-mon[28477]: Deploying daemon osd.0 on smithi007 2024-04-21T08:36:35.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:35 smithi016 ceph-mon[31259]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:35.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:35 smithi077 ceph-mon[31615]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:35.627 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:35 smithi007 ceph-mon[28477]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:37.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:37 smithi007 ceph-mon[28477]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:37.553 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:37 smithi016 ceph-mon[31259]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:37.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:37 smithi077 ceph-mon[31615]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:38 smithi007 ceph-mon[28477]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:38 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:38.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:38 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:38.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:38 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:38 smithi016 ceph-mon[31259]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:38 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:38 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:38 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:39.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:38 smithi077 ceph-mon[31615]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:39.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:38 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:39.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:38 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:39.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:38 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:40.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:40 smithi007 ceph-mon[28477]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:40.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:40.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:40.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:40.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:40.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:40.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.975+0000 7fdba8ff9700 1 -- 172.21.15.7:0/975134082 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdbac06ae30 con 0x7fdb94039890 2024-04-21T08:36:40.976 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 0 on host 'smithi007' 2024-04-21T08:36:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.977+0000 7fdbb1da0700 1 -- 172.21.15.7:0/975134082 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdb94039890 msgr2=0x7fdb9403bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.977+0000 7fdbb1da0700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdb94039890 0x7fdb9403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdba0009c60 tx=0x7fdba0009260 comp rx=0 tx=0).stop 2024-04-21T08:36:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.977+0000 7fdbb1da0700 1 -- 172.21.15.7:0/975134082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbac1019a0 msgr2=0x7fdbac105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.977+0000 7fdbb1da0700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbac1019a0 0x7fdbac105a70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c005b40 tx=0x7fdb9c009580 comp rx=0 tx=0).stop 2024-04-21T08:36:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.977+0000 7fdbb1da0700 1 -- 172.21.15.7:0/975134082 shutdown_connections 2024-04-21T08:36:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.977+0000 7fdbb1da0700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdb94039890 0x7fdb9403bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.977+0000 7fdbb1da0700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbac103540 0x7fdbac104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.977+0000 7fdbb1da0700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbac1019a0 0x7fdbac105a70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.977+0000 7fdbb1da0700 1 --2- 172.21.15.7:0/975134082 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac102ba0 0x7fdbac104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.977+0000 7fdbb1da0700 1 -- 172.21.15.7:0/975134082 >> 172.21.15.7:0/975134082 conn(0x7fdbac0fcf40 msgr2=0x7fdbac06a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:40.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.978+0000 7fdbb1da0700 1 -- 172.21.15.7:0/975134082 shutdown_connections 2024-04-21T08:36:40.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:36:40.978+0000 7fdbb1da0700 1 -- 172.21.15.7:0/975134082 wait complete. 2024-04-21T08:36:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:40 smithi016 ceph-mon[31259]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:41.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:40 smithi077 ceph-mon[31615]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:41.538 DEBUG:teuthology.orchestra.run.smithi007:osd.0> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.0.service 2024-04-21T08:36:41.540 INFO:tasks.cephadm:Deploying osd.1 on smithi016 with /dev/vg_nvme/lv_4... 2024-04-21T08:36:41.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:36:41.795 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:36:42.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:41 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:42.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:41 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:42.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:41 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:42.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:41 smithi007 ceph-mon[28477]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:36:42.231 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:41 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:42.231 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:41 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:42.231 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:41 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:42.231 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:41 smithi016 ceph-mon[31259]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:36:42.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:41 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:42.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:41 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:42.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:41 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:42.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:41 smithi077 ceph-mon[31615]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:36:43.050 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:42 smithi016 ceph-mon[31259]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.050 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:42 smithi016 ceph-mon[31259]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:36:43.050 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:42 smithi016 ceph-mon[31259]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:36:43.050 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:42 smithi016 ceph-mon[31259]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:36:43.050 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:42 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:43.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:42 smithi007 ceph-mon[28477]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:42 smithi007 ceph-mon[28477]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:36:43.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:42 smithi007 ceph-mon[28477]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:36:43.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:42 smithi007 ceph-mon[28477]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:36:43.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:42 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:43.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:42 smithi077 ceph-mon[31615]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:42 smithi077 ceph-mon[31615]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:36:43.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:42 smithi077 ceph-mon[31615]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:36:43.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:42 smithi077 ceph-mon[31615]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:36:43.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:42 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:36:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:36:43.612 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-21T08:36:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-21T08:36:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00880817 s, 1.2 GB/s 2024-04-21T08:36:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-21T08:36:43.859 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:36:43 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0[41045]: 2024-04-21T08:36:43.462+0000 7f81e3a72700 -1 osd.0 0 waiting for initial osdmap 2024-04-21T08:36:43.861 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:36:43 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0[41045]: 2024-04-21T08:36:43.466+0000 7f81db85d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:36:44.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:43 smithi007 ceph-mon[28477]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:36:44.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:43 smithi007 ceph-mon[28477]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:36:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:43 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:43 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:43 smithi007 ceph-mon[28477]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' 2024-04-21T08:36:44.195 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:43 smithi016 ceph-mon[31259]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:36:44.195 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:43 smithi016 ceph-mon[31259]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:36:44.195 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:43 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:44.195 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:43 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:44.195 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:43 smithi016 ceph-mon[31259]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' 2024-04-21T08:36:44.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-04-21T08:36:44.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:43 smithi077 ceph-mon[31615]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:36:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:43 smithi077 ceph-mon[31615]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:36:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:43 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:43 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:43 smithi077 ceph-mon[31615]: from='osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241]' entity='osd.0' 2024-04-21T08:36:44.457 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:36:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:36:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:36:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241] boot 2024-04-21T08:36:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:36:45.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:45.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:45.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:45.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:45.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:36:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:36:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241] boot 2024-04-21T08:36:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:36:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:45.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:45.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:36:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:36:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: osd.0 [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241] boot 2024-04-21T08:36:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:36:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:36:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:45.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:45.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:45.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:36:45.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.892+0000 7f515c92a700 1 -- 172.21.15.16:0/1029385107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51581035c0 msgr2=0x7f51581059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:45.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.892+0000 7f515c92a700 1 --2- 172.21.15.16:0/1029385107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51581035c0 0x7f51581059b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5148009990 tx=0x7f5148009ca0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.892+0000 7f515c92a700 1 -- 172.21.15.16:0/1029385107 shutdown_connections 2024-04-21T08:36:45.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.892+0000 7f515c92a700 1 --2- 172.21.15.16:0/1029385107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5158108820 0x7f515810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.892+0000 7f515c92a700 1 --2- 172.21.15.16:0/1029385107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51581035c0 0x7f51581059b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.892+0000 7f515c92a700 1 --2- 172.21.15.16:0/1029385107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5158105ef0 0x7f51581082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.892+0000 7f515c92a700 1 -- 172.21.15.16:0/1029385107 >> 172.21.15.16:0/1029385107 conn(0x7f51580fcf30 msgr2=0x7f51580ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.892+0000 7f515c92a700 1 -- 172.21.15.16:0/1029385107 shutdown_connections 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.892+0000 7f515c92a700 1 -- 172.21.15.16:0/1029385107 wait complete. 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.893+0000 7f515c92a700 1 Processor -- start 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.893+0000 7f515c92a700 1 -- start start 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.893+0000 7f515c92a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51581035c0 0x7f5158102f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.893+0000 7f515c92a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5158105ef0 0x7f5158101570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.893+0000 7f515c92a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5158108820 0x7f5158101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.893+0000 7f515c92a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f515810e4f0 con 0x7f5158105ef0 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.893+0000 7f515c92a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f515810e370 con 0x7f51581035c0 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.893+0000 7f515c92a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f515810e670 con 0x7f5158108820 2024-04-21T08:36:45.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f515659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51581035c0 0x7f5158102f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f515659c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51581035c0 0x7f5158102f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56542/0 (socket says 172.21.15.16:56542) 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f515659c700 1 -- 172.21.15.16:0/1892900615 learned_addr learned my addr 172.21.15.16:0/1892900615 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f5156d9d700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5158108820 0x7f5158101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f5155d9b700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5158105ef0 0x7f5158101570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f515659c700 1 -- 172.21.15.16:0/1892900615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5158108820 msgr2=0x7f5158101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f515659c700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5158108820 0x7f5158101ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f515659c700 1 -- 172.21.15.16:0/1892900615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5158105ef0 msgr2=0x7f5158101570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f515659c700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5158105ef0 0x7f5158101570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f515659c700 1 -- 172.21.15.16:0/1892900615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5158102360 con 0x7f51581035c0 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f515659c700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51581035c0 0x7f5158102f20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f51480036f0 tx=0x7f51480057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f514f7fe700 1 -- 172.21.15.16:0/1892900615 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5148027070 con 0x7f51581035c0 2024-04-21T08:36:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f514f7fe700 1 -- 172.21.15.16:0/1892900615 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5148022470 con 0x7f51581035c0 2024-04-21T08:36:45.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f515c92a700 1 -- 172.21.15.16:0/1892900615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51581a2bb0 con 0x7f51581035c0 2024-04-21T08:36:45.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f514f7fe700 1 -- 172.21.15.16:0/1892900615 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514800f6c0 con 0x7f51581035c0 2024-04-21T08:36:45.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.894+0000 7f515c92a700 1 -- 172.21.15.16:0/1892900615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51581a3000 con 0x7f51581035c0 2024-04-21T08:36:45.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.895+0000 7f515c92a700 1 -- 172.21.15.16:0/1892900615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5158069730 con 0x7f51581035c0 2024-04-21T08:36:45.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.896+0000 7f514f7fe700 1 -- 172.21.15.16:0/1892900615 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7f514802a070 con 0x7f51581035c0 2024-04-21T08:36:45.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.897+0000 7f514f7fe700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f51440a3c30 0x7f51440a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:45.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.897+0000 7f514f7fe700 1 -- 172.21.15.16:0/1892900615 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f51480cfdb0 con 0x7f51581035c0 2024-04-21T08:36:45.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.897+0000 7f5155d9b700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f51440a3c30 0x7f51440a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:45.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.898+0000 7f5155d9b700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f51440a3c30 0x7f51440a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5140005ea0 tx=0x7f5140005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:45.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:45.901+0000 7f514f7fe700 1 -- 172.21.15.16:0/1892900615 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5148031080 con 0x7f51581035c0 2024-04-21T08:36:46.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:45 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi007 2024-04-21T08:36:46.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:45 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi007 to 17093M 2024-04-21T08:36:46.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:45 smithi016 ceph-mon[31259]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:36:46.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:36:46.120+0000 7f515c92a700 1 -- 172.21.15.16:0/1892900615 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f5158062e80 con 0x7f51440a3c30 2024-04-21T08:36:46.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:45 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi007 2024-04-21T08:36:46.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:45 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi007 to 17093M 2024-04-21T08:36:46.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:45 smithi077 ceph-mon[31615]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:36:46.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:45 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi007 2024-04-21T08:36:46.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:45 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi007 to 17093M 2024-04-21T08:36:46.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:45 smithi007 ceph-mon[28477]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:36:47.014 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:46 smithi016 ceph-mon[31259]: pgmap v47: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:47.015 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:46 smithi016 ceph-mon[31259]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:47.015 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:47.015 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:47.015 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:47.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:46 smithi077 ceph-mon[31615]: pgmap v47: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:47.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:46 smithi077 ceph-mon[31615]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:47.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:47.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:47.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:47.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:46 smithi007 ceph-mon[28477]: pgmap v47: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:47.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:46 smithi007 ceph-mon[28477]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:47.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:47.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:47.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:48 smithi016 ceph-mon[31259]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:48 smithi016 ceph-mon[31259]: from='client.? 172.21.15.16:0/3701215931' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d953784-5cd8-4152-b25e-c1b36ccc8635"}]: dispatch 2024-04-21T08:36:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:48 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d953784-5cd8-4152-b25e-c1b36ccc8635"}]: dispatch 2024-04-21T08:36:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:48 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d953784-5cd8-4152-b25e-c1b36ccc8635"}]': finished 2024-04-21T08:36:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:48 smithi016 ceph-mon[31259]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:36:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:48 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:36:49.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:48 smithi077 ceph-mon[31615]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:48 smithi077 ceph-mon[31615]: from='client.? 172.21.15.16:0/3701215931' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d953784-5cd8-4152-b25e-c1b36ccc8635"}]: dispatch 2024-04-21T08:36:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:48 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d953784-5cd8-4152-b25e-c1b36ccc8635"}]: dispatch 2024-04-21T08:36:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:48 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d953784-5cd8-4152-b25e-c1b36ccc8635"}]': finished 2024-04-21T08:36:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:48 smithi077 ceph-mon[31615]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:36:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:48 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:36:49.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:48 smithi007 ceph-mon[28477]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:49.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:48 smithi007 ceph-mon[28477]: from='client.? 172.21.15.16:0/3701215931' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d953784-5cd8-4152-b25e-c1b36ccc8635"}]: dispatch 2024-04-21T08:36:49.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:48 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d953784-5cd8-4152-b25e-c1b36ccc8635"}]: dispatch 2024-04-21T08:36:49.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:48 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d953784-5cd8-4152-b25e-c1b36ccc8635"}]': finished 2024-04-21T08:36:49.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:48 smithi007 ceph-mon[28477]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:36:49.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:36:50.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:49 smithi016 ceph-mon[31259]: from='client.? 172.21.15.16:0/2158310808' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:36:50.351 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:49 smithi077 ceph-mon[31615]: from='client.? 172.21.15.16:0/2158310808' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:36:50.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:49 smithi007 ceph-mon[28477]: from='client.? 172.21.15.16:0/2158310808' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:36:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:50 smithi016 ceph-mon[31259]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:51.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:50 smithi077 ceph-mon[31615]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:51.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:50 smithi007 ceph-mon[28477]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:52 smithi016 ceph-mon[31259]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:53.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:52 smithi077 ceph-mon[31615]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:53.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:52 smithi007 ceph-mon[28477]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:55.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:54 smithi016 ceph-mon[31259]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:55.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:54 smithi077 ceph-mon[31615]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:55.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:54 smithi007 ceph-mon[28477]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:56.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:55 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:36:56.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:55 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:56.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:55 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:36:56.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:55 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:56.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:55 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:36:56.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:55 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:57.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:56 smithi016 ceph-mon[31259]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:57.161 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:56 smithi016 ceph-mon[31259]: Deploying daemon osd.1 on smithi016 2024-04-21T08:36:57.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:56 smithi077 ceph-mon[31615]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:57.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:56 smithi077 ceph-mon[31615]: Deploying daemon osd.1 on smithi016 2024-04-21T08:36:57.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:56 smithi007 ceph-mon[28477]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:57.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:56 smithi007 ceph-mon[28477]: Deploying daemon osd.1 on smithi016 2024-04-21T08:36:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:36:58 smithi016 ceph-mon[31259]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:59.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:36:58 smithi077 ceph-mon[31615]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:36:59.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:36:58 smithi007 ceph-mon[28477]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:01.246 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:00 smithi016 ceph-mon[31259]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:01.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:00 smithi077 ceph-mon[31615]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:01.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:00 smithi007 ceph-mon[28477]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:02.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:02.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:02.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:02.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:02 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:02.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:02 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:02.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:02 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:02.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:02 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:02.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:02 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:02.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:02 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:03.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:03 smithi016 ceph-mon[31259]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:03.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:03 smithi077 ceph-mon[31615]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:03.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:03 smithi007 ceph-mon[28477]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:04.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.890+0000 7f514f7fe700 1 -- 172.21.15.16:0/1892900615 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5158062e80 con 0x7f51440a3c30 2024-04-21T08:37:04.890 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 1 on host 'smithi016' 2024-04-21T08:37:04.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 -- 172.21.15.16:0/1892900615 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f51440a3c30 msgr2=0x7f51440a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:04.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f51440a3c30 0x7f51440a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5140005ea0 tx=0x7f5140005e30 comp rx=0 tx=0).stop 2024-04-21T08:37:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 -- 172.21.15.16:0/1892900615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51581035c0 msgr2=0x7f5158102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51581035c0 0x7f5158102f20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f51480036f0 tx=0x7f51480057c0 comp rx=0 tx=0).stop 2024-04-21T08:37:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 -- 172.21.15.16:0/1892900615 shutdown_connections 2024-04-21T08:37:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f51440a3c30 0x7f51440a60f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5158108820 0x7f5158101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51581035c0 0x7f5158102f20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 --2- 172.21.15.16:0/1892900615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5158105ef0 0x7f5158101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 -- 172.21.15.16:0/1892900615 >> 172.21.15.16:0/1892900615 conn(0x7f51580fcf30 msgr2=0x7f5158103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 -- 172.21.15.16:0/1892900615 shutdown_connections 2024-04-21T08:37:04.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:04.893+0000 7f515c92a700 1 -- 172.21.15.16:0/1892900615 wait complete. 2024-04-21T08:37:05.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:04 smithi016 ceph-mon[31259]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:05.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:05.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:05.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.124 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.125 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:05.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:04 smithi077 ceph-mon[31615]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:05.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:04 smithi007 ceph-mon[28477]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:05.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:05.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:05.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:05.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:05.419 DEBUG:teuthology.orchestra.run.smithi016:osd.1> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.1.service 2024-04-21T08:37:05.422 INFO:tasks.cephadm:Deploying osd.2 on smithi077 with /dev/vg_nvme/lv_4... 2024-04-21T08:37:05.422 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:37:05.661 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:37:06.093 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:05 smithi077 ceph-mon[31615]: from='osd.1 [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:06.094 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:05 smithi077 ceph-mon[31615]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:06.230 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:05 smithi016 ceph-mon[31259]: from='osd.1 [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:06.231 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:05 smithi016 ceph-mon[31259]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:06.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:05 smithi007 ceph-mon[28477]: from='osd.1 [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:06.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:05 smithi007 ceph-mon[28477]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:06 smithi016 ceph-mon[31259]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:06 smithi016 ceph-mon[31259]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:37:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:06 smithi016 ceph-mon[31259]: from='osd.1 [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:37:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:06 smithi016 ceph-mon[31259]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:37:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:06 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:06 smithi016 ceph-mon[31259]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:37:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:06 smithi077 ceph-mon[31615]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:06 smithi077 ceph-mon[31615]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:37:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:06 smithi077 ceph-mon[31615]: from='osd.1 [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:37:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:06 smithi077 ceph-mon[31615]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:37:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:06 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:06 smithi077 ceph-mon[31615]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:37:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:06 smithi007 ceph-mon[28477]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:06 smithi007 ceph-mon[28477]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:37:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:06 smithi007 ceph-mon[28477]: from='osd.1 [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:37:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:06 smithi007 ceph-mon[28477]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:37:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:06 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:07.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:06 smithi007 ceph-mon[28477]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:37:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:37:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:37:07.508 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-21T08:37:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-21T08:37:07.508 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112387 s, 933 MB/s 2024-04-21T08:37:07.508 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-21T08:37:07.741 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:37:07 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1[35871]: 2024-04-21T08:37:07.432+0000 7fdaf0ae5700 -1 osd.1 0 waiting for initial osdmap 2024-04-21T08:37:07.741 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:37:07 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1[35871]: 2024-04-21T08:37:07.436+0000 7fdae88d0700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:37:08.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:07 smithi016 ceph-mon[31259]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:37:08.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:07 smithi016 ceph-mon[31259]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:37:08.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:08.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:08.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:07 smithi016 ceph-mon[31259]: from='osd.1 ' entity='osd.1' 2024-04-21T08:37:08.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:07 smithi077 ceph-mon[31615]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:37:08.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:07 smithi077 ceph-mon[31615]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:37:08.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:08.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:08.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:07 smithi077 ceph-mon[31615]: from='osd.1 ' entity='osd.1' 2024-04-21T08:37:08.173 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_4 2024-04-21T08:37:08.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:07 smithi007 ceph-mon[28477]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:37:08.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:07 smithi007 ceph-mon[28477]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:37:08.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:08.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:08.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:07 smithi007 ceph-mon[28477]: from='osd.1 ' entity='osd.1' 2024-04-21T08:37:08.415 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:37:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:37:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:37:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: osd.1 [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202] boot 2024-04-21T08:37:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:37:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:09.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:09.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:09.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:37:09.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:37:09.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:09.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:09.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: osd.1 [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202] boot 2024-04-21T08:37:09.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:37:09.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:09.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:09.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:09.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:09.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:37:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:37:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: osd.1 [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202] boot 2024-04-21T08:37:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:37:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:09.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:09.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:09.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:09.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 -- 172.21.15.77:0/3356207980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93b8102aa0 msgr2=0x7f93b8102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:09.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 --2- 172.21.15.77:0/3356207980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93b8102aa0 0x7f93b8102ec0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f93a8009990 tx=0x7f93a8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:37:09.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 -- 172.21.15.77:0/3356207980 shutdown_connections 2024-04-21T08:37:09.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 --2- 172.21.15.77:0/3356207980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93b8102aa0 0x7f93b8102ec0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:09.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 --2- 172.21.15.77:0/3356207980 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93b8104640 0x7f93b810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:09.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 --2- 172.21.15.77:0/3356207980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93b8103ca0 0x7f93b8104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:09.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 -- 172.21.15.77:0/3356207980 >> 172.21.15.77:0/3356207980 conn(0x7f93b80fe040 msgr2=0x7f93b8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 -- 172.21.15.77:0/3356207980 shutdown_connections 2024-04-21T08:37:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 -- 172.21.15.77:0/3356207980 wait complete. 2024-04-21T08:37:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 Processor -- start 2024-04-21T08:37:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 -- start start 2024-04-21T08:37:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.958+0000 7f93be849700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93b8102aa0 0x7f93b8198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93be849700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93b8103ca0 0x7f93b8199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93be849700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93b8104640 0x7f93b81a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93be849700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f93b810e530 con 0x7f93b8102aa0 2024-04-21T08:37:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93be849700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f93b810e3b0 con 0x7f93b8103ca0 2024-04-21T08:37:09.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93be849700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f93b810e6b0 con 0x7f93b8104640 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93bcde6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93b8104640 0x7f93b81a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93bcde6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93b8104640 0x7f93b81a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41216/0 (socket says 172.21.15.77:41216) 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93bcde6700 1 -- 172.21.15.77:0/3372045386 learned_addr learned my addr 172.21.15.77:0/3372045386 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93b77fe700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93b8103ca0 0x7f93b8199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93b7fff700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93b8102aa0 0x7f93b8198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93bcde6700 1 -- 172.21.15.77:0/3372045386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93b8103ca0 msgr2=0x7f93b8199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93bcde6700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93b8103ca0 0x7f93b8199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93bcde6700 1 -- 172.21.15.77:0/3372045386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93b8102aa0 msgr2=0x7f93b8198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93bcde6700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93b8102aa0 0x7f93b8198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93bcde6700 1 -- 172.21.15.77:0/3372045386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b81a0840 con 0x7f93b8104640 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93bcde6700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93b8104640 0x7f93b81a01a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f93b0007d40 tx=0x7f93b000a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:09.961 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93b57fa700 1 -- 172.21.15.77:0/3372045386 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b0016020 con 0x7f93b8104640 2024-04-21T08:37:09.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93be849700 1 -- 172.21.15.77:0/3372045386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b81a0b30 con 0x7f93b8104640 2024-04-21T08:37:09.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93b57fa700 1 -- 172.21.15.77:0/3372045386 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93b000acb0 con 0x7f93b8104640 2024-04-21T08:37:09.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93b57fa700 1 -- 172.21.15.77:0/3372045386 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b00115e0 con 0x7f93b8104640 2024-04-21T08:37:09.962 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.959+0000 7f93be849700 1 -- 172.21.15.77:0/3372045386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b81a1020 con 0x7f93b8104640 2024-04-21T08:37:09.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.960+0000 7f93be849700 1 -- 172.21.15.77:0/3372045386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b8069730 con 0x7f93b8104640 2024-04-21T08:37:09.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.961+0000 7f93b57fa700 1 -- 172.21.15.77:0/3372045386 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7f93b0019020 con 0x7f93b8104640 2024-04-21T08:37:09.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.962+0000 7f93b57fa700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93a00a3bb0 0x7f93a00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:09.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.962+0000 7f93b57fa700 1 -- 172.21.15.77:0/3372045386 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7f93b00cea60 con 0x7f93b8104640 2024-04-21T08:37:09.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.962+0000 7f93b7fff700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93a00a3bb0 0x7f93a00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:09.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.963+0000 7f93b7fff700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93a00a3bb0 0x7f93a00a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f93a80038a0 tx=0x7f93a8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:09.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:09.967+0000 7f93b57fa700 1 -- 172.21.15.77:0/3372045386 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f93b0020020 con 0x7f93b8104640 2024-04-21T08:37:10.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:09 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi016 2024-04-21T08:37:10.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:09 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi016 to 17093M 2024-04-21T08:37:10.179 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:10.180+0000 7f93be849700 1 -- 172.21.15.77:0/3372045386 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f93b8062e80 con 0x7f93a00a3bb0 2024-04-21T08:37:10.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:09 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi016 2024-04-21T08:37:10.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:09 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi016 to 17093M 2024-04-21T08:37:10.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:09 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi016 2024-04-21T08:37:10.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:09 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi016 to 17093M 2024-04-21T08:37:11.073 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:10 smithi077 ceph-mon[31615]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:11.073 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:10 smithi077 ceph-mon[31615]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:37:11.073 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:10 smithi077 ceph-mon[31615]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:11.073 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:11.073 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:11.073 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:10 smithi016 ceph-mon[31259]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:10 smithi016 ceph-mon[31259]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:37:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:10 smithi016 ceph-mon[31259]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:11.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:10 smithi007 ceph-mon[28477]: pgmap v63: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:11.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:10 smithi007 ceph-mon[28477]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:37:11.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:10 smithi007 ceph-mon[28477]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:11.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:11.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:11.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:12 smithi016 ceph-mon[31259]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:12 smithi016 ceph-mon[31259]: from='client.? 172.21.15.77:0/2473376691' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb1a5dad-fdc9-4a5b-a41d-de604769e356"}]: dispatch 2024-04-21T08:37:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:12 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb1a5dad-fdc9-4a5b-a41d-de604769e356"}]: dispatch 2024-04-21T08:37:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:12 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb1a5dad-fdc9-4a5b-a41d-de604769e356"}]': finished 2024-04-21T08:37:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:12 smithi016 ceph-mon[31259]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:37:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:12 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:13.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:12 smithi077 ceph-mon[31615]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:13.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:12 smithi077 ceph-mon[31615]: from='client.? 172.21.15.77:0/2473376691' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb1a5dad-fdc9-4a5b-a41d-de604769e356"}]: dispatch 2024-04-21T08:37:13.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:12 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb1a5dad-fdc9-4a5b-a41d-de604769e356"}]: dispatch 2024-04-21T08:37:13.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:12 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb1a5dad-fdc9-4a5b-a41d-de604769e356"}]': finished 2024-04-21T08:37:13.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:12 smithi077 ceph-mon[31615]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:37:13.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:12 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:13.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:12 smithi007 ceph-mon[28477]: pgmap v65: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:13.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:12 smithi007 ceph-mon[28477]: from='client.? 172.21.15.77:0/2473376691' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb1a5dad-fdc9-4a5b-a41d-de604769e356"}]: dispatch 2024-04-21T08:37:13.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:12 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb1a5dad-fdc9-4a5b-a41d-de604769e356"}]: dispatch 2024-04-21T08:37:13.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:12 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb1a5dad-fdc9-4a5b-a41d-de604769e356"}]': finished 2024-04-21T08:37:13.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:12 smithi007 ceph-mon[28477]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:37:13.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:14.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:13 smithi016 ceph-mon[31259]: from='client.? 172.21.15.77:0/3418972418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:14.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:13 smithi077 ceph-mon[31615]: from='client.? 172.21.15.77:0/3418972418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:14.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:13 smithi007 ceph-mon[28477]: from='client.? 172.21.15.77:0/3418972418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:14 smithi016 ceph-mon[31259]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:15.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:14 smithi077 ceph-mon[31615]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:15.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:14 smithi007 ceph-mon[28477]: pgmap v67: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:17.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:16 smithi077 ceph-mon[31615]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:16 smithi016 ceph-mon[31259]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:17.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:16 smithi007 ceph-mon[28477]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:18 smithi016 ceph-mon[31259]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:19.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:18 smithi077 ceph-mon[31615]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:19.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:18 smithi007 ceph-mon[28477]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:21.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:20 smithi077 ceph-mon[31615]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:21.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:37:21.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:21.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:20 smithi077 ceph-mon[31615]: Deploying daemon osd.2 on smithi077 2024-04-21T08:37:21.307 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:20 smithi016 ceph-mon[31259]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:21.307 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:37:21.307 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:21.307 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:20 smithi016 ceph-mon[31259]: Deploying daemon osd.2 on smithi077 2024-04-21T08:37:21.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:20 smithi007 ceph-mon[28477]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:21.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:37:21.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:21.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:20 smithi007 ceph-mon[28477]: Deploying daemon osd.2 on smithi077 2024-04-21T08:37:23.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:22 smithi077 ceph-mon[31615]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:22 smithi016 ceph-mon[31259]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:23.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:22 smithi007 ceph-mon[28477]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:25.210 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:24 smithi077 ceph-mon[31615]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:25.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:24 smithi016 ceph-mon[31259]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:25.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:24 smithi007 ceph-mon[28477]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:26.743 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:26.743 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:26.743 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:26.743 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:26 smithi077 ceph-mon[31615]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:26 smithi016 ceph-mon[31259]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:26.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:26.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:26.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:26.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:26 smithi007 ceph-mon[28477]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:28.851 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 2 on host 'smithi077' 2024-04-21T08:37:28.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.851+0000 7f93b57fa700 1 -- 172.21.15.77:0/3372045386 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f93b8062e80 con 0x7f93a00a3bb0 2024-04-21T08:37:28.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 -- 172.21.15.77:0/3372045386 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93a00a3bb0 msgr2=0x7f93a00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:28.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93a00a3bb0 0x7f93a00a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f93a80038a0 tx=0x7f93a8005fb0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 -- 172.21.15.77:0/3372045386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93b8104640 msgr2=0x7f93b81a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:28.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93b8104640 0x7f93b81a01a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f93b0007d40 tx=0x7f93b000a340 comp rx=0 tx=0).stop 2024-04-21T08:37:28.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 -- 172.21.15.77:0/3372045386 shutdown_connections 2024-04-21T08:37:28.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93a00a3bb0 0x7f93a00a6070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93b8104640 0x7f93b81a01a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93b8102aa0 0x7f93b8198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 --2- 172.21.15.77:0/3372045386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93b8103ca0 0x7f93b8199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:28.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 -- 172.21.15.77:0/3372045386 >> 172.21.15.77:0/3372045386 conn(0x7f93b80fe040 msgr2=0x7f93b81070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:28.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 -- 172.21.15.77:0/3372045386 shutdown_connections 2024-04-21T08:37:28.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:37:28.854+0000 7f93be849700 1 -- 172.21.15.77:0/3372045386 wait complete. 2024-04-21T08:37:29.465 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:29 smithi077 ceph-mon[31615]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:29.465 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:29 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.465 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:29 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.465 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:29 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:29.465 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:29 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:29.465 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:29 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.465 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:29 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.466 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:29 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.466 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:29 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:29.466 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:29 smithi077 ceph-mon[31615]: from='osd.2 [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:37:29.466 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:29 smithi077 ceph-mon[31615]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:37:29.491 DEBUG:teuthology.orchestra.run.smithi077:osd.2> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.2.service 2024-04-21T08:37:29.493 INFO:tasks.cephadm:Deploying osd.3 on smithi007 with /dev/vg_nvme/lv_3... 2024-04-21T08:37:29.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:37:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:37:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:29 smithi016 ceph-mon[31259]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:29 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:29 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:29 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:29 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:29 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:29 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:29 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:29 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:29 smithi016 ceph-mon[31259]: from='osd.2 [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:37:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:29 smithi016 ceph-mon[31259]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:37:29.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:29 smithi007 ceph-mon[28477]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:29.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:29 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:29 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:29 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:29.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:29 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:29.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:29 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:29 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:29 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:29.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:29 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:29.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:29 smithi007 ceph-mon[28477]: from='osd.2 [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:37:29.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:29 smithi007 ceph-mon[28477]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:37:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:30 smithi077 ceph-mon[31615]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:30 smithi077 ceph-mon[31615]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:37:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:30 smithi077 ceph-mon[31615]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:37:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:30 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:30 smithi077 ceph-mon[31615]: from='osd.2 [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:37:30.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:30 smithi077 ceph-mon[31615]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:37:30.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:37:30 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2[36765]: 2024-04-21T08:37:30.823+0000 7f25a9bb3700 -1 osd.2 0 waiting for initial osdmap 2024-04-21T08:37:30.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:37:30 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2[36765]: 2024-04-21T08:37:30.829+0000 7f25a59a6700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:37:31.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:30 smithi007 ceph-mon[28477]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:31.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:30 smithi007 ceph-mon[28477]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:37:31.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:30 smithi007 ceph-mon[28477]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:37:31.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:30 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:31.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:30 smithi007 ceph-mon[28477]: from='osd.2 [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:37:31.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:30 smithi007 ceph-mon[28477]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:37:31.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:30 smithi016 ceph-mon[31259]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:31.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:30 smithi016 ceph-mon[31259]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:37:31.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:30 smithi016 ceph-mon[31259]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:37:31.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:30 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:31.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:30 smithi016 ceph-mon[31259]: from='osd.2 [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:37:31.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:30 smithi016 ceph-mon[31259]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00859113 s, 1.2 GB/s 2024-04-21T08:37:31.698 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-21T08:37:31.995 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:31 smithi077 ceph-mon[31615]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:37:31.996 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:31 smithi077 ceph-mon[31615]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:37:31.996 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:31 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:31.996 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:31 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:32.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:31 smithi007 ceph-mon[28477]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:37:32.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:31 smithi007 ceph-mon[28477]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:37:32.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:31 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:32.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:31 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:32.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_3 2024-04-21T08:37:32.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:31 smithi016 ceph-mon[31259]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:37:32.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:31 smithi016 ceph-mon[31259]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:37:32.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:31 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:32.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:31 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:32.555 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:37:32.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:37:32.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:37:32.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: pgmap v78: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:32.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:32.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: osd.2 [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623] boot 2024-04-21T08:37:32.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:37:32.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:32.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi077 2024-04-21T08:37:32.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:32.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:32.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:32.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi077 to 17088M 2024-04-21T08:37:32.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:32.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:32.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:32.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:32 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: pgmap v78: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: osd.2 [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623] boot 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi077 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi077 to 17088M 2024-04-21T08:37:33.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:33.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:33.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:33.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:32 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:37:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:37:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: pgmap v78: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: osd.2 [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623] boot 2024-04-21T08:37:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:37:33.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:33.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi077 2024-04-21T08:37:33.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:33.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:33.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:33.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi077 to 17088M 2024-04-21T08:37:33.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:33.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:33.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:33.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:32 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:34.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.040+0000 7f6874f2c700 1 -- 172.21.15.7:0/3050224395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6870103cc0 msgr2=0x7f6870104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:34.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.040+0000 7f6874f2c700 1 --2- 172.21.15.7:0/3050224395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6870103cc0 0x7f6870104120 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f68640099c0 tx=0x7f6864009cd0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.041+0000 7f6874f2c700 1 -- 172.21.15.7:0/3050224395 shutdown_connections 2024-04-21T08:37:34.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.041+0000 7f6874f2c700 1 --2- 172.21.15.7:0/3050224395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6870104660 0x7f687010af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.041+0000 7f6874f2c700 1 --2- 172.21.15.7:0/3050224395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6870102ac0 0x7f6870102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.041+0000 7f6874f2c700 1 --2- 172.21.15.7:0/3050224395 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6870103cc0 0x7f6870104120 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.041+0000 7f6874f2c700 1 -- 172.21.15.7:0/3050224395 >> 172.21.15.7:0/3050224395 conn(0x7f68700fe060 msgr2=0x7f68701004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:34.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.041+0000 7f6874f2c700 1 -- 172.21.15.7:0/3050224395 shutdown_connections 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.041+0000 7f6874f2c700 1 -- 172.21.15.7:0/3050224395 wait complete. 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.041+0000 7f6874f2c700 1 Processor -- start 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.041+0000 7f6874f2c700 1 -- start start 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f6874f2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6870102ac0 0x7f6870198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f6874f2c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6870103cc0 0x7f6870199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f6874f2c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6870104660 0x7f68701a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f6874f2c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f687010e530 con 0x7f6870102ac0 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f6874f2c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f687010e3b0 con 0x7f6870103cc0 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f6874f2c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f687010e6b0 con 0x7f6870104660 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f686e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6870102ac0 0x7f6870198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f686e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6870102ac0 0x7f6870198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53662/0 (socket says 172.21.15.7:53662) 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f686e59c700 1 -- 172.21.15.7:0/2566397741 learned_addr learned my addr 172.21.15.7:0/2566397741 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:37:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f686ed9d700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6870104660 0x7f68701a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f686dd9b700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6870103cc0 0x7f6870199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f686e59c700 1 -- 172.21.15.7:0/2566397741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6870104660 msgr2=0x7f68701a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f686e59c700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6870104660 0x7f68701a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f686e59c700 1 -- 172.21.15.7:0/2566397741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6870103cc0 msgr2=0x7f6870199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f686e59c700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6870103cc0 0x7f6870199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f686e59c700 1 -- 172.21.15.7:0/2566397741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68701a0870 con 0x7f6870102ac0 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.042+0000 7f686e59c700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6870102ac0 0x7f6870198c20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f686000d980 tx=0x7f686000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.043+0000 7f685f7fe700 1 -- 172.21.15.7:0/2566397741 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6860017030 con 0x7f6870102ac0 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.043+0000 7f6874f2c700 1 -- 172.21.15.7:0/2566397741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68701a0b60 con 0x7f6870102ac0 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.043+0000 7f685f7fe700 1 -- 172.21.15.7:0/2566397741 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68600044d0 con 0x7f6870102ac0 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.043+0000 7f685f7fe700 1 -- 172.21.15.7:0/2566397741 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6860003970 con 0x7f6870102ac0 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.043+0000 7f6874f2c700 1 -- 172.21.15.7:0/2566397741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68701a10a0 con 0x7f6870102ac0 2024-04-21T08:37:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.044+0000 7f6874f2c700 1 -- 172.21.15.7:0/2566397741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6870069730 con 0x7f6870102ac0 2024-04-21T08:37:34.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.045+0000 7f685f7fe700 1 -- 172.21.15.7:0/2566397741 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7f686001a070 con 0x7f6870102ac0 2024-04-21T08:37:34.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.046+0000 7f685f7fe700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f68580a3bb0 0x7f68580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:34.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.046+0000 7f686dd9b700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f68580a3bb0 0x7f68580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:34.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.046+0000 7f685f7fe700 1 -- 172.21.15.7:0/2566397741 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 2851+0+0 (secure 0 0 0) 0x7f68600ce9b0 con 0x7f6870102ac0 2024-04-21T08:37:34.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.046+0000 7f686dd9b700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f68580a3bb0 0x7f68580a6070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6864009990 tx=0x7f686400b870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:34.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.050+0000 7f685f7fe700 1 -- 172.21.15.7:0/2566397741 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f686009c9c0 con 0x7f6870102ac0 2024-04-21T08:37:34.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:34.265+0000 7f6874f2c700 1 -- 172.21.15.7:0/2566397741 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6870062e80 con 0x7f68580a3bb0 2024-04-21T08:37:34.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:34 smithi007 ceph-mon[28477]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:37:34.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:34 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:37:34.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:34 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:34.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:34 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:34.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:34 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:34.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:34 smithi016 ceph-mon[31259]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:37:34.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:34 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:37:34.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:34 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:34.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:34 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:34.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:34 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:34.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:34 smithi077 ceph-mon[31615]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:37:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:34 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:37:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:34 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:34 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:34 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:35.492 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 ceph-mon[28477]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:35.492 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 ceph-mon[28477]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:35.492 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:37:35.493 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 ceph-mon[28477]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:37:35.493 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:37:35.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40403]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:37:35.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40403]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:35.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40403]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:35.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40403]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:35.633 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 ceph-mon[31615]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:35.633 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 ceph-mon[31615]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:35.633 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:37:35.633 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 ceph-mon[31615]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:37:35.633 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:37:35.750 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 ceph-mon[31259]: pgmap v81: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:35.750 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 ceph-mon[31259]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:35.750 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:37:35.750 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 ceph-mon[31259]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:37:35.750 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:37:35.750 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39558]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:37:35.750 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39558]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:35.751 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39558]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:35.751 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39558]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:35.751 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39561]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:37:35.751 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39561]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:35.751 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39561]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:35.751 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39561]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:35.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45473]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:37:35.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45473]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:35.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45473]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:35.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45473]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:35.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45476]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:37:35.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45476]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:35.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45476]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:35.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45476]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:35.903 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45466]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:37:35.903 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45466]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:35.903 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45466]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:35.903 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45466]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:35.903 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45469]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:37:35.903 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45469]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:35.903 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45469]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:35.904 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:37:35 smithi007 sudo[45469]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:36.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39564]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:37:36.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39564]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:36.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39564]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:36.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39564]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:36.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39567]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:37:36.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39567]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:36.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39567]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:36.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:35 smithi016 sudo[39567]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:36.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40406]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:37:36.104 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40406]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:36.104 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40406]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:36.104 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40406]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:36.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40409]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:37:36.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40409]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:36.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40409]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:36.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40409]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:36.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40412]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:37:36.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40412]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:37:36.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40412]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:37:36.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:35 smithi077 sudo[40412]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:37:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:36.552+0000 7f685f7fe700 1 -- 172.21.15.7:0/2566397741 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6860098930 con 0x7f6870102ac0 2024-04-21T08:37:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:37:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:37:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:37:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:37:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:37:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:37:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:37:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:37:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:37:36.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:37:36.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:37:36.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:37:36.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:37:36.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:37:36.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:37:36.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:37:36.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:36 smithi007 ceph-mon[28477]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:37:36.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:37:36.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:37:36.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:36.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:36.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:36.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:37:36.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:37:36.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:37:36.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:37:36.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:37:36.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:37:36.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:37:36.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:37:36.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:37:36.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:37:36.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:37:36.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:37:36.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:37:36.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:37:36.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:36 smithi016 ceph-mon[31259]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:37:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:37:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:37:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:37:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:37:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:37:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:37:36.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:37:36.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:37:36.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:37:36.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:37:36.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:37:36.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:37:36.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:37:36.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:37:36.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:37:36.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:37:36.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:36 smithi077 ceph-mon[31615]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:37:37.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:37 smithi007 ceph-mon[28477]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:37.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:37 smithi007 ceph-mon[28477]: osdmap e22: 3 total, 3 up, 3 in 2024-04-21T08:37:37.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:37 smithi007 ceph-mon[28477]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:37:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:37 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/473978752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b911243-4599-4632-9186-43b2714c8d39"}]: dispatch 2024-04-21T08:37:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:37 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/473978752' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b911243-4599-4632-9186-43b2714c8d39"}]': finished 2024-04-21T08:37:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:37 smithi007 ceph-mon[28477]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:37:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:37 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:37 smithi016 ceph-mon[31259]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:37 smithi016 ceph-mon[31259]: osdmap e22: 3 total, 3 up, 3 in 2024-04-21T08:37:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:37 smithi016 ceph-mon[31259]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:37:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:37 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/473978752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b911243-4599-4632-9186-43b2714c8d39"}]: dispatch 2024-04-21T08:37:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:37 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/473978752' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b911243-4599-4632-9186-43b2714c8d39"}]': finished 2024-04-21T08:37:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:37 smithi016 ceph-mon[31259]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:37:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:37 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:37.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:37 smithi077 ceph-mon[31615]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:37.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:37 smithi077 ceph-mon[31615]: osdmap e22: 3 total, 3 up, 3 in 2024-04-21T08:37:37.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:37 smithi077 ceph-mon[31615]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:37:37.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:37 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/473978752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b911243-4599-4632-9186-43b2714c8d39"}]: dispatch 2024-04-21T08:37:37.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:37 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/473978752' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b911243-4599-4632-9186-43b2714c8d39"}]': finished 2024-04-21T08:37:37.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:37 smithi077 ceph-mon[31615]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:37:37.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:37 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:38.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:38 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4270161601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:38.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:38 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4270161601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:38.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:38 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4270161601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:39.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:39 smithi007 ceph-mon[28477]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:39.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:39 smithi016 ceph-mon[31259]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:39.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:39 smithi077 ceph-mon[31615]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:41.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:41 smithi007 ceph-mon[28477]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:41.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:41 smithi016 ceph-mon[31259]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:41.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:41 smithi077 ceph-mon[31615]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:43.643 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:43 smithi007 ceph-mon[28477]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:43.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:43 smithi016 ceph-mon[31259]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:43.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:43 smithi077 ceph-mon[31615]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:45.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:45 smithi007 ceph-mon[28477]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:45.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:45 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:37:45.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:45 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:45.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:45 smithi016 ceph-mon[31259]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:45.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:45 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:37:45.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:45 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:45.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:45 smithi077 ceph-mon[31615]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:45.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:45 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:37:45.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:45 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:46.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:46 smithi007 ceph-mon[28477]: Deploying daemon osd.3 on smithi007 2024-04-21T08:37:46.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:46 smithi016 ceph-mon[31259]: Deploying daemon osd.3 on smithi007 2024-04-21T08:37:46.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:46 smithi077 ceph-mon[31615]: Deploying daemon osd.3 on smithi007 2024-04-21T08:37:47.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:47 smithi007 ceph-mon[28477]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:47.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:47 smithi016 ceph-mon[31259]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:47.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:47 smithi077 ceph-mon[31615]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:49.615 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:49 smithi007 ceph-mon[28477]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:49.669 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:49 smithi077 ceph-mon[31615]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:49.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:49 smithi016 ceph-mon[31259]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:51.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:50 smithi077 ceph-mon[31615]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:51.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:50 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:51.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:50 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:51.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:50 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:51.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:50 smithi007 ceph-mon[28477]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:51.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:50 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:51.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:50 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:51.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:50 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:50 smithi016 ceph-mon[31259]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:50 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:50 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:50 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:52 smithi016 ceph-mon[31259]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:53.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:52 smithi007 ceph-mon[28477]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:53.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:53.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:53.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:53.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:53.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:53.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:52 smithi077 ceph-mon[31615]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:53.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:53.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:53.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:53.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:53.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:53.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.670+0000 7f685f7fe700 1 -- 172.21.15.7:0/2566397741 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6870062e80 con 0x7f68580a3bb0 2024-04-21T08:37:53.670 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 3 on host 'smithi007' 2024-04-21T08:37:53.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 -- 172.21.15.7:0/2566397741 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f68580a3bb0 msgr2=0x7f68580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f68580a3bb0 0x7f68580a6070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6864009990 tx=0x7f686400b870 comp rx=0 tx=0).stop 2024-04-21T08:37:53.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 -- 172.21.15.7:0/2566397741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6870102ac0 msgr2=0x7f6870198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:53.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6870102ac0 0x7f6870198c20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f686000d980 tx=0x7f686000dc90 comp rx=0 tx=0).stop 2024-04-21T08:37:53.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 -- 172.21.15.7:0/2566397741 shutdown_connections 2024-04-21T08:37:53.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f68580a3bb0 0x7f68580a6070 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6870104660 0x7f68701a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6870102ac0 0x7f6870198c20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 --2- 172.21.15.7:0/2566397741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6870103cc0 0x7f6870199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:53.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 -- 172.21.15.7:0/2566397741 >> 172.21.15.7:0/2566397741 conn(0x7f68700fe060 msgr2=0x7f6870107030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:53.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 -- 172.21.15.7:0/2566397741 shutdown_connections 2024-04-21T08:37:53.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:37:53.672+0000 7f6874f2c700 1 -- 172.21.15.7:0/2566397741 wait complete. 2024-04-21T08:37:54.295 DEBUG:teuthology.orchestra.run.smithi007:osd.3> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.3.service 2024-04-21T08:37:54.297 INFO:tasks.cephadm:Deploying osd.4 on smithi016 with /dev/vg_nvme/lv_3... 2024-04-21T08:37:54.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:37:54.553 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:37:54.898 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:54 smithi016 ceph-mon[31259]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:54.898 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:54.898 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:54.899 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:54.899 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:54 smithi016 ceph-mon[31259]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:37:54.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:54 smithi007 ceph-mon[28477]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:54.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:54.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:54.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:54.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:54 smithi007 ceph-mon[28477]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:37:55.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:54 smithi077 ceph-mon[31615]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:54 smithi077 ceph-mon[31615]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:37:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:55 smithi007 ceph-mon[28477]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:37:55.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:55 smithi007 ceph-mon[28477]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:37:55.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:55 smithi007 ceph-mon[28477]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:37:55.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:55 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:55.732 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:37:55 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3[47619]: 2024-04-21T08:37:55.653+0000 7f18ad382700 -1 osd.3 0 waiting for initial osdmap 2024-04-21T08:37:55.732 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:37:55 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3[47619]: 2024-04-21T08:37:55.660+0000 7f18a9175700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:37:55.950 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:55 smithi016 ceph-mon[31259]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:37:55.950 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:55 smithi016 ceph-mon[31259]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:37:55.950 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:55 smithi016 ceph-mon[31259]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:37:55.950 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:55 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:56.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:55 smithi077 ceph-mon[31615]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:37:56.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:55 smithi077 ceph-mon[31615]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:37:56.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:55 smithi077 ceph-mon[31615]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:37:56.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:55 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:56.552 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:37:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:37:56.553 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-21T08:37:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-21T08:37:56.553 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00834039 s, 1.3 GB/s 2024-04-21T08:37:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-21T08:37:56.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:56 smithi007 ceph-mon[28477]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:56.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:56 smithi007 ceph-mon[28477]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:37:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:56 smithi007 ceph-mon[28477]: osdmap e25: 4 total, 3 up, 4 in 2024-04-21T08:37:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:56 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:56 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:56 smithi007 ceph-mon[28477]: osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881] boot 2024-04-21T08:37:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:56 smithi007 ceph-mon[28477]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:37:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:56 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:56 smithi016 ceph-mon[31259]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:56 smithi016 ceph-mon[31259]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:37:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:56 smithi016 ceph-mon[31259]: osdmap e25: 4 total, 3 up, 4 in 2024-04-21T08:37:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:56 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:56 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:56 smithi016 ceph-mon[31259]: osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881] boot 2024-04-21T08:37:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:56 smithi016 ceph-mon[31259]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:37:56.918 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:56 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:57.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:56 smithi077 ceph-mon[31615]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:37:57.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:56 smithi077 ceph-mon[31615]: from='osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:37:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:56 smithi077 ceph-mon[31615]: osdmap e25: 4 total, 3 up, 4 in 2024-04-21T08:37:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:56 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:56 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:56 smithi077 ceph-mon[31615]: osd.3 [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881] boot 2024-04-21T08:37:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:56 smithi077 ceph-mon[31615]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:37:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:56 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:37:57.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-04-21T08:37:57.477 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:37:57.827 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi007 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi007 to 8546M 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:57.828 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:57.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:37:57.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:37:57.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:37:57.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi007 2024-04-21T08:37:57.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:57.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:57.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:57.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:57.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi007 to 8546M 2024-04-21T08:37:57.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:57.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:57.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:57.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi007 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi007 to 8546M 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:37:59.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.107+0000 7fc0f071b700 1 -- 172.21.15.16:0/2866016519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e8108790 msgr2=0x7fc0e810ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.107+0000 7fc0f071b700 1 --2- 172.21.15.16:0/2866016519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e8108790 0x7fc0e810ac00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000b0a0 tx=0x7fc0e000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.107+0000 7fc0f071b700 1 -- 172.21.15.16:0/2866016519 shutdown_connections 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.107+0000 7fc0f071b700 1 --2- 172.21.15.16:0/2866016519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e8103530 0x7fc0e8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.107+0000 7fc0f071b700 1 --2- 172.21.15.16:0/2866016519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e8108790 0x7fc0e810ac00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.107+0000 7fc0f071b700 1 --2- 172.21.15.16:0/2866016519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0e8105e60 0x7fc0e8108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.107+0000 7fc0f071b700 1 -- 172.21.15.16:0/2866016519 >> 172.21.15.16:0/2866016519 conn(0x7fc0e80fcf40 msgr2=0x7fc0e80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.107+0000 7fc0f071b700 1 -- 172.21.15.16:0/2866016519 shutdown_connections 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.107+0000 7fc0f071b700 1 -- 172.21.15.16:0/2866016519 wait complete. 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0f071b700 1 Processor -- start 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0f071b700 1 -- start start 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0f071b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0e8103530 0x7fc0e8102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0f071b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e8105e60 0x7fc0e81014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0f071b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e8108790 0x7fc0e8101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0f071b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e810e580 con 0x7fc0e8103530 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0f071b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e810e400 con 0x7fc0e8105e60 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0f071b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e810e700 con 0x7fc0e8108790 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0edcb6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e8105e60 0x7fc0e81014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0edcb6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e8105e60 0x7fc0e81014e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42712/0 (socket says 172.21.15.16:42712) 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0edcb6700 1 -- 172.21.15.16:0/1597882295 learned_addr learned my addr 172.21.15.16:0/1597882295 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0ee4b7700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0e8103530 0x7fc0e8102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0eecb8700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e8108790 0x7fc0e8101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0edcb6700 1 -- 172.21.15.16:0/1597882295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e8108790 msgr2=0x7fc0e8101a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0edcb6700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e8108790 0x7fc0e8101a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0edcb6700 1 -- 172.21.15.16:0/1597882295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0e8103530 msgr2=0x7fc0e8102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:59.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0edcb6700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0e8103530 0x7fc0e8102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.108+0000 7fc0edcb6700 1 -- 172.21.15.16:0/1597882295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e81022d0 con 0x7fc0e8105e60 2024-04-21T08:37:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.109+0000 7fc0eecb8700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e8108790 0x7fc0e8101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:37:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.109+0000 7fc0edcb6700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e8105e60 0x7fc0e81014e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400d980 tx=0x7fc0e400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.109+0000 7fc0df7fe700 1 -- 172.21.15.16:0/1597882295 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4016070 con 0x7fc0e8105e60 2024-04-21T08:37:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.109+0000 7fc0df7fe700 1 -- 172.21.15.16:0/1597882295 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0e4004830 con 0x7fc0e8105e60 2024-04-21T08:37:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.109+0000 7fc0df7fe700 1 -- 172.21.15.16:0/1597882295 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e40054c0 con 0x7fc0e8105e60 2024-04-21T08:37:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.109+0000 7fc0f071b700 1 -- 172.21.15.16:0/1597882295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e81a2de0 con 0x7fc0e8105e60 2024-04-21T08:37:59.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.109+0000 7fc0f071b700 1 -- 172.21.15.16:0/1597882295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e81a3290 con 0x7fc0e8105e60 2024-04-21T08:37:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.110+0000 7fc0f071b700 1 -- 172.21.15.16:0/1597882295 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e8069730 con 0x7fc0e8105e60 2024-04-21T08:37:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.111+0000 7fc0df7fe700 1 -- 172.21.15.16:0/1597882295 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc0e4004220 con 0x7fc0e8105e60 2024-04-21T08:37:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.112+0000 7fc0df7fe700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc0d40a3cc0 0x7fc0d40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.112+0000 7fc0df7fe700 1 -- 172.21.15.16:0/1597882295 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7fc0e40cf000 con 0x7fc0e8105e60 2024-04-21T08:37:59.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.112+0000 7fc0ee4b7700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc0d40a3cc0 0x7fc0d40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:59.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.112+0000 7fc0ee4b7700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc0d40a3cc0 0x7fc0d40a6180 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8009a10 tx=0x7fc0d8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.117+0000 7fc0df7fe700 1 -- 172.21.15.16:0/1597882295 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc0e401f020 con 0x7fc0e8105e60 2024-04-21T08:37:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:58 smithi016 ceph-mon[31259]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:37:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:37:58 smithi016 ceph-mon[31259]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:37:59.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:37:59.336+0000 7fc0f071b700 1 -- 172.21.15.16:0/1597882295 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fc0e8062e80 con 0x7fc0d40a3cc0 2024-04-21T08:37:59.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:58 smithi077 ceph-mon[31615]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:37:59.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:37:58 smithi077 ceph-mon[31615]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:37:59.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:58 smithi007 ceph-mon[28477]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:37:59.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:37:58 smithi007 ceph-mon[28477]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:38:00.248 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:00 smithi016 ceph-mon[31259]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:38:00.248 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:00 smithi016 ceph-mon[31259]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:00.248 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:00 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:00.248 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:00 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:00.248 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:00 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:00.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:00 smithi077 ceph-mon[31615]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:38:00.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:00 smithi077 ceph-mon[31615]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:00.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:00 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:00.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:00 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:00.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:00 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:00.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:00 smithi007 ceph-mon[28477]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:38:00.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:00 smithi007 ceph-mon[28477]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:00.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:00 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:00.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:00 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:00.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:00 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:01.283 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:01 smithi016 ceph-mon[31259]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-04-21T08:38:01.283 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:01 smithi016 ceph-mon[31259]: osdmap e30: 4 total, 4 up, 4 in 2024-04-21T08:38:01.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:01 smithi077 ceph-mon[31615]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-04-21T08:38:01.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:01 smithi077 ceph-mon[31615]: osdmap e30: 4 total, 4 up, 4 in 2024-04-21T08:38:01.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:01 smithi007 ceph-mon[28477]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-04-21T08:38:01.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:01 smithi007 ceph-mon[28477]: osdmap e30: 4 total, 4 up, 4 in 2024-04-21T08:38:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:03 smithi016 ceph-mon[31259]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-21T08:38:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:03 smithi016 ceph-mon[31259]: from='client.? 172.21.15.16:0/903666693' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb"}]: dispatch 2024-04-21T08:38:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:03 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb"}]: dispatch 2024-04-21T08:38:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:03 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb"}]': finished 2024-04-21T08:38:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:03 smithi016 ceph-mon[31259]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:38:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:03 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:03 smithi016 ceph-mon[31259]: from='client.? 172.21.15.16:0/3635082832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:03 smithi077 ceph-mon[31615]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-21T08:38:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:03 smithi077 ceph-mon[31615]: from='client.? 172.21.15.16:0/903666693' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb"}]: dispatch 2024-04-21T08:38:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:03 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb"}]: dispatch 2024-04-21T08:38:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:03 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb"}]': finished 2024-04-21T08:38:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:03 smithi077 ceph-mon[31615]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:38:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:03 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:03 smithi077 ceph-mon[31615]: from='client.? 172.21.15.16:0/3635082832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:03.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:03 smithi007 ceph-mon[28477]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-21T08:38:03.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:03 smithi007 ceph-mon[28477]: from='client.? 172.21.15.16:0/903666693' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb"}]: dispatch 2024-04-21T08:38:03.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:03 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb"}]: dispatch 2024-04-21T08:38:03.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:03 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb"}]': finished 2024-04-21T08:38:03.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:03 smithi007 ceph-mon[28477]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:38:03.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:03 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:03.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:03 smithi007 ceph-mon[28477]: from='client.? 172.21.15.16:0/3635082832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:05.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:05 smithi016 ceph-mon[31259]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-21T08:38:05.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:05 smithi077 ceph-mon[31615]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-21T08:38:05.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:05 smithi007 ceph-mon[28477]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-21T08:38:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:07 smithi016 ceph-mon[31259]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:07.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:07 smithi077 ceph-mon[31615]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:07 smithi007 ceph-mon[28477]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:09.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:09 smithi016 ceph-mon[31259]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:09.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:09 smithi077 ceph-mon[31615]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:09.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:09 smithi007 ceph-mon[28477]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:10.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:38:10.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:10.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:38:10.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:10.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:38:10.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:11 smithi016 ceph-mon[31259]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:11 smithi016 ceph-mon[31259]: Deploying daemon osd.4 on smithi016 2024-04-21T08:38:11.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:11 smithi077 ceph-mon[31615]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:11 smithi077 ceph-mon[31615]: Deploying daemon osd.4 on smithi016 2024-04-21T08:38:11.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:11 smithi007 ceph-mon[28477]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:11.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:11 smithi007 ceph-mon[28477]: Deploying daemon osd.4 on smithi016 2024-04-21T08:38:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:13 smithi016 ceph-mon[31259]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:13.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:13 smithi077 ceph-mon[31615]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:13.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:13 smithi007 ceph-mon[28477]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:15 smithi016 ceph-mon[31259]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:15.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:15 smithi077 ceph-mon[31615]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:15.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:15 smithi007 ceph-mon[28477]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:16.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:16.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:16.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:16.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:16.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:16.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:16.680 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:16.680 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:16.680 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:17.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:17 smithi016 ceph-mon[31259]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:17.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:17 smithi077 ceph-mon[31615]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:17.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:17 smithi007 ceph-mon[28477]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:19.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.105+0000 7fc0df7fe700 1 -- 172.21.15.16:0/1597882295 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc0e8062e80 con 0x7fc0d40a3cc0 2024-04-21T08:38:19.105 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 4 on host 'smithi016' 2024-04-21T08:38:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.107+0000 7fc0f071b700 1 -- 172.21.15.16:0/1597882295 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc0d40a3cc0 msgr2=0x7fc0d40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.107+0000 7fc0f071b700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc0d40a3cc0 0x7fc0d40a6180 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8009a10 tx=0x7fc0d8009190 comp rx=0 tx=0).stop 2024-04-21T08:38:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.108+0000 7fc0f071b700 1 -- 172.21.15.16:0/1597882295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e8105e60 msgr2=0x7fc0e81014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.108+0000 7fc0f071b700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e8105e60 0x7fc0e81014e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400d980 tx=0x7fc0e400dc90 comp rx=0 tx=0).stop 2024-04-21T08:38:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.108+0000 7fc0f071b700 1 -- 172.21.15.16:0/1597882295 shutdown_connections 2024-04-21T08:38:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.108+0000 7fc0f071b700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc0d40a3cc0 0x7fc0d40a6180 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.108+0000 7fc0f071b700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc0e8108790 0x7fc0e8101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:19.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.108+0000 7fc0f071b700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0e8103530 0x7fc0e8102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:19.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.108+0000 7fc0f071b700 1 --2- 172.21.15.16:0/1597882295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0e8105e60 0x7fc0e81014e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:19.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.108+0000 7fc0f071b700 1 -- 172.21.15.16:0/1597882295 >> 172.21.15.16:0/1597882295 conn(0x7fc0e80fcf40 msgr2=0x7fc0e8103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:19.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.108+0000 7fc0f071b700 1 -- 172.21.15.16:0/1597882295 shutdown_connections 2024-04-21T08:38:19.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:38:19.108+0000 7fc0f071b700 1 -- 172.21.15.16:0/1597882295 wait complete. 2024-04-21T08:38:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:18 smithi016 ceph-mon[31259]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:19.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:18 smithi077 ceph-mon[31615]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:19.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:19.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:19.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:19.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:19.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:19.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:18 smithi007 ceph-mon[28477]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:19.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:19.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:19.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:19.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:19.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:19.925 DEBUG:teuthology.orchestra.run.smithi016:osd.4> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.4.service 2024-04-21T08:38:19.927 INFO:tasks.cephadm:Deploying osd.5 on smithi077 with /dev/vg_nvme/lv_3... 2024-04-21T08:38:19.927 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:38:20.155 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:20.155 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:20.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:20 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:20.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:20 smithi016 ceph-mon[31259]: from='osd.4 [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:20.156 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:20 smithi016 ceph-mon[31259]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:20.170 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:38:20.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:20.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:20.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:20 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:20.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:20 smithi077 ceph-mon[31615]: from='osd.4 [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:20.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:20 smithi077 ceph-mon[31615]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:20.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:20.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:20.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:20 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:20.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:20 smithi007 ceph-mon[28477]: from='osd.4 [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:20.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:20 smithi007 ceph-mon[28477]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:21.338 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:21 smithi016 ceph-mon[31259]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:21.338 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:21 smithi016 ceph-mon[31259]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:38:21.338 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:21 smithi016 ceph-mon[31259]: from='osd.4 [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:38:21.338 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:21 smithi016 ceph-mon[31259]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:38:21.338 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:21 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:21.338 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:21 smithi016 ceph-mon[31259]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:38:21.352 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:21 smithi077 ceph-mon[31615]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:21.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:21 smithi077 ceph-mon[31615]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:38:21.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:21 smithi077 ceph-mon[31615]: from='osd.4 [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:38:21.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:21 smithi077 ceph-mon[31615]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:38:21.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:21 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:21.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:21 smithi077 ceph-mon[31615]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:38:21.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:21 smithi007 ceph-mon[28477]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:21.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:21 smithi007 ceph-mon[28477]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:38:21.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:21 smithi007 ceph-mon[28477]: from='osd.4 [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:38:21.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:21 smithi007 ceph-mon[28477]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:38:21.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:21 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:21.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:21 smithi007 ceph-mon[28477]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:38:21.641 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:38:21 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4[42366]: 2024-04-21T08:38:21.527+0000 7f0f9afbf700 -1 osd.4 0 waiting for initial osdmap 2024-04-21T08:38:21.641 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:38:21 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4[42366]: 2024-04-21T08:38:21.534+0000 7f0f92daa700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:38:22.088 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:38:22.088 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:38:22.088 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-21T08:38:22.088 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-21T08:38:22.088 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114512 s, 916 MB/s 2024-04-21T08:38:22.088 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-21T08:38:22.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:22 smithi077 ceph-mon[31615]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:38:22.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:22 smithi077 ceph-mon[31615]: osdmap e33: 5 total, 4 up, 5 in 2024-04-21T08:38:22.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:22.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:22.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:22 smithi077 ceph-mon[31615]: from='osd.4 ' entity='osd.4' 2024-04-21T08:38:22.397 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:22 smithi016 ceph-mon[31259]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:38:22.398 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:22 smithi016 ceph-mon[31259]: osdmap e33: 5 total, 4 up, 5 in 2024-04-21T08:38:22.398 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:22.398 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:22.398 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:22 smithi016 ceph-mon[31259]: from='osd.4 ' entity='osd.4' 2024-04-21T08:38:22.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:22 smithi007 ceph-mon[28477]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:38:22.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:22 smithi007 ceph-mon[28477]: osdmap e33: 5 total, 4 up, 5 in 2024-04-21T08:38:22.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:22.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:22.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:22 smithi007 ceph-mon[28477]: from='osd.4 ' entity='osd.4' 2024-04-21T08:38:22.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_3 2024-04-21T08:38:22.968 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:38:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:38:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:38:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: osd.4 [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848] boot 2024-04-21T08:38:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:38:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:23.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:23.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:23.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:23.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:23.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:23.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:23.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:23.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:38:23.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:38:23.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:23.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:23.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: osd.4 [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848] boot 2024-04-21T08:38:23.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:38:23.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:23.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:23.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:23.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:23.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:23.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:23.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:23.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:23.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:23.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:38:23.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:38:23.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:23.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:23.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: osd.4 [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848] boot 2024-04-21T08:38:23.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:38:23.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:23.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:23.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:23.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:23.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:23.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:23.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:23.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:23.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:24.309 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:24 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi016 2024-04-21T08:38:24.310 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:24 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi016 to 8546M 2024-04-21T08:38:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:24 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi016 2024-04-21T08:38:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:24 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi016 to 8546M 2024-04-21T08:38:24.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.504+0000 7fab6d395700 1 -- 172.21.15.77:0/3390942983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab68103df0 msgr2=0x7fab68104250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:24.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.504+0000 7fab6d395700 1 --2- 172.21.15.77:0/3390942983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab68103df0 0x7fab68104250 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fab50009990 tx=0x7fab50009ca0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.504+0000 7fab6d395700 1 -- 172.21.15.77:0/3390942983 shutdown_connections 2024-04-21T08:38:24.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.504+0000 7fab6d395700 1 --2- 172.21.15.77:0/3390942983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab68104790 0x7fab6810b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.504+0000 7fab6d395700 1 --2- 172.21.15.77:0/3390942983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab68100d70 0x7fab68101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.504+0000 7fab6d395700 1 --2- 172.21.15.77:0/3390942983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab68103df0 0x7fab68104250 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.504+0000 7fab6d395700 1 -- 172.21.15.77:0/3390942983 >> 172.21.15.77:0/3390942983 conn(0x7fab680fc6d0 msgr2=0x7fab680feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:24.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.504+0000 7fab6d395700 1 -- 172.21.15.77:0/3390942983 shutdown_connections 2024-04-21T08:38:24.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.504+0000 7fab6d395700 1 -- 172.21.15.77:0/3390942983 wait complete. 2024-04-21T08:38:24.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.504+0000 7fab6d395700 1 Processor -- start 2024-04-21T08:38:24.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab6d395700 1 -- start start 2024-04-21T08:38:24.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab6d395700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab68100d70 0x7fab68103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:24.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab6d395700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab68103df0 0x7fab68101da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:24.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab6d395700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab68104790 0x7fab681022e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:24.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab6d395700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fab6810e520 con 0x7fab68103df0 2024-04-21T08:38:24.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab6d395700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fab6810e3a0 con 0x7fab68100d70 2024-04-21T08:38:24.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab6d395700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fab6810e6a0 con 0x7fab68104790 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab677fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab68104790 0x7fab681022e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab677fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab68104790 0x7fab681022e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52952/0 (socket says 172.21.15.77:52952) 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab677fe700 1 -- 172.21.15.77:0/27815065 learned_addr learned my addr 172.21.15.77:0/27815065 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab667fc700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab68103df0 0x7fab68101da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab66ffd700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab68100d70 0x7fab68103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab677fe700 1 -- 172.21.15.77:0/27815065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab68100d70 msgr2=0x7fab68103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab677fe700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab68100d70 0x7fab68103750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab677fe700 1 -- 172.21.15.77:0/27815065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab68103df0 msgr2=0x7fab68101da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab677fe700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab68103df0 0x7fab68101da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab677fe700 1 -- 172.21.15.77:0/27815065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab68102b90 con 0x7fab68104790 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.505+0000 7fab677fe700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab68104790 0x7fab681022e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fab6000d980 tx=0x7fab6000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.506+0000 7fab5ffff700 1 -- 172.21.15.77:0/27815065 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60016070 con 0x7fab68104790 2024-04-21T08:38:24.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.506+0000 7fab5ffff700 1 -- 172.21.15.77:0/27815065 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab60004830 con 0x7fab68104790 2024-04-21T08:38:24.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.506+0000 7fab6d395700 1 -- 172.21.15.77:0/27815065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab681a2d50 con 0x7fab68104790 2024-04-21T08:38:24.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.506+0000 7fab5ffff700 1 -- 172.21.15.77:0/27815065 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab600054b0 con 0x7fab68104790 2024-04-21T08:38:24.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.506+0000 7fab6d395700 1 -- 172.21.15.77:0/27815065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab681a3200 con 0x7fab68104790 2024-04-21T08:38:24.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.507+0000 7fab6d395700 1 -- 172.21.15.77:0/27815065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab68069730 con 0x7fab68104790 2024-04-21T08:38:24.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.508+0000 7fab5ffff700 1 -- 172.21.15.77:0/27815065 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fab600049e0 con 0x7fab68104790 2024-04-21T08:38:24.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.508+0000 7fab5ffff700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fab540a3cc0 0x7fab540a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:24.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.509+0000 7fab5ffff700 1 -- 172.21.15.77:0/27815065 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7fab600cf1a0 con 0x7fab68104790 2024-04-21T08:38:24.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.509+0000 7fab66ffd700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fab540a3cc0 0x7fab540a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:24.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.509+0000 7fab66ffd700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fab540a3cc0 0x7fab540a6180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fab6806a460 tx=0x7fab58009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:24.513 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.513+0000 7fab5ffff700 1 -- 172.21.15.77:0/27815065 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fab6001f020 con 0x7fab68104790 2024-04-21T08:38:24.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:24 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi016 2024-04-21T08:38:24.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:24 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi016 to 8546M 2024-04-21T08:38:24.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:24.720+0000 7fab6d395700 1 -- 172.21.15.77:0/27815065 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fab68062e80 con 0x7fab540a3cc0 2024-04-21T08:38:25.326 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:25 smithi077 ceph-mon[31615]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:25.326 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:25 smithi077 ceph-mon[31615]: osdmap e35: 5 total, 5 up, 5 in 2024-04-21T08:38:25.326 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:25 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:25.326 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:25 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:25.326 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:25 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:25.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:25 smithi007 ceph-mon[28477]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:25.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:25 smithi007 ceph-mon[28477]: osdmap e35: 5 total, 5 up, 5 in 2024-04-21T08:38:25.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:25 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:25.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:25 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:25.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:25 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:25 smithi016 ceph-mon[31259]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:25 smithi016 ceph-mon[31259]: osdmap e35: 5 total, 5 up, 5 in 2024-04-21T08:38:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:25 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:25 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:25 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:26.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:26 smithi077 ceph-mon[31615]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:26.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:26 smithi007 ceph-mon[28477]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:26.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:26 smithi016 ceph-mon[31259]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:27.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:27 smithi077 ceph-mon[31615]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:27.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:27 smithi007 ceph-mon[28477]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:27.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:27 smithi016 ceph-mon[31259]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:28.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:28 smithi077 ceph-mon[31615]: from='client.? 172.21.15.77:0/268236504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e0c8bb-dee9-4c3b-83f7-fd536ebdee21"}]: dispatch 2024-04-21T08:38:28.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:28 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e0c8bb-dee9-4c3b-83f7-fd536ebdee21"}]: dispatch 2024-04-21T08:38:28.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:28 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93e0c8bb-dee9-4c3b-83f7-fd536ebdee21"}]': finished 2024-04-21T08:38:28.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:28 smithi077 ceph-mon[31615]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:38:28.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:28 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:28.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:28 smithi007 ceph-mon[28477]: from='client.? 172.21.15.77:0/268236504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e0c8bb-dee9-4c3b-83f7-fd536ebdee21"}]: dispatch 2024-04-21T08:38:28.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:28 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e0c8bb-dee9-4c3b-83f7-fd536ebdee21"}]: dispatch 2024-04-21T08:38:28.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:28 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93e0c8bb-dee9-4c3b-83f7-fd536ebdee21"}]': finished 2024-04-21T08:38:28.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:28 smithi007 ceph-mon[28477]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:38:28.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:28 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:28.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:28 smithi016 ceph-mon[31259]: from='client.? 172.21.15.77:0/268236504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e0c8bb-dee9-4c3b-83f7-fd536ebdee21"}]: dispatch 2024-04-21T08:38:28.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:28 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e0c8bb-dee9-4c3b-83f7-fd536ebdee21"}]: dispatch 2024-04-21T08:38:28.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:28 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93e0c8bb-dee9-4c3b-83f7-fd536ebdee21"}]': finished 2024-04-21T08:38:28.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:28 smithi016 ceph-mon[31259]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:38:28.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:28 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:29.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:29 smithi077 ceph-mon[31615]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:29.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:29 smithi077 ceph-mon[31615]: from='client.? 172.21.15.77:0/1535346446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:29.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:29 smithi007 ceph-mon[28477]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:29.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:29 smithi007 ceph-mon[28477]: from='client.? 172.21.15.77:0/1535346446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:29.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:29 smithi016 ceph-mon[31259]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:29.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:29 smithi016 ceph-mon[31259]: from='client.? 172.21.15.77:0/1535346446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:31.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:31 smithi077 ceph-mon[31615]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:31.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:31 smithi007 ceph-mon[28477]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:31.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:31 smithi016 ceph-mon[31259]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:33.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:33 smithi077 ceph-mon[31615]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:33.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:33 smithi007 ceph-mon[28477]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:33.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:33 smithi016 ceph-mon[31259]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:35.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:35 smithi077 ceph-mon[31615]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:35.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:35 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:38:35.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:35 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:35.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:35 smithi007 ceph-mon[28477]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:35.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:35 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:38:35.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:35 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:35.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:35 smithi016 ceph-mon[31259]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:35.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:35 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:38:35.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:35 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:36.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:36 smithi077 ceph-mon[31615]: Deploying daemon osd.5 on smithi077 2024-04-21T08:38:36.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:36 smithi007 ceph-mon[28477]: Deploying daemon osd.5 on smithi077 2024-04-21T08:38:36.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:36 smithi016 ceph-mon[31259]: Deploying daemon osd.5 on smithi077 2024-04-21T08:38:37.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:37 smithi077 ceph-mon[31615]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:37.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:37 smithi007 ceph-mon[28477]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:37.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:37 smithi016 ceph-mon[31259]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:39.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:39 smithi077 ceph-mon[31615]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:39.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:39 smithi007 ceph-mon[28477]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:39.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:39 smithi016 ceph-mon[31259]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:41.254 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:41 smithi077 ceph-mon[31615]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:41.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:41 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:41.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:41 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:41.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:41 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:41.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:41 smithi007 ceph-mon[28477]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:41.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:41 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:41.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:41 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:41.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:41 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:41.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:41 smithi016 ceph-mon[31259]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:41.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:41 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:41.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:41 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:41.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:41 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:43.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:43 smithi077 ceph-mon[31615]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:43.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:43 smithi007 ceph-mon[28477]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:43.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:43 smithi016 ceph-mon[31259]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:44.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.643+0000 7fab5ffff700 1 -- 172.21.15.77:0/27815065 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fab68062e80 con 0x7fab540a3cc0 2024-04-21T08:38:44.643 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 5 on host 'smithi077' 2024-04-21T08:38:44.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.645+0000 7fab6d395700 1 -- 172.21.15.77:0/27815065 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fab540a3cc0 msgr2=0x7fab540a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:44.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.645+0000 7fab6d395700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fab540a3cc0 0x7fab540a6180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fab6806a460 tx=0x7fab58009040 comp rx=0 tx=0).stop 2024-04-21T08:38:44.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.645+0000 7fab6d395700 1 -- 172.21.15.77:0/27815065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab68104790 msgr2=0x7fab681022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:44.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.645+0000 7fab6d395700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab68104790 0x7fab681022e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fab6000d980 tx=0x7fab6000dc90 comp rx=0 tx=0).stop 2024-04-21T08:38:44.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.645+0000 7fab6d395700 1 -- 172.21.15.77:0/27815065 shutdown_connections 2024-04-21T08:38:44.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.645+0000 7fab6d395700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fab540a3cc0 0x7fab540a6180 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:44.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.645+0000 7fab6d395700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab68104790 0x7fab681022e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:44.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.645+0000 7fab6d395700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab68100d70 0x7fab68103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:44.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.645+0000 7fab6d395700 1 --2- 172.21.15.77:0/27815065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab68103df0 0x7fab68101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:44.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.645+0000 7fab6d395700 1 -- 172.21.15.77:0/27815065 >> 172.21.15.77:0/27815065 conn(0x7fab680fc6d0 msgr2=0x7fab68107170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:44.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.646+0000 7fab6d395700 1 -- 172.21.15.77:0/27815065 shutdown_connections 2024-04-21T08:38:44.646 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:38:44.646+0000 7fab6d395700 1 -- 172.21.15.77:0/27815065 wait complete. 2024-04-21T08:38:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:44 smithi016 ceph-mon[31259]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:44 smithi016 ceph-mon[31259]: from='osd.5 [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:38:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:44 smithi016 ceph-mon[31259]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:38:44.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:44.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:44.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:44.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:44.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:44.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:44 smithi077 ceph-mon[31615]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:44.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:44 smithi077 ceph-mon[31615]: from='osd.5 [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:38:44.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:44 smithi077 ceph-mon[31615]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:38:44.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:44.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:44.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:44.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:44.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:44.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:44 smithi007 ceph-mon[28477]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:44.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:44 smithi007 ceph-mon[28477]: from='osd.5 [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:38:44.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:44 smithi007 ceph-mon[28477]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:38:45.329 DEBUG:teuthology.orchestra.run.smithi077:osd.5> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.5.service 2024-04-21T08:38:45.331 INFO:tasks.cephadm:Deploying osd.6 on smithi007 with /dev/vg_nvme/lv_2... 2024-04-21T08:38:45.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:38:45.590 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:45.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:45 smithi007 ceph-mon[28477]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:38:45.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:45 smithi007 ceph-mon[28477]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:38:45.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:45 smithi007 ceph-mon[28477]: from='osd.5 [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:38:45.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:45 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:45.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:45 smithi007 ceph-mon[28477]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:38:45.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:45 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:45.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:45 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:45.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:45 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:45.839 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:45 smithi077 ceph-mon[31615]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:38:45.839 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:45 smithi077 ceph-mon[31615]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:38:45.839 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:45 smithi077 ceph-mon[31615]: from='osd.5 [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:38:45.839 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:45 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:45.839 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:45 smithi077 ceph-mon[31615]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:38:45.839 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:45 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:45.839 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:45 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:45.839 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:45 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:45.840 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:38:45 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:38:45.568+0000 7ff32adf5700 -1 osd.5 0 waiting for initial osdmap 2024-04-21T08:38:45.840 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:38:45 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:38:45.575+0000 7ff326be8700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:38:46.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:45 smithi016 ceph-mon[31259]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:38:46.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:45 smithi016 ceph-mon[31259]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:38:46.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:45 smithi016 ceph-mon[31259]: from='osd.5 [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:38:46.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:45 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:46.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:45 smithi016 ceph-mon[31259]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:38:46.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:45 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:46.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:45 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:46.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:45 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:46 smithi077 ceph-mon[31615]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:38:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:46 smithi077 ceph-mon[31615]: osdmap e38: 6 total, 5 up, 6 in 2024-04-21T08:38:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:46 smithi077 ceph-mon[31615]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:46 smithi077 ceph-mon[31615]: osd.5 [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994] boot 2024-04-21T08:38:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:46 smithi077 ceph-mon[31615]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:38:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:46.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:46 smithi007 ceph-mon[28477]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:38:46.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:46 smithi007 ceph-mon[28477]: osdmap e38: 6 total, 5 up, 6 in 2024-04-21T08:38:46.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:46.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:46.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:46 smithi007 ceph-mon[28477]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:46.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:46 smithi007 ceph-mon[28477]: osd.5 [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994] boot 2024-04-21T08:38:46.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:46 smithi007 ceph-mon[28477]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:38:46.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:46 smithi016 ceph-mon[31259]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:38:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:46 smithi016 ceph-mon[31259]: osdmap e38: 6 total, 5 up, 6 in 2024-04-21T08:38:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:46 smithi016 ceph-mon[31259]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:38:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:46 smithi016 ceph-mon[31259]: osd.5 [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994] boot 2024-04-21T08:38:47.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:46 smithi016 ceph-mon[31259]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:38:47.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:38:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:38:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:38:47.662 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-21T08:38:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-21T08:38:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00825678 s, 1.3 GB/s 2024-04-21T08:38:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-21T08:38:48.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_2 2024-04-21T08:38:48.266 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:47 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:38:48.266 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:47 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:38:48.266 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:47 smithi007 ceph-mon[28477]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:38:48.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:47 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:38:48.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:47 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:38:48.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:47 smithi016 ceph-mon[31259]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:38:48.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:48 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:38:48.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:48 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:38:48.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:48 smithi077 ceph-mon[31615]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:38:48.487 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:49.130 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:48 smithi007 ceph-mon[28477]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:38:49.130 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:48 smithi007 ceph-mon[28477]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:38:49.130 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.130 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.130 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:49.130 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:49.130 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.130 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:49.130 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:49.130 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:49 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:49 smithi016 ceph-mon[31259]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:38:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:49 smithi016 ceph-mon[31259]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:38:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:49 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:49 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:49 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:49 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:49 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:49 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:49.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:49 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:49.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:49 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:49 smithi077 ceph-mon[31615]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:38:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:49 smithi077 ceph-mon[31615]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:38:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:49 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:49 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:49 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:49 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:49 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:49 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:49 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:49 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:38:49.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.908+0000 7f88677f8700 1 -- 172.21.15.7:0/2250661540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8860104660 msgr2=0x7f886010af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:49.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.908+0000 7f88677f8700 1 --2- 172.21.15.7:0/2250661540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8860104660 0x7f886010af10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f885800b0d0 tx=0x7f885800b3e0 comp rx=0 tx=0).stop 2024-04-21T08:38:49.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.908+0000 7f88677f8700 1 -- 172.21.15.7:0/2250661540 shutdown_connections 2024-04-21T08:38:49.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.908+0000 7f88677f8700 1 --2- 172.21.15.7:0/2250661540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8860102ac0 0x7f8860102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:49.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.908+0000 7f88677f8700 1 --2- 172.21.15.7:0/2250661540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8860104660 0x7f886010af10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.908+0000 7f88677f8700 1 --2- 172.21.15.7:0/2250661540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8860103cc0 0x7f8860104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.908+0000 7f88677f8700 1 -- 172.21.15.7:0/2250661540 >> 172.21.15.7:0/2250661540 conn(0x7f88600fe060 msgr2=0x7f88601004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.908+0000 7f88677f8700 1 -- 172.21.15.7:0/2250661540 shutdown_connections 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.908+0000 7f88677f8700 1 -- 172.21.15.7:0/2250661540 wait complete. 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f88677f8700 1 Processor -- start 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f88677f8700 1 -- start start 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f88677f8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8860102ac0 0x7f8860198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f88677f8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8860103cc0 0x7f8860199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f88677f8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8860104660 0x7f88601a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f88677f8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f886010e540 con 0x7f8860103cc0 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f88677f8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f886010e3c0 con 0x7f8860104660 2024-04-21T08:38:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f88677f8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f886010e6c0 con 0x7f8860102ac0 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f8864d93700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8860103cc0 0x7f8860199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f8864d93700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8860103cc0 0x7f8860199070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50722/0 (socket says 172.21.15.7:50722) 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f8864d93700 1 -- 172.21.15.7:0/1706030619 learned_addr learned my addr 172.21.15.7:0/1706030619 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.909+0000 7f8865594700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8860102ac0 0x7f8860198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f8865d95700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8860104660 0x7f88601a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f8864d93700 1 -- 172.21.15.7:0/1706030619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8860102ac0 msgr2=0x7f8860198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f8864d93700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8860102ac0 0x7f8860198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f8864d93700 1 -- 172.21.15.7:0/1706030619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8860104660 msgr2=0x7f88601a00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f8864d93700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8860104660 0x7f88601a00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f8864d93700 1 -- 172.21.15.7:0/1706030619 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88601a0790 con 0x7f8860103cc0 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f8864d93700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8860103cc0 0x7f8860199070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f885000e9d0 tx=0x7f885000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f88567fc700 1 -- 172.21.15.7:0/1706030619 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885000cbd0 con 0x7f8860103cc0 2024-04-21T08:38:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f88677f8700 1 -- 172.21.15.7:0/1706030619 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88601a0a80 con 0x7f8860103cc0 2024-04-21T08:38:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f88567fc700 1 -- 172.21.15.7:0/1706030619 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f885000cd70 con 0x7f8860103cc0 2024-04-21T08:38:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f88567fc700 1 -- 172.21.15.7:0/1706030619 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850018680 con 0x7f8860103cc0 2024-04-21T08:38:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.910+0000 7f88677f8700 1 -- 172.21.15.7:0/1706030619 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88601a1010 con 0x7f8860103cc0 2024-04-21T08:38:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.911+0000 7f88677f8700 1 -- 172.21.15.7:0/1706030619 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8860069730 con 0x7f8860103cc0 2024-04-21T08:38:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.912+0000 7f88567fc700 1 -- 172.21.15.7:0/1706030619 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f8850010430 con 0x7f8860103cc0 2024-04-21T08:38:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.913+0000 7f88567fc700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f884c0a3c40 0x7f884c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.913+0000 7f8865594700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f884c0a3c40 0x7f884c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.913+0000 7f88567fc700 1 -- 172.21.15.7:0/1706030619 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f88500cfc20 con 0x7f8860103cc0 2024-04-21T08:38:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.913+0000 7f8865594700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f884c0a3c40 0x7f884c0a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8860103b20 tx=0x7f885c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:49.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:49.917+0000 7f88567fc700 1 -- 172.21.15.7:0/1706030619 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f885009d4c0 con 0x7f8860103cc0 2024-04-21T08:38:50.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:38:50.139+0000 7f88677f8700 1 -- 172.21.15.7:0/1706030619 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8860062f40 con 0x7f884c0a3c40 2024-04-21T08:38:50.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:50 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi077 2024-04-21T08:38:50.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:50 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi077 to 8544M 2024-04-21T08:38:50.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:50 smithi016 ceph-mon[31259]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:38:50.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:50 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi077 2024-04-21T08:38:50.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:50 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi077 to 8544M 2024-04-21T08:38:50.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:50 smithi077 ceph-mon[31615]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:38:50.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:50 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi077 2024-04-21T08:38:50.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:50 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi077 to 8544M 2024-04-21T08:38:50.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:50 smithi007 ceph-mon[28477]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:38:51.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:51 smithi007 ceph-mon[28477]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:38:51.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:51 smithi007 ceph-mon[28477]: osdmap e43: 6 total, 6 up, 6 in 2024-04-21T08:38:51.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:51 smithi007 ceph-mon[28477]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:51.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:51 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:51.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:51 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:51.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:51 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:51 smithi016 ceph-mon[31259]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:38:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:51 smithi016 ceph-mon[31259]: osdmap e43: 6 total, 6 up, 6 in 2024-04-21T08:38:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:51 smithi016 ceph-mon[31259]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:51 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:51 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:51 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:51.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:51 smithi077 ceph-mon[31615]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:38:51.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:51 smithi077 ceph-mon[31615]: osdmap e43: 6 total, 6 up, 6 in 2024-04-21T08:38:51.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:51 smithi077 ceph-mon[31615]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:51.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:51 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:51.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:51 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:51.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:51 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:53.273 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:53 smithi007 ceph-mon[28477]: pgmap v144: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-21T08:38:53.274 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:53 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1131703038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1f722c8-d99f-47c0-8d9c-35d391a82058"}]: dispatch 2024-04-21T08:38:53.274 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:53 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1131703038' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1f722c8-d99f-47c0-8d9c-35d391a82058"}]': finished 2024-04-21T08:38:53.274 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:53 smithi007 ceph-mon[28477]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:38:53.274 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:53 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:38:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:53 smithi016 ceph-mon[31259]: pgmap v144: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-21T08:38:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:53 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1131703038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1f722c8-d99f-47c0-8d9c-35d391a82058"}]: dispatch 2024-04-21T08:38:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:53 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1131703038' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1f722c8-d99f-47c0-8d9c-35d391a82058"}]': finished 2024-04-21T08:38:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:53 smithi016 ceph-mon[31259]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:38:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:53 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:38:53.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:53 smithi077 ceph-mon[31615]: pgmap v144: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-21T08:38:53.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:53 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1131703038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1f722c8-d99f-47c0-8d9c-35d391a82058"}]: dispatch 2024-04-21T08:38:53.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:53 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1131703038' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1f722c8-d99f-47c0-8d9c-35d391a82058"}]': finished 2024-04-21T08:38:53.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:53 smithi077 ceph-mon[31615]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:38:53.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:53 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:38:54.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:54 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4030209018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:54.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:54 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4030209018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:54.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:54 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4030209018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:55.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:55 smithi016 ceph-mon[31259]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-21T08:38:55.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:55 smithi077 ceph-mon[31615]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-21T08:38:55.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:55 smithi007 ceph-mon[28477]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-21T08:38:57.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:57 smithi016 ceph-mon[31259]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:38:57.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:57 smithi007 ceph-mon[28477]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:38:57.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:57 smithi077 ceph-mon[31615]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:38:59.212 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:38:59 smithi007 ceph-mon[28477]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:38:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:38:59 smithi016 ceph-mon[31259]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:38:59.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:38:59 smithi077 ceph-mon[31615]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:01.299 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:01 smithi007 ceph-mon[28477]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:01.299 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:39:01.299 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:01.300 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:01 smithi007 ceph-mon[28477]: Deploying daemon osd.6 on smithi007 2024-04-21T08:39:01.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:01 smithi016 ceph-mon[31259]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:01.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:39:01.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:01.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:01 smithi016 ceph-mon[31259]: Deploying daemon osd.6 on smithi007 2024-04-21T08:39:01.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:01 smithi077 ceph-mon[31615]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:01.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:39:01.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:01.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:01 smithi077 ceph-mon[31615]: Deploying daemon osd.6 on smithi007 2024-04-21T08:39:03.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:03 smithi007 ceph-mon[28477]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:03 smithi016 ceph-mon[31259]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:03.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:03 smithi077 ceph-mon[31615]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:05.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:05 smithi007 ceph-mon[28477]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:05.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:05 smithi016 ceph-mon[31259]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:05.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:05 smithi077 ceph-mon[31615]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:07 smithi007 ceph-mon[28477]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:07.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:07.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:07.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:07 smithi016 ceph-mon[31259]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:07.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:07.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:07.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:07.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:07 smithi077 ceph-mon[31615]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:07.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:07.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:07.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:09.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:09 smithi007 ceph-mon[28477]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:09.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:09 smithi016 ceph-mon[31259]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:09.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:09.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:09.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:09.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:09.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:09.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:09 smithi077 ceph-mon[31615]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:09.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:09.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:09.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:09.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:09.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:10.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.110+0000 7f88567fc700 1 -- 172.21.15.7:0/1706030619 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8860062f40 con 0x7f884c0a3c40 2024-04-21T08:39:10.109 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 6 on host 'smithi007' 2024-04-21T08:39:10.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 -- 172.21.15.7:0/1706030619 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f884c0a3c40 msgr2=0x7f884c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f884c0a3c40 0x7f884c0a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8860103b20 tx=0x7f885c00b410 comp rx=0 tx=0).stop 2024-04-21T08:39:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 -- 172.21.15.7:0/1706030619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8860103cc0 msgr2=0x7f8860199070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8860103cc0 0x7f8860199070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f885000e9d0 tx=0x7f885000ece0 comp rx=0 tx=0).stop 2024-04-21T08:39:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 -- 172.21.15.7:0/1706030619 shutdown_connections 2024-04-21T08:39:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f884c0a3c40 0x7f884c0a6100 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8860102ac0 0x7f8860198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8860104660 0x7f88601a00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 --2- 172.21.15.7:0/1706030619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8860103cc0 0x7f8860199070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 -- 172.21.15.7:0/1706030619 >> 172.21.15.7:0/1706030619 conn(0x7f88600fe060 msgr2=0x7f8860106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 -- 172.21.15.7:0/1706030619 shutdown_connections 2024-04-21T08:39:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:39:10.112+0000 7f88677f8700 1 -- 172.21.15.7:0/1706030619 wait complete. 2024-04-21T08:39:10.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:10 smithi007 ceph-mon[28477]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:39:10.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:10.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:10.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:10.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:10 smithi016 ceph-mon[31259]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:39:10.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:10.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:10.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:10.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:10 smithi077 ceph-mon[31615]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:39:10.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:10.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:10.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:10.811 DEBUG:teuthology.orchestra.run.smithi007:osd.6> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.6.service 2024-04-21T08:39:10.854 INFO:tasks.cephadm:Deploying osd.7 on smithi016 with /dev/vg_nvme/lv_2... 2024-04-21T08:39:10.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:39:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:39:11.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:11 smithi007 ceph-mon[28477]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:11.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:11 smithi007 ceph-mon[28477]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:39:11.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:11 smithi007 ceph-mon[28477]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:39:11.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:11 smithi007 ceph-mon[28477]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:39:11.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:11 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:11.481 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:11 smithi016 ceph-mon[31259]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:11.481 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:11 smithi016 ceph-mon[31259]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:39:11.481 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:11 smithi016 ceph-mon[31259]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:39:11.481 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:11 smithi016 ceph-mon[31259]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:39:11.481 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:11 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:11.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:11 smithi077 ceph-mon[31615]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:11.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:11 smithi077 ceph-mon[31615]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:39:11.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:11 smithi077 ceph-mon[31615]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:39:11.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:11 smithi077 ceph-mon[31615]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:39:11.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:11 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:11.902 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:39:11 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:39:11.603+0000 7fa3495e8700 -1 osd.6 0 waiting for initial osdmap 2024-04-21T08:39:11.902 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:39:11 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:39:11.614+0000 7fa3413d3700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:39:12.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:12 smithi007 ceph-mon[28477]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:39:12.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:12 smithi007 ceph-mon[28477]: osdmap e46: 7 total, 6 up, 7 in 2024-04-21T08:39:12.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:12.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:12.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:12 smithi007 ceph-mon[28477]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' 2024-04-21T08:39:12.293 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:12 smithi016 ceph-mon[31259]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:39:12.293 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:12 smithi016 ceph-mon[31259]: osdmap e46: 7 total, 6 up, 7 in 2024-04-21T08:39:12.294 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:12 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:12.294 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:12 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:12.294 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:12 smithi016 ceph-mon[31259]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' 2024-04-21T08:39:12.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:12 smithi077 ceph-mon[31615]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:39:12.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:12 smithi077 ceph-mon[31615]: osdmap e46: 7 total, 6 up, 7 in 2024-04-21T08:39:12.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:12 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:12.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:12 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:12.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:12 smithi077 ceph-mon[31615]: from='osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687]' entity='osd.6' 2024-04-21T08:39:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:39:13.208 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:39:13.208 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-21T08:39:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-21T08:39:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00851771 s, 1.2 GB/s 2024-04-21T08:39:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-21T08:39:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:13 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:39:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:13 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:39:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:13 smithi007 ceph-mon[28477]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:13 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:13 smithi007 ceph-mon[28477]: osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687] boot 2024-04-21T08:39:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:13 smithi007 ceph-mon[28477]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:39:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:13 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:13.525 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:13 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:39:13.525 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:13 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:39:13.525 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:13 smithi016 ceph-mon[31259]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:13.526 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:13 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:13.526 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:13 smithi016 ceph-mon[31259]: osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687] boot 2024-04-21T08:39:13.526 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:13 smithi016 ceph-mon[31259]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:39:13.526 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:13 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:13.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:13 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:39:13.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:13 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:39:13.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:13 smithi077 ceph-mon[31615]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:13.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:13 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:13.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:13 smithi077 ceph-mon[31615]: osd.6 [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687] boot 2024-04-21T08:39:13.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:13 smithi077 ceph-mon[31615]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:39:13.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:13 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:13.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-04-21T08:39:14.081 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:39:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: osdmap e48: 7 total, 7 up, 7 in 2024-04-21T08:39:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi007 2024-04-21T08:39:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi007 to 5697M 2024-04-21T08:39:14.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:14.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:14.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:14.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:14.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: osdmap e48: 7 total, 7 up, 7 in 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi007 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi007 to 5697M 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:15.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: osdmap e48: 7 total, 7 up, 7 in 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi007 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi007 to 5697M 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:15.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.698+0000 7f6559497700 1 -- 172.21.15.16:0/2295848359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6554104660 msgr2=0x7f655410af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:15.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.698+0000 7f6559497700 1 --2- 172.21.15.16:0/2295848359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6554104660 0x7f655410af10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f65440099c0 tx=0x7f6544009cd0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.698+0000 7f6559497700 1 -- 172.21.15.16:0/2295848359 shutdown_connections 2024-04-21T08:39:15.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.698+0000 7f6559497700 1 --2- 172.21.15.16:0/2295848359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6554104660 0x7f655410af10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.698+0000 7f6559497700 1 --2- 172.21.15.16:0/2295848359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6554102ac0 0x7f6554102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.698+0000 7f6559497700 1 --2- 172.21.15.16:0/2295848359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6554103cc0 0x7f6554104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.698+0000 7f6559497700 1 -- 172.21.15.16:0/2295848359 >> 172.21.15.16:0/2295848359 conn(0x7f65540fe060 msgr2=0x7f65541004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:15.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.698+0000 7f6559497700 1 -- 172.21.15.16:0/2295848359 shutdown_connections 2024-04-21T08:39:15.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.698+0000 7f6559497700 1 -- 172.21.15.16:0/2295848359 wait complete. 2024-04-21T08:39:15.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f6559497700 1 Processor -- start 2024-04-21T08:39:15.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f6559497700 1 -- start start 2024-04-21T08:39:15.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f6559497700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6554102ac0 0x7f6554198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:15.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f6559497700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6554103cc0 0x7f65541990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:15.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f6559497700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6554104660 0x7f65541a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:15.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f6559497700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f655410e5b0 con 0x7f6554102ac0 2024-04-21T08:39:15.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f6559497700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f655410e430 con 0x7f6554103cc0 2024-04-21T08:39:15.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f6559497700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f655410e730 con 0x7f6554104660 2024-04-21T08:39:15.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f65527fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6554103cc0 0x7f65541990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:15.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f6552ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6554102ac0 0x7f6554198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:15.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f65527fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6554103cc0 0x7f65541990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33936/0 (socket says 172.21.15.16:33936) 2024-04-21T08:39:15.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f65527fc700 1 -- 172.21.15.16:0/1506405597 learned_addr learned my addr 172.21.15.16:0/1506405597 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-21T08:39:15.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.699+0000 7f65537fe700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6554104660 0x7f65541a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f65527fc700 1 -- 172.21.15.16:0/1506405597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6554104660 msgr2=0x7f65541a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f65527fc700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6554104660 0x7f65541a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f65527fc700 1 -- 172.21.15.16:0/1506405597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6554102ac0 msgr2=0x7f6554198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f65527fc700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6554102ac0 0x7f6554198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f65527fc700 1 -- 172.21.15.16:0/1506405597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65541a0800 con 0x7f6554103cc0 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f65537fe700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6554104660 0x7f65541a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f6552ffd700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6554102ac0 0x7f6554198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f65527fc700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6554103cc0 0x7f65541990e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f653c00b530 tx=0x7f653c00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f654bfff700 1 -- 172.21.15.16:0/1506405597 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653c015020 con 0x7f6554103cc0 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f654bfff700 1 -- 172.21.15.16:0/1506405597 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f653c011ce0 con 0x7f6554103cc0 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f6559497700 1 -- 172.21.15.16:0/1506405597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65541a0af0 con 0x7f6554103cc0 2024-04-21T08:39:15.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f654bfff700 1 -- 172.21.15.16:0/1506405597 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653c00f2e0 con 0x7f6554103cc0 2024-04-21T08:39:15.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.700+0000 7f6559497700 1 -- 172.21.15.16:0/1506405597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65541a1080 con 0x7f6554103cc0 2024-04-21T08:39:15.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.701+0000 7f6559497700 1 -- 172.21.15.16:0/1506405597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6554069730 con 0x7f6554103cc0 2024-04-21T08:39:15.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.702+0000 7f654bfff700 1 -- 172.21.15.16:0/1506405597 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f653c018020 con 0x7f6554103cc0 2024-04-21T08:39:15.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.703+0000 7f654bfff700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f65400a3cc0 0x7f65400a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:15.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.703+0000 7f654bfff700 1 -- 172.21.15.16:0/1506405597 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f653c0cef00 con 0x7f6554103cc0 2024-04-21T08:39:15.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.703+0000 7f6552ffd700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f65400a3cc0 0x7f65400a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:15.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.703+0000 7f6552ffd700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f65400a3cc0 0x7f65400a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6554103b20 tx=0x7f654c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:15.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.707+0000 7f654bfff700 1 -- 172.21.15.16:0/1506405597 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f653c01f020 con 0x7f6554103cc0 2024-04-21T08:39:15.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:15.925+0000 7f6559497700 1 -- 172.21.15.16:0/1506405597 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6554062e80 con 0x7f65400a3cc0 2024-04-21T08:39:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:16 smithi016 ceph-mon[31259]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:16 smithi016 ceph-mon[31259]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:16.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:16 smithi007 ceph-mon[28477]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:16 smithi007 ceph-mon[28477]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:17.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:16 smithi077 ceph-mon[31615]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:17.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:16 smithi077 ceph-mon[31615]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:17.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:17.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:17.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:18.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:18 smithi007 ceph-mon[28477]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:19.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:18 smithi016 ceph-mon[31259]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:19.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:18 smithi077 ceph-mon[31615]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:19.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:19 smithi007 ceph-mon[28477]: from='client.? 172.21.15.16:0/1127912506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce43f302-448d-4896-8a13-3f52413a2252"}]: dispatch 2024-04-21T08:39:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:19 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce43f302-448d-4896-8a13-3f52413a2252"}]: dispatch 2024-04-21T08:39:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:19 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce43f302-448d-4896-8a13-3f52413a2252"}]': finished 2024-04-21T08:39:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:19 smithi007 ceph-mon[28477]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:39:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:19 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:19 smithi007 ceph-mon[28477]: from='client.? 172.21.15.16:0/3981122169' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:20.004 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:19 smithi077 ceph-mon[31615]: from='client.? 172.21.15.16:0/1127912506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce43f302-448d-4896-8a13-3f52413a2252"}]: dispatch 2024-04-21T08:39:20.004 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:19 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce43f302-448d-4896-8a13-3f52413a2252"}]: dispatch 2024-04-21T08:39:20.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:19 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce43f302-448d-4896-8a13-3f52413a2252"}]': finished 2024-04-21T08:39:20.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:19 smithi077 ceph-mon[31615]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:39:20.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:19 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:20.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:19 smithi077 ceph-mon[31615]: from='client.? 172.21.15.16:0/3981122169' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:20.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:19 smithi016 ceph-mon[31259]: from='client.? 172.21.15.16:0/1127912506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce43f302-448d-4896-8a13-3f52413a2252"}]: dispatch 2024-04-21T08:39:20.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:19 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce43f302-448d-4896-8a13-3f52413a2252"}]: dispatch 2024-04-21T08:39:20.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:19 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce43f302-448d-4896-8a13-3f52413a2252"}]': finished 2024-04-21T08:39:20.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:19 smithi016 ceph-mon[31259]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:39:20.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:19 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:20.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:19 smithi016 ceph-mon[31259]: from='client.? 172.21.15.16:0/3981122169' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:21.026 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:20 smithi077 ceph-mon[31615]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:21.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:20 smithi016 ceph-mon[31259]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:21.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:20 smithi007 ceph-mon[28477]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:22 smithi016 ceph-mon[31259]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:23.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:22 smithi077 ceph-mon[31615]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:23.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:22 smithi007 ceph-mon[28477]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:25.035 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:24 smithi016 ceph-mon[31259]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:25.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:24 smithi077 ceph-mon[31615]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:25.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:24 smithi007 ceph-mon[28477]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:27.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:26 smithi016 ceph-mon[31259]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:27.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:39:27.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:27.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:26 smithi016 ceph-mon[31259]: Deploying daemon osd.7 on smithi016 2024-04-21T08:39:27.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:26 smithi077 ceph-mon[31615]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:27.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:39:27.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:27.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:26 smithi077 ceph-mon[31615]: Deploying daemon osd.7 on smithi016 2024-04-21T08:39:27.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:26 smithi007 ceph-mon[28477]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:27.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:39:27.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:27.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:26 smithi007 ceph-mon[28477]: Deploying daemon osd.7 on smithi016 2024-04-21T08:39:29.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:28 smithi016 ceph-mon[31259]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:29.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:28 smithi077 ceph-mon[31615]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:29.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:28 smithi007 ceph-mon[28477]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:30 smithi016 ceph-mon[31259]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:31.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:30 smithi077 ceph-mon[31615]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:31.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:30 smithi007 ceph-mon[28477]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:33 smithi016 ceph-mon[31259]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:33.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:33 smithi077 ceph-mon[31615]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:33.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:33.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:33.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:33.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:33 smithi007 ceph-mon[28477]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:33.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:33.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:33.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:35.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:35 smithi016 ceph-mon[31259]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:35.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:35 smithi077 ceph-mon[31615]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:35.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:35 smithi007 ceph-mon[28477]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:35.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.898+0000 7f654bfff700 1 -- 172.21.15.16:0/1506405597 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6554062e80 con 0x7f65400a3cc0 2024-04-21T08:39:35.898 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 7 on host 'smithi016' 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 -- 172.21.15.16:0/1506405597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f65400a3cc0 msgr2=0x7f65400a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f65400a3cc0 0x7f65400a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6554103b20 tx=0x7f654c00b040 comp rx=0 tx=0).stop 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 -- 172.21.15.16:0/1506405597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6554103cc0 msgr2=0x7f65541990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6554103cc0 0x7f65541990e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f653c00b530 tx=0x7f653c00b840 comp rx=0 tx=0).stop 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 -- 172.21.15.16:0/1506405597 shutdown_connections 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f65400a3cc0 0x7f65400a6180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6554104660 0x7f65541a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6554102ac0 0x7f6554198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 --2- 172.21.15.16:0/1506405597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6554103cc0 0x7f65541990e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 -- 172.21.15.16:0/1506405597 >> 172.21.15.16:0/1506405597 conn(0x7f65540fe060 msgr2=0x7f6554107040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 -- 172.21.15.16:0/1506405597 shutdown_connections 2024-04-21T08:39:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:39:35.901+0000 7f6559497700 1 -- 172.21.15.16:0/1506405597 wait complete. 2024-04-21T08:39:36.507 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.507 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.507 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:36.507 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:36.507 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.507 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.507 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.507 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:36 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:36.508 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:36 smithi016 ceph-mon[31259]: from='osd.7 [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:39:36.508 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:36 smithi016 ceph-mon[31259]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:39:36.534 DEBUG:teuthology.orchestra.run.smithi016:osd.7> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.7.service 2024-04-21T08:39:36.537 INFO:tasks.cephadm:Deploying osd.8 on smithi077 with /dev/vg_nvme/lv_2... 2024-04-21T08:39:36.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:39:36.564 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.564 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.564 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:36.564 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:36.564 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.564 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.565 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.565 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:36 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:36.565 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:36 smithi077 ceph-mon[31615]: from='osd.7 [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:39:36.565 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:36 smithi077 ceph-mon[31615]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:39:36.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:36.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:36.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:36.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:36 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:36.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:36 smithi007 ceph-mon[28477]: from='osd.7 [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:39:36.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:36 smithi007 ceph-mon[28477]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:39:36.785 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:39:37.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:37 smithi016 ceph-mon[31259]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:37.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:37 smithi016 ceph-mon[31259]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:39:37.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:37 smithi016 ceph-mon[31259]: from='osd.7 [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:39:37.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:37 smithi016 ceph-mon[31259]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:39:37.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:37 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:37.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:37 smithi016 ceph-mon[31259]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:39:37.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:37 smithi077 ceph-mon[31615]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:37 smithi077 ceph-mon[31615]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:39:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:37 smithi077 ceph-mon[31615]: from='osd.7 [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:39:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:37 smithi077 ceph-mon[31615]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:39:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:37 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:37 smithi077 ceph-mon[31615]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:39:37.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:37 smithi007 ceph-mon[28477]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:37 smithi007 ceph-mon[28477]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:39:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:37 smithi007 ceph-mon[28477]: from='osd.7 [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:39:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:37 smithi007 ceph-mon[28477]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:39:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:37 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:37 smithi007 ceph-mon[28477]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:39:38.801 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:39:38.801 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:39:38.801 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-21T08:39:38.801 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-21T08:39:38.801 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114827 s, 913 MB/s 2024-04-21T08:39:38.801 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-21T08:39:38.804 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:39:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:39:38.391+0000 7f0c24788700 -1 osd.7 0 waiting for initial osdmap 2024-04-21T08:39:38.804 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:39:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:39:38.400+0000 7f0c1c573700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:39:39.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:38 smithi007 ceph-mon[28477]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:39.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:38 smithi007 ceph-mon[28477]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:39:39.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:38 smithi007 ceph-mon[28477]: osdmap e51: 8 total, 7 up, 8 in 2024-04-21T08:39:39.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:38 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:39.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:38 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:39.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:38 smithi007 ceph-mon[28477]: from='osd.7 ' entity='osd.7' 2024-04-21T08:39:39.181 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:38 smithi016 ceph-mon[31259]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:39.181 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:38 smithi016 ceph-mon[31259]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:39:39.181 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:38 smithi016 ceph-mon[31259]: osdmap e51: 8 total, 7 up, 8 in 2024-04-21T08:39:39.181 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:38 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:39.181 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:38 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:39.181 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:38 smithi016 ceph-mon[31259]: from='osd.7 ' entity='osd.7' 2024-04-21T08:39:39.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:38 smithi077 ceph-mon[31615]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:39.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:38 smithi077 ceph-mon[31615]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:39:39.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:38 smithi077 ceph-mon[31615]: osdmap e51: 8 total, 7 up, 8 in 2024-04-21T08:39:39.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:38 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:39.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:38 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:39.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:38 smithi077 ceph-mon[31615]: from='osd.7 ' entity='osd.7' 2024-04-21T08:39:39.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_2 2024-04-21T08:39:39.819 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:39:40.094 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:39:40.094 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:39:40.094 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: osd.7 [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881] boot 2024-04-21T08:39:40.094 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:39:40.094 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:40.094 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.095 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.095 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:40.095 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:40.095 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:40.095 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.095 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:40.095 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:40.095 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:39 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:39:40.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: osd.7 [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881] boot 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:40.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:39 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:39:40.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:39:40.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: osd.7 [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881] boot 2024-04-21T08:39:40.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:39:40.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:40.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:40.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:40.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:40.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:40.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:40.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:39 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:40.979 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:40 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi016 2024-04-21T08:39:40.979 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:40 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi016 to 5697M 2024-04-21T08:39:40.979 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:40 smithi077 ceph-mon[31615]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:40.980 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:40 smithi077 ceph-mon[31615]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:39:41.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:40 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi016 2024-04-21T08:39:41.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:40 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi016 to 5697M 2024-04-21T08:39:41.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:40 smithi007 ceph-mon[28477]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:40 smithi007 ceph-mon[28477]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:39:41.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:40 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi016 2024-04-21T08:39:41.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:40 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi016 to 5697M 2024-04-21T08:39:41.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:40 smithi016 ceph-mon[31259]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:41.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:40 smithi016 ceph-mon[31259]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:39:41.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.527+0000 7f3c9330f700 1 -- 172.21.15.77:0/1161729476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c8c103520 msgr2=0x7f3c8c105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:41.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.527+0000 7f3c9330f700 1 --2- 172.21.15.77:0/1161729476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c8c103520 0x7f3c8c105910 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3c88009990 tx=0x7f3c88009ca0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.527+0000 7f3c9330f700 1 -- 172.21.15.77:0/1161729476 shutdown_connections 2024-04-21T08:39:41.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.527+0000 7f3c9330f700 1 --2- 172.21.15.77:0/1161729476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c8c103520 0x7f3c8c105910 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.527+0000 7f3c9330f700 1 --2- 172.21.15.77:0/1161729476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c8c108780 0x7f3c8c10abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.527+0000 7f3c9330f700 1 --2- 172.21.15.77:0/1161729476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c105e50 0x7f3c8c108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.527+0000 7f3c9330f700 1 -- 172.21.15.77:0/1161729476 >> 172.21.15.77:0/1161729476 conn(0x7f3c8c0fcf30 msgr2=0x7f3c8c0ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:41.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c9330f700 1 -- 172.21.15.77:0/1161729476 shutdown_connections 2024-04-21T08:39:41.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c9330f700 1 -- 172.21.15.77:0/1161729476 wait complete. 2024-04-21T08:39:41.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c9330f700 1 Processor -- start 2024-04-21T08:39:41.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c9330f700 1 -- start start 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c9330f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c8c103520 0x7f3c8c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c9330f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c105e50 0x7f3c8c199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c9330f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c8c108780 0x7f3c8c1a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c9330f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c10e510 con 0x7f3c8c103520 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c9330f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c10e390 con 0x7f3c8c105e50 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c9330f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c10e690 con 0x7f3c8c108780 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c918ac700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c8c108780 0x7f3c8c1a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c918ac700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c8c108780 0x7f3c8c1a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33842/0 (socket says 172.21.15.77:33842) 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c918ac700 1 -- 172.21.15.77:0/3612856243 learned_addr learned my addr 172.21.15.77:0/3612856243 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.528+0000 7f3c908aa700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c105e50 0x7f3c8c199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c910ab700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c8c103520 0x7f3c8c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c918ac700 1 -- 172.21.15.77:0/3612856243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c105e50 msgr2=0x7f3c8c199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c918ac700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c105e50 0x7f3c8c199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c918ac700 1 -- 172.21.15.77:0/3612856243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c8c103520 msgr2=0x7f3c8c198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c918ac700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c8c103520 0x7f3c8c198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c918ac700 1 -- 172.21.15.77:0/3612856243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8c1a0820 con 0x7f3c8c108780 2024-04-21T08:39:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c910ab700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c8c103520 0x7f3c8c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c918ac700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c8c108780 0x7f3c8c1a0180 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400efc0 tx=0x7f3c8400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c827fc700 1 -- 172.21.15.77:0/3612856243 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c840092e0 con 0x7f3c8c108780 2024-04-21T08:39:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c827fc700 1 -- 172.21.15.77:0/3612856243 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c84004bb0 con 0x7f3c8c108780 2024-04-21T08:39:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c9330f700 1 -- 172.21.15.77:0/3612856243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c8c1a0ab0 con 0x7f3c8c108780 2024-04-21T08:39:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c827fc700 1 -- 172.21.15.77:0/3612856243 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c84004280 con 0x7f3c8c108780 2024-04-21T08:39:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.529+0000 7f3c9330f700 1 -- 172.21.15.77:0/3612856243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c8c1a1040 con 0x7f3c8c108780 2024-04-21T08:39:41.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.530+0000 7f3c9330f700 1 -- 172.21.15.77:0/3612856243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c8c069730 con 0x7f3c8c108780 2024-04-21T08:39:41.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.531+0000 7f3c827fc700 1 -- 172.21.15.77:0/3612856243 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f3c8401b030 con 0x7f3c8c108780 2024-04-21T08:39:41.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.532+0000 7f3c827fc700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3c780a3cc0 0x7f3c780a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:41.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.532+0000 7f3c827fc700 1 -- 172.21.15.77:0/3612856243 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7f3c840cf3a0 con 0x7f3c8c108780 2024-04-21T08:39:41.532 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.532+0000 7f3c910ab700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3c780a3cc0 0x7f3c780a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:41.533 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.533+0000 7f3c910ab700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3c780a3cc0 0x7f3c780a6180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800b2e0 tx=0x7f3c8801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:41.536 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.537+0000 7f3c827fc700 1 -- 172.21.15.77:0/3612856243 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c84022020 con 0x7f3c8c108780 2024-04-21T08:39:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:39:41.742+0000 7f3c9330f700 1 -- 172.21.15.77:0/3612856243 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3c8c062e80 con 0x7f3c780a3cc0 2024-04-21T08:39:42.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:41 smithi016 ceph-mon[31259]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:39:42.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:41 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:42.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:41 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:42.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:41 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:42.352 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:41 smithi077 ceph-mon[31615]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:39:42.352 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:41 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:42.352 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:41 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:42.352 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:41 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:42.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:41 smithi007 ceph-mon[28477]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:39:42.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:41 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:42.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:41 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:42.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:41 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:43.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:42 smithi007 ceph-mon[28477]: pgmap v180: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:43.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:42 smithi007 ceph-mon[28477]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:43.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:42 smithi007 ceph-mon[28477]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:39:43.303 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:42 smithi016 ceph-mon[31259]: pgmap v180: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:43.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:42 smithi016 ceph-mon[31259]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:43.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:42 smithi016 ceph-mon[31259]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:39:43.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:42 smithi077 ceph-mon[31615]: pgmap v180: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:43.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:42 smithi077 ceph-mon[31615]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:43.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:42 smithi077 ceph-mon[31615]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:39:44.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:43 smithi077 ceph-mon[31615]: osdmap e56: 8 total, 8 up, 8 in 2024-04-21T08:39:44.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:43 smithi016 ceph-mon[31259]: osdmap e56: 8 total, 8 up, 8 in 2024-04-21T08:39:44.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:43 smithi007 ceph-mon[28477]: osdmap e56: 8 total, 8 up, 8 in 2024-04-21T08:39:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:44 smithi016 ceph-mon[31259]: pgmap v183: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:44 smithi016 ceph-mon[31259]: from='client.? 172.21.15.77:0/4163673609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84019e97-520b-4134-93dc-67d0654f29fe"}]: dispatch 2024-04-21T08:39:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:44 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84019e97-520b-4134-93dc-67d0654f29fe"}]: dispatch 2024-04-21T08:39:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:44 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84019e97-520b-4134-93dc-67d0654f29fe"}]': finished 2024-04-21T08:39:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:44 smithi016 ceph-mon[31259]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:39:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:39:45.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:44 smithi077 ceph-mon[31615]: pgmap v183: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:45.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:44 smithi077 ceph-mon[31615]: from='client.? 172.21.15.77:0/4163673609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84019e97-520b-4134-93dc-67d0654f29fe"}]: dispatch 2024-04-21T08:39:45.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:44 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84019e97-520b-4134-93dc-67d0654f29fe"}]: dispatch 2024-04-21T08:39:45.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:44 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84019e97-520b-4134-93dc-67d0654f29fe"}]': finished 2024-04-21T08:39:45.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:44 smithi077 ceph-mon[31615]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:39:45.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:39:45.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:44 smithi007 ceph-mon[28477]: pgmap v183: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 263 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:45.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:44 smithi007 ceph-mon[28477]: from='client.? 172.21.15.77:0/4163673609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84019e97-520b-4134-93dc-67d0654f29fe"}]: dispatch 2024-04-21T08:39:45.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:44 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84019e97-520b-4134-93dc-67d0654f29fe"}]: dispatch 2024-04-21T08:39:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:44 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84019e97-520b-4134-93dc-67d0654f29fe"}]': finished 2024-04-21T08:39:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:44 smithi007 ceph-mon[28477]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:39:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:39:46.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:45 smithi016 ceph-mon[31259]: from='client.? 172.21.15.77:0/1648406175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:46.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:45 smithi077 ceph-mon[31615]: from='client.? 172.21.15.77:0/1648406175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:46.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:45 smithi007 ceph-mon[28477]: from='client.? 172.21.15.77:0/1648406175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:46 smithi016 ceph-mon[31259]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:47.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:46 smithi077 ceph-mon[31615]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:47.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:46 smithi007 ceph-mon[28477]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:48 smithi016 ceph-mon[31259]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:48 smithi077 ceph-mon[31615]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:49.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:48 smithi007 ceph-mon[28477]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:51.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:50 smithi077 ceph-mon[31615]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:50 smithi016 ceph-mon[31259]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:51.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:50 smithi007 ceph-mon[28477]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:53.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:52 smithi077 ceph-mon[31615]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:39:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:52 smithi077 ceph-mon[31615]: Deploying daemon osd.8 on smithi077 2024-04-21T08:39:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:52 smithi016 ceph-mon[31259]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:39:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:52 smithi016 ceph-mon[31259]: Deploying daemon osd.8 on smithi077 2024-04-21T08:39:53.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:52 smithi007 ceph-mon[28477]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:53.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:39:53.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:53.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:52 smithi007 ceph-mon[28477]: Deploying daemon osd.8 on smithi077 2024-04-21T08:39:55.095 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:54 smithi077 ceph-mon[31615]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:55.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:54 smithi016 ceph-mon[31259]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:55.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:54 smithi007 ceph-mon[28477]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:57.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:56 smithi016 ceph-mon[31259]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:57.321 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:56 smithi077 ceph-mon[31615]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:57.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:56 smithi007 ceph-mon[28477]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:58 smithi016 ceph-mon[31259]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:58 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:58 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:39:58 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:58 smithi077 ceph-mon[31615]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:39:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:59.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:58 smithi007 ceph-mon[28477]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:39:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:39:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:39:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:01.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:00 smithi077 ceph-mon[31615]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:01.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:00 smithi077 ceph-mon[31615]: overall HEALTH_OK 2024-04-21T08:40:01.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:00 smithi016 ceph-mon[31259]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:01.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:00 smithi016 ceph-mon[31259]: overall HEALTH_OK 2024-04-21T08:40:01.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:00 smithi007 ceph-mon[28477]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:01.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:00 smithi007 ceph-mon[28477]: overall HEALTH_OK 2024-04-21T08:40:01.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.868+0000 7f3c827fc700 1 -- 172.21.15.77:0/3612856243 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3c8c062e80 con 0x7f3c780a3cc0 2024-04-21T08:40:01.868 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 8 on host 'smithi077' 2024-04-21T08:40:01.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.870+0000 7f3c9330f700 1 -- 172.21.15.77:0/3612856243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3c780a3cc0 msgr2=0x7f3c780a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.870+0000 7f3c9330f700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3c780a3cc0 0x7f3c780a6180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800b2e0 tx=0x7f3c8801a040 comp rx=0 tx=0).stop 2024-04-21T08:40:01.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.870+0000 7f3c9330f700 1 -- 172.21.15.77:0/3612856243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c8c108780 msgr2=0x7f3c8c1a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:01.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.870+0000 7f3c9330f700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c8c108780 0x7f3c8c1a0180 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400efc0 tx=0x7f3c8400c3a0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.871+0000 7f3c9330f700 1 -- 172.21.15.77:0/3612856243 shutdown_connections 2024-04-21T08:40:01.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.871+0000 7f3c9330f700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3c780a3cc0 0x7f3c780a6180 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.871+0000 7f3c9330f700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c8c108780 0x7f3c8c1a0180 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.871+0000 7f3c9330f700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c8c103520 0x7f3c8c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.871+0000 7f3c9330f700 1 --2- 172.21.15.77:0/3612856243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c8c105e50 0x7f3c8c199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:01.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.871+0000 7f3c9330f700 1 -- 172.21.15.77:0/3612856243 >> 172.21.15.77:0/3612856243 conn(0x7f3c8c0fcf30 msgr2=0x7f3c8c1053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:01.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.871+0000 7f3c9330f700 1 -- 172.21.15.77:0/3612856243 shutdown_connections 2024-04-21T08:40:01.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:40:01.871+0000 7f3c9330f700 1 -- 172.21.15.77:0/3612856243 wait complete. 2024-04-21T08:40:02.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:02 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:02 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:02 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:02.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:02 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:02.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:02 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:02 smithi077 ceph-mon[31615]: from='osd.8 [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:02.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:02 smithi077 ceph-mon[31615]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:02.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:02 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:02 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:02.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:02 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:02.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:02.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:02 smithi016 ceph-mon[31259]: from='osd.8 [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:02.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:02 smithi016 ceph-mon[31259]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:02.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:02.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:02 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:02 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:02 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:02 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:02.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:02 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:02.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:02 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:02 smithi007 ceph-mon[28477]: from='osd.8 [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:02.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:02 smithi007 ceph-mon[28477]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:02.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:02 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:02 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:02.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:02 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:02.598 DEBUG:teuthology.orchestra.run.smithi077:osd.8> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.8.service 2024-04-21T08:40:02.600 INFO:tasks.cephadm:Deploying osd.9 on smithi007 with /dev/vg_nvme/lv_1... 2024-04-21T08:40:02.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:40:02.865 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:40:03.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:03 smithi007 ceph-mon[28477]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:03.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:03 smithi007 ceph-mon[28477]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:40:03.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:03 smithi007 ceph-mon[28477]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:40:03.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:03 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:03.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:03 smithi007 ceph-mon[28477]: from='osd.8 [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:40:03.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:03 smithi007 ceph-mon[28477]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:40:03.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:03 smithi077 ceph-mon[31615]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:03 smithi077 ceph-mon[31615]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:40:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:03 smithi077 ceph-mon[31615]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:40:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:03 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:03 smithi077 ceph-mon[31615]: from='osd.8 [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:40:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:03 smithi077 ceph-mon[31615]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:40:03.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:03 smithi016 ceph-mon[31259]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:03.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:03 smithi016 ceph-mon[31259]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:40:03.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:03 smithi016 ceph-mon[31259]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:40:03.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:03 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:03.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:03 smithi016 ceph-mon[31259]: from='osd.8 [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:40:03.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:03 smithi016 ceph-mon[31259]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:40:03.835 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:40:03 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:40:03.466+0000 7f42b1b10700 -1 osd.8 0 waiting for initial osdmap 2024-04-21T08:40:03.835 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:40:03 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:40:03.478+0000 7f42a98fb700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:40:04.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:04 smithi077 ceph-mon[31615]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:40:04.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:04 smithi077 ceph-mon[31615]: osdmap e59: 9 total, 8 up, 9 in 2024-04-21T08:40:04.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:04.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:04.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:04 smithi077 ceph-mon[31615]: from='osd.8 ' entity='osd.8' 2024-04-21T08:40:04.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:04 smithi007 ceph-mon[28477]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:40:04.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:04 smithi007 ceph-mon[28477]: osdmap e59: 9 total, 8 up, 9 in 2024-04-21T08:40:04.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:04.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:04.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:04 smithi007 ceph-mon[28477]: from='osd.8 ' entity='osd.8' 2024-04-21T08:40:04.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:04 smithi016 ceph-mon[31259]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:40:04.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:04 smithi016 ceph-mon[31259]: osdmap e59: 9 total, 8 up, 9 in 2024-04-21T08:40:04.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:04.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:04.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:04 smithi016 ceph-mon[31259]: from='osd.8 ' entity='osd.8' 2024-04-21T08:40:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:40:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:40:05.164 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-21T08:40:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-21T08:40:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00842944 s, 1.2 GB/s 2024-04-21T08:40:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-21T08:40:05.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:05 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:40:05.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:05 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:40:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:05 smithi077 ceph-mon[31615]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:05 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:05 smithi077 ceph-mon[31615]: osd.8 [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717] boot 2024-04-21T08:40:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:05 smithi077 ceph-mon[31615]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:40:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:05 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:05.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:05 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:40:05.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:05 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:40:05.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:05 smithi007 ceph-mon[28477]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:05.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:05 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:05.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:05 smithi007 ceph-mon[28477]: osd.8 [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717] boot 2024-04-21T08:40:05.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:05 smithi007 ceph-mon[28477]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:40:05.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:05 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:05.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:05 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:40:05.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:05 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:40:05.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:05 smithi016 ceph-mon[31259]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:05.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:05 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:05.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:05 smithi016 ceph-mon[31259]: osd.8 [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717] boot 2024-04-21T08:40:05.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:05 smithi016 ceph-mon[31259]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:40:05.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:05 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:05.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_1 2024-04-21T08:40:06.052 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:40:06.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:06 smithi077 ceph-mon[31615]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:40:06.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:06 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:06.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:06 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:06.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:06 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:06.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:06 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:06.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:06 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:06.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:06 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:06.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:06 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:06.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:06 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:06.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:06 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:06.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:06 smithi007 ceph-mon[28477]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:40:06.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:06 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:06.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:06 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:06.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:06 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:06.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:06 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:06.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:06 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:06.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:06 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:06.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:06 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:06.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:06 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:06.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:06 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:06.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:06 smithi016 ceph-mon[31259]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:40:06.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:06 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:06.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:06 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:06.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:06 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:06.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:06 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:06.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:06 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:06.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:06 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:06.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:06 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:06.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:06 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:06.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:06 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:07 smithi077 ceph-mon[31615]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:07 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi077 2024-04-21T08:40:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:07 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi077 to 5696M 2024-04-21T08:40:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:07 smithi077 ceph-mon[31615]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:40:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:07 smithi007 ceph-mon[28477]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:07 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi077 2024-04-21T08:40:07.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:07 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi077 to 5696M 2024-04-21T08:40:07.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:07 smithi007 ceph-mon[28477]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:40:07.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:07 smithi016 ceph-mon[31259]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:07.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:07 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi077 2024-04-21T08:40:07.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:07 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi077 to 5696M 2024-04-21T08:40:07.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:07 smithi016 ceph-mon[31259]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:40:07.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.684+0000 7f1df515d700 1 -- 172.21.15.7:0/875607302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1df0103550 msgr2=0x7f1df0105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.684+0000 7f1df515d700 1 --2- 172.21.15.7:0/875607302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1df0103550 0x7f1df0105940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009990 tx=0x7f1de0009ca0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.684+0000 7f1df515d700 1 -- 172.21.15.7:0/875607302 shutdown_connections 2024-04-21T08:40:07.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.684+0000 7f1df515d700 1 --2- 172.21.15.7:0/875607302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df01087b0 0x7f1df010ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.684+0000 7f1df515d700 1 --2- 172.21.15.7:0/875607302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1df0103550 0x7f1df0105940 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.684+0000 7f1df515d700 1 --2- 172.21.15.7:0/875607302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df0105e80 0x7f1df0108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.684+0000 7f1df515d700 1 -- 172.21.15.7:0/875607302 >> 172.21.15.7:0/875607302 conn(0x7f1df00fcf60 msgr2=0x7f1df00ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:07.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.684+0000 7f1df515d700 1 -- 172.21.15.7:0/875607302 shutdown_connections 2024-04-21T08:40:07.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.684+0000 7f1df515d700 1 -- 172.21.15.7:0/875607302 wait complete. 2024-04-21T08:40:07.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.685+0000 7f1df515d700 1 Processor -- start 2024-04-21T08:40:07.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.685+0000 7f1df515d700 1 -- start start 2024-04-21T08:40:07.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.685+0000 7f1df515d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1df0103550 0x7f1df0198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.685+0000 7f1df515d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df0105e80 0x7f1df0199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.685+0000 7f1df515d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df01087b0 0x7f1df01a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.685+0000 7f1df515d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1df010e5f0 con 0x7f1df0103550 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.685+0000 7f1df515d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1df010e470 con 0x7f1df01087b0 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.685+0000 7f1df515d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1df010e770 con 0x7f1df0105e80 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.685+0000 7f1deed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1df0103550 0x7f1df0198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.685+0000 7f1deed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1df0103550 0x7f1df0198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37030/0 (socket says 172.21.15.7:37030) 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.685+0000 7f1deed9d700 1 -- 172.21.15.7:0/2869162283 learned_addr learned my addr 172.21.15.7:0/2869162283 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1dee59c700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df0105e80 0x7f1df0199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1def59e700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df01087b0 0x7f1df01a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1deed9d700 1 -- 172.21.15.7:0/2869162283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df0105e80 msgr2=0x7f1df0199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1deed9d700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df0105e80 0x7f1df0199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1deed9d700 1 -- 172.21.15.7:0/2869162283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df01087b0 msgr2=0x7f1df01a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1deed9d700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df01087b0 0x7f1df01a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:07.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1deed9d700 1 -- 172.21.15.7:0/2869162283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df01a08a0 con 0x7f1df0103550 2024-04-21T08:40:07.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1def59e700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df01087b0 0x7f1df01a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:07.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1deed9d700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1df0103550 0x7f1df0198c50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1de0006010 tx=0x7f1de00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:07.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1dd7fff700 1 -- 172.21.15.7:0/2869162283 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de0027070 con 0x7f1df0103550 2024-04-21T08:40:07.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1dd7fff700 1 -- 172.21.15.7:0/2869162283 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1de0004080 con 0x7f1df0103550 2024-04-21T08:40:07.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1dd7fff700 1 -- 172.21.15.7:0/2869162283 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de0022e00 con 0x7f1df0103550 2024-04-21T08:40:07.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1df515d700 1 -- 172.21.15.7:0/2869162283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df01a0b30 con 0x7f1df0103550 2024-04-21T08:40:07.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.686+0000 7f1df515d700 1 -- 172.21.15.7:0/2869162283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df01a0fb0 con 0x7f1df0103550 2024-04-21T08:40:07.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.687+0000 7f1df515d700 1 -- 172.21.15.7:0/2869162283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df002df80 con 0x7f1df0103550 2024-04-21T08:40:07.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.689+0000 7f1dd7fff700 1 -- 172.21.15.7:0/2869162283 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f1de002a070 con 0x7f1df0103550 2024-04-21T08:40:07.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.689+0000 7f1dd7fff700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1dd80a3c40 0x7f1dd80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:07.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.689+0000 7f1dee59c700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1dd80a3c40 0x7f1dd80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:07.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.689+0000 7f1dd7fff700 1 -- 172.21.15.7:0/2869162283 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f1de00d0d10 con 0x7f1df0103550 2024-04-21T08:40:07.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.690+0000 7f1dee59c700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1dd80a3c40 0x7f1dd80a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009a10 tx=0x7f1de4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:07.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.693+0000 7f1dd7fff700 1 -- 172.21.15.7:0/2869162283 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1de0013070 con 0x7f1df0103550 2024-04-21T08:40:07.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:07.914+0000 7f1df515d700 1 -- 172.21.15.7:0/2869162283 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1df0062e80 con 0x7f1dd80a3c40 2024-04-21T08:40:08.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:08 smithi077 ceph-mon[31615]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:40:08.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:08.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:08.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:08 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:08.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:08 smithi007 ceph-mon[28477]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:40:08.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:08.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:08.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:08 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:08.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:08 smithi016 ceph-mon[31259]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:40:08.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:08.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:08.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:08 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:09.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:09 smithi007 ceph-mon[28477]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:09.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:09 smithi007 ceph-mon[28477]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:09.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:09 smithi007 ceph-mon[28477]: osdmap e64: 9 total, 9 up, 9 in 2024-04-21T08:40:09.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:09 smithi077 ceph-mon[31615]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:09.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:09 smithi077 ceph-mon[31615]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:09.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:09 smithi077 ceph-mon[31615]: osdmap e64: 9 total, 9 up, 9 in 2024-04-21T08:40:09.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:09 smithi016 ceph-mon[31259]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:09.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:09 smithi016 ceph-mon[31259]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:09.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:09 smithi016 ceph-mon[31259]: osdmap e64: 9 total, 9 up, 9 in 2024-04-21T08:40:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:11 smithi077 ceph-mon[31615]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-21T08:40:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:11 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/51115565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6"}]: dispatch 2024-04-21T08:40:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:11 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/51115565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6"}]': finished 2024-04-21T08:40:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:11 smithi077 ceph-mon[31615]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:40:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:11 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:11.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:11 smithi007 ceph-mon[28477]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-21T08:40:11.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:11 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/51115565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6"}]: dispatch 2024-04-21T08:40:11.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:11 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/51115565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6"}]': finished 2024-04-21T08:40:11.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:11 smithi007 ceph-mon[28477]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:40:11.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:11 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:11.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:11 smithi016 ceph-mon[31259]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-21T08:40:11.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:11 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/51115565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6"}]: dispatch 2024-04-21T08:40:11.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:11 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/51115565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6"}]': finished 2024-04-21T08:40:11.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:11 smithi016 ceph-mon[31259]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:40:11.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:11 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:12.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:12 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4164118535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:12.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:12 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4164118535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:12.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:12 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4164118535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:13.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:13 smithi077 ceph-mon[31615]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-21T08:40:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:13 smithi007 ceph-mon[28477]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-21T08:40:13.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:13 smithi016 ceph-mon[31259]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-21T08:40:15.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:15 smithi007 ceph-mon[28477]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-21T08:40:15.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:15 smithi016 ceph-mon[31259]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-21T08:40:15.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:15 smithi077 ceph-mon[31615]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-21T08:40:17.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:17 smithi007 ceph-mon[28477]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:40:17.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:17 smithi016 ceph-mon[31259]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:40:17.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:17 smithi077 ceph-mon[31615]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:40:19.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:19 smithi007 ceph-mon[28477]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T08:40:19.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:19 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:40:19.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:19 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:19.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:19 smithi007 ceph-mon[28477]: Deploying daemon osd.9 on smithi007 2024-04-21T08:40:19.553 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:19 smithi016 ceph-mon[31259]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T08:40:19.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:19 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:40:19.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:19 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:19.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:19 smithi016 ceph-mon[31259]: Deploying daemon osd.9 on smithi007 2024-04-21T08:40:19.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:19 smithi077 ceph-mon[31615]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T08:40:19.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:19 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:40:19.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:19 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:19.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:19 smithi077 ceph-mon[31615]: Deploying daemon osd.9 on smithi007 2024-04-21T08:40:21.378 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:21 smithi077 ceph-mon[31615]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:21.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:21 smithi007 ceph-mon[28477]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:21.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:21 smithi016 ceph-mon[31259]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:23.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:23 smithi007 ceph-mon[28477]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:23.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:23 smithi016 ceph-mon[31259]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:23.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:23 smithi077 ceph-mon[31615]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:25.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:25 smithi007 ceph-mon[28477]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:25.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:25 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:25.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:25 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:25.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:25 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:25.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:25 smithi016 ceph-mon[31259]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:25.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:25 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:25.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:25 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:25.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:25 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:25.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:25 smithi077 ceph-mon[31615]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:25.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:25 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:25.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:25 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:25.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:25 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:27 smithi007 ceph-mon[28477]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:27.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:27 smithi016 ceph-mon[31259]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:27.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:27 smithi077 ceph-mon[31615]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:27.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.971+0000 7f1dd7fff700 1 -- 172.21.15.7:0/2869162283 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1df0062e80 con 0x7f1dd80a3c40 2024-04-21T08:40:27.971 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 9 on host 'smithi007' 2024-04-21T08:40:27.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 -- 172.21.15.7:0/2869162283 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1dd80a3c40 msgr2=0x7f1dd80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:27.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1dd80a3c40 0x7f1dd80a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009a10 tx=0x7f1de4009190 comp rx=0 tx=0).stop 2024-04-21T08:40:27.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 -- 172.21.15.7:0/2869162283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1df0103550 msgr2=0x7f1df0198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:27.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1df0103550 0x7f1df0198c50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1de0006010 tx=0x7f1de00047b0 comp rx=0 tx=0).stop 2024-04-21T08:40:27.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 -- 172.21.15.7:0/2869162283 shutdown_connections 2024-04-21T08:40:27.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1dd80a3c40 0x7f1dd80a6100 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1df01087b0 0x7f1df01a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1df0103550 0x7f1df0198c50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 --2- 172.21.15.7:0/2869162283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1df0105e80 0x7f1df0199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 -- 172.21.15.7:0/2869162283 >> 172.21.15.7:0/2869162283 conn(0x7f1df00fcf60 msgr2=0x7f1df0103f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 -- 172.21.15.7:0/2869162283 shutdown_connections 2024-04-21T08:40:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:40:27.975+0000 7f1df515d700 1 -- 172.21.15.7:0/2869162283 wait complete. 2024-04-21T08:40:28.575 DEBUG:teuthology.orchestra.run.smithi007:osd.9> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.9.service 2024-04-21T08:40:28.577 INFO:tasks.cephadm:Deploying osd.10 on smithi016 with /dev/vg_nvme/lv_1... 2024-04-21T08:40:28.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:40:28.835 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:40:29.198 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:28 smithi016 ceph-mon[31259]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:29.198 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:28 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:29.199 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:28 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:29.199 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:28 smithi016 ceph-mon[31259]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:40:29.199 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:28 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:29.199 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:28 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:29.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:28 smithi077 ceph-mon[31615]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:29.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:28 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:29.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:28 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:29.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:28 smithi077 ceph-mon[31615]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:40:29.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:28 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:29.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:28 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:29.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:28 smithi007 ceph-mon[28477]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:29.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:28 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:29.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:28 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:29.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:28 smithi007 ceph-mon[28477]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:40:29.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:28 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:29.365 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:28 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:30.258 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:29 smithi016 ceph-mon[31259]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:40:30.259 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:29 smithi016 ceph-mon[31259]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:40:30.259 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:29 smithi016 ceph-mon[31259]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:40:30.259 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:29 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:30.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:29 smithi007 ceph-mon[28477]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:40:30.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:29 smithi007 ceph-mon[28477]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:40:30.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:29 smithi007 ceph-mon[28477]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:40:30.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:29 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:30.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:29 smithi077 ceph-mon[31615]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:40:30.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:29 smithi077 ceph-mon[31615]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:40:30.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:29 smithi077 ceph-mon[31615]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-21T08:40:30.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:29 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:30.593 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:40:30 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:40:30.363+0000 7f96fbfd3700 -1 osd.9 0 waiting for initial osdmap 2024-04-21T08:40:30.593 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:40:30 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:40:30.377+0000 7f96f3dbe700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:40:30.844 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:40:30.844 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:40:30.844 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-21T08:40:30.844 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-21T08:40:30.844 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00833956 s, 1.3 GB/s 2024-04-21T08:40:30.844 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-21T08:40:31.235 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:30 smithi016 ceph-mon[31259]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:31.235 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:30 smithi016 ceph-mon[31259]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:40:31.235 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:30 smithi016 ceph-mon[31259]: osdmap e67: 10 total, 9 up, 10 in 2024-04-21T08:40:31.235 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:30 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:31.235 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:30 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:31.235 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:30 smithi016 ceph-mon[31259]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' 2024-04-21T08:40:31.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:30 smithi077 ceph-mon[31615]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:31.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:30 smithi077 ceph-mon[31615]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:40:31.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:30 smithi077 ceph-mon[31615]: osdmap e67: 10 total, 9 up, 10 in 2024-04-21T08:40:31.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:30 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:31.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:30 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:31.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:30 smithi077 ceph-mon[31615]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' 2024-04-21T08:40:31.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:30 smithi007 ceph-mon[28477]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:31.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:30 smithi007 ceph-mon[28477]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-21T08:40:31.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:30 smithi007 ceph-mon[28477]: osdmap e67: 10 total, 9 up, 10 in 2024-04-21T08:40:31.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:30 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:31.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:30 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:31.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:30 smithi007 ceph-mon[28477]: from='osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831]' entity='osd.9' 2024-04-21T08:40:31.589 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2024-04-21T08:40:31.838 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:40:32.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:31 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:40:32.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:31 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:40:32.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:31 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:32.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:31 smithi077 ceph-mon[31615]: osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831] boot 2024-04-21T08:40:32.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:31 smithi077 ceph-mon[31615]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:40:32.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:31 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:32.214 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:31 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:40:32.214 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:31 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:40:32.214 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:31 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:32.214 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:31 smithi016 ceph-mon[31259]: osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831] boot 2024-04-21T08:40:32.214 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:31 smithi016 ceph-mon[31259]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:40:32.214 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:31 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:32.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:31 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:40:32.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:31 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:40:32.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:31 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:32.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:31 smithi007 ceph-mon[28477]: osd.9 [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831] boot 2024-04-21T08:40:32.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:31 smithi007 ceph-mon[28477]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:40:32.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:31 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:33.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.596+0000 7f543a42f700 1 -- 172.21.15.16:0/542220258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5434108810 msgr2=0x7f543410ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.596+0000 7f543a42f700 1 --2- 172.21.15.16:0/542220258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5434108810 0x7f543410ac20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f542c00b0a0 tx=0x7f542c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.596+0000 7f543a42f700 1 -- 172.21.15.16:0/542220258 shutdown_connections 2024-04-21T08:40:33.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.596+0000 7f543a42f700 1 --2- 172.21.15.16:0/542220258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54341035b0 0x7f54341059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.596+0000 7f543a42f700 1 --2- 172.21.15.16:0/542220258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5434108810 0x7f543410ac20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.596+0000 7f543a42f700 1 --2- 172.21.15.16:0/542220258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5434105ee0 0x7f54341082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.596+0000 7f543a42f700 1 -- 172.21.15.16:0/542220258 >> 172.21.15.16:0/542220258 conn(0x7f54340fcf60 msgr2=0x7f54340ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:33.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.596+0000 7f543a42f700 1 -- 172.21.15.16:0/542220258 shutdown_connections 2024-04-21T08:40:33.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.596+0000 7f543a42f700 1 -- 172.21.15.16:0/542220258 wait complete. 2024-04-21T08:40:33.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.597+0000 7f543a42f700 1 Processor -- start 2024-04-21T08:40:33.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.597+0000 7f543a42f700 1 -- start start 2024-04-21T08:40:33.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.597+0000 7f543a42f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54341035b0 0x7f5434198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.597+0000 7f543a42f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5434105ee0 0x7f5434199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.597+0000 7f543a42f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5434108810 0x7f54341a00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.597+0000 7f543a42f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f543410e520 con 0x7f5434105ee0 2024-04-21T08:40:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.597+0000 7f543a42f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f543410e3a0 con 0x7f5434108810 2024-04-21T08:40:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.597+0000 7f543a42f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f543410e6a0 con 0x7f54341035b0 2024-04-21T08:40:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.597+0000 7f54389cc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5434108810 0x7f54341a00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.597+0000 7f54389cc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5434108810 0x7f54341a00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40914/0 (socket says 172.21.15.16:40914) 2024-04-21T08:40:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.597+0000 7f54389cc700 1 -- 172.21.15.16:0/934381944 learned_addr learned my addr 172.21.15.16:0/934381944 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-21T08:40:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f54337fe700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5434105ee0 0x7f5434199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f5433fff700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54341035b0 0x7f5434198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f54389cc700 1 -- 172.21.15.16:0/934381944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54341035b0 msgr2=0x7f5434198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f54389cc700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54341035b0 0x7f5434198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f54389cc700 1 -- 172.21.15.16:0/934381944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5434105ee0 msgr2=0x7f5434199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f54389cc700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5434105ee0 0x7f5434199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f54389cc700 1 -- 172.21.15.16:0/934381944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54341a0740 con 0x7f5434108810 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f54389cc700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5434108810 0x7f54341a00a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f542c0062a0 tx=0x7f542c004e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f54317fa700 1 -- 172.21.15.16:0/934381944 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c021070 con 0x7f5434108810 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f543a42f700 1 -- 172.21.15.16:0/934381944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54341a09d0 con 0x7f5434108810 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f54317fa700 1 -- 172.21.15.16:0/934381944 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f542c007c00 con 0x7f5434108810 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f54317fa700 1 -- 172.21.15.16:0/934381944 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c004770 con 0x7f5434108810 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.598+0000 7f543a42f700 1 -- 172.21.15.16:0/934381944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54341a0f10 con 0x7f5434108810 2024-04-21T08:40:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.599+0000 7f543a42f700 1 -- 172.21.15.16:0/934381944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5434069730 con 0x7f5434108810 2024-04-21T08:40:33.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.600+0000 7f54317fa700 1 -- 172.21.15.16:0/934381944 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f542c003a00 con 0x7f5434108810 2024-04-21T08:40:33.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.601+0000 7f54317fa700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f541c0a3c40 0x7f541c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:33.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.601+0000 7f54317fa700 1 -- 172.21.15.16:0/934381944 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f542c00f020 con 0x7f5434108810 2024-04-21T08:40:33.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.602+0000 7f5433fff700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f541c0a3c40 0x7f541c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:33.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.602+0000 7f5433fff700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f541c0a3c40 0x7f541c0a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f542400ac50 tx=0x7f542400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:33.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.605+0000 7f54317fa700 1 -- 172.21.15.16:0/934381944 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f542c02a020 con 0x7f5434108810 2024-04-21T08:40:33.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:33.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: osdmap e69: 10 total, 10 up, 10 in 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:33.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:33 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:33.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:33.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:33.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: osdmap e69: 10 total, 10 up, 10 in 2024-04-21T08:40:33.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:33.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:33.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:33 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:33.821+0000 7f543a42f700 1 -- 172.21.15.16:0/934381944 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5434062e80 con 0x7f541c0a3c40 2024-04-21T08:40:33.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:33.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: osdmap e69: 10 total, 10 up, 10 in 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:33 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:34.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:34 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi007 2024-04-21T08:40:34.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:34 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi007 to 4273M 2024-04-21T08:40:34.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:34 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:34.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:34 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:34.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:34 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:34.690 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:34 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi007 2024-04-21T08:40:34.690 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:34 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi007 to 4273M 2024-04-21T08:40:34.691 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:34 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:34.691 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:34 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:34.691 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:34 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:34.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:34 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi007 2024-04-21T08:40:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:34 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi007 to 4273M 2024-04-21T08:40:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:34 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:34 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:34 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:35.546 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:35 smithi016 ceph-mon[31259]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:35.546 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:35 smithi016 ceph-mon[31259]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:35.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:35 smithi007 ceph-mon[28477]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:35.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:35 smithi007 ceph-mon[28477]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:35.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:35 smithi077 ceph-mon[31615]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:35.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:35 smithi077 ceph-mon[31615]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:37.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:37 smithi007 ceph-mon[28477]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:37.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:37 smithi007 ceph-mon[28477]: from='client.? 172.21.15.16:0/1561441607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4b74336-65ed-4ff2-a634-217a0e7ce75c"}]: dispatch 2024-04-21T08:40:37.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:37 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4b74336-65ed-4ff2-a634-217a0e7ce75c"}]: dispatch 2024-04-21T08:40:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:37 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4b74336-65ed-4ff2-a634-217a0e7ce75c"}]': finished 2024-04-21T08:40:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:37 smithi007 ceph-mon[28477]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:40:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:37 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:37 smithi016 ceph-mon[31259]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:37 smithi016 ceph-mon[31259]: from='client.? 172.21.15.16:0/1561441607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4b74336-65ed-4ff2-a634-217a0e7ce75c"}]: dispatch 2024-04-21T08:40:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:37 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4b74336-65ed-4ff2-a634-217a0e7ce75c"}]: dispatch 2024-04-21T08:40:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:37 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4b74336-65ed-4ff2-a634-217a0e7ce75c"}]': finished 2024-04-21T08:40:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:37 smithi016 ceph-mon[31259]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:40:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:37 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:37.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:37 smithi077 ceph-mon[31615]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:37.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:37 smithi077 ceph-mon[31615]: from='client.? 172.21.15.16:0/1561441607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4b74336-65ed-4ff2-a634-217a0e7ce75c"}]: dispatch 2024-04-21T08:40:37.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:37 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d4b74336-65ed-4ff2-a634-217a0e7ce75c"}]: dispatch 2024-04-21T08:40:37.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:37 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d4b74336-65ed-4ff2-a634-217a0e7ce75c"}]': finished 2024-04-21T08:40:37.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:37 smithi077 ceph-mon[31615]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:40:37.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:37 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:38.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:38 smithi007 ceph-mon[28477]: from='client.? 172.21.15.16:0/1966817275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:38.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:38 smithi016 ceph-mon[31259]: from='client.? 172.21.15.16:0/1966817275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:38.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:38 smithi077 ceph-mon[31615]: from='client.? 172.21.15.16:0/1966817275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:39.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:39 smithi007 ceph-mon[28477]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:39.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:39 smithi016 ceph-mon[31259]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:39.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:39 smithi077 ceph-mon[31615]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:41.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:41 smithi007 ceph-mon[28477]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:41.802 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:41 smithi016 ceph-mon[31259]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:41.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:41 smithi077 ceph-mon[31615]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:43.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:43 smithi007 ceph-mon[28477]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:43.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:43 smithi016 ceph-mon[31259]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:43.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:43 smithi077 ceph-mon[31615]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:44.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:40:44.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:44.803 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:40:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:44.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:40:44.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:45.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:45 smithi007 ceph-mon[28477]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:45.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:45 smithi007 ceph-mon[28477]: Deploying daemon osd.10 on smithi016 2024-04-21T08:40:45.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:45 smithi016 ceph-mon[31259]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:45.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:45 smithi016 ceph-mon[31259]: Deploying daemon osd.10 on smithi016 2024-04-21T08:40:45.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:45 smithi077 ceph-mon[31615]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:45.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:45 smithi077 ceph-mon[31615]: Deploying daemon osd.10 on smithi016 2024-04-21T08:40:47.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:47 smithi007 ceph-mon[28477]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:47.754 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:47 smithi016 ceph-mon[31259]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:47.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:47 smithi077 ceph-mon[31615]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:49.628 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:49 smithi016 ceph-mon[31259]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:49.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:49 smithi077 ceph-mon[31615]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:49.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:49 smithi007 ceph-mon[28477]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:51.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:51 smithi007 ceph-mon[28477]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:51.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:51 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:51.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:51 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:51.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:51 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:51.655 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:51 smithi077 ceph-mon[31615]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:51.655 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:51 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:51.655 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:51 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:51.655 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:51 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:51.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:51 smithi016 ceph-mon[31259]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:51.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:51 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:51.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:51 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:51.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:51 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:53.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:53 smithi016 ceph-mon[31259]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:53.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:53 smithi007 ceph-mon[28477]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:53.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:53 smithi077 ceph-mon[31615]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:54.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.392+0000 7f54317fa700 1 -- 172.21.15.16:0/934381944 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f5434062e80 con 0x7f541c0a3c40 2024-04-21T08:40:54.392 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 10 on host 'smithi016' 2024-04-21T08:40:54.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 -- 172.21.15.16:0/934381944 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f541c0a3c40 msgr2=0x7f541c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:54.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f541c0a3c40 0x7f541c0a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f542400ac50 tx=0x7f542400a380 comp rx=0 tx=0).stop 2024-04-21T08:40:54.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 -- 172.21.15.16:0/934381944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5434108810 msgr2=0x7f54341a00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:54.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5434108810 0x7f54341a00a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f542c0062a0 tx=0x7f542c004e40 comp rx=0 tx=0).stop 2024-04-21T08:40:54.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 -- 172.21.15.16:0/934381944 shutdown_connections 2024-04-21T08:40:54.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f541c0a3c40 0x7f541c0a6100 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:54.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54341035b0 0x7f5434198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:54.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5434108810 0x7f54341a00a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:54.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 --2- 172.21.15.16:0/934381944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5434105ee0 0x7f5434199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:54.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 -- 172.21.15.16:0/934381944 >> 172.21.15.16:0/934381944 conn(0x7f54340fcf60 msgr2=0x7f5434103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:54.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 -- 172.21.15.16:0/934381944 shutdown_connections 2024-04-21T08:40:54.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-21T08:40:54.395+0000 7f543a42f700 1 -- 172.21.15.16:0/934381944 wait complete. 2024-04-21T08:40:54.788 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.789 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.789 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:54.789 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:54.789 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.789 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:54 smithi007 ceph-mon[28477]: from='osd.10 [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:40:54.789 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:54 smithi007 ceph-mon[28477]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:40:54.789 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.789 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.789 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:54 smithi016 ceph-mon[31259]: from='osd.10 [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:40:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:54 smithi016 ceph-mon[31259]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:40:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:54.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:54.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:54.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:54 smithi077 ceph-mon[31615]: from='osd.10 [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:40:54.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:54 smithi077 ceph-mon[31615]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:40:54.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:54.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:55.175 DEBUG:teuthology.orchestra.run.smithi016:osd.10> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.10.service 2024-04-21T08:40:55.177 INFO:tasks.cephadm:Deploying osd.11 on smithi077 with /dev/vg_nvme/lv_1... 2024-04-21T08:40:55.178 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:40:55.421 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:40:55.743 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:55 smithi016 ceph-mon[31259]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:55.743 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:55 smithi016 ceph-mon[31259]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:40:55.743 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:55 smithi016 ceph-mon[31259]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:40:55.743 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:55 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:55.743 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:55 smithi016 ceph-mon[31259]: from='osd.10 [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:40:55.743 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:55 smithi016 ceph-mon[31259]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:40:55.807 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:55 smithi077 ceph-mon[31615]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:55.807 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:55 smithi077 ceph-mon[31615]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:40:55.807 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:55 smithi077 ceph-mon[31615]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:40:55.807 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:55 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:55.807 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:55 smithi077 ceph-mon[31615]: from='osd.10 [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:40:55.808 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:55 smithi077 ceph-mon[31615]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:40:55.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:55 smithi007 ceph-mon[28477]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:55.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:55 smithi007 ceph-mon[28477]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:40:55.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:55 smithi007 ceph-mon[28477]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:40:55.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:55 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:55.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:55 smithi007 ceph-mon[28477]: from='osd.10 [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:40:55.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:55 smithi007 ceph-mon[28477]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-21T08:40:55.997 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:40:55 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:40:55.987+0000 7fe271abd700 -1 osd.10 0 waiting for initial osdmap 2024-04-21T08:40:55.998 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:40:55 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:40:55.997+0000 7fe2690a7700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:40:56.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:56 smithi016 ceph-mon[31259]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:40:56.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:56 smithi016 ceph-mon[31259]: osdmap e72: 11 total, 10 up, 11 in 2024-04-21T08:40:56.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:56 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:56.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:56 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:56.671 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:56 smithi016 ceph-mon[31259]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:56.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:56 smithi016 ceph-mon[31259]: from='osd.10 ' entity='osd.10' 2024-04-21T08:40:56.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:56 smithi016 ceph-mon[31259]: osd.10 [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333] boot 2024-04-21T08:40:56.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:56 smithi016 ceph-mon[31259]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:40:56.672 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:56 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:56.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:56 smithi077 ceph-mon[31615]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:40:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:56 smithi077 ceph-mon[31615]: osdmap e72: 11 total, 10 up, 11 in 2024-04-21T08:40:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:56 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:56 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:56 smithi077 ceph-mon[31615]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:56 smithi077 ceph-mon[31615]: from='osd.10 ' entity='osd.10' 2024-04-21T08:40:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:56 smithi077 ceph-mon[31615]: osd.10 [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333] boot 2024-04-21T08:40:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:56 smithi077 ceph-mon[31615]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:40:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:56 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:56.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:56 smithi007 ceph-mon[28477]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-21T08:40:56.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:56 smithi007 ceph-mon[28477]: osdmap e72: 11 total, 10 up, 11 in 2024-04-21T08:40:56.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:56 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:56 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:56 smithi007 ceph-mon[28477]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:40:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:56 smithi007 ceph-mon[28477]: from='osd.10 ' entity='osd.10' 2024-04-21T08:40:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:56 smithi007 ceph-mon[28477]: osd.10 [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333] boot 2024-04-21T08:40:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:56 smithi007 ceph-mon[28477]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:40:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:56 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:40:57.607 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:40:57.607 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:40:57.607 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-21T08:40:57.607 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-21T08:40:57.607 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131179 s, 799 MB/s 2024-04-21T08:40:57.607 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-21T08:40:58.196 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:58 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:40:58.196 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:58 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:40:58.196 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:58 smithi016 ceph-mon[31259]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:40:58.262 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:58 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:40:58.262 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:58 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:40:58.262 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:58 smithi077 ceph-mon[31615]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:40:58.292 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_1 2024-04-21T08:40:58.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:58 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:40:58.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:58 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:40:58.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:58 smithi007 ceph-mon[28477]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:40:58.528 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:40:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:40:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:40:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi016 2024-04-21T08:40:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:59.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:59.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi016 to 4273M 2024-04-21T08:40:59.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:59.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:59.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:59.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:40:59 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:59.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:40:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:40:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi016 2024-04-21T08:40:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi016 to 4273M 2024-04-21T08:40:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:59.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:59.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:59.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:40:59 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:59.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:40:59.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:40:59.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi016 2024-04-21T08:40:59.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi016 to 4273M 2024-04-21T08:40:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:40:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:40:59 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:00.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:00 smithi016 ceph-mon[31259]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:41:00.349 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:00 smithi077 ceph-mon[31615]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:41:00.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.352+0000 7f8e88570700 1 -- 172.21.15.77:0/2797406031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e80103540 msgr2=0x7f8e80109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.352+0000 7f8e88570700 1 --2- 172.21.15.77:0/2797406031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e80103540 0x7f8e80109df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8e7800b0a0 tx=0x7f8e7800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.352+0000 7f8e88570700 1 -- 172.21.15.77:0/2797406031 shutdown_connections 2024-04-21T08:41:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.352+0000 7f8e88570700 1 --2- 172.21.15.77:0/2797406031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e80103540 0x7f8e80109df0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.352+0000 7f8e88570700 1 --2- 172.21.15.77:0/2797406031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e801019a0 0x7f8e80101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.352+0000 7f8e88570700 1 --2- 172.21.15.77:0/2797406031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e80102ba0 0x7f8e80103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.352+0000 7f8e88570700 1 -- 172.21.15.77:0/2797406031 >> 172.21.15.77:0/2797406031 conn(0x7f8e800fcf40 msgr2=0x7f8e800ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.352+0000 7f8e88570700 1 -- 172.21.15.77:0/2797406031 shutdown_connections 2024-04-21T08:41:00.353 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.352+0000 7f8e88570700 1 -- 172.21.15.77:0/2797406031 wait complete. 2024-04-21T08:41:00.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e88570700 1 Processor -- start 2024-04-21T08:41:00.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e88570700 1 -- start start 2024-04-21T08:41:00.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e88570700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e801019a0 0x7f8e80198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e88570700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e80102ba0 0x7f8e80199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e88570700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e80103540 0x7f8e801a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e88570700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8010d530 con 0x7f8e80102ba0 2024-04-21T08:41:00.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e88570700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8010d3b0 con 0x7f8e801019a0 2024-04-21T08:41:00.356 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e88570700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8010d6b0 con 0x7f8e80103540 2024-04-21T08:41:00.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e86b0d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e80103540 0x7f8e801a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e86b0d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e80103540 0x7f8e801a0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51118/0 (socket says 172.21.15.77:51118) 2024-04-21T08:41:00.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e86b0d700 1 -- 172.21.15.77:0/160732397 learned_addr learned my addr 172.21.15.77:0/160732397 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-21T08:41:00.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e8630c700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e801019a0 0x7f8e80198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e85b0b700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e80102ba0 0x7f8e80199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e86b0d700 1 -- 172.21.15.77:0/160732397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e801019a0 msgr2=0x7f8e80198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e86b0d700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e801019a0 0x7f8e80198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e86b0d700 1 -- 172.21.15.77:0/160732397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e80102ba0 msgr2=0x7f8e80199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:00.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e86b0d700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e80102ba0 0x7f8e80199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:00.357 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e86b0d700 1 -- 172.21.15.77:0/160732397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e80103dc0 con 0x7f8e80103540 2024-04-21T08:41:00.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.353+0000 7f8e86b0d700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e80103540 0x7f8e801a0300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e7800ced0 tx=0x7f8e7800cf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:00.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.354+0000 7f8e777fe700 1 -- 172.21.15.77:0/160732397 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e78021070 con 0x7f8e80103540 2024-04-21T08:41:00.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.354+0000 7f8e88570700 1 -- 172.21.15.77:0/160732397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e80104050 con 0x7f8e80103540 2024-04-21T08:41:00.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.354+0000 7f8e777fe700 1 -- 172.21.15.77:0/160732397 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e78007d00 con 0x7f8e80103540 2024-04-21T08:41:00.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.354+0000 7f8e777fe700 1 -- 172.21.15.77:0/160732397 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e780052e0 con 0x7f8e80103540 2024-04-21T08:41:00.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.354+0000 7f8e88570700 1 -- 172.21.15.77:0/160732397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e80104660 con 0x7f8e80103540 2024-04-21T08:41:00.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.355+0000 7f8e88570700 1 -- 172.21.15.77:0/160732397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e80069730 con 0x7f8e80103540 2024-04-21T08:41:00.358 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.356+0000 7f8e777fe700 1 -- 172.21.15.77:0/160732397 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f8e78024070 con 0x7f8e80103540 2024-04-21T08:41:00.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.356+0000 7f8e777fe700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8e6c0a3cc0 0x7f8e6c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:00.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.356+0000 7f8e777fe700 1 -- 172.21.15.77:0/160732397 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6694+0+0 (secure 0 0 0) 0x7f8e7800f020 con 0x7f8e80103540 2024-04-21T08:41:00.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.356+0000 7f8e8630c700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8e6c0a3cc0 0x7f8e6c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:00.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.357+0000 7f8e8630c700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8e6c0a3cc0 0x7f8e6c0a6180 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8e80102a00 tx=0x7f8e70009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:00.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.361+0000 7f8e777fe700 1 -- 172.21.15.77:0/160732397 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8e7802b020 con 0x7f8e80103540 2024-04-21T08:41:00.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:00 smithi007 ceph-mon[28477]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:41:00.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:00.565+0000 7f8e88570700 1 -- 172.21.15.77:0/160732397 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f8e8006ae60 con 0x7f8e6c0a3cc0 2024-04-21T08:41:01.288 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:01 smithi077 ceph-mon[31615]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 359 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:41:01.288 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:01 smithi077 ceph-mon[31615]: osdmap e77: 11 total, 11 up, 11 in 2024-04-21T08:41:01.288 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:41:01.289 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:41:01.289 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:01.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:01 smithi016 ceph-mon[31259]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 359 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:41:01.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:01 smithi016 ceph-mon[31259]: osdmap e77: 11 total, 11 up, 11 in 2024-04-21T08:41:01.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:41:01.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:41:01.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:01.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:01 smithi007 ceph-mon[28477]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 359 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:41:01.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:01 smithi007 ceph-mon[28477]: osdmap e77: 11 total, 11 up, 11 in 2024-04-21T08:41:01.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:41:01.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:41:01.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:02.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:02 smithi016 ceph-mon[31259]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:02.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:02 smithi077 ceph-mon[31615]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:02.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:02 smithi007 ceph-mon[28477]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:03 smithi016 ceph-mon[31259]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-21T08:41:03.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:03 smithi077 ceph-mon[31615]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-21T08:41:03.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:03 smithi007 ceph-mon[28477]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-21T08:41:04.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:04 smithi016 ceph-mon[31259]: from='client.? 172.21.15.77:0/2616806260' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c"}]: dispatch 2024-04-21T08:41:04.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:04 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c"}]: dispatch 2024-04-21T08:41:04.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:04 smithi016 ceph-mon[31259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c"}]': finished 2024-04-21T08:41:04.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:04 smithi016 ceph-mon[31259]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:41:04.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:04.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:04 smithi077 ceph-mon[31615]: from='client.? 172.21.15.77:0/2616806260' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c"}]: dispatch 2024-04-21T08:41:04.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:04 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c"}]: dispatch 2024-04-21T08:41:04.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:04 smithi077 ceph-mon[31615]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c"}]': finished 2024-04-21T08:41:04.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:04 smithi077 ceph-mon[31615]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:41:04.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:04.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:04 smithi007 ceph-mon[28477]: from='client.? 172.21.15.77:0/2616806260' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c"}]: dispatch 2024-04-21T08:41:04.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:04 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c"}]: dispatch 2024-04-21T08:41:04.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:04 smithi007 ceph-mon[28477]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c"}]': finished 2024-04-21T08:41:04.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:04 smithi007 ceph-mon[28477]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:41:04.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:05.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:05 smithi016 ceph-mon[31259]: pgmap v245: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-21T08:41:05.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:05 smithi016 ceph-mon[31259]: from='client.? 172.21.15.77:0/4235687132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:41:05.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:05 smithi077 ceph-mon[31615]: pgmap v245: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-21T08:41:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:05 smithi077 ceph-mon[31615]: from='client.? 172.21.15.77:0/4235687132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:41:05.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:05 smithi007 ceph-mon[28477]: pgmap v245: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-21T08:41:05.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:05 smithi007 ceph-mon[28477]: from='client.? 172.21.15.77:0/4235687132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:41:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:07 smithi016 ceph-mon[31259]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:07 smithi077 ceph-mon[31615]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:07 smithi007 ceph-mon[28477]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:09.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:09 smithi016 ceph-mon[31259]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:09.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:09 smithi077 ceph-mon[31615]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:09.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:09 smithi007 ceph-mon[28477]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:11.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:11 smithi077 ceph-mon[31615]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:11.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:11 smithi007 ceph-mon[28477]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:11.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:11 smithi016 ceph-mon[31259]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:12.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:12 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:41:12.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:12 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:12.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:12 smithi077 ceph-mon[31615]: Deploying daemon osd.11 on smithi077 2024-04-21T08:41:12.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:41:12.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:12.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:12 smithi007 ceph-mon[28477]: Deploying daemon osd.11 on smithi077 2024-04-21T08:41:12.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:12 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:41:12.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:12 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:12.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:12 smithi016 ceph-mon[31259]: Deploying daemon osd.11 on smithi077 2024-04-21T08:41:13.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:13 smithi077 ceph-mon[31615]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:13.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:13 smithi007 ceph-mon[28477]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:13.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:13 smithi016 ceph-mon[31259]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:15.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:15 smithi077 ceph-mon[31615]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:15.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:15 smithi007 ceph-mon[28477]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:15.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:15 smithi016 ceph-mon[31259]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:17.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:17 smithi077 ceph-mon[31615]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:17.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:17 smithi007 ceph-mon[28477]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:17.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:17 smithi016 ceph-mon[31259]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:18.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:18.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:18.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:18.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:18.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:18.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:18.655 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:18.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:18.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:19.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:19 smithi077 ceph-mon[31615]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:19.553 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:19 smithi016 ceph-mon[31259]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:19.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:19 smithi007 ceph-mon[28477]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:20.939 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 11 on host 'smithi077' 2024-04-21T08:41:20.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.939+0000 7f8e777fe700 1 -- 172.21.15.77:0/160732397 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f8e8006ae60 con 0x7f8e6c0a3cc0 2024-04-21T08:41:20.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 -- 172.21.15.77:0/160732397 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8e6c0a3cc0 msgr2=0x7f8e6c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:20.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8e6c0a3cc0 0x7f8e6c0a6180 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8e80102a00 tx=0x7f8e70009190 comp rx=0 tx=0).stop 2024-04-21T08:41:20.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 -- 172.21.15.77:0/160732397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e80103540 msgr2=0x7f8e801a0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:20.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e80103540 0x7f8e801a0300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e7800ced0 tx=0x7f8e7800cf00 comp rx=0 tx=0).stop 2024-04-21T08:41:20.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 -- 172.21.15.77:0/160732397 shutdown_connections 2024-04-21T08:41:20.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8e6c0a3cc0 0x7f8e6c0a6180 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:20.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8e80103540 0x7f8e801a0300 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:20.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e801019a0 0x7f8e80198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:20.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 --2- 172.21.15.77:0/160732397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e80102ba0 0x7f8e80199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:20.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 -- 172.21.15.77:0/160732397 >> 172.21.15.77:0/160732397 conn(0x7f8e800fcf40 msgr2=0x7f8e8006a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:20.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 -- 172.21.15.77:0/160732397 shutdown_connections 2024-04-21T08:41:20.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:20.942+0000 7f8e88570700 1 -- 172.21.15.77:0/160732397 wait complete. 2024-04-21T08:41:21.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:21 smithi016 ceph-mon[31259]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:21.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:21 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:21.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:21 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:21.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:21 smithi016 ceph-mon[31259]: from='osd.11 [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:41:21.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:21 smithi016 ceph-mon[31259]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:41:21.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:21 smithi077 ceph-mon[31615]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:21.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:21 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:21.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:21 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:21.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:21 smithi077 ceph-mon[31615]: from='osd.11 [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:41:21.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:21 smithi077 ceph-mon[31615]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:41:21.650 DEBUG:teuthology.orchestra.run.smithi077:osd.11> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.11.service 2024-04-21T08:41:21.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:21 smithi007 ceph-mon[28477]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:21.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:21 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:21.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:21 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:21.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:21 smithi007 ceph-mon[28477]: from='osd.11 [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:41:21.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:21 smithi007 ceph-mon[28477]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:41:21.653 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-21T08:41:21.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-21T08:41:21.926 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:41:23.024 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:22 smithi077 ceph-mon[31615]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:23.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:23.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:23.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:22 smithi077 ceph-mon[31615]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:41:23.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:22 smithi077 ceph-mon[31615]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:41:23.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:22 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:23.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:22 smithi077 ceph-mon[31615]: from='osd.11 [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:41:23.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:22 smithi077 ceph-mon[31615]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:41:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:22 smithi016 ceph-mon[31259]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:22 smithi016 ceph-mon[31259]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:41:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:22 smithi016 ceph-mon[31259]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:41:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:22 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:22 smithi016 ceph-mon[31259]: from='osd.11 [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:41:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:22 smithi016 ceph-mon[31259]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:41:23.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:22 smithi007 ceph-mon[28477]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:23.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:23.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:23.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:22 smithi007 ceph-mon[28477]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:41:23.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:22 smithi007 ceph-mon[28477]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:41:23.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:22 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:23.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:22 smithi007 ceph-mon[28477]: from='osd.11 [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:41:23.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:22 smithi007 ceph-mon[28477]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-21T08:41:23.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.652+0000 7f582d310700 1 -- 172.21.15.7:0/3091470688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5828102ba0 msgr2=0x7f5828103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:23.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.652+0000 7f582d310700 1 --2- 172.21.15.7:0/3091470688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5828102ba0 0x7f5828103000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f58100099c0 tx=0x7f5810009cd0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.652+0000 7f582d310700 1 -- 172.21.15.7:0/3091470688 shutdown_connections 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.652+0000 7f582d310700 1 --2- 172.21.15.7:0/3091470688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5828103540 0x7f5828109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.652+0000 7f582d310700 1 --2- 172.21.15.7:0/3091470688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58281019a0 0x7f5828101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.652+0000 7f582d310700 1 --2- 172.21.15.7:0/3091470688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5828102ba0 0x7f5828103000 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.652+0000 7f582d310700 1 -- 172.21.15.7:0/3091470688 >> 172.21.15.7:0/3091470688 conn(0x7f58280fcf40 msgr2=0x7f58280ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.652+0000 7f582d310700 1 -- 172.21.15.7:0/3091470688 shutdown_connections 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.652+0000 7f582d310700 1 -- 172.21.15.7:0/3091470688 wait complete. 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.652+0000 7f582d310700 1 Processor -- start 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f582d310700 1 -- start start 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f582d310700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58281019a0 0x7f5828105930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f582d310700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5828102ba0 0x7f5828103f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f582d310700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5828103540 0x7f58281044c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:23.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f582d310700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f582810d410 con 0x7f5828103540 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f582d310700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f582810d290 con 0x7f58281019a0 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f582d310700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f582810d590 con 0x7f5828102ba0 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f58267fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5828102ba0 0x7f5828103f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f58277fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5828103540 0x7f58281044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f58267fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5828102ba0 0x7f5828103f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:33072/0 (socket says 172.21.15.7:33072) 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f58267fc700 1 -- 172.21.15.7:0/1613899252 learned_addr learned my addr 172.21.15.7:0/1613899252 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f5826ffd700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58281019a0 0x7f5828105930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f58277fe700 1 -- 172.21.15.7:0/1613899252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5828102ba0 msgr2=0x7f5828103f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f58277fe700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5828102ba0 0x7f5828103f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f58277fe700 1 -- 172.21.15.7:0/1613899252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58281019a0 msgr2=0x7f5828105930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f58277fe700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58281019a0 0x7f5828105930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f58277fe700 1 -- 172.21.15.7:0/1613899252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5828104d70 con 0x7f5828103540 2024-04-21T08:41:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f58267fc700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5828102ba0 0x7f5828103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:41:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f5826ffd700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58281019a0 0x7f5828105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.653+0000 7f58277fe700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5828103540 0x7f58281044c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f582000e970 tx=0x7f582000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.654+0000 7f581ffff700 1 -- 172.21.15.7:0/1613899252 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582000cbd0 con 0x7f5828103540 2024-04-21T08:41:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.654+0000 7f581ffff700 1 -- 172.21.15.7:0/1613899252 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f582000ee60 con 0x7f5828103540 2024-04-21T08:41:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.654+0000 7f582d310700 1 -- 172.21.15.7:0/1613899252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58281a2dc0 con 0x7f5828103540 2024-04-21T08:41:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.654+0000 7f581ffff700 1 -- 172.21.15.7:0/1613899252 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5820005020 con 0x7f5828103540 2024-04-21T08:41:23.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.654+0000 7f582d310700 1 -- 172.21.15.7:0/1613899252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58281a3220 con 0x7f5828103540 2024-04-21T08:41:23.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.655+0000 7f582d310700 1 -- 172.21.15.7:0/1613899252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5828069760 con 0x7f5828103540 2024-04-21T08:41:23.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.656+0000 7f581ffff700 1 -- 172.21.15.7:0/1613899252 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f58200051c0 con 0x7f5828103540 2024-04-21T08:41:23.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.657+0000 7f581ffff700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f58140a3c40 0x7f58140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:23.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.657+0000 7f5826ffd700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f58140a3c40 0x7f58140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:23.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.657+0000 7f581ffff700 1 -- 172.21.15.7:0/1613899252 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7f58200cfe10 con 0x7f5828103540 2024-04-21T08:41:23.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.657+0000 7f5826ffd700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f58140a3c40 0x7f58140a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5828102a00 tx=0x7f5818009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.661+0000 7f581ffff700 1 -- 172.21.15.7:0/1613899252 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5828069760 con 0x7f5828103540 2024-04-21T08:41:23.739 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:41:23 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:41:23.402+0000 7f9b8009a700 -1 osd.11 0 waiting for initial osdmap 2024-04-21T08:41:23.739 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:41:23 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:41:23.418+0000 7f9b77e85700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:41:23.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.867+0000 7f582d310700 1 -- 172.21.15.7:0/1613899252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f58281a3750 con 0x7f5828103540 2024-04-21T08:41:23.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.868+0000 7f581ffff700 1 -- 172.21.15.7:0/1613899252 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f582009cda0 con 0x7f5828103540 2024-04-21T08:41:23.867 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:23.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 -- 172.21.15.7:0/1613899252 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f58140a3c40 msgr2=0x7f58140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:23.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f58140a3c40 0x7f58140a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5828102a00 tx=0x7f5818009040 comp rx=0 tx=0).stop 2024-04-21T08:41:23.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 -- 172.21.15.7:0/1613899252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5828103540 msgr2=0x7f58281044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:23.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5828103540 0x7f58281044c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f582000e970 tx=0x7f582000ec80 comp rx=0 tx=0).stop 2024-04-21T08:41:23.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 -- 172.21.15.7:0/1613899252 shutdown_connections 2024-04-21T08:41:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f58140a3c40 0x7f58140a6100 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5828103540 0x7f58281044c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f58281019a0 0x7f5828105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 --2- 172.21.15.7:0/1613899252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5828102ba0 0x7f5828103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 -- 172.21.15.7:0/1613899252 >> 172.21.15.7:0/1613899252 conn(0x7f58280fcf40 msgr2=0x7f582806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 -- 172.21.15.7:0/1613899252 shutdown_connections 2024-04-21T08:41:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:23.870+0000 7f582d310700 1 -- 172.21.15.7:0/1613899252 wait complete. 2024-04-21T08:41:23.989 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:23 smithi077 ceph-mon[31615]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:41:23.989 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:23 smithi077 ceph-mon[31615]: osdmap e80: 12 total, 11 up, 12 in 2024-04-21T08:41:23.989 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:23.989 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:23 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:23.989 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:23 smithi077 ceph-mon[31615]: from='osd.11 ' entity='osd.11' 2024-04-21T08:41:23.989 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:23 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1613899252' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:41:24.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:23 smithi007 ceph-mon[28477]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:41:24.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:23 smithi007 ceph-mon[28477]: osdmap e80: 12 total, 11 up, 12 in 2024-04-21T08:41:24.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:24.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:23 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:24.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:23 smithi007 ceph-mon[28477]: from='osd.11 ' entity='osd.11' 2024-04-21T08:41:24.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:23 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1613899252' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:41:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:23 smithi016 ceph-mon[31259]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-21T08:41:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:23 smithi016 ceph-mon[31259]: osdmap e80: 12 total, 11 up, 12 in 2024-04-21T08:41:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:23 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:23 smithi016 ceph-mon[31259]: from='osd.11 ' entity='osd.11' 2024-04-21T08:41:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:23 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1613899252' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:41:24.512 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1713688856,"num_in_osds":12,"osd_in_since":1713688863,"num_remapped_pgs":0} 2024-04-21T08:41:25.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:24 smithi007 ceph-mon[28477]: purged_snaps scrub starts 2024-04-21T08:41:25.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:24 smithi007 ceph-mon[28477]: purged_snaps scrub ok 2024-04-21T08:41:25.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:24 smithi007 ceph-mon[28477]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:25.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:25.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:24 smithi007 ceph-mon[28477]: osd.11 [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303] boot 2024-04-21T08:41:25.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:24 smithi007 ceph-mon[28477]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:41:25.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:24 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:25.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:24 smithi016 ceph-mon[31259]: purged_snaps scrub starts 2024-04-21T08:41:25.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:24 smithi016 ceph-mon[31259]: purged_snaps scrub ok 2024-04-21T08:41:25.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:24 smithi016 ceph-mon[31259]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:25.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:25.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:24 smithi016 ceph-mon[31259]: osd.11 [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303] boot 2024-04-21T08:41:25.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:24 smithi016 ceph-mon[31259]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:41:25.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:24 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:25.310 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:24 smithi077 ceph-mon[31615]: purged_snaps scrub starts 2024-04-21T08:41:25.310 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:24 smithi077 ceph-mon[31615]: purged_snaps scrub ok 2024-04-21T08:41:25.310 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:24 smithi077 ceph-mon[31615]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:25.311 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:25.311 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:24 smithi077 ceph-mon[31615]: osd.11 [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303] boot 2024-04-21T08:41:25.311 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:24 smithi077 ceph-mon[31615]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:41:25.311 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:24 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:25.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-21T08:41:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:41:26.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: Detected new or changed devices on smithi077 2024-04-21T08:41:26.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi077 to 4272M 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:26.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:26 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: Detected new or changed devices on smithi077 2024-04-21T08:41:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi077 to 4272M 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:26.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:26 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: Detected new or changed devices on smithi077 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi077 to 4272M 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:26.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:41:26.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:26.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:26.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:26 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.427+0000 7f533aef5700 1 -- 172.21.15.7:0/1604710377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5334101990 msgr2=0x7f5334101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.427+0000 7f533aef5700 1 --2- 172.21.15.7:0/1604710377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5334101990 0x7f5334101db0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f53280099c0 tx=0x7f5328009cd0 comp rx=0 tx=0).stop 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.427+0000 7f533aef5700 1 -- 172.21.15.7:0/1604710377 shutdown_connections 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.427+0000 7f533aef5700 1 --2- 172.21.15.7:0/1604710377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5334103530 0x7f5334109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.427+0000 7f533aef5700 1 --2- 172.21.15.7:0/1604710377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5334101990 0x7f5334101db0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.427+0000 7f533aef5700 1 --2- 172.21.15.7:0/1604710377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5334102b90 0x7f5334102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.427+0000 7f533aef5700 1 -- 172.21.15.7:0/1604710377 >> 172.21.15.7:0/1604710377 conn(0x7f53340fcf30 msgr2=0x7f53340ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.427+0000 7f533aef5700 1 -- 172.21.15.7:0/1604710377 shutdown_connections 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.427+0000 7f533aef5700 1 -- 172.21.15.7:0/1604710377 wait complete. 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f533aef5700 1 Processor -- start 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f533aef5700 1 -- start start 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f533aef5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5334101990 0x7f53341058d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f533aef5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5334102b90 0x7f5334103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f533aef5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5334103530 0x7f53341044b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f533aef5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f533410d420 con 0x7f5334103530 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f533aef5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f533410d2a0 con 0x7f5334102b90 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f533aef5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f533410d5a0 con 0x7f5334101990 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f5339492700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5334103530 0x7f53341044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f5333fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5334102b90 0x7f5334103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f5339492700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5334103530 0x7f53341044b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35134/0 (socket says 172.21.15.7:35134) 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f5339492700 1 -- 172.21.15.7:0/4003604719 learned_addr learned my addr 172.21.15.7:0/4003604719 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.428+0000 7f5338c91700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5334101990 0x7f53341058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.429+0000 7f5339492700 1 -- 172.21.15.7:0/4003604719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5334101990 msgr2=0x7f53341058d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.429+0000 7f5339492700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5334101990 0x7f53341058d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.429+0000 7f5339492700 1 -- 172.21.15.7:0/4003604719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5334102b90 msgr2=0x7f5334103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.429+0000 7f5339492700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5334102b90 0x7f5334103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.429+0000 7f5339492700 1 -- 172.21.15.7:0/4003604719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5334104d80 con 0x7f5334103530 2024-04-21T08:41:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.429+0000 7f5339492700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5334103530 0x7f53341044b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f532c007ee0 tx=0x7f532c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.429+0000 7f5331ffb700 1 -- 172.21.15.7:0/4003604719 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c004bb0 con 0x7f5334103530 2024-04-21T08:41:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.429+0000 7f5331ffb700 1 -- 172.21.15.7:0/4003604719 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f532c004d50 con 0x7f5334103530 2024-04-21T08:41:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.429+0000 7f533aef5700 1 -- 172.21.15.7:0/4003604719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53341a2d10 con 0x7f5334103530 2024-04-21T08:41:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.429+0000 7f5331ffb700 1 -- 172.21.15.7:0/4003604719 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c005920 con 0x7f5334103530 2024-04-21T08:41:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.429+0000 7f533aef5700 1 -- 172.21.15.7:0/4003604719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53341a3170 con 0x7f5334103530 2024-04-21T08:41:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.430+0000 7f533aef5700 1 -- 172.21.15.7:0/4003604719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53341a3400 con 0x7f5334103530 2024-04-21T08:41:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.431+0000 7f5331ffb700 1 -- 172.21.15.7:0/4003604719 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f532c01a020 con 0x7f5334103530 2024-04-21T08:41:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.432+0000 7f5331ffb700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f53240a3c40 0x7f53240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.432+0000 7f5331ffb700 1 -- 172.21.15.7:0/4003604719 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f532c0cf5b0 con 0x7f5334103530 2024-04-21T08:41:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.432+0000 7f5338c91700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f53240a3c40 0x7f53240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.432+0000 7f5338c91700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f53240a3c40 0x7f53240a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f53280038c0 tx=0x7f532800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:27.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.437+0000 7f5331ffb700 1 -- 172.21.15.7:0/4003604719 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f532c010070 con 0x7f5334103530 2024-04-21T08:41:27.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.648+0000 7f533aef5700 1 -- 172.21.15.7:0/4003604719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f53341a3ba0 con 0x7f5334103530 2024-04-21T08:41:27.648 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:27 smithi007 ceph-mon[28477]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:27.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.649+0000 7f5331ffb700 1 -- 172.21.15.7:0/4003604719 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f532c09c430 con 0x7f5334103530 2024-04-21T08:41:27.649 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:27.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 -- 172.21.15.7:0/4003604719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f53240a3c40 msgr2=0x7f53240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:27.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f53240a3c40 0x7f53240a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f53280038c0 tx=0x7f532800b540 comp rx=0 tx=0).stop 2024-04-21T08:41:27.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 -- 172.21.15.7:0/4003604719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5334103530 msgr2=0x7f53341044b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:27.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5334103530 0x7f53341044b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f532c007ee0 tx=0x7f532c00d3b0 comp rx=0 tx=0).stop 2024-04-21T08:41:27.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 -- 172.21.15.7:0/4003604719 shutdown_connections 2024-04-21T08:41:27.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f53240a3c40 0x7f53240a6100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:27.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5334101990 0x7f53341058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:27.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5334103530 0x7f53341044b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:27.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 --2- 172.21.15.7:0/4003604719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5334102b90 0x7f5334103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:27.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 -- 172.21.15.7:0/4003604719 >> 172.21.15.7:0/4003604719 conn(0x7f53340fcf30 msgr2=0x7f533406af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:27.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 -- 172.21.15.7:0/4003604719 shutdown_connections 2024-04-21T08:41:27.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:27.652+0000 7f533aef5700 1 -- 172.21.15.7:0/4003604719 wait complete. 2024-04-21T08:41:27.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:27 smithi016 ceph-mon[31259]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:27.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:27 smithi077 ceph-mon[31615]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:28.363 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1713688884,"num_in_osds":12,"osd_in_since":1713688863,"num_remapped_pgs":0} 2024-04-21T08:41:28.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:41:28.610 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:41:28.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:28 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4003604719' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:41:28.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:28 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4003604719' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:41:28.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:28 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4003604719' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:41:29.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:29 smithi007 ceph-mon[28477]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:29 smithi016 ceph-mon[31259]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:29.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:29 smithi077 ceph-mon[31615]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.200+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/906078053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc108a60 msgr2=0x7ff5dc108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.200+0000 7ff5e3b9a700 1 --2- 172.21.15.7:0/906078053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc108a60 0x7ff5dc108e40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8009990 tx=0x7ff5d8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.200+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/906078053 shutdown_connections 2024-04-21T08:41:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.200+0000 7ff5e3b9a700 1 --2- 172.21.15.7:0/906078053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5dc06bbb0 0x7ff5dc10f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.200+0000 7ff5e3b9a700 1 --2- 172.21.15.7:0/906078053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc108a60 0x7ff5dc108e40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.200+0000 7ff5e3b9a700 1 --2- 172.21.15.7:0/906078053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc06b1f0 0x7ff5dc06b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.200+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/906078053 >> 172.21.15.7:0/906078053 conn(0x7ff5dc0789e0 msgr2=0x7ff5dc078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.200+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/906078053 shutdown_connections 2024-04-21T08:41:30.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.200+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/906078053 wait complete. 2024-04-21T08:41:30.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e3b9a700 1 Processor -- start 2024-04-21T08:41:30.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e3b9a700 1 -- start start 2024-04-21T08:41:30.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e3b9a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc06b1f0 0x7ff5dc1991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:30.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e3b9a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5dc06bbb0 0x7ff5dc1996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e3b9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc108a60 0x7ff5dc19da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e3b9a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc112c40 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e3b9a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc112ac0 con 0x7ff5dc06bbb0 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e3b9a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc112dc0 con 0x7ff5dc108a60 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e1936700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc06b1f0 0x7ff5dc1991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e1936700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc06b1f0 0x7ff5dc1991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35150/0 (socket says 172.21.15.7:35150) 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e1936700 1 -- 172.21.15.7:0/1416700867 learned_addr learned my addr 172.21.15.7:0/1416700867 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e1135700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5dc06bbb0 0x7ff5dc1996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.201+0000 7ff5e2137700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc108a60 0x7ff5dc19da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5e1936700 1 -- 172.21.15.7:0/1416700867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc108a60 msgr2=0x7ff5dc19da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5e1936700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc108a60 0x7ff5dc19da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5e1936700 1 -- 172.21.15.7:0/1416700867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5dc06bbb0 msgr2=0x7ff5dc1996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5e1936700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5dc06bbb0 0x7ff5dc1996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5e1936700 1 -- 172.21.15.7:0/1416700867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5dc19e1c0 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5e2137700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc108a60 0x7ff5dc19da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:30.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5e1936700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc06b1f0 0x7ff5dc1991b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8000c00 tx=0x7ff5d8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:30.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5d2ffd700 1 -- 172.21.15.7:0/1416700867 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8027070 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/1416700867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5dc19e450 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5d2ffd700 1 -- 172.21.15.7:0/1416700867 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d8022470 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5d2ffd700 1 -- 172.21.15.7:0/1416700867 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8017650 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.202+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/1416700867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5dc1a5d30 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.203+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/1416700867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5dc10b060 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.204+0000 7ff5d2ffd700 1 -- 172.21.15.7:0/1416700867 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7ff5d802a070 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.205+0000 7ff5d2ffd700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5c80a3c40 0x7ff5c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.205+0000 7ff5e1135700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5c80a3c40 0x7ff5c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.205+0000 7ff5d2ffd700 1 -- 172.21.15.7:0/1416700867 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff5d80d0d10 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.205+0000 7ff5e1135700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5c80a3c40 0x7ff5c80a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc079000 tx=0x7ff5cc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:30.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.209+0000 7ff5d2ffd700 1 -- 172.21.15.7:0/1416700867 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff5d8013070 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.418+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/1416700867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5dc19a510 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.419+0000 7ff5d2ffd700 1 -- 172.21.15.7:0/1416700867 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18289 (secure 0 0 0) 0x7ff5d809dc20 con 0x7ff5dc06b1f0 2024-04-21T08:41:30.418 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:30.419 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":82,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:41:25.912790+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:41:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/1416700867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5c80a3c40 msgr2=0x7ff5c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:30.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5c80a3c40 0x7ff5c80a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc079000 tx=0x7ff5cc0073d0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/1416700867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc06b1f0 msgr2=0x7ff5dc1991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc06b1f0 0x7ff5dc1991b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8000c00 tx=0x7ff5d8009580 comp rx=0 tx=0).stop 2024-04-21T08:41:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/1416700867 shutdown_connections 2024-04-21T08:41:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff5c80a3c40 0x7ff5c80a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc108a60 0x7ff5dc19da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc06b1f0 0x7ff5dc1991b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 --2- 172.21.15.7:0/1416700867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5dc06bbb0 0x7ff5dc1996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/1416700867 >> 172.21.15.7:0/1416700867 conn(0x7ff5dc0789e0 msgr2=0x7ff5dc0ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/1416700867 shutdown_connections 2024-04-21T08:41:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:30.421+0000 7ff5e3b9a700 1 -- 172.21.15.7:0/1416700867 wait complete. 2024-04-21T08:41:31.101 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:41:31.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-21T08:41:31.350 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:41:31.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:31 smithi016 ceph-mon[31259]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:31.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:31 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1416700867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:41:31.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:31 smithi007 ceph-mon[28477]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:31.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:31 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1416700867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:41:31.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:31 smithi077 ceph-mon[31615]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:31.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:31 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1416700867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:41:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.084+0000 7f4cfe4de700 1 -- 172.21.15.7:0/910262314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cf8108a50 msgr2=0x7f4cf8108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.084+0000 7f4cfe4de700 1 --2- 172.21.15.7:0/910262314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cf8108a50 0x7f4cf8108e30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009990 tx=0x7f4ce8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:41:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.084+0000 7f4cfe4de700 1 -- 172.21.15.7:0/910262314 shutdown_connections 2024-04-21T08:41:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.084+0000 7f4cfe4de700 1 --2- 172.21.15.7:0/910262314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf8103450 0x7f4cf810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.084+0000 7f4cfe4de700 1 --2- 172.21.15.7:0/910262314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cf8108a50 0x7f4cf8108e30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.084+0000 7f4cfe4de700 1 --2- 172.21.15.7:0/910262314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8102a90 0x7f4cf8102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.084+0000 7f4cfe4de700 1 -- 172.21.15.7:0/910262314 >> 172.21.15.7:0/910262314 conn(0x7f4cf8079ba0 msgr2=0x7f4cf8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.084+0000 7f4cfe4de700 1 -- 172.21.15.7:0/910262314 shutdown_connections 2024-04-21T08:41:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.084+0000 7f4cfe4de700 1 -- 172.21.15.7:0/910262314 wait complete. 2024-04-21T08:41:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.085+0000 7f4cfe4de700 1 Processor -- start 2024-04-21T08:41:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.085+0000 7f4cfe4de700 1 -- start start 2024-04-21T08:41:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.085+0000 7f4cfe4de700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf8102a90 0x7f4cf81991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfe4de700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8103450 0x7f4cf8199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfe4de700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cf8108a50 0x7f4cf819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfe4de700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf8112c00 con 0x7f4cf8108a50 2024-04-21T08:41:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfe4de700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf8112a80 con 0x7f4cf8102a90 2024-04-21T08:41:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfe4de700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf8112d80 con 0x7f4cf8103450 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfca7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cf8108a50 0x7f4cf819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfca7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cf8108a50 0x7f4cf819dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35166/0 (socket says 172.21.15.7:35166) 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cf77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8103450 0x7f4cf8199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfca7b700 1 -- 172.21.15.7:0/611289931 learned_addr learned my addr 172.21.15.7:0/611289931 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cf7fff700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf8102a90 0x7f4cf81991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfca7b700 1 -- 172.21.15.7:0/611289931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8103450 msgr2=0x7f4cf8199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfca7b700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8103450 0x7f4cf8199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfca7b700 1 -- 172.21.15.7:0/611289931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf8102a90 msgr2=0x7f4cf81991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfca7b700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf8102a90 0x7f4cf81991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfca7b700 1 -- 172.21.15.7:0/611289931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf819e200 con 0x7f4cf8108a50 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.086+0000 7f4cfca7b700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cf8108a50 0x7f4cf819dac0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000ada0 tx=0x7f4cf000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.087+0000 7f4cf57fa700 1 -- 172.21.15.7:0/611289931 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0017020 con 0x7f4cf8108a50 2024-04-21T08:41:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.087+0000 7f4cfe4de700 1 -- 172.21.15.7:0/611289931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf819e4f0 con 0x7f4cf8108a50 2024-04-21T08:41:33.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.087+0000 7f4cfe4de700 1 -- 172.21.15.7:0/611289931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf81a5d70 con 0x7f4cf8108a50 2024-04-21T08:41:33.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.087+0000 7f4cf57fa700 1 -- 172.21.15.7:0/611289931 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cf000cd80 con 0x7f4cf8108a50 2024-04-21T08:41:33.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.087+0000 7f4cf57fa700 1 -- 172.21.15.7:0/611289931 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0005280 con 0x7f4cf8108a50 2024-04-21T08:41:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.088+0000 7f4cfe4de700 1 -- 172.21.15.7:0/611289931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cf8069730 con 0x7f4cf8108a50 2024-04-21T08:41:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.089+0000 7f4cf57fa700 1 -- 172.21.15.7:0/611289931 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f4cf000c8d0 con 0x7f4cf8108a50 2024-04-21T08:41:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.090+0000 7f4cf57fa700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ce00a3cc0 0x7f4ce00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.090+0000 7f4cf7fff700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ce00a3cc0 0x7f4ce00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.090+0000 7f4cf57fa700 1 -- 172.21.15.7:0/611289931 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4cf00cf520 con 0x7f4cf8108a50 2024-04-21T08:41:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.090+0000 7f4cf7fff700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ce00a3cc0 0x7f4ce00a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80098e0 tx=0x7f4ce8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:33.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.095+0000 7f4cf57fa700 1 -- 172.21.15.7:0/611289931 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4cf0021020 con 0x7f4cf8108a50 2024-04-21T08:41:33.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.302+0000 7f4cfe4de700 1 -- 172.21.15.7:0/611289931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4cf802df80 con 0x7f4cf8108a50 2024-04-21T08:41:33.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.302+0000 7f4cf57fa700 1 -- 172.21.15.7:0/611289931 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f4cf009c430 con 0x7f4cf8108a50 2024-04-21T08:41:33.302 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.304+0000 7f4cfe4de700 1 -- 172.21.15.7:0/611289931 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ce00a3cc0 msgr2=0x7f4ce00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.304+0000 7f4cfe4de700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ce00a3cc0 0x7f4ce00a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80098e0 tx=0x7f4ce8005960 comp rx=0 tx=0).stop 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.304+0000 7f4cfe4de700 1 -- 172.21.15.7:0/611289931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cf8108a50 msgr2=0x7f4cf819dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.304+0000 7f4cfe4de700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cf8108a50 0x7f4cf819dac0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000ada0 tx=0x7f4cf000c330 comp rx=0 tx=0).stop 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.304+0000 7f4cfe4de700 1 -- 172.21.15.7:0/611289931 shutdown_connections 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.304+0000 7f4cfe4de700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ce00a3cc0 0x7f4ce00a6180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.304+0000 7f4cfe4de700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cf8108a50 0x7f4cf819dac0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.304+0000 7f4cfe4de700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cf8102a90 0x7f4cf81991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.304+0000 7f4cfe4de700 1 --2- 172.21.15.7:0/611289931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8103450 0x7f4cf8199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.304+0000 7f4cfe4de700 1 -- 172.21.15.7:0/611289931 >> 172.21.15.7:0/611289931 conn(0x7f4cf8079ba0 msgr2=0x7f4cf81019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.304+0000 7f4cfe4de700 1 -- 172.21.15.7:0/611289931 shutdown_connections 2024-04-21T08:41:33.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:33.305+0000 7f4cfe4de700 1 -- 172.21.15.7:0/611289931 wait complete. 2024-04-21T08:41:33.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:33 smithi077 ceph-mon[31615]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:33.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:33 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/611289931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:41:33.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:33 smithi007 ceph-mon[28477]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:33.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:33 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/611289931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:41:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:33 smithi016 ceph-mon[31259]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:33 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/611289931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:41:33.978 INFO:tasks.cephadm:Adding mds.a on smithi007 2024-04-21T08:41:33.978 INFO:tasks.cephadm:Adding mds.d on smithi007 2024-04-21T08:41:33.979 INFO:tasks.cephadm:Adding mds.g on smithi007 2024-04-21T08:41:33.979 INFO:tasks.cephadm:Adding mds.j on smithi007 2024-04-21T08:41:33.979 INFO:tasks.cephadm:Adding mds.b on smithi016 2024-04-21T08:41:33.979 INFO:tasks.cephadm:Adding mds.e on smithi016 2024-04-21T08:41:33.979 INFO:tasks.cephadm:Adding mds.h on smithi016 2024-04-21T08:41:33.979 INFO:tasks.cephadm:Adding mds.k on smithi016 2024-04-21T08:41:33.979 INFO:tasks.cephadm:Adding mds.c on smithi077 2024-04-21T08:41:33.979 INFO:tasks.cephadm:Adding mds.f on smithi077 2024-04-21T08:41:33.979 INFO:tasks.cephadm:Adding mds.i on smithi077 2024-04-21T08:41:33.979 INFO:tasks.cephadm:Adding mds.l on smithi077 2024-04-21T08:41:33.979 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l' 2024-04-21T08:41:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:41:35.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:35 smithi007 ceph-mon[28477]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:35.657 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:35 smithi077 ceph-mon[31615]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:35.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:35 smithi016 ceph-mon[31259]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:36.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.896+0000 7f29d8a6c700 1 -- 172.21.15.77:0/2144042484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d4103ca0 msgr2=0x7f29d4104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:36.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.896+0000 7f29d8a6c700 1 --2- 172.21.15.77:0/2144042484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d4103ca0 0x7f29d4104100 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f29c8009990 tx=0x7f29c8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:41:36.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.896+0000 7f29d8a6c700 1 -- 172.21.15.77:0/2144042484 shutdown_connections 2024-04-21T08:41:36.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.896+0000 7f29d8a6c700 1 --2- 172.21.15.77:0/2144042484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29d4104640 0x7f29d410aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.896+0000 7f29d8a6c700 1 --2- 172.21.15.77:0/2144042484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29d4102aa0 0x7f29d4102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.896+0000 7f29d8a6c700 1 --2- 172.21.15.77:0/2144042484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d4103ca0 0x7f29d4104100 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.896+0000 7f29d8a6c700 1 -- 172.21.15.77:0/2144042484 >> 172.21.15.77:0/2144042484 conn(0x7f29d40fe040 msgr2=0x7f29d4100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.896+0000 7f29d8a6c700 1 -- 172.21.15.77:0/2144042484 shutdown_connections 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.896+0000 7f29d8a6c700 1 -- 172.21.15.77:0/2144042484 wait complete. 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d8a6c700 1 Processor -- start 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d8a6c700 1 -- start start 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d8a6c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d4102aa0 0x7f29d4198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d8a6c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29d4103ca0 0x7f29d41991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d8a6c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29d4104640 0x7f29d41a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d8a6c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f29d410e590 con 0x7f29d4104640 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d8a6c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29d410e410 con 0x7f29d4103ca0 2024-04-21T08:41:36.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d8a6c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f29d410e710 con 0x7f29d4102aa0 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d4102aa0 0x7f29d4198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d4102aa0 0x7f29d4198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35384/0 (socket says 172.21.15.77:35384) 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d259c700 1 -- 172.21.15.77:0/835430072 learned_addr learned my addr 172.21.15.77:0/835430072 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d2d9d700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29d4104640 0x7f29d41a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d1d9b700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29d4103ca0 0x7f29d41991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d259c700 1 -- 172.21.15.77:0/835430072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29d4103ca0 msgr2=0x7f29d41991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d259c700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29d4103ca0 0x7f29d41991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d259c700 1 -- 172.21.15.77:0/835430072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29d4104640 msgr2=0x7f29d41a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d259c700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29d4104640 0x7f29d41a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d259c700 1 -- 172.21.15.77:0/835430072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d41a0880 con 0x7f29d4102aa0 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d2d9d700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29d4104640 0x7f29d41a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d259c700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d4102aa0 0x7f29d4198c80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29c400d980 tx=0x7f29c400dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:36.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29c37fe700 1 -- 172.21.15.77:0/835430072 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c4017030 con 0x7f29d4102aa0 2024-04-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29d8a6c700 1 -- 172.21.15.77:0/835430072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d41a0b70 con 0x7f29d4102aa0 2024-04-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29c37fe700 1 -- 172.21.15.77:0/835430072 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29c40044d0 con 0x7f29d4102aa0 2024-04-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.897+0000 7f29c37fe700 1 -- 172.21.15.77:0/835430072 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c4003aa0 con 0x7f29d4102aa0 2024-04-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.898+0000 7f29d8a6c700 1 -- 172.21.15.77:0/835430072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d41a1100 con 0x7f29d4102aa0 2024-04-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.898+0000 7f29d8a6c700 1 -- 172.21.15.77:0/835430072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d4069730 con 0x7f29d4102aa0 2024-04-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.900+0000 7f29c37fe700 1 -- 172.21.15.77:0/835430072 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f29c401a070 con 0x7f29d4102aa0 2024-04-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.900+0000 7f29c37fe700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f29bc0a3c40 0x7f29bc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.900+0000 7f29c37fe700 1 -- 172.21.15.77:0/835430072 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f29c40cf970 con 0x7f29d4102aa0 2024-04-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.900+0000 7f29d1d9b700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f29bc0a3c40 0x7f29bc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.901+0000 7f29d1d9b700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f29bc0a3c40 0x7f29bc0a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f29c800b5c0 tx=0x7f29c8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:36.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:35.905+0000 7f29c37fe700 1 -- 172.21.15.77:0/835430072 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29c409c7f0 con 0x7f29d4102aa0 2024-04-21T08:41:36.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.119+0000 7f29d8a6c700 1 -- 172.21.15.77:0/835430072 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l", "target": ["mon-mgr", ""]}) -- 0x7f29d4063080 con 0x7f29bc0a3c40 2024-04-21T08:41:36.126 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.127+0000 7f29c37fe700 1 -- 172.21.15.77:0/835430072 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f29d4063080 con 0x7f29bc0a3c40 2024-04-21T08:41:36.126 INFO:teuthology.orchestra.run.smithi077.stdout:Scheduled mds.all update... 2024-04-21T08:41:36.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.129+0000 7f29d8a6c700 1 -- 172.21.15.77:0/835430072 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f29bc0a3c40 msgr2=0x7f29bc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:36.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.129+0000 7f29d8a6c700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f29bc0a3c40 0x7f29bc0a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f29c800b5c0 tx=0x7f29c8005fd0 comp rx=0 tx=0).stop 2024-04-21T08:41:36.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.130+0000 7f29d8a6c700 1 -- 172.21.15.77:0/835430072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d4102aa0 msgr2=0x7f29d4198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:36.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.130+0000 7f29d8a6c700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d4102aa0 0x7f29d4198c80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29c400d980 tx=0x7f29c400dd40 comp rx=0 tx=0).stop 2024-04-21T08:41:36.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.130+0000 7f29d8a6c700 1 -- 172.21.15.77:0/835430072 shutdown_connections 2024-04-21T08:41:36.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.130+0000 7f29d8a6c700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f29bc0a3c40 0x7f29bc0a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:36.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.130+0000 7f29d8a6c700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29d4102aa0 0x7f29d4198c80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:36.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.130+0000 7f29d8a6c700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29d4104640 0x7f29d41a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:36.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.130+0000 7f29d8a6c700 1 --2- 172.21.15.77:0/835430072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29d4103ca0 0x7f29d41991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:36.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.130+0000 7f29d8a6c700 1 -- 172.21.15.77:0/835430072 >> 172.21.15.77:0/835430072 conn(0x7f29d40fe040 msgr2=0x7f29d41070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:36.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.130+0000 7f29d8a6c700 1 -- 172.21.15.77:0/835430072 shutdown_connections 2024-04-21T08:41:36.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-21T08:41:36.130+0000 7f29d8a6c700 1 -- 172.21.15.77:0/835430072 wait complete. 2024-04-21T08:41:36.850 DEBUG:teuthology.orchestra.run.smithi007:mds.a> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.a.service 2024-04-21T08:41:36.853 DEBUG:teuthology.orchestra.run.smithi007:mds.d> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.d.service 2024-04-21T08:41:36.896 DEBUG:teuthology.orchestra.run.smithi007:mds.g> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.g.service 2024-04-21T08:41:36.939 DEBUG:teuthology.orchestra.run.smithi007:mds.j> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.j.service 2024-04-21T08:41:36.983 DEBUG:teuthology.orchestra.run.smithi016:mds.b> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.b.service 2024-04-21T08:41:36.986 DEBUG:teuthology.orchestra.run.smithi016:mds.e> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.e.service 2024-04-21T08:41:37.030 DEBUG:teuthology.orchestra.run.smithi016:mds.h> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.h.service 2024-04-21T08:41:37.073 DEBUG:teuthology.orchestra.run.smithi016:mds.k> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.k.service 2024-04-21T08:41:37.116 DEBUG:teuthology.orchestra.run.smithi077:mds.c> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.c.service 2024-04-21T08:41:37.118 DEBUG:teuthology.orchestra.run.smithi077:mds.f> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.f.service 2024-04-21T08:41:37.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:37 smithi007 ceph-mon[28477]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:37.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:37 smithi007 ceph-mon[28477]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:37.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:37 smithi007 ceph-mon[28477]: Saving service mds.all spec with placement smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l;count:12 2024-04-21T08:41:37.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:37 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:37.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:37 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:37.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:37 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:37.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:37 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:37.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:37 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:37.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:37 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:37.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:37 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:37.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:37 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:37.161 DEBUG:teuthology.orchestra.run.smithi077:mds.i> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.i.service 2024-04-21T08:41:37.204 DEBUG:teuthology.orchestra.run.smithi077:mds.l> sudo journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.l.service 2024-04-21T08:41:37.247 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-21T08:41:37.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:41:37.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:37 smithi016 ceph-mon[31259]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:37.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:37 smithi016 ceph-mon[31259]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:37.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:37 smithi016 ceph-mon[31259]: Saving service mds.all spec with placement smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l;count:12 2024-04-21T08:41:37.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:37 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:37.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:37 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:37.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:37 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:37.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:37 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:37.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:37 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:37.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:37 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:37.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:37 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:37.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:37 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:37.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.364+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/774597412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c48101ff0 msgr2=0x7f5c48102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.364+0000 7f5c4d1f4640 1 --2- 172.21.15.7:0/774597412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c48101ff0 0x7f5c48102470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009900 tx=0x7f5c3c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:41:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.364+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/774597412 shutdown_connections 2024-04-21T08:41:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.364+0000 7f5c4d1f4640 1 --2- 172.21.15.7:0/774597412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c481029b0 0x7f5c481092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.364+0000 7f5c4d1f4640 1 --2- 172.21.15.7:0/774597412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c48101ff0 0x7f5c48102470 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.364+0000 7f5c4d1f4640 1 --2- 172.21.15.7:0/774597412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c48100ea0 0x7f5c481012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.364+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/774597412 >> 172.21.15.7:0/774597412 conn(0x7f5c480fc410 msgr2=0x7f5c480fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.364+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/774597412 shutdown_connections 2024-04-21T08:41:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.364+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/774597412 wait complete. 2024-04-21T08:41:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4d1f4640 1 Processor -- start 2024-04-21T08:41:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4d1f4640 1 -- start start 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4d1f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c48100ea0 0x7f5c4806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4d1f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c48101ff0 0x7f5c4819f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4d1f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c481029b0 0x7f5c481a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4d1f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4810c9f0 con 0x7f5c481029b0 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4d1f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4810c870 con 0x7f5c48100ea0 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4d1f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4810cb70 con 0x7f5c48101ff0 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c481029b0 0x7f5c481a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c481029b0 0x7f5c481a1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45260/0 (socket says 172.21.15.7:45260) 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4759e640 1 -- 172.21.15.7:0/3290088757 learned_addr learned my addr 172.21.15.7:0/3290088757 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c4659c640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c48101ff0 0x7f5c4819f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.365+0000 7f5c46d9d640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c48100ea0 0x7f5c4806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.366+0000 7f5c4759e640 1 -- 172.21.15.7:0/3290088757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c48101ff0 msgr2=0x7f5c4819f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.366+0000 7f5c4759e640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c48101ff0 0x7f5c4819f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.366+0000 7f5c4759e640 1 -- 172.21.15.7:0/3290088757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c48100ea0 msgr2=0x7f5c4806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.366+0000 7f5c4759e640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c48100ea0 0x7f5c4806b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.366+0000 7f5c4759e640 1 -- 172.21.15.7:0/3290088757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c481a1ff0 con 0x7f5c481029b0 2024-04-21T08:41:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.366+0000 7f5c4759e640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c481029b0 0x7f5c481a1800 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000d750 tx=0x7f5c4000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.366+0000 7f5c2ffff640 1 -- 172.21.15.7:0/3290088757 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c40017070 con 0x7f5c481029b0 2024-04-21T08:41:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.366+0000 7f5c2ffff640 1 -- 172.21.15.7:0/3290088757 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c400054d0 con 0x7f5c481029b0 2024-04-21T08:41:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.366+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/3290088757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c481a2280 con 0x7f5c481029b0 2024-04-21T08:41:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.366+0000 7f5c2ffff640 1 -- 172.21.15.7:0/3290088757 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c40004c60 con 0x7f5c481029b0 2024-04-21T08:41:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.366+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/3290088757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c481a27a0 con 0x7f5c481029b0 2024-04-21T08:41:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.367+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/3290088757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c48108230 con 0x7f5c481029b0 2024-04-21T08:41:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.368+0000 7f5c2ffff640 1 -- 172.21.15.7:0/3290088757 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f5c40009bd0 con 0x7f5c481029b0 2024-04-21T08:41:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.369+0000 7f5c2ffff640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5c300a3be0 0x7f5c300a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.369+0000 7f5c46d9d640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5c300a3be0 0x7f5c300a60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.369+0000 7f5c2ffff640 1 -- 172.21.15.7:0/3290088757 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5c400cf8d0 con 0x7f5c481029b0 2024-04-21T08:41:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.369+0000 7f5c46d9d640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5c300a3be0 0x7f5c300a60a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5c38005fd0 tx=0x7f5c38009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:37.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.373+0000 7f5c2ffff640 1 -- 172.21.15.7:0/3290088757 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c4009c750 con 0x7f5c481029b0 2024-04-21T08:41:37.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.563+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/3290088757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c48062b50 con 0x7f5c481029b0 2024-04-21T08:41:37.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.564+0000 7f5c2ffff640 1 -- 172.21.15.7:0/3290088757 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18289 (secure 0 0 0) 0x7f5c4009c050 con 0x7f5c481029b0 2024-04-21T08:41:37.563 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:37.564 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":82,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:41:25.912790+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:41:37.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/3290088757 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5c300a3be0 msgr2=0x7f5c300a60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5c300a3be0 0x7f5c300a60a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5c38005fd0 tx=0x7f5c38009500 comp rx=0 tx=0).stop 2024-04-21T08:41:37.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/3290088757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c481029b0 msgr2=0x7f5c481a1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c481029b0 0x7f5c481a1800 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000d750 tx=0x7f5c4000dc10 comp rx=0 tx=0).stop 2024-04-21T08:41:37.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/3290088757 shutdown_connections 2024-04-21T08:41:37.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5c300a3be0 0x7f5c300a60a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c481029b0 0x7f5c481a1800 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c48101ff0 0x7f5c4819f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 --2- 172.21.15.7:0/3290088757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c48100ea0 0x7f5c4806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/3290088757 >> 172.21.15.7:0/3290088757 conn(0x7f5c480fc410 msgr2=0x7f5c48105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:37.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/3290088757 shutdown_connections 2024-04-21T08:41:37.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.566+0000 7f5c4d1f4640 1 -- 172.21.15.7:0/3290088757 wait complete. 2024-04-21T08:41:37.577 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:41:37.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:41:37.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:37 smithi077 ceph-mon[31615]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:37 smithi077 ceph-mon[31615]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:37 smithi077 ceph-mon[31615]: Saving service mds.all spec with placement smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l;count:12 2024-04-21T08:41:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:37 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:37 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:37 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:37 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:37 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:37 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:37 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:37 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:37.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.683+0000 7fe737d3d640 1 -- 172.21.15.7:0/1651528583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe730101fe0 msgr2=0x7fe730102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.683+0000 7fe737d3d640 1 --2- 172.21.15.7:0/1651528583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe730101fe0 0x7fe730102460 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe72c009900 tx=0x7fe72c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.683+0000 7fe737d3d640 1 -- 172.21.15.7:0/1651528583 shutdown_connections 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.683+0000 7fe737d3d640 1 --2- 172.21.15.7:0/1651528583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7301029a0 0x7fe7301092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.683+0000 7fe737d3d640 1 --2- 172.21.15.7:0/1651528583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe730101fe0 0x7fe730102460 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.683+0000 7fe737d3d640 1 --2- 172.21.15.7:0/1651528583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe730100e90 0x7fe7301012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.683+0000 7fe737d3d640 1 -- 172.21.15.7:0/1651528583 >> 172.21.15.7:0/1651528583 conn(0x7fe7300fc420 msgr2=0x7fe7300fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.683+0000 7fe737d3d640 1 -- 172.21.15.7:0/1651528583 shutdown_connections 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.683+0000 7fe737d3d640 1 -- 172.21.15.7:0/1651528583 wait complete. 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.683+0000 7fe737d3d640 1 Processor -- start 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.683+0000 7fe737d3d640 1 -- start start 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe737d3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe730100e90 0x7fe730198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe737d3d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe730101fe0 0x7fe730198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe737d3d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7301029a0 0x7fe73019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe737d3d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe73010dac0 con 0x7fe7301029a0 2024-04-21T08:41:37.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe737d3d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe73010d940 con 0x7fe730100e90 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe737d3d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe73010dc40 con 0x7fe730101fe0 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe7362da640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7301029a0 0x7fe73019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe7362da640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7301029a0 0x7fe73019f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45280/0 (socket says 172.21.15.7:45280) 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe7362da640 1 -- 172.21.15.7:0/3780351053 learned_addr learned my addr 172.21.15.7:0/3780351053 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe7352d8640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe730101fe0 0x7fe730198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe735ad9640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe730100e90 0x7fe730198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe7362da640 1 -- 172.21.15.7:0/3780351053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe730101fe0 msgr2=0x7fe730198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe7362da640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe730101fe0 0x7fe730198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe7362da640 1 -- 172.21.15.7:0/3780351053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe730100e90 msgr2=0x7fe730198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe7362da640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe730100e90 0x7fe730198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe7362da640 1 -- 172.21.15.7:0/3780351053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73019fe70 con 0x7fe7301029a0 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe7362da640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7301029a0 0x7fe73019f800 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe72800cad0 tx=0x7fe72800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe726ffd640 1 -- 172.21.15.7:0/3780351053 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe72800ae20 con 0x7fe7301029a0 2024-04-21T08:41:37.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe737d3d640 1 -- 172.21.15.7:0/3780351053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7301a0160 con 0x7fe7301029a0 2024-04-21T08:41:37.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe726ffd640 1 -- 172.21.15.7:0/3780351053 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe728005020 con 0x7fe7301029a0 2024-04-21T08:41:37.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe726ffd640 1 -- 172.21.15.7:0/3780351053 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe72800f6b0 con 0x7fe7301029a0 2024-04-21T08:41:37.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.684+0000 7fe737d3d640 1 -- 172.21.15.7:0/3780351053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7301a0680 con 0x7fe7301029a0 2024-04-21T08:41:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.686+0000 7fe737d3d640 1 -- 172.21.15.7:0/3780351053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7301082c0 con 0x7fe7301029a0 2024-04-21T08:41:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.687+0000 7fe726ffd640 1 -- 172.21.15.7:0/3780351053 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fe7280051d0 con 0x7fe7301029a0 2024-04-21T08:41:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.687+0000 7fe726ffd640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe71c0a3bb0 0x7fe71c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.687+0000 7fe735ad9640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe71c0a3bb0 0x7fe71c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.687+0000 7fe726ffd640 1 -- 172.21.15.7:0/3780351053 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe7280d00d0 con 0x7fe7301029a0 2024-04-21T08:41:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.688+0000 7fe735ad9640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe71c0a3bb0 0x7fe71c0a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe720005fd0 tx=0x7fe720005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:37.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.692+0000 7fe726ffd640 1 -- 172.21.15.7:0/3780351053 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe728014030 con 0x7fe7301029a0 2024-04-21T08:41:37.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.878+0000 7fe737d3d640 1 -- 172.21.15.7:0/3780351053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe730062b50 con 0x7fe7301029a0 2024-04-21T08:41:37.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.878+0000 7fe726ffd640 1 -- 172.21.15.7:0/3780351053 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fe72809cf50 con 0x7fe7301029a0 2024-04-21T08:41:37.877 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:41:37.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 -- 172.21.15.7:0/3780351053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe71c0a3bb0 msgr2=0x7fe71c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe71c0a3bb0 0x7fe71c0a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe720005fd0 tx=0x7fe720005850 comp rx=0 tx=0).stop 2024-04-21T08:41:37.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 -- 172.21.15.7:0/3780351053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7301029a0 msgr2=0x7fe73019f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7301029a0 0x7fe73019f800 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe72800cad0 tx=0x7fe72800cf90 comp rx=0 tx=0).stop 2024-04-21T08:41:37.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 -- 172.21.15.7:0/3780351053 shutdown_connections 2024-04-21T08:41:37.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe71c0a3bb0 0x7fe71c0a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7301029a0 0x7fe73019f800 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe730101fe0 0x7fe730198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 --2- 172.21.15.7:0/3780351053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe730100e90 0x7fe730198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 -- 172.21.15.7:0/3780351053 >> 172.21.15.7:0/3780351053 conn(0x7fe7300fc420 msgr2=0x7fe730105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:37.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 -- 172.21.15.7:0/3780351053 shutdown_connections 2024-04-21T08:41:37.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.880+0000 7fe737d3d640 1 -- 172.21.15.7:0/3780351053 wait complete. 2024-04-21T08:41:37.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.992+0000 7fbe394d5640 1 -- 172.21.15.7:0/2624415284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe34103b50 msgr2=0x7fbe3410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.992+0000 7fbe394d5640 1 --2- 172.21.15.7:0/2624415284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe34103b50 0x7fbe3410a3e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009ca0 tx=0x7fbe2c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.993+0000 7fbe394d5640 1 -- 172.21.15.7:0/2624415284 shutdown_connections 2024-04-21T08:41:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.993+0000 7fbe394d5640 1 --2- 172.21.15.7:0/2624415284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe34103b50 0x7fbe3410a3e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.993+0000 7fbe394d5640 1 --2- 172.21.15.7:0/2624415284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe34103190 0x7fbe34103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.993+0000 7fbe394d5640 1 --2- 172.21.15.7:0/2624415284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe34101fa0 0x7fbe341023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.993+0000 7fbe394d5640 1 -- 172.21.15.7:0/2624415284 >> 172.21.15.7:0/2624415284 conn(0x7fbe340fd570 msgr2=0x7fbe340ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.993+0000 7fbe394d5640 1 -- 172.21.15.7:0/2624415284 shutdown_connections 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.993+0000 7fbe394d5640 1 -- 172.21.15.7:0/2624415284 wait complete. 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.993+0000 7fbe394d5640 1 Processor -- start 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.993+0000 7fbe394d5640 1 -- start start 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe394d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe34101fa0 0x7fbe341980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe394d5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe34103190 0x7fbe34198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe394d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe34103b50 0x7fbe3419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe394d5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3410dae0 con 0x7fbe34103190 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe394d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3410d960 con 0x7fbe34103b50 2024-04-21T08:41:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe394d5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3410dc60 con 0x7fbe34101fa0 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe327fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe34103190 0x7fbe34198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe327fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe34103190 0x7fbe34198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45298/0 (socket says 172.21.15.7:45298) 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe327fc640 1 -- 172.21.15.7:0/4093769127 learned_addr learned my addr 172.21.15.7:0/4093769127 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe337fe640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe34103b50 0x7fbe3419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe32ffd640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe34101fa0 0x7fbe341980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe327fc640 1 -- 172.21.15.7:0/4093769127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe34101fa0 msgr2=0x7fbe341980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe327fc640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe34101fa0 0x7fbe341980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe327fc640 1 -- 172.21.15.7:0/4093769127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe34103b50 msgr2=0x7fbe3419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe327fc640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe34103b50 0x7fbe3419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe327fc640 1 -- 172.21.15.7:0/4093769127 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3419fd00 con 0x7fbe34103190 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe327fc640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe34103190 0x7fbe34198620 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400ea60 tx=0x7fbe2400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe1bfff640 1 -- 172.21.15.7:0/4093769127 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2400cde0 con 0x7fbe34103190 2024-04-21T08:41:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe394d5640 1 -- 172.21.15.7:0/4093769127 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe3419fff0 con 0x7fbe34103190 2024-04-21T08:41:37.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe1bfff640 1 -- 172.21.15.7:0/4093769127 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe240035f0 con 0x7fbe34103190 2024-04-21T08:41:37.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe1bfff640 1 -- 172.21.15.7:0/4093769127 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe24010690 con 0x7fbe34103190 2024-04-21T08:41:37.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.994+0000 7fbe394d5640 1 -- 172.21.15.7:0/4093769127 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe341a04b0 con 0x7fbe34103190 2024-04-21T08:41:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.995+0000 7fbe394d5640 1 -- 172.21.15.7:0/4093769127 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe341092c0 con 0x7fbe34103190 2024-04-21T08:41:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.997+0000 7fbe1bfff640 1 -- 172.21.15.7:0/4093769127 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fbe240037a0 con 0x7fbe34103190 2024-04-21T08:41:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.997+0000 7fbe1bfff640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe1c0a3bb0 0x7fbe1c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.997+0000 7fbe32ffd640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe1c0a3bb0 0x7fbe1c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.997+0000 7fbe1bfff640 1 -- 172.21.15.7:0/4093769127 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbe240d02b0 con 0x7fbe34103190 2024-04-21T08:41:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:37.998+0000 7fbe32ffd640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe1c0a3bb0 0x7fbe1c0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbe28005fd0 tx=0x7fbe28005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:38.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.002+0000 7fbe1bfff640 1 -- 172.21.15.7:0/4093769127 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe2409d130 con 0x7fbe34103190 2024-04-21T08:41:38.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:38 smithi007 ceph-mon[28477]: Deploying daemon mds.l on smithi077 2024-04-21T08:41:38.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:38 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3290088757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:41:38.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:38 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3780351053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:41:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.229+0000 7fbe394d5640 1 -- 172.21.15.7:0/4093769127 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe3402d5c0 con 0x7fbe34103190 2024-04-21T08:41:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.229+0000 7fbe1bfff640 1 -- 172.21.15.7:0/4093769127 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fbe2409ca30 con 0x7fbe34103190 2024-04-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-21T08:41:38.229 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 1 2024-04-21T08:41:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.231+0000 7fbe394d5640 1 -- 172.21.15.7:0/4093769127 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe1c0a3bb0 msgr2=0x7fbe1c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.231+0000 7fbe394d5640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe1c0a3bb0 0x7fbe1c0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbe28005fd0 tx=0x7fbe28005850 comp rx=0 tx=0).stop 2024-04-21T08:41:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.231+0000 7fbe394d5640 1 -- 172.21.15.7:0/4093769127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe34103190 msgr2=0x7fbe34198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.231+0000 7fbe394d5640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe34103190 0x7fbe34198620 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400ea60 tx=0x7fbe2400ef20 comp rx=0 tx=0).stop 2024-04-21T08:41:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.231+0000 7fbe394d5640 1 -- 172.21.15.7:0/4093769127 shutdown_connections 2024-04-21T08:41:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.231+0000 7fbe394d5640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe1c0a3bb0 0x7fbe1c0a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.231+0000 7fbe394d5640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe34103b50 0x7fbe3419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.231+0000 7fbe394d5640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe34103190 0x7fbe34198620 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.231+0000 7fbe394d5640 1 --2- 172.21.15.7:0/4093769127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe34101fa0 0x7fbe341980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.231+0000 7fbe394d5640 1 -- 172.21.15.7:0/4093769127 >> 172.21.15.7:0/4093769127 conn(0x7fbe340fd570 msgr2=0x7fbe34106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:38.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.231+0000 7fbe394d5640 1 -- 172.21.15.7:0/4093769127 shutdown_connections 2024-04-21T08:41:38.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:38.232+0000 7fbe394d5640 1 -- 172.21.15.7:0/4093769127 wait complete. 2024-04-21T08:41:38.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:38 smithi077 ceph-mon[31615]: Deploying daemon mds.l on smithi077 2024-04-21T08:41:38.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:38 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3290088757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:41:38.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:38 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3780351053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:41:38.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:38 smithi016 ceph-mon[31259]: Deploying daemon mds.l on smithi077 2024-04-21T08:41:38.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:38 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3290088757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:41:38.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:38 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3780351053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:41:39.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:39 smithi007 ceph-mon[28477]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:39.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:39 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4093769127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:39.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:39 smithi077 ceph-mon[31615]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:39.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:39 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4093769127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:39.550 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 08:41:39 smithi077 systemd[1]: Starting Ceph mds.l for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:41:39.553 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:39 smithi016 ceph-mon[31259]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:39.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:39 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4093769127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:39.853 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 08:41:39 smithi077 podman[61722]: 2024-04-21 08:41:39.453931182 +0000 UTC m=+0.018303431 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:41:39.853 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 08:41:39 smithi077 podman[61722]: 2024-04-21 08:41:39.592874039 +0000 UTC m=+0.157246316 container create ae73de27efd79a63a1aeca8140c8f90f73804939126cc08f9250b0fafe28d3b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:41:39.853 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 08:41:39 smithi077 podman[61722]: 2024-04-21 08:41:39.854355347 +0000 UTC m=+0.418727582 container init ae73de27efd79a63a1aeca8140c8f90f73804939126cc08f9250b0fafe28d3b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-l, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:41:40.103 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 08:41:39 smithi077 podman[61722]: 2024-04-21 08:41:39.857106995 +0000 UTC m=+0.421479241 container start ae73de27efd79a63a1aeca8140c8f90f73804939126cc08f9250b0fafe28d3b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-l, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-21T08:41:40.104 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 08:41:39 smithi077 ceph-mds[61742]: -- 172.21.15.77:0/3623223287 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56545cee5a40 con 0x56545ceb9000 2024-04-21T08:41:40.104 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 08:41:39 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-l[61738]: starting mds.l at 2024-04-21T08:41:40.104 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 08:41:39 smithi077 bash[61722]: ae73de27efd79a63a1aeca8140c8f90f73804939126cc08f9250b0fafe28d3b8 2024-04-21T08:41:40.104 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 08:41:39 smithi077 systemd[1]: Started Ceph mds.l for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:41:40.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.358+0000 7f287faa5640 1 -- 172.21.15.7:0/1940921657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2878101ff0 msgr2=0x7f2878102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.358+0000 7f287faa5640 1 --2- 172.21.15.7:0/1940921657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2878101ff0 0x7f2878102470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2868009900 tx=0x7f286802bc30 comp rx=0 tx=0).stop 2024-04-21T08:41:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.358+0000 7f287faa5640 1 -- 172.21.15.7:0/1940921657 shutdown_connections 2024-04-21T08:41:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.358+0000 7f287faa5640 1 --2- 172.21.15.7:0/1940921657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28781029b0 0x7f28781092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.358+0000 7f287faa5640 1 --2- 172.21.15.7:0/1940921657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2878101ff0 0x7f2878102470 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.358+0000 7f287faa5640 1 --2- 172.21.15.7:0/1940921657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2878100ea0 0x7f28781012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.358+0000 7f287faa5640 1 -- 172.21.15.7:0/1940921657 >> 172.21.15.7:0/1940921657 conn(0x7f28780fc410 msgr2=0x7f28780fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.358+0000 7f287faa5640 1 -- 172.21.15.7:0/1940921657 shutdown_connections 2024-04-21T08:41:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.358+0000 7f287faa5640 1 -- 172.21.15.7:0/1940921657 wait complete. 2024-04-21T08:41:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.358+0000 7f287faa5640 1 Processor -- start 2024-04-21T08:41:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.358+0000 7f287faa5640 1 -- start start 2024-04-21T08:41:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287faa5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2878100ea0 0x7f2878198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287faa5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2878101ff0 0x7f28781986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287faa5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28781029b0 0x7f287819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287faa5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f287810db10 con 0x7f2878101ff0 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287faa5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f287810d990 con 0x7f2878100ea0 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287faa5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f287810dc90 con 0x7f28781029b0 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287d040640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2878101ff0 0x7f28781986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287d841640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2878100ea0 0x7f2878198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287d040640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2878101ff0 0x7f28781986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45308/0 (socket says 172.21.15.7:45308) 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287d040640 1 -- 172.21.15.7:0/3085396159 learned_addr learned my addr 172.21.15.7:0/3085396159 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287e042640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28781029b0 0x7f287819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287d040640 1 -- 172.21.15.7:0/3085396159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28781029b0 msgr2=0x7f287819f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287d040640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28781029b0 0x7f287819f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287d040640 1 -- 172.21.15.7:0/3085396159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2878100ea0 msgr2=0x7f2878198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287d040640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2878100ea0 0x7f2878198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287d040640 1 -- 172.21.15.7:0/3085396159 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f287819fda0 con 0x7f2878101ff0 2024-04-21T08:41:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287d841640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2878100ea0 0x7f2878198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:41:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287d040640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2878101ff0 0x7f28781986c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2868000c00 tx=0x7f2868004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f286effd640 1 -- 172.21.15.7:0/3085396159 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2868043020 con 0x7f2878101ff0 2024-04-21T08:41:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f286effd640 1 -- 172.21.15.7:0/3085396159 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2868004440 con 0x7f2878101ff0 2024-04-21T08:41:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287faa5640 1 -- 172.21.15.7:0/3085396159 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28781a0030 con 0x7f2878101ff0 2024-04-21T08:41:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f286effd640 1 -- 172.21.15.7:0/3085396159 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2868003c90 con 0x7f2878101ff0 2024-04-21T08:41:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.359+0000 7f287faa5640 1 -- 172.21.15.7:0/3085396159 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28781a05d0 con 0x7f2878101ff0 2024-04-21T08:41:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.360+0000 7f287faa5640 1 -- 172.21.15.7:0/3085396159 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28781081b0 con 0x7f2878101ff0 2024-04-21T08:41:40.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.361+0000 7f286effd640 1 -- 172.21.15.7:0/3085396159 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f2868046020 con 0x7f2878101ff0 2024-04-21T08:41:40.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.362+0000 7f286effd640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28640a3c30 0x7f28640a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:40.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.362+0000 7f287d841640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28640a3c30 0x7f28640a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:40.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.362+0000 7f286effd640 1 -- 172.21.15.7:0/3085396159 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f28680f2100 con 0x7f2878101ff0 2024-04-21T08:41:40.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.363+0000 7f287d841640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28640a3c30 0x7f28640a60f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2874009fc0 tx=0x7f2874005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:40.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.367+0000 7f286effd640 1 -- 172.21.15.7:0/3085396159 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f286804d020 con 0x7f2878101ff0 2024-04-21T08:41:40.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.594+0000 7f287faa5640 1 -- 172.21.15.7:0/3085396159 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f287802d5c0 con 0x7f2878101ff0 2024-04-21T08:41:40.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.595+0000 7f286effd640 1 -- 172.21.15.7:0/3085396159 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7f28680beee0 con 0x7f2878101ff0 2024-04-21T08:41:40.594 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:40.594 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:41:40.595 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 2 2024-04-21T08:41:40.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 -- 172.21.15.7:0/3085396159 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28640a3c30 msgr2=0x7f28640a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:40.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28640a3c30 0x7f28640a60f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2874009fc0 tx=0x7f2874005f90 comp rx=0 tx=0).stop 2024-04-21T08:41:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 -- 172.21.15.7:0/3085396159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2878101ff0 msgr2=0x7f28781986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2878101ff0 0x7f28781986c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2868000c00 tx=0x7f2868004040 comp rx=0 tx=0).stop 2024-04-21T08:41:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 -- 172.21.15.7:0/3085396159 shutdown_connections 2024-04-21T08:41:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28640a3c30 0x7f28640a60f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28781029b0 0x7f287819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2878101ff0 0x7f28781986c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 --2- 172.21.15.7:0/3085396159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2878100ea0 0x7f2878198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 -- 172.21.15.7:0/3085396159 >> 172.21.15.7:0/3085396159 conn(0x7f28780fc410 msgr2=0x7f2878105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 -- 172.21.15.7:0/3085396159 shutdown_connections 2024-04-21T08:41:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:40.597+0000 7f287faa5640 1 -- 172.21.15.7:0/3085396159 wait complete. 2024-04-21T08:41:40.973 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:40.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.77:6832/1817896021,v1:172.21.15.77:6833/1817896021] up:boot 2024-04-21T08:41:40.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: fsmap 1 up:standby 2024-04-21T08:41:40.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:41:40.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:40.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:40.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:40.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:40.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:40.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:40.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: Deploying daemon mds.k on smithi016 2024-04-21T08:41:40.974 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:40 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3085396159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:41.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.77:6832/1817896021,v1:172.21.15.77:6833/1817896021] up:boot 2024-04-21T08:41:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: fsmap 1 up:standby 2024-04-21T08:41:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:41:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: Deploying daemon mds.k on smithi016 2024-04-21T08:41:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:40 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3085396159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:41.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.77:6832/1817896021,v1:172.21.15.77:6833/1817896021] up:boot 2024-04-21T08:41:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: fsmap 1 up:standby 2024-04-21T08:41:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:41:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: Deploying daemon mds.k on smithi016 2024-04-21T08:41:41.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:40 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3085396159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:42.554 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:41:42 smithi016 systemd[1]: Starting Ceph mds.k for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:41:42.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:42.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.724+0000 7f6cf7b22640 1 -- 172.21.15.7:0/1153614595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cf0100e80 msgr2=0x7f6cf01012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.724+0000 7f6cf7b22640 1 --2- 172.21.15.7:0/1153614595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cf0100e80 0x7f6cf01012a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009900 tx=0x7f6ce802bc30 comp rx=0 tx=0).stop 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.724+0000 7f6cf7b22640 1 -- 172.21.15.7:0/1153614595 shutdown_connections 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.724+0000 7f6cf7b22640 1 --2- 172.21.15.7:0/1153614595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cf0102990 0x7f6cf01092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.724+0000 7f6cf7b22640 1 --2- 172.21.15.7:0/1153614595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf0101fd0 0x7f6cf0102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.724+0000 7f6cf7b22640 1 --2- 172.21.15.7:0/1153614595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cf0100e80 0x7f6cf01012a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.724+0000 7f6cf7b22640 1 -- 172.21.15.7:0/1153614595 >> 172.21.15.7:0/1153614595 conn(0x7f6cf00fc3f0 msgr2=0x7f6cf00fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.724+0000 7f6cf7b22640 1 -- 172.21.15.7:0/1153614595 shutdown_connections 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.724+0000 7f6cf7b22640 1 -- 172.21.15.7:0/1153614595 wait complete. 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf7b22640 1 Processor -- start 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf7b22640 1 -- start start 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf7b22640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cf0100e80 0x7f6cf0104db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:42.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf7b22640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cf0101fd0 0x7f6cf0103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf7b22640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf0102990 0x7f6cf0103980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf7b22640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf010c9d0 con 0x7f6cf0100e80 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf7b22640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf010c850 con 0x7f6cf0102990 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf7b22640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf010cb50 con 0x7f6cf0101fd0 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf58be640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cf0100e80 0x7f6cf0104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf58be640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cf0100e80 0x7f6cf0104db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45322/0 (socket says 172.21.15.7:45322) 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf58be640 1 -- 172.21.15.7:0/2502662971 learned_addr learned my addr 172.21.15.7:0/2502662971 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf60bf640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf0102990 0x7f6cf0103980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.725+0000 7f6cf50bd640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cf0101fd0 0x7f6cf0103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.726+0000 7f6cf60bf640 1 -- 172.21.15.7:0/2502662971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cf0101fd0 msgr2=0x7f6cf0103440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.726+0000 7f6cf60bf640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cf0101fd0 0x7f6cf0103440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.726+0000 7f6cf60bf640 1 -- 172.21.15.7:0/2502662971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cf0100e80 msgr2=0x7f6cf0104db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:42.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.726+0000 7f6cf60bf640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cf0100e80 0x7f6cf0104db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.726+0000 7f6cf60bf640 1 -- 172.21.15.7:0/2502662971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf0104240 con 0x7f6cf0102990 2024-04-21T08:41:42.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.726+0000 7f6cf60bf640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf0102990 0x7f6cf0103980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009fd0 tx=0x7f6ce000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:42.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.726+0000 7f6ceeffd640 1 -- 172.21.15.7:0/2502662971 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce000ce20 con 0x7f6cf0102990 2024-04-21T08:41:42.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.726+0000 7f6ceeffd640 1 -- 172.21.15.7:0/2502662971 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ce0005020 con 0x7f6cf0102990 2024-04-21T08:41:42.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.726+0000 7f6cf7b22640 1 -- 172.21.15.7:0/2502662971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf01a2310 con 0x7f6cf0102990 2024-04-21T08:41:42.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.726+0000 7f6ceeffd640 1 -- 172.21.15.7:0/2502662971 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0010690 con 0x7f6cf0102990 2024-04-21T08:41:42.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.726+0000 7f6cf7b22640 1 -- 172.21.15.7:0/2502662971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf01a2830 con 0x7f6cf0102990 2024-04-21T08:41:42.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.727+0000 7f6cf7b22640 1 -- 172.21.15.7:0/2502662971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf01a2af0 con 0x7f6cf0102990 2024-04-21T08:41:42.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.728+0000 7f6ceeffd640 1 -- 172.21.15.7:0/2502662971 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6ce00054d0 con 0x7f6cf0102990 2024-04-21T08:41:42.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.729+0000 7f6ceeffd640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6cd80a3bb0 0x7f6cd80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:42.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.729+0000 7f6ceeffd640 1 -- 172.21.15.7:0/2502662971 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6ce00d0a60 con 0x7f6cf0102990 2024-04-21T08:41:42.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.729+0000 7f6cf58be640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6cd80a3bb0 0x7f6cd80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:42.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.729+0000 7f6cf58be640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6cd80a3bb0 0x7f6cd80a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80096c0 tx=0x7f6ce802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:42.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.733+0000 7f6ceeffd640 1 -- 172.21.15.7:0/2502662971 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ce009d880 con 0x7f6cf0102990 2024-04-21T08:41:42.866 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:41:42 smithi016 podman[60223]: 2024-04-21 08:41:42.493849172 +0000 UTC m=+0.018062322 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:41:42.867 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:41:42 smithi016 podman[60223]: 2024-04-21 08:41:42.61549584 +0000 UTC m=+0.139708979 container create 06fcf0ad3eaa8889d63aba9728cede48dadcddce2cf7740b71313c2a92138243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-04-21T08:41:42.867 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:41:42 smithi016 podman[60223]: 2024-04-21 08:41:42.868168048 +0000 UTC m=+0.392381189 container init 06fcf0ad3eaa8889d63aba9728cede48dadcddce2cf7740b71313c2a92138243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-21T08:41:42.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.961+0000 7f6cf7b22640 1 -- 172.21.15.7:0/2502662971 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cf01a2de0 con 0x7f6cf0102990 2024-04-21T08:41:42.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.962+0000 7f6ceeffd640 1 -- 172.21.15.7:0/2502662971 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7f6ce009d180 con 0x7f6cf0102990 2024-04-21T08:41:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:42.961 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:41:42.962 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 2 2024-04-21T08:41:42.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 -- 172.21.15.7:0/2502662971 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6cd80a3bb0 msgr2=0x7f6cd80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:42.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6cd80a3bb0 0x7f6cd80a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80096c0 tx=0x7f6ce802c7e0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 -- 172.21.15.7:0/2502662971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf0102990 msgr2=0x7f6cf0103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:42.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf0102990 0x7f6cf0103980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009fd0 tx=0x7f6ce000ef10 comp rx=0 tx=0).stop 2024-04-21T08:41:42.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 -- 172.21.15.7:0/2502662971 shutdown_connections 2024-04-21T08:41:42.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6cd80a3bb0 0x7f6cd80a6070 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cf0102990 0x7f6cf0103980 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cf0101fd0 0x7f6cf0103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 --2- 172.21.15.7:0/2502662971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cf0100e80 0x7f6cf0104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 -- 172.21.15.7:0/2502662971 >> 172.21.15.7:0/2502662971 conn(0x7f6cf00fc3f0 msgr2=0x7f6cf00fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:42.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 -- 172.21.15.7:0/2502662971 shutdown_connections 2024-04-21T08:41:42.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:42.964+0000 7f6cf7b22640 1 -- 172.21.15.7:0/2502662971 wait complete. 2024-04-21T08:41:43.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:42 smithi007 ceph-mon[28477]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:43.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:42 smithi016 ceph-mon[31259]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:43.304 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:41:42 smithi016 podman[60223]: 2024-04-21 08:41:42.871046246 +0000 UTC m=+0.395259387 container start 06fcf0ad3eaa8889d63aba9728cede48dadcddce2cf7740b71313c2a92138243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-21T08:41:43.304 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:41:42 smithi016 ceph-mds[60242]: -- 172.21.15.16:0/2544788480 <== mon.2 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558b85323a40 con 0x558b852f7000 2024-04-21T08:41:43.304 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:41:42 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-k[60238]: starting mds.k at 2024-04-21T08:41:43.305 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:41:43 smithi016 bash[60223]: 06fcf0ad3eaa8889d63aba9728cede48dadcddce2cf7740b71313c2a92138243 2024-04-21T08:41:43.305 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:41:43 smithi016 systemd[1]: Started Ceph mds.k for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:41:43.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:42 smithi077 ceph-mon[31615]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:44.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:43 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2502662971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:44.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:43 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:44.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:43 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:44.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:43 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:44.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:43 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:44.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:43 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:44.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:43 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:44.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:43 smithi007 ceph-mon[28477]: Deploying daemon mds.j on smithi007 2024-04-21T08:41:44.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:43 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2502662971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:44.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:43 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:44.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:43 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:44.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:43 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:44.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:43 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:44.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:43 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:44.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:43 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:44.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:43 smithi016 ceph-mon[31259]: Deploying daemon mds.j on smithi007 2024-04-21T08:41:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:43 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2502662971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:43 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:43 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:43 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:43 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:43 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:43 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:44.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:43 smithi077 ceph-mon[31615]: Deploying daemon mds.j on smithi007 2024-04-21T08:41:44.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:45.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:44 smithi007 ceph-mon[28477]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:45.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:44 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.16:6832/2178684530,v1:172.21.15.16:6833/2178684530] up:boot 2024-04-21T08:41:45.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:44 smithi007 ceph-mon[28477]: fsmap 2 up:standby 2024-04-21T08:41:45.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:44 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:41:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.190+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3812486427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00a4102990 msgr2=0x7f00a41092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.190+0000 7f00ac0ba640 1 --2- 172.21.15.7:0/3812486427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00a4102990 0x7f00a41092b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f009c009ca0 tx=0x7f009c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.190+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3812486427 shutdown_connections 2024-04-21T08:41:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.190+0000 7f00ac0ba640 1 --2- 172.21.15.7:0/3812486427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00a4102990 0x7f00a41092b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.190+0000 7f00ac0ba640 1 --2- 172.21.15.7:0/3812486427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00a4101fd0 0x7f00a4102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.190+0000 7f00ac0ba640 1 --2- 172.21.15.7:0/3812486427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a4100e80 0x7f00a41012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.190+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3812486427 >> 172.21.15.7:0/3812486427 conn(0x7f00a40fc3f0 msgr2=0x7f00a40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.190+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3812486427 shutdown_connections 2024-04-21T08:41:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.190+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3812486427 wait complete. 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00ac0ba640 1 Processor -- start 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00ac0ba640 1 -- start start 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00ac0ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00a4100e80 0x7f00a4104e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00ac0ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00a4101fd0 0x7f00a4103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00ac0ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a4102990 0x7f00a4103980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00ac0ba640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f00a410c9c0 con 0x7f00a4100e80 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00ac0ba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00a410c840 con 0x7f00a4102990 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00ac0ba640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f00a410cb40 con 0x7f00a4101fd0 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00a9e56640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00a4100e80 0x7f00a4104e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00a9e56640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00a4100e80 0x7f00a4104e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49370/0 (socket says 172.21.15.7:49370) 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00a9e56640 1 -- 172.21.15.7:0/3706272207 learned_addr learned my addr 172.21.15.7:0/3706272207 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00aa657640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a4102990 0x7f00a4103980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00a9655640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00a4101fd0 0x7f00a4103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00a9e56640 1 -- 172.21.15.7:0/3706272207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00a4101fd0 msgr2=0x7f00a4103440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00a9e56640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00a4101fd0 0x7f00a4103440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00a9e56640 1 -- 172.21.15.7:0/3706272207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a4102990 msgr2=0x7f00a4103980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00a9e56640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a4102990 0x7f00a4103980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00a9e56640 1 -- 172.21.15.7:0/3706272207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a4104210 con 0x7f00a4100e80 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00a9655640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00a4101fd0 0x7f00a4103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.191+0000 7f00a9e56640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00a4100e80 0x7f00a4104e00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f00a000e9f0 tx=0x7f00a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.192+0000 7f009affd640 1 -- 172.21.15.7:0/3706272207 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a000cd80 con 0x7f00a4100e80 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.192+0000 7f009affd640 1 -- 172.21.15.7:0/3706272207 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00a00108e0 con 0x7f00a4100e80 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.192+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3706272207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a41a22b0 con 0x7f00a4100e80 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.192+0000 7f009affd640 1 -- 172.21.15.7:0/3706272207 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a0018980 con 0x7f00a4100e80 2024-04-21T08:41:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.192+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3706272207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a41a2850 con 0x7f00a4100e80 2024-04-21T08:41:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.193+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3706272207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a41a2b10 con 0x7f00a4100e80 2024-04-21T08:41:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.194+0000 7f009affd640 1 -- 172.21.15.7:0/3706272207 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f00a0010430 con 0x7f00a4100e80 2024-04-21T08:41:45.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.195+0000 7f009affd640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f00900a3c30 0x7f00900a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:45.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.195+0000 7f00a9655640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f00900a3c30 0x7f00900a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:45.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.195+0000 7f009affd640 1 -- 172.21.15.7:0/3706272207 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f00a00d07e0 con 0x7f00a4100e80 2024-04-21T08:41:45.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.195+0000 7f00a9655640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f00900a3c30 0x7f00900a60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0094005d00 tx=0x7f0094005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:45.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.198+0000 7f009affd640 1 -- 172.21.15.7:0/3706272207 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f00a009d5c0 con 0x7f00a4100e80 2024-04-21T08:41:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:44 smithi016 ceph-mon[31259]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:44 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.16:6832/2178684530,v1:172.21.15.16:6833/2178684530] up:boot 2024-04-21T08:41:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:44 smithi016 ceph-mon[31259]: fsmap 2 up:standby 2024-04-21T08:41:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:44 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:41:45.314 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:41:45 smithi007 systemd[1]: Starting Ceph mds.j for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:41:45.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:44 smithi077 ceph-mon[31615]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:45.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:44 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.16:6832/2178684530,v1:172.21.15.16:6833/2178684530] up:boot 2024-04-21T08:41:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:44 smithi077 ceph-mon[31615]: fsmap 2 up:standby 2024-04-21T08:41:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:44 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:41:45.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.434+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3706272207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00a41a2ce0 con 0x7f00a4100e80 2024-04-21T08:41:45.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.435+0000 7f009affd640 1 -- 172.21.15.7:0/3706272207 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f00a009cec0 con 0x7f00a4100e80 2024-04-21T08:41:45.434 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:45.435 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:41:45.435 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 3 2024-04-21T08:41:45.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3706272207 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f00900a3c30 msgr2=0x7f00900a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:45.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f00900a3c30 0x7f00900a60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0094005d00 tx=0x7f0094005c70 comp rx=0 tx=0).stop 2024-04-21T08:41:45.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3706272207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00a4100e80 msgr2=0x7f00a4104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:45.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00a4100e80 0x7f00a4104e00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f00a000e9f0 tx=0x7f00a000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3706272207 shutdown_connections 2024-04-21T08:41:45.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f00900a3c30 0x7f00900a60f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00a4102990 0x7f00a4103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00a4101fd0 0x7f00a4103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 --2- 172.21.15.7:0/3706272207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00a4100e80 0x7f00a4104e00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3706272207 >> 172.21.15.7:0/3706272207 conn(0x7f00a40fc3f0 msgr2=0x7f00a40fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:45.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3706272207 shutdown_connections 2024-04-21T08:41:45.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:45.438+0000 7f00ac0ba640 1 -- 172.21.15.7:0/3706272207 wait complete. 2024-04-21T08:41:45.570 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:41:45 smithi007 podman[67319]: 2024-04-21 08:41:45.449452742 +0000 UTC m=+0.024669596 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:41:45.570 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:41:45 smithi007 podman[67319]: 2024-04-21 08:41:45.571693125 +0000 UTC m=+0.146910001 container create b54239fdaa877e0d8e1435ad5c16cd04f9db6ee0fd41413a56294ebf14564809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-04-21T08:41:45.873 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:41:45 smithi007 podman[67319]: 2024-04-21 08:41:45.782942176 +0000 UTC m=+0.358159011 container init b54239fdaa877e0d8e1435ad5c16cd04f9db6ee0fd41413a56294ebf14564809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-21T08:41:45.873 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:41:45 smithi007 podman[67319]: 2024-04-21 08:41:45.785881163 +0000 UTC m=+0.361098007 container start b54239fdaa877e0d8e1435ad5c16cd04f9db6ee0fd41413a56294ebf14564809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-j, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-21T08:41:45.873 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:41:45 smithi007 ceph-mds[67340]: -- 172.21.15.7:0/1712989970 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5577fb4c7a40 con 0x5577fb49ac00 2024-04-21T08:41:45.873 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:41:45 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-j[67336]: starting mds.j at 2024-04-21T08:41:45.874 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:41:45 smithi007 bash[67319]: b54239fdaa877e0d8e1435ad5c16cd04f9db6ee0fd41413a56294ebf14564809 2024-04-21T08:41:46.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:45 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3706272207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:46.151 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:41:45 smithi007 systemd[1]: Started Ceph mds.j for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:41:46.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:45 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3706272207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:46.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:45 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3706272207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:47.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:46 smithi077 ceph-mon[31615]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:46 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.7:6834/2001227851,v1:172.21.15.7:6835/2001227851] up:boot 2024-04-21T08:41:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:46 smithi077 ceph-mon[31615]: fsmap 3 up:standby 2024-04-21T08:41:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:41:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:46 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:46 smithi077 ceph-mon[31615]: Deploying daemon mds.d on smithi007 2024-04-21T08:41:47.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:46 smithi007 ceph-mon[28477]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:47.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:46 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.7:6834/2001227851,v1:172.21.15.7:6835/2001227851] up:boot 2024-04-21T08:41:47.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:46 smithi007 ceph-mon[28477]: fsmap 3 up:standby 2024-04-21T08:41:47.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:41:47.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:47.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:47.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:47.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:47.282 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:47.282 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:46 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:47.282 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:46 smithi007 ceph-mon[28477]: Deploying daemon mds.d on smithi007 2024-04-21T08:41:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:46 smithi016 ceph-mon[31259]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:46 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.7:6834/2001227851,v1:172.21.15.7:6835/2001227851] up:boot 2024-04-21T08:41:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:46 smithi016 ceph-mon[31259]: fsmap 3 up:standby 2024-04-21T08:41:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:41:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:46 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:46 smithi016 ceph-mon[31259]: Deploying daemon mds.d on smithi007 2024-04-21T08:41:47.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:47.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.567+0000 7f4b1647a640 1 -- 172.21.15.7:0/4198044178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b10102a00 msgr2=0x7f4b10104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:47.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.567+0000 7f4b1647a640 1 --2- 172.21.15.7:0/4198044178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b10102a00 0x7f4b10104df0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009900 tx=0x7f4b0002bc30 comp rx=0 tx=0).stop 2024-04-21T08:41:47.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.567+0000 7f4b1647a640 1 -- 172.21.15.7:0/4198044178 shutdown_connections 2024-04-21T08:41:47.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.567+0000 7f4b1647a640 1 --2- 172.21.15.7:0/4198044178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b10107c60 0x7f4b1010a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:47.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.567+0000 7f4b1647a640 1 --2- 172.21.15.7:0/4198044178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b10105330 0x7f4b10107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:47.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.567+0000 7f4b1647a640 1 --2- 172.21.15.7:0/4198044178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b10102a00 0x7f4b10104df0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:47.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.567+0000 7f4b1647a640 1 -- 172.21.15.7:0/4198044178 >> 172.21.15.7:0/4198044178 conn(0x7f4b100fc3f0 msgr2=0x7f4b100fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:47.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.567+0000 7f4b1647a640 1 -- 172.21.15.7:0/4198044178 shutdown_connections 2024-04-21T08:41:47.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.567+0000 7f4b1647a640 1 -- 172.21.15.7:0/4198044178 wait complete. 2024-04-21T08:41:47.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.567+0000 7f4b1647a640 1 Processor -- start 2024-04-21T08:41:47.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.567+0000 7f4b1647a640 1 -- start start 2024-04-21T08:41:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b1647a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b10102a00 0x7f4b10198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b1647a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b10105330 0x7f4b101986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b1647a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b10107c60 0x7f4b1019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b1647a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1010dab0 con 0x7f4b10107c60 2024-04-21T08:41:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b1647a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1010d930 con 0x7f4b10105330 2024-04-21T08:41:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b1647a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1010dc30 con 0x7f4b10102a00 2024-04-21T08:41:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b14a17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b10107c60 0x7f4b1019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b14a17640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b10107c60 0x7f4b1019f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49398/0 (socket says 172.21.15.7:49398) 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b14a17640 1 -- 172.21.15.7:0/896437350 learned_addr learned my addr 172.21.15.7:0/896437350 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b0f7fe640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b10105330 0x7f4b101986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b0ffff640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b10102a00 0x7f4b10198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b14a17640 1 -- 172.21.15.7:0/896437350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b10102a00 msgr2=0x7f4b10198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b14a17640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b10102a00 0x7f4b10198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b14a17640 1 -- 172.21.15.7:0/896437350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b10105330 msgr2=0x7f4b101986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b14a17640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b10105330 0x7f4b101986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b14a17640 1 -- 172.21.15.7:0/896437350 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1019fda0 con 0x7f4b10107c60 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.568+0000 7f4b14a17640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b10107c60 0x7f4b1019f730 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800eb10 tx=0x7f4b0800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.569+0000 7f4b0d7fa640 1 -- 172.21.15.7:0/896437350 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b08004b20 con 0x7f4b10107c60 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.569+0000 7f4b1647a640 1 -- 172.21.15.7:0/896437350 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b101a0090 con 0x7f4b10107c60 2024-04-21T08:41:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.569+0000 7f4b1647a640 1 -- 172.21.15.7:0/896437350 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b101a05e0 con 0x7f4b10107c60 2024-04-21T08:41:47.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.569+0000 7f4b0d7fa640 1 -- 172.21.15.7:0/896437350 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b08004cc0 con 0x7f4b10107c60 2024-04-21T08:41:47.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.569+0000 7f4b0d7fa640 1 -- 172.21.15.7:0/896437350 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b08010670 con 0x7f4b10107c60 2024-04-21T08:41:47.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.570+0000 7f4b1647a640 1 -- 172.21.15.7:0/896437350 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af00052c0 con 0x7f4b10107c60 2024-04-21T08:41:47.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.571+0000 7f4b0d7fa640 1 -- 172.21.15.7:0/896437350 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f4b08010810 con 0x7f4b10107c60 2024-04-21T08:41:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.572+0000 7f4b0d7fa640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4afc0a3c30 0x7f4afc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.572+0000 7f4b0ffff640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4afc0a3c30 0x7f4afc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.572+0000 7f4b0d7fa640 1 -- 172.21.15.7:0/896437350 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4b080d0630 con 0x7f4b10107c60 2024-04-21T08:41:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.572+0000 7f4b0ffff640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4afc0a3c30 0x7f4afc0a60f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009230 tx=0x7f4b00005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:47.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.575+0000 7f4b0d7fa640 1 -- 172.21.15.7:0/896437350 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b08053ff0 con 0x7f4b10107c60 2024-04-21T08:41:47.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.814+0000 7f4b1647a640 1 -- 172.21.15.7:0/896437350 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4af00055b0 con 0x7f4b10107c60 2024-04-21T08:41:47.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.815+0000 7f4b0d7fa640 1 -- 172.21.15.7:0/896437350 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3040 (secure 0 0 0) 0x7f4b0809d410 con 0x7f4b10107c60 2024-04-21T08:41:47.814 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:47.814 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:41:47.815 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 4 2024-04-21T08:41:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 -- 172.21.15.7:0/896437350 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4afc0a3c30 msgr2=0x7f4afc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4afc0a3c30 0x7f4afc0a60f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009230 tx=0x7f4b00005fd0 comp rx=0 tx=0).stop 2024-04-21T08:41:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 -- 172.21.15.7:0/896437350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b10107c60 msgr2=0x7f4b1019f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b10107c60 0x7f4b1019f730 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800eb10 tx=0x7f4b0800c2d0 comp rx=0 tx=0).stop 2024-04-21T08:41:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 -- 172.21.15.7:0/896437350 shutdown_connections 2024-04-21T08:41:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4afc0a3c30 0x7f4afc0a60f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b10107c60 0x7f4b1019f730 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b10105330 0x7f4b101986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 --2- 172.21.15.7:0/896437350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b10102a00 0x7f4b10198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 -- 172.21.15.7:0/896437350 >> 172.21.15.7:0/896437350 conn(0x7f4b100fc3f0 msgr2=0x7f4b10103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:47.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 -- 172.21.15.7:0/896437350 shutdown_connections 2024-04-21T08:41:47.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:47.817+0000 7f4b1647a640 1 -- 172.21.15.7:0/896437350 wait complete. 2024-04-21T08:41:48.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:47 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/896437350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:48.235 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:41:48 smithi007 systemd[1]: Starting Ceph mds.d for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:41:48.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:47 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/896437350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:48.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:47 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/896437350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:48.505 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:41:48 smithi007 podman[67675]: 2024-04-21 08:41:48.139124634 +0000 UTC m=+0.017747554 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:41:48.506 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:41:48 smithi007 podman[67675]: 2024-04-21 08:41:48.2526164 +0000 UTC m=+0.131239359 container create 6812fd59cc5d014656f46b92a96b39317c26b04b5ccabea195a913dfabf62c2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-21T08:41:48.506 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:41:48 smithi007 podman[67675]: 2024-04-21 08:41:48.480222167 +0000 UTC m=+0.358845085 container init 6812fd59cc5d014656f46b92a96b39317c26b04b5ccabea195a913dfabf62c2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-d, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:41:48.506 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:41:48 smithi007 podman[67675]: 2024-04-21 08:41:48.482933518 +0000 UTC m=+0.361556436 container start 6812fd59cc5d014656f46b92a96b39317c26b04b5ccabea195a913dfabf62c2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:41:48.506 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:41:48 smithi007 ceph-mds[67695]: -- 172.21.15.7:0/3673378292 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564b071b9a40 con 0x564b0718d000 2024-04-21T08:41:48.762 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:41:48 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-d[67691]: starting mds.d at 2024-04-21T08:41:49.151 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:41:48 smithi007 bash[67675]: 6812fd59cc5d014656f46b92a96b39317c26b04b5ccabea195a913dfabf62c2d 2024-04-21T08:41:49.151 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:41:48 smithi007 systemd[1]: Started Ceph mds.d for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:41:49.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:48 smithi007 ceph-mon[28477]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:49.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:49.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:49.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:49.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:49.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:49.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:48 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:49.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:48 smithi016 ceph-mon[31259]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:49.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:48 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:49.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:48 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:49.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:48 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:49.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:48 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:49.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:48 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:49.206 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:48 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:48 smithi077 ceph-mon[31615]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:48 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:48 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:48 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:48 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:48 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:48 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:49.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.951+0000 7f3bba83a640 1 -- 172.21.15.7:0/141302245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bb41021f0 msgr2=0x7f3bb4108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.951+0000 7f3bba83a640 1 --2- 172.21.15.7:0/141302245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bb41021f0 0x7f3bb4108a80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009ca0 tx=0x7f3bac02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.951+0000 7f3bba83a640 1 -- 172.21.15.7:0/141302245 shutdown_connections 2024-04-21T08:41:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.951+0000 7f3bba83a640 1 --2- 172.21.15.7:0/141302245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bb41021f0 0x7f3bb4108a80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.951+0000 7f3bba83a640 1 --2- 172.21.15.7:0/141302245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bb4101830 0x7f3bb4101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.951+0000 7f3bba83a640 1 --2- 172.21.15.7:0/141302245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb4100640 0x7f3bb4100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.951+0000 7f3bba83a640 1 -- 172.21.15.7:0/141302245 >> 172.21.15.7:0/141302245 conn(0x7f3bb40fbbb0 msgr2=0x7f3bb40fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.951+0000 7f3bba83a640 1 -- 172.21.15.7:0/141302245 shutdown_connections 2024-04-21T08:41:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.951+0000 7f3bba83a640 1 -- 172.21.15.7:0/141302245 wait complete. 2024-04-21T08:41:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bba83a640 1 Processor -- start 2024-04-21T08:41:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bba83a640 1 -- start start 2024-04-21T08:41:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bba83a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bb4100640 0x7f3bb419a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bba83a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bb4101830 0x7f3bb419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bba83a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb41021f0 0x7f3bb41a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bba83a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb410ca00 con 0x7f3bb4100640 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bba83a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb410c880 con 0x7f3bb4101830 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bba83a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb410cb80 con 0x7f3bb41021f0 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bb3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bb4100640 0x7f3bb419a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bb3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bb4100640 0x7f3bb419a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49410/0 (socket says 172.21.15.7:49410) 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bb3fff640 1 -- 172.21.15.7:0/1153708401 learned_addr learned my addr 172.21.15.7:0/1153708401 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.952+0000 7f3bb37fe640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bb4101830 0x7f3bb419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bb8dd7640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb41021f0 0x7f3bb41a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bb37fe640 1 -- 172.21.15.7:0/1153708401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb41021f0 msgr2=0x7f3bb41a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bb37fe640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb41021f0 0x7f3bb41a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bb37fe640 1 -- 172.21.15.7:0/1153708401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bb4100640 msgr2=0x7f3bb419a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bb37fe640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bb4100640 0x7f3bb419a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bb37fe640 1 -- 172.21.15.7:0/1153708401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb41a2000 con 0x7f3bb4101830 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bb3fff640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bb4100640 0x7f3bb419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:41:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bb37fe640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bb4101830 0x7f3bb419a920 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400e9f0 tx=0x7f3ba400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bb17fa640 1 -- 172.21.15.7:0/1153708401 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba400ccc0 con 0x7f3bb4101830 2024-04-21T08:41:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bb17fa640 1 -- 172.21.15.7:0/1153708401 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ba40108e0 con 0x7f3bb4101830 2024-04-21T08:41:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bba83a640 1 -- 172.21.15.7:0/1153708401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bb41a22f0 con 0x7f3bb4101830 2024-04-21T08:41:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bb17fa640 1 -- 172.21.15.7:0/1153708401 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4018880 con 0x7f3bb4101830 2024-04-21T08:41:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.953+0000 7f3bba83a640 1 -- 172.21.15.7:0/1153708401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bb41a2840 con 0x7f3bb4101830 2024-04-21T08:41:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.954+0000 7f3bba83a640 1 -- 172.21.15.7:0/1153708401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bb41a2aa0 con 0x7f3bb4101830 2024-04-21T08:41:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.955+0000 7f3bb17fa640 1 -- 172.21.15.7:0/1153708401 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f3ba4010430 con 0x7f3bb4101830 2024-04-21T08:41:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.956+0000 7f3bb17fa640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3b9c0a3bb0 0x7f3b9c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.956+0000 7f3bb3fff640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3b9c0a3bb0 0x7f3b9c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.956+0000 7f3bb17fa640 1 -- 172.21.15.7:0/1153708401 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f3ba40d03a0 con 0x7f3bb4101830 2024-04-21T08:41:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.956+0000 7f3bb3fff640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3b9c0a3bb0 0x7f3b9c0a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3ba80058c0 tx=0x7f3ba8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:49.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:49.960+0000 7f3bb17fa640 1 -- 172.21.15.7:0/1153708401 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3ba409d180 con 0x7f3bb4101830 2024-04-21T08:41:50.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:49 smithi016 ceph-mon[31259]: Deploying daemon mds.e on smithi016 2024-04-21T08:41:50.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:49 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.7:6836/1152814354,v1:172.21.15.7:6837/1152814354] up:boot 2024-04-21T08:41:50.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:49 smithi016 ceph-mon[31259]: fsmap 4 up:standby 2024-04-21T08:41:50.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:49 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:41:50.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:49 smithi007 ceph-mon[28477]: Deploying daemon mds.e on smithi016 2024-04-21T08:41:50.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:49 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.7:6836/1152814354,v1:172.21.15.7:6837/1152814354] up:boot 2024-04-21T08:41:50.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:49 smithi007 ceph-mon[28477]: fsmap 4 up:standby 2024-04-21T08:41:50.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:49 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:41:50.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.189+0000 7f3bba83a640 1 -- 172.21.15.7:0/1153708401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bb41a3170 con 0x7f3bb4101830 2024-04-21T08:41:50.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.190+0000 7f3bb17fa640 1 -- 172.21.15.7:0/1153708401 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3862 (secure 0 0 0) 0x7f3ba40bd0c0 con 0x7f3bb4101830 2024-04-21T08:41:50.190 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:50.190 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:41:50.190 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 5 2024-04-21T08:41:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.192+0000 7f3bba83a640 1 -- 172.21.15.7:0/1153708401 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3b9c0a3bb0 msgr2=0x7f3b9c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.192+0000 7f3bba83a640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3b9c0a3bb0 0x7f3b9c0a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3ba80058c0 tx=0x7f3ba8005f90 comp rx=0 tx=0).stop 2024-04-21T08:41:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.192+0000 7f3bba83a640 1 -- 172.21.15.7:0/1153708401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bb4101830 msgr2=0x7f3bb419a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.192+0000 7f3bba83a640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bb4101830 0x7f3bb419a920 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400e9f0 tx=0x7f3ba400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:41:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.192+0000 7f3bba83a640 1 -- 172.21.15.7:0/1153708401 shutdown_connections 2024-04-21T08:41:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.192+0000 7f3bba83a640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3b9c0a3bb0 0x7f3b9c0a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.192+0000 7f3bba83a640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3bb41021f0 0x7f3bb41a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.192+0000 7f3bba83a640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3bb4101830 0x7f3bb419a920 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.192+0000 7f3bba83a640 1 --2- 172.21.15.7:0/1153708401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bb4100640 0x7f3bb419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.192+0000 7f3bba83a640 1 -- 172.21.15.7:0/1153708401 >> 172.21.15.7:0/1153708401 conn(0x7f3bb40fbbb0 msgr2=0x7f3bb40fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.192+0000 7f3bba83a640 1 -- 172.21.15.7:0/1153708401 shutdown_connections 2024-04-21T08:41:50.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:50.193+0000 7f3bba83a640 1 -- 172.21.15.7:0/1153708401 wait complete. 2024-04-21T08:41:50.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:49 smithi077 ceph-mon[31615]: Deploying daemon mds.e on smithi016 2024-04-21T08:41:50.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:49 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.7:6836/1152814354,v1:172.21.15.7:6837/1152814354] up:boot 2024-04-21T08:41:50.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:49 smithi077 ceph-mon[31615]: fsmap 4 up:standby 2024-04-21T08:41:50.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:49 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:41:51.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:50 smithi077 ceph-mon[31615]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:51.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:50 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1153708401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:50 smithi016 ceph-mon[31259]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:50 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1153708401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:51.304 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:41:51 smithi016 systemd[1]: Starting Ceph mds.e for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:41:51.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:50 smithi007 ceph-mon[28477]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:51.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:50 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1153708401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:51.597 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:41:51 smithi016 podman[60540]: 2024-04-21 08:41:51.249785322 +0000 UTC m=+0.018429722 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:41:51.598 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:41:51 smithi016 podman[60540]: 2024-04-21 08:41:51.363119697 +0000 UTC m=+0.131764085 container create 01e19dd24fe98e83914f151bd50eb300aca92f11ebc0de41cd167540502f34a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-21T08:41:51.598 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:41:51 smithi016 podman[60540]: 2024-04-21 08:41:51.59898526 +0000 UTC m=+0.367629641 container init 01e19dd24fe98e83914f151bd50eb300aca92f11ebc0de41cd167540502f34a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-21T08:41:52.054 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:41:51 smithi016 podman[60540]: 2024-04-21 08:41:51.60161851 +0000 UTC m=+0.370262891 container start 01e19dd24fe98e83914f151bd50eb300aca92f11ebc0de41cd167540502f34a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:41:52.054 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:41:51 smithi016 ceph-mds[60559]: -- 172.21.15.16:0/2209381544 <== mon.2 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5617450fba40 con 0x5617450cf000 2024-04-21T08:41:52.054 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:41:51 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-e[60555]: starting mds.e at 2024-04-21T08:41:52.054 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:41:51 smithi016 bash[60540]: 01e19dd24fe98e83914f151bd50eb300aca92f11ebc0de41cd167540502f34a1 2024-04-21T08:41:52.054 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:41:51 smithi016 systemd[1]: Started Ceph mds.e for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:41:52.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:52.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 -- 172.21.15.7:0/3477880958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f0102990 msgr2=0x7f46f01092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 --2- 172.21.15.7:0/3477880958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f0102990 0x7f46f01092b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46e8009ca0 tx=0x7f46e802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 -- 172.21.15.7:0/3477880958 shutdown_connections 2024-04-21T08:41:52.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 --2- 172.21.15.7:0/3477880958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f0102990 0x7f46f01092b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 --2- 172.21.15.7:0/3477880958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f0101fd0 0x7f46f0102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 --2- 172.21.15.7:0/3477880958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46f0100e80 0x7f46f01012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 -- 172.21.15.7:0/3477880958 >> 172.21.15.7:0/3477880958 conn(0x7f46f00fc3f0 msgr2=0x7f46f00fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 -- 172.21.15.7:0/3477880958 shutdown_connections 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 -- 172.21.15.7:0/3477880958 wait complete. 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 Processor -- start 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 -- start start 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46f0100e80 0x7f46f0104e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f0101fd0 0x7f46f0103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f0102990 0x7f46f0103980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f46f010c9e0 con 0x7f46f0102990 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.320+0000 7f46f6b7e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46f010c860 con 0x7f46f0101fd0 2024-04-21T08:41:52.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f6b7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f46f010cb60 con 0x7f46f0100e80 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f511b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f0102990 0x7f46f0103980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f511b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f0102990 0x7f46f0103980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49436/0 (socket says 172.21.15.7:49436) 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f511b640 1 -- 172.21.15.7:0/1103596742 learned_addr learned my addr 172.21.15.7:0/1103596742 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f491a640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46f0100e80 0x7f46f0104e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46effff640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f0101fd0 0x7f46f0103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f511b640 1 -- 172.21.15.7:0/1103596742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46f0100e80 msgr2=0x7f46f0104e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f511b640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46f0100e80 0x7f46f0104e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f511b640 1 -- 172.21.15.7:0/1103596742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f0101fd0 msgr2=0x7f46f0103440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f511b640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f0101fd0 0x7f46f0103440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f511b640 1 -- 172.21.15.7:0/1103596742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f0104210 con 0x7f46f0102990 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f511b640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f0102990 0x7f46f0103980 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f46e802c700 tx=0x7f46e80093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46edffb640 1 -- 172.21.15.7:0/1103596742 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e8041020 con 0x7f46f0102990 2024-04-21T08:41:52.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46edffb640 1 -- 172.21.15.7:0/1103596742 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46e8034d60 con 0x7f46f0102990 2024-04-21T08:41:52.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46f6b7e640 1 -- 172.21.15.7:0/1103596742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f01a2210 con 0x7f46f0102990 2024-04-21T08:41:52.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.321+0000 7f46edffb640 1 -- 172.21.15.7:0/1103596742 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e8039040 con 0x7f46f0102990 2024-04-21T08:41:52.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.322+0000 7f46f6b7e640 1 -- 172.21.15.7:0/1103596742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f01a2760 con 0x7f46f0102990 2024-04-21T08:41:52.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.323+0000 7f46f6b7e640 1 -- 172.21.15.7:0/1103596742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f01a2a80 con 0x7f46f0102990 2024-04-21T08:41:52.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.324+0000 7f46edffb640 1 -- 172.21.15.7:0/1103596742 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f46e8044020 con 0x7f46f0102990 2024-04-21T08:41:52.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.324+0000 7f46edffb640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f46d80a3bb0 0x7f46d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:52.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.325+0000 7f46edffb640 1 -- 172.21.15.7:0/1103596742 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f46e80f22e0 con 0x7f46f0102990 2024-04-21T08:41:52.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.325+0000 7f46f491a640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f46d80a3bb0 0x7f46d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:52.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.325+0000 7f46f491a640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f46d80a3bb0 0x7f46d80a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f46e0009bd0 tx=0x7f46e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:52.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.328+0000 7f46edffb640 1 -- 172.21.15.7:0/1103596742 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46e804b020 con 0x7f46f0102990 2024-04-21T08:41:52.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.556+0000 7f46f6b7e640 1 -- 172.21.15.7:0/1103596742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46f01a2c50 con 0x7f46f0102990 2024-04-21T08:41:52.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.556+0000 7f46edffb640 1 -- 172.21.15.7:0/1103596742 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4687 (secure 0 0 0) 0x7f46e80bf160 con 0x7f46f0102990 2024-04-21T08:41:52.556 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:52.556 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-21T08:41:52.556 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 6 2024-04-21T08:41:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 -- 172.21.15.7:0/1103596742 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f46d80a3bb0 msgr2=0x7f46d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f46d80a3bb0 0x7f46d80a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f46e0009bd0 tx=0x7f46e0005f90 comp rx=0 tx=0).stop 2024-04-21T08:41:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 -- 172.21.15.7:0/1103596742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f0102990 msgr2=0x7f46f0103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f0102990 0x7f46f0103980 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f46e802c700 tx=0x7f46e80093f0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 -- 172.21.15.7:0/1103596742 shutdown_connections 2024-04-21T08:41:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f46d80a3bb0 0x7f46d80a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46f0102990 0x7f46f0103980 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46f0101fd0 0x7f46f0103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 --2- 172.21.15.7:0/1103596742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46f0100e80 0x7f46f0104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 -- 172.21.15.7:0/1103596742 >> 172.21.15.7:0/1103596742 conn(0x7f46f00fc3f0 msgr2=0x7f46f00fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 -- 172.21.15.7:0/1103596742 shutdown_connections 2024-04-21T08:41:52.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:52.558+0000 7f46f6b7e640 1 -- 172.21.15.7:0/1103596742 wait complete. 2024-04-21T08:41:52.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: Deploying daemon mds.f on smithi077 2024-04-21T08:41:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.16:6834/4020622072,v1:172.21.15.16:6835/4020622072] up:boot 2024-04-21T08:41:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: fsmap 5 up:standby 2024-04-21T08:41:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:41:52.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:52 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1103596742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:53.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:53.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:53.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:53.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:53.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: Deploying daemon mds.f on smithi077 2024-04-21T08:41:53.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.16:6834/4020622072,v1:172.21.15.16:6835/4020622072] up:boot 2024-04-21T08:41:53.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: fsmap 5 up:standby 2024-04-21T08:41:53.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:41:53.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:52 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1103596742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:53.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: Deploying daemon mds.f on smithi077 2024-04-21T08:41:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.16:6834/4020622072,v1:172.21.15.16:6835/4020622072] up:boot 2024-04-21T08:41:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: fsmap 5 up:standby 2024-04-21T08:41:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:41:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:52 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1103596742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:54.104 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:41:53 smithi077 systemd[1]: Starting Ceph mds.f for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:41:54.427 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:41:54 smithi077 podman[62043]: 2024-04-21 08:41:54.061924843 +0000 UTC m=+0.017577928 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:41:54.427 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:41:54 smithi077 podman[62043]: 2024-04-21 08:41:54.182966321 +0000 UTC m=+0.138619389 container create decaa5941cd00e0ab21bbd3541ac93764c36b567e5bd39d8d1e2ad24983042f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:41:54.427 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:41:54 smithi077 podman[62043]: 2024-04-21 08:41:54.427785922 +0000 UTC m=+0.383438990 container init decaa5941cd00e0ab21bbd3541ac93764c36b567e5bd39d8d1e2ad24983042f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-21T08:41:54.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:54.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.679+0000 7f5a69d75640 1 -- 172.21.15.7:0/1612802283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a64107c80 msgr2=0x7f5a6410a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:54.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.679+0000 7f5a69d75640 1 --2- 172.21.15.7:0/1612802283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a64107c80 0x7f5a6410a0d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009ca0 tx=0x7f5a5c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.679+0000 7f5a69d75640 1 -- 172.21.15.7:0/1612802283 shutdown_connections 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.679+0000 7f5a69d75640 1 --2- 172.21.15.7:0/1612802283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a64107c80 0x7f5a6410a0d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.679+0000 7f5a69d75640 1 --2- 172.21.15.7:0/1612802283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a64105350 0x7f5a64107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.679+0000 7f5a69d75640 1 --2- 172.21.15.7:0/1612802283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a64102a20 0x7f5a64104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.679+0000 7f5a69d75640 1 -- 172.21.15.7:0/1612802283 >> 172.21.15.7:0/1612802283 conn(0x7f5a640fc410 msgr2=0x7f5a640fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.679+0000 7f5a69d75640 1 -- 172.21.15.7:0/1612802283 shutdown_connections 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.679+0000 7f5a69d75640 1 -- 172.21.15.7:0/1612802283 wait complete. 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.680+0000 7f5a69d75640 1 Processor -- start 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.680+0000 7f5a69d75640 1 -- start start 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.680+0000 7f5a69d75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a64102a20 0x7f5a64198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.680+0000 7f5a69d75640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a64105350 0x7f5a64198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.680+0000 7f5a69d75640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a64107c80 0x7f5a6419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:54.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.680+0000 7f5a69d75640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6410dae0 con 0x7f5a64105350 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.680+0000 7f5a69d75640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6410d960 con 0x7f5a64102a20 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.680+0000 7f5a69d75640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6410dc60 con 0x7f5a64107c80 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a62ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a64105350 0x7f5a64198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a62ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a64105350 0x7f5a64198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34758/0 (socket says 172.21.15.7:34758) 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a62ffd640 1 -- 172.21.15.7:0/2468464227 learned_addr learned my addr 172.21.15.7:0/2468464227 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a63fff640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a64107c80 0x7f5a6419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a637fe640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a64102a20 0x7f5a64198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a62ffd640 1 -- 172.21.15.7:0/2468464227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a64107c80 msgr2=0x7f5a6419f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a62ffd640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a64107c80 0x7f5a6419f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a62ffd640 1 -- 172.21.15.7:0/2468464227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a64102a20 msgr2=0x7f5a64198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a62ffd640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a64102a20 0x7f5a64198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a62ffd640 1 -- 172.21.15.7:0/2468464227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6419fd70 con 0x7f5a64105350 2024-04-21T08:41:54.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a62ffd640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a64105350 0x7f5a64198690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800e9f0 tx=0x7f5a5800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:54.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a60ff9640 1 -- 172.21.15.7:0/2468464227 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5800ccc0 con 0x7f5a64105350 2024-04-21T08:41:54.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a60ff9640 1 -- 172.21.15.7:0/2468464227 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a580054d0 con 0x7f5a64105350 2024-04-21T08:41:54.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a69d75640 1 -- 172.21.15.7:0/2468464227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a641a0060 con 0x7f5a64105350 2024-04-21T08:41:54.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a60ff9640 1 -- 172.21.15.7:0/2468464227 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a58010430 con 0x7f5a64105350 2024-04-21T08:41:54.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.681+0000 7f5a69d75640 1 -- 172.21.15.7:0/2468464227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a641a0560 con 0x7f5a64105350 2024-04-21T08:41:54.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.682+0000 7f5a69d75640 1 -- 172.21.15.7:0/2468464227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a640fdf00 con 0x7f5a64105350 2024-04-21T08:41:54.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.683+0000 7f5a60ff9640 1 -- 172.21.15.7:0/2468464227 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f5a580046c0 con 0x7f5a64105350 2024-04-21T08:41:54.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.684+0000 7f5a60ff9640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5a4c0a3bb0 0x7f5a4c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:54.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.684+0000 7f5a637fe640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5a4c0a3bb0 0x7f5a4c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:54.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.684+0000 7f5a60ff9640 1 -- 172.21.15.7:0/2468464227 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5a580d0350 con 0x7f5a64105350 2024-04-21T08:41:54.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.685+0000 7f5a637fe640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5a4c0a3bb0 0x7f5a4c0a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5a54005fd0 tx=0x7f5a54005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:54.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.689+0000 7f5a60ff9640 1 -- 172.21.15.7:0/2468464227 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a5809d1d0 con 0x7f5a64105350 2024-04-21T08:41:54.759 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:41:54 smithi077 podman[62043]: 2024-04-21 08:41:54.430604957 +0000 UTC m=+0.386258032 container start decaa5941cd00e0ab21bbd3541ac93764c36b567e5bd39d8d1e2ad24983042f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-21T08:41:54.759 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:41:54 smithi077 ceph-mds[62062]: -- 172.21.15.77:0/3295489393 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f3d2e77a40 con 0x55f3d2e4a800 2024-04-21T08:41:54.759 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:41:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-f[62058]: starting mds.f at 2024-04-21T08:41:54.760 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:41:54 smithi077 bash[62043]: decaa5941cd00e0ab21bbd3541ac93764c36b567e5bd39d8d1e2ad24983042f8 2024-04-21T08:41:54.760 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:41:54 smithi077 systemd[1]: Started Ceph mds.f for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:41:54.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.916+0000 7f5a69d75640 1 -- 172.21.15.7:0/2468464227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a6402d5c0 con 0x7f5a64105350 2024-04-21T08:41:54.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.917+0000 7f5a60ff9640 1 -- 172.21.15.7:0/2468464227 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5512 (secure 0 0 0) 0x7f5a5809cad0 con 0x7f5a64105350 2024-04-21T08:41:54.917 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:54.917 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T08:41:54.917 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 7 2024-04-21T08:41:54.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 -- 172.21.15.7:0/2468464227 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5a4c0a3bb0 msgr2=0x7f5a4c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:54.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5a4c0a3bb0 0x7f5a4c0a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5a54005fd0 tx=0x7f5a54005850 comp rx=0 tx=0).stop 2024-04-21T08:41:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 -- 172.21.15.7:0/2468464227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a64105350 msgr2=0x7f5a64198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a64105350 0x7f5a64198690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800e9f0 tx=0x7f5a5800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 -- 172.21.15.7:0/2468464227 shutdown_connections 2024-04-21T08:41:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5a4c0a3bb0 0x7f5a4c0a6070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a64107c80 0x7f5a6419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a64105350 0x7f5a64198690 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 --2- 172.21.15.7:0/2468464227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a64102a20 0x7f5a64198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 -- 172.21.15.7:0/2468464227 >> 172.21.15.7:0/2468464227 conn(0x7f5a640fc410 msgr2=0x7f5a64103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 -- 172.21.15.7:0/2468464227 shutdown_connections 2024-04-21T08:41:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:54.919+0000 7f5a69d75640 1 -- 172.21.15.7:0/2468464227 wait complete. 2024-04-21T08:41:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:54 smithi016 ceph-mon[31259]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:55.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:54 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:55.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:54 smithi077 ceph-mon[31615]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:54 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:55.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:54 smithi007 ceph-mon[28477]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:55.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:55.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:55.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:55.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:55.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:55.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:54 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:56.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:55 smithi016 ceph-mon[31259]: Deploying daemon mds.c on smithi077 2024-04-21T08:41:56.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:55 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.77:6834/2205305943,v1:172.21.15.77:6835/2205305943] up:boot 2024-04-21T08:41:56.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:55 smithi016 ceph-mon[31259]: fsmap 6 up:standby 2024-04-21T08:41:56.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:55 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:41:56.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:55 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2468464227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:56.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:55 smithi077 ceph-mon[31615]: Deploying daemon mds.c on smithi077 2024-04-21T08:41:56.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:55 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.77:6834/2205305943,v1:172.21.15.77:6835/2205305943] up:boot 2024-04-21T08:41:56.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:55 smithi077 ceph-mon[31615]: fsmap 6 up:standby 2024-04-21T08:41:56.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:55 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:41:56.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:55 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2468464227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:56.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:55 smithi007 ceph-mon[28477]: Deploying daemon mds.c on smithi077 2024-04-21T08:41:56.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:55 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.77:6834/2205305943,v1:172.21.15.77:6835/2205305943] up:boot 2024-04-21T08:41:56.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:55 smithi007 ceph-mon[28477]: fsmap 6 up:standby 2024-04-21T08:41:56.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:55 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:41:56.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:55 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2468464227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:56.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:56 smithi077 ceph-mon[31615]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:56.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:57.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.047+0000 7facb57fb640 1 -- 172.21.15.7:0/4250368884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facb0100e80 msgr2=0x7facb01012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:57.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.047+0000 7facb57fb640 1 --2- 172.21.15.7:0/4250368884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facb0100e80 0x7facb01012a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7faca0009900 tx=0x7faca002bc30 comp rx=0 tx=0).stop 2024-04-21T08:41:57.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.047+0000 7facb57fb640 1 -- 172.21.15.7:0/4250368884 shutdown_connections 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.047+0000 7facb57fb640 1 --2- 172.21.15.7:0/4250368884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0102990 0x7facb01092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.047+0000 7facb57fb640 1 --2- 172.21.15.7:0/4250368884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb0101fd0 0x7facb0102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.047+0000 7facb57fb640 1 --2- 172.21.15.7:0/4250368884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facb0100e80 0x7facb01012a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.047+0000 7facb57fb640 1 -- 172.21.15.7:0/4250368884 >> 172.21.15.7:0/4250368884 conn(0x7facb00fc3f0 msgr2=0x7facb00fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.047+0000 7facb57fb640 1 -- 172.21.15.7:0/4250368884 shutdown_connections 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.047+0000 7facb57fb640 1 -- 172.21.15.7:0/4250368884 wait complete. 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.047+0000 7facb57fb640 1 Processor -- start 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facb57fb640 1 -- start start 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facb57fb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100e80 0x7facb0069dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facb57fb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb0101fd0 0x7facb019f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facb57fb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facb0102990 0x7facb01a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facb57fb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7facb010c9b0 con 0x7facb0102990 2024-04-21T08:41:57.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facb57fb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7facb010c830 con 0x7facb0101fd0 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facb57fb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7facb010cb30 con 0x7facb0100e80 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facaf7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facb0102990 0x7facb01a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facaeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100e80 0x7facb0069dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facaf7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facb0102990 0x7facb01a1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34772/0 (socket says 172.21.15.7:34772) 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facaf7fe640 1 -- 172.21.15.7:0/3668941412 learned_addr learned my addr 172.21.15.7:0/3668941412 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facae7fc640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb0101fd0 0x7facb019f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facaf7fe640 1 -- 172.21.15.7:0/3668941412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100e80 msgr2=0x7facb0069dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facaf7fe640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100e80 0x7facb0069dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facaf7fe640 1 -- 172.21.15.7:0/3668941412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb0101fd0 msgr2=0x7facb019f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facaf7fe640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb0101fd0 0x7facb019f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facaf7fe640 1 -- 172.21.15.7:0/3668941412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb01a2010 con 0x7facb0102990 2024-04-21T08:41:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facae7fc640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb0101fd0 0x7facb019f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7facaf7fe640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facb0102990 0x7facb01a1820 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7faca800eaa0 tx=0x7faca800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.048+0000 7fac97fff640 1 -- 172.21.15.7:0/3668941412 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca800cdb0 con 0x7facb0102990 2024-04-21T08:41:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.049+0000 7facb57fb640 1 -- 172.21.15.7:0/3668941412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb01a22a0 con 0x7facb0102990 2024-04-21T08:41:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.049+0000 7fac97fff640 1 -- 172.21.15.7:0/3668941412 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faca8005020 con 0x7facb0102990 2024-04-21T08:41:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.049+0000 7fac97fff640 1 -- 172.21.15.7:0/3668941412 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca80116d0 con 0x7facb0102990 2024-04-21T08:41:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.049+0000 7facb57fb640 1 -- 172.21.15.7:0/3668941412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facb01a27f0 con 0x7facb0102990 2024-04-21T08:41:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.050+0000 7facb57fb640 1 -- 172.21.15.7:0/3668941412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb0108230 con 0x7facb0102990 2024-04-21T08:41:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.051+0000 7fac97fff640 1 -- 172.21.15.7:0/3668941412 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7faca80051d0 con 0x7facb0102990 2024-04-21T08:41:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.051+0000 7fac97fff640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac980a3be0 0x7fac980a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.051+0000 7facaeffd640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac980a3be0 0x7fac980a60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.051+0000 7fac97fff640 1 -- 172.21.15.7:0/3668941412 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7faca80d00a0 con 0x7facb0102990 2024-04-21T08:41:57.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.052+0000 7facaeffd640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac980a3be0 0x7fac980a60a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faca0009230 tx=0x7faca0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:57.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:56 smithi016 ceph-mon[31259]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:57.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.056+0000 7fac97fff640 1 -- 172.21.15.7:0/3668941412 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faca8016030 con 0x7facb0102990 2024-04-21T08:41:57.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:56 smithi007 ceph-mon[28477]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.288+0000 7facb57fb640 1 -- 172.21.15.7:0/3668941412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facb002d770 con 0x7facb0102990 2024-04-21T08:41:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.289+0000 7fac97fff640 1 -- 172.21.15.7:0/3668941412 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5512 (secure 0 0 0) 0x7faca809cf20 con 0x7facb0102990 2024-04-21T08:41:57.288 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:57.289 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T08:41:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 7 2024-04-21T08:41:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 -- 172.21.15.7:0/3668941412 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac980a3be0 msgr2=0x7fac980a60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac980a3be0 0x7fac980a60a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faca0009230 tx=0x7faca0005fd0 comp rx=0 tx=0).stop 2024-04-21T08:41:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 -- 172.21.15.7:0/3668941412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facb0102990 msgr2=0x7facb01a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facb0102990 0x7facb01a1820 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7faca800eaa0 tx=0x7faca800ef60 comp rx=0 tx=0).stop 2024-04-21T08:41:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 -- 172.21.15.7:0/3668941412 shutdown_connections 2024-04-21T08:41:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fac980a3be0 0x7fac980a60a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7facb0102990 0x7facb01a1820 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7facb0101fd0 0x7facb019f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 --2- 172.21.15.7:0/3668941412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facb0100e80 0x7facb0069dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 -- 172.21.15.7:0/3668941412 >> 172.21.15.7:0/3668941412 conn(0x7facb00fc3f0 msgr2=0x7facb0105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 -- 172.21.15.7:0/3668941412 shutdown_connections 2024-04-21T08:41:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:57.291+0000 7facb57fb640 1 -- 172.21.15.7:0/3668941412 wait complete. 2024-04-21T08:41:57.604 INFO:journalctl@ceph.mds.c.smithi077.stdout:Apr 21 08:41:57 smithi077 systemd[1]: Started Ceph mds.c for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:41:58.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:57 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3668941412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:58.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:58.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:58.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:58.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:58.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:58.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:57 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:58.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:57 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3668941412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:58.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:57 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:58.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:57 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3668941412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:41:58.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:58.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:58.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:41:58.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:41:58.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:41:58.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:57 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:58 smithi016 ceph-mon[31259]: Deploying daemon mds.h on smithi016 2024-04-21T08:41:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:58 smithi016 ceph-mon[31259]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:58 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.77:6836/674822179,v1:172.21.15.77:6837/674822179] up:boot 2024-04-21T08:41:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:58 smithi016 ceph-mon[31259]: fsmap 7 up:standby 2024-04-21T08:41:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:58 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:41:59.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:58 smithi077 ceph-mon[31615]: Deploying daemon mds.h on smithi016 2024-04-21T08:41:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:58 smithi077 ceph-mon[31615]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:58 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.77:6836/674822179,v1:172.21.15.77:6837/674822179] up:boot 2024-04-21T08:41:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:58 smithi077 ceph-mon[31615]: fsmap 7 up:standby 2024-04-21T08:41:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:58 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:41:59.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:58 smithi007 ceph-mon[28477]: Deploying daemon mds.h on smithi016 2024-04-21T08:41:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:58 smithi007 ceph-mon[28477]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:41:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:58 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.77:6836/674822179,v1:172.21.15.77:6837/674822179] up:boot 2024-04-21T08:41:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:58 smithi007 ceph-mon[28477]: fsmap 7 up:standby 2024-04-21T08:41:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:58 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:41:59.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:41:59.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.417+0000 7f3473f52640 1 -- 172.21.15.7:0/3272260947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c103180 msgr2=0x7f346c1055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:59.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.417+0000 7f3473f52640 1 --2- 172.21.15.7:0/3272260947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c103180 0x7f346c1055f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3460009900 tx=0x7f346002bc30 comp rx=0 tx=0).stop 2024-04-21T08:41:59.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.417+0000 7f3473f52640 1 -- 172.21.15.7:0/3272260947 shutdown_connections 2024-04-21T08:41:59.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.417+0000 7f3473f52640 1 --2- 172.21.15.7:0/3272260947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f346c105b30 0x7f346c105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.417+0000 7f3473f52640 1 --2- 172.21.15.7:0/3272260947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c103180 0x7f346c1055f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.417+0000 7f3473f52640 1 --2- 172.21.15.7:0/3272260947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f346c100810 0x7f346c102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.417+0000 7f3473f52640 1 -- 172.21.15.7:0/3272260947 >> 172.21.15.7:0/3272260947 conn(0x7f346c0fc410 msgr2=0x7f346c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.417+0000 7f3473f52640 1 -- 172.21.15.7:0/3272260947 shutdown_connections 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.417+0000 7f3473f52640 1 -- 172.21.15.7:0/3272260947 wait complete. 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f3473f52640 1 Processor -- start 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f3473f52640 1 -- start start 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f3473f52640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c100810 0x7f346c19e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f3473f52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f346c103180 0x7f346c19ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f3473f52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f346c105b30 0x7f346c1a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f3473f52640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f346c1140b0 con 0x7f346c100810 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f3473f52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f346c113f30 con 0x7f346c103180 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f3473f52640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f346c114230 con 0x7f346c105b30 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f3471cee640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c100810 0x7f346c19e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f3471cee640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c100810 0x7f346c19e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34798/0 (socket says 172.21.15.7:34798) 2024-04-21T08:41:59.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f3471cee640 1 -- 172.21.15.7:0/3405232695 learned_addr learned my addr 172.21.15.7:0/3405232695 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f34714ed640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f346c103180 0x7f346c19ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.418+0000 7f34724ef640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f346c105b30 0x7f346c1a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.419+0000 7f3471cee640 1 -- 172.21.15.7:0/3405232695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f346c105b30 msgr2=0x7f346c1a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.419+0000 7f3471cee640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f346c105b30 0x7f346c1a5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.419+0000 7f3471cee640 1 -- 172.21.15.7:0/3405232695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f346c103180 msgr2=0x7f346c19ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.419+0000 7f3471cee640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f346c103180 0x7f346c19ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.419+0000 7f3471cee640 1 -- 172.21.15.7:0/3405232695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f346c1a6370 con 0x7f346c100810 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.419+0000 7f3471cee640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c100810 0x7f346c19e750 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f346800d950 tx=0x7f346800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.419+0000 7f345effd640 1 -- 172.21.15.7:0/3405232695 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3468017030 con 0x7f346c100810 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.419+0000 7f345effd640 1 -- 172.21.15.7:0/3405232695 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3468004020 con 0x7f346c100810 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.419+0000 7f3473f52640 1 -- 172.21.15.7:0/3405232695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f346c1a6660 con 0x7f346c100810 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.419+0000 7f345effd640 1 -- 172.21.15.7:0/3405232695 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34680047a0 con 0x7f346c100810 2024-04-21T08:41:59.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.419+0000 7f3473f52640 1 -- 172.21.15.7:0/3405232695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f346c1a6b80 con 0x7f346c100810 2024-04-21T08:41:59.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.420+0000 7f3473f52640 1 -- 172.21.15.7:0/3405232695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f346c0fdf00 con 0x7f346c100810 2024-04-21T08:41:59.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.421+0000 7f345effd640 1 -- 172.21.15.7:0/3405232695 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f3468009a30 con 0x7f346c100810 2024-04-21T08:41:59.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.422+0000 7f345effd640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f34580a3bb0 0x7f34580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:59.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.422+0000 7f345effd640 1 -- 172.21.15.7:0/3405232695 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f34680cffe0 con 0x7f346c100810 2024-04-21T08:41:59.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.422+0000 7f34714ed640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f34580a3bb0 0x7f34580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:59.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.422+0000 7f34714ed640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f34580a3bb0 0x7f34580a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3460009230 tx=0x7f3460005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:59.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.426+0000 7f345effd640 1 -- 172.21.15.7:0/3405232695 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f346809ce60 con 0x7f346c100810 2024-04-21T08:41:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.652+0000 7f3473f52640 1 -- 172.21.15.7:0/3405232695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f346c02d5c0 con 0x7f346c100810 2024-04-21T08:41:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.652+0000 7f345effd640 1 -- 172.21.15.7:0/3405232695 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6334 (secure 0 0 0) 0x7f346809c760 con 0x7f346c100810 2024-04-21T08:41:59.652 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:41:59.652 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T08:41:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 8 2024-04-21T08:41:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 -- 172.21.15.7:0/3405232695 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f34580a3bb0 msgr2=0x7f34580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f34580a3bb0 0x7f34580a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3460009230 tx=0x7f3460005fd0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 -- 172.21.15.7:0/3405232695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c100810 msgr2=0x7f346c19e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c100810 0x7f346c19e750 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f346800d950 tx=0x7f346800de10 comp rx=0 tx=0).stop 2024-04-21T08:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 -- 172.21.15.7:0/3405232695 shutdown_connections 2024-04-21T08:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f34580a3bb0 0x7f34580a6070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f346c105b30 0x7f346c1a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f346c103180 0x7f346c19ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 --2- 172.21.15.7:0/3405232695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c100810 0x7f346c19e750 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 -- 172.21.15.7:0/3405232695 >> 172.21.15.7:0/3405232695 conn(0x7f346c0fc410 msgr2=0x7f346c1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 -- 172.21.15.7:0/3405232695 shutdown_connections 2024-04-21T08:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:41:59.654+0000 7f3473f52640 1 -- 172.21.15.7:0/3405232695 wait complete. 2024-04-21T08:41:59.986 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:41:59 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3405232695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:00.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:41:59 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3405232695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:00.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:41:59 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3405232695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:00.256 INFO:journalctl@ceph.mds.h.smithi016.stdout:Apr 21 08:42:00 smithi016 podman[60851]: 2024-04-21 08:42:00.234022361 +0000 UTC m=+0.362392705 container start 937012de44324d5c1412350062a1858d153dffcfd167f9a273e46ee374a44df1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-21T08:42:00.257 INFO:journalctl@ceph.mds.h.smithi016.stdout:Apr 21 08:42:00 smithi016 ceph-mds[60869]: -- 172.21.15.16:0/2430433487 <== mon.2 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563e115dda40 con 0x563e115b0c00 2024-04-21T08:42:00.554 INFO:journalctl@ceph.mds.h.smithi016.stdout:Apr 21 08:42:00 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-h[60865]: starting mds.h at 2024-04-21T08:42:00.554 INFO:journalctl@ceph.mds.h.smithi016.stdout:Apr 21 08:42:00 smithi016 bash[60851]: 937012de44324d5c1412350062a1858d153dffcfd167f9a273e46ee374a44df1 2024-04-21T08:42:00.554 INFO:journalctl@ceph.mds.h.smithi016.stdout:Apr 21 08:42:00 smithi016 systemd[1]: Started Ceph mds.h for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:01.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:01 smithi077 ceph-mon[31615]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:01.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:01.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:01.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:01.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:01.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:01.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:01.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:01 smithi077 ceph-mon[31615]: Deploying daemon mds.i on smithi077 2024-04-21T08:42:01.605 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:01 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.16:6836/3699203122,v1:172.21.15.16:6837/3699203122] up:boot 2024-04-21T08:42:01.605 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:01 smithi077 ceph-mon[31615]: fsmap 8 up:standby 2024-04-21T08:42:01.605 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:01 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:42:01.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:01 smithi007 ceph-mon[28477]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:01.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:01.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:01.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:01.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:01.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:01.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:01.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:01 smithi007 ceph-mon[28477]: Deploying daemon mds.i on smithi077 2024-04-21T08:42:01.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:01 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.16:6836/3699203122,v1:172.21.15.16:6837/3699203122] up:boot 2024-04-21T08:42:01.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:01 smithi007 ceph-mon[28477]: fsmap 8 up:standby 2024-04-21T08:42:01.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:01 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:42:01.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:01.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.782+0000 7f701eb8f640 1 -- 172.21.15.7:0/1522398627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7018101fd0 msgr2=0x7f7018102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:01.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.782+0000 7f701eb8f640 1 --2- 172.21.15.7:0/1522398627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7018101fd0 0x7f7018102450 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7004009900 tx=0x7f700402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:01.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.782+0000 7f701eb8f640 1 -- 172.21.15.7:0/1522398627 shutdown_connections 2024-04-21T08:42:01.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.782+0000 7f701eb8f640 1 --2- 172.21.15.7:0/1522398627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7018102990 0x7f70181092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:01.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.782+0000 7f701eb8f640 1 --2- 172.21.15.7:0/1522398627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7018101fd0 0x7f7018102450 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:01.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.782+0000 7f701eb8f640 1 --2- 172.21.15.7:0/1522398627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7018100e80 0x7f70181012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.782+0000 7f701eb8f640 1 -- 172.21.15.7:0/1522398627 >> 172.21.15.7:0/1522398627 conn(0x7f70180fc3f0 msgr2=0x7f70180fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.782+0000 7f701eb8f640 1 -- 172.21.15.7:0/1522398627 shutdown_connections 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.782+0000 7f701eb8f640 1 -- 172.21.15.7:0/1522398627 wait complete. 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.782+0000 7f701eb8f640 1 Processor -- start 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.782+0000 7f701eb8f640 1 -- start start 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f701eb8f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7018100e80 0x7f7018069dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f701eb8f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7018101fd0 0x7f701819f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f701eb8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7018102990 0x7f70181a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f701eb8f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f701810ca10 con 0x7f7018101fd0 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f701eb8f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f701810c890 con 0x7f7018100e80 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f701eb8f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f701810cb90 con 0x7f7018102990 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7017fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7018101fd0 0x7f701819f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7017fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7018101fd0 0x7f701819f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34814/0 (socket says 172.21.15.7:34814) 2024-04-21T08:42:01.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7017fff640 1 -- 172.21.15.7:0/37578384 learned_addr learned my addr 172.21.15.7:0/37578384 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f701d12c640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7018102990 0x7f70181a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f701c92b640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7018100e80 0x7f7018069dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7017fff640 1 -- 172.21.15.7:0/37578384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7018102990 msgr2=0x7f70181a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7017fff640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7018102990 0x7f70181a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7017fff640 1 -- 172.21.15.7:0/37578384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7018100e80 msgr2=0x7f7018069dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7017fff640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7018100e80 0x7f7018069dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7017fff640 1 -- 172.21.15.7:0/37578384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70181a2010 con 0x7f7018101fd0 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f701c92b640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7018100e80 0x7f7018069dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7017fff640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7018101fd0 0x7f701819f2d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7004000c00 tx=0x7f70040040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7015ffb640 1 -- 172.21.15.7:0/37578384 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7004043020 con 0x7f7018101fd0 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7015ffb640 1 -- 172.21.15.7:0/37578384 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70040044c0 con 0x7f7018101fd0 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f701eb8f640 1 -- 172.21.15.7:0/37578384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70181a2240 con 0x7f7018101fd0 2024-04-21T08:42:01.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.783+0000 7f7015ffb640 1 -- 172.21.15.7:0/37578384 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7004003c60 con 0x7f7018101fd0 2024-04-21T08:42:01.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.784+0000 7f701eb8f640 1 -- 172.21.15.7:0/37578384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70181a27e0 con 0x7f7018101fd0 2024-04-21T08:42:01.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.785+0000 7f701eb8f640 1 -- 172.21.15.7:0/37578384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7018108190 con 0x7f7018101fd0 2024-04-21T08:42:01.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.786+0000 7f7015ffb640 1 -- 172.21.15.7:0/37578384 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f7004046020 con 0x7f7018101fd0 2024-04-21T08:42:01.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.786+0000 7f7015ffb640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f70080a3c60 0x7f70080a6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:01.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.786+0000 7f701c92b640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f70080a3c60 0x7f70080a6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:01.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.786+0000 7f7015ffb640 1 -- 172.21.15.7:0/37578384 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f70040f2130 con 0x7f7018101fd0 2024-04-21T08:42:01.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.787+0000 7f701c92b640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f70080a3c60 0x7f70080a6120 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f700c005d00 tx=0x7f700c005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:01.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:01.791+0000 7f7015ffb640 1 -- 172.21.15.7:0/37578384 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f700404d020 con 0x7f7018101fd0 2024-04-21T08:42:01.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:01 smithi016 ceph-mon[31259]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:01.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:01.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:01.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:01.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:01.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:01.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:01.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:01 smithi016 ceph-mon[31259]: Deploying daemon mds.i on smithi077 2024-04-21T08:42:01.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:01 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.16:6836/3699203122,v1:172.21.15.16:6837/3699203122] up:boot 2024-04-21T08:42:01.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:01 smithi016 ceph-mon[31259]: fsmap 8 up:standby 2024-04-21T08:42:01.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:01 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:42:02.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.017+0000 7f701eb8f640 1 -- 172.21.15.7:0/37578384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f701802d5c0 con 0x7f7018101fd0 2024-04-21T08:42:02.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.018+0000 7f7015ffb640 1 -- 172.21.15.7:0/37578384 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7159 (secure 0 0 0) 0x7f70040befb0 con 0x7f7018101fd0 2024-04-21T08:42:02.017 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:02.017 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T08:42:02.018 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 9 2024-04-21T08:42:02.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 -- 172.21.15.7:0/37578384 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f70080a3c60 msgr2=0x7f70080a6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:02.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f70080a3c60 0x7f70080a6120 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f700c005d00 tx=0x7f700c005ed0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 -- 172.21.15.7:0/37578384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7018101fd0 msgr2=0x7f701819f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7018101fd0 0x7f701819f2d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7004000c00 tx=0x7f70040040c0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 -- 172.21.15.7:0/37578384 shutdown_connections 2024-04-21T08:42:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f70080a3c60 0x7f70080a6120 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7018102990 0x7f70181a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7018101fd0 0x7f701819f2d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 --2- 172.21.15.7:0/37578384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7018100e80 0x7f7018069dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 -- 172.21.15.7:0/37578384 >> 172.21.15.7:0/37578384 conn(0x7f70180fc3f0 msgr2=0x7f7018105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 -- 172.21.15.7:0/37578384 shutdown_connections 2024-04-21T08:42:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:02.020+0000 7f701eb8f640 1 -- 172.21.15.7:0/37578384 wait complete. 2024-04-21T08:42:02.585 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:02 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/37578384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:02.585 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:42:02 smithi077 systemd[1]: Starting Ceph mds.i for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:02.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:02 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/37578384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:02.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:02 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/37578384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:02.854 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:42:02 smithi077 podman[62657]: 2024-04-21 08:42:02.645019046 +0000 UTC m=+0.018332324 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:02.854 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:42:02 smithi077 podman[62657]: 2024-04-21 08:42:02.754354174 +0000 UTC m=+0.127667450 container create 92989efcacbcc14b2ff5d02b9b55dce83b21cd87f391f46eddac1f592c09f7eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-21T08:42:03.353 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:42:02 smithi077 podman[62657]: 2024-04-21 08:42:02.999020113 +0000 UTC m=+0.372333391 container init 92989efcacbcc14b2ff5d02b9b55dce83b21cd87f391f46eddac1f592c09f7eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-21T08:42:03.354 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:42:03 smithi077 podman[62657]: 2024-04-21 08:42:03.002196044 +0000 UTC m=+0.375509321 container start 92989efcacbcc14b2ff5d02b9b55dce83b21cd87f391f46eddac1f592c09f7eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-21T08:42:03.354 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:42:03 smithi077 ceph-mds[62676]: -- 172.21.15.77:0/808700090 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5588421afa40 con 0x558842182c00 2024-04-21T08:42:03.354 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:42:03 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i[62672]: starting mds.i at 2024-04-21T08:42:03.354 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:42:03 smithi077 bash[62657]: 92989efcacbcc14b2ff5d02b9b55dce83b21cd87f391f46eddac1f592c09f7eb 2024-04-21T08:42:03.354 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:42:03 smithi077 systemd[1]: Started Ceph mds.i for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:03.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:03 smithi007 ceph-mon[28477]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:03.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:03 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:03.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:03 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:03.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:03 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:03.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:03 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:03.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:03 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:03.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:03 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:03 smithi016 ceph-mon[31259]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:03 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:03 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:03 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:03 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:03.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:03 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:03.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:03 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:03.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:03 smithi077 ceph-mon[31615]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:03 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:03 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:03 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:03 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:03 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:03 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:04.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.141+0000 7f2b20ceb640 1 -- 172.21.15.7:0/2234287196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c103180 msgr2=0x7f2b1c103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.141+0000 7f2b20ceb640 1 --2- 172.21.15.7:0/2234287196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c103180 0x7f2b1c103600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009900 tx=0x7f2b0402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.141+0000 7f2b20ceb640 1 -- 172.21.15.7:0/2234287196 shutdown_connections 2024-04-21T08:42:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.141+0000 7f2b20ceb640 1 --2- 172.21.15.7:0/2234287196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b1c103b40 0x7f2b1c10a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.141+0000 7f2b20ceb640 1 --2- 172.21.15.7:0/2234287196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c103180 0x7f2b1c103600 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.141+0000 7f2b20ceb640 1 --2- 172.21.15.7:0/2234287196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c101f90 0x7f2b1c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.141+0000 7f2b20ceb640 1 -- 172.21.15.7:0/2234287196 >> 172.21.15.7:0/2234287196 conn(0x7f2b1c0fd520 msgr2=0x7f2b1c0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.141+0000 7f2b20ceb640 1 -- 172.21.15.7:0/2234287196 shutdown_connections 2024-04-21T08:42:04.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.141+0000 7f2b20ceb640 1 -- 172.21.15.7:0/2234287196 wait complete. 2024-04-21T08:42:04.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b20ceb640 1 Processor -- start 2024-04-21T08:42:04.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b20ceb640 1 -- start start 2024-04-21T08:42:04.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b20ceb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c101f90 0x7f2b1c198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:04.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b20ceb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b1c103180 0x7f2b1c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:04.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b20ceb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c103b40 0x7f2b1c19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b20ceb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c10dad0 con 0x7f2b1c103b40 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b20ceb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c10d950 con 0x7f2b1c101f90 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b20ceb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c10dc50 con 0x7f2b1c103180 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b1ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c103b40 0x7f2b1c19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b1ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c103b40 0x7f2b1c19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34832/0 (socket says 172.21.15.7:34832) 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b1ad9d640 1 -- 172.21.15.7:0/3306181095 learned_addr learned my addr 172.21.15.7:0/3306181095 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b1a59c640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c101f90 0x7f2b1c198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.142+0000 7f2b11d9b640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b1c103180 0x7f2b1c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.143+0000 7f2b1ad9d640 1 -- 172.21.15.7:0/3306181095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b1c103180 msgr2=0x7f2b1c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.143+0000 7f2b1ad9d640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b1c103180 0x7f2b1c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.143+0000 7f2b1ad9d640 1 -- 172.21.15.7:0/3306181095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c101f90 msgr2=0x7f2b1c198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.143+0000 7f2b1ad9d640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c101f90 0x7f2b1c198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:04.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.143+0000 7f2b1ad9d640 1 -- 172.21.15.7:0/3306181095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c19fdb0 con 0x7f2b1c103b40 2024-04-21T08:42:04.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.143+0000 7f2b1ad9d640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c103b40 0x7f2b1c19f740 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400cad0 tx=0x7f2b1400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:04.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.143+0000 7f2b13fff640 1 -- 172.21.15.7:0/3306181095 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1400ae20 con 0x7f2b1c103b40 2024-04-21T08:42:04.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.143+0000 7f2b13fff640 1 -- 172.21.15.7:0/3306181095 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b14005020 con 0x7f2b1c103b40 2024-04-21T08:42:04.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.143+0000 7f2b20ceb640 1 -- 172.21.15.7:0/3306181095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b1c1a00a0 con 0x7f2b1c103b40 2024-04-21T08:42:04.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.143+0000 7f2b13fff640 1 -- 172.21.15.7:0/3306181095 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1400f6b0 con 0x7f2b1c103b40 2024-04-21T08:42:04.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.143+0000 7f2b20ceb640 1 -- 172.21.15.7:0/3306181095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b1c1a05c0 con 0x7f2b1c103b40 2024-04-21T08:42:04.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.144+0000 7f2b20ceb640 1 -- 172.21.15.7:0/3306181095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b1c1092d0 con 0x7f2b1c103b40 2024-04-21T08:42:04.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.145+0000 7f2b13fff640 1 -- 172.21.15.7:0/3306181095 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f2b140051d0 con 0x7f2b1c103b40 2024-04-21T08:42:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.146+0000 7f2b13fff640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2b080a3bb0 0x7f2b080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.146+0000 7f2b13fff640 1 -- 172.21.15.7:0/3306181095 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2b140cff70 con 0x7f2b1c103b40 2024-04-21T08:42:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.146+0000 7f2b1a59c640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2b080a3bb0 0x7f2b080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:04.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.146+0000 7f2b1a59c640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2b080a3bb0 0x7f2b080a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c005fd0 tx=0x7f2b0c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:04.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.150+0000 7f2b13fff640 1 -- 172.21.15.7:0/3306181095 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b14014030 con 0x7f2b1c103b40 2024-04-21T08:42:04.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.380+0000 7f2b20ceb640 1 -- 172.21.15.7:0/3306181095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b1c062b50 con 0x7f2b1c103b40 2024-04-21T08:42:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.380+0000 7f2b13fff640 1 -- 172.21.15.7:0/3306181095 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7983 (secure 0 0 0) 0x7f2b1409cdf0 con 0x7f2b1c103b40 2024-04-21T08:42:04.380 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:04.380 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:42:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 10 2024-04-21T08:42:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 -- 172.21.15.7:0/3306181095 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2b080a3bb0 msgr2=0x7f2b080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2b080a3bb0 0x7f2b080a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c005fd0 tx=0x7f2b0c005850 comp rx=0 tx=0).stop 2024-04-21T08:42:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 -- 172.21.15.7:0/3306181095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c103b40 msgr2=0x7f2b1c19f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c103b40 0x7f2b1c19f740 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400cad0 tx=0x7f2b1400cf90 comp rx=0 tx=0).stop 2024-04-21T08:42:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 -- 172.21.15.7:0/3306181095 shutdown_connections 2024-04-21T08:42:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2b080a3bb0 0x7f2b080a6070 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b1c103b40 0x7f2b1c19f740 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b1c103180 0x7f2b1c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 --2- 172.21.15.7:0/3306181095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1c101f90 0x7f2b1c198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 -- 172.21.15.7:0/3306181095 >> 172.21.15.7:0/3306181095 conn(0x7f2b1c0fd520 msgr2=0x7f2b1c1064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 -- 172.21.15.7:0/3306181095 shutdown_connections 2024-04-21T08:42:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:04.383+0000 7f2b20ceb640 1 -- 172.21.15.7:0/3306181095 wait complete. 2024-04-21T08:42:04.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:04 smithi007 ceph-mon[28477]: Deploying daemon mds.g on smithi007 2024-04-21T08:42:04.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:04 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.77:6838/516970569,v1:172.21.15.77:6839/516970569] up:boot 2024-04-21T08:42:04.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:04 smithi007 ceph-mon[28477]: fsmap 9 up:standby 2024-04-21T08:42:04.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:04 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:42:04.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:04 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3306181095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:04 smithi016 ceph-mon[31259]: Deploying daemon mds.g on smithi007 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:04 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.77:6838/516970569,v1:172.21.15.77:6839/516970569] up:boot 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:04 smithi016 ceph-mon[31259]: fsmap 9 up:standby 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:04 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:04 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3306181095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:04.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:04 smithi077 ceph-mon[31615]: Deploying daemon mds.g on smithi007 2024-04-21T08:42:04.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:04 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.77:6838/516970569,v1:172.21.15.77:6839/516970569] up:boot 2024-04-21T08:42:04.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:04 smithi077 ceph-mon[31615]: fsmap 9 up:standby 2024-04-21T08:42:04.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:04 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:42:04.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:04 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3306181095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:05.655 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:05 smithi007 ceph-mon[28477]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:05.655 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:42:05 smithi007 systemd[1]: Starting Ceph mds.g for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:05.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:05 smithi016 ceph-mon[31259]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:05.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:05 smithi077 ceph-mon[31615]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:05.918 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:42:05 smithi007 podman[68273]: 2024-04-21 08:42:05.572307027 +0000 UTC m=+0.018781714 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:05.919 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:42:05 smithi007 podman[68273]: 2024-04-21 08:42:05.683140315 +0000 UTC m=+0.129614997 container create 0cca774d80e46bb950acc667f8d3390e682343ca291366cb88f8374857098785 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-g, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:42:06.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:06.401 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:42:05 smithi007 podman[68273]: 2024-04-21 08:42:05.91951947 +0000 UTC m=+0.365994151 container init 0cca774d80e46bb950acc667f8d3390e682343ca291366cb88f8374857098785 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-04-21T08:42:06.401 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:42:05 smithi007 podman[68273]: 2024-04-21 08:42:05.922492584 +0000 UTC m=+0.368967278 container start 0cca774d80e46bb950acc667f8d3390e682343ca291366cb88f8374857098785 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-21T08:42:06.402 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:42:05 smithi007 ceph-mds[68293]: -- 172.21.15.7:0/2737829246 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5620becf7a40 con 0x5620beccac00 2024-04-21T08:42:06.402 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:42:05 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-g[68289]: starting mds.g at 2024-04-21T08:42:06.402 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:42:06 smithi007 bash[68273]: 0cca774d80e46bb950acc667f8d3390e682343ca291366cb88f8374857098785 2024-04-21T08:42:06.402 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:42:06 smithi007 systemd[1]: Started Ceph mds.g for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.503+0000 7f0db4b96640 1 -- 172.21.15.7:0/3802971560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db0101b70 msgr2=0x7f0db0101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.503+0000 7f0db4b96640 1 --2- 172.21.15.7:0/3802971560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db0101b70 0x7f0db0101ff0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009ca0 tx=0x7f0da802c180 comp rx=0 tx=0).stop 2024-04-21T08:42:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.503+0000 7f0db4b96640 1 -- 172.21.15.7:0/3802971560 shutdown_connections 2024-04-21T08:42:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.503+0000 7f0db4b96640 1 --2- 172.21.15.7:0/3802971560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db0101b70 0x7f0db0101ff0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.503+0000 7f0db4b96640 1 --2- 172.21.15.7:0/3802971560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0db01011d0 0x7f0db0101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.503+0000 7f0db4b96640 1 --2- 172.21.15.7:0/3802971560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db0100870 0x7f0db0100c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.503+0000 7f0db4b96640 1 -- 172.21.15.7:0/3802971560 >> 172.21.15.7:0/3802971560 conn(0x7f0db00fc3f0 msgr2=0x7f0db00fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.503+0000 7f0db4b96640 1 -- 172.21.15.7:0/3802971560 shutdown_connections 2024-04-21T08:42:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.503+0000 7f0db4b96640 1 -- 172.21.15.7:0/3802971560 wait complete. 2024-04-21T08:42:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.504+0000 7f0db4b96640 1 Processor -- start 2024-04-21T08:42:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.504+0000 7f0db4b96640 1 -- start start 2024-04-21T08:42:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.504+0000 7f0db4b96640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db0100870 0x7f0db019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.504+0000 7f0db4b96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0db01011d0 0x7f0db019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.504+0000 7f0db4b96640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db0101b70 0x7f0db01a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.504+0000 7f0db4b96640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0db010fcc0 con 0x7f0db0100870 2024-04-21T08:42:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.504+0000 7f0db4b96640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0db010fb40 con 0x7f0db0101b70 2024-04-21T08:42:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.504+0000 7f0db4b96640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0db010fe40 con 0x7f0db01011d0 2024-04-21T08:42:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.504+0000 7f0dae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db0100870 0x7f0db019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.504+0000 7f0dae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db0100870 0x7f0db019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35102/0 (socket says 172.21.15.7:35102) 2024-04-21T08:42:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.504+0000 7f0dae59c640 1 -- 172.21.15.7:0/3871189289 learned_addr learned my addr 172.21.15.7:0/3871189289 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0dadd9b640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0db01011d0 0x7f0db019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0daed9d640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db0101b70 0x7f0db01a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0dae59c640 1 -- 172.21.15.7:0/3871189289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0db01011d0 msgr2=0x7f0db019a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0dae59c640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0db01011d0 0x7f0db019a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0dae59c640 1 -- 172.21.15.7:0/3871189289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db0101b70 msgr2=0x7f0db01a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0dae59c640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db0101b70 0x7f0db01a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0dae59c640 1 -- 172.21.15.7:0/3871189289 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db01a1eb0 con 0x7f0db0100870 2024-04-21T08:42:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0dadd9b640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0db01011d0 0x7f0db019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0dae59c640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db0100870 0x7f0db019a290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0da400d950 tx=0x7f0da400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0d9f7fe640 1 -- 172.21.15.7:0/3871189289 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da4017070 con 0x7f0db0100870 2024-04-21T08:42:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0db4b96640 1 -- 172.21.15.7:0/3871189289 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0db01a21a0 con 0x7f0db0100870 2024-04-21T08:42:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0d9f7fe640 1 -- 172.21.15.7:0/3871189289 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0da4009c30 con 0x7f0db0100870 2024-04-21T08:42:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0d9f7fe640 1 -- 172.21.15.7:0/3871189289 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da400f620 con 0x7f0db0100870 2024-04-21T08:42:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.505+0000 7f0db4b96640 1 -- 172.21.15.7:0/3871189289 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0db01a2740 con 0x7f0db0100870 2024-04-21T08:42:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.506+0000 7f0db4b96640 1 -- 172.21.15.7:0/3871189289 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0db010b5c0 con 0x7f0db0100870 2024-04-21T08:42:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.507+0000 7f0d9f7fe640 1 -- 172.21.15.7:0/3871189289 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f0da4010460 con 0x7f0db0100870 2024-04-21T08:42:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.508+0000 7f0d9f7fe640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0d980a3bb0 0x7f0d980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.508+0000 7f0d9f7fe640 1 -- 172.21.15.7:0/3871189289 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0da40cfce0 con 0x7f0db0100870 2024-04-21T08:42:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.508+0000 7f0dadd9b640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0d980a3bb0 0x7f0d980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:06.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.509+0000 7f0dadd9b640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0d980a3bb0 0x7f0d980a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0db019b7a0 tx=0x7f0da0005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:06.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.513+0000 7f0d9f7fe640 1 -- 172.21.15.7:0/3871189289 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0da409cac0 con 0x7f0db0100870 2024-04-21T08:42:06.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.759+0000 7f0db4b96640 1 -- 172.21.15.7:0/3871189289 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0db002d5c0 con 0x7f0db0100870 2024-04-21T08:42:06.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.760+0000 7f0d9f7fe640 1 -- 172.21.15.7:0/3871189289 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8806 (secure 0 0 0) 0x7f0da4009de0 con 0x7f0db0100870 2024-04-21T08:42:06.760 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:06.760 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:42:06.762 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 11 2024-04-21T08:42:06.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 -- 172.21.15.7:0/3871189289 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0d980a3bb0 msgr2=0x7f0d980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:06.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0d980a3bb0 0x7f0d980a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0db019b7a0 tx=0x7f0da0005d90 comp rx=0 tx=0).stop 2024-04-21T08:42:06.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 -- 172.21.15.7:0/3871189289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db0100870 msgr2=0x7f0db019a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db0100870 0x7f0db019a290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0da400d950 tx=0x7f0da400de10 comp rx=0 tx=0).stop 2024-04-21T08:42:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 -- 172.21.15.7:0/3871189289 shutdown_connections 2024-04-21T08:42:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0d980a3bb0 0x7f0d980a6070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db0101b70 0x7f0db01a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0db01011d0 0x7f0db019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 --2- 172.21.15.7:0/3871189289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db0100870 0x7f0db019a290 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 -- 172.21.15.7:0/3871189289 >> 172.21.15.7:0/3871189289 conn(0x7f0db00fc3f0 msgr2=0x7f0db00fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 -- 172.21.15.7:0/3871189289 shutdown_connections 2024-04-21T08:42:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:06.762+0000 7f0db4b96640 1 -- 172.21.15.7:0/3871189289 wait complete. 2024-04-21T08:42:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:07.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:07.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:07.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:07.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:07.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: Deploying daemon mds.a on smithi007 2024-04-21T08:42:07.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.7:6838/2635298189,v1:172.21.15.7:6839/2635298189] up:boot 2024-04-21T08:42:07.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: fsmap 10 up:standby 2024-04-21T08:42:07.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:42:07.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:07 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3871189289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: Deploying daemon mds.a on smithi007 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.7:6838/2635298189,v1:172.21.15.7:6839/2635298189] up:boot 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: fsmap 10 up:standby 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:42:07.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:07 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3871189289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:07.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:07.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:07.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:07.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:07.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: Deploying daemon mds.a on smithi007 2024-04-21T08:42:07.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.7:6838/2635298189,v1:172.21.15.7:6839/2635298189] up:boot 2024-04-21T08:42:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: fsmap 10 up:standby 2024-04-21T08:42:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:42:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:07 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3871189289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:08.151 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:42:08 smithi007 systemd[1]: Starting Ceph mds.a for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:08.490 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:42:08 smithi007 podman[68623]: 2024-04-21 08:42:08.253379777 +0000 UTC m=+0.020057458 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:08.490 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:42:08 smithi007 podman[68623]: 2024-04-21 08:42:08.363750151 +0000 UTC m=+0.130427811 container create 7efc3f16e9ed2eb155f19edd0c090e921ef04e358f32eb6115b54be336a8524c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-04-21T08:42:08.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:08.798 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:42:08 smithi007 podman[68623]: 2024-04-21 08:42:08.616572465 +0000 UTC m=+0.383250117 container init 7efc3f16e9ed2eb155f19edd0c090e921ef04e358f32eb6115b54be336a8524c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-21T08:42:08.799 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:42:08 smithi007 podman[68623]: 2024-04-21 08:42:08.619694032 +0000 UTC m=+0.386371692 container start 7efc3f16e9ed2eb155f19edd0c090e921ef04e358f32eb6115b54be336a8524c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:42:08.799 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:42:08 smithi007 ceph-mds[68652]: -- 172.21.15.7:0/1186916906 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556ef53c7a40 con 0x556ef539a800 2024-04-21T08:42:08.799 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:42:08 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-a[68646]: starting mds.a at 2024-04-21T08:42:08.799 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:42:08 smithi007 bash[68623]: 7efc3f16e9ed2eb155f19edd0c090e921ef04e358f32eb6115b54be336a8524c 2024-04-21T08:42:08.799 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:42:08 smithi007 systemd[1]: Started Ceph mds.a for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:08.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.881+0000 7f6eabd59640 1 -- 172.21.15.7:0/2959574021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ea4102990 msgr2=0x7f6ea41092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:08.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.881+0000 7f6eabd59640 1 --2- 172.21.15.7:0/2959574021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ea4102990 0x7f6ea41092b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009ca0 tx=0x7f6e9c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:08.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 -- 172.21.15.7:0/2959574021 shutdown_connections 2024-04-21T08:42:08.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 --2- 172.21.15.7:0/2959574021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ea4102990 0x7f6ea41092b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:08.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 --2- 172.21.15.7:0/2959574021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea4101fd0 0x7f6ea4102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:08.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 --2- 172.21.15.7:0/2959574021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ea4100e80 0x7f6ea41012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:08.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 -- 172.21.15.7:0/2959574021 >> 172.21.15.7:0/2959574021 conn(0x7f6ea40fc3f0 msgr2=0x7f6ea40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:08.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 -- 172.21.15.7:0/2959574021 shutdown_connections 2024-04-21T08:42:08.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 -- 172.21.15.7:0/2959574021 wait complete. 2024-04-21T08:42:08.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 Processor -- start 2024-04-21T08:42:08.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 -- start start 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ea4100e80 0x7f6ea4069dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ea4101fd0 0x7f6ea419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea4102990 0x7f6ea419f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea410c9c0 con 0x7f6ea4100e80 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea410c840 con 0x7f6ea4101fd0 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.882+0000 7f6eabd59640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea410cb40 con 0x7f6ea4102990 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6ea9af5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ea4100e80 0x7f6ea4069dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6ea9af5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ea4100e80 0x7f6ea4069dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35146/0 (socket says 172.21.15.7:35146) 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6ea9af5640 1 -- 172.21.15.7:0/2976619549 learned_addr learned my addr 172.21.15.7:0/2976619549 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6ea92f4640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ea4101fd0 0x7f6ea419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6eaa2f6640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea4102990 0x7f6ea419f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6ea9af5640 1 -- 172.21.15.7:0/2976619549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea4102990 msgr2=0x7f6ea419f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:08.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6ea9af5640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea4102990 0x7f6ea419f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6ea9af5640 1 -- 172.21.15.7:0/2976619549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ea4101fd0 msgr2=0x7f6ea419cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6ea9af5640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ea4101fd0 0x7f6ea419cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6ea9af5640 1 -- 172.21.15.7:0/2976619549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea419fcd0 con 0x7f6ea4100e80 2024-04-21T08:42:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6ea9af5640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ea4100e80 0x7f6ea4069dc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000e9f0 tx=0x7f6ea000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6e9affd640 1 -- 172.21.15.7:0/2976619549 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea000cd80 con 0x7f6ea4100e80 2024-04-21T08:42:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6eabd59640 1 -- 172.21.15.7:0/2976619549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea419ff60 con 0x7f6ea4100e80 2024-04-21T08:42:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6e9affd640 1 -- 172.21.15.7:0/2976619549 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ea00108e0 con 0x7f6ea4100e80 2024-04-21T08:42:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6e9affd640 1 -- 172.21.15.7:0/2976619549 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0018980 con 0x7f6ea4100e80 2024-04-21T08:42:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.883+0000 7f6eabd59640 1 -- 172.21.15.7:0/2976619549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea41a03c0 con 0x7f6ea4100e80 2024-04-21T08:42:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.884+0000 7f6eabd59640 1 -- 172.21.15.7:0/2976619549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ea4068e30 con 0x7f6ea4100e80 2024-04-21T08:42:08.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.886+0000 7f6e9affd640 1 -- 172.21.15.7:0/2976619549 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6ea0010430 con 0x7f6ea4100e80 2024-04-21T08:42:08.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.886+0000 7f6e9affd640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6e900a3bb0 0x7f6e900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:08.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.887+0000 7f6ea92f4640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6e900a3bb0 0x7f6e900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:08.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.887+0000 7f6e9affd640 1 -- 172.21.15.7:0/2976619549 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6ea00d0790 con 0x7f6ea4100e80 2024-04-21T08:42:08.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.887+0000 7f6ea92f4640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6e900a3bb0 0x7f6e900a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4069b30 tx=0x7f6e94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:08.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:08.891+0000 7f6e9affd640 1 -- 172.21.15.7:0/2976619549 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ea4068e30 con 0x7f6ea4100e80 2024-04-21T08:42:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.121+0000 7f6eabd59640 1 -- 172.21.15.7:0/2976619549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ea402d5c0 con 0x7f6ea4100e80 2024-04-21T08:42:09.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.122+0000 7f6e9affd640 1 -- 172.21.15.7:0/2976619549 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9629 (secure 0 0 0) 0x7f6ea009d610 con 0x7f6ea4100e80 2024-04-21T08:42:09.124 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:09.124 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:42:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 12 2024-04-21T08:42:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 -- 172.21.15.7:0/2976619549 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6e900a3bb0 msgr2=0x7f6e900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6e900a3bb0 0x7f6e900a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4069b30 tx=0x7f6e94005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 -- 172.21.15.7:0/2976619549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ea4100e80 msgr2=0x7f6ea4069dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ea4100e80 0x7f6ea4069dc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000e9f0 tx=0x7f6ea000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 -- 172.21.15.7:0/2976619549 shutdown_connections 2024-04-21T08:42:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6e900a3bb0 0x7f6e900a6070 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea4102990 0x7f6ea419f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ea4101fd0 0x7f6ea419cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 --2- 172.21.15.7:0/2976619549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ea4100e80 0x7f6ea4069dc0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 -- 172.21.15.7:0/2976619549 >> 172.21.15.7:0/2976619549 conn(0x7f6ea40fc3f0 msgr2=0x7f6ea4105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:09.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 -- 172.21.15.7:0/2976619549 shutdown_connections 2024-04-21T08:42:09.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:09.124+0000 7f6eabd59640 1 -- 172.21.15.7:0/2976619549 wait complete. 2024-04-21T08:42:09.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:09 smithi077 ceph-mon[31615]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:09.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:09.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:09.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:09.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:09.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:09.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:09 smithi007 ceph-mon[28477]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:09.431 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:09 smithi016 ceph-mon[31259]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:09.432 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:09.432 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:09.432 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:09.432 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:09.432 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:09.432 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:10.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:10 smithi077 ceph-mon[31615]: Deploying daemon mds.b on smithi016 2024-04-21T08:42:10.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:10 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.7:6840/3576958984,v1:172.21.15.7:6841/3576958984] up:boot 2024-04-21T08:42:10.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:10 smithi077 ceph-mon[31615]: fsmap 11 up:standby 2024-04-21T08:42:10.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:42:10.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:10 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2976619549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:10.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:10 smithi007 ceph-mon[28477]: Deploying daemon mds.b on smithi016 2024-04-21T08:42:10.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:10 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.7:6840/3576958984,v1:172.21.15.7:6841/3576958984] up:boot 2024-04-21T08:42:10.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:10 smithi007 ceph-mon[28477]: fsmap 11 up:standby 2024-04-21T08:42:10.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:42:10.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:10 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2976619549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:10.451 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:10 smithi016 ceph-mon[31259]: Deploying daemon mds.b on smithi016 2024-04-21T08:42:10.451 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:10 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.7:6840/3576958984,v1:172.21.15.7:6841/3576958984] up:boot 2024-04-21T08:42:10.451 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:10 smithi016 ceph-mon[31259]: fsmap 11 up:standby 2024-04-21T08:42:10.451 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:42:10.451 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:10 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2976619549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:11.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:11.233 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:11 smithi016 ceph-mon[31259]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.251+0000 7fbcbda06640 1 -- 172.21.15.7:0/816614031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcb81053b0 msgr2=0x7fbcb81077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.251+0000 7fbcbda06640 1 --2- 172.21.15.7:0/816614031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcb81053b0 0x7fbcb81077a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbca8009900 tx=0x7fbca802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.251+0000 7fbcbda06640 1 -- 172.21.15.7:0/816614031 shutdown_connections 2024-04-21T08:42:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.251+0000 7fbcbda06640 1 --2- 172.21.15.7:0/816614031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb8107ce0 0x7fbcb810a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.251+0000 7fbcbda06640 1 --2- 172.21.15.7:0/816614031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcb81053b0 0x7fbcb81077a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.251+0000 7fbcbda06640 1 --2- 172.21.15.7:0/816614031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb8102a80 0x7fbcb8104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.251+0000 7fbcbda06640 1 -- 172.21.15.7:0/816614031 >> 172.21.15.7:0/816614031 conn(0x7fbcb80fc410 msgr2=0x7fbcb80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.251+0000 7fbcbda06640 1 -- 172.21.15.7:0/816614031 shutdown_connections 2024-04-21T08:42:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.251+0000 7fbcbda06640 1 -- 172.21.15.7:0/816614031 wait complete. 2024-04-21T08:42:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcbda06640 1 Processor -- start 2024-04-21T08:42:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcbda06640 1 -- start start 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcbda06640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb8102a80 0x7fbcb8102350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcbda06640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcb81053b0 0x7fbcb8100a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcbda06640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb8107ce0 0x7fbcb8100f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcbda06640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb810db30 con 0x7fbcb81053b0 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcbda06640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb810d9b0 con 0x7fbcb8107ce0 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcbda06640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb810dcb0 con 0x7fbcb8102a80 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcb67fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcb81053b0 0x7fbcb8100a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcb67fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcb81053b0 0x7fbcb8100a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35160/0 (socket says 172.21.15.7:35160) 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcb67fc640 1 -- 172.21.15.7:0/3058586543 learned_addr learned my addr 172.21.15.7:0/3058586543 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcb6ffd640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb8102a80 0x7fbcb8102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcb77fe640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb8107ce0 0x7fbcb8100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcb67fc640 1 -- 172.21.15.7:0/3058586543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb8102a80 msgr2=0x7fbcb8102350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:11.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcb67fc640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb8102a80 0x7fbcb8102350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:11.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcb67fc640 1 -- 172.21.15.7:0/3058586543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb8107ce0 msgr2=0x7fbcb8100f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:11.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcb67fc640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb8107ce0 0x7fbcb8100f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:11.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcb67fc640 1 -- 172.21.15.7:0/3058586543 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb8101770 con 0x7fbcb81053b0 2024-04-21T08:42:11.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.252+0000 7fbcb6ffd640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb8102a80 0x7fbcb8102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:11.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.253+0000 7fbcb67fc640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcb81053b0 0x7fbcb8100a30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbca8000c00 tx=0x7fbca8005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:11.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.253+0000 7fbcbca04640 1 -- 172.21.15.7:0/3058586543 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8043020 con 0x7fbcb81053b0 2024-04-21T08:42:11.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.253+0000 7fbcbca04640 1 -- 172.21.15.7:0/3058586543 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbca80043d0 con 0x7fbcb81053b0 2024-04-21T08:42:11.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.253+0000 7fbcbda06640 1 -- 172.21.15.7:0/3058586543 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcb81a2280 con 0x7fbcb81053b0 2024-04-21T08:42:11.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.253+0000 7fbcbca04640 1 -- 172.21.15.7:0/3058586543 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8003b20 con 0x7fbcb81053b0 2024-04-21T08:42:11.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.253+0000 7fbcbda06640 1 -- 172.21.15.7:0/3058586543 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcb81a27f0 con 0x7fbcb81053b0 2024-04-21T08:42:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.254+0000 7fbcbda06640 1 -- 172.21.15.7:0/3058586543 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcb80fdf00 con 0x7fbcb81053b0 2024-04-21T08:42:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.255+0000 7fbcbca04640 1 -- 172.21.15.7:0/3058586543 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fbca8046020 con 0x7fbcb81053b0 2024-04-21T08:42:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.256+0000 7fbcbca04640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbca00a3c30 0x7fbca00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.256+0000 7fbcb6ffd640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbca00a3c30 0x7fbca00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:11.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.256+0000 7fbcbca04640 1 -- 172.21.15.7:0/3058586543 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbca80f2130 con 0x7fbcb81053b0 2024-04-21T08:42:11.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.256+0000 7fbcb6ffd640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbca00a3c30 0x7fbca00a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009c20 tx=0x7fbcac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:11.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.260+0000 7fbcbca04640 1 -- 172.21.15.7:0/3058586543 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbca804d020 con 0x7fbcb81053b0 2024-04-21T08:42:11.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:11 smithi077 ceph-mon[31615]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:11.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:11 smithi007 ceph-mon[28477]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:11.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.487+0000 7fbcbda06640 1 -- 172.21.15.7:0/3058586543 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcb802d810 con 0x7fbcb81053b0 2024-04-21T08:42:11.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.488+0000 7fbcbca04640 1 -- 172.21.15.7:0/3058586543 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9629 (secure 0 0 0) 0x7fbca803c350 con 0x7fbcb81053b0 2024-04-21T08:42:11.487 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:11.487 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:42:11.488 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 12 2024-04-21T08:42:11.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 -- 172.21.15.7:0/3058586543 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbca00a3c30 msgr2=0x7fbca00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:11.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbca00a3c30 0x7fbca00a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009c20 tx=0x7fbcac005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:11.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 -- 172.21.15.7:0/3058586543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcb81053b0 msgr2=0x7fbcb8100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:11.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcb81053b0 0x7fbcb8100a30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbca8000c00 tx=0x7fbca8005e60 comp rx=0 tx=0).stop 2024-04-21T08:42:11.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 -- 172.21.15.7:0/3058586543 shutdown_connections 2024-04-21T08:42:11.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbca00a3c30 0x7fbca00a60f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:11.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcb8107ce0 0x7fbcb8100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:11.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbcb81053b0 0x7fbcb8100a30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:11.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 --2- 172.21.15.7:0/3058586543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbcb8102a80 0x7fbcb8102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:11.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 -- 172.21.15.7:0/3058586543 >> 172.21.15.7:0/3058586543 conn(0x7fbcb80fc410 msgr2=0x7fbcb81033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:11.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 -- 172.21.15.7:0/3058586543 shutdown_connections 2024-04-21T08:42:11.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:11.490+0000 7fbcbda06640 1 -- 172.21.15.7:0/3058586543 wait complete. 2024-04-21T08:42:11.511 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:42:11 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-b[61177]: starting mds.b at 2024-04-21T08:42:11.804 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:42:11 smithi016 bash[61162]: e079e6b260924e434acad7ed2ecff37d91351259b950920ad10831287e08e49d 2024-04-21T08:42:11.804 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:42:11 smithi016 systemd[1]: Started Ceph mds.b for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:12.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:12 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3058586543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:12.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:12 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:12 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:12 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:12 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:12 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:12 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:12.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:12 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3058586543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:12.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:12 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:12.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:12 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3058586543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:12.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:12 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:12 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:12 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:12 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:12 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:12.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:12 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:13 smithi016 ceph-mon[31259]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:13 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] up:boot 2024-04-21T08:42:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:13 smithi016 ceph-mon[31259]: fsmap 12 up:standby 2024-04-21T08:42:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:13 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:42:13.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:13 smithi077 ceph-mon[31615]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:13.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:13 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] up:boot 2024-04-21T08:42:13.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:13 smithi077 ceph-mon[31615]: fsmap 12 up:standby 2024-04-21T08:42:13.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:13 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:42:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:13 smithi007 ceph-mon[28477]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:13 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] up:boot 2024-04-21T08:42:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:13 smithi007 ceph-mon[28477]: fsmap 12 up:standby 2024-04-21T08:42:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:13 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:42:13.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.609+0000 7fe3077b6640 1 -- 172.21.15.7:0/291794276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe300102040 msgr2=0x7fe3001024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.609+0000 7fe3077b6640 1 --2- 172.21.15.7:0/291794276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe300102040 0x7fe3001024c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009900 tx=0x7fe2f402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 -- 172.21.15.7:0/291794276 shutdown_connections 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 --2- 172.21.15.7:0/291794276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe300102a00 0x7fe300109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 --2- 172.21.15.7:0/291794276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe300102040 0x7fe3001024c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 --2- 172.21.15.7:0/291794276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe300100e60 0x7fe300101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 -- 172.21.15.7:0/291794276 >> 172.21.15.7:0/291794276 conn(0x7fe3000fc460 msgr2=0x7fe3000fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 -- 172.21.15.7:0/291794276 shutdown_connections 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 -- 172.21.15.7:0/291794276 wait complete. 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 Processor -- start 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 -- start start 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe300100e60 0x7fe30006b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe300102040 0x7fe30019cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe300102a00 0x7fe30019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.610+0000 7fe3077b6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe30010ca10 con 0x7fe300100e60 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe3077b6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe30010c890 con 0x7fe300102040 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe3077b6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe30010cb90 con 0x7fe300102a00 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe305552640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe300100e60 0x7fe30006b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe305552640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe300100e60 0x7fe30006b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35168/0 (socket says 172.21.15.7:35168) 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe305552640 1 -- 172.21.15.7:0/329896101 learned_addr learned my addr 172.21.15.7:0/329896101 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe304d51640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe300102040 0x7fe30019cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe305d53640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe300102a00 0x7fe30019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe304d51640 1 -- 172.21.15.7:0/329896101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe300102a00 msgr2=0x7fe30019f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe304d51640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe300102a00 0x7fe30019f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe304d51640 1 -- 172.21.15.7:0/329896101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe300100e60 msgr2=0x7fe30006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe304d51640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe300100e60 0x7fe30006b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe304d51640 1 -- 172.21.15.7:0/329896101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30019fe30 con 0x7fe300102040 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe305552640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe300100e60 0x7fe30006b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:42:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe304d51640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe300102040 0x7fe30019cf00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4000c00 tx=0x7fe2f4005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe2f27fc640 1 -- 172.21.15.7:0/329896101 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4043020 con 0x7fe300102040 2024-04-21T08:42:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe2f27fc640 1 -- 172.21.15.7:0/329896101 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2f4004350 con 0x7fe300102040 2024-04-21T08:42:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe3077b6640 1 -- 172.21.15.7:0/329896101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3001a0060 con 0x7fe300102040 2024-04-21T08:42:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe2f27fc640 1 -- 172.21.15.7:0/329896101 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4003b50 con 0x7fe300102040 2024-04-21T08:42:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.611+0000 7fe3077b6640 1 -- 172.21.15.7:0/329896101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3001a05b0 con 0x7fe300102040 2024-04-21T08:42:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.612+0000 7fe3077b6640 1 -- 172.21.15.7:0/329896101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe300108200 con 0x7fe300102040 2024-04-21T08:42:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.613+0000 7fe2f27fc640 1 -- 172.21.15.7:0/329896101 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fe2f4046020 con 0x7fe300102040 2024-04-21T08:42:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.614+0000 7fe2f27fc640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2ec0a3bb0 0x7fe2ec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.614+0000 7fe305552640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2ec0a3bb0 0x7fe2ec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.614+0000 7fe2f27fc640 1 -- 172.21.15.7:0/329896101 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe2f40f1ff0 con 0x7fe300102040 2024-04-21T08:42:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.614+0000 7fe305552640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2ec0a3bb0 0x7fe2ec0a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0058c0 tx=0x7fe2fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.619+0000 7fe2f27fc640 1 -- 172.21.15.7:0/329896101 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2f404d020 con 0x7fe300102040 2024-04-21T08:42:13.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.846+0000 7fe3077b6640 1 -- 172.21.15.7:0/329896101 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe30002d770 con 0x7fe300102040 2024-04-21T08:42:13.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.847+0000 7fe2f27fc640 1 -- 172.21.15.7:0/329896101 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10455 (secure 0 0 0) 0x7fe2f40bee70 con 0x7fe300102040 2024-04-21T08:42:13.846 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:13.846 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24533,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-21T08:42:13.847 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 13 2024-04-21T08:42:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 -- 172.21.15.7:0/329896101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2ec0a3bb0 msgr2=0x7fe2ec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2ec0a3bb0 0x7fe2ec0a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0058c0 tx=0x7fe2fc005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 -- 172.21.15.7:0/329896101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe300102040 msgr2=0x7fe30019cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe300102040 0x7fe30019cf00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4000c00 tx=0x7fe2f4005dc0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 -- 172.21.15.7:0/329896101 shutdown_connections 2024-04-21T08:42:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2ec0a3bb0 0x7fe2ec0a6070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe300102a00 0x7fe30019f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe300102040 0x7fe30019cf00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 --2- 172.21.15.7:0/329896101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe300100e60 0x7fe30006b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 -- 172.21.15.7:0/329896101 >> 172.21.15.7:0/329896101 conn(0x7fe3000fc460 msgr2=0x7fe3001053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 -- 172.21.15.7:0/329896101 shutdown_connections 2024-04-21T08:42:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.849+0000 7fe3077b6640 1 -- 172.21.15.7:0/329896101 wait complete. 2024-04-21T08:42:13.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:42:13.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.964+0000 7f0c7c532640 1 -- 172.21.15.7:0/508767444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c741053d0 msgr2=0x7f0c741077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.964+0000 7f0c7c532640 1 --2- 172.21.15.7:0/508767444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c741053d0 0x7f0c741077c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009900 tx=0x7f0c6402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.964+0000 7f0c7c532640 1 -- 172.21.15.7:0/508767444 shutdown_connections 2024-04-21T08:42:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.964+0000 7f0c7c532640 1 --2- 172.21.15.7:0/508767444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74107d00 0x7f0c7410a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.964+0000 7f0c7c532640 1 --2- 172.21.15.7:0/508767444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c741053d0 0x7f0c741077c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.964+0000 7f0c7c532640 1 --2- 172.21.15.7:0/508767444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c74102aa0 0x7f0c74104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.964+0000 7f0c7c532640 1 -- 172.21.15.7:0/508767444 >> 172.21.15.7:0/508767444 conn(0x7f0c740fc3f0 msgr2=0x7f0c740fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.964+0000 7f0c7c532640 1 -- 172.21.15.7:0/508767444 shutdown_connections 2024-04-21T08:42:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.964+0000 7f0c7c532640 1 -- 172.21.15.7:0/508767444 wait complete. 2024-04-21T08:42:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7c532640 1 Processor -- start 2024-04-21T08:42:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7c532640 1 -- start start 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7c532640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74102aa0 0x7f0c741981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7c532640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c741053d0 0x7f0c74198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7c532640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c74107d00 0x7f0c7419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7c532640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7410dae0 con 0x7f0c74107d00 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7c532640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7410d960 con 0x7f0c74102aa0 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7c532640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7410dc60 con 0x7f0c741053d0 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7aacf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c74107d00 0x7f0c7419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c79acd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c741053d0 0x7f0c74198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7a2ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74102aa0 0x7f0c741981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7aacf640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c74107d00 0x7f0c7419f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35186/0 (socket says 172.21.15.7:35186) 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7aacf640 1 -- 172.21.15.7:0/1482532961 learned_addr learned my addr 172.21.15.7:0/1482532961 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c79acd640 1 -- 172.21.15.7:0/1482532961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74102aa0 msgr2=0x7f0c741981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c79acd640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74102aa0 0x7f0c741981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c79acd640 1 -- 172.21.15.7:0/1482532961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c74107d00 msgr2=0x7f0c7419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c79acd640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c74107d00 0x7f0c7419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c79acd640 1 -- 172.21.15.7:0/1482532961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7419fdf0 con 0x7f0c741053d0 2024-04-21T08:42:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c7aacf640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c74107d00 0x7f0c7419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:42:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.965+0000 7f0c79acd640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c741053d0 0x7f0c74198710 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0c64036040 tx=0x7f0c64004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.966+0000 7f0c6b7fe640 1 -- 172.21.15.7:0/1482532961 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c64043020 con 0x7f0c741053d0 2024-04-21T08:42:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.966+0000 7f0c6b7fe640 1 -- 172.21.15.7:0/1482532961 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c64004440 con 0x7f0c741053d0 2024-04-21T08:42:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.966+0000 7f0c7c532640 1 -- 172.21.15.7:0/1482532961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c741a0080 con 0x7f0c741053d0 2024-04-21T08:42:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.966+0000 7f0c6b7fe640 1 -- 172.21.15.7:0/1482532961 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c64003c20 con 0x7f0c741053d0 2024-04-21T08:42:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.966+0000 7f0c7c532640 1 -- 172.21.15.7:0/1482532961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c741a05a0 con 0x7f0c741053d0 2024-04-21T08:42:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.967+0000 7f0c7c532640 1 -- 172.21.15.7:0/1482532961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c740fdee0 con 0x7f0c741053d0 2024-04-21T08:42:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.969+0000 7f0c6b7fe640 1 -- 172.21.15.7:0/1482532961 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f0c64046020 con 0x7f0c741053d0 2024-04-21T08:42:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.969+0000 7f0c6b7fe640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0c600a3bb0 0x7f0c600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.969+0000 7f0c6b7fe640 1 -- 172.21.15.7:0/1482532961 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0c640f2080 con 0x7f0c741053d0 2024-04-21T08:42:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.969+0000 7f0c7a2ce640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0c600a3bb0 0x7f0c600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.970+0000 7f0c7a2ce640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0c600a3bb0 0x7f0c600a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0c70009bd0 tx=0x7f0c70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:13.974+0000 7f0c6b7fe640 1 -- 172.21.15.7:0/1482532961 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0c6404d020 con 0x7f0c741053d0 2024-04-21T08:42:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.164+0000 7f0c7c532640 1 -- 172.21.15.7:0/1482532961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c7402d810 con 0x7f0c741053d0 2024-04-21T08:42:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.165+0000 7f0c6b7fe640 1 -- 172.21.15.7:0/1482532961 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18289 (secure 0 0 0) 0x7f0c640bef00 con 0x7f0c741053d0 2024-04-21T08:42:14.164 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:14.165 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":82,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:41:25.912790+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 -- 172.21.15.7:0/1482532961 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0c600a3bb0 msgr2=0x7f0c600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0c600a3bb0 0x7f0c600a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0c70009bd0 tx=0x7f0c70005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 -- 172.21.15.7:0/1482532961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c741053d0 msgr2=0x7f0c74198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c741053d0 0x7f0c74198710 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0c64036040 tx=0x7f0c64004040 comp rx=0 tx=0).stop 2024-04-21T08:42:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 -- 172.21.15.7:0/1482532961 shutdown_connections 2024-04-21T08:42:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0c600a3bb0 0x7f0c600a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c74107d00 0x7f0c7419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c741053d0 0x7f0c74198710 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 --2- 172.21.15.7:0/1482532961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c74102aa0 0x7f0c741981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 -- 172.21.15.7:0/1482532961 >> 172.21.15.7:0/1482532961 conn(0x7f0c740fc3f0 msgr2=0x7f0c74103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 -- 172.21.15.7:0/1482532961 shutdown_connections 2024-04-21T08:42:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.167+0000 7f0c7c532640 1 -- 172.21.15.7:0/1482532961 wait complete. 2024-04-21T08:42:14.178 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:42:14.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:42:14.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.296+0000 7f30c75a4640 1 -- 172.21.15.7:0/2330917112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c01029b0 msgr2=0x7f30c01092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.296+0000 7f30c75a4640 1 --2- 172.21.15.7:0/2330917112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c01029b0 0x7f30c01092d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009ca0 tx=0x7f30b802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.296+0000 7f30c75a4640 1 -- 172.21.15.7:0/2330917112 shutdown_connections 2024-04-21T08:42:14.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.296+0000 7f30c75a4640 1 --2- 172.21.15.7:0/2330917112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c01029b0 0x7f30c01092d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.296+0000 7f30c75a4640 1 --2- 172.21.15.7:0/2330917112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c0101ff0 0x7f30c0102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.296+0000 7f30c75a4640 1 --2- 172.21.15.7:0/2330917112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c0100ea0 0x7f30c01012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.296+0000 7f30c75a4640 1 -- 172.21.15.7:0/2330917112 >> 172.21.15.7:0/2330917112 conn(0x7f30c00fc410 msgr2=0x7f30c00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.296+0000 7f30c75a4640 1 -- 172.21.15.7:0/2330917112 shutdown_connections 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.296+0000 7f30c75a4640 1 -- 172.21.15.7:0/2330917112 wait complete. 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.296+0000 7f30c75a4640 1 Processor -- start 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.296+0000 7f30c75a4640 1 -- start start 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c75a4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c0100ea0 0x7f30c006b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c75a4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c0101ff0 0x7f30c019cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c75a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c01029b0 0x7f30c019f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c75a4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f30c010c9f0 con 0x7f30c0100ea0 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c75a4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30c010c870 con 0x7f30c01029b0 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c75a4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f30c010cb70 con 0x7f30c0101ff0 2024-04-21T08:42:14.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c5340640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c0100ea0 0x7f30c006b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c5340640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c0100ea0 0x7f30c006b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35206/0 (socket says 172.21.15.7:35206) 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c5340640 1 -- 172.21.15.7:0/4008542656 learned_addr learned my addr 172.21.15.7:0/4008542656 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c4b3f640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c0101ff0 0x7f30c019cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c5b41640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c01029b0 0x7f30c019f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c5340640 1 -- 172.21.15.7:0/4008542656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c0101ff0 msgr2=0x7f30c019cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c5340640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c0101ff0 0x7f30c019cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c5340640 1 -- 172.21.15.7:0/4008542656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c01029b0 msgr2=0x7f30c019f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c5340640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c01029b0 0x7f30c019f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c5340640 1 -- 172.21.15.7:0/4008542656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c019fe10 con 0x7f30c0100ea0 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c5340640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c0100ea0 0x7f30c006b1f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f30bc00e9f0 tx=0x7f30bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30b67fc640 1 -- 172.21.15.7:0/4008542656 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30bc00cd80 con 0x7f30c0100ea0 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30b67fc640 1 -- 172.21.15.7:0/4008542656 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30bc0108e0 con 0x7f30c0100ea0 2024-04-21T08:42:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30c75a4640 1 -- 172.21.15.7:0/4008542656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30c01a00a0 con 0x7f30c0100ea0 2024-04-21T08:42:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.297+0000 7f30b67fc640 1 -- 172.21.15.7:0/4008542656 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30bc018980 con 0x7f30c0100ea0 2024-04-21T08:42:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.298+0000 7f30c75a4640 1 -- 172.21.15.7:0/4008542656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30c01a05f0 con 0x7f30c0100ea0 2024-04-21T08:42:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.298+0000 7f30c75a4640 1 -- 172.21.15.7:0/4008542656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30c0108230 con 0x7f30c0100ea0 2024-04-21T08:42:14.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.300+0000 7f30b67fc640 1 -- 172.21.15.7:0/4008542656 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f30bc010430 con 0x7f30c0100ea0 2024-04-21T08:42:14.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.300+0000 7f30b67fc640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30ac0a3bb0 0x7f30ac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.301+0000 7f30c4b3f640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30ac0a3bb0 0x7f30ac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.301+0000 7f30b67fc640 1 -- 172.21.15.7:0/4008542656 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f30bc0d0230 con 0x7f30c0100ea0 2024-04-21T08:42:14.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.301+0000 7f30c4b3f640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30ac0a3bb0 0x7f30ac0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f30c006af60 tx=0x7f30b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.306+0000 7f30b67fc640 1 -- 172.21.15.7:0/4008542656 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f30bc09ca60 con 0x7f30c0100ea0 2024-04-21T08:42:14.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.503+0000 7f30c75a4640 1 -- 172.21.15.7:0/4008542656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f30c002d830 con 0x7f30c0100ea0 2024-04-21T08:42:14.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.503+0000 7f30b67fc640 1 -- 172.21.15.7:0/4008542656 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f30bc0a0d60 con 0x7f30c0100ea0 2024-04-21T08:42:14.503 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:42:14.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 -- 172.21.15.7:0/4008542656 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30ac0a3bb0 msgr2=0x7f30ac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30ac0a3bb0 0x7f30ac0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f30c006af60 tx=0x7f30b0005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 -- 172.21.15.7:0/4008542656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c0100ea0 msgr2=0x7f30c006b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c0100ea0 0x7f30c006b1f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f30bc00e9f0 tx=0x7f30bc00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 -- 172.21.15.7:0/4008542656 shutdown_connections 2024-04-21T08:42:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30ac0a3bb0 0x7f30ac0a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30c01029b0 0x7f30c019f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30c0101ff0 0x7f30c019cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 --2- 172.21.15.7:0/4008542656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c0100ea0 0x7f30c006b1f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 -- 172.21.15.7:0/4008542656 >> 172.21.15.7:0/4008542656 conn(0x7f30c00fc410 msgr2=0x7f30c0105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 -- 172.21.15.7:0/4008542656 shutdown_connections 2024-04-21T08:42:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.506+0000 7f30c75a4640 1 -- 172.21.15.7:0/4008542656 wait complete. 2024-04-21T08:42:14.518 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-21T08:42:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.623+0000 7fa052c48640 1 -- 172.21.15.7:0/1865123844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa04c101180 msgr2=0x7fa04c101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.623+0000 7fa052c48640 1 --2- 172.21.15.7:0/1865123844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa04c101180 0x7fa04c101600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa038009860 tx=0x7fa03802bb70 comp rx=0 tx=0).stop 2024-04-21T08:42:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.623+0000 7fa052c48640 1 -- 172.21.15.7:0/1865123844 shutdown_connections 2024-04-21T08:42:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.623+0000 7fa052c48640 1 --2- 172.21.15.7:0/1865123844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa04c101b40 0x7fa04c1093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.623+0000 7fa052c48640 1 --2- 172.21.15.7:0/1865123844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa04c101180 0x7fa04c101600 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.623+0000 7fa052c48640 1 --2- 172.21.15.7:0/1865123844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa04c102450 0x7fa04c102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.623+0000 7fa052c48640 1 -- 172.21.15.7:0/1865123844 >> 172.21.15.7:0/1865123844 conn(0x7fa04c0fbbb0 msgr2=0x7fa04c0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.624+0000 7fa052c48640 1 -- 172.21.15.7:0/1865123844 shutdown_connections 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.624+0000 7fa052c48640 1 -- 172.21.15.7:0/1865123844 wait complete. 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.624+0000 7fa052c48640 1 Processor -- start 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.624+0000 7fa052c48640 1 -- start start 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.624+0000 7fa052c48640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa04c101180 0x7fa04c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.624+0000 7fa052c48640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa04c101b40 0x7fa04c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.624+0000 7fa052c48640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa04c102450 0x7fa04c19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.624+0000 7fa052c48640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c10db20 con 0x7fa04c101180 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.624+0000 7fa052c48640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c10d9a0 con 0x7fa04c102450 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.624+0000 7fa052c48640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c10dca0 con 0x7fa04c101b40 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa0509e4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa04c101180 0x7fa04c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa0509e4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa04c101180 0x7fa04c1981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35222/0 (socket says 172.21.15.7:35222) 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa0509e4640 1 -- 172.21.15.7:0/3319208630 learned_addr learned my addr 172.21.15.7:0/3319208630 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa04bfff640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa04c101b40 0x7fa04c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa0511e5640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa04c102450 0x7fa04c19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa0509e4640 1 -- 172.21.15.7:0/3319208630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa04c101b40 msgr2=0x7fa04c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa0509e4640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa04c101b40 0x7fa04c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa0509e4640 1 -- 172.21.15.7:0/3319208630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa04c102450 msgr2=0x7fa04c19f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa0509e4640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa04c102450 0x7fa04c19f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa0509e4640 1 -- 172.21.15.7:0/3319208630 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04c19fe00 con 0x7fa04c101180 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa04bfff640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa04c101b40 0x7fa04c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa0511e5640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa04c102450 0x7fa04c19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa0509e4640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa04c101180 0x7fa04c1981e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa04000e9c0 tx=0x7fa04000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa049ffb640 1 -- 172.21.15.7:0/3319208630 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04000cde0 con 0x7fa04c101180 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa049ffb640 1 -- 172.21.15.7:0/3319208630 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0400108e0 con 0x7fa04c101180 2024-04-21T08:42:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa052c48640 1 -- 172.21.15.7:0/3319208630 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa04c1a00f0 con 0x7fa04c101180 2024-04-21T08:42:14.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.625+0000 7fa049ffb640 1 -- 172.21.15.7:0/3319208630 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0400188d0 con 0x7fa04c101180 2024-04-21T08:42:14.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.626+0000 7fa052c48640 1 -- 172.21.15.7:0/3319208630 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa04c1a0610 con 0x7fa04c101180 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.627+0000 7fa052c48640 1 -- 172.21.15.7:0/3319208630 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa04c108340 con 0x7fa04c101180 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.628+0000 7fa049ffb640 1 -- 172.21.15.7:0/3319208630 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fa04001f020 con 0x7fa04c101180 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.629+0000 7fa049ffb640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa03c0a3bb0 0x7fa03c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.629+0000 7fa04bfff640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa03c0a3bb0 0x7fa03c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.629+0000 7fa049ffb640 1 -- 172.21.15.7:0/3319208630 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa0400d06f0 con 0x7fa04c101180 2024-04-21T08:42:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.629+0000 7fa04bfff640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa03c0a3bb0 0x7fa03c0a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa03802c180 tx=0x7fa0380349b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.633+0000 7fa049ffb640 1 -- 172.21.15.7:0/3319208630 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa04009d550 con 0x7fa04c101180 2024-04-21T08:42:14.757 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.757 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.757 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.757 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.757 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.757 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.757 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:14.757 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:14.758 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.758 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:14.758 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/329896101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:14.758 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:14 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1482532961' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:14.758 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.758 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.758 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.758 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.759 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.759 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.759 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:14.759 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:14.759 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.759 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:14.759 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/329896101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:14.759 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:14 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1482532961' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:14.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:14.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:14.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:14.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:14.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/329896101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:14.807 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:14 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1482532961' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.828+0000 7fa052c48640 1 -- 172.21.15.7:0/3319208630 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa04c02d5c0 con 0x7fa04c101180 2024-04-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.829+0000 7fa049ffb640 1 -- 172.21.15.7:0/3319208630 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fa04009ce50 con 0x7fa04c101180 2024-04-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-04-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 1, 2024-04-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": ".mgr" 2024-04-21T08:42:14.828 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:42:14.829 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-04-21T08:42:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 -- 172.21.15.7:0/3319208630 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa03c0a3bb0 msgr2=0x7fa03c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa03c0a3bb0 0x7fa03c0a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa03802c180 tx=0x7fa0380349b0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 -- 172.21.15.7:0/3319208630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa04c101180 msgr2=0x7fa04c1981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa04c101180 0x7fa04c1981e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa04000e9c0 tx=0x7fa04000ee80 comp rx=0 tx=0).stop 2024-04-21T08:42:14.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 -- 172.21.15.7:0/3319208630 shutdown_connections 2024-04-21T08:42:14.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa03c0a3bb0 0x7fa03c0a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa04c102450 0x7fa04c19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa04c101b40 0x7fa04c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 --2- 172.21.15.7:0/3319208630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa04c101180 0x7fa04c1981e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 -- 172.21.15.7:0/3319208630 >> 172.21.15.7:0/3319208630 conn(0x7fa04c0fbbb0 msgr2=0x7fa04c105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 -- 172.21.15.7:0/3319208630 shutdown_connections 2024-04-21T08:42:14.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.831+0000 7fa052c48640 1 -- 172.21.15.7:0/3319208630 wait complete. 2024-04-21T08:42:14.841 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-21T08:42:14.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-21T08:42:14.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.948+0000 7f893d104640 1 -- 172.21.15.7:0/2828926050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8938103b60 msgr2=0x7f893810a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.948+0000 7f893d104640 1 --2- 172.21.15.7:0/2828926050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8938103b60 0x7f893810a3f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8930009ca0 tx=0x7f893002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.948+0000 7f893d104640 1 -- 172.21.15.7:0/2828926050 shutdown_connections 2024-04-21T08:42:14.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.948+0000 7f893d104640 1 --2- 172.21.15.7:0/2828926050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8938103b60 0x7f893810a3f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.948+0000 7f893d104640 1 --2- 172.21.15.7:0/2828926050 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89381031a0 0x7f8938103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.948+0000 7f893d104640 1 --2- 172.21.15.7:0/2828926050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8938101fb0 0x7f89381023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.948+0000 7f893d104640 1 -- 172.21.15.7:0/2828926050 >> 172.21.15.7:0/2828926050 conn(0x7f89380fd520 msgr2=0x7f89380ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.948+0000 7f893d104640 1 -- 172.21.15.7:0/2828926050 shutdown_connections 2024-04-21T08:42:14.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.948+0000 7f893d104640 1 -- 172.21.15.7:0/2828926050 wait complete. 2024-04-21T08:42:14.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.948+0000 7f893d104640 1 Processor -- start 2024-04-21T08:42:14.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f893d104640 1 -- start start 2024-04-21T08:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f893d104640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8938101fb0 0x7f8938198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f893d104640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89381031a0 0x7f89381986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f893d104640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8938103b60 0x7f893819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f893d104640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f893810daf0 con 0x7f89381031a0 2024-04-21T08:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f893d104640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f893810d970 con 0x7f8938101fb0 2024-04-21T08:42:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f893d104640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f893810dc70 con 0x7f8938103b60 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f8936d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8938101fb0 0x7f8938198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f893659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89381031a0 0x7f89381986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f8936d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8938101fb0 0x7f8938198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:58330/0 (socket says 172.21.15.7:58330) 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f8936d9d640 1 -- 172.21.15.7:0/4219723848 learned_addr learned my addr 172.21.15.7:0/4219723848 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f893759e640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8938103b60 0x7f893819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f8936d9d640 1 -- 172.21.15.7:0/4219723848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8938103b60 msgr2=0x7f893819f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f8936d9d640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8938103b60 0x7f893819f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f8936d9d640 1 -- 172.21.15.7:0/4219723848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89381031a0 msgr2=0x7f89381986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f8936d9d640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89381031a0 0x7f89381986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f8936d9d640 1 -- 172.21.15.7:0/4219723848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893819fdb0 con 0x7f8938101fb0 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f893659c640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89381031a0 0x7f89381986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:42:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.949+0000 7f8936d9d640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8938101fb0 0x7f8938198190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f892800e9c0 tx=0x7f892800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.950+0000 7f891ffff640 1 -- 172.21.15.7:0/4219723848 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892800cd80 con 0x7f8938101fb0 2024-04-21T08:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.950+0000 7f891ffff640 1 -- 172.21.15.7:0/4219723848 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89280035f0 con 0x7f8938101fb0 2024-04-21T08:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.950+0000 7f893d104640 1 -- 172.21.15.7:0/4219723848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89381a00a0 con 0x7f8938101fb0 2024-04-21T08:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.950+0000 7f891ffff640 1 -- 172.21.15.7:0/4219723848 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928010690 con 0x7f8938101fb0 2024-04-21T08:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.950+0000 7f893d104640 1 -- 172.21.15.7:0/4219723848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89381a05f0 con 0x7f8938101fb0 2024-04-21T08:42:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.951+0000 7f893d104640 1 -- 172.21.15.7:0/4219723848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89381092d0 con 0x7f8938101fb0 2024-04-21T08:42:14.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.952+0000 7f891ffff640 1 -- 172.21.15.7:0/4219723848 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f89280037a0 con 0x7f8938101fb0 2024-04-21T08:42:14.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.952+0000 7f891ffff640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89200a3bb0 0x7f89200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:14.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.952+0000 7f893659c640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89200a3bb0 0x7f89200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:14.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.952+0000 7f891ffff640 1 -- 172.21.15.7:0/4219723848 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f89280d0310 con 0x7f8938101fb0 2024-04-21T08:42:14.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.953+0000 7f893659c640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89200a3bb0 0x7f89200a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f89381996a0 tx=0x7f892c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:14.956+0000 7f891ffff640 1 -- 172.21.15.7:0/4219723848 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f892809d190 con 0x7f8938101fb0 2024-04-21T08:42:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.159+0000 7f893d104640 1 -- 172.21.15.7:0/4219723848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f893802d5c0 con 0x7f8938101fb0 2024-04-21T08:42:15.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.617+0000 7f891ffff640 1 -- 172.21.15.7:0/4219723848 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f892809ca90 con 0x7f8938101fb0 2024-04-21T08:42:15.616 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_metadata' created 2024-04-21T08:42:15.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.620+0000 7f893d104640 1 -- 172.21.15.7:0/4219723848 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89200a3bb0 msgr2=0x7f89200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.620+0000 7f893d104640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89200a3bb0 0x7f89200a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f89381996a0 tx=0x7f892c005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:15.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.620+0000 7f893d104640 1 -- 172.21.15.7:0/4219723848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8938101fb0 msgr2=0x7f8938198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.620+0000 7f893d104640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8938101fb0 0x7f8938198190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f892800e9c0 tx=0x7f892800ee80 comp rx=0 tx=0).stop 2024-04-21T08:42:15.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.620+0000 7f893d104640 1 -- 172.21.15.7:0/4219723848 shutdown_connections 2024-04-21T08:42:15.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.620+0000 7f893d104640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89200a3bb0 0x7f89200a6070 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f89381996a0 tx=0x7f892c005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:15.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.620+0000 7f893d104640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8938103b60 0x7f893819f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.620+0000 7f893d104640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89381031a0 0x7f89381986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.620+0000 7f893d104640 1 --2- 172.21.15.7:0/4219723848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8938101fb0 0x7f8938198190 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f892800e9c0 tx=0x7f892800ee80 comp rx=0 tx=0).stop 2024-04-21T08:42:15.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.620+0000 7f893d104640 1 -- 172.21.15.7:0/4219723848 >> 172.21.15.7:0/4219723848 conn(0x7f89380fd520 msgr2=0x7f89381064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:15.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.621+0000 7f893d104640 1 -- 172.21.15.7:0/4219723848 shutdown_connections 2024-04-21T08:42:15.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.621+0000 7f893d104640 1 -- 172.21.15.7:0/4219723848 wait complete. 2024-04-21T08:42:15.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-21T08:42:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.742+0000 7fb462da4640 1 -- 172.21.15.7:0/638551898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb45c101b80 msgr2=0x7fb45c102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.742+0000 7fb462da4640 1 --2- 172.21.15.7:0/638551898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb45c101b80 0x7fb45c102000 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb454009ca0 tx=0x7fb45402c180 comp rx=0 tx=0).stop 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 -- 172.21.15.7:0/638551898 shutdown_connections 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 --2- 172.21.15.7:0/638551898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb45c101b80 0x7fb45c102000 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 --2- 172.21.15.7:0/638551898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c1011e0 0x7fb45c101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 --2- 172.21.15.7:0/638551898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c100880 0x7fb45c100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 -- 172.21.15.7:0/638551898 >> 172.21.15.7:0/638551898 conn(0x7fb45c0fc420 msgr2=0x7fb45c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 -- 172.21.15.7:0/638551898 shutdown_connections 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 -- 172.21.15.7:0/638551898 wait complete. 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 Processor -- start 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 -- start start 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c100880 0x7fb45c19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c1011e0 0x7fb45c19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb45c101b80 0x7fb45c1a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c10fcd0 con 0x7fb45c101b80 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c10fb50 con 0x7fb45c1011e0 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb462da4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c10fe50 con 0x7fb45c100880 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb461341640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb45c101b80 0x7fb45c1a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.743+0000 7fb461341640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb45c101b80 0x7fb45c1a1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37266/0 (socket says 172.21.15.7:37266) 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb461341640 1 -- 172.21.15.7:0/2347002193 learned_addr learned my addr 172.21.15.7:0/2347002193 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb460b40640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c100880 0x7fb45c19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb45bfff640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c1011e0 0x7fb45c19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb461341640 1 -- 172.21.15.7:0/2347002193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c100880 msgr2=0x7fb45c19a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb461341640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c100880 0x7fb45c19a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb461341640 1 -- 172.21.15.7:0/2347002193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c1011e0 msgr2=0x7fb45c19a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb461341640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c1011e0 0x7fb45c19a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb461341640 1 -- 172.21.15.7:0/2347002193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45c1a1fc0 con 0x7fb45c101b80 2024-04-21T08:42:15.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb45bfff640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c1011e0 0x7fb45c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:15.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb460b40640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c100880 0x7fb45c19a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:42:15.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb461341640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb45c101b80 0x7fb45c1a1950 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb454005e30 tx=0x7fb454009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:15.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb459ffb640 1 -- 172.21.15.7:0/2347002193 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb454039040 con 0x7fb45c101b80 2024-04-21T08:42:15.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb459ffb640 1 -- 172.21.15.7:0/2347002193 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb454044070 con 0x7fb45c101b80 2024-04-21T08:42:15.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb462da4640 1 -- 172.21.15.7:0/2347002193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45c1a2250 con 0x7fb45c101b80 2024-04-21T08:42:15.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb459ffb640 1 -- 172.21.15.7:0/2347002193 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4540328a0 con 0x7fb45c101b80 2024-04-21T08:42:15.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.744+0000 7fb462da4640 1 -- 172.21.15.7:0/2347002193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb45c1a2790 con 0x7fb45c101b80 2024-04-21T08:42:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.745+0000 7fb462da4640 1 -- 172.21.15.7:0/2347002193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb45c10b5d0 con 0x7fb45c101b80 2024-04-21T08:42:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.746+0000 7fb459ffb640 1 -- 172.21.15.7:0/2347002193 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fb454003c70 con 0x7fb45c101b80 2024-04-21T08:42:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.747+0000 7fb459ffb640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb4440a3c30 0x7fb4440a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.747+0000 7fb460b40640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb4440a3c30 0x7fb4440a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.747+0000 7fb459ffb640 1 -- 172.21.15.7:0/2347002193 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7fb4540f25f0 con 0x7fb45c101b80 2024-04-21T08:42:15.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.748+0000 7fb460b40640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb4440a3c30 0x7fb4440a60f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb44c009fc0 tx=0x7fb44c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:15.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.752+0000 7fb459ffb640 1 -- 172.21.15.7:0/2347002193 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb4540bf390 con 0x7fb45c101b80 2024-04-21T08:42:15.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:15 smithi016 ceph-mon[31259]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:15.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:15 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4008542656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:15.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:15 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:15 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3319208630' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:42:15.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:15 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4219723848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:42:15.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:15 smithi016 ceph-mon[31259]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:42:15.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:15 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:15 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:15 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:15 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:15 smithi077 ceph-mon[31615]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:15.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:15 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4008542656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:15.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:15 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:15 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3319208630' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:42:15.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:15 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4219723848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:42:15.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:15 smithi077 ceph-mon[31615]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:42:15.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:15 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:15 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:15 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:15 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:15 smithi007 ceph-mon[28477]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:15.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:15 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4008542656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:15.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:15 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:15 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3319208630' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:42:15.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:15 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4219723848' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:42:15.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:15 smithi007 ceph-mon[28477]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:42:15.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:15 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:15 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:15 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:15 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:15.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:15.939+0000 7fb462da4640 1 -- 172.21.15.7:0/2347002193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fb45c062b50 con 0x7fb45c101b80 2024-04-21T08:42:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.620+0000 7fb459ffb640 1 -- 172.21.15.7:0/2347002193 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) ==== 171+0+0 (secure 0 0 0) 0x7fb4540bec90 con 0x7fb45c101b80 2024-04-21T08:42:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data' created 2024-04-21T08:42:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.624+0000 7fb462da4640 1 -- 172.21.15.7:0/2347002193 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb4440a3c30 msgr2=0x7fb4440a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.624+0000 7fb462da4640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb4440a3c30 0x7fb4440a60f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb44c009fc0 tx=0x7fb44c005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.624+0000 7fb462da4640 1 -- 172.21.15.7:0/2347002193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb45c101b80 msgr2=0x7fb45c1a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.624+0000 7fb462da4640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb45c101b80 0x7fb45c1a1950 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb454005e30 tx=0x7fb454009360 comp rx=0 tx=0).stop 2024-04-21T08:42:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.624+0000 7fb462da4640 1 -- 172.21.15.7:0/2347002193 shutdown_connections 2024-04-21T08:42:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.624+0000 7fb462da4640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb4440a3c30 0x7fb4440a60f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.624+0000 7fb462da4640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb45c101b80 0x7fb45c1a1950 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.624+0000 7fb462da4640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb45c1011e0 0x7fb45c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.624+0000 7fb462da4640 1 --2- 172.21.15.7:0/2347002193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb45c100880 0x7fb45c19a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.624+0000 7fb462da4640 1 -- 172.21.15.7:0/2347002193 >> 172.21.15.7:0/2347002193 conn(0x7fb45c0fc420 msgr2=0x7fb45c0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.624+0000 7fb462da4640 1 -- 172.21.15.7:0/2347002193 shutdown_connections 2024-04-21T08:42:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.625+0000 7fb462da4640 1 -- 172.21.15.7:0/2347002193 wait complete. 2024-04-21T08:42:16.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-21T08:42:16.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.758+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1973362583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e90075f10 msgr2=0x7f2e90074360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.758+0000 7f2e95ed2640 1 --2- 172.21.15.7:0/1973362583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e90075f10 0x7f2e90074360 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009900 tx=0x7f2e7802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.758+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1973362583 shutdown_connections 2024-04-21T08:42:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.758+0000 7f2e95ed2640 1 --2- 172.21.15.7:0/1973362583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e90109e40 0x7f2e9010c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.758+0000 7f2e95ed2640 1 --2- 172.21.15.7:0/1973362583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e90074930 0x7f2e90074d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.758+0000 7f2e95ed2640 1 --2- 172.21.15.7:0/1973362583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e90075f10 0x7f2e90074360 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.758+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1973362583 >> 172.21.15.7:0/1973362583 conn(0x7f2e900fd4e0 msgr2=0x7f2e900ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.758+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1973362583 shutdown_connections 2024-04-21T08:42:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.758+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1973362583 wait complete. 2024-04-21T08:42:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.759+0000 7f2e95ed2640 1 Processor -- start 2024-04-21T08:42:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.759+0000 7f2e95ed2640 1 -- start start 2024-04-21T08:42:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.759+0000 7f2e95ed2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e90074930 0x7f2e9019a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.759+0000 7f2e95ed2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e90075f10 0x7f2e9019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.759+0000 7f2e95ed2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e90109e40 0x7f2e901a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.759+0000 7f2e95ed2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9010fce0 con 0x7f2e90074930 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.759+0000 7f2e95ed2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9010fb60 con 0x7f2e90109e40 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.759+0000 7f2e95ed2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9010fe60 con 0x7f2e90075f10 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.759+0000 7f2e8f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e90074930 0x7f2e9019a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.759+0000 7f2e8f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e90074930 0x7f2e9019a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37284/0 (socket says 172.21.15.7:37284) 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.759+0000 7f2e8f7fe640 1 -- 172.21.15.7:0/1673501322 learned_addr learned my addr 172.21.15.7:0/1673501322 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8effd640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e90075f10 0x7f2e9019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8ffff640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e90109e40 0x7f2e901a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8f7fe640 1 -- 172.21.15.7:0/1673501322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e90075f10 msgr2=0x7f2e9019a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8f7fe640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e90075f10 0x7f2e9019a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8f7fe640 1 -- 172.21.15.7:0/1673501322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e90109e40 msgr2=0x7f2e901a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8f7fe640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e90109e40 0x7f2e901a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8f7fe640 1 -- 172.21.15.7:0/1673501322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e901a1ff0 con 0x7f2e90074930 2024-04-21T08:42:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8ffff640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e90109e40 0x7f2e901a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8f7fe640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e90074930 0x7f2e9019a420 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006010 tx=0x7f2e7802d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8cff9640 1 -- 172.21.15.7:0/1673501322 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78043020 con 0x7f2e90074930 2024-04-21T08:42:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1673501322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e901a2280 con 0x7f2e90074930 2024-04-21T08:42:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8cff9640 1 -- 172.21.15.7:0/1673501322 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e7802dd90 con 0x7f2e90074930 2024-04-21T08:42:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e8cff9640 1 -- 172.21.15.7:0/1673501322 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78003bb0 con 0x7f2e90074930 2024-04-21T08:42:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.760+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1673501322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e901a2820 con 0x7f2e90074930 2024-04-21T08:42:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.761+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1673501322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e90108c00 con 0x7f2e90074930 2024-04-21T08:42:16.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.762+0000 7f2e8cff9640 1 -- 172.21.15.7:0/1673501322 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f2e78046020 con 0x7f2e90074930 2024-04-21T08:42:16.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.763+0000 7f2e8cff9640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2e7c0a3bb0 0x7f2e7c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.763+0000 7f2e8cff9640 1 -- 172.21.15.7:0/1673501322 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f2e780f2ca0 con 0x7f2e90074930 2024-04-21T08:42:16.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.763+0000 7f2e8effd640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2e7c0a3bb0 0x7f2e7c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.764+0000 7f2e8effd640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2e7c0a3bb0 0x7f2e7c0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2e80005fd0 tx=0x7f2e80005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:16.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.767+0000 7f2e8cff9640 1 -- 172.21.15.7:0/1673501322 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2e7804d020 con 0x7f2e90074930 2024-04-21T08:42:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:16 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:16 smithi016 ceph-mon[31259]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:42:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:16 smithi016 ceph-mon[31259]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:42:16.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:16 smithi016 ceph-mon[31259]: pgmap v286: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:16.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:16 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2347002193' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:42:16.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:16.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:16 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:16 smithi077 ceph-mon[31615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:42:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:16 smithi077 ceph-mon[31615]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:42:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:16 smithi077 ceph-mon[31615]: pgmap v286: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:16 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2347002193' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:42:16.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:16.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:16 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:42:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:16 smithi007 ceph-mon[28477]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:42:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:16 smithi007 ceph-mon[28477]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:42:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:16 smithi007 ceph-mon[28477]: pgmap v286: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:16 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2347002193' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:42:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:16.994+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1673501322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f2e9002d5c0 con 0x7f2e90074930 2024-04-21T08:42:17.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.621+0000 7f2e8cff9640 1 -- 172.21.15.7:0/1673501322 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f2e780bf790 con 0x7f2e90074930 2024-04-21T08:42:17.622 INFO:teuthology.orchestra.run.smithi007.stderr:new fs with metadata pool 2 and data pool 3 2024-04-21T08:42:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1673501322 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2e7c0a3bb0 msgr2=0x7f2e7c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2e7c0a3bb0 0x7f2e7c0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2e80005fd0 tx=0x7f2e80005850 comp rx=0 tx=0).stop 2024-04-21T08:42:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1673501322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e90074930 msgr2=0x7f2e9019a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e90074930 0x7f2e9019a420 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006010 tx=0x7f2e7802d990 comp rx=0 tx=0).stop 2024-04-21T08:42:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1673501322 shutdown_connections 2024-04-21T08:42:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2e7c0a3bb0 0x7f2e7c0a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e90109e40 0x7f2e901a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e90075f10 0x7f2e9019a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 --2- 172.21.15.7:0/1673501322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e90074930 0x7f2e9019a420 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1673501322 >> 172.21.15.7:0/1673501322 conn(0x7f2e900fd4e0 msgr2=0x7f2e90105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:17.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1673501322 shutdown_connections 2024-04-21T08:42:17.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.624+0000 7f2e95ed2640 1 -- 172.21.15.7:0/1673501322 wait complete. 2024-04-21T08:42:17.641 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-21T08:42:17.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-21T08:42:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.754+0000 7fa504d52640 1 -- 172.21.15.7:0/1796199445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5001053a0 msgr2=0x7fa500107790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.754+0000 7fa504d52640 1 --2- 172.21.15.7:0/1796199445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5001053a0 0x7fa500107790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4009900 tx=0x7fa4f402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.754+0000 7fa504d52640 1 -- 172.21.15.7:0/1796199445 shutdown_connections 2024-04-21T08:42:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.754+0000 7fa504d52640 1 --2- 172.21.15.7:0/1796199445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500107cd0 0x7fa50010a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.754+0000 7fa504d52640 1 --2- 172.21.15.7:0/1796199445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5001053a0 0x7fa500107790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.754+0000 7fa504d52640 1 --2- 172.21.15.7:0/1796199445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500102a70 0x7fa500104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.754+0000 7fa504d52640 1 -- 172.21.15.7:0/1796199445 >> 172.21.15.7:0/1796199445 conn(0x7fa5000fc420 msgr2=0x7fa5000fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.754+0000 7fa504d52640 1 -- 172.21.15.7:0/1796199445 shutdown_connections 2024-04-21T08:42:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.754+0000 7fa504d52640 1 -- 172.21.15.7:0/1796199445 wait complete. 2024-04-21T08:42:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa504d52640 1 Processor -- start 2024-04-21T08:42:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa504d52640 1 -- start start 2024-04-21T08:42:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa504d52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500102a70 0x7fa500198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa504d52640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5001053a0 0x7fa5001987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa504d52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500107cd0 0x7fa50019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa504d52640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa50010dac0 con 0x7fa5001053a0 2024-04-21T08:42:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa504d52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa50010d940 con 0x7fa500107cd0 2024-04-21T08:42:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa504d52640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa50010dc40 con 0x7fa500102a70 2024-04-21T08:42:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa4fdd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5001053a0 0x7fa5001987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa4fdd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5001053a0 0x7fa5001987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37300/0 (socket says 172.21.15.7:37300) 2024-04-21T08:42:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa4fdd9b640 1 -- 172.21.15.7:0/3982624867 learned_addr learned my addr 172.21.15.7:0/3982624867 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa4fdd9b640 1 -- 172.21.15.7:0/3982624867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500102a70 msgr2=0x7fa500198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.755+0000 7fa4fdd9b640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500102a70 0x7fa500198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.756+0000 7fa4fdd9b640 1 -- 172.21.15.7:0/3982624867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500107cd0 msgr2=0x7fa50019f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.756+0000 7fa4fdd9b640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500107cd0 0x7fa50019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.756+0000 7fa4fdd9b640 1 -- 172.21.15.7:0/3982624867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50019fe90 con 0x7fa5001053a0 2024-04-21T08:42:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.756+0000 7fa4fdd9b640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5001053a0 0x7fa5001987b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4000c00 tx=0x7fa4f4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.756+0000 7fa4ef7fe640 1 -- 172.21.15.7:0/3982624867 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4043020 con 0x7fa5001053a0 2024-04-21T08:42:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.756+0000 7fa4ef7fe640 1 -- 172.21.15.7:0/3982624867 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4f4004aa0 con 0x7fa5001053a0 2024-04-21T08:42:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.756+0000 7fa504d52640 1 -- 172.21.15.7:0/3982624867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5001a0120 con 0x7fa5001053a0 2024-04-21T08:42:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.756+0000 7fa4ef7fe640 1 -- 172.21.15.7:0/3982624867 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4003d00 con 0x7fa5001053a0 2024-04-21T08:42:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.756+0000 7fa504d52640 1 -- 172.21.15.7:0/3982624867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5001a0670 con 0x7fa5001053a0 2024-04-21T08:42:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.757+0000 7fa504d52640 1 -- 172.21.15.7:0/3982624867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5000fdf10 con 0x7fa5001053a0 2024-04-21T08:42:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.758+0000 7fa4ef7fe640 1 -- 172.21.15.7:0/3982624867 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fa4f4046020 con 0x7fa5001053a0 2024-04-21T08:42:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.759+0000 7fa4ef7fe640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa4e80a3bb0 0x7fa4e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.759+0000 7fa4fe59c640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa4e80a3bb0 0x7fa4e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.759+0000 7fa4ef7fe640 1 -- 172.21.15.7:0/3982624867 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa4f40f2990 con 0x7fa5001053a0 2024-04-21T08:42:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.759+0000 7fa4fe59c640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa4e80a3bb0 0x7fa4e80a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0005fd0 tx=0x7fa4f0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.763+0000 7fa4ef7fe640 1 -- 172.21.15.7:0/3982624867 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa4f404d020 con 0x7fa5001053a0 2024-04-21T08:42:17.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:17 smithi016 ceph-mon[31259]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:42:17.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:17 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2347002193' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:42:17.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:17 smithi016 ceph-mon[31259]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:42:17.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:17 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1673501322' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:42:17.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:17 smithi016 ceph-mon[31259]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:42:17.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:17 smithi077 ceph-mon[31615]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:42:17.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:17 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2347002193' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:42:17.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:17 smithi077 ceph-mon[31615]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:42:17.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:17 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1673501322' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:42:17.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:17 smithi077 ceph-mon[31615]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:42:17.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:17 smithi007 ceph-mon[28477]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:42:17.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:17 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2347002193' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:42:17.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:17 smithi007 ceph-mon[28477]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:42:17.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:17 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1673501322' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:42:17.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:17 smithi007 ceph-mon[28477]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:42:17.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:17.958+0000 7fa504d52640 1 -- 172.21.15.7:0/3982624867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fa50002d5c0 con 0x7fa5001053a0 2024-04-21T08:42:18.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.524+0000 7fa4ef7fe640 1 -- 172.21.15.7:0/3982624867 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7fa4f40bf420 con 0x7fa5001053a0 2024-04-21T08:42:18.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.527+0000 7fa504d52640 1 -- 172.21.15.7:0/3982624867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa4e80a3bb0 msgr2=0x7fa4e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:18.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.527+0000 7fa504d52640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa4e80a3bb0 0x7fa4e80a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0005fd0 tx=0x7fa4f0005850 comp rx=0 tx=0).stop 2024-04-21T08:42:18.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.527+0000 7fa504d52640 1 -- 172.21.15.7:0/3982624867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5001053a0 msgr2=0x7fa5001987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:18.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.527+0000 7fa504d52640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5001053a0 0x7fa5001987b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4000c00 tx=0x7fa4f4004270 comp rx=0 tx=0).stop 2024-04-21T08:42:18.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.527+0000 7fa504d52640 1 -- 172.21.15.7:0/3982624867 shutdown_connections 2024-04-21T08:42:18.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.527+0000 7fa504d52640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa4e80a3bb0 0x7fa4e80a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:18.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.527+0000 7fa504d52640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa500107cd0 0x7fa50019f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:18.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.527+0000 7fa504d52640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5001053a0 0x7fa5001987b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:18.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.527+0000 7fa504d52640 1 --2- 172.21.15.7:0/3982624867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa500102a70 0x7fa500198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:18.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.527+0000 7fa504d52640 1 -- 172.21.15.7:0/3982624867 >> 172.21.15.7:0/3982624867 conn(0x7fa5000fc420 msgr2=0x7fa500103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:18.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.528+0000 7fa504d52640 1 -- 172.21.15.7:0/3982624867 shutdown_connections 2024-04-21T08:42:18.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.528+0000 7fa504d52640 1 -- 172.21.15.7:0/3982624867 wait complete. 2024-04-21T08:42:18.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-21T08:42:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.659+0000 7fc4e284e640 1 -- 172.21.15.7:0/1714323568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc107c90 msgr2=0x7fc4dc10a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.659+0000 7fc4e284e640 1 --2- 172.21.15.7:0/1714323568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc107c90 0x7fc4dc10a0e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4009ca0 tx=0x7fc4d402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.659+0000 7fc4e284e640 1 -- 172.21.15.7:0/1714323568 shutdown_connections 2024-04-21T08:42:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.659+0000 7fc4e284e640 1 --2- 172.21.15.7:0/1714323568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc107c90 0x7fc4dc10a0e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.659+0000 7fc4e284e640 1 --2- 172.21.15.7:0/1714323568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc105360 0x7fc4dc107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.659+0000 7fc4e284e640 1 --2- 172.21.15.7:0/1714323568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4dc102a30 0x7fc4dc104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.659+0000 7fc4e284e640 1 -- 172.21.15.7:0/1714323568 >> 172.21.15.7:0/1714323568 conn(0x7fc4dc0fc460 msgr2=0x7fc4dc0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.659+0000 7fc4e284e640 1 -- 172.21.15.7:0/1714323568 shutdown_connections 2024-04-21T08:42:18.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.659+0000 7fc4e284e640 1 -- 172.21.15.7:0/1714323568 wait complete. 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.659+0000 7fc4e284e640 1 Processor -- start 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.659+0000 7fc4e284e640 1 -- start start 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e284e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102a30 0x7fc4dc198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e284e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4dc105360 0x7fc4dc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e284e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc107c90 0x7fc4dc19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e284e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc10daf0 con 0x7fc4dc107c90 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e284e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc10d970 con 0x7fc4dc102a30 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e284e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc10dc70 con 0x7fc4dc105360 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e0deb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc107c90 0x7fc4dc19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e0deb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc107c90 0x7fc4dc19f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37308/0 (socket says 172.21.15.7:37308) 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e0deb640 1 -- 172.21.15.7:0/3507958382 learned_addr learned my addr 172.21.15.7:0/3507958382 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4dbfff640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102a30 0x7fc4dc198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:18.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4db7fe640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4dc105360 0x7fc4dc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e0deb640 1 -- 172.21.15.7:0/3507958382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4dc105360 msgr2=0x7fc4dc1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e0deb640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4dc105360 0x7fc4dc1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e0deb640 1 -- 172.21.15.7:0/3507958382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102a30 msgr2=0x7fc4dc198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e0deb640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102a30 0x7fc4dc198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e0deb640 1 -- 172.21.15.7:0/3507958382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4dc19fd60 con 0x7fc4dc107c90 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e0deb640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc107c90 0x7fc4dc19f6f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4032520 tx=0x7fc4d40347b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4d97fa640 1 -- 172.21.15.7:0/3507958382 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4039040 con 0x7fc4dc107c90 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4d97fa640 1 -- 172.21.15.7:0/3507958382 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4d4044070 con 0x7fc4dc107c90 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.660+0000 7fc4e284e640 1 -- 172.21.15.7:0/3507958382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4dc19fff0 con 0x7fc4dc107c90 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.661+0000 7fc4d97fa640 1 -- 172.21.15.7:0/3507958382 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d40042a0 con 0x7fc4dc107c90 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.661+0000 7fc4e284e640 1 -- 172.21.15.7:0/3507958382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4dc1a0450 con 0x7fc4dc107c90 2024-04-21T08:42:18.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.662+0000 7fc4e284e640 1 -- 172.21.15.7:0/3507958382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4dc0fdf50 con 0x7fc4dc107c90 2024-04-21T08:42:18.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.663+0000 7fc4d97fa640 1 -- 172.21.15.7:0/3507958382 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc4d4003970 con 0x7fc4dc107c90 2024-04-21T08:42:18.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.663+0000 7fc4d97fa640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4c40a3bb0 0x7fc4c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:18.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.663+0000 7fc4dbfff640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4c40a3bb0 0x7fc4c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:18.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.664+0000 7fc4d97fa640 1 -- 172.21.15.7:0/3507958382 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7fc4d40f2790 con 0x7fc4dc107c90 2024-04-21T08:42:18.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.664+0000 7fc4dbfff640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4c40a3bb0 0x7fc4c40a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc005fd0 tx=0x7fc4cc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:18.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.668+0000 7fc4d97fa640 1 -- 172.21.15.7:0/3507958382 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4d4035d30 con 0x7fc4dc107c90 2024-04-21T08:42:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:18 smithi016 ceph-mon[31259]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:42:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:18 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1673501322' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:42:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:18 smithi016 ceph-mon[31259]: pgmap v289: 97 pgs: 18 creating+peering, 6 active+clean, 73 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:18 smithi016 ceph-mon[31259]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:42:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:18 smithi016 ceph-mon[31259]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:42:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:18 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3982624867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:42:18.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:18 smithi077 ceph-mon[31615]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:42:18.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:18 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1673501322' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:42:18.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:18 smithi077 ceph-mon[31615]: pgmap v289: 97 pgs: 18 creating+peering, 6 active+clean, 73 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:18.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:18 smithi077 ceph-mon[31615]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:42:18.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:18 smithi077 ceph-mon[31615]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:42:18.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:18 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3982624867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:42:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:18.856+0000 7fc4e284e640 1 -- 172.21.15.7:0/3507958382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc4dc062bf0 con 0x7fc4dc107c90 2024-04-21T08:42:18.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:18 smithi007 ceph-mon[28477]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:42:18.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:18 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1673501322' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:42:18.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:18 smithi007 ceph-mon[28477]: pgmap v289: 97 pgs: 18 creating+peering, 6 active+clean, 73 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:18.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:18 smithi007 ceph-mon[28477]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:42:18.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:18 smithi007 ceph-mon[28477]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:42:18.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:18 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3982624867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:42:19.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:19 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3982624867' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:42:19.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:19 smithi016 ceph-mon[31259]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:42:19.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:19 smithi016 ceph-mon[31259]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:42:19.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:19 smithi016 ceph-mon[31259]: Cluster is now healthy 2024-04-21T08:42:19.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:19 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] up:active 2024-04-21T08:42:19.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:19 smithi016 ceph-mon[31259]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:19.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:19 smithi016 ceph-mon[31259]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:19.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:19 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3507958382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:42:19.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:19 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3982624867' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:42:19.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:19 smithi077 ceph-mon[31615]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:42:19.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:19 smithi077 ceph-mon[31615]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:42:19.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:19 smithi077 ceph-mon[31615]: Cluster is now healthy 2024-04-21T08:42:19.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:19 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] up:active 2024-04-21T08:42:19.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:19 smithi077 ceph-mon[31615]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:19.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:19 smithi077 ceph-mon[31615]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:19.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:19 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3507958382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:42:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:19 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3982624867' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:42:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:19 smithi007 ceph-mon[28477]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:42:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:19 smithi007 ceph-mon[28477]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:42:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:19 smithi007 ceph-mon[28477]: Cluster is now healthy 2024-04-21T08:42:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:19 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] up:active 2024-04-21T08:42:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:19 smithi007 ceph-mon[28477]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:19 smithi007 ceph-mon[28477]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:19.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:19 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3507958382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:42:20.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.637+0000 7fc4d97fa640 1 -- 172.21.15.7:0/3507958382 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7fc4d40bf140 con 0x7fc4dc107c90 2024-04-21T08:42:20.638 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data_ec' created 2024-04-21T08:42:20.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 -- 172.21.15.7:0/3507958382 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4c40a3bb0 msgr2=0x7fc4c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4c40a3bb0 0x7fc4c40a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc005fd0 tx=0x7fc4cc00b390 comp rx=0 tx=0).stop 2024-04-21T08:42:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 -- 172.21.15.7:0/3507958382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc107c90 msgr2=0x7fc4dc19f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc107c90 0x7fc4dc19f6f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4032520 tx=0x7fc4d40347b0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 -- 172.21.15.7:0/3507958382 shutdown_connections 2024-04-21T08:42:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4c40a3bb0 0x7fc4c40a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc107c90 0x7fc4dc19f6f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4dc105360 0x7fc4dc1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 --2- 172.21.15.7:0/3507958382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4dc102a30 0x7fc4dc198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 -- 172.21.15.7:0/3507958382 >> 172.21.15.7:0/3507958382 conn(0x7fc4dc0fc460 msgr2=0x7fc4dc1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 -- 172.21.15.7:0/3507958382 shutdown_connections 2024-04-21T08:42:20.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.641+0000 7fc4e284e640 1 -- 172.21.15.7:0/3507958382 wait complete. 2024-04-21T08:42:20.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-21T08:42:20.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.773+0000 7fba19493640 1 -- 172.21.15.7:0/2010465581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba14103b60 msgr2=0x7fba1410a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:20.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.773+0000 7fba19493640 1 --2- 172.21.15.7:0/2010465581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba14103b60 0x7fba1410a3f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009ca0 tx=0x7fba0c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.773+0000 7fba19493640 1 -- 172.21.15.7:0/2010465581 shutdown_connections 2024-04-21T08:42:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.773+0000 7fba19493640 1 --2- 172.21.15.7:0/2010465581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba14103b60 0x7fba1410a3f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.773+0000 7fba19493640 1 --2- 172.21.15.7:0/2010465581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba141031a0 0x7fba14103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.773+0000 7fba19493640 1 --2- 172.21.15.7:0/2010465581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba14101fb0 0x7fba141023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.773+0000 7fba19493640 1 -- 172.21.15.7:0/2010465581 >> 172.21.15.7:0/2010465581 conn(0x7fba140fd520 msgr2=0x7fba140ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.773+0000 7fba19493640 1 -- 172.21.15.7:0/2010465581 shutdown_connections 2024-04-21T08:42:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.773+0000 7fba19493640 1 -- 172.21.15.7:0/2010465581 wait complete. 2024-04-21T08:42:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.773+0000 7fba19493640 1 Processor -- start 2024-04-21T08:42:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba19493640 1 -- start start 2024-04-21T08:42:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba19493640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14101fb0 0x7fba14075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba19493640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba141031a0 0x7fba14074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba19493640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba14103b60 0x7fba14074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba19493640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fba1410daf0 con 0x7fba141031a0 2024-04-21T08:42:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba19493640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba1410d970 con 0x7fba14103b60 2024-04-21T08:42:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba19493640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fba1410dc70 con 0x7fba14101fb0 2024-04-21T08:42:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba0bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba141031a0 0x7fba14074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba0bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba141031a0 0x7fba14074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37328/0 (socket says 172.21.15.7:37328) 2024-04-21T08:42:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba0bfff640 1 -- 172.21.15.7:0/3524411761 learned_addr learned my addr 172.21.15.7:0/3524411761 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba137fe640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba14103b60 0x7fba14074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba12ffd640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14101fb0 0x7fba14075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba0bfff640 1 -- 172.21.15.7:0/3524411761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14101fb0 msgr2=0x7fba14075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba0bfff640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14101fb0 0x7fba14075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba0bfff640 1 -- 172.21.15.7:0/3524411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba14103b60 msgr2=0x7fba14074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba0bfff640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba14103b60 0x7fba14074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba0bfff640 1 -- 172.21.15.7:0/3524411761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba14075110 con 0x7fba141031a0 2024-04-21T08:42:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba0bfff640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba141031a0 0x7fba14074340 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00e9f0 tx=0x7fb9fc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.774+0000 7fba10ff9640 1 -- 172.21.15.7:0/3524411761 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc00cd80 con 0x7fba141031a0 2024-04-21T08:42:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.775+0000 7fba10ff9640 1 -- 172.21.15.7:0/3524411761 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9fc0054d0 con 0x7fba141031a0 2024-04-21T08:42:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.775+0000 7fba10ff9640 1 -- 172.21.15.7:0/3524411761 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc010690 con 0x7fba141031a0 2024-04-21T08:42:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.775+0000 7fba19493640 1 -- 172.21.15.7:0/3524411761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba141a22b0 con 0x7fba141031a0 2024-04-21T08:42:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.775+0000 7fba19493640 1 -- 172.21.15.7:0/3524411761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba141a27b0 con 0x7fba141031a0 2024-04-21T08:42:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.776+0000 7fba19493640 1 -- 172.21.15.7:0/3524411761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba141093a0 con 0x7fba141031a0 2024-04-21T08:42:20.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.777+0000 7fba10ff9640 1 -- 172.21.15.7:0/3524411761 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fb9fc004e30 con 0x7fba141031a0 2024-04-21T08:42:20.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.777+0000 7fba10ff9640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba000a3c30 0x7fba000a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:20.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.778+0000 7fba10ff9640 1 -- 172.21.15.7:0/3524411761 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7fb9fc0d0c70 con 0x7fba141031a0 2024-04-21T08:42:20.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.778+0000 7fba12ffd640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba000a3c30 0x7fba000a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.778+0000 7fba12ffd640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba000a3c30 0x7fba000a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fba14103000 tx=0x7fba04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.781+0000 7fba10ff9640 1 -- 172.21.15.7:0/3524411761 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb9fc053ff0 con 0x7fba141031a0 2024-04-21T08:42:20.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:20 smithi007 ceph-mon[28477]: pgmap v291: 97 pgs: 18 creating+peering, 45 active+clean, 34 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-04-21T08:42:20.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:20 smithi007 ceph-mon[28477]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:42:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:20.968+0000 7fba19493640 1 -- 172.21.15.7:0/3524411761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fba14062b50 con 0x7fba141031a0 2024-04-21T08:42:21.003 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:20 smithi016 ceph-mon[31259]: pgmap v291: 97 pgs: 18 creating+peering, 45 active+clean, 34 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-04-21T08:42:21.003 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:20 smithi016 ceph-mon[31259]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:42:21.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:20 smithi077 ceph-mon[31615]: pgmap v291: 97 pgs: 18 creating+peering, 45 active+clean, 34 unknown; 577 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-04-21T08:42:21.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:20 smithi077 ceph-mon[31615]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:42:21.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.639+0000 7fba10ff9640 1 -- 172.21.15.7:0/3524411761 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) ==== 169+0+0 (secure 0 0 0) 0x7fb9fc09d3d0 con 0x7fba141031a0 2024-04-21T08:42:21.640 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 4 allow_ec_overwrites to true 2024-04-21T08:42:21.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.643+0000 7fba19493640 1 -- 172.21.15.7:0/3524411761 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba000a3c30 msgr2=0x7fba000a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:21.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.643+0000 7fba19493640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba000a3c30 0x7fba000a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fba14103000 tx=0x7fba04005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:21.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.643+0000 7fba19493640 1 -- 172.21.15.7:0/3524411761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba141031a0 msgr2=0x7fba14074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:21.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.643+0000 7fba19493640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba141031a0 0x7fba14074340 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00e9f0 tx=0x7fb9fc00edf0 comp rx=0 tx=0).stop 2024-04-21T08:42:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.643+0000 7fba19493640 1 -- 172.21.15.7:0/3524411761 shutdown_connections 2024-04-21T08:42:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.643+0000 7fba19493640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba000a3c30 0x7fba000a60f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.643+0000 7fba19493640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba14103b60 0x7fba14074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.643+0000 7fba19493640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba141031a0 0x7fba14074340 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.643+0000 7fba19493640 1 --2- 172.21.15.7:0/3524411761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba14101fb0 0x7fba14075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.643+0000 7fba19493640 1 -- 172.21.15.7:0/3524411761 >> 172.21.15.7:0/3524411761 conn(0x7fba140fd520 msgr2=0x7fba14106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.643+0000 7fba19493640 1 -- 172.21.15.7:0/3524411761 shutdown_connections 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.644+0000 7fba19493640 1 -- 172.21.15.7:0/3524411761 wait complete. 2024-04-21T08:42:21.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-21T08:42:21.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.777+0000 7effbf952640 1 -- 172.21.15.7:0/2015438844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8103180 msgr2=0x7effb81055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:21.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.777+0000 7effbf952640 1 --2- 172.21.15.7:0/2015438844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8103180 0x7effb81055f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7effa8009900 tx=0x7effa802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.777+0000 7effbf952640 1 -- 172.21.15.7:0/2015438844 shutdown_connections 2024-04-21T08:42:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.777+0000 7effbf952640 1 --2- 172.21.15.7:0/2015438844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb8105b30 0x7effb8105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.777+0000 7effbf952640 1 --2- 172.21.15.7:0/2015438844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8103180 0x7effb81055f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.777+0000 7effbf952640 1 --2- 172.21.15.7:0/2015438844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb8100810 0x7effb8102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.777+0000 7effbf952640 1 -- 172.21.15.7:0/2015438844 >> 172.21.15.7:0/2015438844 conn(0x7effb80fc430 msgr2=0x7effb80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.777+0000 7effbf952640 1 -- 172.21.15.7:0/2015438844 shutdown_connections 2024-04-21T08:42:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.777+0000 7effbf952640 1 -- 172.21.15.7:0/2015438844 wait complete. 2024-04-21T08:42:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbf952640 1 Processor -- start 2024-04-21T08:42:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbf952640 1 -- start start 2024-04-21T08:42:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbf952640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb8100810 0x7effb819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:21.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbf952640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb8103180 0x7effb819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbf952640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8105b30 0x7effb81a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbf952640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7effb81140c0 con 0x7effb8105b30 2024-04-21T08:42:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbf952640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7effb8113f40 con 0x7effb8103180 2024-04-21T08:42:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbf952640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7effb8114240 con 0x7effb8100810 2024-04-21T08:42:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbdeef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8105b30 0x7effb81a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbdeef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8105b30 0x7effb81a5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37346/0 (socket says 172.21.15.7:37346) 2024-04-21T08:42:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbdeef640 1 -- 172.21.15.7:0/3819543900 learned_addr learned my addr 172.21.15.7:0/3819543900 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbceed640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb8103180 0x7effb819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.778+0000 7effbd6ee640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb8100810 0x7effb819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effbdeef640 1 -- 172.21.15.7:0/3819543900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb8100810 msgr2=0x7effb819e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:21.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effbdeef640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb8100810 0x7effb819e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:21.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effbdeef640 1 -- 172.21.15.7:0/3819543900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb8103180 msgr2=0x7effb819eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:21.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effbdeef640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb8103180 0x7effb819eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:21.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effbdeef640 1 -- 172.21.15.7:0/3819543900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb81a6430 con 0x7effb8105b30 2024-04-21T08:42:21.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effbceed640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb8103180 0x7effb819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:42:21.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effbdeef640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8105b30 0x7effb81a5dc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7effb000cad0 tx=0x7effb000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:21.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effae7fc640 1 -- 172.21.15.7:0/3819543900 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb000ae20 con 0x7effb8105b30 2024-04-21T08:42:21.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effbf952640 1 -- 172.21.15.7:0/3819543900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb81a6720 con 0x7effb8105b30 2024-04-21T08:42:21.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effae7fc640 1 -- 172.21.15.7:0/3819543900 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effb0005020 con 0x7effb8105b30 2024-04-21T08:42:21.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effae7fc640 1 -- 172.21.15.7:0/3819543900 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb000f6b0 con 0x7effb8105b30 2024-04-21T08:42:21.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.779+0000 7effbf952640 1 -- 172.21.15.7:0/3819543900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb81a6c90 con 0x7effb8105b30 2024-04-21T08:42:21.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.780+0000 7effbf952640 1 -- 172.21.15.7:0/3819543900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb80fdf20 con 0x7effb8105b30 2024-04-21T08:42:21.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.781+0000 7effae7fc640 1 -- 172.21.15.7:0/3819543900 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7effb00051d0 con 0x7effb8105b30 2024-04-21T08:42:21.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.782+0000 7effae7fc640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7effa40a3bb0 0x7effa40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:21.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.782+0000 7effbd6ee640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7effa40a3bb0 0x7effa40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:21.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.782+0000 7effae7fc640 1 -- 172.21.15.7:0/3819543900 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7effb00d0930 con 0x7effb8105b30 2024-04-21T08:42:21.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.782+0000 7effbd6ee640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7effa40a3bb0 0x7effa40a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7effb4009bd0 tx=0x7effb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:21.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:21.786+0000 7effae7fc640 1 -- 172.21.15.7:0/3819543900 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7effb0014030 con 0x7effb8105b30 2024-04-21T08:42:21.977 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:21 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3507958382' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:42:21.977 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:21 smithi016 ceph-mon[31259]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:42:21.977 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:21 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3524411761' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:42:21.977 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:21 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3524411761' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:42:21.977 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:21 smithi016 ceph-mon[31259]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:42:22.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:21 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3507958382' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:42:22.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:21 smithi077 ceph-mon[31615]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:42:22.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:21 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3524411761' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:42:22.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:21 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3524411761' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:42:22.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:21 smithi077 ceph-mon[31615]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:42:22.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.020+0000 7effbf952640 1 -- 172.21.15.7:0/3819543900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7effb8062b50 con 0x7effb8105b30 2024-04-21T08:42:22.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:21 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3507958382' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:42:22.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:21 smithi007 ceph-mon[28477]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:42:22.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:21 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3524411761' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:42:22.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:21 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3524411761' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:42:22.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:21 smithi007 ceph-mon[28477]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:42:22.304 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:42:22 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-b[61177]: 2024-04-21T08:42:22.078+0000 7fb5f067c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-21T08:42:22.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.659+0000 7effae7fc640 1 -- 172.21.15.7:0/3819543900 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7effb009caa0 con 0x7effb8105b30 2024-04-21T08:42:22.660 INFO:teuthology.orchestra.run.smithi007.stderr:added data pool 4 to fsmap 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 -- 172.21.15.7:0/3819543900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7effa40a3bb0 msgr2=0x7effa40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7effa40a3bb0 0x7effa40a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7effb4009bd0 tx=0x7effb4005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 -- 172.21.15.7:0/3819543900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8105b30 msgr2=0x7effb81a5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8105b30 0x7effb81a5dc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7effb000cad0 tx=0x7effb000cf90 comp rx=0 tx=0).stop 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 -- 172.21.15.7:0/3819543900 shutdown_connections 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7effa40a3bb0 0x7effa40a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8105b30 0x7effb81a5dc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effb8103180 0x7effb819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 --2- 172.21.15.7:0/3819543900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effb8100810 0x7effb819e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 -- 172.21.15.7:0/3819543900 >> 172.21.15.7:0/3819543900 conn(0x7effb80fc430 msgr2=0x7effb81010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 -- 172.21.15.7:0/3819543900 shutdown_connections 2024-04-21T08:42:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.663+0000 7effbf952640 1 -- 172.21.15.7:0/3819543900 wait complete. 2024-04-21T08:42:22.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.786+0000 7f6db2270640 1 -- 172.21.15.7:0/950996191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dac101b50 msgr2=0x7f6dac101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.786+0000 7f6db2270640 1 --2- 172.21.15.7:0/950996191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dac101b50 0x7f6dac101fd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009ca0 tx=0x7f6da402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.786+0000 7f6db2270640 1 -- 172.21.15.7:0/950996191 shutdown_connections 2024-04-21T08:42:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.786+0000 7f6db2270640 1 --2- 172.21.15.7:0/950996191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dac101b50 0x7f6dac101fd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.786+0000 7f6db2270640 1 --2- 172.21.15.7:0/950996191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac1011b0 0x7f6dac101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.786+0000 7f6db2270640 1 --2- 172.21.15.7:0/950996191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dac100850 0x7f6dac100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.786+0000 7f6db2270640 1 -- 172.21.15.7:0/950996191 >> 172.21.15.7:0/950996191 conn(0x7f6dac0fc3f0 msgr2=0x7f6dac0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.786+0000 7f6db2270640 1 -- 172.21.15.7:0/950996191 shutdown_connections 2024-04-21T08:42:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.786+0000 7f6db2270640 1 -- 172.21.15.7:0/950996191 wait complete. 2024-04-21T08:42:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.786+0000 7f6db2270640 1 Processor -- start 2024-04-21T08:42:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.786+0000 7f6db2270640 1 -- start start 2024-04-21T08:42:22.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db2270640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac100850 0x7f6dac19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:22.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db2270640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dac1011b0 0x7f6dac19a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:22.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db2270640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dac101b50 0x7f6dac1a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:22.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db2270640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac10fcd0 con 0x7f6dac101b50 2024-04-21T08:42:22.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db2270640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac10fb50 con 0x7f6dac100850 2024-04-21T08:42:22.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db2270640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac10fe50 con 0x7f6dac1011b0 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db080d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dac101b50 0x7f6dac1a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db080d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dac101b50 0x7f6dac1a18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37366/0 (socket says 172.21.15.7:37366) 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db080d640 1 -- 172.21.15.7:0/2509138192 learned_addr learned my addr 172.21.15.7:0/2509138192 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6dab7fe640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dac1011b0 0x7f6dac19a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6dabfff640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac100850 0x7f6dac19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db080d640 1 -- 172.21.15.7:0/2509138192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dac1011b0 msgr2=0x7f6dac19a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db080d640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dac1011b0 0x7f6dac19a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db080d640 1 -- 172.21.15.7:0/2509138192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac100850 msgr2=0x7f6dac19a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db080d640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac100850 0x7f6dac19a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db080d640 1 -- 172.21.15.7:0/2509138192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dac1a1f30 con 0x7f6dac101b50 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.787+0000 7f6db080d640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dac101b50 0x7f6dac1a18c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6da402c700 tx=0x7f6da40093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:22.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.788+0000 7f6da97fa640 1 -- 172.21.15.7:0/2509138192 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4041020 con 0x7f6dac101b50 2024-04-21T08:42:22.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.788+0000 7f6db2270640 1 -- 172.21.15.7:0/2509138192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dac1a21c0 con 0x7f6dac101b50 2024-04-21T08:42:22.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.788+0000 7f6da97fa640 1 -- 172.21.15.7:0/2509138192 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6da4034d60 con 0x7f6dac101b50 2024-04-21T08:42:22.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.788+0000 7f6da97fa640 1 -- 172.21.15.7:0/2509138192 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4039040 con 0x7f6dac101b50 2024-04-21T08:42:22.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.788+0000 7f6db2270640 1 -- 172.21.15.7:0/2509138192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dac1a2700 con 0x7f6dac101b50 2024-04-21T08:42:22.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.789+0000 7f6db2270640 1 -- 172.21.15.7:0/2509138192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dac10b750 con 0x7f6dac101b50 2024-04-21T08:42:22.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.790+0000 7f6da97fa640 1 -- 172.21.15.7:0/2509138192 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6da4044020 con 0x7f6dac101b50 2024-04-21T08:42:22.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.790+0000 7f6da97fa640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6d940a3bb0 0x7f6d940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:22.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.791+0000 7f6dabfff640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6d940a3bb0 0x7f6d940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:22.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.791+0000 7f6da97fa640 1 -- 172.21.15.7:0/2509138192 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6da40f2ab0 con 0x7f6dac101b50 2024-04-21T08:42:22.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.791+0000 7f6dabfff640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6d940a3bb0 0x7f6d940a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009b60 tx=0x7f6d9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:22.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:22.794+0000 7f6da97fa640 1 -- 172.21.15.7:0/2509138192 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6da404b020 con 0x7f6dac101b50 2024-04-21T08:42:23.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.021+0000 7f6db2270640 1 -- 172.21.15.7:0/2509138192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6dac062b50 con 0x7f6dac101b50 2024-04-21T08:42:23.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.021+0000 7f6da97fa640 1 -- 172.21.15.7:0/2509138192 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11773 (secure 0 0 0) 0x7f6da40bf250 con 0x7f6dac101b50 2024-04-21T08:42:23.022 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:23.022 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:22.023279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:42:23.022 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 17 2024-04-21T08:42:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.023+0000 7f6db2270640 1 -- 172.21.15.7:0/2509138192 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6d940a3bb0 msgr2=0x7f6d940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.023+0000 7f6db2270640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6d940a3bb0 0x7f6d940a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009b60 tx=0x7f6d9c005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.023+0000 7f6db2270640 1 -- 172.21.15.7:0/2509138192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dac101b50 msgr2=0x7f6dac1a18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.023+0000 7f6db2270640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dac101b50 0x7f6dac1a18c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6da402c700 tx=0x7f6da40093f0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.024+0000 7f6db2270640 1 -- 172.21.15.7:0/2509138192 shutdown_connections 2024-04-21T08:42:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.024+0000 7f6db2270640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6d940a3bb0 0x7f6d940a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.024+0000 7f6db2270640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dac101b50 0x7f6dac1a18c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.024+0000 7f6db2270640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dac1011b0 0x7f6dac19a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.024+0000 7f6db2270640 1 --2- 172.21.15.7:0/2509138192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6dac100850 0x7f6dac19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.024+0000 7f6db2270640 1 -- 172.21.15.7:0/2509138192 >> 172.21.15.7:0/2509138192 conn(0x7f6dac0fc3f0 msgr2=0x7f6dac0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.024+0000 7f6db2270640 1 -- 172.21.15.7:0/2509138192 shutdown_connections 2024-04-21T08:42:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.024+0000 7f6db2270640 1 -- 172.21.15.7:0/2509138192 wait complete. 2024-04-21T08:42:23.034 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:22 smithi007 ceph-mon[28477]: pgmap v294: 129 pgs: 14 creating+peering, 87 active+clean, 28 unknown; 578 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-21T08:42:23.034 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:22 smithi007 ceph-mon[28477]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:42:23.034 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:22 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3819543900' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:42:23.035 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:42:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:22 smithi016 ceph-mon[31259]: pgmap v294: 129 pgs: 14 creating+peering, 87 active+clean, 28 unknown; 578 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-21T08:42:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:22 smithi016 ceph-mon[31259]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:42:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:22 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3819543900' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:42:23.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:22 smithi077 ceph-mon[31615]: pgmap v294: 129 pgs: 14 creating+peering, 87 active+clean, 28 unknown; 578 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-21T08:42:23.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:22 smithi077 ceph-mon[31615]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:42:23.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:22 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3819543900' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:42:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.137+0000 7f7a16723640 1 -- 172.21.15.7:0/1259209770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10101180 msgr2=0x7f7a10101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.137+0000 7f7a16723640 1 --2- 172.21.15.7:0/1259209770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10101180 0x7f7a10101600 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009900 tx=0x7f7a0402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.137+0000 7f7a16723640 1 -- 172.21.15.7:0/1259209770 shutdown_connections 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.137+0000 7f7a16723640 1 --2- 172.21.15.7:0/1259209770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a10101b40 0x7f7a101093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.137+0000 7f7a16723640 1 --2- 172.21.15.7:0/1259209770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10101180 0x7f7a10101600 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.137+0000 7f7a16723640 1 --2- 172.21.15.7:0/1259209770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a10102450 0x7f7a10102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.137+0000 7f7a16723640 1 -- 172.21.15.7:0/1259209770 >> 172.21.15.7:0/1259209770 conn(0x7f7a100fbbb0 msgr2=0x7f7a100fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.137+0000 7f7a16723640 1 -- 172.21.15.7:0/1259209770 shutdown_connections 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.137+0000 7f7a16723640 1 -- 172.21.15.7:0/1259209770 wait complete. 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.137+0000 7f7a16723640 1 Processor -- start 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.137+0000 7f7a16723640 1 -- start start 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a16723640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10101180 0x7f7a10100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a16723640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a10101b40 0x7f7a1019cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a16723640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a10102450 0x7f7a1019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a16723640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1010c9f0 con 0x7f7a10101180 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a16723640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1010c870 con 0x7f7a10102450 2024-04-21T08:42:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a16723640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1010cb70 con 0x7f7a10101b40 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a0ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10101180 0x7f7a10100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a0ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10101180 0x7f7a10100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37378/0 (socket says 172.21.15.7:37378) 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a0ffff640 1 -- 172.21.15.7:0/3040687526 learned_addr learned my addr 172.21.15.7:0/3040687526 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a0f7fe640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a10101b40 0x7f7a1019cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a0ffff640 1 -- 172.21.15.7:0/3040687526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a10101b40 msgr2=0x7f7a1019cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a0ffff640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a10101b40 0x7f7a1019cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a0ffff640 1 -- 172.21.15.7:0/3040687526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a10102450 msgr2=0x7f7a1019f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a14cc0640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a10102450 0x7f7a1019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a0ffff640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a10102450 0x7f7a1019f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a0ffff640 1 -- 172.21.15.7:0/3040687526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1019fe00 con 0x7f7a10101180 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a0ffff640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10101180 0x7f7a10100c00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000e9c0 tx=0x7f7a0000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.138+0000 7f7a0d7fa640 1 -- 172.21.15.7:0/3040687526 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0000cd80 con 0x7f7a10101180 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.139+0000 7f7a0d7fa640 1 -- 172.21.15.7:0/3040687526 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a000035f0 con 0x7f7a10101180 2024-04-21T08:42:23.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.139+0000 7f7a16723640 1 -- 172.21.15.7:0/3040687526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a101a00f0 con 0x7f7a10101180 2024-04-21T08:42:23.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.139+0000 7f7a0d7fa640 1 -- 172.21.15.7:0/3040687526 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a00010690 con 0x7f7a10101180 2024-04-21T08:42:23.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.139+0000 7f7a16723640 1 -- 172.21.15.7:0/3040687526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a101a05c0 con 0x7f7a10101180 2024-04-21T08:42:23.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.140+0000 7f7a16723640 1 -- 172.21.15.7:0/3040687526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a10108340 con 0x7f7a10101180 2024-04-21T08:42:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.141+0000 7f7a0d7fa640 1 -- 172.21.15.7:0/3040687526 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f7a00004000 con 0x7f7a10101180 2024-04-21T08:42:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.142+0000 7f7a0d7fa640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79f80a3bb0 0x7f79f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.142+0000 7f7a0f7fe640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79f80a3bb0 0x7f79f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.142+0000 7f7a0d7fa640 1 -- 172.21.15.7:0/3040687526 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f7a000d0c40 con 0x7f7a10101180 2024-04-21T08:42:23.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.142+0000 7f7a0f7fe640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79f80a3bb0 0x7f79f80a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009230 tx=0x7f7a04005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:23.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.146+0000 7f7a0d7fa640 1 -- 172.21.15.7:0/3040687526 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a0009d380 con 0x7f7a10101180 2024-04-21T08:42:23.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.337+0000 7f7a16723640 1 -- 172.21.15.7:0/3040687526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7a10062b50 con 0x7f7a10101180 2024-04-21T08:42:23.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.338+0000 7f7a0d7fa640 1 -- 172.21.15.7:0/3040687526 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23961 (secure 0 0 0) 0x7f7a0009cc80 con 0x7f7a10101180 2024-04-21T08:42:23.339 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:23.339 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":90,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:42:22.023111+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:23.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 -- 172.21.15.7:0/3040687526 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79f80a3bb0 msgr2=0x7f79f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79f80a3bb0 0x7f79f80a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009230 tx=0x7f7a04005fd0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 -- 172.21.15.7:0/3040687526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10101180 msgr2=0x7f7a10100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10101180 0x7f7a10100c00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000e9c0 tx=0x7f7a0000ee80 comp rx=0 tx=0).stop 2024-04-21T08:42:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 -- 172.21.15.7:0/3040687526 shutdown_connections 2024-04-21T08:42:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79f80a3bb0 0x7f79f80a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a10102450 0x7f7a1019f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a10101b40 0x7f7a1019cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 --2- 172.21.15.7:0/3040687526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a10101180 0x7f7a10100c00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 -- 172.21.15.7:0/3040687526 >> 172.21.15.7:0/3040687526 conn(0x7f7a100fbbb0 msgr2=0x7f7a10105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 -- 172.21.15.7:0/3040687526 shutdown_connections 2024-04-21T08:42:23.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.340+0000 7f7a16723640 1 -- 172.21.15.7:0/3040687526 wait complete. 2024-04-21T08:42:23.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:42:23.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.456+0000 7fc4c21d5640 1 -- 172.21.15.7:0/2170746936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4bc107d00 msgr2=0x7fc4bc10a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.456+0000 7fc4c21d5640 1 --2- 172.21.15.7:0/2170746936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4bc107d00 0x7fc4bc10a0f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4009ca0 tx=0x7fc4b402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.456+0000 7fc4c21d5640 1 -- 172.21.15.7:0/2170746936 shutdown_connections 2024-04-21T08:42:23.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.456+0000 7fc4c21d5640 1 --2- 172.21.15.7:0/2170746936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4bc107d00 0x7fc4bc10a0f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.456+0000 7fc4c21d5640 1 --2- 172.21.15.7:0/2170746936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc1053d0 0x7fc4bc1077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.456+0000 7fc4c21d5640 1 --2- 172.21.15.7:0/2170746936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4bc102aa0 0x7fc4bc104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.456+0000 7fc4c21d5640 1 -- 172.21.15.7:0/2170746936 >> 172.21.15.7:0/2170746936 conn(0x7fc4bc0fc3f0 msgr2=0x7fc4bc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:23.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.456+0000 7fc4c21d5640 1 -- 172.21.15.7:0/2170746936 shutdown_connections 2024-04-21T08:42:23.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.456+0000 7fc4c21d5640 1 -- 172.21.15.7:0/2170746936 wait complete. 2024-04-21T08:42:23.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4c21d5640 1 Processor -- start 2024-04-21T08:42:23.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4c21d5640 1 -- start start 2024-04-21T08:42:23.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4c21d5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4bc102aa0 0x7fc4bc198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4c21d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc1053d0 0x7fc4bc198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4c21d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4bc107d00 0x7fc4bc19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4c21d5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc10dae0 con 0x7fc4bc102aa0 2024-04-21T08:42:23.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4c21d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc10d960 con 0x7fc4bc1053d0 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4c21d5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc10dc60 con 0x7fc4bc107d00 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4bc102aa0 0x7fc4bc198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4bc102aa0 0x7fc4bc198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37392/0 (socket says 172.21.15.7:37392) 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4bb7fe640 1 -- 172.21.15.7:0/1039983912 learned_addr learned my addr 172.21.15.7:0/1039983912 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4bbfff640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4bc107d00 0x7fc4bc19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.457+0000 7fc4baffd640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc1053d0 0x7fc4bc198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.458+0000 7fc4bbfff640 1 -- 172.21.15.7:0/1039983912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc1053d0 msgr2=0x7fc4bc198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.458+0000 7fc4bbfff640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc1053d0 0x7fc4bc198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.458+0000 7fc4bbfff640 1 -- 172.21.15.7:0/1039983912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4bc102aa0 msgr2=0x7fc4bc198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.458+0000 7fc4bbfff640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4bc102aa0 0x7fc4bc198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.458+0000 7fc4bbfff640 1 -- 172.21.15.7:0/1039983912 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4bc19fd50 con 0x7fc4bc107d00 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.458+0000 7fc4bbfff640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4bc107d00 0x7fc4bc19f6e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc4b402c700 tx=0x7fc4b4034760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:23.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.458+0000 7fc4b8ff9640 1 -- 172.21.15.7:0/1039983912 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4039040 con 0x7fc4bc107d00 2024-04-21T08:42:23.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.458+0000 7fc4b8ff9640 1 -- 172.21.15.7:0/1039983912 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4b40325c0 con 0x7fc4bc107d00 2024-04-21T08:42:23.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.458+0000 7fc4c21d5640 1 -- 172.21.15.7:0/1039983912 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4bc19ffe0 con 0x7fc4bc107d00 2024-04-21T08:42:23.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.458+0000 7fc4b8ff9640 1 -- 172.21.15.7:0/1039983912 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40042a0 con 0x7fc4bc107d00 2024-04-21T08:42:23.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.458+0000 7fc4c21d5640 1 -- 172.21.15.7:0/1039983912 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4bc1a0440 con 0x7fc4bc107d00 2024-04-21T08:42:23.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.459+0000 7fc4c21d5640 1 -- 172.21.15.7:0/1039983912 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4bc0fdee0 con 0x7fc4bc107d00 2024-04-21T08:42:23.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.460+0000 7fc4b8ff9640 1 -- 172.21.15.7:0/1039983912 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc4b4046030 con 0x7fc4bc107d00 2024-04-21T08:42:23.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.461+0000 7fc4b8ff9640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4a40a3bb0 0x7fc4a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.461+0000 7fc4b8ff9640 1 -- 172.21.15.7:0/1039983912 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc4b40f1f60 con 0x7fc4bc107d00 2024-04-21T08:42:23.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.461+0000 7fc4bb7fe640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4a40a3bb0 0x7fc4a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.462+0000 7fc4bb7fe640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4a40a3bb0 0x7fc4a40a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00bca0 tx=0x7fc4ac00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:23.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.466+0000 7fc4b8ff9640 1 -- 172.21.15.7:0/1039983912 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4b404d060 con 0x7fc4bc107d00 2024-04-21T08:42:23.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.655+0000 7fc4c21d5640 1 -- 172.21.15.7:0/1039983912 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc4bc062b50 con 0x7fc4bc107d00 2024-04-21T08:42:23.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.657+0000 7fc4b8ff9640 1 -- 172.21.15.7:0/1039983912 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23961 (secure 0 0 0) 0x7fc4b40be790 con 0x7fc4bc107d00 2024-04-21T08:42:23.658 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:23.658 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":90,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:42:22.023111+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 -- 172.21.15.7:0/1039983912 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4a40a3bb0 msgr2=0x7fc4a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4a40a3bb0 0x7fc4a40a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00bca0 tx=0x7fc4ac00b410 comp rx=0 tx=0).stop 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 -- 172.21.15.7:0/1039983912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4bc107d00 msgr2=0x7fc4bc19f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4bc107d00 0x7fc4bc19f6e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc4b402c700 tx=0x7fc4b4034760 comp rx=0 tx=0).stop 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 -- 172.21.15.7:0/1039983912 shutdown_connections 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4a40a3bb0 0x7fc4a40a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4bc107d00 0x7fc4bc19f6e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4bc1053d0 0x7fc4bc198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 --2- 172.21.15.7:0/1039983912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4bc102aa0 0x7fc4bc198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 -- 172.21.15.7:0/1039983912 >> 172.21.15.7:0/1039983912 conn(0x7fc4bc0fc3f0 msgr2=0x7fc4bc1033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 -- 172.21.15.7:0/1039983912 shutdown_connections 2024-04-21T08:42:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.659+0000 7fc4c21d5640 1 -- 172.21.15.7:0/1039983912 wait complete. 2024-04-21T08:42:23.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.786+0000 7f42a7567640 1 -- 172.21.15.7:0/1338571683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42a0103180 msgr2=0x7f42a0103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.786+0000 7f42a7567640 1 --2- 172.21.15.7:0/1338571683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42a0103180 0x7f42a0103600 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4290009900 tx=0x7f429002bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 -- 172.21.15.7:0/1338571683 shutdown_connections 2024-04-21T08:42:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 --2- 172.21.15.7:0/1338571683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0103b40 0x7f42a010a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 --2- 172.21.15.7:0/1338571683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42a0103180 0x7f42a0103600 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 --2- 172.21.15.7:0/1338571683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a0101f90 0x7f42a01023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 -- 172.21.15.7:0/1338571683 >> 172.21.15.7:0/1338571683 conn(0x7f42a00fd520 msgr2=0x7f42a00ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 -- 172.21.15.7:0/1338571683 shutdown_connections 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 -- 172.21.15.7:0/1338571683 wait complete. 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 Processor -- start 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 -- start start 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42a0101f90 0x7f42a0198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a0103180 0x7f42a01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0103b40 0x7f42a019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f42a010dad0 con 0x7f42a0101f90 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42a010d950 con 0x7f42a0103b40 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a7567640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f42a010dc50 con 0x7f42a0103180 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a5303640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42a0101f90 0x7f42a0198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.787+0000 7f42a5303640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42a0101f90 0x7f42a0198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37410/0 (socket says 172.21.15.7:37410) 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42a5303640 1 -- 172.21.15.7:0/1079549357 learned_addr learned my addr 172.21.15.7:0/1079549357 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42a5303640 1 -- 172.21.15.7:0/1079549357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a0103180 msgr2=0x7f42a01986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42a4b02640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a0103180 0x7f42a01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42a5303640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a0103180 0x7f42a01986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42a5303640 1 -- 172.21.15.7:0/1079549357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0103b40 msgr2=0x7f42a019f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42a5b04640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0103b40 0x7f42a019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42a5303640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0103b40 0x7f42a019f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42a5303640 1 -- 172.21.15.7:0/1079549357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a019fda0 con 0x7f42a0101f90 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42a5303640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42a0101f90 0x7f42a0198180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f429c00e9c0 tx=0x7f429c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42967fc640 1 -- 172.21.15.7:0/1079549357 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429c00cd80 con 0x7f42a0101f90 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42a7567640 1 -- 172.21.15.7:0/1079549357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a01a0090 con 0x7f42a0101f90 2024-04-21T08:42:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42967fc640 1 -- 172.21.15.7:0/1079549357 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f429c0035f0 con 0x7f42a0101f90 2024-04-21T08:42:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42967fc640 1 -- 172.21.15.7:0/1079549357 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429c010690 con 0x7f42a0101f90 2024-04-21T08:42:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.788+0000 7f42a7567640 1 -- 172.21.15.7:0/1079549357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a01a0590 con 0x7f42a0101f90 2024-04-21T08:42:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.789+0000 7f42a7567640 1 -- 172.21.15.7:0/1079549357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a0109330 con 0x7f42a0101f90 2024-04-21T08:42:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.790+0000 7f42967fc640 1 -- 172.21.15.7:0/1079549357 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f429c004000 con 0x7f42a0101f90 2024-04-21T08:42:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.791+0000 7f42967fc640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f428c0a3bb0 0x7f428c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.791+0000 7f42967fc640 1 -- 172.21.15.7:0/1079549357 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f429c0d0ca0 con 0x7f42a0101f90 2024-04-21T08:42:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.791+0000 7f42a4b02640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f428c0a3bb0 0x7f428c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.791+0000 7f42a4b02640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f428c0a3bb0 0x7f428c0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4290036040 tx=0x7f4290000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:23.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:23.795+0000 7f42967fc640 1 -- 172.21.15.7:0/1079549357 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f429c09d440 con 0x7f42a0101f90 2024-04-21T08:42:24.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.020+0000 7f42a7567640 1 -- 172.21.15.7:0/1079549357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42a002d5c0 con 0x7f42a0101f90 2024-04-21T08:42:24.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.020+0000 7f42967fc640 1 -- 172.21.15.7:0/1079549357 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11773 (secure 0 0 0) 0x7f429c09cd40 con 0x7f42a0101f90 2024-04-21T08:42:24.021 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:24.021 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:22.023279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:42:24.021 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 17 2024-04-21T08:42:24.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.022+0000 7f42a7567640 1 -- 172.21.15.7:0/1079549357 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f428c0a3bb0 msgr2=0x7f428c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.022+0000 7f42a7567640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f428c0a3bb0 0x7f428c0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4290036040 tx=0x7f4290000bc0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.022+0000 7f42a7567640 1 -- 172.21.15.7:0/1079549357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42a0101f90 msgr2=0x7f42a0198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.022+0000 7f42a7567640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42a0101f90 0x7f42a0198180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f429c00e9c0 tx=0x7f429c00ee80 comp rx=0 tx=0).stop 2024-04-21T08:42:24.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.022+0000 7f42a7567640 1 -- 172.21.15.7:0/1079549357 shutdown_connections 2024-04-21T08:42:24.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.022+0000 7f42a7567640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f428c0a3bb0 0x7f428c0a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.022+0000 7f42a7567640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42a0103b40 0x7f42a019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.022+0000 7f42a7567640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42a0103180 0x7f42a01986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.022+0000 7f42a7567640 1 --2- 172.21.15.7:0/1079549357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42a0101f90 0x7f42a0198180 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.022+0000 7f42a7567640 1 -- 172.21.15.7:0/1079549357 >> 172.21.15.7:0/1079549357 conn(0x7f42a00fd520 msgr2=0x7f42a0106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:24.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.023+0000 7f42a7567640 1 -- 172.21.15.7:0/1079549357 shutdown_connections 2024-04-21T08:42:24.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.023+0000 7f42a7567640 1 -- 172.21.15.7:0/1079549357 wait complete. 2024-04-21T08:42:24.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:23 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3819543900' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:42:24.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:23 smithi007 ceph-mon[28477]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:42:24.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:23 smithi007 ceph-mon[28477]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:24.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:23 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2509138192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:24.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:23 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3040687526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:24.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:23 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1039983912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:24.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:42:24.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:23 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3819543900' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:42:24.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:23 smithi016 ceph-mon[31259]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:42:24.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:23 smithi016 ceph-mon[31259]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:24.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:23 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2509138192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:24.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:23 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3040687526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:24.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:23 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1039983912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:24.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:23 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3819543900' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:42:24.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:23 smithi077 ceph-mon[31615]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:42:24.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:23 smithi077 ceph-mon[31615]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:24.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:23 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2509138192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:24.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:23 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3040687526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:24.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:23 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1039983912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:24.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.135+0000 7f9df778f640 1 -- 172.21.15.7:0/2240666988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df0103b50 msgr2=0x7f9df010a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.135+0000 7f9df778f640 1 --2- 172.21.15.7:0/2240666988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df0103b50 0x7f9df010a3e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009ca0 tx=0x7f9de802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.135+0000 7f9df778f640 1 -- 172.21.15.7:0/2240666988 shutdown_connections 2024-04-21T08:42:24.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.135+0000 7f9df778f640 1 --2- 172.21.15.7:0/2240666988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df0103b50 0x7f9df010a3e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.135+0000 7f9df778f640 1 --2- 172.21.15.7:0/2240666988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103190 0x7f9df0103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.135+0000 7f9df778f640 1 --2- 172.21.15.7:0/2240666988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df0101fa0 0x7f9df01023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.135+0000 7f9df778f640 1 -- 172.21.15.7:0/2240666988 >> 172.21.15.7:0/2240666988 conn(0x7f9df00fd570 msgr2=0x7f9df00ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:24.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.135+0000 7f9df778f640 1 -- 172.21.15.7:0/2240666988 shutdown_connections 2024-04-21T08:42:24.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.135+0000 7f9df778f640 1 -- 172.21.15.7:0/2240666988 wait complete. 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df778f640 1 Processor -- start 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df778f640 1 -- start start 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df778f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df0101fa0 0x7f9df0198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df778f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df0103190 0x7f9df0198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df778f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103b50 0x7f9df019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df778f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9df010dae0 con 0x7f9df0103190 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df778f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9df010d960 con 0x7f9df0103b50 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df778f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9df010dc60 con 0x7f9df0101fa0 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df4d2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df0103190 0x7f9df0198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df4d2a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df0103190 0x7f9df0198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37424/0 (socket says 172.21.15.7:37424) 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df4d2a640 1 -- 172.21.15.7:0/2384302551 learned_addr learned my addr 172.21.15.7:0/2384302551 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:24.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df552b640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df0101fa0 0x7f9df0198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df5d2c640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103b50 0x7f9df019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df4d2a640 1 -- 172.21.15.7:0/2384302551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df0101fa0 msgr2=0x7f9df0198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df4d2a640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df0101fa0 0x7f9df0198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df4d2a640 1 -- 172.21.15.7:0/2384302551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103b50 msgr2=0x7f9df019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df4d2a640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103b50 0x7f9df019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df4d2a640 1 -- 172.21.15.7:0/2384302551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df019fd20 con 0x7f9df0103190 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df5d2c640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103b50 0x7f9df019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df552b640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df0101fa0 0x7f9df0198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.136+0000 7f9df4d2a640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df0103190 0x7f9df0198640 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9de000e9f0 tx=0x7f9de000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.137+0000 7f9de67fc640 1 -- 172.21.15.7:0/2384302551 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de000cd80 con 0x7f9df0103190 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.137+0000 7f9de67fc640 1 -- 172.21.15.7:0/2384302551 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9de00108e0 con 0x7f9df0103190 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.137+0000 7f9df778f640 1 -- 172.21.15.7:0/2384302551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df019ffb0 con 0x7f9df0103190 2024-04-21T08:42:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.137+0000 7f9de67fc640 1 -- 172.21.15.7:0/2384302551 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0018980 con 0x7f9df0103190 2024-04-21T08:42:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.137+0000 7f9df778f640 1 -- 172.21.15.7:0/2384302551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df01a04d0 con 0x7f9df0103190 2024-04-21T08:42:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.138+0000 7f9df778f640 1 -- 172.21.15.7:0/2384302551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df0109260 con 0x7f9df0103190 2024-04-21T08:42:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.139+0000 7f9de67fc640 1 -- 172.21.15.7:0/2384302551 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f9de0010430 con 0x7f9df0103190 2024-04-21T08:42:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.139+0000 7f9de67fc640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9ddc0a3bb0 0x7f9ddc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.139+0000 7f9df552b640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9ddc0a3bb0 0x7f9ddc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.139+0000 7f9de67fc640 1 -- 172.21.15.7:0/2384302551 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f9de00d0f20 con 0x7f9df0103190 2024-04-21T08:42:24.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.140+0000 7f9df552b640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9ddc0a3bb0 0x7f9ddc0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0058c0 tx=0x7f9dec005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:24.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.143+0000 7f9de67fc640 1 -- 172.21.15.7:0/2384302551 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9de009d6c0 con 0x7f9df0103190 2024-04-21T08:42:24.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.329+0000 7f9df778f640 1 -- 172.21.15.7:0/2384302551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9df002d810 con 0x7f9df0103190 2024-04-21T08:42:24.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.331+0000 7f9de67fc640 1 -- 172.21.15.7:0/2384302551 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23961 (secure 0 0 0) 0x7f9de009cfc0 con 0x7f9df0103190 2024-04-21T08:42:24.332 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:24.332 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":90,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:42:22.023111+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:24.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 -- 172.21.15.7:0/2384302551 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9ddc0a3bb0 msgr2=0x7f9ddc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9ddc0a3bb0 0x7f9ddc0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9dec0058c0 tx=0x7f9dec005850 comp rx=0 tx=0).stop 2024-04-21T08:42:24.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 -- 172.21.15.7:0/2384302551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df0103190 msgr2=0x7f9df0198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df0103190 0x7f9df0198640 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9de000e9f0 tx=0x7f9de000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 -- 172.21.15.7:0/2384302551 shutdown_connections 2024-04-21T08:42:24.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9ddc0a3bb0 0x7f9ddc0a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103b50 0x7f9df019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df0103190 0x7f9df0198640 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 --2- 172.21.15.7:0/2384302551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df0101fa0 0x7f9df0198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 -- 172.21.15.7:0/2384302551 >> 172.21.15.7:0/2384302551 conn(0x7f9df00fd570 msgr2=0x7f9df0106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:24.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 -- 172.21.15.7:0/2384302551 shutdown_connections 2024-04-21T08:42:24.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.333+0000 7f9df778f640 1 -- 172.21.15.7:0/2384302551 wait complete. 2024-04-21T08:42:24.345 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-21T08:42:16.996517+0000', 'modified': '2024-04-21T08:42:22.023279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24533}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24533': {'gid': 24533, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6839/1703654929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 1703654929}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 1703654929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24533, 'qdb_cluster': [24533]} 2024-04-21T08:42:24.345 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T08:42:24.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:42:24.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.448+0000 7f8274d7e640 1 -- 172.21.15.7:0/4193517937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270102a20 msgr2=0x7f8270104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.448+0000 7f8274d7e640 1 --2- 172.21.15.7:0/4193517937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270102a20 0x7f8270104e10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8260009ca0 tx=0x7f826002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.448+0000 7f8274d7e640 1 -- 172.21.15.7:0/4193517937 shutdown_connections 2024-04-21T08:42:24.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.448+0000 7f8274d7e640 1 --2- 172.21.15.7:0/4193517937 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8270107c80 0x7f827010a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.448+0000 7f8274d7e640 1 --2- 172.21.15.7:0/4193517937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8270105350 0x7f8270107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.448+0000 7f8274d7e640 1 --2- 172.21.15.7:0/4193517937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270102a20 0x7f8270104e10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.448+0000 7f8274d7e640 1 -- 172.21.15.7:0/4193517937 >> 172.21.15.7:0/4193517937 conn(0x7f82700fc430 msgr2=0x7f82700fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.449+0000 7f8274d7e640 1 -- 172.21.15.7:0/4193517937 shutdown_connections 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.449+0000 7f8274d7e640 1 -- 172.21.15.7:0/4193517937 wait complete. 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.449+0000 7f8274d7e640 1 Processor -- start 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.449+0000 7f8274d7e640 1 -- start start 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.449+0000 7f8274d7e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270102a20 0x7f82701981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.449+0000 7f8274d7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8270105350 0x7f8270198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.449+0000 7f8274d7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8270107c80 0x7f827019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.449+0000 7f8274d7e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f827010dae0 con 0x7f8270102a20 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.449+0000 7f8274d7e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f827010d960 con 0x7f8270107c80 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.449+0000 7f8274d7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f827010dc60 con 0x7f8270105350 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.449+0000 7f826e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270102a20 0x7f82701981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f826e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270102a20 0x7f82701981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37440/0 (socket says 172.21.15.7:37440) 2024-04-21T08:42:24.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f826e59c640 1 -- 172.21.15.7:0/4021578611 learned_addr learned my addr 172.21.15.7:0/4021578611 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f826ed9d640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8270107c80 0x7f827019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f826dd9b640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8270105350 0x7f8270198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f826e59c640 1 -- 172.21.15.7:0/4021578611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8270105350 msgr2=0x7f8270198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f826e59c640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8270105350 0x7f8270198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f826e59c640 1 -- 172.21.15.7:0/4021578611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8270107c80 msgr2=0x7f827019f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f826e59c640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8270107c80 0x7f827019f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f826e59c640 1 -- 172.21.15.7:0/4021578611 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827019fe10 con 0x7f8270102a20 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f826ed9d640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8270107c80 0x7f827019f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f826e59c640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270102a20 0x7f82701981f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8260032380 tx=0x7f8260034650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f825f7fe640 1 -- 172.21.15.7:0/4021578611 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8260041020 con 0x7f8270102a20 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f825f7fe640 1 -- 172.21.15.7:0/4021578611 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8260034d60 con 0x7f8270102a20 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f8274d7e640 1 -- 172.21.15.7:0/4021578611 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82701a00a0 con 0x7f8270102a20 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f825f7fe640 1 -- 172.21.15.7:0/4021578611 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8260039040 con 0x7f8270102a20 2024-04-21T08:42:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.450+0000 7f8274d7e640 1 -- 172.21.15.7:0/4021578611 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82701a05e0 con 0x7f8270102a20 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.452+0000 7f825f7fe640 1 -- 172.21.15.7:0/4021578611 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f8260044020 con 0x7f8270102a20 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.453+0000 7f827796a740 1 Processor -- start 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.453+0000 7f827796a740 1 -- start start 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.453+0000 7f827796a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaeb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.453+0000 7f827796a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 0x5557ebaebcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.453+0000 7f827796a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebaf2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.453+0000 7f827796a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5557eba4fb10 con 0x5557ebaeae40 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.453+0000 7f827796a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5557eb9ed240 con 0x5557ebaeb830 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.453+0000 7f827796a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5557eba4bad0 con 0x5557ebaec2d0 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.453+0000 7f826ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaeb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.453+0000 7f826ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaeb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37456/0 (socket says 172.21.15.7:37456) 2024-04-21T08:42:24.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.453+0000 7f826ed9d640 1 -- 172.21.15.7:0/2599056872 learned_addr learned my addr 172.21.15.7:0/2599056872 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f825f7fe640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82580a3c30 0x7f82580a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f826e59c640 1 --2- 172.21.15.7:0/2599056872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 0x5557ebaebcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f826ed9d640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82580a3c30 0x7f82580a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f826dd9b640 1 --2- 172.21.15.7:0/2599056872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebaf2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f826ed9d640 1 -- 172.21.15.7:0/2599056872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 msgr2=0x5557ebaf2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f826ed9d640 1 --2- 172.21.15.7:0/2599056872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebaf2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f826ed9d640 1 -- 172.21.15.7:0/2599056872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 msgr2=0x5557ebaebcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f826ed9d640 1 --2- 172.21.15.7:0/2599056872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 0x5557ebaebcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f826ed9d640 1 -- 172.21.15.7:0/2599056872 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557eb7622d0 con 0x5557ebaeae40 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f826dd9b640 1 --2- 172.21.15.7:0/2599056872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebaf2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f826e59c640 1 --2- 172.21.15.7:0/2599056872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 0x5557ebaebcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f825f7fe640 1 -- 172.21.15.7:0/4021578611 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f82600f2530 con 0x7f8270102a20 2024-04-21T08:42:24.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f826ed9d640 1 --2- 172.21.15.7:0/2599056872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaeb260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f82680137c0 tx=0x7f8268013c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373168e83cf8089f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f825cff9640 1 -- 172.21.15.7:0/2599056872 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826801d7c0 con 0x5557ebaeae40 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f825cff9640 1 -- 172.21.15.7:0/2599056872 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8268022070 con 0x5557ebaeae40 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.454+0000 7f825cff9640 1 -- 172.21.15.7:0/2599056872 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826801c8d0 con 0x5557ebaeae40 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f827796a740 1 -- 172.21.15.7:0/2599056872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 msgr2=0x5557ebaeb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f826ed9d640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82580a3c30 0x7f82580a60f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8268021000 tx=0x7f826801f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f827796a740 1 --2- 172.21.15.7:0/2599056872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaeb260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f82680137c0 tx=0x7f8268013c80 comp rx=0 tx=0).stop 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f827796a740 1 -- 172.21.15.7:0/2599056872 shutdown_connections 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f827796a740 1 --2- 172.21.15.7:0/2599056872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebaf2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f827796a740 1 --2- 172.21.15.7:0/2599056872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 0x5557ebaebcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f827796a740 1 --2- 172.21.15.7:0/2599056872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaeb260 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f827796a740 1 -- 172.21.15.7:0/2599056872 >> 172.21.15.7:0/2599056872 conn(0x5557ebae6f40 msgr2=0x5557ebae5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:24.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f827796a740 1 -- 172.21.15.7:0/2599056872 shutdown_connections 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f827796a740 1 -- 172.21.15.7:0/2599056872 wait complete. 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f827796a740 1 -- start start 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.455+0000 7f827796a740 1 Processor -- start 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f827796a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeae40 0x5557ebaf4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f827796a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebaf53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f827796a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebafc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f827796a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5557ebaf34b0 con 0x5557ebaeb830 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f827796a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5557eb9d8190 con 0x5557ebaeae40 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f827796a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5557ebaf3630 con 0x5557ebaec2d0 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebaf53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f827796a740 10 client.0 _ll_register_callbacks cb 0x5557eb9d38b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebaf53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37472/0 (socket says 172.21.15.7:37472) 2024-04-21T08:42:24.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826dd9b640 1 -- 172.21.15.7:0/1844939044 learned_addr learned my addr 172.21.15.7:0/1844939044 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826e59c640 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeae40 0x5557ebaf4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826ed9d640 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebafc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826dd9b640 1 -- 172.21.15.7:0/1844939044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 msgr2=0x5557ebafc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826dd9b640 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebafc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826dd9b640 1 -- 172.21.15.7:0/1844939044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeae40 msgr2=0x5557ebaf4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826dd9b640 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeae40 0x5557ebaf4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826dd9b640 1 -- 172.21.15.7:0/1844939044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557eb82c6b0 con 0x5557ebaeb830 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826ed9d640 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebafc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f826dd9b640 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebaf53f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8264011690 tx=0x7f8264011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe2dc59ae57dd81 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f8255ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f8255ffb640 1 -- 172.21.15.7:0/1844939044 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8264004220 con 0x5557ebaeb830 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f827796a740 1 -- 172.21.15.7:0/1844939044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8268022070 con 0x5557ebaeb830 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.456+0000 7f8255ffb640 1 -- 172.21.15.7:0/1844939044 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f82640043c0 con 0x5557ebaeb830 2024-04-21T08:42:24.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f8255ffb640 1 -- 172.21.15.7:0/1844939044 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f8264012780 con 0x5557ebaeb830 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f827796a740 10 client.14778 fetch_fsmap learned FSMap version 17 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f827796a740 1 -- 172.21.15.7:0/1844939044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5557ebaff7e0 con 0x5557ebaeb830 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f8255ffb640 1 -- 172.21.15.7:0/1844939044 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8268022070 con 0x5557ebaeb830 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f8255ffb640 1 -- 172.21.15.7:0/1844939044 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f8264003c30 con 0x5557ebaeb830 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f827796a740 10 client.14778 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f827796a740 10 client.14778 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f827796a740 20 client.14778 populate_metadata read hostname 'smithi007' 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f827796a740 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5557ebb01ee0 0x5557ebb03350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f827796a740 4 client.14778 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f827796a740 1 -- 172.21.15.7:0/1844939044 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557eba5d530 con 0x5557ebb01ee0 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.457+0000 7f826e59c640 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5557ebb01ee0 0x5557ebb03350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.458+0000 7f826e59c640 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5557ebb01ee0 0x5557ebb03350 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afd10c0fcb9c6e29 server_cookie=5978c5e2ab424 in_seq=0 out_seq=0 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.458+0000 7f8255ffb640 10 client.14778 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:42:24.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.458+0000 7f8255ffb640 1 -- 172.21.15.7:0/1844939044 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5557eba5d530 con 0x5557ebb01ee0 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.458+0000 7f8255ffb640 10 client.14778 handle_command_reply: tid=0 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.458+0000 7f827796a740 1 client.14778 shutdown 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f827796a740 1 -- 172.21.15.7:0/1844939044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 msgr2=0x5557ebaf53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f827796a740 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebaf53f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8264011690 tx=0x7f8264011a90 comp rx=0 tx=0).stop 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f827796a740 1 -- 172.21.15.7:0/1844939044 shutdown_connections 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f827796a740 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5557ebb01ee0 0x5557ebb03350 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f827796a740 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebafc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f827796a740 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebaf53f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f8255ffb640 0 client.14778 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f827796a740 1 --2- 172.21.15.7:0/1844939044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeae40 0x5557ebaf4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f827796a740 1 -- 172.21.15.7:0/1844939044 >> 172.21.15.7:0/1844939044 conn(0x5557ebae6f40 msgr2=0x5557ebae4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:24.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f827796a740 1 -- 172.21.15.7:0/1844939044 shutdown_connections 2024-04-21T08:42:24.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f827796a740 1 -- 172.21.15.7:0/1844939044 wait complete. 2024-04-21T08:42:24.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.459+0000 7f827796a740 20 client.14778 trim_cache size 0 max 16384 2024-04-21T08:42:24.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f827796a740 1 Processor -- start 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f827796a740 1 -- start start 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f827796a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeae40 0x5557ebaeee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f827796a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebafc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f827796a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebaef7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f827796a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5557ebb01690 con 0x5557ebaeb830 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f827796a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5557ebb01d60 con 0x5557ebaeae40 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f827796a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5557ebb02430 con 0x5557ebaec2d0 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f826dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebafc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f826dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebafc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37484/0 (socket says 172.21.15.7:37484) 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f826dd9b640 1 -- 172.21.15.7:0/2292281616 learned_addr learned my addr 172.21.15.7:0/2292281616 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f826dd9b640 1 -- 172.21.15.7:0/2292281616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 msgr2=0x5557ebaef7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f826dd9b640 1 --2- 172.21.15.7:0/2292281616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebaef7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f826dd9b640 1 -- 172.21.15.7:0/2292281616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeae40 msgr2=0x5557ebaeee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f826dd9b640 1 --2- 172.21.15.7:0/2292281616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeae40 0x5557ebaeee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.477+0000 7f826dd9b640 1 -- 172.21.15.7:0/2292281616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557ebaf42c0 con 0x5557ebaeb830 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f826dd9b640 1 --2- 172.21.15.7:0/2292281616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebafc3f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8264009f30 tx=0x7f8264009420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce92af67462c0ecd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f8257fff640 1 -- 172.21.15.7:0/2292281616 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8264004220 con 0x5557ebaeb830 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f8257fff640 1 -- 172.21.15.7:0/2292281616 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8264012dd0 con 0x5557ebaeb830 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f8257fff640 1 -- 172.21.15.7:0/2292281616 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826400f750 con 0x5557ebaeb830 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f827796a740 1 -- 172.21.15.7:0/2292281616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 msgr2=0x5557ebafc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f827796a740 1 --2- 172.21.15.7:0/2292281616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebafc3f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8264009f30 tx=0x7f8264009420 comp rx=0 tx=0).stop 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f827796a740 1 -- 172.21.15.7:0/2292281616 shutdown_connections 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f827796a740 1 --2- 172.21.15.7:0/2292281616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebaef7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f827796a740 1 --2- 172.21.15.7:0/2292281616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeb830 0x5557ebafc3f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f827796a740 1 --2- 172.21.15.7:0/2292281616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeae40 0x5557ebaeee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f827796a740 1 -- 172.21.15.7:0/2292281616 >> 172.21.15.7:0/2292281616 conn(0x5557ebae6f40 msgr2=0x5557ebae0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f827796a740 1 -- 172.21.15.7:0/2292281616 shutdown_connections 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.478+0000 7f827796a740 1 -- 172.21.15.7:0/2292281616 wait complete. 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f827796a740 1 -- start start 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f827796a740 1 Processor -- start 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f827796a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaf7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f827796a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 0x5557ebaf8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f827796a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebafb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f827796a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5557ebb031d0 con 0x5557ebaeae40 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f827796a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5557ebb02b00 con 0x5557ebaeb830 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f827796a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5557ebb038a0 con 0x5557ebaec2d0 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f827796a740 10 client.0 _ll_register_callbacks cb 0x7f8264009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaf7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaf7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37490/0 (socket says 172.21.15.7:37490) 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826e59c640 1 -- 172.21.15.7:0/1519930715 learned_addr learned my addr 172.21.15.7:0/1519930715 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826dd9b640 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 0x5557ebaf8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826e59c640 1 -- 172.21.15.7:0/1519930715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 msgr2=0x5557ebafb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826ed9d640 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebafb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826e59c640 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebafb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826e59c640 1 -- 172.21.15.7:0/1519930715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 msgr2=0x5557ebaf8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826e59c640 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 0x5557ebaf8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826e59c640 1 -- 172.21.15.7:0/1519930715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557ebafb660 con 0x5557ebaeae40 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826dd9b640 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 0x5557ebaf8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f826e59c640 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaf7b10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f82600a9130 tx=0x7f826008cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815bdaef659ab88 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.479+0000 7f8255ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f8255ffb640 1 -- 172.21.15.7:0/1519930715 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82600348f0 con 0x5557ebaeae40 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f8255ffb640 1 -- 172.21.15.7:0/1519930715 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f82600babe0 con 0x5557ebaeae40 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f827796a740 1 -- 172.21.15.7:0/1519930715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8264012dd0 con 0x5557ebaeae40 2024-04-21T08:42:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f8255ffb640 1 -- 172.21.15.7:0/1519930715 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f82600890d0 con 0x5557ebaeae40 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f827796a740 10 client.14784 fetch_fsmap learned FSMap version 17 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f827796a740 1 -- 172.21.15.7:0/1519930715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5557ebb67e90 con 0x5557ebaeae40 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f8255ffb640 1 -- 172.21.15.7:0/1519930715 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8264012dd0 con 0x5557ebaeae40 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f8255ffb640 1 -- 172.21.15.7:0/1519930715 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f82600f2070 con 0x5557ebaeae40 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f827796a740 10 client.14784 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f827796a740 10 client.14784 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f827796a740 20 client.14784 populate_metadata read hostname 'smithi007' 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f827796a740 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5557ebb6c9f0 0x5557ebb6b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f827796a740 4 client.14784 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f827796a740 1 -- 172.21.15.7:0/1519930715 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "status"}) -- 0x5557eba573a0 con 0x5557ebb6c9f0 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.480+0000 7f826dd9b640 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5557ebb6c9f0 0x5557ebb6b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f8255ffb640 10 client.14784 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:42:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f826dd9b640 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5557ebb6c9f0 0x5557ebb6b700 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e959827641a930 server_cookie=c0395c7ceaeb024 in_seq=0 out_seq=0 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f8255ffb640 1 -- 172.21.15.7:0/1519930715 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5557eba573a0 con 0x5557ebb6c9f0 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f8255ffb640 10 client.14784 handle_command_reply: tid=0 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 1 client.14784 shutdown 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 1 -- 172.21.15.7:0/1519930715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 msgr2=0x5557ebaf7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaf7b10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f82600a9130 tx=0x7f826008cf70 comp rx=0 tx=0).stop 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 1 -- 172.21.15.7:0/1519930715 shutdown_connections 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5557ebb6c9f0 0x5557ebb6b700 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5557ebaec2d0 0x5557ebafb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f8255ffb640 0 client.14784 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5557ebaeb830 0x5557ebaf8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 1 --2- 172.21.15.7:0/1519930715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557ebaeae40 0x5557ebaf7b10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 1 -- 172.21.15.7:0/1519930715 >> 172.21.15.7:0/1519930715 conn(0x5557ebae6f40 msgr2=0x5557ebaef7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 1 -- 172.21.15.7:0/1519930715 shutdown_connections 2024-04-21T08:42:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 1 -- 172.21.15.7:0/1519930715 wait complete. 2024-04-21T08:42:24.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.481+0000 7f827796a740 20 client.14784 trim_cache size 0 max 16384 2024-04-21T08:42:24.489 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:42:24.489 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "c9c60282-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:42:24.489 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2024-04-21T08:42:24.489 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24533, 2024-04-21T08:42:24.489 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-04-21T08:42:24.489 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-04-21T08:42:24.489 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-04-21T08:42:24.489 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 6.8581780569999999, 2024-04-21T08:42:24.489 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 17, 2024-04-21T08:42:24.490 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 90, 2024-04-21T08:42:24.490 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:42:24.490 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 12.999661445999999 2024-04-21T08:42:24.490 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 -- 172.21.15.7:0/4021578611 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82580a3c30 msgr2=0x7f82580a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82580a3c30 0x7f82580a60f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8268021000 tx=0x7f826801f000 comp rx=0 tx=0).stop 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 -- 172.21.15.7:0/4021578611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270102a20 msgr2=0x7f82701981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270102a20 0x7f82701981f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8260032380 tx=0x7f8260034650 comp rx=0 tx=0).stop 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 -- 172.21.15.7:0/4021578611 shutdown_connections 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82580a3c30 0x7f82580a60f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8270107c80 0x7f827019f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8270105350 0x7f8270198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 --2- 172.21.15.7:0/4021578611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8270102a20 0x7f82701981f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 -- 172.21.15.7:0/4021578611 >> 172.21.15.7:0/4021578611 conn(0x7f82700fc430 msgr2=0x7f82701034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 -- 172.21.15.7:0/4021578611 shutdown_connections 2024-04-21T08:42:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:24.482+0000 7f8274d7e640 1 -- 172.21.15.7:0/4021578611 wait complete. 2024-04-21T08:42:24.493 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T08:42:24.493 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:42:24.493 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph fs ls 2024-04-21T08:42:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:25.004 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:24 smithi007 ceph-mon[28477]: pgmap v297: 129 pgs: 1 creating+activating, 4 creating+peering, 116 active+clean, 8 unknown; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 8 op/s 2024-04-21T08:42:25.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:24 smithi007 ceph-mon[28477]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:42:25.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:24 smithi007 ceph-mon[28477]: Cluster is now healthy 2024-04-21T08:42:25.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:24 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1079549357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:25.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:24 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2384302551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:25.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:24 smithi016 ceph-mon[31259]: pgmap v297: 129 pgs: 1 creating+activating, 4 creating+peering, 116 active+clean, 8 unknown; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 8 op/s 2024-04-21T08:42:25.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:24 smithi016 ceph-mon[31259]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:42:25.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:24 smithi016 ceph-mon[31259]: Cluster is now healthy 2024-04-21T08:42:25.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:24 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1079549357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:25.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:24 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2384302551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:25.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:24 smithi077 ceph-mon[31615]: pgmap v297: 129 pgs: 1 creating+activating, 4 creating+peering, 116 active+clean, 8 unknown; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 8 op/s 2024-04-21T08:42:25.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:24 smithi077 ceph-mon[31615]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:42:25.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:24 smithi077 ceph-mon[31615]: Cluster is now healthy 2024-04-21T08:42:25.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:24 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1079549357' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:25.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:24 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2384302551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:26.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.460+0000 7fd82ccf8700 1 -- 172.21.15.7:0/93945751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd828101940 msgr2=0x7fd828101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:26.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.460+0000 7fd82ccf8700 1 --2- 172.21.15.7:0/93945751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd828101940 0x7fd828101dc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd818009990 tx=0x7fd818009ca0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.460+0000 7fd82ccf8700 1 -- 172.21.15.7:0/93945751 shutdown_connections 2024-04-21T08:42:26.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.460+0000 7fd82ccf8700 1 --2- 172.21.15.7:0/93945751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828107990 0x7fd828107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.460+0000 7fd82ccf8700 1 --2- 172.21.15.7:0/93945751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd828102300 0x7fd82810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.460+0000 7fd82ccf8700 1 --2- 172.21.15.7:0/93945751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd828101940 0x7fd828101dc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.460+0000 7fd82ccf8700 1 -- 172.21.15.7:0/93945751 >> 172.21.15.7:0/93945751 conn(0x7fd8280788f0 msgr2=0x7fd828078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:26.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.460+0000 7fd82ccf8700 1 -- 172.21.15.7:0/93945751 shutdown_connections 2024-04-21T08:42:26.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.460+0000 7fd82ccf8700 1 -- 172.21.15.7:0/93945751 wait complete. 2024-04-21T08:42:26.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.461+0000 7fd82ccf8700 1 Processor -- start 2024-04-21T08:42:26.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.461+0000 7fd82ccf8700 1 -- start start 2024-04-21T08:42:26.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.461+0000 7fd82ccf8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828101940 0x7fd8281034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.461+0000 7fd82ccf8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd828102300 0x7fd828103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.461+0000 7fd82ccf8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd828107990 0x7fd828103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.461+0000 7fd82ccf8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd828111b30 con 0x7fd828107990 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.461+0000 7fd82ccf8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281119b0 con 0x7fd828102300 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.461+0000 7fd82ccf8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd828111cb0 con 0x7fd828101940 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.461+0000 7fd826d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd828107990 0x7fd828103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.461+0000 7fd826d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd828107990 0x7fd828103f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39904/0 (socket says 172.21.15.7:39904) 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.461+0000 7fd826d9d700 1 -- 172.21.15.7:0/1449943936 learned_addr learned my addr 172.21.15.7:0/1449943936 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd82659c700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828101940 0x7fd8281034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd825d9b700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd828102300 0x7fd828103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd826d9d700 1 -- 172.21.15.7:0/1449943936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828101940 msgr2=0x7fd8281034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd826d9d700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828101940 0x7fd8281034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd826d9d700 1 -- 172.21.15.7:0/1449943936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd828102300 msgr2=0x7fd828103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd826d9d700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd828102300 0x7fd828103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd826d9d700 1 -- 172.21.15.7:0/1449943936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8281a7590 con 0x7fd828107990 2024-04-21T08:42:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd825d9b700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd828102300 0x7fd828103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd826d9d700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd828107990 0x7fd828103f70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd820007d70 tx=0x7fd82000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd8177fe700 1 -- 172.21.15.7:0/1449943936 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd820016020 con 0x7fd828107990 2024-04-21T08:42:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd8177fe700 1 -- 172.21.15.7:0/1449943936 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd82000ad40 con 0x7fd828107990 2024-04-21T08:42:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd8177fe700 1 -- 172.21.15.7:0/1449943936 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8200115e0 con 0x7fd828107990 2024-04-21T08:42:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd82ccf8700 1 -- 172.21.15.7:0/1449943936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8281a7880 con 0x7fd828107990 2024-04-21T08:42:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.462+0000 7fd82ccf8700 1 -- 172.21.15.7:0/1449943936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8281a7dc0 con 0x7fd828107990 2024-04-21T08:42:26.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.463+0000 7fd82ccf8700 1 -- 172.21.15.7:0/1449943936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd828069730 con 0x7fd828107990 2024-04-21T08:42:26.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.465+0000 7fd8177fe700 1 -- 172.21.15.7:0/1449943936 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd820019020 con 0x7fd828107990 2024-04-21T08:42:26.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.465+0000 7fd8177fe700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8100a3cc0 0x7fd8100a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:26.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.465+0000 7fd82659c700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8100a3cc0 0x7fd8100a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:26.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.465+0000 7fd8177fe700 1 -- 172.21.15.7:0/1449943936 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd8200d0930 con 0x7fd828107990 2024-04-21T08:42:26.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.466+0000 7fd82659c700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8100a3cc0 0x7fd8100a6180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00abd0 tx=0x7fd81c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:26.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.469+0000 7fd8177fe700 1 -- 172.21.15.7:0/1449943936 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd820020020 con 0x7fd828107990 2024-04-21T08:42:26.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.724+0000 7fd82ccf8700 1 -- 172.21.15.7:0/1449943936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd82802df80 con 0x7fd828107990 2024-04-21T08:42:26.725 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:26 smithi007 ceph-mon[28477]: pgmap v298: 129 pgs: 1 creating+activating, 4 creating+peering, 124 active+clean; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2024-04-21T08:42:26.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.725+0000 7fd8177fe700 1 -- 172.21.15.7:0/1449943936 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fd82009d0d0 con 0x7fd828107990 2024-04-21T08:42:26.726 INFO:teuthology.orchestra.run.smithi007.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T08:42:26.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 -- 172.21.15.7:0/1449943936 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8100a3cc0 msgr2=0x7fd8100a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:26.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8100a3cc0 0x7fd8100a6180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00abd0 tx=0x7fd81c00a380 comp rx=0 tx=0).stop 2024-04-21T08:42:26.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 -- 172.21.15.7:0/1449943936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd828107990 msgr2=0x7fd828103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:26.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd828107990 0x7fd828103f70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd820007d70 tx=0x7fd82000a340 comp rx=0 tx=0).stop 2024-04-21T08:42:26.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 -- 172.21.15.7:0/1449943936 shutdown_connections 2024-04-21T08:42:26.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8100a3cc0 0x7fd8100a6180 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd828101940 0x7fd8281034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd828107990 0x7fd828103f70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 --2- 172.21.15.7:0/1449943936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd828102300 0x7fd828103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 -- 172.21.15.7:0/1449943936 >> 172.21.15.7:0/1449943936 conn(0x7fd8280788f0 msgr2=0x7fd82806a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:26.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 -- 172.21.15.7:0/1449943936 shutdown_connections 2024-04-21T08:42:26.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:26.727+0000 7fd82ccf8700 1 -- 172.21.15.7:0/1449943936 wait complete. 2024-04-21T08:42:27.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:26 smithi016 ceph-mon[31259]: pgmap v298: 129 pgs: 1 creating+activating, 4 creating+peering, 124 active+clean; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2024-04-21T08:42:27.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:26 smithi077 ceph-mon[31615]: pgmap v298: 129 pgs: 1 creating+activating, 4 creating+peering, 124 active+clean; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2024-04-21T08:42:27.450 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-21T08:42:27.450 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T08:42:27.450 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi007.front.sepia.ceph.com 2024-04-21T08:42:27.450 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-21T08:42:27.450 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:42:27.450 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-21T08:42:27.450 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T08:42:27.450 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T08:42:27.450 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T08:42:27.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:27.451 DEBUG:teuthology.orchestra.run.smithi007:> ip addr 2024-04-21T08:42:27.468 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout:2: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f0 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 42176sec preferred_lft 42176sec 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link noprefixroute 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout:3: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2024-04-21T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f0 2024-04-21T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2024-04-21T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f1 2024-04-21T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2024-04-21T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f1 2024-04-21T08:42:27.471 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T08:42:27.471 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:42:27.471 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-04-21T08:42:27.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add name ceph-brx type bridge 2024-04-21T08:42:27.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr flush dev ceph-brx 2024-04-21T08:42:27.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set ceph-brx up 2024-04-21T08:42:27.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T08:42:27.471 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-04-21T08:42:27.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:42:27.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:42:27.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:27.667 DEBUG:teuthology.orchestra.run.smithi007:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T08:42:27.693 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-04-21T08:42:27.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:27.695 DEBUG:teuthology.orchestra.run.smithi007:> route 2024-04-21T08:42:27.765 INFO:teuthology.orchestra.run.smithi007.stdout:Kernel IP routing table 2024-04-21T08:42:27.765 INFO:teuthology.orchestra.run.smithi007.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T08:42:27.765 INFO:teuthology.orchestra.run.smithi007.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-21T08:42:27.765 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-21T08:42:27.765 INFO:teuthology.orchestra.run.smithi007.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T08:42:27.765 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:42:27.766 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-04-21T08:42:27.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-21T08:42:27.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-21T08:42:27.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-21T08:42:27.766 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-04-21T08:42:27.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:42:27.960 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:27 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1449943936' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:42:28.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:42:28.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:28.047 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-04-21T08:42:28.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:27 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1449943936' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:42:28.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:28.062 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list-id 2024-04-21T08:42:28.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:27 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1449943936' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:42:28.118 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:42:28.118 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-04-21T08:42:28.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:42:28.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-21T08:42:28.118 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-04-21T08:42:28.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:42:28.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:42:28.236 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-21T08:42:28.236 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:42:28.236 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-04-21T08:42:28.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-21T08:42:28.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T08:42:28.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-21T08:42:28.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-21T08:42:28.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-21T08:42:28.237 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-04-21T08:42:28.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:42:28.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:42:28.363 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:42:28.363 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-04-21T08:42:28.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set brx.0 up 2024-04-21T08:42:28.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T08:42:28.364 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-04-21T08:42:28.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:42:28.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:42:28.440 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-21T08:42:28.440 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:42:28.441 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:42:28.455 INFO:teuthology.orchestra.run.smithi007.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T08:42:28.455 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:42:28.455 DEBUG:teuthology.orchestra.run.smithi007:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:42:28.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2024-04-21T08:42:28.575 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T08:42:28.631 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2024-04-21T08:42:28.631 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2024-04-21T08:42:28.631 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/selinux 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2024-04-21T08:42:28.632 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/a0f5e2f638aa21b276b9cbfec5aa931cc72863d67cd8bb3b969f2a66243d3a2d/merged 2024-04-21T08:42:28.633 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/d5489e31291d1b3f72e030f0bfbe67f721f48cccc09edc2ef9029db75edb1406/merged 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/0 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/bcfcc69acbe60c207a63adbce7ae90624b8d853921022c2df3b7215027b39f7e/merged 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/80a775629437d59ec89bbca28e609da213d430dca9579db258fe567ee425cab3/merged 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/e7de7116ed0211039967e54be58c48e52a9dd294318d8f85157e08d6b928366b/merged 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/08d8e987a9d769654a6db6a33b3946386d26c82e54f726af547d5b340b4bb1b1/merged 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/d6c148ecb737b74c6046760a001fd7c2deacba602caf8bf463776baa11b028c5/merged 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/869ba15e1bec9a8fd27be7e33609e021d0dfac9873d420e6718e83cee64e511c/merged 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/9125adcde3b2ceffc426d04bd4b6c9ad6b3b53f46f8a8727f97b144ba59da9e0/merged 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/70c0bd4f1e8503b924a5d7ec922d355fc787d4d7df152f4d888b4d16454f5e21/merged 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:42:28.634 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:42:28.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:28.635 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2024-04-21T08:42:28.687 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-21T08:42:28.688 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-21T08:42:28.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2024-04-21T08:42:28.756 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T08:42:28.812 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2024-04-21T08:42:28.812 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2024-04-21T08:42:28.812 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2024-04-21T08:42:28.812 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2024-04-21T08:42:28.812 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2024-04-21T08:42:28.812 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2024-04-21T08:42:28.813 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2024-04-21T08:42:28.813 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2024-04-21T08:42:28.813 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2024-04-21T08:42:28.813 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2024-04-21T08:42:28.813 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2024-04-21T08:42:28.813 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/selinux 2024-04-21T08:42:28.813 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:42:28.813 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2024-04-21T08:42:28.813 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2024-04-21T08:42:28.813 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2024-04-21T08:42:28.814 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2024-04-21T08:42:28.814 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T08:42:28.814 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2024-04-21T08:42:28.814 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2024-04-21T08:42:28.814 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T08:42:28.814 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T08:42:28.814 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T08:42:28.814 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T08:42:28.814 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2024-04-21T08:42:28.814 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2024-04-21T08:42:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay 2024-04-21T08:42:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/a0f5e2f638aa21b276b9cbfec5aa931cc72863d67cd8bb3b969f2a66243d3a2d/merged 2024-04-21T08:42:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/d5489e31291d1b3f72e030f0bfbe67f721f48cccc09edc2ef9029db75edb1406/merged 2024-04-21T08:42:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/0 2024-04-21T08:42:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:42:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/bcfcc69acbe60c207a63adbce7ae90624b8d853921022c2df3b7215027b39f7e/merged 2024-04-21T08:42:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/80a775629437d59ec89bbca28e609da213d430dca9579db258fe567ee425cab3/merged 2024-04-21T08:42:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/e7de7116ed0211039967e54be58c48e52a9dd294318d8f85157e08d6b928366b/merged 2024-04-21T08:42:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/08d8e987a9d769654a6db6a33b3946386d26c82e54f726af547d5b340b4bb1b1/merged 2024-04-21T08:42:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/d6c148ecb737b74c6046760a001fd7c2deacba602caf8bf463776baa11b028c5/merged 2024-04-21T08:42:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/869ba15e1bec9a8fd27be7e33609e021d0dfac9873d420e6718e83cee64e511c/merged 2024-04-21T08:42:28.816 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/9125adcde3b2ceffc426d04bd4b6c9ad6b3b53f46f8a8727f97b144ba59da9e0/merged 2024-04-21T08:42:28.816 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/70c0bd4f1e8503b924a5d7ec922d355fc787d4d7df152f4d888b4d16454f5e21/merged 2024-04-21T08:42:28.816 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2024-04-21T08:42:28.816 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:42:28.816 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:42:28.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:28.816 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2024-04-21T08:42:28.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:2024-04-21T08:42:28.830+0000 7fd234bd5480 -1 init, newargv = 0x55e3c7191060 newargc=15 2024-04-21T08:42:28.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:ceph-fuse[70809]: starting ceph client 2024-04-21T08:42:28.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:ceph-fuse[70809]: starting fuse 2024-04-21T08:42:28.846 INFO:teuthology.orchestra.run.smithi007.stdout:115 2024-04-21T08:42:28.846 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-21T08:42:28.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo stdin-killer -- python3 -c ' 2024-04-21T08:42:28.846 DEBUG:teuthology.orchestra.run.smithi007:> import glob 2024-04-21T08:42:28.846 DEBUG:teuthology.orchestra.run.smithi007:> import re 2024-04-21T08:42:28.846 DEBUG:teuthology.orchestra.run.smithi007:> import os 2024-04-21T08:42:28.846 DEBUG:teuthology.orchestra.run.smithi007:> import subprocess 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> def _find_admin_socket(client_name): 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> files = glob.glob(asok_path) 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> # Given a non-glob path, it better be there 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> if "*" not in asok_path: 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> assert(len(files) == 1) 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> return files[0] 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> for f in files: 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-21T08:42:28.847 DEBUG:teuthology.orchestra.run.smithi007:> contents = proc_f.read() 2024-04-21T08:42:28.848 DEBUG:teuthology.orchestra.run.smithi007:> if mountpoint in contents: 2024-04-21T08:42:28.848 DEBUG:teuthology.orchestra.run.smithi007:> return f 2024-04-21T08:42:28.848 DEBUG:teuthology.orchestra.run.smithi007:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-21T08:42:28.848 DEBUG:teuthology.orchestra.run.smithi007:> 2024-04-21T08:42:28.848 DEBUG:teuthology.orchestra.run.smithi007:> print(_find_admin_socket("client.admin")) 2024-04-21T08:42:28.848 DEBUG:teuthology.orchestra.run.smithi007:> ' 2024-04-21T08:42:28.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:28 smithi007 ceph-mon[28477]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 879 B/s wr, 2 op/s 2024-04-21T08:42:28.967 INFO:teuthology.orchestra.run.smithi007.stdout:/var/run/ceph/ceph-client.admin.70809.asok 2024-04-21T08:42:28.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:42:28.977 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.70809.asok 2024-04-21T08:42:28.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:28.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.70809.asok status 2024-04-21T08:42:29.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:28 smithi016 ceph-mon[31259]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 879 B/s wr, 2 op/s 2024-04-21T08:42:29.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:28 smithi077 ceph-mon[31615]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 879 B/s wr, 2 op/s 2024-04-21T08:42:29.115 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:42:29.115 INFO:teuthology.orchestra.run.smithi007.stdout: "metadata": { 2024-04-21T08:42:29.115 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_sha1": "a630765c1f93c6e61d9b40c4b76437611d5ef29e", 2024-04-21T08:42:29.115 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:42:29.115 INFO:teuthology.orchestra.run.smithi007.stdout: "entity_id": "admin", 2024-04-21T08:42:29.115 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": "smithi007", 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: "pid": "70809", 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: "root": "/" 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_count": 0, 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_pinned_count": 0, 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 14796, 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: "inst": { 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: "name": { 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "client", 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: "num": 14796 2024-04-21T08:42:29.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "192.168.0.1:0", 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3710418408 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "192.168.0.1:0", 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3710418408 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: "inst_str": "client.14796 192.168.0.1:0/3710418408", 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: "addr_str": "192.168.0.1:0/3710418408", 2024-04-21T08:42:29.117 INFO:teuthology.orchestra.run.smithi007.stdout: "inode_count": 1, 2024-04-21T08:42:29.118 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_epoch": 17, 2024-04-21T08:42:29.118 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch": 90, 2024-04-21T08:42:29.118 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch_barrier": 0, 2024-04-21T08:42:29.118 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklisted": false, 2024-04-21T08:42:29.118 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs" 2024-04-21T08:42:29.118 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:42:29.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:29.124 DEBUG:teuthology.orchestra.run.smithi007:> stat --file-system '--printf=%T 2024-04-21T08:42:29.124 DEBUG:teuthology.orchestra.run.smithi007:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:42:29.140 INFO:teuthology.orchestra.run.smithi007.stdout:fuseblk 2024-04-21T08:42:29.140 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:42:29.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:29.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:42:29.206 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-21T08:42:29.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:42:29.320 INFO:teuthology.orchestra.run.smithi007.stdout:# file: . 2024-04-21T08:42:29.320 INFO:teuthology.orchestra.run.smithi007.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-21T08:42:29.320 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:29.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:42:29.326 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:42:29.326 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:42:29.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:42:29.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.485+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/1937185858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1e0100ea0 msgr2=0x7ff1e01012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.485+0000 7ff1e7a6a640 1 --2- 172.21.15.7:0/1937185858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1e0100ea0 0x7ff1e01012c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009900 tx=0x7ff1dc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.485+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/1937185858 shutdown_connections 2024-04-21T08:42:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.485+0000 7ff1e7a6a640 1 --2- 172.21.15.7:0/1937185858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1e01029b0 0x7ff1e01092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.485+0000 7ff1e7a6a640 1 --2- 172.21.15.7:0/1937185858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e0101ff0 0x7ff1e0102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.485+0000 7ff1e7a6a640 1 --2- 172.21.15.7:0/1937185858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1e0100ea0 0x7ff1e01012c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.485+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/1937185858 >> 172.21.15.7:0/1937185858 conn(0x7ff1e00fc410 msgr2=0x7ff1e00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.485+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/1937185858 shutdown_connections 2024-04-21T08:42:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.485+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/1937185858 wait complete. 2024-04-21T08:42:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e7a6a640 1 Processor -- start 2024-04-21T08:42:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e7a6a640 1 -- start start 2024-04-21T08:42:29.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e7a6a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1e0100ea0 0x7ff1e0104e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e7a6a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1e0101ff0 0x7ff1e0103460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e7a6a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e01029b0 0x7ff1e01039a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e7a6a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e010c9f0 con 0x7ff1e0100ea0 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e7a6a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e010c870 con 0x7ff1e01029b0 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e7a6a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e010cb70 con 0x7ff1e0101ff0 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e5806640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1e0100ea0 0x7ff1e0104e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e5806640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1e0100ea0 0x7ff1e0104e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39928/0 (socket says 172.21.15.7:39928) 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e5806640 1 -- 172.21.15.7:0/3983685806 learned_addr learned my addr 172.21.15.7:0/3983685806 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e6007640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e01029b0 0x7ff1e01039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e5005640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1e0101ff0 0x7ff1e0103460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e5806640 1 -- 172.21.15.7:0/3983685806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1e0101ff0 msgr2=0x7ff1e0103460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e5806640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1e0101ff0 0x7ff1e0103460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e5806640 1 -- 172.21.15.7:0/3983685806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e01029b0 msgr2=0x7ff1e01039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e5806640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e01029b0 0x7ff1e01039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e5806640 1 -- 172.21.15.7:0/3983685806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e0104230 con 0x7ff1e0100ea0 2024-04-21T08:42:29.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e6007640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e01029b0 0x7ff1e01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:29.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.486+0000 7ff1e5806640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1e0100ea0 0x7ff1e0104e20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc02c840 tx=0x7ff1dc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:29.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.487+0000 7ff1d6ffd640 1 -- 172.21.15.7:0/3983685806 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc043020 con 0x7ff1e0100ea0 2024-04-21T08:42:29.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.487+0000 7ff1d6ffd640 1 -- 172.21.15.7:0/3983685806 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1dc004a80 con 0x7ff1e0100ea0 2024-04-21T08:42:29.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.487+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/3983685806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e01a2330 con 0x7ff1e0100ea0 2024-04-21T08:42:29.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.487+0000 7ff1d6ffd640 1 -- 172.21.15.7:0/3983685806 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc003c40 con 0x7ff1e0100ea0 2024-04-21T08:42:29.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.487+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/3983685806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e01a2880 con 0x7ff1e0100ea0 2024-04-21T08:42:29.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.488+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/3983685806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1e01a2a80 con 0x7ff1e0100ea0 2024-04-21T08:42:29.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.489+0000 7ff1d6ffd640 1 -- 172.21.15.7:0/3983685806 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7ff1dc046020 con 0x7ff1e0100ea0 2024-04-21T08:42:29.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.491+0000 7ff1d6ffd640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff1cc0a3bb0 0x7ff1cc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.491+0000 7ff1e5005640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff1cc0a3bb0 0x7ff1cc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.491+0000 7ff1d6ffd640 1 -- 172.21.15.7:0/3983685806 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff1dc0f3210 con 0x7ff1e0100ea0 2024-04-21T08:42:29.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.491+0000 7ff1e5005640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff1cc0a3bb0 0x7ff1cc0a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0104b90 tx=0x7ff1d0005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:29.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.494+0000 7ff1d6ffd640 1 -- 172.21.15.7:0/3983685806 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1dc04d020 con 0x7ff1e0100ea0 2024-04-21T08:42:29.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.685+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/3983685806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff1e01a2c50 con 0x7ff1e0100ea0 2024-04-21T08:42:29.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.686+0000 7ff1d6ffd640 1 -- 172.21.15.7:0/3983685806 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23961 (secure 0 0 0) 0x7ff1dc0b9e20 con 0x7ff1e0100ea0 2024-04-21T08:42:29.687 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:29.687 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":90,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:42:22.023111+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:29.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/3983685806 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff1cc0a3bb0 msgr2=0x7ff1cc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff1cc0a3bb0 0x7ff1cc0a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0104b90 tx=0x7ff1d0005c60 comp rx=0 tx=0).stop 2024-04-21T08:42:29.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/3983685806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1e0100ea0 msgr2=0x7ff1e0104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1e0100ea0 0x7ff1e0104e20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc02c840 tx=0x7ff1dc004270 comp rx=0 tx=0).stop 2024-04-21T08:42:29.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/3983685806 shutdown_connections 2024-04-21T08:42:29.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff1cc0a3bb0 0x7ff1cc0a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e01029b0 0x7ff1e01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1e0101ff0 0x7ff1e0103460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 --2- 172.21.15.7:0/3983685806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1e0100ea0 0x7ff1e0104e20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/3983685806 >> 172.21.15.7:0/3983685806 conn(0x7ff1e00fc410 msgr2=0x7ff1e00fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:29.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/3983685806 shutdown_connections 2024-04-21T08:42:29.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.689+0000 7ff1e7a6a640 1 -- 172.21.15.7:0/3983685806 wait complete. 2024-04-21T08:42:29.700 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:42:29.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.806+0000 7f51a7bce640 1 -- 172.21.15.7:0/3312570570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51a0107ce0 msgr2=0x7f51a010a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.806+0000 7f51a7bce640 1 --2- 172.21.15.7:0/3312570570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51a0107ce0 0x7f51a010a0d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5198009ca0 tx=0x7f519802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.806+0000 7f51a7bce640 1 -- 172.21.15.7:0/3312570570 shutdown_connections 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.806+0000 7f51a7bce640 1 --2- 172.21.15.7:0/3312570570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51a0107ce0 0x7f51a010a0d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.806+0000 7f51a7bce640 1 --2- 172.21.15.7:0/3312570570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a01053b0 0x7f51a01077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.806+0000 7f51a7bce640 1 --2- 172.21.15.7:0/3312570570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51a0102a80 0x7f51a0104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.806+0000 7f51a7bce640 1 -- 172.21.15.7:0/3312570570 >> 172.21.15.7:0/3312570570 conn(0x7f51a00fc430 msgr2=0x7f51a00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.806+0000 7f51a7bce640 1 -- 172.21.15.7:0/3312570570 shutdown_connections 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.806+0000 7f51a7bce640 1 -- 172.21.15.7:0/3312570570 wait complete. 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a7bce640 1 Processor -- start 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a7bce640 1 -- start start 2024-04-21T08:42:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a7bce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51a0102a80 0x7f51a0198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a7bce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a01053b0 0x7f51a01986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a7bce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51a0107ce0 0x7f51a019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a7bce640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f51a010dae0 con 0x7f51a0107ce0 2024-04-21T08:42:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a7bce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51a010d960 con 0x7f51a01053b0 2024-04-21T08:42:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a7bce640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f51a010dc60 con 0x7f51a0102a80 2024-04-21T08:42:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a616b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51a0107ce0 0x7f51a019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a616b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51a0107ce0 0x7f51a019f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39948/0 (socket says 172.21.15.7:39948) 2024-04-21T08:42:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a616b640 1 -- 172.21.15.7:0/1532128670 learned_addr learned my addr 172.21.15.7:0/1532128670 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a596a640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51a0102a80 0x7f51a0198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a5169640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a01053b0 0x7f51a01986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a616b640 1 -- 172.21.15.7:0/1532128670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51a0102a80 msgr2=0x7f51a0198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a616b640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51a0102a80 0x7f51a0198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a616b640 1 -- 172.21.15.7:0/1532128670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a01053b0 msgr2=0x7f51a01986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a616b640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a01053b0 0x7f51a01986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.807+0000 7f51a616b640 1 -- 172.21.15.7:0/1532128670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a019fdb0 con 0x7f51a0107ce0 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.808+0000 7f51a616b640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51a0107ce0 0x7f51a019f740 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f519802c700 tx=0x7f5198034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.808+0000 7f5196ffd640 1 -- 172.21.15.7:0/1532128670 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5198039040 con 0x7f51a0107ce0 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.808+0000 7f5196ffd640 1 -- 172.21.15.7:0/1532128670 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5198044070 con 0x7f51a0107ce0 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.808+0000 7f51a7bce640 1 -- 172.21.15.7:0/1532128670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a01a0040 con 0x7f51a0107ce0 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.808+0000 7f5196ffd640 1 -- 172.21.15.7:0/1532128670 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51980042a0 con 0x7f51a0107ce0 2024-04-21T08:42:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.808+0000 7f51a7bce640 1 -- 172.21.15.7:0/1532128670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a01a0580 con 0x7f51a0107ce0 2024-04-21T08:42:29.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.809+0000 7f51a7bce640 1 -- 172.21.15.7:0/1532128670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a00fdf20 con 0x7f51a0107ce0 2024-04-21T08:42:29.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.810+0000 7f5196ffd640 1 -- 172.21.15.7:0/1532128670 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f5198003970 con 0x7f51a0107ce0 2024-04-21T08:42:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.811+0000 7f5196ffd640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f518c0a3c30 0x7f518c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.811+0000 7f51a596a640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f518c0a3c30 0x7f518c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.811+0000 7f5196ffd640 1 -- 172.21.15.7:0/1532128670 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f51980f3480 con 0x7f51a0107ce0 2024-04-21T08:42:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.811+0000 7f51a596a640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f518c0a3c30 0x7f518c0a60f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f519c005fd0 tx=0x7f519c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:29.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:29.814+0000 7f5196ffd640 1 -- 172.21.15.7:0/1532128670 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51980bfbc0 con 0x7f51a0107ce0 2024-04-21T08:42:30.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.001+0000 7f51a7bce640 1 -- 172.21.15.7:0/1532128670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f51a002d5c0 con 0x7f51a0107ce0 2024-04-21T08:42:30.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.001+0000 7f5196ffd640 1 -- 172.21.15.7:0/1532128670 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f51980bf4c0 con 0x7f51a0107ce0 2024-04-21T08:42:30.001 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:42:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 -- 172.21.15.7:0/1532128670 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f518c0a3c30 msgr2=0x7f518c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f518c0a3c30 0x7f518c0a60f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f519c005fd0 tx=0x7f519c00b390 comp rx=0 tx=0).stop 2024-04-21T08:42:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 -- 172.21.15.7:0/1532128670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51a0107ce0 msgr2=0x7f51a019f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51a0107ce0 0x7f51a019f740 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f519802c700 tx=0x7f5198034760 comp rx=0 tx=0).stop 2024-04-21T08:42:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 -- 172.21.15.7:0/1532128670 shutdown_connections 2024-04-21T08:42:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f518c0a3c30 0x7f518c0a60f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51a0107ce0 0x7f51a019f740 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a01053b0 0x7f51a01986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 --2- 172.21.15.7:0/1532128670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51a0102a80 0x7f51a0198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 -- 172.21.15.7:0/1532128670 >> 172.21.15.7:0/1532128670 conn(0x7f51a00fc430 msgr2=0x7f51a01033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:30.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 -- 172.21.15.7:0/1532128670 shutdown_connections 2024-04-21T08:42:30.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.003+0000 7f51a7bce640 1 -- 172.21.15.7:0/1532128670 wait complete. 2024-04-21T08:42:30.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:42:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.119+0000 7f85983a4640 1 -- 172.21.15.7:0/2064404344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8590100e90 msgr2=0x7f85901012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.119+0000 7f85983a4640 1 --2- 172.21.15.7:0/2064404344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8590100e90 0x7f85901012b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f858c009900 tx=0x7f858c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.119+0000 7f85983a4640 1 -- 172.21.15.7:0/2064404344 shutdown_connections 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.119+0000 7f85983a4640 1 --2- 172.21.15.7:0/2064404344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85901029a0 0x7f85901092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.119+0000 7f85983a4640 1 --2- 172.21.15.7:0/2064404344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8590101fe0 0x7f8590102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.119+0000 7f85983a4640 1 --2- 172.21.15.7:0/2064404344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8590100e90 0x7f85901012b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.119+0000 7f85983a4640 1 -- 172.21.15.7:0/2064404344 >> 172.21.15.7:0/2064404344 conn(0x7f85900fc420 msgr2=0x7f85900fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.119+0000 7f85983a4640 1 -- 172.21.15.7:0/2064404344 shutdown_connections 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.119+0000 7f85983a4640 1 -- 172.21.15.7:0/2064404344 wait complete. 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.119+0000 7f85983a4640 1 Processor -- start 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.119+0000 7f85983a4640 1 -- start start 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f85983a4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8590100e90 0x7f85901970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f85983a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8590101fe0 0x7f8590197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f85983a4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85901029a0 0x7f859019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f85983a4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f859010c9c0 con 0x7f85901029a0 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f85983a4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f859010c840 con 0x7f8590101fe0 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f85983a4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f859010cb40 con 0x7f8590100e90 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f8596941640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85901029a0 0x7f859019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f8596941640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85901029a0 0x7f859019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39960/0 (socket says 172.21.15.7:39960) 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f8596941640 1 -- 172.21.15.7:0/1955223087 learned_addr learned my addr 172.21.15.7:0/1955223087 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f8596140640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8590100e90 0x7f85901970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f859593f640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8590101fe0 0x7f8590197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f8596941640 1 -- 172.21.15.7:0/1955223087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8590100e90 msgr2=0x7f85901970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f8596941640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8590100e90 0x7f85901970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f8596941640 1 -- 172.21.15.7:0/1955223087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8590101fe0 msgr2=0x7f8590197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f8596941640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8590101fe0 0x7f8590197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f8596941640 1 -- 172.21.15.7:0/1955223087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f859019ed10 con 0x7f85901029a0 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f859593f640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8590101fe0 0x7f8590197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f8596941640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85901029a0 0x7f859019e6a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8588009fd0 tx=0x7f858800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f85877fe640 1 -- 172.21.15.7:0/1955223087 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8588017070 con 0x7f85901029a0 2024-04-21T08:42:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f85877fe640 1 -- 172.21.15.7:0/1955223087 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8588003e20 con 0x7f85901029a0 2024-04-21T08:42:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f85983a4640 1 -- 172.21.15.7:0/1955223087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f859019efa0 con 0x7f85901029a0 2024-04-21T08:42:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f85877fe640 1 -- 172.21.15.7:0/1955223087 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8588010460 con 0x7f85901029a0 2024-04-21T08:42:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.120+0000 7f85983a4640 1 -- 172.21.15.7:0/1955223087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f859019f470 con 0x7f85901029a0 2024-04-21T08:42:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.121+0000 7f85983a4640 1 -- 172.21.15.7:0/1955223087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8590108310 con 0x7f85901029a0 2024-04-21T08:42:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.122+0000 7f85877fe640 1 -- 172.21.15.7:0/1955223087 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f8588003970 con 0x7f85901029a0 2024-04-21T08:42:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.123+0000 7f85877fe640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f857c0a3bb0 0x7f857c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.123+0000 7f85877fe640 1 -- 172.21.15.7:0/1955223087 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f85880d0450 con 0x7f85901029a0 2024-04-21T08:42:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.123+0000 7f8596140640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f857c0a3bb0 0x7f857c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.124+0000 7f8596140640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f857c0a3bb0 0x7f857c0a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f858c000c00 tx=0x7f858c005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:30.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.127+0000 7f85877fe640 1 -- 172.21.15.7:0/1955223087 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f858809cbf0 con 0x7f85901029a0 2024-04-21T08:42:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.310+0000 7f85983a4640 1 -- 172.21.15.7:0/1955223087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8590062b50 con 0x7f85901029a0 2024-04-21T08:42:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.311+0000 7f85877fe640 1 -- 172.21.15.7:0/1955223087 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f858809c4f0 con 0x7f85901029a0 2024-04-21T08:42:30.310 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:42:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.312+0000 7f85983a4640 1 -- 172.21.15.7:0/1955223087 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f857c0a3bb0 msgr2=0x7f857c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.312+0000 7f85983a4640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f857c0a3bb0 0x7f857c0a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f858c000c00 tx=0x7f858c005be0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.312+0000 7f85983a4640 1 -- 172.21.15.7:0/1955223087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85901029a0 msgr2=0x7f859019e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.312+0000 7f85983a4640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85901029a0 0x7f859019e6a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8588009fd0 tx=0x7f858800b260 comp rx=0 tx=0).stop 2024-04-21T08:42:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.312+0000 7f85983a4640 1 -- 172.21.15.7:0/1955223087 shutdown_connections 2024-04-21T08:42:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.312+0000 7f85983a4640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f857c0a3bb0 0x7f857c0a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.312+0000 7f85983a4640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85901029a0 0x7f859019e6a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.312+0000 7f85983a4640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8590101fe0 0x7f8590197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.312+0000 7f85983a4640 1 --2- 172.21.15.7:0/1955223087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8590100e90 0x7f85901970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.312+0000 7f85983a4640 1 -- 172.21.15.7:0/1955223087 >> 172.21.15.7:0/1955223087 conn(0x7f85900fc420 msgr2=0x7f85901054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:30.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.313+0000 7f85983a4640 1 -- 172.21.15.7:0/1955223087 shutdown_connections 2024-04-21T08:42:30.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.313+0000 7f85983a4640 1 -- 172.21.15.7:0/1955223087 wait complete. 2024-04-21T08:42:30.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:42:30.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.428+0000 7f6082406640 1 -- 172.21.15.7:0/760824134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f607c100ea0 msgr2=0x7f607c1012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.428+0000 7f6082406640 1 --2- 172.21.15.7:0/760824134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f607c100ea0 0x7f607c1012c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f606c009900 tx=0x7f606c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:30.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.428+0000 7f6082406640 1 -- 172.21.15.7:0/760824134 shutdown_connections 2024-04-21T08:42:30.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.428+0000 7f6082406640 1 --2- 172.21.15.7:0/760824134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f607c1029b0 0x7f607c1092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.428+0000 7f6082406640 1 --2- 172.21.15.7:0/760824134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f607c101ff0 0x7f607c102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.428+0000 7f6082406640 1 --2- 172.21.15.7:0/760824134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f607c100ea0 0x7f607c1012c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.428+0000 7f6082406640 1 -- 172.21.15.7:0/760824134 >> 172.21.15.7:0/760824134 conn(0x7f607c0fc430 msgr2=0x7f607c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.428+0000 7f6082406640 1 -- 172.21.15.7:0/760824134 shutdown_connections 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.428+0000 7f6082406640 1 -- 172.21.15.7:0/760824134 wait complete. 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.429+0000 7f6082406640 1 Processor -- start 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.429+0000 7f6082406640 1 -- start start 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.429+0000 7f6082406640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f607c100ea0 0x7f607c10b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.429+0000 7f6082406640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f607c101ff0 0x7f607c109e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.429+0000 7f6082406640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f607c1029b0 0x7f607c10a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.429+0000 7f6082406640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f607c10db40 con 0x7f607c1029b0 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.429+0000 7f6082406640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f607c10d9c0 con 0x7f607c101ff0 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.429+0000 7f6082406640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f607c10dcc0 con 0x7f607c100ea0 2024-04-21T08:42:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.429+0000 7f60809a3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f607c1029b0 0x7f607c10a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.429+0000 7f60809a3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f607c1029b0 0x7f607c10a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39974/0 (socket says 172.21.15.7:39974) 2024-04-21T08:42:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.429+0000 7f60809a3640 1 -- 172.21.15.7:0/2848320571 learned_addr learned my addr 172.21.15.7:0/2848320571 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f607b7fe640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f607c101ff0 0x7f607c109e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f607bfff640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f607c100ea0 0x7f607c10b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f60809a3640 1 -- 172.21.15.7:0/2848320571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f607c100ea0 msgr2=0x7f607c10b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f60809a3640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f607c100ea0 0x7f607c10b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f60809a3640 1 -- 172.21.15.7:0/2848320571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f607c101ff0 msgr2=0x7f607c109e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f60809a3640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f607c101ff0 0x7f607c109e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f60809a3640 1 -- 172.21.15.7:0/2848320571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c10ac80 con 0x7f607c1029b0 2024-04-21T08:42:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f607b7fe640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f607c101ff0 0x7f607c109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:30.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f60809a3640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f607c1029b0 0x7f607c10a3c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f607400cce0 tx=0x7f607400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:30.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f60797fa640 1 -- 172.21.15.7:0/2848320571 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607400f450 con 0x7f607c1029b0 2024-04-21T08:42:30.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f60797fa640 1 -- 172.21.15.7:0/2848320571 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f607400faa0 con 0x7f607c1029b0 2024-04-21T08:42:30.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f6082406640 1 -- 172.21.15.7:0/2848320571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607c1a2310 con 0x7f607c1029b0 2024-04-21T08:42:30.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f60797fa640 1 -- 172.21.15.7:0/2848320571 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074018b70 con 0x7f607c1029b0 2024-04-21T08:42:30.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.430+0000 7f6082406640 1 -- 172.21.15.7:0/2848320571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f607c1a27e0 con 0x7f607c1029b0 2024-04-21T08:42:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.431+0000 7f6082406640 1 -- 172.21.15.7:0/2848320571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f607c1081b0 con 0x7f607c1029b0 2024-04-21T08:42:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.432+0000 7f60797fa640 1 -- 172.21.15.7:0/2848320571 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6074018d10 con 0x7f607c1029b0 2024-04-21T08:42:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.433+0000 7f60797fa640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60640a3bb0 0x7f60640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.433+0000 7f607bfff640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60640a3bb0 0x7f60640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.433+0000 7f60797fa640 1 -- 172.21.15.7:0/2848320571 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f60740d0be0 con 0x7f607c1029b0 2024-04-21T08:42:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.434+0000 7f607bfff640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60640a3bb0 0x7f60640a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f606c0096c0 tx=0x7f606c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:30.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.437+0000 7f60797fa640 1 -- 172.21.15.7:0/2848320571 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6074014030 con 0x7f607c1029b0 2024-04-21T08:42:30.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.623+0000 7f6082406640 1 -- 172.21.15.7:0/2848320571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f607c02d5c0 con 0x7f607c1029b0 2024-04-21T08:42:30.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.624+0000 7f60797fa640 1 -- 172.21.15.7:0/2848320571 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f607409d410 con 0x7f607c1029b0 2024-04-21T08:42:30.623 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:42:30.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 -- 172.21.15.7:0/2848320571 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60640a3bb0 msgr2=0x7f60640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60640a3bb0 0x7f60640a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f606c0096c0 tx=0x7f606c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 -- 172.21.15.7:0/2848320571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f607c1029b0 msgr2=0x7f607c10a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f607c1029b0 0x7f607c10a3c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f607400cce0 tx=0x7f607400a620 comp rx=0 tx=0).stop 2024-04-21T08:42:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 -- 172.21.15.7:0/2848320571 shutdown_connections 2024-04-21T08:42:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60640a3bb0 0x7f60640a6070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f607c1029b0 0x7f607c10a3c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f607c101ff0 0x7f607c109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 --2- 172.21.15.7:0/2848320571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f607c100ea0 0x7f607c10b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 -- 172.21.15.7:0/2848320571 >> 172.21.15.7:0/2848320571 conn(0x7f607c0fc430 msgr2=0x7f607c105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 -- 172.21.15.7:0/2848320571 shutdown_connections 2024-04-21T08:42:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.626+0000 7f6082406640 1 -- 172.21.15.7:0/2848320571 wait complete. 2024-04-21T08:42:30.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.743+0000 7fa27a225640 1 -- 172.21.15.7:0/2089574276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa274102040 msgr2=0x7fa2741024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.743+0000 7fa27a225640 1 --2- 172.21.15.7:0/2089574276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa274102040 0x7fa2741024c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa268009900 tx=0x7fa26802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.743+0000 7fa27a225640 1 -- 172.21.15.7:0/2089574276 shutdown_connections 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.743+0000 7fa27a225640 1 --2- 172.21.15.7:0/2089574276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa274102a00 0x7fa274109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.743+0000 7fa27a225640 1 --2- 172.21.15.7:0/2089574276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa274102040 0x7fa2741024c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.743+0000 7fa27a225640 1 --2- 172.21.15.7:0/2089574276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa274100e60 0x7fa274101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.743+0000 7fa27a225640 1 -- 172.21.15.7:0/2089574276 >> 172.21.15.7:0/2089574276 conn(0x7fa2740fc420 msgr2=0x7fa2740fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.743+0000 7fa27a225640 1 -- 172.21.15.7:0/2089574276 shutdown_connections 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.743+0000 7fa27a225640 1 -- 172.21.15.7:0/2089574276 wait complete. 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.743+0000 7fa27a225640 1 Processor -- start 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.743+0000 7fa27a225640 1 -- start start 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa27a225640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa274100e60 0x7fa27406ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa27a225640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa274102040 0x7fa27419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa27a225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa274102a00 0x7fa27419f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa27a225640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa27410c9f0 con 0x7fa274102040 2024-04-21T08:42:30.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa27a225640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa27410c870 con 0x7fa274102a00 2024-04-21T08:42:30.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa27a225640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa27410cb70 con 0x7fa274100e60 2024-04-21T08:42:30.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa272ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa274102040 0x7fa27419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa272ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa274102040 0x7fa27419cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39992/0 (socket says 172.21.15.7:39992) 2024-04-21T08:42:30.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa272ffd640 1 -- 172.21.15.7:0/1719312616 learned_addr learned my addr 172.21.15.7:0/1719312616 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:30.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa2737fe640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa274100e60 0x7fa27406ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa273fff640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa274102a00 0x7fa27419f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa272ffd640 1 -- 172.21.15.7:0/1719312616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa274100e60 msgr2=0x7fa27406ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa272ffd640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa274100e60 0x7fa27406ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa272ffd640 1 -- 172.21.15.7:0/1719312616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa274102a00 msgr2=0x7fa27419f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa272ffd640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa274102a00 0x7fa27419f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa272ffd640 1 -- 172.21.15.7:0/1719312616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27419fe40 con 0x7fa274102040 2024-04-21T08:42:30.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa272ffd640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa274102040 0x7fa27419cf10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa268006010 tx=0x7fa268004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:30.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa270ff9640 1 -- 172.21.15.7:0/1719312616 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa268043020 con 0x7fa274102040 2024-04-21T08:42:30.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa270ff9640 1 -- 172.21.15.7:0/1719312616 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa268004440 con 0x7fa274102040 2024-04-21T08:42:30.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.744+0000 7fa27a225640 1 -- 172.21.15.7:0/1719312616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2741a0070 con 0x7fa274102040 2024-04-21T08:42:30.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.745+0000 7fa27a225640 1 -- 172.21.15.7:0/1719312616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2741a05c0 con 0x7fa274102040 2024-04-21T08:42:30.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.745+0000 7fa270ff9640 1 -- 172.21.15.7:0/1719312616 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa268003bb0 con 0x7fa274102040 2024-04-21T08:42:30.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.745+0000 7fa27a225640 1 -- 172.21.15.7:0/1719312616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2741082d0 con 0x7fa274102040 2024-04-21T08:42:30.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.747+0000 7fa270ff9640 1 -- 172.21.15.7:0/1719312616 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fa268046020 con 0x7fa274102040 2024-04-21T08:42:30.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.747+0000 7fa270ff9640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa25c0a3bb0 0x7fa25c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:30.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.748+0000 7fa2737fe640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa25c0a3bb0 0x7fa25c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:30.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.748+0000 7fa270ff9640 1 -- 172.21.15.7:0/1719312616 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa2680f2f60 con 0x7fa274102040 2024-04-21T08:42:30.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.748+0000 7fa2737fe640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa25c0a3bb0 0x7fa25c0a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa264005fd0 tx=0x7fa26400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:30.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.751+0000 7fa270ff9640 1 -- 172.21.15.7:0/1719312616 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa26804d020 con 0x7fa274102040 2024-04-21T08:42:30.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:30 smithi007 ceph-mon[28477]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-04-21T08:42:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:30 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3983685806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:30 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1532128670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:30 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1955223087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:42:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:30 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2848320571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:42:30.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.938+0000 7fa27a225640 1 -- 172.21.15.7:0/1719312616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa27402d660 con 0x7fa274102040 2024-04-21T08:42:30.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.939+0000 7fa270ff9640 1 -- 172.21.15.7:0/1719312616 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7fa2680bf700 con 0x7fa274102040 2024-04-21T08:42:30.938 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:42:30.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 -- 172.21.15.7:0/1719312616 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa25c0a3bb0 msgr2=0x7fa25c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa25c0a3bb0 0x7fa25c0a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa264005fd0 tx=0x7fa26400b410 comp rx=0 tx=0).stop 2024-04-21T08:42:30.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 -- 172.21.15.7:0/1719312616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa274102040 msgr2=0x7fa27419cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa274102040 0x7fa27419cf10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa268006010 tx=0x7fa268004040 comp rx=0 tx=0).stop 2024-04-21T08:42:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 -- 172.21.15.7:0/1719312616 shutdown_connections 2024-04-21T08:42:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa25c0a3bb0 0x7fa25c0a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa274102a00 0x7fa27419f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa274102040 0x7fa27419cf10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 --2- 172.21.15.7:0/1719312616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa274100e60 0x7fa27406ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 -- 172.21.15.7:0/1719312616 >> 172.21.15.7:0/1719312616 conn(0x7fa2740fc420 msgr2=0x7fa2741054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 -- 172.21.15.7:0/1719312616 shutdown_connections 2024-04-21T08:42:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:30.941+0000 7fa27a225640 1 -- 172.21.15.7:0/1719312616 wait complete. 2024-04-21T08:42:30.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T08:42:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:30 smithi016 ceph-mon[31259]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-04-21T08:42:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:30 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3983685806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:30 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1532128670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:30 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1955223087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:42:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:30 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2848320571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:42:31.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.055+0000 7f937b2a8640 1 -- 172.21.15.7:0/222018191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9374101fb0 msgr2=0x7f93741023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.055+0000 7f937b2a8640 1 --2- 172.21.15.7:0/222018191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9374101fb0 0x7f93741023d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9370009ca0 tx=0x7f937002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.055+0000 7f937b2a8640 1 -- 172.21.15.7:0/222018191 shutdown_connections 2024-04-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.055+0000 7f937b2a8640 1 --2- 172.21.15.7:0/222018191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9374103b60 0x7f937410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.055+0000 7f937b2a8640 1 --2- 172.21.15.7:0/222018191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93741031a0 0x7f9374103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.055+0000 7f937b2a8640 1 --2- 172.21.15.7:0/222018191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9374101fb0 0x7f93741023d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.055+0000 7f937b2a8640 1 -- 172.21.15.7:0/222018191 >> 172.21.15.7:0/222018191 conn(0x7f93740fd520 msgr2=0x7f93740ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.055+0000 7f937b2a8640 1 -- 172.21.15.7:0/222018191 shutdown_connections 2024-04-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.055+0000 7f937b2a8640 1 -- 172.21.15.7:0/222018191 wait complete. 2024-04-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f937b2a8640 1 Processor -- start 2024-04-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f937b2a8640 1 -- start start 2024-04-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f937b2a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9374101fb0 0x7f9374198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f937b2a8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93741031a0 0x7f93741987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f937b2a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9374103b60 0x7f937419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f937b2a8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f937410daf0 con 0x7f93741031a0 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f937b2a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f937410d970 con 0x7f9374101fb0 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f937b2a8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f937410dc70 con 0x7f9374103b60 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f9378843640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93741031a0 0x7f93741987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f9378843640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93741031a0 0x7f93741987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40004/0 (socket says 172.21.15.7:40004) 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f9378843640 1 -- 172.21.15.7:0/2550480873 learned_addr learned my addr 172.21.15.7:0/2550480873 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f9379044640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9374101fb0 0x7f9374198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f9379845640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9374103b60 0x7f937419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f9378843640 1 -- 172.21.15.7:0/2550480873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9374103b60 msgr2=0x7f937419f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f9378843640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9374103b60 0x7f937419f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f9378843640 1 -- 172.21.15.7:0/2550480873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9374101fb0 msgr2=0x7f9374198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f9378843640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9374101fb0 0x7f9374198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f9378843640 1 -- 172.21.15.7:0/2550480873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f937419fe80 con 0x7f93741031a0 2024-04-21T08:42:31.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.056+0000 7f9379044640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9374101fb0 0x7f9374198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:31.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.057+0000 7f9378843640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93741031a0 0x7f93741987a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f936400e9f0 tx=0x7f936400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:31.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.057+0000 7f936a7fc640 1 -- 172.21.15.7:0/2550480873 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936400cd80 con 0x7f93741031a0 2024-04-21T08:42:31.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.057+0000 7f936a7fc640 1 -- 172.21.15.7:0/2550480873 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93640054d0 con 0x7f93741031a0 2024-04-21T08:42:31.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.057+0000 7f937b2a8640 1 -- 172.21.15.7:0/2550480873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93741a0170 con 0x7f93741031a0 2024-04-21T08:42:31.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.057+0000 7f936a7fc640 1 -- 172.21.15.7:0/2550480873 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364010690 con 0x7f93741031a0 2024-04-21T08:42:31.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.057+0000 7f937b2a8640 1 -- 172.21.15.7:0/2550480873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93741a06c0 con 0x7f93741031a0 2024-04-21T08:42:31.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.058+0000 7f937b2a8640 1 -- 172.21.15.7:0/2550480873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93741093f0 con 0x7f93741031a0 2024-04-21T08:42:31.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.059+0000 7f936a7fc640 1 -- 172.21.15.7:0/2550480873 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f9364004e30 con 0x7f93741031a0 2024-04-21T08:42:31.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.060+0000 7f936a7fc640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93600a3c30 0x7f93600a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.060+0000 7f9379044640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93600a3c30 0x7f93600a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.060+0000 7f936a7fc640 1 -- 172.21.15.7:0/2550480873 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f93640d0d20 con 0x7f93741031a0 2024-04-21T08:42:31.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.060+0000 7f9379044640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93600a3c30 0x7f93600a60f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f937002c700 tx=0x7f93700324c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:31.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.064+0000 7f936a7fc640 1 -- 172.21.15.7:0/2550480873 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9364053ff0 con 0x7f93741031a0 2024-04-21T08:42:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:30 smithi077 ceph-mon[31615]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-04-21T08:42:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:30 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3983685806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:30 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1532128670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:30 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1955223087' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:42:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:30 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2848320571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:42:31.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.263+0000 7f937b2a8640 1 -- 172.21.15.7:0/2550480873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f937402d810 con 0x7f93741031a0 2024-04-21T08:42:31.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.263+0000 7f936a7fc640 1 -- 172.21.15.7:0/2550480873 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+579 (secure 0 0 0) 0x7f936401f340 con 0x7f93741031a0 2024-04-21T08:42:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/1764088272 2024-04-22T08:35:09.541897+0000 2024-04-21T08:42:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3825837962 2024-04-22T08:35:09.541897+0000 2024-04-21T08:42:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2463266160 2024-04-22T08:35:09.541897+0000 2024-04-21T08:42:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3631314876 2024-04-22T08:34:42.863861+0000 2024-04-21T08:42:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/102274072 2024-04-22T08:35:09.541897+0000 2024-04-21T08:42:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/2031045761 2024-04-22T08:34:42.863861+0000 2024-04-21T08:42:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/2031045761 2024-04-22T08:34:42.863861+0000 2024-04-21T08:42:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/865111185 2024-04-22T08:34:42.863861+0000 2024-04-21T08:42:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2778026231 2024-04-22T08:34:42.863861+0000 2024-04-21T08:42:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/102274072 2024-04-22T08:35:09.541897+0000 2024-04-21T08:42:31.263 INFO:teuthology.orchestra.run.smithi007.stderr:listed 10 entries 2024-04-21T08:42:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 -- 172.21.15.7:0/2550480873 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93600a3c30 msgr2=0x7f93600a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93600a3c30 0x7f93600a60f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f937002c700 tx=0x7f93700324c0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 -- 172.21.15.7:0/2550480873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93741031a0 msgr2=0x7f93741987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93741031a0 0x7f93741987a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f936400e9f0 tx=0x7f936400edf0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 -- 172.21.15.7:0/2550480873 shutdown_connections 2024-04-21T08:42:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93600a3c30 0x7f93600a60f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9374103b60 0x7f937419f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93741031a0 0x7f93741987a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 --2- 172.21.15.7:0/2550480873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9374101fb0 0x7f9374198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 -- 172.21.15.7:0/2550480873 >> 172.21.15.7:0/2550480873 conn(0x7f93740fd520 msgr2=0x7f93741065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:31.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 -- 172.21.15.7:0/2550480873 shutdown_connections 2024-04-21T08:42:31.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.265+0000 7f937b2a8640 1 -- 172.21.15.7:0/2550480873 wait complete. 2024-04-21T08:42:31.275 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:42:31.275 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:42:31.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:42:31.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.434+0000 7f896e814640 1 -- 172.21.15.7:0/3150536860 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89681053f0 msgr2=0x7f89681077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.434+0000 7f896e814640 1 --2- 172.21.15.7:0/3150536860 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89681053f0 0x7f89681077e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f895c009900 tx=0x7f895c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.434+0000 7f896e814640 1 -- 172.21.15.7:0/3150536860 shutdown_connections 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.434+0000 7f896e814640 1 --2- 172.21.15.7:0/3150536860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8968107d20 0x7f896810a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.434+0000 7f896e814640 1 --2- 172.21.15.7:0/3150536860 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89681053f0 0x7f89681077e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.434+0000 7f896e814640 1 --2- 172.21.15.7:0/3150536860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968102ac0 0x7f8968104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.434+0000 7f896e814640 1 -- 172.21.15.7:0/3150536860 >> 172.21.15.7:0/3150536860 conn(0x7f89680fc410 msgr2=0x7f89680fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.434+0000 7f896e814640 1 -- 172.21.15.7:0/3150536860 shutdown_connections 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.434+0000 7f896e814640 1 -- 172.21.15.7:0/3150536860 wait complete. 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.434+0000 7f896e814640 1 Processor -- start 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.434+0000 7f896e814640 1 -- start start 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896e814640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968102ac0 0x7f89681981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896e814640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89681053f0 0x7f89681986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896e814640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8968107d20 0x7f896819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896e814640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f896810dac0 con 0x7f8968107d20 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896e814640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f896810d940 con 0x7f8968102ac0 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896e814640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f896810dc40 con 0x7f89681053f0 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896cdb1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8968107d20 0x7f896819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896cdb1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8968107d20 0x7f896819f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40030/0 (socket says 172.21.15.7:40030) 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896cdb1640 1 -- 172.21.15.7:0/4124913060 learned_addr learned my addr 172.21.15.7:0/4124913060 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f8967fff640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968102ac0 0x7f89681981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f89677fe640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89681053f0 0x7f89681986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896cdb1640 1 -- 172.21.15.7:0/4124913060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89681053f0 msgr2=0x7f89681986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896cdb1640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89681053f0 0x7f89681986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896cdb1640 1 -- 172.21.15.7:0/4124913060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968102ac0 msgr2=0x7f89681981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896cdb1640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968102ac0 0x7f89681981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896cdb1640 1 -- 172.21.15.7:0/4124913060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896819fdc0 con 0x7f8968107d20 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f89677fe640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89681053f0 0x7f89681986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:31.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.435+0000 7f896cdb1640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8968107d20 0x7f896819f750 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f896000eaa0 tx=0x7f896000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:31.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.436+0000 7f89657fa640 1 -- 172.21.15.7:0/4124913060 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896000cdb0 con 0x7f8968107d20 2024-04-21T08:42:31.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.436+0000 7f89657fa640 1 -- 172.21.15.7:0/4124913060 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8960005020 con 0x7f8968107d20 2024-04-21T08:42:31.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.436+0000 7f896e814640 1 -- 172.21.15.7:0/4124913060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89681a00b0 con 0x7f8968107d20 2024-04-21T08:42:31.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.436+0000 7f89657fa640 1 -- 172.21.15.7:0/4124913060 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89600116d0 con 0x7f8968107d20 2024-04-21T08:42:31.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.436+0000 7f896e814640 1 -- 172.21.15.7:0/4124913060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89681a0580 con 0x7f8968107d20 2024-04-21T08:42:31.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.437+0000 7f896e814640 1 -- 172.21.15.7:0/4124913060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89680fdf00 con 0x7f8968107d20 2024-04-21T08:42:31.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.438+0000 7f89657fa640 1 -- 172.21.15.7:0/4124913060 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f89600051d0 con 0x7f8968107d20 2024-04-21T08:42:31.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.438+0000 7f89657fa640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89500a3bb0 0x7f89500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.438+0000 7f8967fff640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89500a3bb0 0x7f89500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.438+0000 7f89657fa640 1 -- 172.21.15.7:0/4124913060 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f89600d07d0 con 0x7f8968107d20 2024-04-21T08:42:31.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.439+0000 7f8967fff640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89500a3bb0 0x7f89500a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8958005d00 tx=0x7f895800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:31.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.443+0000 7f89657fa640 1 -- 172.21.15.7:0/4124913060 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8960016030 con 0x7f8968107d20 2024-04-21T08:42:31.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.631+0000 7f896e814640 1 -- 172.21.15.7:0/4124913060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8968062b50 con 0x7f8968107d20 2024-04-21T08:42:31.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.632+0000 7f89657fa640 1 -- 172.21.15.7:0/4124913060 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23961 (secure 0 0 0) 0x7f896009cf70 con 0x7f8968107d20 2024-04-21T08:42:31.632 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:31.632 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":90,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:42:22.023111+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:31.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 -- 172.21.15.7:0/4124913060 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89500a3bb0 msgr2=0x7f89500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89500a3bb0 0x7f89500a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8958005d00 tx=0x7f895800b410 comp rx=0 tx=0).stop 2024-04-21T08:42:31.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 -- 172.21.15.7:0/4124913060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8968107d20 msgr2=0x7f896819f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8968107d20 0x7f896819f750 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f896000eaa0 tx=0x7f896000ef60 comp rx=0 tx=0).stop 2024-04-21T08:42:31.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 -- 172.21.15.7:0/4124913060 shutdown_connections 2024-04-21T08:42:31.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89500a3bb0 0x7f89500a6070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8968107d20 0x7f896819f750 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89681053f0 0x7f89681986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 --2- 172.21.15.7:0/4124913060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8968102ac0 0x7f89681981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 -- 172.21.15.7:0/4124913060 >> 172.21.15.7:0/4124913060 conn(0x7f89680fc410 msgr2=0x7f89681034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:31.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 -- 172.21.15.7:0/4124913060 shutdown_connections 2024-04-21T08:42:31.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.635+0000 7f896e814640 1 -- 172.21.15.7:0/4124913060 wait complete. 2024-04-21T08:42:31.645 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:42:31.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:42:31.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.752+0000 7f9f55d3b640 1 -- 172.21.15.7:0/791817499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f50107c70 msgr2=0x7f9f5010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.752+0000 7f9f55d3b640 1 --2- 172.21.15.7:0/791817499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f50107c70 0x7f9f5010a0c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009ca0 tx=0x7f9f4802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.753+0000 7f9f55d3b640 1 -- 172.21.15.7:0/791817499 shutdown_connections 2024-04-21T08:42:31.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.753+0000 7f9f55d3b640 1 --2- 172.21.15.7:0/791817499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f50107c70 0x7f9f5010a0c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.753+0000 7f9f55d3b640 1 --2- 172.21.15.7:0/791817499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f50105340 0x7f9f50107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.753+0000 7f9f55d3b640 1 --2- 172.21.15.7:0/791817499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f50102a10 0x7f9f50104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.753+0000 7f9f55d3b640 1 -- 172.21.15.7:0/791817499 >> 172.21.15.7:0/791817499 conn(0x7f9f500fc420 msgr2=0x7f9f500fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.753+0000 7f9f55d3b640 1 -- 172.21.15.7:0/791817499 shutdown_connections 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.753+0000 7f9f55d3b640 1 -- 172.21.15.7:0/791817499 wait complete. 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.753+0000 7f9f55d3b640 1 Processor -- start 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.753+0000 7f9f55d3b640 1 -- start start 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f55d3b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f50102a10 0x7f9f50102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f55d3b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f50105340 0x7f9f501009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f55d3b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f50107c70 0x7f9f50100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f55d3b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5010dad0 con 0x7f9f50107c70 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f55d3b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5010d950 con 0x7f9f50102a10 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f55d3b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5010dc50 con 0x7f9f50105340 2024-04-21T08:42:31.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f50107c70 0x7f9f50100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f50107c70 0x7f9f50100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40052/0 (socket says 172.21.15.7:40052) 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4ffff640 1 -- 172.21.15.7:0/3330465784 learned_addr learned my addr 172.21.15.7:0/3330465784 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4f7fe640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f50102a10 0x7f9f50102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4effd640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f50105340 0x7f9f501009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4ffff640 1 -- 172.21.15.7:0/3330465784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f50105340 msgr2=0x7f9f501009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4ffff640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f50105340 0x7f9f501009c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4ffff640 1 -- 172.21.15.7:0/3330465784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f50102a10 msgr2=0x7f9f50102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4ffff640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f50102a10 0x7f9f50102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4ffff640 1 -- 172.21.15.7:0/3330465784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f50101790 con 0x7f9f50107c70 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4effd640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f50105340 0x7f9f501009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4ffff640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f50107c70 0x7f9f50100f00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9f4802c700 tx=0x7f9f480093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4cff9640 1 -- 172.21.15.7:0/3330465784 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f48041020 con 0x7f9f50107c70 2024-04-21T08:42:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4cff9640 1 -- 172.21.15.7:0/3330465784 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f48034d60 con 0x7f9f50107c70 2024-04-21T08:42:31.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f55d3b640 1 -- 172.21.15.7:0/3330465784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f501a2290 con 0x7f9f50107c70 2024-04-21T08:42:31.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f4cff9640 1 -- 172.21.15.7:0/3330465784 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f48039040 con 0x7f9f50107c70 2024-04-21T08:42:31.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.754+0000 7f9f55d3b640 1 -- 172.21.15.7:0/3330465784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f501a2830 con 0x7f9f50107c70 2024-04-21T08:42:31.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.755+0000 7f9f55d3b640 1 -- 172.21.15.7:0/3330465784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f500fdf10 con 0x7f9f50107c70 2024-04-21T08:42:31.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.757+0000 7f9f4cff9640 1 -- 172.21.15.7:0/3330465784 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f9f48044020 con 0x7f9f50107c70 2024-04-21T08:42:31.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.757+0000 7f9f4cff9640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9f380a3c30 0x7f9f380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.757+0000 7f9f4f7fe640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9f380a3c30 0x7f9f380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.757+0000 7f9f4cff9640 1 -- 172.21.15.7:0/3330465784 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f9f480f29c0 con 0x7f9f50107c70 2024-04-21T08:42:31.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.758+0000 7f9f4f7fe640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9f380a3c30 0x7f9f380a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9f40005d00 tx=0x7f9f4000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:31.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.761+0000 7f9f4cff9640 1 -- 172.21.15.7:0/3330465784 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9f4804b020 con 0x7f9f50107c70 2024-04-21T08:42:31.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:31 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1719312616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:42:31.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:31 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2550480873' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:42:31.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:31 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4124913060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:31.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:31 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1719312616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:42:31.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:31 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2550480873' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:42:31.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:31 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4124913060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:31.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.950+0000 7f9f55d3b640 1 -- 172.21.15.7:0/3330465784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9f5002d5c0 con 0x7f9f50107c70 2024-04-21T08:42:31.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.951+0000 7f9f4cff9640 1 -- 172.21.15.7:0/3330465784 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f9f480bf160 con 0x7f9f50107c70 2024-04-21T08:42:31.950 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:42:31.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 -- 172.21.15.7:0/3330465784 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9f380a3c30 msgr2=0x7f9f380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9f380a3c30 0x7f9f380a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9f40005d00 tx=0x7f9f4000b410 comp rx=0 tx=0).stop 2024-04-21T08:42:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 -- 172.21.15.7:0/3330465784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f50107c70 msgr2=0x7f9f50100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f50107c70 0x7f9f50100f00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9f4802c700 tx=0x7f9f480093f0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 -- 172.21.15.7:0/3330465784 shutdown_connections 2024-04-21T08:42:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9f380a3c30 0x7f9f380a60f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f50107c70 0x7f9f50100f00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f50105340 0x7f9f501009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 --2- 172.21.15.7:0/3330465784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f50102a10 0x7f9f50102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 -- 172.21.15.7:0/3330465784 >> 172.21.15.7:0/3330465784 conn(0x7f9f500fc420 msgr2=0x7f9f50103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 -- 172.21.15.7:0/3330465784 shutdown_connections 2024-04-21T08:42:31.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:31.953+0000 7f9f55d3b640 1 -- 172.21.15.7:0/3330465784 wait complete. 2024-04-21T08:42:31.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:42:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.068+0000 7fb93bea5640 1 -- 172.21.15.7:0/3528708590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9341031a0 msgr2=0x7fb934103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.068+0000 7fb93bea5640 1 --2- 172.21.15.7:0/3528708590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9341031a0 0x7fb934103620 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb924009900 tx=0x7fb92402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.068+0000 7fb93bea5640 1 -- 172.21.15.7:0/3528708590 shutdown_connections 2024-04-21T08:42:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.068+0000 7fb93bea5640 1 --2- 172.21.15.7:0/3528708590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb934103b60 0x7fb93410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.068+0000 7fb93bea5640 1 --2- 172.21.15.7:0/3528708590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9341031a0 0x7fb934103620 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.068+0000 7fb93bea5640 1 --2- 172.21.15.7:0/3528708590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb934101fb0 0x7fb9341023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.068+0000 7fb93bea5640 1 -- 172.21.15.7:0/3528708590 >> 172.21.15.7:0/3528708590 conn(0x7fb9340fd520 msgr2=0x7fb9340ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb93bea5640 1 -- 172.21.15.7:0/3528708590 shutdown_connections 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb93bea5640 1 -- 172.21.15.7:0/3528708590 wait complete. 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb93bea5640 1 Processor -- start 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb93bea5640 1 -- start start 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb93bea5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb934101fb0 0x7fb9341981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb93bea5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9341031a0 0x7fb934198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb93bea5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb934103b60 0x7fb93419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb93bea5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb93410daf0 con 0x7fb934101fb0 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb93bea5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb93410d970 con 0x7fb934103b60 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb93bea5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb93410dc70 con 0x7fb9341031a0 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb939c41640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb934101fb0 0x7fb9341981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb939c41640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb934101fb0 0x7fb9341981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40080/0 (socket says 172.21.15.7:40080) 2024-04-21T08:42:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.069+0000 7fb939c41640 1 -- 172.21.15.7:0/2279245575 learned_addr learned my addr 172.21.15.7:0/2279245575 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb93a442640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb934103b60 0x7fb93419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb939440640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9341031a0 0x7fb934198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb939c41640 1 -- 172.21.15.7:0/2279245575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9341031a0 msgr2=0x7fb934198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb939c41640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9341031a0 0x7fb934198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb939c41640 1 -- 172.21.15.7:0/2279245575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb934103b60 msgr2=0x7fb93419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb939c41640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb934103b60 0x7fb93419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb939c41640 1 -- 172.21.15.7:0/2279245575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb93419fdf0 con 0x7fb934101fb0 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb939440640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9341031a0 0x7fb934198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb939c41640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb934101fb0 0x7fb9341981d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb93000d770 tx=0x7fb93000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb92affd640 1 -- 172.21.15.7:0/2279245575 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930017070 con 0x7fb934101fb0 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb92affd640 1 -- 172.21.15.7:0/2279245575 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb930009c30 con 0x7fb934101fb0 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb93bea5640 1 -- 172.21.15.7:0/2279245575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9341a00e0 con 0x7fb934101fb0 2024-04-21T08:42:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb92affd640 1 -- 172.21.15.7:0/2279245575 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930003cc0 con 0x7fb934101fb0 2024-04-21T08:42:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.070+0000 7fb93bea5640 1 -- 172.21.15.7:0/2279245575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9341a0630 con 0x7fb934101fb0 2024-04-21T08:42:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.071+0000 7fb93bea5640 1 -- 172.21.15.7:0/2279245575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9341093a0 con 0x7fb934101fb0 2024-04-21T08:42:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.072+0000 7fb92affd640 1 -- 172.21.15.7:0/2279245575 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fb930003e60 con 0x7fb934101fb0 2024-04-21T08:42:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.073+0000 7fb92affd640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb9200a3bb0 0x7fb9200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.073+0000 7fb939440640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb9200a3bb0 0x7fb9200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.073+0000 7fb92affd640 1 -- 172.21.15.7:0/2279245575 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb9300d0470 con 0x7fb934101fb0 2024-04-21T08:42:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.073+0000 7fb939440640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb9200a3bb0 0x7fb9200a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb9240096c0 tx=0x7fb924000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.076+0000 7fb92affd640 1 -- 172.21.15.7:0/2279245575 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb930009de0 con 0x7fb934101fb0 2024-04-21T08:42:32.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:31 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1719312616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:42:32.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:31 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2550480873' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:42:32.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:31 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4124913060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:32.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.261+0000 7fb93bea5640 1 -- 172.21.15.7:0/2279245575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb934062b80 con 0x7fb934101fb0 2024-04-21T08:42:32.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.262+0000 7fb92affd640 1 -- 172.21.15.7:0/2279245575 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fb93009cbb0 con 0x7fb934101fb0 2024-04-21T08:42:32.261 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:42:32.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 -- 172.21.15.7:0/2279245575 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb9200a3bb0 msgr2=0x7fb9200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb9200a3bb0 0x7fb9200a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb9240096c0 tx=0x7fb924000bc0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 -- 172.21.15.7:0/2279245575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb934101fb0 msgr2=0x7fb9341981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb934101fb0 0x7fb9341981d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb93000d770 tx=0x7fb93000dc30 comp rx=0 tx=0).stop 2024-04-21T08:42:32.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 -- 172.21.15.7:0/2279245575 shutdown_connections 2024-04-21T08:42:32.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb9200a3bb0 0x7fb9200a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb934103b60 0x7fb93419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9341031a0 0x7fb934198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 --2- 172.21.15.7:0/2279245575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb934101fb0 0x7fb9341981d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 -- 172.21.15.7:0/2279245575 >> 172.21.15.7:0/2279245575 conn(0x7fb9340fd520 msgr2=0x7fb934106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:32.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 -- 172.21.15.7:0/2279245575 shutdown_connections 2024-04-21T08:42:32.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.264+0000 7fb93bea5640 1 -- 172.21.15.7:0/2279245575 wait complete. 2024-04-21T08:42:32.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:42:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.380+0000 7fd912406640 1 -- 172.21.15.7:0/2352898697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c105400 msgr2=0x7fd90c1077f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.380+0000 7fd912406640 1 --2- 172.21.15.7:0/2352898697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c105400 0x7fd90c1077f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd900009900 tx=0x7fd90002bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.380+0000 7fd912406640 1 -- 172.21.15.7:0/2352898697 shutdown_connections 2024-04-21T08:42:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.380+0000 7fd912406640 1 --2- 172.21.15.7:0/2352898697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c107d30 0x7fd90c10a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.380+0000 7fd912406640 1 --2- 172.21.15.7:0/2352898697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c105400 0x7fd90c1077f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.380+0000 7fd912406640 1 --2- 172.21.15.7:0/2352898697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c102ad0 0x7fd90c104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.380+0000 7fd912406640 1 -- 172.21.15.7:0/2352898697 >> 172.21.15.7:0/2352898697 conn(0x7fd90c0fc420 msgr2=0x7fd90c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.380+0000 7fd912406640 1 -- 172.21.15.7:0/2352898697 shutdown_connections 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.380+0000 7fd912406640 1 -- 172.21.15.7:0/2352898697 wait complete. 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd912406640 1 Processor -- start 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd912406640 1 -- start start 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd912406640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c102ad0 0x7fd90c102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd912406640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c105400 0x7fd90c100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd912406640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c107d30 0x7fd90c100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd912406640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c10dae0 con 0x7fd90c102ad0 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd912406640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c10d960 con 0x7fd90c105400 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd912406640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd90c10dc60 con 0x7fd90c107d30 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd90bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c102ad0 0x7fd90c102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd90bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c102ad0 0x7fd90c102440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40092/0 (socket says 172.21.15.7:40092) 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd90bfff640 1 -- 172.21.15.7:0/1416476568 learned_addr learned my addr 172.21.15.7:0/1416476568 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd9109a3640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c107d30 0x7fd90c100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd90b7fe640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c105400 0x7fd90c100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd90bfff640 1 -- 172.21.15.7:0/1416476568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c107d30 msgr2=0x7fd90c100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.381+0000 7fd90bfff640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c107d30 0x7fd90c100fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.382+0000 7fd90bfff640 1 -- 172.21.15.7:0/1416476568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c105400 msgr2=0x7fd90c100a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.382+0000 7fd90bfff640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c105400 0x7fd90c100a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.382+0000 7fd90bfff640 1 -- 172.21.15.7:0/1416476568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd90c101850 con 0x7fd90c102ad0 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.382+0000 7fd90bfff640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c102ad0 0x7fd90c102440 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00d950 tx=0x7fd8fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.382+0000 7fd9097fa640 1 -- 172.21.15.7:0/1416476568 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc017030 con 0x7fd90c102ad0 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.382+0000 7fd9097fa640 1 -- 172.21.15.7:0/1416476568 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8fc004020 con 0x7fd90c102ad0 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.382+0000 7fd912406640 1 -- 172.21.15.7:0/1416476568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd90c1a2370 con 0x7fd90c102ad0 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.382+0000 7fd9097fa640 1 -- 172.21.15.7:0/1416476568 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc0047a0 con 0x7fd90c102ad0 2024-04-21T08:42:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.382+0000 7fd912406640 1 -- 172.21.15.7:0/1416476568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd90c1a2890 con 0x7fd90c102ad0 2024-04-21T08:42:32.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.383+0000 7fd912406640 1 -- 172.21.15.7:0/1416476568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd90c068e00 con 0x7fd90c102ad0 2024-04-21T08:42:32.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.384+0000 7fd9097fa640 1 -- 172.21.15.7:0/1416476568 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd8fc009a30 con 0x7fd90c102ad0 2024-04-21T08:42:32.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.385+0000 7fd9097fa640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8f40a3bb0 0x7fd8f40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.385+0000 7fd90b7fe640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8f40a3bb0 0x7fd8f40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.385+0000 7fd9097fa640 1 -- 172.21.15.7:0/1416476568 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd8fc0d0150 con 0x7fd90c102ad0 2024-04-21T08:42:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.386+0000 7fd90b7fe640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8f40a3bb0 0x7fd8f40a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd900006010 tx=0x7fd900005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.390+0000 7fd9097fa640 1 -- 172.21.15.7:0/1416476568 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8fc09c8f0 con 0x7fd90c102ad0 2024-04-21T08:42:32.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.579+0000 7fd912406640 1 -- 172.21.15.7:0/1416476568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd90c062b50 con 0x7fd90c102ad0 2024-04-21T08:42:32.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.579+0000 7fd9097fa640 1 -- 172.21.15.7:0/1416476568 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fd8fc09c1f0 con 0x7fd90c102ad0 2024-04-21T08:42:32.579 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:42:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.581+0000 7fd912406640 1 -- 172.21.15.7:0/1416476568 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8f40a3bb0 msgr2=0x7fd8f40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.581+0000 7fd912406640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8f40a3bb0 0x7fd8f40a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd900006010 tx=0x7fd900005be0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.581+0000 7fd912406640 1 -- 172.21.15.7:0/1416476568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c102ad0 msgr2=0x7fd90c102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.581+0000 7fd912406640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c102ad0 0x7fd90c102440 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00d950 tx=0x7fd8fc00de10 comp rx=0 tx=0).stop 2024-04-21T08:42:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.582+0000 7fd912406640 1 -- 172.21.15.7:0/1416476568 shutdown_connections 2024-04-21T08:42:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.582+0000 7fd912406640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8f40a3bb0 0x7fd8f40a6070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.582+0000 7fd912406640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c107d30 0x7fd90c100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.582+0000 7fd912406640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd90c105400 0x7fd90c100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.582+0000 7fd912406640 1 --2- 172.21.15.7:0/1416476568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c102ad0 0x7fd90c102440 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.582+0000 7fd912406640 1 -- 172.21.15.7:0/1416476568 >> 172.21.15.7:0/1416476568 conn(0x7fd90c0fc420 msgr2=0x7fd90c1035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.582+0000 7fd912406640 1 -- 172.21.15.7:0/1416476568 shutdown_connections 2024-04-21T08:42:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.582+0000 7fd912406640 1 -- 172.21.15.7:0/1416476568 wait complete. 2024-04-21T08:42:32.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:42:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.699+0000 7f5d3c74b640 1 -- 172.21.15.7:0/2712799913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d34101fa0 msgr2=0x7f5d341023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.699+0000 7f5d3c74b640 1 --2- 172.21.15.7:0/2712799913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d34101fa0 0x7f5d341023c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c009900 tx=0x7f5d2c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.699+0000 7f5d3c74b640 1 -- 172.21.15.7:0/2712799913 shutdown_connections 2024-04-21T08:42:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.699+0000 7f5d3c74b640 1 --2- 172.21.15.7:0/2712799913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d34103b50 0x7f5d3410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.699+0000 7f5d3c74b640 1 --2- 172.21.15.7:0/2712799913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d34103190 0x7f5d34103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.699+0000 7f5d3c74b640 1 --2- 172.21.15.7:0/2712799913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d34101fa0 0x7f5d341023c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.699+0000 7f5d3c74b640 1 -- 172.21.15.7:0/2712799913 >> 172.21.15.7:0/2712799913 conn(0x7f5d340fd510 msgr2=0x7f5d340ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.699+0000 7f5d3c74b640 1 -- 172.21.15.7:0/2712799913 shutdown_connections 2024-04-21T08:42:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.699+0000 7f5d3c74b640 1 -- 172.21.15.7:0/2712799913 wait complete. 2024-04-21T08:42:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3c74b640 1 Processor -- start 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3c74b640 1 -- start start 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3c74b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d34101fa0 0x7f5d34198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3c74b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d34103190 0x7f5d341987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3c74b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d34103b50 0x7f5d3419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3c74b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3410db90 con 0x7f5d34103b50 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3c74b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3410da10 con 0x7f5d34101fa0 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3c74b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3410dd10 con 0x7f5d34103190 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3ace8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d34103b50 0x7f5d3419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3ace8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d34103b50 0x7f5d3419f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40110/0 (socket says 172.21.15.7:40110) 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3ace8640 1 -- 172.21.15.7:0/3656480240 learned_addr learned my addr 172.21.15.7:0/3656480240 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d39ce6640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d34103190 0x7f5d341987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.700+0000 7f5d3a4e7640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d34101fa0 0x7f5d34198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.701+0000 7f5d3ace8640 1 -- 172.21.15.7:0/3656480240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d34103190 msgr2=0x7f5d341987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.701+0000 7f5d3ace8640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d34103190 0x7f5d341987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.701+0000 7f5d3ace8640 1 -- 172.21.15.7:0/3656480240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d34101fa0 msgr2=0x7f5d34198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.701+0000 7f5d3ace8640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d34101fa0 0x7f5d34198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.701+0000 7f5d3ace8640 1 -- 172.21.15.7:0/3656480240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3419fe80 con 0x7f5d34103b50 2024-04-21T08:42:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.701+0000 7f5d3ace8640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d34103b50 0x7f5d3419f810 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000eaa0 tx=0x7f5d3000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.701+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/3656480240 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30004d20 con 0x7f5d34103b50 2024-04-21T08:42:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.701+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/3656480240 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d30005020 con 0x7f5d34103b50 2024-04-21T08:42:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.701+0000 7f5d3c74b640 1 -- 172.21.15.7:0/3656480240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d341a0170 con 0x7f5d34103b50 2024-04-21T08:42:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.701+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/3656480240 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d300116b0 con 0x7f5d34103b50 2024-04-21T08:42:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.701+0000 7f5d3c74b640 1 -- 172.21.15.7:0/3656480240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d341a0790 con 0x7f5d34103b50 2024-04-21T08:42:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.702+0000 7f5d3c74b640 1 -- 172.21.15.7:0/3656480240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d34109340 con 0x7f5d34103b50 2024-04-21T08:42:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.703+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/3656480240 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f5d300051d0 con 0x7f5d34103b50 2024-04-21T08:42:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.704+0000 7f5d2b7fe640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5d200a3c30 0x7f5d200a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.704+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/3656480240 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f5d300d0770 con 0x7f5d34103b50 2024-04-21T08:42:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.704+0000 7f5d3a4e7640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5d200a3c30 0x7f5d200a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:32.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.704+0000 7f5d3a4e7640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5d200a3c30 0x7f5d200a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c02c240 tx=0x7f5d2c02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:32.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.708+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/3656480240 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d30016030 con 0x7f5d34103b50 2024-04-21T08:42:32.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.894+0000 7f5d3c74b640 1 -- 172.21.15.7:0/3656480240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5d34062b50 con 0x7f5d34103b50 2024-04-21T08:42:32.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.895+0000 7f5d2b7fe640 1 -- 172.21.15.7:0/3656480240 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f5d3009ceb0 con 0x7f5d34103b50 2024-04-21T08:42:32.894 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:42:32.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.896+0000 7f5d3c74b640 1 -- 172.21.15.7:0/3656480240 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5d200a3c30 msgr2=0x7f5d200a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.896+0000 7f5d3c74b640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5d200a3c30 0x7f5d200a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c02c240 tx=0x7f5d2c02c7e0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.896+0000 7f5d3c74b640 1 -- 172.21.15.7:0/3656480240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d34103b50 msgr2=0x7f5d3419f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.896+0000 7f5d3c74b640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d34103b50 0x7f5d3419f810 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000eaa0 tx=0x7f5d3000eea0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.896+0000 7f5d3c74b640 1 -- 172.21.15.7:0/3656480240 shutdown_connections 2024-04-21T08:42:32.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.896+0000 7f5d3c74b640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5d200a3c30 0x7f5d200a60f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.896+0000 7f5d3c74b640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d34103b50 0x7f5d3419f810 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.896+0000 7f5d3c74b640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d34103190 0x7f5d341987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.896+0000 7f5d3c74b640 1 --2- 172.21.15.7:0/3656480240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d34101fa0 0x7f5d34198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.896+0000 7f5d3c74b640 1 -- 172.21.15.7:0/3656480240 >> 172.21.15.7:0/3656480240 conn(0x7f5d340fd510 msgr2=0x7f5d34106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:32.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.897+0000 7f5d3c74b640 1 -- 172.21.15.7:0/3656480240 shutdown_connections 2024-04-21T08:42:32.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:32.897+0000 7f5d3c74b640 1 -- 172.21.15.7:0/3656480240 wait complete. 2024-04-21T08:42:32.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:32 smithi007 ceph-mon[28477]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 410 B/s wr, 0 op/s 2024-04-21T08:42:32.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:32 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3330465784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:32.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:32 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2279245575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:42:32.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:32 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1416476568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:42:32.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.010+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1393664431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e0107c80 msgr2=0x7f92e010a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.010+0000 7f92e4ed2640 1 --2- 172.21.15.7:0/1393664431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e0107c80 0x7f92e010a0d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009ca0 tx=0x7f92d802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.011+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1393664431 shutdown_connections 2024-04-21T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.011+0000 7f92e4ed2640 1 --2- 172.21.15.7:0/1393664431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e0107c80 0x7f92e010a0d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.011+0000 7f92e4ed2640 1 --2- 172.21.15.7:0/1393664431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0105350 0x7f92e0107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.011+0000 7f92e4ed2640 1 --2- 172.21.15.7:0/1393664431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0102a20 0x7f92e0104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.011+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1393664431 >> 172.21.15.7:0/1393664431 conn(0x7f92e00fc430 msgr2=0x7f92e00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.011+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1393664431 shutdown_connections 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.011+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1393664431 wait complete. 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.011+0000 7f92e4ed2640 1 Processor -- start 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.011+0000 7f92e4ed2640 1 -- start start 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92e4ed2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e0102a20 0x7f92e0102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92e4ed2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0105350 0x7f92e01009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92e4ed2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0107c80 0x7f92e0100f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92e4ed2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f92e010dae0 con 0x7f92e0102a20 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92e4ed2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f92e010d960 con 0x7f92e0107c80 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92e4ed2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f92e010dc60 con 0x7f92e0105350 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92de59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e0102a20 0x7f92e0102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92de59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e0102a20 0x7f92e0102390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40130/0 (socket says 172.21.15.7:40130) 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92de59c640 1 -- 172.21.15.7:0/1700730115 learned_addr learned my addr 172.21.15.7:0/1700730115 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92ddd9b640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0105350 0x7f92e01009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92ded9d640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0107c80 0x7f92e0100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92de59c640 1 -- 172.21.15.7:0/1700730115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0105350 msgr2=0x7f92e01009d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92de59c640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0105350 0x7f92e01009d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92de59c640 1 -- 172.21.15.7:0/1700730115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0107c80 msgr2=0x7f92e0100f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92de59c640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0107c80 0x7f92e0100f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92de59c640 1 -- 172.21.15.7:0/1700730115 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e01017a0 con 0x7f92e0102a20 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92de59c640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e0102a20 0x7f92e0102390 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f92d000d950 tx=0x7f92d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92cf7fe640 1 -- 172.21.15.7:0/1700730115 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d0017070 con 0x7f92e0102a20 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92cf7fe640 1 -- 172.21.15.7:0/1700730115 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92d0009c30 con 0x7f92e0102a20 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1700730115 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e01a22b0 con 0x7f92e0102a20 2024-04-21T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92cf7fe640 1 -- 172.21.15.7:0/1700730115 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d000f620 con 0x7f92e0102a20 2024-04-21T08:42:33.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.012+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1700730115 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92e01a2850 con 0x7f92e0102a20 2024-04-21T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.014+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1700730115 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92e00fdf20 con 0x7f92e0102a20 2024-04-21T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.015+0000 7f92cf7fe640 1 -- 172.21.15.7:0/1700730115 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f92d0010460 con 0x7f92e0102a20 2024-04-21T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.015+0000 7f92cf7fe640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f92c80a3c30 0x7f92c80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.016+0000 7f92cf7fe640 1 -- 172.21.15.7:0/1700730115 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f92d00d0140 con 0x7f92e0102a20 2024-04-21T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.016+0000 7f92ddd9b640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f92c80a3c30 0x7f92c80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.016+0000 7f92ddd9b640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f92c80a3c30 0x7f92c80a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f92e0102100 tx=0x7f92d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.020+0000 7f92cf7fe640 1 -- 172.21.15.7:0/1700730115 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f92d0009de0 con 0x7f92e0102a20 2024-04-21T08:42:33.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:32 smithi016 ceph-mon[31259]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 410 B/s wr, 0 op/s 2024-04-21T08:42:33.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:32 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3330465784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:33.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:32 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2279245575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:42:33.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:32 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1416476568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:42:33.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:32 smithi077 ceph-mon[31615]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 410 B/s wr, 0 op/s 2024-04-21T08:42:33.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:32 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3330465784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:32 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2279245575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:42:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:32 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1416476568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:42:33.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.219+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1700730115 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f92e002d5c0 con 0x7f92e0102a20 2024-04-21T08:42:33.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.219+0000 7f92cf7fe640 1 -- 172.21.15.7:0/1700730115 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+579 (secure 0 0 0) 0x7f92d009c290 con 0x7f92e0102a20 2024-04-21T08:42:33.219 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/1764088272 2024-04-22T08:35:09.541897+0000 2024-04-21T08:42:33.219 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3825837962 2024-04-22T08:35:09.541897+0000 2024-04-21T08:42:33.219 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2463266160 2024-04-22T08:35:09.541897+0000 2024-04-21T08:42:33.219 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3631314876 2024-04-22T08:34:42.863861+0000 2024-04-21T08:42:33.219 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/102274072 2024-04-22T08:35:09.541897+0000 2024-04-21T08:42:33.219 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/2031045761 2024-04-22T08:34:42.863861+0000 2024-04-21T08:42:33.219 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/2031045761 2024-04-22T08:34:42.863861+0000 2024-04-21T08:42:33.219 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/865111185 2024-04-22T08:34:42.863861+0000 2024-04-21T08:42:33.219 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2778026231 2024-04-22T08:34:42.863861+0000 2024-04-21T08:42:33.219 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/102274072 2024-04-22T08:35:09.541897+0000 2024-04-21T08:42:33.220 INFO:teuthology.orchestra.run.smithi007.stderr:listed 10 entries 2024-04-21T08:42:33.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1700730115 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f92c80a3c30 msgr2=0x7f92c80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:33.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f92c80a3c30 0x7f92c80a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f92e0102100 tx=0x7f92d4005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:33.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1700730115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e0102a20 msgr2=0x7f92e0102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:33.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e0102a20 0x7f92e0102390 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f92d000d950 tx=0x7f92d000de10 comp rx=0 tx=0).stop 2024-04-21T08:42:33.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1700730115 shutdown_connections 2024-04-21T08:42:33.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f92c80a3c30 0x7f92c80a60f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:33.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92e0107c80 0x7f92e0100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:33.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92e0105350 0x7f92e01009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:33.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 --2- 172.21.15.7:0/1700730115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e0102a20 0x7f92e0102390 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:33.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1700730115 >> 172.21.15.7:0/1700730115 conn(0x7f92e00fc430 msgr2=0x7f92e0103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:33.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1700730115 shutdown_connections 2024-04-21T08:42:33.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:33.221+0000 7f92e4ed2640 1 -- 172.21.15.7:0/1700730115 wait complete. 2024-04-21T08:42:33.231 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi007.front.sepia.ceph.com... 2024-04-21T08:42:33.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:33.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:42:33.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:ceph-fuse[70809]: fuse finished with error 0 and tester_r 0 2024-04-21T08:42:33.256 INFO:teuthology.orchestra.run:waiting for 300 2024-04-21T08:42:34.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:33 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3656480240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:42:34.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:33 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1700730115' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:42:34.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:33 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3656480240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:42:34.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:33 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1700730115' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:42:34.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:33 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3656480240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:42:34.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:33 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1700730115' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:42:35.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:34 smithi016 ceph-mon[31259]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 280 B/s wr, 0 op/s 2024-04-21T08:42:35.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:34 smithi077 ceph-mon[31615]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 280 B/s wr, 0 op/s 2024-04-21T08:42:35.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:34 smithi007 ceph-mon[28477]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 280 B/s wr, 0 op/s 2024-04-21T08:42:37.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:36 smithi016 ceph-mon[31259]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T08:42:37.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:36 smithi077 ceph-mon[31615]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T08:42:37.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:36 smithi007 ceph-mon[28477]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T08:42:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:38 smithi016 ceph-mon[31259]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:42:39.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:38 smithi077 ceph-mon[31615]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:42:39.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:38 smithi007 ceph-mon[28477]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:42:39.358 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi007.front.sepia.ceph.com 2024-04-21T08:42:39.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:42:39.358 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-21T08:42:39.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:42:39.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.544+0000 7f0be3a26640 1 -- 172.21.15.7:0/3887388164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bdc1007f0 msgr2=0x7f0bdc100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.544+0000 7f0be3a26640 1 --2- 172.21.15.7:0/3887388164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bdc1007f0 0x7f0bdc100c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009900 tx=0x7f0bd802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:39.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.544+0000 7f0be3a26640 1 -- 172.21.15.7:0/3887388164 shutdown_connections 2024-04-21T08:42:39.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.544+0000 7f0be3a26640 1 --2- 172.21.15.7:0/3887388164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bdc101af0 0x7f0bdc101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.544+0000 7f0be3a26640 1 --2- 172.21.15.7:0/3887388164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bdc101150 0x7f0bdc1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.544+0000 7f0be3a26640 1 --2- 172.21.15.7:0/3887388164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bdc1007f0 0x7f0bdc100c10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.544+0000 7f0be3a26640 1 -- 172.21.15.7:0/3887388164 >> 172.21.15.7:0/3887388164 conn(0x7f0bdc0fc3f0 msgr2=0x7f0bdc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:39.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.544+0000 7f0be3a26640 1 -- 172.21.15.7:0/3887388164 shutdown_connections 2024-04-21T08:42:39.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.544+0000 7f0be3a26640 1 -- 172.21.15.7:0/3887388164 wait complete. 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.544+0000 7f0be3a26640 1 Processor -- start 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.544+0000 7f0be3a26640 1 -- start start 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be3a26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bdc1007f0 0x7f0bdc19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be3a26640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bdc101150 0x7f0bdc19a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be3a26640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bdc101af0 0x7f0bdc1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be3a26640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc10fd20 con 0x7f0bdc101af0 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be3a26640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc10fba0 con 0x7f0bdc1007f0 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be3a26640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc10fea0 con 0x7f0bdc101150 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be1fc3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bdc101af0 0x7f0bdc1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be1fc3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bdc101af0 0x7f0bdc1a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35960/0 (socket says 172.21.15.7:35960) 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be1fc3640 1 -- 172.21.15.7:0/239521938 learned_addr learned my addr 172.21.15.7:0/239521938 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be0fc1640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bdc101150 0x7f0bdc19a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:39.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be17c2640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bdc1007f0 0x7f0bdc19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:39.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be1fc3640 1 -- 172.21.15.7:0/239521938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bdc101150 msgr2=0x7f0bdc19a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be1fc3640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bdc101150 0x7f0bdc19a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be1fc3640 1 -- 172.21.15.7:0/239521938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bdc1007f0 msgr2=0x7f0bdc19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be1fc3640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bdc1007f0 0x7f0bdc19a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be1fc3640 1 -- 172.21.15.7:0/239521938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc1a1f90 con 0x7f0bdc101af0 2024-04-21T08:42:39.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be1fc3640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bdc101af0 0x7f0bdc1a1920 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400eb10 tx=0x7f0bd400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:39.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0bd27fc640 1 -- 172.21.15.7:0/239521938 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4004b20 con 0x7f0bdc101af0 2024-04-21T08:42:39.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0bd27fc640 1 -- 172.21.15.7:0/239521938 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bd4004cc0 con 0x7f0bdc101af0 2024-04-21T08:42:39.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0be3a26640 1 -- 172.21.15.7:0/239521938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bdc1a2280 con 0x7f0bdc101af0 2024-04-21T08:42:39.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.545+0000 7f0bd27fc640 1 -- 172.21.15.7:0/239521938 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4010670 con 0x7f0bdc101af0 2024-04-21T08:42:39.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.546+0000 7f0be3a26640 1 -- 172.21.15.7:0/239521938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bdc1a27d0 con 0x7f0bdc101af0 2024-04-21T08:42:39.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.546+0000 7f0be3a26640 1 -- 172.21.15.7:0/239521938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bdc10b540 con 0x7f0bdc101af0 2024-04-21T08:42:39.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.548+0000 7f0bd27fc640 1 -- 172.21.15.7:0/239521938 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f0bd4005000 con 0x7f0bdc101af0 2024-04-21T08:42:39.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.548+0000 7f0bd27fc640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0bc80a3bb0 0x7f0bc80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.548+0000 7f0bd27fc640 1 -- 172.21.15.7:0/239521938 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f0bd40d0da0 con 0x7f0bdc101af0 2024-04-21T08:42:39.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.548+0000 7f0be17c2640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0bc80a3bb0 0x7f0bc80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:39.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.549+0000 7f0be17c2640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0bc80a3bb0 0x7f0bc80a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8006010 tx=0x7f0bd8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:39.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.552+0000 7f0bd27fc640 1 -- 172.21.15.7:0/239521938 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0bd4053ff0 con 0x7f0bdc101af0 2024-04-21T08:42:39.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.741+0000 7f0be3a26640 1 -- 172.21.15.7:0/239521938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0bdc062b50 con 0x7f0bdc101af0 2024-04-21T08:42:39.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.743+0000 7f0bd27fc640 1 -- 172.21.15.7:0/239521938 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23961 (secure 0 0 0) 0x7f0bd409d540 con 0x7f0bdc101af0 2024-04-21T08:42:39.743 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:39.743 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":90,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:42:22.023111+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:39.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 -- 172.21.15.7:0/239521938 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0bc80a3bb0 msgr2=0x7f0bc80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0bc80a3bb0 0x7f0bc80a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8006010 tx=0x7f0bd8000bc0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 -- 172.21.15.7:0/239521938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bdc101af0 msgr2=0x7f0bdc1a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bdc101af0 0x7f0bdc1a1920 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400eb10 tx=0x7f0bd400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 -- 172.21.15.7:0/239521938 shutdown_connections 2024-04-21T08:42:39.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f0bc80a3bb0 0x7f0bc80a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bdc101af0 0x7f0bdc1a1920 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bdc101150 0x7f0bdc19a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 --2- 172.21.15.7:0/239521938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0bdc1007f0 0x7f0bdc19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 -- 172.21.15.7:0/239521938 >> 172.21.15.7:0/239521938 conn(0x7f0bdc0fc3f0 msgr2=0x7f0bdc0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:39.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 -- 172.21.15.7:0/239521938 shutdown_connections 2024-04-21T08:42:39.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.745+0000 7f0be3a26640 1 -- 172.21.15.7:0/239521938 wait complete. 2024-04-21T08:42:39.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:42:39.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 -- 172.21.15.7:0/339424887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1878101fa0 msgr2=0x7f18781023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 --2- 172.21.15.7:0/339424887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1878101fa0 0x7f18781023c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1870009ca0 tx=0x7f187002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 -- 172.21.15.7:0/339424887 shutdown_connections 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 --2- 172.21.15.7:0/339424887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1878103b50 0x7f187810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 --2- 172.21.15.7:0/339424887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1878103190 0x7f1878103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 --2- 172.21.15.7:0/339424887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1878101fa0 0x7f18781023c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 -- 172.21.15.7:0/339424887 >> 172.21.15.7:0/339424887 conn(0x7f18780fd510 msgr2=0x7f18780ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 -- 172.21.15.7:0/339424887 shutdown_connections 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 -- 172.21.15.7:0/339424887 wait complete. 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 Processor -- start 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 -- start start 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1878101fa0 0x7f18781981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1878103190 0x7f18781986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1878103b50 0x7f187819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.859+0000 7f187fa5c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f187810dae0 con 0x7f1878101fa0 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187fa5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f187810d960 con 0x7f1878103190 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187fa5c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f187810dc60 con 0x7f1878103b50 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187d7f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1878101fa0 0x7f18781981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187d7f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1878101fa0 0x7f18781981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35974/0 (socket says 172.21.15.7:35974) 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187d7f8640 1 -- 172.21.15.7:0/697118579 learned_addr learned my addr 172.21.15.7:0/697118579 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187cff7640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1878103190 0x7f18781986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187dff9640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1878103b50 0x7f187819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187d7f8640 1 -- 172.21.15.7:0/697118579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1878103b50 msgr2=0x7f187819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187d7f8640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1878103b50 0x7f187819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187d7f8640 1 -- 172.21.15.7:0/697118579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1878103190 msgr2=0x7f18781986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187d7f8640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1878103190 0x7f18781986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187d7f8640 1 -- 172.21.15.7:0/697118579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187819fdd0 con 0x7f1878101fa0 2024-04-21T08:42:39.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187d7f8640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1878101fa0 0x7f18781981b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1870033000 tx=0x7f1870034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:39.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f186e7fc640 1 -- 172.21.15.7:0/697118579 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1870039040 con 0x7f1878101fa0 2024-04-21T08:42:39.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187fa5c640 1 -- 172.21.15.7:0/697118579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18781a0060 con 0x7f1878101fa0 2024-04-21T08:42:39.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f186e7fc640 1 -- 172.21.15.7:0/697118579 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1870044070 con 0x7f1878101fa0 2024-04-21T08:42:39.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f186e7fc640 1 -- 172.21.15.7:0/697118579 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18700042a0 con 0x7f1878101fa0 2024-04-21T08:42:39.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.860+0000 7f187fa5c640 1 -- 172.21.15.7:0/697118579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18781a0550 con 0x7f1878101fa0 2024-04-21T08:42:39.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.861+0000 7f187fa5c640 1 -- 172.21.15.7:0/697118579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1878109390 con 0x7f1878101fa0 2024-04-21T08:42:39.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.862+0000 7f186e7fc640 1 -- 172.21.15.7:0/697118579 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f1870003970 con 0x7f1878101fa0 2024-04-21T08:42:39.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.863+0000 7f186e7fc640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f18640a3bb0 0x7f18640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.863+0000 7f187cff7640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f18640a3bb0 0x7f18640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:39.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.863+0000 7f186e7fc640 1 -- 172.21.15.7:0/697118579 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f18700f2b10 con 0x7f1878101fa0 2024-04-21T08:42:39.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.863+0000 7f187cff7640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f18640a3bb0 0x7f18640a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f18781996c0 tx=0x7f186800a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:39.868+0000 7f186e7fc640 1 -- 172.21.15.7:0/697118579 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18700bf250 con 0x7f1878101fa0 2024-04-21T08:42:40.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.055+0000 7f187fa5c640 1 -- 172.21.15.7:0/697118579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1878062b50 con 0x7f1878101fa0 2024-04-21T08:42:40.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.056+0000 7f186e7fc640 1 -- 172.21.15.7:0/697118579 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23961 (secure 0 0 0) 0x7f18700beb50 con 0x7f1878101fa0 2024-04-21T08:42:40.056 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:40.056 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":90,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:42:22.023111+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:40.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.058+0000 7f187fa5c640 1 -- 172.21.15.7:0/697118579 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f18640a3bb0 msgr2=0x7f18640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.058+0000 7f187fa5c640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f18640a3bb0 0x7f18640a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f18781996c0 tx=0x7f186800a4c0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.058+0000 7f187fa5c640 1 -- 172.21.15.7:0/697118579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1878101fa0 msgr2=0x7f18781981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.058+0000 7f187fa5c640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1878101fa0 0x7f18781981b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1870033000 tx=0x7f1870034760 comp rx=0 tx=0).stop 2024-04-21T08:42:40.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.058+0000 7f187fa5c640 1 -- 172.21.15.7:0/697118579 shutdown_connections 2024-04-21T08:42:40.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.058+0000 7f187fa5c640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f18640a3bb0 0x7f18640a6070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.058+0000 7f187fa5c640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1878103b50 0x7f187819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.058+0000 7f187fa5c640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1878103190 0x7f18781986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.058+0000 7f187fa5c640 1 --2- 172.21.15.7:0/697118579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1878101fa0 0x7f18781981b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.058+0000 7f187fa5c640 1 -- 172.21.15.7:0/697118579 >> 172.21.15.7:0/697118579 conn(0x7f18780fd510 msgr2=0x7f1878106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.059+0000 7f187fa5c640 1 -- 172.21.15.7:0/697118579 shutdown_connections 2024-04-21T08:42:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.059+0000 7f187fa5c640 1 -- 172.21.15.7:0/697118579 wait complete. 2024-04-21T08:42:40.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-21T08:42:40.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.175+0000 7f6925ba4640 1 -- 172.21.15.7:0/3428391029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69201029b0 msgr2=0x7f69201092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.175+0000 7f6925ba4640 1 --2- 172.21.15.7:0/3428391029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69201029b0 0x7f69201092d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6918009ca0 tx=0x7f691802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.175+0000 7f6925ba4640 1 -- 172.21.15.7:0/3428391029 shutdown_connections 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.175+0000 7f6925ba4640 1 --2- 172.21.15.7:0/3428391029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69201029b0 0x7f69201092d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.175+0000 7f6925ba4640 1 --2- 172.21.15.7:0/3428391029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6920101ff0 0x7f6920102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.175+0000 7f6925ba4640 1 --2- 172.21.15.7:0/3428391029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6920100ea0 0x7f69201012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.175+0000 7f6925ba4640 1 -- 172.21.15.7:0/3428391029 >> 172.21.15.7:0/3428391029 conn(0x7f69200fc430 msgr2=0x7f69200fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.175+0000 7f6925ba4640 1 -- 172.21.15.7:0/3428391029 shutdown_connections 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.175+0000 7f6925ba4640 1 -- 172.21.15.7:0/3428391029 wait complete. 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f6925ba4640 1 Processor -- start 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f6925ba4640 1 -- start start 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f6925ba4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6920100ea0 0x7f692006b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f6925ba4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6920101ff0 0x7f692019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f6925ba4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69201029b0 0x7f692019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f6925ba4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f692010c9e0 con 0x7f6920100ea0 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f6925ba4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f692010c860 con 0x7f69201029b0 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f6925ba4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f692010cb60 con 0x7f6920101ff0 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f691f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6920100ea0 0x7f692006b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f691f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6920100ea0 0x7f692006b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35980/0 (socket says 172.21.15.7:35980) 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f691f7fe640 1 -- 172.21.15.7:0/2267066747 learned_addr learned my addr 172.21.15.7:0/2267066747 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f691f7fe640 1 -- 172.21.15.7:0/2267066747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6920101ff0 msgr2=0x7f692019ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f691f7fe640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6920101ff0 0x7f692019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f691f7fe640 1 -- 172.21.15.7:0/2267066747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69201029b0 msgr2=0x7f692019f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f691f7fe640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69201029b0 0x7f692019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.176+0000 7f691f7fe640 1 -- 172.21.15.7:0/2267066747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692019fdc0 con 0x7f6920100ea0 2024-04-21T08:42:40.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.177+0000 7f691f7fe640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6920100ea0 0x7f692006b190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f691000c920 tx=0x7f691000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:40.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.177+0000 7f691d7fa640 1 -- 172.21.15.7:0/2267066747 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910007c70 con 0x7f6920100ea0 2024-04-21T08:42:40.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.177+0000 7f691d7fa640 1 -- 172.21.15.7:0/2267066747 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6910007e10 con 0x7f6920100ea0 2024-04-21T08:42:40.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.177+0000 7f6925ba4640 1 -- 172.21.15.7:0/2267066747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692019fff0 con 0x7f6920100ea0 2024-04-21T08:42:40.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.177+0000 7f691d7fa640 1 -- 172.21.15.7:0/2267066747 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910018820 con 0x7f6920100ea0 2024-04-21T08:42:40.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.177+0000 7f6925ba4640 1 -- 172.21.15.7:0/2267066747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69201a0540 con 0x7f6920100ea0 2024-04-21T08:42:40.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.178+0000 7f6925ba4640 1 -- 172.21.15.7:0/2267066747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6920108150 con 0x7f6920100ea0 2024-04-21T08:42:40.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.179+0000 7f691d7fa640 1 -- 172.21.15.7:0/2267066747 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f691000f450 con 0x7f6920100ea0 2024-04-21T08:42:40.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.179+0000 7f691d7fa640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f690c0a3bb0 0x7f690c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.180+0000 7f691d7fa640 1 -- 172.21.15.7:0/2267066747 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f69100d0800 con 0x7f6920100ea0 2024-04-21T08:42:40.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.180+0000 7f6917fff640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f690c0a3bb0 0x7f690c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:40.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.180+0000 7f6917fff640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f690c0a3bb0 0x7f690c0a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f692006af00 tx=0x7f6908009630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:40.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.184+0000 7f691d7fa640 1 -- 172.21.15.7:0/2267066747 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6910014030 con 0x7f6920100ea0 2024-04-21T08:42:40.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.411+0000 7f6925ba4640 1 -- 172.21.15.7:0/2267066747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6920062c50 con 0x7f6920100ea0 2024-04-21T08:42:40.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.709+0000 7f691d7fa640 1 -- 172.21.15.7:0/2267066747 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f691009cfa0 con 0x7f6920100ea0 2024-04-21T08:42:40.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 -- 172.21.15.7:0/2267066747 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f690c0a3bb0 msgr2=0x7f690c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f690c0a3bb0 0x7f690c0a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f692006af00 tx=0x7f6908009630 comp rx=0 tx=0).stop 2024-04-21T08:42:40.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 -- 172.21.15.7:0/2267066747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6920100ea0 msgr2=0x7f692006b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6920100ea0 0x7f692006b190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f691000c920 tx=0x7f691000cde0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 -- 172.21.15.7:0/2267066747 shutdown_connections 2024-04-21T08:42:40.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f690c0a3bb0 0x7f690c0a6070 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69201029b0 0x7f692019f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6920101ff0 0x7f692019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 --2- 172.21.15.7:0/2267066747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6920100ea0 0x7f692006b190 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 -- 172.21.15.7:0/2267066747 >> 172.21.15.7:0/2267066747 conn(0x7f69200fc430 msgr2=0x7f6920105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:40.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 -- 172.21.15.7:0/2267066747 shutdown_connections 2024-04-21T08:42:40.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.711+0000 7f6925ba4640 1 -- 172.21.15.7:0/2267066747 wait complete. 2024-04-21T08:42:40.723 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-21T08:42:40.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-21T08:42:40.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.830+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/2641211469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a74100e90 msgr2=0x7f6a741012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.830+0000 7f6a7bc6a640 1 --2- 172.21.15.7:0/2641211469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a74100e90 0x7f6a741012b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009900 tx=0x7f6a6c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.831+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/2641211469 shutdown_connections 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.831+0000 7f6a7bc6a640 1 --2- 172.21.15.7:0/2641211469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a741029a0 0x7f6a741092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.831+0000 7f6a7bc6a640 1 --2- 172.21.15.7:0/2641211469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a74101fe0 0x7f6a74102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.831+0000 7f6a7bc6a640 1 --2- 172.21.15.7:0/2641211469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a74100e90 0x7f6a741012b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.831+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/2641211469 >> 172.21.15.7:0/2641211469 conn(0x7f6a740fc420 msgr2=0x7f6a740fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.831+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/2641211469 shutdown_connections 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.831+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/2641211469 wait complete. 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.831+0000 7f6a7bc6a640 1 Processor -- start 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.831+0000 7f6a7bc6a640 1 -- start start 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.831+0000 7f6a7bc6a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a74100e90 0x7f6a74104e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.831+0000 7f6a7bc6a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a74101fe0 0x7f6a74103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a7bc6a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a741029a0 0x7f6a74103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a7bc6a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7410c9e0 con 0x7f6a74101fe0 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a7bc6a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7410c860 con 0x7f6a74100e90 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a7bc6a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7410cb60 con 0x7f6a741029a0 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a79205640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a74101fe0 0x7f6a74103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a79205640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a74101fe0 0x7f6a74103450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36006/0 (socket says 172.21.15.7:36006) 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a79205640 1 -- 172.21.15.7:0/1090387774 learned_addr learned my addr 172.21.15.7:0/1090387774 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a79a06640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a74100e90 0x7f6a74104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a7a207640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a741029a0 0x7f6a74103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a79205640 1 -- 172.21.15.7:0/1090387774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a741029a0 msgr2=0x7f6a74103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a79205640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a741029a0 0x7f6a74103990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a79205640 1 -- 172.21.15.7:0/1090387774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a74100e90 msgr2=0x7f6a74104e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a79205640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a74100e90 0x7f6a74104e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a79205640 1 -- 172.21.15.7:0/1090387774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a74104220 con 0x7f6a74101fe0 2024-04-21T08:42:40.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a7a207640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a741029a0 0x7f6a74103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:40.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a79205640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a74101fe0 0x7f6a74103450 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400d950 tx=0x7f6a6400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:40.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a6affd640 1 -- 172.21.15.7:0/1090387774 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64017070 con 0x7f6a74101fe0 2024-04-21T08:42:40.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a6affd640 1 -- 172.21.15.7:0/1090387774 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a64009c30 con 0x7f6a74101fe0 2024-04-21T08:42:40.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/1090387774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a741a2360 con 0x7f6a74101fe0 2024-04-21T08:42:40.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a6affd640 1 -- 172.21.15.7:0/1090387774 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400f620 con 0x7f6a74101fe0 2024-04-21T08:42:40.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.832+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/1090387774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a741a2860 con 0x7f6a74101fe0 2024-04-21T08:42:40.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.833+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/1090387774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a741a2b20 con 0x7f6a74101fe0 2024-04-21T08:42:40.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.834+0000 7f6a6affd640 1 -- 172.21.15.7:0/1090387774 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6a6401a070 con 0x7f6a74101fe0 2024-04-21T08:42:40.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.835+0000 7f6a6affd640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6a600a3bb0 0x7f6a600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.835+0000 7f6a79a06640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6a600a3bb0 0x7f6a600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:40.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.835+0000 7f6a6affd640 1 -- 172.21.15.7:0/1090387774 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6a640d00e0 con 0x7f6a74101fe0 2024-04-21T08:42:40.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.835+0000 7f6a79a06640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6a600a3bb0 0x7f6a600a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009230 tx=0x7f6a6c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:40.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:40.839+0000 7f6a6affd640 1 -- 172.21.15.7:0/1090387774 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a64009de0 con 0x7f6a74101fe0 2024-04-21T08:42:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:40 smithi016 ceph-mon[31259]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:42:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:40 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/239521938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:40 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/697118579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:40 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2267066747' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:42:41.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.066+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/1090387774 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f6a741a2f10 con 0x7f6a74101fe0 2024-04-21T08:42:41.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:40 smithi077 ceph-mon[31615]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:42:41.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:40 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/239521938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:40 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/697118579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:40 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2267066747' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:42:41.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:40 smithi007 ceph-mon[28477]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:42:41.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:40 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/239521938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:41.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:40 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/697118579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:40 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2267066747' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:42:41.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.711+0000 7f6a6affd640 1 -- 172.21.15.7:0/1090387774 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f6a741a2f10 con 0x7f6a74101fe0 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/1090387774 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6a600a3bb0 msgr2=0x7f6a600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6a600a3bb0 0x7f6a600a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009230 tx=0x7f6a6c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/1090387774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a74101fe0 msgr2=0x7f6a74103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a74101fe0 0x7f6a74103450 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400d950 tx=0x7f6a6400de10 comp rx=0 tx=0).stop 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/1090387774 shutdown_connections 2024-04-21T08:42:41.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6a600a3bb0 0x7f6a600a6070 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a741029a0 0x7f6a74103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a74101fe0 0x7f6a74103450 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 --2- 172.21.15.7:0/1090387774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a74100e90 0x7f6a74104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/1090387774 >> 172.21.15.7:0/1090387774 conn(0x7f6a740fc420 msgr2=0x7f6a74069250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:41.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/1090387774 shutdown_connections 2024-04-21T08:42:41.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.714+0000 7f6a7bc6a640 1 -- 172.21.15.7:0/1090387774 wait complete. 2024-04-21T08:42:41.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-21T08:42:41.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.840+0000 7fcb6592b640 1 -- 172.21.15.7:0/3024469270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb601029b0 msgr2=0x7fcb601092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:41.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.840+0000 7fcb6592b640 1 --2- 172.21.15.7:0/3024469270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb601029b0 0x7fcb601092d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009ca0 tx=0x7fcb5802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.840+0000 7fcb6592b640 1 -- 172.21.15.7:0/3024469270 shutdown_connections 2024-04-21T08:42:41.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.840+0000 7fcb6592b640 1 --2- 172.21.15.7:0/3024469270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb601029b0 0x7fcb601092d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.840+0000 7fcb6592b640 1 --2- 172.21.15.7:0/3024469270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb60101ff0 0x7fcb60102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.840+0000 7fcb6592b640 1 --2- 172.21.15.7:0/3024469270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb60100ea0 0x7fcb601012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.840+0000 7fcb6592b640 1 -- 172.21.15.7:0/3024469270 >> 172.21.15.7:0/3024469270 conn(0x7fcb600fc430 msgr2=0x7fcb600fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.840+0000 7fcb6592b640 1 -- 172.21.15.7:0/3024469270 shutdown_connections 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.840+0000 7fcb6592b640 1 -- 172.21.15.7:0/3024469270 wait complete. 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb6592b640 1 Processor -- start 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb6592b640 1 -- start start 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb6592b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb60100ea0 0x7fcb6006b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb6592b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb60101ff0 0x7fcb6019f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb6592b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb601029b0 0x7fcb601a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb6592b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6010ca00 con 0x7fcb60100ea0 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb6592b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6010c880 con 0x7fcb601029b0 2024-04-21T08:42:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb6592b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6010cb80 con 0x7fcb60101ff0 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb5effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb60100ea0 0x7fcb6006b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb5effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb60100ea0 0x7fcb6006b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36028/0 (socket says 172.21.15.7:36028) 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb5effd640 1 -- 172.21.15.7:0/1918233574 learned_addr learned my addr 172.21.15.7:0/1918233574 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.841+0000 7fcb5e7fc640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb60101ff0 0x7fcb6019f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb5f7fe640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb601029b0 0x7fcb601a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb5effd640 1 -- 172.21.15.7:0/1918233574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb60101ff0 msgr2=0x7fcb6019f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb5effd640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb60101ff0 0x7fcb6019f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb5effd640 1 -- 172.21.15.7:0/1918233574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb601029b0 msgr2=0x7fcb601a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb5effd640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb601029b0 0x7fcb601a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb5effd640 1 -- 172.21.15.7:0/1918233574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb601a1f90 con 0x7fcb60100ea0 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb5f7fe640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb601029b0 0x7fcb601a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb5effd640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb60100ea0 0x7fcb6006b140 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000d770 tx=0x7fcb5000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:41.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb64929640 1 -- 172.21.15.7:0/1918233574 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb50017070 con 0x7fcb60100ea0 2024-04-21T08:42:41.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb64929640 1 -- 172.21.15.7:0/1918233574 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb500054d0 con 0x7fcb60100ea0 2024-04-21T08:42:41.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb6592b640 1 -- 172.21.15.7:0/1918233574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb601a2280 con 0x7fcb60100ea0 2024-04-21T08:42:41.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb64929640 1 -- 172.21.15.7:0/1918233574 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb50004c30 con 0x7fcb60100ea0 2024-04-21T08:42:41.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.842+0000 7fcb6592b640 1 -- 172.21.15.7:0/1918233574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb601a27d0 con 0x7fcb60100ea0 2024-04-21T08:42:41.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.843+0000 7fcb6592b640 1 -- 172.21.15.7:0/1918233574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb601081b0 con 0x7fcb60100ea0 2024-04-21T08:42:41.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.844+0000 7fcb64929640 1 -- 172.21.15.7:0/1918233574 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fcb50004dd0 con 0x7fcb60100ea0 2024-04-21T08:42:41.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.845+0000 7fcb64929640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb480a3c60 0x7fcb480a6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:41.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.845+0000 7fcb5e7fc640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb480a3c60 0x7fcb480a6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:41.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.845+0000 7fcb64929640 1 -- 172.21.15.7:0/1918233574 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcb500d0180 con 0x7fcb60100ea0 2024-04-21T08:42:41.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.845+0000 7fcb5e7fc640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb480a3c60 0x7fcb480a6120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcb54005fd0 tx=0x7fcb54005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:41.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:41.850+0000 7fcb64929640 1 -- 172.21.15.7:0/1918233574 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb5009c8c0 con 0x7fcb60100ea0 2024-04-21T08:42:42.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:41 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2267066747' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:42:42.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:41 smithi016 ceph-mon[31259]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:42.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:41 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1090387774' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:42:42.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.079+0000 7fcb6592b640 1 -- 172.21.15.7:0/1918233574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fcb6002d5c0 con 0x7fcb60100ea0 2024-04-21T08:42:42.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:41 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2267066747' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:42:42.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:41 smithi077 ceph-mon[31615]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:41 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1090387774' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:42:42.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:41 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2267066747' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:42:42.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:41 smithi007 ceph-mon[28477]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:42.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:41 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1090387774' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:42:42.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.717+0000 7fcb64929640 1 -- 172.21.15.7:0/1918233574 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7fcb5009c1c0 con 0x7fcb60100ea0 2024-04-21T08:42:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.720+0000 7fcb6592b640 1 -- 172.21.15.7:0/1918233574 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb480a3c60 msgr2=0x7fcb480a6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.720+0000 7fcb6592b640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb480a3c60 0x7fcb480a6120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcb54005fd0 tx=0x7fcb54005850 comp rx=0 tx=0).stop 2024-04-21T08:42:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.720+0000 7fcb6592b640 1 -- 172.21.15.7:0/1918233574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb60100ea0 msgr2=0x7fcb6006b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.720+0000 7fcb6592b640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb60100ea0 0x7fcb6006b140 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000d770 tx=0x7fcb5000dc30 comp rx=0 tx=0).stop 2024-04-21T08:42:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.720+0000 7fcb6592b640 1 -- 172.21.15.7:0/1918233574 shutdown_connections 2024-04-21T08:42:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.720+0000 7fcb6592b640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb480a3c60 0x7fcb480a6120 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.720+0000 7fcb6592b640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb601029b0 0x7fcb601a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.720+0000 7fcb6592b640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb60101ff0 0x7fcb6019f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.720+0000 7fcb6592b640 1 --2- 172.21.15.7:0/1918233574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb60100ea0 0x7fcb6006b140 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.720+0000 7fcb6592b640 1 -- 172.21.15.7:0/1918233574 >> 172.21.15.7:0/1918233574 conn(0x7fcb600fc430 msgr2=0x7fcb60105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:42.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.721+0000 7fcb6592b640 1 -- 172.21.15.7:0/1918233574 shutdown_connections 2024-04-21T08:42:42.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.721+0000 7fcb6592b640 1 -- 172.21.15.7:0/1918233574 wait complete. 2024-04-21T08:42:42.734 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-04-21T08:42:42.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-21T08:42:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.847+0000 7f7826650640 1 -- 172.21.15.7:0/1366776709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7820101ab0 msgr2=0x7f7820109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.847+0000 7f7826650640 1 --2- 172.21.15.7:0/1366776709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7820101ab0 0x7f7820109350 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7818009ca0 tx=0x7f781802c140 comp rx=0 tx=0).stop 2024-04-21T08:42:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.847+0000 7f7826650640 1 -- 172.21.15.7:0/1366776709 shutdown_connections 2024-04-21T08:42:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.847+0000 7f7826650640 1 --2- 172.21.15.7:0/1366776709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7820101ab0 0x7f7820109350 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.847+0000 7f7826650640 1 --2- 172.21.15.7:0/1366776709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78201010f0 0x7f7820101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.847+0000 7f7826650640 1 --2- 172.21.15.7:0/1366776709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7820102530 0x7f7820102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.847+0000 7f7826650640 1 -- 172.21.15.7:0/1366776709 >> 172.21.15.7:0/1366776709 conn(0x7f78200fbb80 msgr2=0x7f78200fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.847+0000 7f7826650640 1 -- 172.21.15.7:0/1366776709 shutdown_connections 2024-04-21T08:42:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.847+0000 7f7826650640 1 -- 172.21.15.7:0/1366776709 wait complete. 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.848+0000 7f7826650640 1 Processor -- start 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.848+0000 7f7826650640 1 -- start start 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.848+0000 7f7826650640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78201010f0 0x7f78201981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.848+0000 7f7826650640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7820101ab0 0x7f7820198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.848+0000 7f7826650640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7820102530 0x7f782019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.848+0000 7f7826650640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f782010ca00 con 0x7f7820102530 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.848+0000 7f7826650640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f782010c880 con 0x7f78201010f0 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.848+0000 7f7826650640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f782010cb80 con 0x7f7820101ab0 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.848+0000 7f7824bed640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7820102530 0x7f782019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.848+0000 7f7824bed640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7820102530 0x7f782019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36042/0 (socket says 172.21.15.7:36042) 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.848+0000 7f7824bed640 1 -- 172.21.15.7:0/2183043725 learned_addr learned my addr 172.21.15.7:0/2183043725 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f781ffff640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78201010f0 0x7f78201981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f7824bed640 1 -- 172.21.15.7:0/2183043725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7820101ab0 msgr2=0x7f7820198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f7824bed640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7820101ab0 0x7f7820198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f7824bed640 1 -- 172.21.15.7:0/2183043725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78201010f0 msgr2=0x7f78201981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f7824bed640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78201010f0 0x7f78201981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f7824bed640 1 -- 172.21.15.7:0/2183043725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782019fe00 con 0x7f7820102530 2024-04-21T08:42:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f781ffff640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78201010f0 0x7f78201981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f7824bed640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7820102530 0x7f782019f790 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f781802c750 tx=0x7f7818035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f781d7fa640 1 -- 172.21.15.7:0/2183043725 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818039040 con 0x7f7820102530 2024-04-21T08:42:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f781d7fa640 1 -- 172.21.15.7:0/2183043725 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7818044070 con 0x7f7820102530 2024-04-21T08:42:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f7826650640 1 -- 172.21.15.7:0/2183043725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78201a0090 con 0x7f7820102530 2024-04-21T08:42:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f781d7fa640 1 -- 172.21.15.7:0/2183043725 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78180044d0 con 0x7f7820102530 2024-04-21T08:42:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.849+0000 7f7826650640 1 -- 172.21.15.7:0/2183043725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78201a05d0 con 0x7f7820102530 2024-04-21T08:42:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.850+0000 7f7826650640 1 -- 172.21.15.7:0/2183043725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78201a08f0 con 0x7f7820102530 2024-04-21T08:42:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.851+0000 7f781d7fa640 1 -- 172.21.15.7:0/2183043725 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f78180327f0 con 0x7f7820102530 2024-04-21T08:42:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.852+0000 7f781d7fa640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f78080a3c30 0x7f78080a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.852+0000 7f781ffff640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f78080a3c30 0x7f78080a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.852+0000 7f781d7fa640 1 -- 172.21.15.7:0/2183043725 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f78180f2f50 con 0x7f7820102530 2024-04-21T08:42:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.853+0000 7f781ffff640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f78080a3c30 0x7f78080a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f78100058c0 tx=0x7f7810005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:42.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:42.857+0000 7f781d7fa640 1 -- 172.21.15.7:0/2183043725 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78180bf690 con 0x7f7820102530 2024-04-21T08:42:43.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:42 smithi007 ceph-mon[28477]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:42:43.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:42 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1090387774' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:42:43.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:42 smithi007 ceph-mon[28477]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:43.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:42 smithi007 ceph-mon[28477]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:42:43.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:42 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1918233574' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:42:43.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:42 smithi016 ceph-mon[31259]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:42:43.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:42 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1090387774' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:42:43.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:42 smithi016 ceph-mon[31259]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:43.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:42 smithi016 ceph-mon[31259]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:42:43.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:42 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1918233574' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:42:43.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.071+0000 7f7826650640 1 -- 172.21.15.7:0/2183043725 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f78200fddb0 con 0x7f78080a3c30 2024-04-21T08:42:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.093+0000 7f781d7fa640 1 -- 172.21.15.7:0/2183043725 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f78200fddb0 con 0x7f78080a3c30 2024-04-21T08:42:43.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 -- 172.21.15.7:0/2183043725 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f78080a3c30 msgr2=0x7f78080a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f78080a3c30 0x7f78080a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f78100058c0 tx=0x7f7810005850 comp rx=0 tx=0).stop 2024-04-21T08:42:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 -- 172.21.15.7:0/2183043725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7820102530 msgr2=0x7f782019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7820102530 0x7f782019f790 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f781802c750 tx=0x7f7818035d60 comp rx=0 tx=0).stop 2024-04-21T08:42:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 -- 172.21.15.7:0/2183043725 shutdown_connections 2024-04-21T08:42:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f78080a3c30 0x7f78080a60f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7820102530 0x7f782019f790 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7820101ab0 0x7f7820198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 --2- 172.21.15.7:0/2183043725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78201010f0 0x7f78201981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 -- 172.21.15.7:0/2183043725 >> 172.21.15.7:0/2183043725 conn(0x7f78200fbb80 msgr2=0x7f78200fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 -- 172.21.15.7:0/2183043725 shutdown_connections 2024-04-21T08:42:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.095+0000 7f7826650640 1 -- 172.21.15.7:0/2183043725 wait complete. 2024-04-21T08:42:43.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:42 smithi077 ceph-mon[31615]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:42:43.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:42 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1090387774' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:42:43.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:42 smithi077 ceph-mon[31615]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:42:43.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:42 smithi077 ceph-mon[31615]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:42:43.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:42 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1918233574' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:42:43.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-21T08:42:43.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.216+0000 7f4c640d5640 1 -- 172.21.15.7:0/2429034709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c5c100e90 msgr2=0x7f4c5c1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.216+0000 7f4c640d5640 1 --2- 172.21.15.7:0/2429034709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c5c100e90 0x7f4c5c1012b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009900 tx=0x7f4c5802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:43.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.216+0000 7f4c640d5640 1 -- 172.21.15.7:0/2429034709 shutdown_connections 2024-04-21T08:42:43.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.216+0000 7f4c640d5640 1 --2- 172.21.15.7:0/2429034709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c1029a0 0x7f4c5c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.216+0000 7f4c640d5640 1 --2- 172.21.15.7:0/2429034709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5c101fe0 0x7f4c5c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.216+0000 7f4c640d5640 1 --2- 172.21.15.7:0/2429034709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c5c100e90 0x7f4c5c1012b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.216+0000 7f4c640d5640 1 -- 172.21.15.7:0/2429034709 >> 172.21.15.7:0/2429034709 conn(0x7f4c5c0fc420 msgr2=0x7f4c5c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:43.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.216+0000 7f4c640d5640 1 -- 172.21.15.7:0/2429034709 shutdown_connections 2024-04-21T08:42:43.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.216+0000 7f4c640d5640 1 -- 172.21.15.7:0/2429034709 wait complete. 2024-04-21T08:42:43.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c640d5640 1 Processor -- start 2024-04-21T08:42:43.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c640d5640 1 -- start start 2024-04-21T08:42:43.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c640d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c100e90 0x7f4c5c197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c640d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5c101fe0 0x7f4c5c1976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c640d5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c5c1029a0 0x7f4c5c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c640d5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c10c9e0 con 0x7f4c5c1029a0 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c640d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c10c860 con 0x7f4c5c100e90 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c640d5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c10cb60 con 0x7f4c5c101fe0 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c62672640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c5c1029a0 0x7f4c5c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c62672640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c5c1029a0 0x7f4c5c19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36072/0 (socket says 172.21.15.7:36072) 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c62672640 1 -- 172.21.15.7:0/1299668349 learned_addr learned my addr 172.21.15.7:0/1299668349 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c61e71640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c100e90 0x7f4c5c197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c61670640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5c101fe0 0x7f4c5c1976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c62672640 1 -- 172.21.15.7:0/1299668349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5c101fe0 msgr2=0x7f4c5c1976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c62672640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5c101fe0 0x7f4c5c1976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c62672640 1 -- 172.21.15.7:0/1299668349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c100e90 msgr2=0x7f4c5c197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c62672640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c100e90 0x7f4c5c197180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.217+0000 7f4c62672640 1 -- 172.21.15.7:0/1299668349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5c19ed50 con 0x7f4c5c1029a0 2024-04-21T08:42:43.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.218+0000 7f4c62672640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c5c1029a0 0x7f4c5c19e6e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400eaa0 tx=0x7f4c5400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:43.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.218+0000 7f4c52ffd640 1 -- 172.21.15.7:0/1299668349 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5400cdb0 con 0x7f4c5c1029a0 2024-04-21T08:42:43.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.218+0000 7f4c52ffd640 1 -- 172.21.15.7:0/1299668349 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c54005020 con 0x7f4c5c1029a0 2024-04-21T08:42:43.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.218+0000 7f4c640d5640 1 -- 172.21.15.7:0/1299668349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5c19efe0 con 0x7f4c5c1029a0 2024-04-21T08:42:43.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.218+0000 7f4c52ffd640 1 -- 172.21.15.7:0/1299668349 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c540116d0 con 0x7f4c5c1029a0 2024-04-21T08:42:43.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.218+0000 7f4c640d5640 1 -- 172.21.15.7:0/1299668349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c5c19f4b0 con 0x7f4c5c1029a0 2024-04-21T08:42:43.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.219+0000 7f4c640d5640 1 -- 172.21.15.7:0/1299668349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c5c108310 con 0x7f4c5c1029a0 2024-04-21T08:42:43.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.220+0000 7f4c52ffd640 1 -- 172.21.15.7:0/1299668349 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f4c540051d0 con 0x7f4c5c1029a0 2024-04-21T08:42:43.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.221+0000 7f4c52ffd640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4c480a3bb0 0x7f4c480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.221+0000 7f4c61e71640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4c480a3bb0 0x7f4c480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.221+0000 7f4c52ffd640 1 -- 172.21.15.7:0/1299668349 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4c540d0840 con 0x7f4c5c1029a0 2024-04-21T08:42:43.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.221+0000 7f4c61e71640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4c480a3bb0 0x7f4c480a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009230 tx=0x7f4c58005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:43.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.226+0000 7f4c52ffd640 1 -- 172.21.15.7:0/1299668349 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c54016030 con 0x7f4c5c1029a0 2024-04-21T08:42:43.391 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:42:43 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:42:43.076+0000 7f73f943b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:42:43.391 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:42:43 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:42:43.076+0000 7f73f943b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:42:43.391 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:42:43 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:42:43.076+0000 7f73f943b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:42:43.391 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:42:43 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:42:43.076+0000 7f73f943b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:42:43.391 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:42:43 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:42:43.076+0000 7f73f943b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:42:43.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.441+0000 7f4c640d5640 1 -- 172.21.15.7:0/1299668349 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4c5c062390 con 0x7f4c480a3bb0 2024-04-21T08:42:43.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.470+0000 7f4c52ffd640 1 -- 172.21.15.7:0/1299668349 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4c5c062390 con 0x7f4c480a3bb0 2024-04-21T08:42:43.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 -- 172.21.15.7:0/1299668349 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4c480a3bb0 msgr2=0x7f4c480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4c480a3bb0 0x7f4c480a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009230 tx=0x7f4c58005fd0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 -- 172.21.15.7:0/1299668349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c5c1029a0 msgr2=0x7f4c5c19e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c5c1029a0 0x7f4c5c19e6e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400eaa0 tx=0x7f4c5400ef60 comp rx=0 tx=0).stop 2024-04-21T08:42:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 -- 172.21.15.7:0/1299668349 shutdown_connections 2024-04-21T08:42:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4c480a3bb0 0x7f4c480a6070 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c5c1029a0 0x7f4c5c19e6e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5c101fe0 0x7f4c5c1976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 --2- 172.21.15.7:0/1299668349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c5c100e90 0x7f4c5c197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 -- 172.21.15.7:0/1299668349 >> 172.21.15.7:0/1299668349 conn(0x7f4c5c0fc420 msgr2=0x7f4c5c1054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 -- 172.21.15.7:0/1299668349 shutdown_connections 2024-04-21T08:42:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.472+0000 7f4c640d5640 1 -- 172.21.15.7:0/1299668349 wait complete. 2024-04-21T08:42:43.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-21T08:42:43.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.594+0000 7fb309e48640 1 -- 172.21.15.7:0/2129631721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb304102a20 msgr2=0x7fb304109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.594+0000 7fb309e48640 1 --2- 172.21.15.7:0/2129631721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb304102a20 0x7fb304109340 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc009c90 tx=0x7fb2fc02e2b0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.594+0000 7fb309e48640 1 -- 172.21.15.7:0/2129631721 shutdown_connections 2024-04-21T08:42:43.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.594+0000 7fb309e48640 1 --2- 172.21.15.7:0/2129631721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb304102a20 0x7fb304109340 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.594+0000 7fb309e48640 1 --2- 172.21.15.7:0/2129631721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304102060 0x7fb3041024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.594+0000 7fb309e48640 1 --2- 172.21.15.7:0/2129631721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb304100e80 0x7fb3041012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.594+0000 7fb309e48640 1 -- 172.21.15.7:0/2129631721 >> 172.21.15.7:0/2129631721 conn(0x7fb3040fc420 msgr2=0x7fb3040fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:43.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.594+0000 7fb309e48640 1 -- 172.21.15.7:0/2129631721 shutdown_connections 2024-04-21T08:42:43.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.594+0000 7fb309e48640 1 -- 172.21.15.7:0/2129631721 wait complete. 2024-04-21T08:42:43.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb309e48640 1 Processor -- start 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb309e48640 1 -- start start 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb309e48640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb304100e80 0x7fb304195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb309e48640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304102060 0x7fb304196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb309e48640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb304102a20 0x7fb30419d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb309e48640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb30410b930 con 0x7fb304100e80 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb309e48640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb30410b7b0 con 0x7fb304102a20 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb309e48640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb30410bab0 con 0x7fb304102060 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb3037fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb304100e80 0x7fb304195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb3037fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb304100e80 0x7fb304195f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36092/0 (socket says 172.21.15.7:36092) 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb3037fe640 1 -- 172.21.15.7:0/792181550 learned_addr learned my addr 172.21.15.7:0/792181550 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb2fbfff640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304102060 0x7fb304196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb303fff640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb304102a20 0x7fb30419d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb3037fe640 1 -- 172.21.15.7:0/792181550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304102060 msgr2=0x7fb304196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.595+0000 7fb3037fe640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304102060 0x7fb304196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.596+0000 7fb3037fe640 1 -- 172.21.15.7:0/792181550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb304102a20 msgr2=0x7fb30419d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.596+0000 7fb3037fe640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb304102a20 0x7fb30419d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.596+0000 7fb3037fe640 1 -- 172.21.15.7:0/792181550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30419db50 con 0x7fb304100e80 2024-04-21T08:42:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.596+0000 7fb303fff640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb304102a20 0x7fb30419d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.596+0000 7fb3037fe640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb304100e80 0x7fb304195f30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400e9c0 tx=0x7fb2f400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.596+0000 7fb3017fa640 1 -- 172.21.15.7:0/792181550 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f400cd80 con 0x7fb304100e80 2024-04-21T08:42:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.596+0000 7fb309e48640 1 -- 172.21.15.7:0/792181550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb30419de40 con 0x7fb304100e80 2024-04-21T08:42:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.596+0000 7fb3017fa640 1 -- 172.21.15.7:0/792181550 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2f40035f0 con 0x7fb304100e80 2024-04-21T08:42:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.596+0000 7fb3017fa640 1 -- 172.21.15.7:0/792181550 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4010690 con 0x7fb304100e80 2024-04-21T08:42:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.596+0000 7fb309e48640 1 -- 172.21.15.7:0/792181550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb30419e2a0 con 0x7fb304100e80 2024-04-21T08:42:43.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.597+0000 7fb309e48640 1 -- 172.21.15.7:0/792181550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb304108220 con 0x7fb304100e80 2024-04-21T08:42:43.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.598+0000 7fb3017fa640 1 -- 172.21.15.7:0/792181550 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fb2f40037a0 con 0x7fb304100e80 2024-04-21T08:42:43.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.599+0000 7fb3017fa640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb2f00a3bb0 0x7fb2f00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.599+0000 7fb2fbfff640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb2f00a3bb0 0x7fb2f00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.599+0000 7fb3017fa640 1 -- 172.21.15.7:0/792181550 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb2f40d0ab0 con 0x7fb304100e80 2024-04-21T08:42:43.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.599+0000 7fb2fbfff640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb2f00a3bb0 0x7fb2f00a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb304197440 tx=0x7fb2ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:43.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.603+0000 7fb3017fa640 1 -- 172.21.15.7:0/792181550 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2f409d1f0 con 0x7fb304100e80 2024-04-21T08:42:43.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.741+0000 7fb3017fa640 1 -- 172.21.15.7:0/792181550 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fb2f4099160 con 0x7fb304100e80 2024-04-21T08:42:43.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.829+0000 7fb309e48640 1 -- 172.21.15.7:0/792181550 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fb304062390 con 0x7fb2f00a3bb0 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.835+0000 7fb3017fa640 1 -- 172.21.15.7:0/792181550 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fb304062390 con 0x7fb2f00a3bb0 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi007.stdout:/volumes/qa/sv_0/5462656b-cdec-4b97-8747-25a4dbede0ff 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 -- 172.21.15.7:0/792181550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb2f00a3bb0 msgr2=0x7fb2f00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb2f00a3bb0 0x7fb2f00a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb304197440 tx=0x7fb2ec005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 -- 172.21.15.7:0/792181550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb304100e80 msgr2=0x7fb304195f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb304100e80 0x7fb304195f30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400e9c0 tx=0x7fb2f400ee80 comp rx=0 tx=0).stop 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 -- 172.21.15.7:0/792181550 shutdown_connections 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb2f00a3bb0 0x7fb2f00a6070 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb304102a20 0x7fb30419d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304102060 0x7fb304196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 --2- 172.21.15.7:0/792181550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb304100e80 0x7fb304195f30 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 -- 172.21.15.7:0/792181550 >> 172.21.15.7:0/792181550 conn(0x7fb3040fc420 msgr2=0x7fb3041053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 -- 172.21.15.7:0/792181550 shutdown_connections 2024-04-21T08:42:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.837+0000 7fb309e48640 1 -- 172.21.15.7:0/792181550 wait complete. 2024-04-21T08:42:43.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-21T08:42:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.956+0000 7ffa952eb640 1 -- 172.21.15.7:0/325885237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa90100e90 msgr2=0x7ffa901012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.956+0000 7ffa952eb640 1 --2- 172.21.15.7:0/325885237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa90100e90 0x7ffa901012b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ffa84009900 tx=0x7ffa8402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.956+0000 7ffa952eb640 1 -- 172.21.15.7:0/325885237 shutdown_connections 2024-04-21T08:42:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.956+0000 7ffa952eb640 1 --2- 172.21.15.7:0/325885237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa901029a0 0x7ffa901092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.956+0000 7ffa952eb640 1 --2- 172.21.15.7:0/325885237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa90101fe0 0x7ffa90102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.956+0000 7ffa952eb640 1 --2- 172.21.15.7:0/325885237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa90100e90 0x7ffa901012b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.956+0000 7ffa952eb640 1 -- 172.21.15.7:0/325885237 >> 172.21.15.7:0/325885237 conn(0x7ffa900fc420 msgr2=0x7ffa900fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.956+0000 7ffa952eb640 1 -- 172.21.15.7:0/325885237 shutdown_connections 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.956+0000 7ffa952eb640 1 -- 172.21.15.7:0/325885237 wait complete. 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.957+0000 7ffa952eb640 1 Processor -- start 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.957+0000 7ffa952eb640 1 -- start start 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.957+0000 7ffa952eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa90100e90 0x7ffa90104e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.957+0000 7ffa952eb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa90101fe0 0x7ffa90103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.957+0000 7ffa952eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa901029a0 0x7ffa90103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.957+0000 7ffa952eb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9010c9e0 con 0x7ffa90101fe0 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.957+0000 7ffa952eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9010c860 con 0x7ffa901029a0 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.957+0000 7ffa952eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9010cb60 con 0x7ffa90100e90 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.957+0000 7ffa8e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa90101fe0 0x7ffa90103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.957+0000 7ffa8e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa90101fe0 0x7ffa90103450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36106/0 (socket says 172.21.15.7:36106) 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.957+0000 7ffa8e7fc640 1 -- 172.21.15.7:0/2127681958 learned_addr learned my addr 172.21.15.7:0/2127681958 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa8effd640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa90100e90 0x7ffa90104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa8e7fc640 1 -- 172.21.15.7:0/2127681958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa90100e90 msgr2=0x7ffa90104e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa8e7fc640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa90100e90 0x7ffa90104e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa8e7fc640 1 -- 172.21.15.7:0/2127681958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa901029a0 msgr2=0x7ffa90103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa8f7fe640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa901029a0 0x7ffa90103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa8e7fc640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa901029a0 0x7ffa90103990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa8e7fc640 1 -- 172.21.15.7:0/2127681958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa90104220 con 0x7ffa90101fe0 2024-04-21T08:42:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa8e7fc640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa90101fe0 0x7ffa90103450 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000d950 tx=0x7ffa8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa77fff640 1 -- 172.21.15.7:0/2127681958 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa80017020 con 0x7ffa90101fe0 2024-04-21T08:42:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa77fff640 1 -- 172.21.15.7:0/2127681958 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa80009c30 con 0x7ffa90101fe0 2024-04-21T08:42:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa952eb640 1 -- 172.21.15.7:0/2127681958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa901a2330 con 0x7ffa90101fe0 2024-04-21T08:42:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa77fff640 1 -- 172.21.15.7:0/2127681958 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8000f6b0 con 0x7ffa90101fe0 2024-04-21T08:42:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.958+0000 7ffa952eb640 1 -- 172.21.15.7:0/2127681958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa901a2880 con 0x7ffa90101fe0 2024-04-21T08:42:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.959+0000 7ffa952eb640 1 -- 172.21.15.7:0/2127681958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa901a2ba0 con 0x7ffa90101fe0 2024-04-21T08:42:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.960+0000 7ffa77fff640 1 -- 172.21.15.7:0/2127681958 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7ffa8001a020 con 0x7ffa90101fe0 2024-04-21T08:42:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.961+0000 7ffa77fff640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ffa780a3c40 0x7ffa780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.961+0000 7ffa77fff640 1 -- 172.21.15.7:0/2127681958 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ffa800d0580 con 0x7ffa90101fe0 2024-04-21T08:42:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.961+0000 7ffa8effd640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ffa780a3c40 0x7ffa780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.961+0000 7ffa8effd640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ffa780a3c40 0x7ffa780a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffa84000c00 tx=0x7ffa84005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:43.965+0000 7ffa77fff640 1 -- 172.21.15.7:0/2127681958 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa80009de0 con 0x7ffa90101fe0 2024-04-21T08:42:44.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:43 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1918233574' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:42:44.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:43 smithi016 ceph-mon[31259]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:42:44.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:43 smithi016 ceph-mon[31259]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:44.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:43 smithi016 ceph-mon[31259]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:44.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:43 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1918233574' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:42:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:43 smithi077 ceph-mon[31615]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:42:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:43 smithi077 ceph-mon[31615]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:43 smithi077 ceph-mon[31615]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:44.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:43 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1918233574' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:42:44.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:43 smithi007 ceph-mon[28477]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:42:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:43 smithi007 ceph-mon[28477]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:43 smithi007 ceph-mon[28477]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:44.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.180+0000 7ffa952eb640 1 -- 172.21.15.7:0/2127681958 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ffa90069990 con 0x7ffa780a3c40 2024-04-21T08:42:44.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.207+0000 7ffa77fff640 1 -- 172.21.15.7:0/2127681958 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ffa90069990 con 0x7ffa780a3c40 2024-04-21T08:42:44.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 -- 172.21.15.7:0/2127681958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ffa780a3c40 msgr2=0x7ffa780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ffa780a3c40 0x7ffa780a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffa84000c00 tx=0x7ffa84005be0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 -- 172.21.15.7:0/2127681958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa90101fe0 msgr2=0x7ffa90103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa90101fe0 0x7ffa90103450 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000d950 tx=0x7ffa8000de10 comp rx=0 tx=0).stop 2024-04-21T08:42:44.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 -- 172.21.15.7:0/2127681958 shutdown_connections 2024-04-21T08:42:44.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ffa780a3c40 0x7ffa780a6100 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa901029a0 0x7ffa90103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa90101fe0 0x7ffa90103450 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 --2- 172.21.15.7:0/2127681958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa90100e90 0x7ffa90104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 -- 172.21.15.7:0/2127681958 >> 172.21.15.7:0/2127681958 conn(0x7ffa900fc420 msgr2=0x7ffa90069280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:44.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 -- 172.21.15.7:0/2127681958 shutdown_connections 2024-04-21T08:42:44.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.210+0000 7ffa952eb640 1 -- 172.21.15.7:0/2127681958 wait complete. 2024-04-21T08:42:44.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-21T08:42:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.331+0000 7f83e6c7a640 1 -- 172.21.15.7:0/4090706082 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83e0102a90 msgr2=0x7f83e0104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.331+0000 7f83e6c7a640 1 --2- 172.21.15.7:0/4090706082 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83e0102a90 0x7f83e0104e80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009900 tx=0x7f83d402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.331+0000 7f83e6c7a640 1 -- 172.21.15.7:0/4090706082 shutdown_connections 2024-04-21T08:42:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.331+0000 7f83e6c7a640 1 --2- 172.21.15.7:0/4090706082 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83e0107cf0 0x7f83e010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.331+0000 7f83e6c7a640 1 --2- 172.21.15.7:0/4090706082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e01053c0 0x7f83e01077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.331+0000 7f83e6c7a640 1 --2- 172.21.15.7:0/4090706082 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83e0102a90 0x7f83e0104e80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.331+0000 7f83e6c7a640 1 -- 172.21.15.7:0/4090706082 >> 172.21.15.7:0/4090706082 conn(0x7f83e00fc3c0 msgr2=0x7f83e00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.331+0000 7f83e6c7a640 1 -- 172.21.15.7:0/4090706082 shutdown_connections 2024-04-21T08:42:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.331+0000 7f83e6c7a640 1 -- 172.21.15.7:0/4090706082 wait complete. 2024-04-21T08:42:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83e6c7a640 1 Processor -- start 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83e6c7a640 1 -- start start 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83e6c7a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e0102a90 0x7f83e01981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83e6c7a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83e01053c0 0x7f83e01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83e6c7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83e0107cf0 0x7f83e019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83e6c7a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f83e010dac0 con 0x7f83e01053c0 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83e6c7a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83e010d940 con 0x7f83e0102a90 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83e6c7a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f83e010dc40 con 0x7f83e0107cf0 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83dffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83e01053c0 0x7f83e01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83dffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83e01053c0 0x7f83e01986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36122/0 (socket says 172.21.15.7:36122) 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83dffff640 1 -- 172.21.15.7:0/2158450542 learned_addr learned my addr 172.21.15.7:0/2158450542 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83e4a16640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e0102a90 0x7f83e01981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83e5217640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83e0107cf0 0x7f83e019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83dffff640 1 -- 172.21.15.7:0/2158450542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83e0107cf0 msgr2=0x7f83e019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83dffff640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83e0107cf0 0x7f83e019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83dffff640 1 -- 172.21.15.7:0/2158450542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e0102a90 msgr2=0x7f83e01981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83dffff640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e0102a90 0x7f83e01981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.332+0000 7f83dffff640 1 -- 172.21.15.7:0/2158450542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e019fdc0 con 0x7f83e01053c0 2024-04-21T08:42:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.333+0000 7f83dffff640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83e01053c0 0x7f83e01986e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f83cc00e9c0 tx=0x7f83cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.333+0000 7f83ddffb640 1 -- 172.21.15.7:0/2158450542 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc00cd80 con 0x7f83e01053c0 2024-04-21T08:42:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.333+0000 7f83e6c7a640 1 -- 172.21.15.7:0/2158450542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e01a00b0 con 0x7f83e01053c0 2024-04-21T08:42:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.333+0000 7f83ddffb640 1 -- 172.21.15.7:0/2158450542 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83cc0054d0 con 0x7f83e01053c0 2024-04-21T08:42:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.333+0000 7f83ddffb640 1 -- 172.21.15.7:0/2158450542 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc010430 con 0x7f83e01053c0 2024-04-21T08:42:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.333+0000 7f83e6c7a640 1 -- 172.21.15.7:0/2158450542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e01a0600 con 0x7f83e01053c0 2024-04-21T08:42:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.334+0000 7f83e6c7a640 1 -- 172.21.15.7:0/2158450542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e00fdeb0 con 0x7f83e01053c0 2024-04-21T08:42:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.335+0000 7f83ddffb640 1 -- 172.21.15.7:0/2158450542 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f83cc004fd0 con 0x7f83e01053c0 2024-04-21T08:42:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.335+0000 7f83ddffb640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f83d00a3c40 0x7f83d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.336+0000 7f83ddffb640 1 -- 172.21.15.7:0/2158450542 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f83cc0d0bc0 con 0x7f83e01053c0 2024-04-21T08:42:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.336+0000 7f83e4a16640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f83d00a3c40 0x7f83d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:44.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.336+0000 7f83e4a16640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f83d00a3c40 0x7f83d00a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009230 tx=0x7f83d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:44.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.340+0000 7f83ddffb640 1 -- 172.21.15.7:0/2158450542 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f83cc09d200 con 0x7f83e01053c0 2024-04-21T08:42:44.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.560+0000 7f83e6c7a640 1 -- 172.21.15.7:0/2158450542 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f83e0062390 con 0x7f83d00a3c40 2024-04-21T08:42:44.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.565+0000 7f83ddffb640 1 -- 172.21.15.7:0/2158450542 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f83e0062390 con 0x7f83d00a3c40 2024-04-21T08:42:44.564 INFO:teuthology.orchestra.run.smithi007.stdout:/volumes/qa/sv_1/38a4aff8-de86-46c2-80e8-f71cbab9c8ff 2024-04-21T08:42:44.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 -- 172.21.15.7:0/2158450542 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f83d00a3c40 msgr2=0x7f83d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f83d00a3c40 0x7f83d00a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f83d4009230 tx=0x7f83d4005fd0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 -- 172.21.15.7:0/2158450542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83e01053c0 msgr2=0x7f83e01986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83e01053c0 0x7f83e01986e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f83cc00e9c0 tx=0x7f83cc00ee80 comp rx=0 tx=0).stop 2024-04-21T08:42:44.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 -- 172.21.15.7:0/2158450542 shutdown_connections 2024-04-21T08:42:44.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f83d00a3c40 0x7f83d00a6100 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83e0107cf0 0x7f83e019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83e01053c0 0x7f83e01986e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 --2- 172.21.15.7:0/2158450542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83e0102a90 0x7f83e01981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 -- 172.21.15.7:0/2158450542 >> 172.21.15.7:0/2158450542 conn(0x7f83e00fc3c0 msgr2=0x7f83e0103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:44.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 -- 172.21.15.7:0/2158450542 shutdown_connections 2024-04-21T08:42:44.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.567+0000 7f83e6c7a640 1 -- 172.21.15.7:0/2158450542 wait complete. 2024-04-21T08:42:44.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:44.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.680+0000 7fdbf13f4640 1 -- 172.21.15.7:0/3641925296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbec105330 msgr2=0x7fdbec107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.680+0000 7fdbf13f4640 1 --2- 172.21.15.7:0/3641925296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbec105330 0x7fdbec107720 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009900 tx=0x7fdbe002bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.680+0000 7fdbf13f4640 1 -- 172.21.15.7:0/3641925296 shutdown_connections 2024-04-21T08:42:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.680+0000 7fdbf13f4640 1 --2- 172.21.15.7:0/3641925296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbec107c60 0x7fdbec10a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.680+0000 7fdbf13f4640 1 --2- 172.21.15.7:0/3641925296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbec105330 0x7fdbec107720 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.680+0000 7fdbf13f4640 1 --2- 172.21.15.7:0/3641925296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec102a00 0x7fdbec104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.680+0000 7fdbf13f4640 1 -- 172.21.15.7:0/3641925296 >> 172.21.15.7:0/3641925296 conn(0x7fdbec0fc3f0 msgr2=0x7fdbec0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.680+0000 7fdbf13f4640 1 -- 172.21.15.7:0/3641925296 shutdown_connections 2024-04-21T08:42:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.680+0000 7fdbf13f4640 1 -- 172.21.15.7:0/3641925296 wait complete. 2024-04-21T08:42:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.680+0000 7fdbf13f4640 1 Processor -- start 2024-04-21T08:42:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbf13f4640 1 -- start start 2024-04-21T08:42:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbf13f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec102a00 0x7fdbec198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbf13f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbec105330 0x7fdbec198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbf13f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbec107c60 0x7fdbec19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbf13f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdbec10db00 con 0x7fdbec105330 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbf13f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbec10d980 con 0x7fdbec107c60 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbf13f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdbec10dc80 con 0x7fdbec102a00 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbea7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbec105330 0x7fdbec198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbea7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbec105330 0x7fdbec198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50248/0 (socket says 172.21.15.7:50248) 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbea7fc640 1 -- 172.21.15.7:0/345078780 learned_addr learned my addr 172.21.15.7:0/345078780 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbeaffd640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec102a00 0x7fdbec198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbea7fc640 1 -- 172.21.15.7:0/345078780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec102a00 msgr2=0x7fdbec198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbeb7fe640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbec107c60 0x7fdbec19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbea7fc640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec102a00 0x7fdbec198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbea7fc640 1 -- 172.21.15.7:0/345078780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbec107c60 msgr2=0x7fdbec19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbea7fc640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbec107c60 0x7fdbec19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbea7fc640 1 -- 172.21.15.7:0/345078780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbec19fe20 con 0x7fdbec105330 2024-04-21T08:42:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.681+0000 7fdbea7fc640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbec105330 0x7fdbec198740 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0000c00 tx=0x7fdbe0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.682+0000 7fdbd3fff640 1 -- 172.21.15.7:0/345078780 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0043020 con 0x7fdbec105330 2024-04-21T08:42:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.682+0000 7fdbd3fff640 1 -- 172.21.15.7:0/345078780 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbe0004aa0 con 0x7fdbec105330 2024-04-21T08:42:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.682+0000 7fdbf13f4640 1 -- 172.21.15.7:0/345078780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbec1a00b0 con 0x7fdbec105330 2024-04-21T08:42:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.682+0000 7fdbd3fff640 1 -- 172.21.15.7:0/345078780 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0003d00 con 0x7fdbec105330 2024-04-21T08:42:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.682+0000 7fdbf13f4640 1 -- 172.21.15.7:0/345078780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbec1a0600 con 0x7fdbec105330 2024-04-21T08:42:44.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.683+0000 7fdbf13f4640 1 -- 172.21.15.7:0/345078780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbec0fdee0 con 0x7fdbec105330 2024-04-21T08:42:44.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.684+0000 7fdbd3fff640 1 -- 172.21.15.7:0/345078780 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fdbe0046020 con 0x7fdbec105330 2024-04-21T08:42:44.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.684+0000 7fdbd3fff640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdbd40a3c40 0x7fdbd40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:44.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.685+0000 7fdbeaffd640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdbd40a3c40 0x7fdbd40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:44.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.685+0000 7fdbd3fff640 1 -- 172.21.15.7:0/345078780 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fdbe00f2c60 con 0x7fdbec105330 2024-04-21T08:42:44.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.685+0000 7fdbeaffd640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdbd40a3c40 0x7fdbd40a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc005d00 tx=0x7fdbdc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:44.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.689+0000 7fdbd3fff640 1 -- 172.21.15.7:0/345078780 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdbe004d020 con 0x7fdbec105330 2024-04-21T08:42:44.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:44 smithi007 ceph-mon[28477]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 170 B/s wr, 0 op/s 2024-04-21T08:42:44.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:44 smithi007 ceph-mon[28477]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:42:44.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:44 smithi007 ceph-mon[28477]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:44.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:44 smithi007 ceph-mon[28477]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:44.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.916+0000 7fdbf13f4640 1 -- 172.21.15.7:0/345078780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbec02d810 con 0x7fdbec105330 2024-04-21T08:42:44.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.917+0000 7fdbd3fff640 1 -- 172.21.15.7:0/345078780 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7fdbe00bf3b0 con 0x7fdbec105330 2024-04-21T08:42:44.916 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:42:44.917 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 -- 172.21.15.7:0/345078780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdbd40a3c40 msgr2=0x7fdbd40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdbd40a3c40 0x7fdbd40a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc005d00 tx=0x7fdbdc00b410 comp rx=0 tx=0).stop 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 -- 172.21.15.7:0/345078780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbec105330 msgr2=0x7fdbec198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbec105330 0x7fdbec198740 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0000c00 tx=0x7fdbe0004270 comp rx=0 tx=0).stop 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 -- 172.21.15.7:0/345078780 shutdown_connections 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdbd40a3c40 0x7fdbd40a6100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbec107c60 0x7fdbec19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbec105330 0x7fdbec198740 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 --2- 172.21.15.7:0/345078780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbec102a00 0x7fdbec198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 -- 172.21.15.7:0/345078780 >> 172.21.15.7:0/345078780 conn(0x7fdbec0fc3f0 msgr2=0x7fdbec1034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 -- 172.21.15.7:0/345078780 shutdown_connections 2024-04-21T08:42:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:44.919+0000 7fdbf13f4640 1 -- 172.21.15.7:0/345078780 wait complete. 2024-04-21T08:42:44.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:42:45.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.032+0000 7f4acb59e640 1 -- 172.21.15.7:0/450258603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4acc100c90 msgr2=0x7f4acc1010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.032+0000 7f4acb59e640 1 --2- 172.21.15.7:0/450258603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4acc100c90 0x7f4acc1010b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009900 tx=0x7f4abc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:45.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.032+0000 7f4acb59e640 1 -- 172.21.15.7:0/450258603 shutdown_connections 2024-04-21T08:42:45.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.032+0000 7f4acb59e640 1 --2- 172.21.15.7:0/450258603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4acc102840 0x7f4acc1090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.032+0000 7f4acb59e640 1 --2- 172.21.15.7:0/450258603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4acc101e80 0x7f4acc102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.032+0000 7f4acb59e640 1 --2- 172.21.15.7:0/450258603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4acc100c90 0x7f4acc1010b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.032+0000 7f4acb59e640 1 -- 172.21.15.7:0/450258603 >> 172.21.15.7:0/450258603 conn(0x7f4acc0fc220 msgr2=0x7f4acc0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:45.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.032+0000 7f4acb59e640 1 -- 172.21.15.7:0/450258603 shutdown_connections 2024-04-21T08:42:45.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.032+0000 7f4acb59e640 1 -- 172.21.15.7:0/450258603 wait complete. 2024-04-21T08:42:45.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.032+0000 7f4acb59e640 1 Processor -- start 2024-04-21T08:42:45.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.032+0000 7f4acb59e640 1 -- start start 2024-04-21T08:42:45.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acb59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4acc100c90 0x7f4acc069de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acb59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4acc101e80 0x7f4acc19cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acb59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4acc102840 0x7f4acc19f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acb59e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc10c7e0 con 0x7f4acc102840 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acb59e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc10c660 con 0x7f4acc101e80 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acb59e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc10c960 con 0x7f4acc100c90 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4acc102840 0x7f4acc19f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4acc102840 0x7f4acc19f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50270/0 (socket says 172.21.15.7:50270) 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acad9d640 1 -- 172.21.15.7:0/247816535 learned_addr learned my addr 172.21.15.7:0/247816535 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4aca59c640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4acc100c90 0x7f4acc069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4ac9d9b640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4acc101e80 0x7f4acc19cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acad9d640 1 -- 172.21.15.7:0/247816535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4acc100c90 msgr2=0x7f4acc069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acad9d640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4acc100c90 0x7f4acc069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acad9d640 1 -- 172.21.15.7:0/247816535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4acc101e80 msgr2=0x7f4acc19cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acad9d640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4acc101e80 0x7f4acc19cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acad9d640 1 -- 172.21.15.7:0/247816535 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc19fa70 con 0x7f4acc102840 2024-04-21T08:42:45.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4ac9d9b640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4acc101e80 0x7f4acc19cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:45.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acad9d640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4acc102840 0x7f4acc19f1b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400eaa0 tx=0x7f4ac400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:45.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4ac37fe640 1 -- 172.21.15.7:0/247816535 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac400cdb0 con 0x7f4acc102840 2024-04-21T08:42:45.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4ac37fe640 1 -- 172.21.15.7:0/247816535 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ac4005020 con 0x7f4acc102840 2024-04-21T08:42:45.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.033+0000 7f4acb59e640 1 -- 172.21.15.7:0/247816535 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4acc19fd60 con 0x7f4acc102840 2024-04-21T08:42:45.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.034+0000 7f4ac37fe640 1 -- 172.21.15.7:0/247816535 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac40116d0 con 0x7f4acc102840 2024-04-21T08:42:45.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.034+0000 7f4acb59e640 1 -- 172.21.15.7:0/247816535 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4acc1a02b0 con 0x7f4acc102840 2024-04-21T08:42:45.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.035+0000 7f4acb59e640 1 -- 172.21.15.7:0/247816535 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4acc107f50 con 0x7f4acc102840 2024-04-21T08:42:45.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.036+0000 7f4ac37fe640 1 -- 172.21.15.7:0/247816535 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f4ac40051d0 con 0x7f4acc102840 2024-04-21T08:42:45.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.036+0000 7f4ac37fe640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ab80a3d70 0x7f4ab80a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.036+0000 7f4aca59c640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ab80a3d70 0x7f4ab80a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.036+0000 7f4ac37fe640 1 -- 172.21.15.7:0/247816535 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4ac40d0a00 con 0x7f4acc102840 2024-04-21T08:42:45.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.037+0000 7f4aca59c640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ab80a3d70 0x7f4ab80a6230 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4abc02c240 tx=0x7f4abc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:45.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.041+0000 7f4ac37fe640 1 -- 172.21.15.7:0/247816535 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ac40d2e20 con 0x7f4acc102840 2024-04-21T08:42:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:44 smithi016 ceph-mon[31259]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 170 B/s wr, 0 op/s 2024-04-21T08:42:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:44 smithi016 ceph-mon[31259]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:42:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:44 smithi016 ceph-mon[31259]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:44 smithi016 ceph-mon[31259]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:45.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:44 smithi077 ceph-mon[31615]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 170 B/s wr, 0 op/s 2024-04-21T08:42:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:44 smithi077 ceph-mon[31615]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:42:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:44 smithi077 ceph-mon[31615]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:44 smithi077 ceph-mon[31615]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:45.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.228+0000 7f4acb59e640 1 -- 172.21.15.7:0/247816535 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4acc02d790 con 0x7f4acc102840 2024-04-21T08:42:45.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.230+0000 7f4ac37fe640 1 -- 172.21.15.7:0/247816535 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23961 (secure 0 0 0) 0x7f4ac4016030 con 0x7f4acc102840 2024-04-21T08:42:45.230 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:45.230 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":90,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:42:22.023111+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:45.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 -- 172.21.15.7:0/247816535 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ab80a3d70 msgr2=0x7f4ab80a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ab80a3d70 0x7f4ab80a6230 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4abc02c240 tx=0x7f4abc005fd0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 -- 172.21.15.7:0/247816535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4acc102840 msgr2=0x7f4acc19f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4acc102840 0x7f4acc19f1b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400eaa0 tx=0x7f4ac400ef60 comp rx=0 tx=0).stop 2024-04-21T08:42:45.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 -- 172.21.15.7:0/247816535 shutdown_connections 2024-04-21T08:42:45.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4ab80a3d70 0x7f4ab80a6230 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4acc102840 0x7f4acc19f1b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4acc101e80 0x7f4acc19cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 --2- 172.21.15.7:0/247816535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4acc100c90 0x7f4acc069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 -- 172.21.15.7:0/247816535 >> 172.21.15.7:0/247816535 conn(0x7f4acc0fc220 msgr2=0x7f4acc105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:45.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 -- 172.21.15.7:0/247816535 shutdown_connections 2024-04-21T08:42:45.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.232+0000 7f4acb59e640 1 -- 172.21.15.7:0/247816535 wait complete. 2024-04-21T08:42:45.243 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-21T08:42:45.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:42:45.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.345+0000 7f5244ffd640 1 -- 172.21.15.7:0/595944293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240103190 msgr2=0x7f5240103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.345+0000 7f5244ffd640 1 --2- 172.21.15.7:0/595944293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240103190 0x7f5240103610 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5234009900 tx=0x7f523402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:45.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.345+0000 7f5244ffd640 1 -- 172.21.15.7:0/595944293 shutdown_connections 2024-04-21T08:42:45.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.345+0000 7f5244ffd640 1 --2- 172.21.15.7:0/595944293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5240103b50 0x7f524010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.345+0000 7f5244ffd640 1 --2- 172.21.15.7:0/595944293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240103190 0x7f5240103610 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.345+0000 7f5244ffd640 1 --2- 172.21.15.7:0/595944293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5240101fa0 0x7f52401023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.345+0000 7f5244ffd640 1 -- 172.21.15.7:0/595944293 >> 172.21.15.7:0/595944293 conn(0x7f52400fd570 msgr2=0x7f52400ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.346+0000 7f5244ffd640 1 -- 172.21.15.7:0/595944293 shutdown_connections 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.346+0000 7f5244ffd640 1 -- 172.21.15.7:0/595944293 wait complete. 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.346+0000 7f5244ffd640 1 Processor -- start 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.346+0000 7f5244ffd640 1 -- start start 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.346+0000 7f5244ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5240101fa0 0x7f5240198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.346+0000 7f5244ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5240103190 0x7f5240198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.346+0000 7f5244ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240103b50 0x7f524019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.346+0000 7f5244ffd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f524010dae0 con 0x7f5240103b50 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f5244ffd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f524010d960 con 0x7f5240103190 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f5244ffd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f524010dc60 con 0x7f5240101fa0 2024-04-21T08:42:45.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f523ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240103b50 0x7f524019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f523ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240103b50 0x7f524019f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50274/0 (socket says 172.21.15.7:50274) 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f523ed9d640 1 -- 172.21.15.7:0/1218962962 learned_addr learned my addr 172.21.15.7:0/1218962962 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f523dd9b640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5240103190 0x7f5240198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f523e59c640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5240101fa0 0x7f5240198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f523ed9d640 1 -- 172.21.15.7:0/1218962962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5240101fa0 msgr2=0x7f5240198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f523ed9d640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5240101fa0 0x7f5240198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f523ed9d640 1 -- 172.21.15.7:0/1218962962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5240103190 msgr2=0x7f5240198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f523ed9d640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5240103190 0x7f5240198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f523ed9d640 1 -- 172.21.15.7:0/1218962962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524019fd20 con 0x7f5240103b50 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f523ed9d640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240103b50 0x7f524019f6b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f523800cce0 tx=0x7f523800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f522f7fe640 1 -- 172.21.15.7:0/1218962962 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523800f450 con 0x7f5240103b50 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f522f7fe640 1 -- 172.21.15.7:0/1218962962 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f523800faa0 con 0x7f5240103b50 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f5244ffd640 1 -- 172.21.15.7:0/1218962962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f524019ffb0 con 0x7f5240103b50 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f522f7fe640 1 -- 172.21.15.7:0/1218962962 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5238018b70 con 0x7f5240103b50 2024-04-21T08:42:45.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.347+0000 7f5244ffd640 1 -- 172.21.15.7:0/1218962962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52401a04d0 con 0x7f5240103b50 2024-04-21T08:42:45.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.349+0000 7f5244ffd640 1 -- 172.21.15.7:0/1218962962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52401092c0 con 0x7f5240103b50 2024-04-21T08:42:45.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.350+0000 7f522f7fe640 1 -- 172.21.15.7:0/1218962962 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f5238018d10 con 0x7f5240103b50 2024-04-21T08:42:45.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.350+0000 7f522f7fe640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52280a3cc0 0x7f52280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.351+0000 7f523e59c640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52280a3cc0 0x7f52280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.351+0000 7f522f7fe640 1 -- 172.21.15.7:0/1218962962 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f52380d0bb0 con 0x7f5240103b50 2024-04-21T08:42:45.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.351+0000 7f523e59c640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52280a3cc0 0x7f52280a6180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5230005fd0 tx=0x7f5230005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:45.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.355+0000 7f522f7fe640 1 -- 172.21.15.7:0/1218962962 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5238014030 con 0x7f5240103b50 2024-04-21T08:42:45.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.537+0000 7f5244ffd640 1 -- 172.21.15.7:0/1218962962 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5240062390 con 0x7f52280a3cc0 2024-04-21T08:42:45.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.547+0000 7f522f7fe640 1 -- 172.21.15.7:0/1218962962 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340465 (secure 0 0 0) 0x7f5240062390 con 0x7f52280a3cc0 2024-04-21T08:42:45.550 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:45.552 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-21T08:42:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.550+0000 7f5244ffd640 1 -- 172.21.15.7:0/1218962962 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52280a3cc0 msgr2=0x7f52280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.550+0000 7f5244ffd640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52280a3cc0 0x7f52280a6180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5230005fd0 tx=0x7f5230005850 comp rx=0 tx=0).stop 2024-04-21T08:42:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.550+0000 7f5244ffd640 1 -- 172.21.15.7:0/1218962962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240103b50 msgr2=0x7f524019f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.550+0000 7f5244ffd640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240103b50 0x7f524019f6b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f523800cce0 tx=0x7f523800a620 comp rx=0 tx=0).stop 2024-04-21T08:42:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.550+0000 7f5244ffd640 1 -- 172.21.15.7:0/1218962962 shutdown_connections 2024-04-21T08:42:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.550+0000 7f5244ffd640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52280a3cc0 0x7f52280a6180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.550+0000 7f5244ffd640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5240103b50 0x7f524019f6b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.550+0000 7f5244ffd640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5240103190 0x7f5240198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.550+0000 7f5244ffd640 1 --2- 172.21.15.7:0/1218962962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5240101fa0 0x7f5240198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.550+0000 7f5244ffd640 1 -- 172.21.15.7:0/1218962962 >> 172.21.15.7:0/1218962962 conn(0x7f52400fd570 msgr2=0x7f5240106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:45.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.551+0000 7f5244ffd640 1 -- 172.21.15.7:0/1218962962 shutdown_connections 2024-04-21T08:42:45.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.551+0000 7f5244ffd640 1 -- 172.21.15.7:0/1218962962 wait complete. 2024-04-21T08:42:45.561 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-04-21T08:42:43.633595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":138,"num_read_kb":121,"num_write":230,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":402444,"kb_used_data":11616,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124539380,"statfs":{"total":1151940427776,"available":1151528325120,"internally_reserved":0,"allocated":11894784,"data_stored":7548216,"data_compressed":284238,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004031"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768636+0000","last_change":"2024-04-21T08:42:17.545701+0000","last_active":"2024-04-21T08:42:22.768636+0000","last_peered":"2024-04-21T08:42:22.768636+0000","last_clean":"2024-04-21T08:42:22.768636+0000","last_became_active":"2024-04-21T08:42:17.545157+0000","last_became_peered":"2024-04-21T08:42:17.545157+0000","last_unstale":"2024-04-21T08:42:22.768636+0000","last_undegraded":"2024-04-21T08:42:22.768636+0000","last_fullsized":"2024-04-21T08:42:22.768636+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:05:41.952879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091903+0000","last_change":"2024-04-21T08:42:17.541153+0000","last_active":"2024-04-21T08:42:23.091903+0000","last_peered":"2024-04-21T08:42:23.091903+0000","last_clean":"2024-04-21T08:42:23.091903+0000","last_became_active":"2024-04-21T08:42:17.540771+0000","last_became_peered":"2024-04-21T08:42:17.540771+0000","last_unstale":"2024-04-21T08:42:23.091903+0000","last_undegraded":"2024-04-21T08:42:23.091903+0000","last_fullsized":"2024-04-21T08:42:23.091903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:35:06.077367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091371+0000","last_change":"2024-04-21T08:42:17.545525+0000","last_active":"2024-04-21T08:42:23.091371+0000","last_peered":"2024-04-21T08:42:23.091371+0000","last_clean":"2024-04-21T08:42:23.091371+0000","last_became_active":"2024-04-21T08:42:17.545168+0000","last_became_peered":"2024-04-21T08:42:17.545168+0000","last_unstale":"2024-04-21T08:42:23.091371+0000","last_undegraded":"2024-04-21T08:42:23.091371+0000","last_fullsized":"2024-04-21T08:42:23.091371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:51.080385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091605+0000","last_change":"2024-04-21T08:42:17.543917+0000","last_active":"2024-04-21T08:42:23.091605+0000","last_peered":"2024-04-21T08:42:23.091605+0000","last_clean":"2024-04-21T08:42:23.091605+0000","last_became_active":"2024-04-21T08:42:17.543513+0000","last_became_peered":"2024-04-21T08:42:17.543513+0000","last_unstale":"2024-04-21T08:42:23.091605+0000","last_undegraded":"2024-04-21T08:42:23.091605+0000","last_fullsized":"2024-04-21T08:42:23.091605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:23.058600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951634+0000","last_change":"2024-04-21T08:42:17.549737+0000","last_active":"2024-04-21T08:42:22.951634+0000","last_peered":"2024-04-21T08:42:22.951634+0000","last_clean":"2024-04-21T08:42:22.951634+0000","last_became_active":"2024-04-21T08:42:17.549396+0000","last_became_peered":"2024-04-21T08:42:17.549396+0000","last_unstale":"2024-04-21T08:42:22.951634+0000","last_undegraded":"2024-04-21T08:42:22.951634+0000","last_fullsized":"2024-04-21T08:42:22.951634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:01:51.879960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771397+0000","last_change":"2024-04-21T08:42:17.538676+0000","last_active":"2024-04-21T08:42:22.771397+0000","last_peered":"2024-04-21T08:42:22.771397+0000","last_clean":"2024-04-21T08:42:22.771397+0000","last_became_active":"2024-04-21T08:42:17.538252+0000","last_became_peered":"2024-04-21T08:42:17.538252+0000","last_unstale":"2024-04-21T08:42:22.771397+0000","last_undegraded":"2024-04-21T08:42:22.771397+0000","last_fullsized":"2024-04-21T08:42:22.771397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:10:24.081715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091199+0000","last_change":"2024-04-21T08:42:17.541172+0000","last_active":"2024-04-21T08:42:23.091199+0000","last_peered":"2024-04-21T08:42:23.091199+0000","last_clean":"2024-04-21T08:42:23.091199+0000","last_became_active":"2024-04-21T08:42:17.540829+0000","last_became_peered":"2024-04-21T08:42:17.540829+0000","last_unstale":"2024-04-21T08:42:23.091199+0000","last_undegraded":"2024-04-21T08:42:23.091199+0000","last_fullsized":"2024-04-21T08:42:23.091199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:16:12.633980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090834+0000","last_change":"2024-04-21T08:42:17.549744+0000","last_active":"2024-04-21T08:42:23.090834+0000","last_peered":"2024-04-21T08:42:23.090834+0000","last_clean":"2024-04-21T08:42:23.090834+0000","last_became_active":"2024-04-21T08:42:17.549378+0000","last_became_peered":"2024-04-21T08:42:17.549378+0000","last_unstale":"2024-04-21T08:42:23.090834+0000","last_undegraded":"2024-04-21T08:42:23.090834+0000","last_fullsized":"2024-04-21T08:42:23.090834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:03:07.702143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664941+0000","last_change":"2024-04-21T08:42:17.548675+0000","last_active":"2024-04-21T08:42:22.664941+0000","last_peered":"2024-04-21T08:42:22.664941+0000","last_clean":"2024-04-21T08:42:22.664941+0000","last_became_active":"2024-04-21T08:42:17.547809+0000","last_became_peered":"2024-04-21T08:42:17.547809+0000","last_unstale":"2024-04-21T08:42:22.664941+0000","last_undegraded":"2024-04-21T08:42:22.664941+0000","last_fullsized":"2024-04-21T08:42:22.664941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:23.343473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771859+0000","last_change":"2024-04-21T08:42:17.551012+0000","last_active":"2024-04-21T08:42:22.771859+0000","last_peered":"2024-04-21T08:42:22.771859+0000","last_clean":"2024-04-21T08:42:22.771859+0000","last_became_active":"2024-04-21T08:42:17.550627+0000","last_became_peered":"2024-04-21T08:42:17.550627+0000","last_unstale":"2024-04-21T08:42:22.771859+0000","last_undegraded":"2024-04-21T08:42:22.771859+0000","last_fullsized":"2024-04-21T08:42:22.771859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:57.501588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091724+0000","last_change":"2024-04-21T08:42:17.546192+0000","last_active":"2024-04-21T08:42:23.091724+0000","last_peered":"2024-04-21T08:42:23.091724+0000","last_clean":"2024-04-21T08:42:23.091724+0000","last_became_active":"2024-04-21T08:42:17.545808+0000","last_became_peered":"2024-04-21T08:42:17.545808+0000","last_unstale":"2024-04-21T08:42:23.091724+0000","last_undegraded":"2024-04-21T08:42:23.091724+0000","last_fullsized":"2024-04-21T08:42:23.091724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:54.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091491+0000","last_change":"2024-04-21T08:42:17.553351+0000","last_active":"2024-04-21T08:42:23.091491+0000","last_peered":"2024-04-21T08:42:23.091491+0000","last_clean":"2024-04-21T08:42:23.091491+0000","last_became_active":"2024-04-21T08:42:17.550507+0000","last_became_peered":"2024-04-21T08:42:17.550507+0000","last_unstale":"2024-04-21T08:42:23.091491+0000","last_undegraded":"2024-04-21T08:42:23.091491+0000","last_fullsized":"2024-04-21T08:42:23.091491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:25.561375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664867+0000","last_change":"2024-04-21T08:42:17.540533+0000","last_active":"2024-04-21T08:42:22.664867+0000","last_peered":"2024-04-21T08:42:22.664867+0000","last_clean":"2024-04-21T08:42:22.664867+0000","last_became_active":"2024-04-21T08:42:17.539940+0000","last_became_peered":"2024-04-21T08:42:17.539940+0000","last_unstale":"2024-04-21T08:42:22.664867+0000","last_undegraded":"2024-04-21T08:42:22.664867+0000","last_fullsized":"2024-04-21T08:42:22.664867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:45:19.102899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091016+0000","last_change":"2024-04-21T08:42:17.537794+0000","last_active":"2024-04-21T08:42:23.091016+0000","last_peered":"2024-04-21T08:42:23.091016+0000","last_clean":"2024-04-21T08:42:23.091016+0000","last_became_active":"2024-04-21T08:42:17.537376+0000","last_became_peered":"2024-04-21T08:42:17.537376+0000","last_unstale":"2024-04-21T08:42:23.091016+0000","last_undegraded":"2024-04-21T08:42:23.091016+0000","last_fullsized":"2024-04-21T08:42:23.091016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:57.430036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091693+0000","last_change":"2024-04-21T08:42:17.550574+0000","last_active":"2024-04-21T08:42:23.091693+0000","last_peered":"2024-04-21T08:42:23.091693+0000","last_clean":"2024-04-21T08:42:23.091693+0000","last_became_active":"2024-04-21T08:42:17.550205+0000","last_became_peered":"2024-04-21T08:42:17.550205+0000","last_unstale":"2024-04-21T08:42:23.091693+0000","last_undegraded":"2024-04-21T08:42:23.091693+0000","last_fullsized":"2024-04-21T08:42:23.091693+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:41:08.226477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664131+0000","last_change":"2024-04-21T08:42:17.542239+0000","last_active":"2024-04-21T08:42:22.664131+0000","last_peered":"2024-04-21T08:42:22.664131+0000","last_clean":"2024-04-21T08:42:22.664131+0000","last_became_active":"2024-04-21T08:42:17.541848+0000","last_became_peered":"2024-04-21T08:42:17.541848+0000","last_unstale":"2024-04-21T08:42:22.664131+0000","last_undegraded":"2024-04-21T08:42:22.664131+0000","last_fullsized":"2024-04-21T08:42:22.664131+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:47.717451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768476+0000","last_change":"2024-04-21T08:42:17.553200+0000","last_active":"2024-04-21T08:42:22.768476+0000","last_peered":"2024-04-21T08:42:22.768476+0000","last_clean":"2024-04-21T08:42:22.768476+0000","last_became_active":"2024-04-21T08:42:17.552913+0000","last_became_peered":"2024-04-21T08:42:17.552913+0000","last_unstale":"2024-04-21T08:42:22.768476+0000","last_undegraded":"2024-04-21T08:42:22.768476+0000","last_fullsized":"2024-04-21T08:42:22.768476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:39:04.186928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680572+0000","last_change":"2024-04-21T08:42:17.534881+0000","last_active":"2024-04-21T08:42:22.680572+0000","last_peered":"2024-04-21T08:42:22.680572+0000","last_clean":"2024-04-21T08:42:22.680572+0000","last_became_active":"2024-04-21T08:42:17.534496+0000","last_became_peered":"2024-04-21T08:42:17.534496+0000","last_unstale":"2024-04-21T08:42:22.680572+0000","last_undegraded":"2024-04-21T08:42:22.680572+0000","last_fullsized":"2024-04-21T08:42:22.680572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:50:19.409391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090574+0000","last_change":"2024-04-21T08:42:17.540480+0000","last_active":"2024-04-21T08:42:23.090574+0000","last_peered":"2024-04-21T08:42:23.090574+0000","last_clean":"2024-04-21T08:42:23.090574+0000","last_became_active":"2024-04-21T08:42:17.539742+0000","last_became_peered":"2024-04-21T08:42:17.539742+0000","last_unstale":"2024-04-21T08:42:23.090574+0000","last_undegraded":"2024-04-21T08:42:23.090574+0000","last_fullsized":"2024-04-21T08:42:23.090574+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:00.238976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090525+0000","last_change":"2024-04-21T08:42:21.661514+0000","last_active":"2024-04-21T08:42:23.090525+0000","last_peered":"2024-04-21T08:42:23.090525+0000","last_clean":"2024-04-21T08:42:23.090525+0000","last_became_active":"2024-04-21T08:42:21.661095+0000","last_became_peered":"2024-04-21T08:42:21.661095+0000","last_unstale":"2024-04-21T08:42:23.090525+0000","last_undegraded":"2024-04-21T08:42:23.090525+0000","last_fullsized":"2024-04-21T08:42:23.090525+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:27:28.933446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680638+0000","last_change":"2024-04-21T08:42:16.639888+0000","last_active":"2024-04-21T08:42:22.680638+0000","last_peered":"2024-04-21T08:42:22.680638+0000","last_clean":"2024-04-21T08:42:22.680638+0000","last_became_active":"2024-04-21T08:42:16.639403+0000","last_became_peered":"2024-04-21T08:42:16.639403+0000","last_unstale":"2024-04-21T08:42:22.680638+0000","last_undegraded":"2024-04-21T08:42:22.680638+0000","last_fullsized":"2024-04-21T08:42:22.680638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:08:17.164924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091798+0000","last_change":"2024-04-21T08:42:17.542557+0000","last_active":"2024-04-21T08:42:23.091798+0000","last_peered":"2024-04-21T08:42:23.091798+0000","last_clean":"2024-04-21T08:42:23.091798+0000","last_became_active":"2024-04-21T08:42:17.542196+0000","last_became_peered":"2024-04-21T08:42:17.542196+0000","last_unstale":"2024-04-21T08:42:23.091798+0000","last_undegraded":"2024-04-21T08:42:23.091798+0000","last_fullsized":"2024-04-21T08:42:23.091798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:26:33.153918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091987+0000","last_change":"2024-04-21T08:42:21.673197+0000","last_active":"2024-04-21T08:42:23.091987+0000","last_peered":"2024-04-21T08:42:23.091987+0000","last_clean":"2024-04-21T08:42:23.091987+0000","last_became_active":"2024-04-21T08:42:21.672793+0000","last_became_peered":"2024-04-21T08:42:21.672793+0000","last_unstale":"2024-04-21T08:42:23.091987+0000","last_undegraded":"2024-04-21T08:42:23.091987+0000","last_fullsized":"2024-04-21T08:42:23.091987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:44:29.788459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092057+0000","last_change":"2024-04-21T08:42:16.648577+0000","last_active":"2024-04-21T08:42:23.092057+0000","last_peered":"2024-04-21T08:42:23.092057+0000","last_clean":"2024-04-21T08:42:23.092057+0000","last_became_active":"2024-04-21T08:42:16.646716+0000","last_became_peered":"2024-04-21T08:42:16.646716+0000","last_unstale":"2024-04-21T08:42:23.092057+0000","last_undegraded":"2024-04-21T08:42:23.092057+0000","last_fullsized":"2024-04-21T08:42:23.092057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:10.327078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092011+0000","last_change":"2024-04-21T08:42:17.545430+0000","last_active":"2024-04-21T08:42:23.092011+0000","last_peered":"2024-04-21T08:42:23.092011+0000","last_clean":"2024-04-21T08:42:23.092011+0000","last_became_active":"2024-04-21T08:42:17.545057+0000","last_became_peered":"2024-04-21T08:42:17.545057+0000","last_unstale":"2024-04-21T08:42:23.092011+0000","last_undegraded":"2024-04-21T08:42:23.092011+0000","last_fullsized":"2024-04-21T08:42:23.092011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:44:59.030830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091987+0000","last_change":"2024-04-21T08:42:21.673196+0000","last_active":"2024-04-21T08:42:23.091987+0000","last_peered":"2024-04-21T08:42:23.091987+0000","last_clean":"2024-04-21T08:42:23.091987+0000","last_became_active":"2024-04-21T08:42:21.672808+0000","last_became_peered":"2024-04-21T08:42:21.672808+0000","last_unstale":"2024-04-21T08:42:23.091987+0000","last_undegraded":"2024-04-21T08:42:23.091987+0000","last_fullsized":"2024-04-21T08:42:23.091987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:39.482353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951824+0000","last_change":"2024-04-21T08:42:16.641611+0000","last_active":"2024-04-21T08:42:22.951824+0000","last_peered":"2024-04-21T08:42:22.951824+0000","last_clean":"2024-04-21T08:42:22.951824+0000","last_became_active":"2024-04-21T08:42:16.641221+0000","last_became_peered":"2024-04-21T08:42:16.641221+0000","last_unstale":"2024-04-21T08:42:22.951824+0000","last_undegraded":"2024-04-21T08:42:22.951824+0000","last_fullsized":"2024-04-21T08:42:22.951824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:45.496532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664491+0000","last_change":"2024-04-21T08:42:17.532986+0000","last_active":"2024-04-21T08:42:22.664491+0000","last_peered":"2024-04-21T08:42:22.664491+0000","last_clean":"2024-04-21T08:42:22.664491+0000","last_became_active":"2024-04-21T08:42:17.532583+0000","last_became_peered":"2024-04-21T08:42:17.532583+0000","last_unstale":"2024-04-21T08:42:22.664491+0000","last_undegraded":"2024-04-21T08:42:22.664491+0000","last_fullsized":"2024-04-21T08:42:22.664491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:43.780935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092145+0000","last_change":"2024-04-21T08:42:21.667459+0000","last_active":"2024-04-21T08:42:23.092145+0000","last_peered":"2024-04-21T08:42:23.092145+0000","last_clean":"2024-04-21T08:42:23.092145+0000","last_became_active":"2024-04-21T08:42:21.667183+0000","last_became_peered":"2024-04-21T08:42:21.667183+0000","last_unstale":"2024-04-21T08:42:23.092145+0000","last_undegraded":"2024-04-21T08:42:23.092145+0000","last_fullsized":"2024-04-21T08:42:23.092145+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:32:23.626267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951843+0000","last_change":"2024-04-21T08:42:16.645210+0000","last_active":"2024-04-21T08:42:22.951843+0000","last_peered":"2024-04-21T08:42:22.951843+0000","last_clean":"2024-04-21T08:42:22.951843+0000","last_became_active":"2024-04-21T08:42:16.644747+0000","last_became_peered":"2024-04-21T08:42:16.644747+0000","last_unstale":"2024-04-21T08:42:22.951843+0000","last_undegraded":"2024-04-21T08:42:22.951843+0000","last_fullsized":"2024-04-21T08:42:22.951843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:34:45.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664370+0000","last_change":"2024-04-21T08:42:17.547110+0000","last_active":"2024-04-21T08:42:22.664370+0000","last_peered":"2024-04-21T08:42:22.664370+0000","last_clean":"2024-04-21T08:42:22.664370+0000","last_became_active":"2024-04-21T08:42:17.546715+0000","last_became_peered":"2024-04-21T08:42:17.546715+0000","last_unstale":"2024-04-21T08:42:22.664370+0000","last_undegraded":"2024-04-21T08:42:22.664370+0000","last_fullsized":"2024-04-21T08:42:22.664370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:26:51.780113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090612+0000","last_change":"2024-04-21T08:42:21.660739+0000","last_active":"2024-04-21T08:42:23.090612+0000","last_peered":"2024-04-21T08:42:23.090612+0000","last_clean":"2024-04-21T08:42:23.090612+0000","last_became_active":"2024-04-21T08:42:21.660307+0000","last_became_peered":"2024-04-21T08:42:21.660307+0000","last_unstale":"2024-04-21T08:42:23.090612+0000","last_undegraded":"2024-04-21T08:42:23.090612+0000","last_fullsized":"2024-04-21T08:42:23.090612+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:26.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664111+0000","last_change":"2024-04-21T08:42:16.639865+0000","last_active":"2024-04-21T08:42:22.664111+0000","last_peered":"2024-04-21T08:42:22.664111+0000","last_clean":"2024-04-21T08:42:22.664111+0000","last_became_active":"2024-04-21T08:42:16.639539+0000","last_became_peered":"2024-04-21T08:42:16.639539+0000","last_unstale":"2024-04-21T08:42:22.664111+0000","last_undegraded":"2024-04-21T08:42:22.664111+0000","last_fullsized":"2024-04-21T08:42:22.664111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:33.645493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951535+0000","last_change":"2024-04-21T08:42:17.542315+0000","last_active":"2024-04-21T08:42:22.951535+0000","last_peered":"2024-04-21T08:42:22.951535+0000","last_clean":"2024-04-21T08:42:22.951535+0000","last_became_active":"2024-04-21T08:42:17.541895+0000","last_became_peered":"2024-04-21T08:42:17.541895+0000","last_unstale":"2024-04-21T08:42:22.951535+0000","last_undegraded":"2024-04-21T08:42:22.951535+0000","last_fullsized":"2024-04-21T08:42:22.951535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:23:33.909499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090736+0000","last_change":"2024-04-21T08:42:21.661541+0000","last_active":"2024-04-21T08:42:23.090736+0000","last_peered":"2024-04-21T08:42:23.090736+0000","last_clean":"2024-04-21T08:42:23.090736+0000","last_became_active":"2024-04-21T08:42:21.661121+0000","last_became_peered":"2024-04-21T08:42:21.661121+0000","last_unstale":"2024-04-21T08:42:23.090736+0000","last_undegraded":"2024-04-21T08:42:23.090736+0000","last_fullsized":"2024-04-21T08:42:23.090736+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:28.624203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771355+0000","last_change":"2024-04-21T08:42:16.644402+0000","last_active":"2024-04-21T08:42:22.771355+0000","last_peered":"2024-04-21T08:42:22.771355+0000","last_clean":"2024-04-21T08:42:22.771355+0000","last_became_active":"2024-04-21T08:42:16.643928+0000","last_became_peered":"2024-04-21T08:42:16.643928+0000","last_unstale":"2024-04-21T08:42:22.771355+0000","last_undegraded":"2024-04-21T08:42:22.771355+0000","last_fullsized":"2024-04-21T08:42:22.771355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:54:14.154892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664231+0000","last_change":"2024-04-21T08:42:17.547770+0000","last_active":"2024-04-21T08:42:22.664231+0000","last_peered":"2024-04-21T08:42:22.664231+0000","last_clean":"2024-04-21T08:42:22.664231+0000","last_became_active":"2024-04-21T08:42:17.547440+0000","last_became_peered":"2024-04-21T08:42:17.547440+0000","last_unstale":"2024-04-21T08:42:22.664231+0000","last_undegraded":"2024-04-21T08:42:22.664231+0000","last_fullsized":"2024-04-21T08:42:22.664231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:37:35.555737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768567+0000","last_change":"2024-04-21T08:42:21.670684+0000","last_active":"2024-04-21T08:42:22.768567+0000","last_peered":"2024-04-21T08:42:22.768567+0000","last_clean":"2024-04-21T08:42:22.768567+0000","last_became_active":"2024-04-21T08:42:21.670260+0000","last_became_peered":"2024-04-21T08:42:21.670260+0000","last_unstale":"2024-04-21T08:42:22.768567+0000","last_undegraded":"2024-04-21T08:42:22.768567+0000","last_fullsized":"2024-04-21T08:42:22.768567+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:45:06.192040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664440+0000","last_change":"2024-04-21T08:42:16.639632+0000","last_active":"2024-04-21T08:42:22.664440+0000","last_peered":"2024-04-21T08:42:22.664440+0000","last_clean":"2024-04-21T08:42:22.664440+0000","last_became_active":"2024-04-21T08:42:16.639301+0000","last_became_peered":"2024-04-21T08:42:16.639301+0000","last_unstale":"2024-04-21T08:42:22.664440+0000","last_undegraded":"2024-04-21T08:42:22.664440+0000","last_fullsized":"2024-04-21T08:42:22.664440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:58.838667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090950+0000","last_change":"2024-04-21T08:42:17.546109+0000","last_active":"2024-04-21T08:42:23.090950+0000","last_peered":"2024-04-21T08:42:23.090950+0000","last_clean":"2024-04-21T08:42:23.090950+0000","last_became_active":"2024-04-21T08:42:17.545673+0000","last_became_peered":"2024-04-21T08:42:17.545673+0000","last_unstale":"2024-04-21T08:42:23.090950+0000","last_undegraded":"2024-04-21T08:42:23.090950+0000","last_fullsized":"2024-04-21T08:42:23.090950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:18.947848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091744+0000","last_change":"2024-04-21T08:42:21.666114+0000","last_active":"2024-04-21T08:42:23.091744+0000","last_peered":"2024-04-21T08:42:23.091744+0000","last_clean":"2024-04-21T08:42:23.091744+0000","last_became_active":"2024-04-21T08:42:21.665020+0000","last_became_peered":"2024-04-21T08:42:21.665020+0000","last_unstale":"2024-04-21T08:42:23.091744+0000","last_undegraded":"2024-04-21T08:42:23.091744+0000","last_fullsized":"2024-04-21T08:42:23.091744+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:33.644178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664153+0000","last_change":"2024-04-21T08:42:16.639118+0000","last_active":"2024-04-21T08:42:22.664153+0000","last_peered":"2024-04-21T08:42:22.664153+0000","last_clean":"2024-04-21T08:42:22.664153+0000","last_became_active":"2024-04-21T08:42:16.638792+0000","last_became_peered":"2024-04-21T08:42:16.638792+0000","last_unstale":"2024-04-21T08:42:22.664153+0000","last_undegraded":"2024-04-21T08:42:22.664153+0000","last_fullsized":"2024-04-21T08:42:22.664153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:55.568712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680193+0000","last_change":"2024-04-21T08:42:17.539688+0000","last_active":"2024-04-21T08:42:22.680193+0000","last_peered":"2024-04-21T08:42:22.680193+0000","last_clean":"2024-04-21T08:42:22.680193+0000","last_became_active":"2024-04-21T08:42:17.539335+0000","last_became_peered":"2024-04-21T08:42:17.539335+0000","last_unstale":"2024-04-21T08:42:22.680193+0000","last_undegraded":"2024-04-21T08:42:22.680193+0000","last_fullsized":"2024-04-21T08:42:22.680193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:27:51.520232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091240+0000","last_change":"2024-04-21T08:42:21.660570+0000","last_active":"2024-04-21T08:42:23.091240+0000","last_peered":"2024-04-21T08:42:23.091240+0000","last_clean":"2024-04-21T08:42:23.091240+0000","last_became_active":"2024-04-21T08:42:21.659398+0000","last_became_peered":"2024-04-21T08:42:21.659398+0000","last_unstale":"2024-04-21T08:42:23.091240+0000","last_undegraded":"2024-04-21T08:42:23.091240+0000","last_fullsized":"2024-04-21T08:42:23.091240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:55.840334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771677+0000","last_change":"2024-04-21T08:42:16.648192+0000","last_active":"2024-04-21T08:42:22.771677+0000","last_peered":"2024-04-21T08:42:22.771677+0000","last_clean":"2024-04-21T08:42:22.771677+0000","last_became_active":"2024-04-21T08:42:16.647666+0000","last_became_peered":"2024-04-21T08:42:16.647666+0000","last_unstale":"2024-04-21T08:42:22.771677+0000","last_undegraded":"2024-04-21T08:42:22.771677+0000","last_fullsized":"2024-04-21T08:42:22.771677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:33.344680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664948+0000","last_change":"2024-04-21T08:42:17.545925+0000","last_active":"2024-04-21T08:42:22.664948+0000","last_peered":"2024-04-21T08:42:22.664948+0000","last_clean":"2024-04-21T08:42:22.664948+0000","last_became_active":"2024-04-21T08:42:17.545555+0000","last_became_peered":"2024-04-21T08:42:17.545555+0000","last_unstale":"2024-04-21T08:42:22.664948+0000","last_undegraded":"2024-04-21T08:42:22.664948+0000","last_fullsized":"2024-04-21T08:42:22.664948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:13:07.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680076+0000","last_change":"2024-04-21T08:42:21.662757+0000","last_active":"2024-04-21T08:42:22.680076+0000","last_peered":"2024-04-21T08:42:22.680076+0000","last_clean":"2024-04-21T08:42:22.680076+0000","last_became_active":"2024-04-21T08:42:21.662306+0000","last_became_peered":"2024-04-21T08:42:21.662306+0000","last_unstale":"2024-04-21T08:42:22.680076+0000","last_undegraded":"2024-04-21T08:42:22.680076+0000","last_fullsized":"2024-04-21T08:42:22.680076+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:26.856147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091702+0000","last_change":"2024-04-21T08:42:16.644191+0000","last_active":"2024-04-21T08:42:23.091702+0000","last_peered":"2024-04-21T08:42:23.091702+0000","last_clean":"2024-04-21T08:42:23.091702+0000","last_became_active":"2024-04-21T08:42:16.643697+0000","last_became_peered":"2024-04-21T08:42:16.643697+0000","last_unstale":"2024-04-21T08:42:23.091702+0000","last_undegraded":"2024-04-21T08:42:23.091702+0000","last_fullsized":"2024-04-21T08:42:23.091702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:48.680680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092262+0000","last_change":"2024-04-21T08:42:17.551026+0000","last_active":"2024-04-21T08:42:23.092262+0000","last_peered":"2024-04-21T08:42:23.092262+0000","last_clean":"2024-04-21T08:42:23.092262+0000","last_became_active":"2024-04-21T08:42:17.550461+0000","last_became_peered":"2024-04-21T08:42:17.550461+0000","last_unstale":"2024-04-21T08:42:23.092262+0000","last_undegraded":"2024-04-21T08:42:23.092262+0000","last_fullsized":"2024-04-21T08:42:23.092262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:09.736721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092346+0000","last_change":"2024-04-21T08:42:21.670561+0000","last_active":"2024-04-21T08:42:23.092346+0000","last_peered":"2024-04-21T08:42:23.092346+0000","last_clean":"2024-04-21T08:42:23.092346+0000","last_became_active":"2024-04-21T08:42:21.670170+0000","last_became_peered":"2024-04-21T08:42:21.670170+0000","last_unstale":"2024-04-21T08:42:23.092346+0000","last_undegraded":"2024-04-21T08:42:23.092346+0000","last_fullsized":"2024-04-21T08:42:23.092346+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:35:54.538013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664307+0000","last_change":"2024-04-21T08:42:16.637319+0000","last_active":"2024-04-21T08:42:22.664307+0000","last_peered":"2024-04-21T08:42:22.664307+0000","last_clean":"2024-04-21T08:42:22.664307+0000","last_became_active":"2024-04-21T08:42:16.636940+0000","last_became_peered":"2024-04-21T08:42:16.636940+0000","last_unstale":"2024-04-21T08:42:22.664307+0000","last_undegraded":"2024-04-21T08:42:22.664307+0000","last_fullsized":"2024-04-21T08:42:22.664307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:59:09.826786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664745+0000","last_change":"2024-04-21T08:42:17.546202+0000","last_active":"2024-04-21T08:42:22.664745+0000","last_peered":"2024-04-21T08:42:22.664745+0000","last_clean":"2024-04-21T08:42:22.664745+0000","last_became_active":"2024-04-21T08:42:17.545781+0000","last_became_peered":"2024-04-21T08:42:17.545781+0000","last_unstale":"2024-04-21T08:42:22.664745+0000","last_undegraded":"2024-04-21T08:42:22.664745+0000","last_fullsized":"2024-04-21T08:42:22.664745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:23.990951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680338+0000","last_change":"2024-04-21T08:42:21.662822+0000","last_active":"2024-04-21T08:42:22.680338+0000","last_peered":"2024-04-21T08:42:22.680338+0000","last_clean":"2024-04-21T08:42:22.680338+0000","last_became_active":"2024-04-21T08:42:21.662398+0000","last_became_peered":"2024-04-21T08:42:21.662398+0000","last_unstale":"2024-04-21T08:42:22.680338+0000","last_undegraded":"2024-04-21T08:42:22.680338+0000","last_fullsized":"2024-04-21T08:42:22.680338+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:38:40.499294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'10","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:33.258766+0000","last_change":"2024-04-21T08:42:16.636330+0000","last_active":"2024-04-21T08:42:33.258766+0000","last_peered":"2024-04-21T08:42:33.258766+0000","last_clean":"2024-04-21T08:42:33.258766+0000","last_became_active":"2024-04-21T08:42:16.635762+0000","last_became_peered":"2024-04-21T08:42:16.635762+0000","last_unstale":"2024-04-21T08:42:33.258766+0000","last_undegraded":"2024-04-21T08:42:33.258766+0000","last_fullsized":"2024-04-21T08:42:33.258766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:31.137923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951660+0000","last_change":"2024-04-21T08:42:17.543033+0000","last_active":"2024-04-21T08:42:22.951660+0000","last_peered":"2024-04-21T08:42:22.951660+0000","last_clean":"2024-04-21T08:42:22.951660+0000","last_became_active":"2024-04-21T08:42:17.542692+0000","last_became_peered":"2024-04-21T08:42:17.542692+0000","last_unstale":"2024-04-21T08:42:22.951660+0000","last_undegraded":"2024-04-21T08:42:22.951660+0000","last_fullsized":"2024-04-21T08:42:22.951660+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:33:32.111999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664742+0000","last_change":"2024-04-21T08:42:21.668831+0000","last_active":"2024-04-21T08:42:22.664742+0000","last_peered":"2024-04-21T08:42:22.664742+0000","last_clean":"2024-04-21T08:42:22.664742+0000","last_became_active":"2024-04-21T08:42:21.668485+0000","last_became_peered":"2024-04-21T08:42:21.668485+0000","last_unstale":"2024-04-21T08:42:22.664742+0000","last_undegraded":"2024-04-21T08:42:22.664742+0000","last_fullsized":"2024-04-21T08:42:22.664742+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:49:29.049707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090476+0000","last_change":"2024-04-21T08:42:16.640288+0000","last_active":"2024-04-21T08:42:23.090476+0000","last_peered":"2024-04-21T08:42:23.090476+0000","last_clean":"2024-04-21T08:42:23.090476+0000","last_became_active":"2024-04-21T08:42:16.639909+0000","last_became_peered":"2024-04-21T08:42:16.639909+0000","last_unstale":"2024-04-21T08:42:23.090476+0000","last_undegraded":"2024-04-21T08:42:23.090476+0000","last_fullsized":"2024-04-21T08:42:23.090476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:33.839442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664531+0000","last_change":"2024-04-21T08:42:17.541154+0000","last_active":"2024-04-21T08:42:22.664531+0000","last_peered":"2024-04-21T08:42:22.664531+0000","last_clean":"2024-04-21T08:42:22.664531+0000","last_became_active":"2024-04-21T08:42:17.540747+0000","last_became_peered":"2024-04-21T08:42:17.540747+0000","last_unstale":"2024-04-21T08:42:22.664531+0000","last_undegraded":"2024-04-21T08:42:22.664531+0000","last_fullsized":"2024-04-21T08:42:22.664531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:14:50.641144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768232+0000","last_change":"2024-04-21T08:42:21.670767+0000","last_active":"2024-04-21T08:42:22.768232+0000","last_peered":"2024-04-21T08:42:22.768232+0000","last_clean":"2024-04-21T08:42:22.768232+0000","last_became_active":"2024-04-21T08:42:21.670338+0000","last_became_peered":"2024-04-21T08:42:21.670338+0000","last_unstale":"2024-04-21T08:42:22.768232+0000","last_undegraded":"2024-04-21T08:42:22.768232+0000","last_fullsized":"2024-04-21T08:42:22.768232+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:38:55.275603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664133+0000","last_change":"2024-04-21T08:42:16.640144+0000","last_active":"2024-04-21T08:42:22.664133+0000","last_peered":"2024-04-21T08:42:22.664133+0000","last_clean":"2024-04-21T08:42:22.664133+0000","last_became_active":"2024-04-21T08:42:16.639707+0000","last_became_peered":"2024-04-21T08:42:16.639707+0000","last_unstale":"2024-04-21T08:42:22.664133+0000","last_undegraded":"2024-04-21T08:42:22.664133+0000","last_fullsized":"2024-04-21T08:42:22.664133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:11:32.785519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091827+0000","last_change":"2024-04-21T08:42:17.555019+0000","last_active":"2024-04-21T08:42:23.091827+0000","last_peered":"2024-04-21T08:42:23.091827+0000","last_clean":"2024-04-21T08:42:23.091827+0000","last_became_active":"2024-04-21T08:42:17.554645+0000","last_became_peered":"2024-04-21T08:42:17.554645+0000","last_unstale":"2024-04-21T08:42:23.091827+0000","last_undegraded":"2024-04-21T08:42:23.091827+0000","last_fullsized":"2024-04-21T08:42:23.091827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:42:30.353945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680387+0000","last_change":"2024-04-21T08:42:21.663567+0000","last_active":"2024-04-21T08:42:22.680387+0000","last_peered":"2024-04-21T08:42:22.680387+0000","last_clean":"2024-04-21T08:42:22.680387+0000","last_became_active":"2024-04-21T08:42:21.663203+0000","last_became_peered":"2024-04-21T08:42:21.663203+0000","last_unstale":"2024-04-21T08:42:22.680387+0000","last_undegraded":"2024-04-21T08:42:22.680387+0000","last_fullsized":"2024-04-21T08:42:22.680387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:50.760336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091780+0000","last_change":"2024-04-21T08:42:16.640407+0000","last_active":"2024-04-21T08:42:23.091780+0000","last_peered":"2024-04-21T08:42:23.091780+0000","last_clean":"2024-04-21T08:42:23.091780+0000","last_became_active":"2024-04-21T08:42:16.640015+0000","last_became_peered":"2024-04-21T08:42:16.640015+0000","last_unstale":"2024-04-21T08:42:23.091780+0000","last_undegraded":"2024-04-21T08:42:23.091780+0000","last_fullsized":"2024-04-21T08:42:23.091780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:23:10.647184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091167+0000","last_change":"2024-04-21T08:42:17.545438+0000","last_active":"2024-04-21T08:42:23.091167+0000","last_peered":"2024-04-21T08:42:23.091167+0000","last_clean":"2024-04-21T08:42:23.091167+0000","last_became_active":"2024-04-21T08:42:17.544258+0000","last_became_peered":"2024-04-21T08:42:17.544258+0000","last_unstale":"2024-04-21T08:42:23.091167+0000","last_undegraded":"2024-04-21T08:42:23.091167+0000","last_fullsized":"2024-04-21T08:42:23.091167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:43.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.665171+0000","last_change":"2024-04-21T08:42:21.671000+0000","last_active":"2024-04-21T08:42:22.665171+0000","last_peered":"2024-04-21T08:42:22.665171+0000","last_clean":"2024-04-21T08:42:22.665171+0000","last_became_active":"2024-04-21T08:42:21.670380+0000","last_became_peered":"2024-04-21T08:42:21.670380+0000","last_unstale":"2024-04-21T08:42:22.665171+0000","last_undegraded":"2024-04-21T08:42:22.665171+0000","last_fullsized":"2024-04-21T08:42:22.665171+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:57.075686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951408+0000","last_change":"2024-04-21T08:42:16.649599+0000","last_active":"2024-04-21T08:42:22.951408+0000","last_peered":"2024-04-21T08:42:22.951408+0000","last_clean":"2024-04-21T08:42:22.951408+0000","last_became_active":"2024-04-21T08:42:16.649232+0000","last_became_peered":"2024-04-21T08:42:16.649232+0000","last_unstale":"2024-04-21T08:42:22.951408+0000","last_undegraded":"2024-04-21T08:42:22.951408+0000","last_fullsized":"2024-04-21T08:42:22.951408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:21.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664102+0000","last_change":"2024-04-21T08:42:17.545910+0000","last_active":"2024-04-21T08:42:22.664102+0000","last_peered":"2024-04-21T08:42:22.664102+0000","last_clean":"2024-04-21T08:42:22.664102+0000","last_became_active":"2024-04-21T08:42:17.545582+0000","last_became_peered":"2024-04-21T08:42:17.545582+0000","last_unstale":"2024-04-21T08:42:22.664102+0000","last_undegraded":"2024-04-21T08:42:22.664102+0000","last_fullsized":"2024-04-21T08:42:22.664102+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:25.466146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.663988+0000","last_change":"2024-04-21T08:42:21.664811+0000","last_active":"2024-04-21T08:42:22.663988+0000","last_peered":"2024-04-21T08:42:22.663988+0000","last_clean":"2024-04-21T08:42:22.663988+0000","last_became_active":"2024-04-21T08:42:21.664332+0000","last_became_peered":"2024-04-21T08:42:21.664332+0000","last_unstale":"2024-04-21T08:42:22.663988+0000","last_undegraded":"2024-04-21T08:42:22.663988+0000","last_fullsized":"2024-04-21T08:42:22.663988+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:40.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951943+0000","last_change":"2024-04-21T08:42:16.641233+0000","last_active":"2024-04-21T08:42:22.951943+0000","last_peered":"2024-04-21T08:42:22.951943+0000","last_clean":"2024-04-21T08:42:22.951943+0000","last_became_active":"2024-04-21T08:42:16.640873+0000","last_became_peered":"2024-04-21T08:42:16.640873+0000","last_unstale":"2024-04-21T08:42:22.951943+0000","last_undegraded":"2024-04-21T08:42:22.951943+0000","last_fullsized":"2024-04-21T08:42:22.951943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:00.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768416+0000","last_change":"2024-04-21T08:42:17.550438+0000","last_active":"2024-04-21T08:42:22.768416+0000","last_peered":"2024-04-21T08:42:22.768416+0000","last_clean":"2024-04-21T08:42:22.768416+0000","last_became_active":"2024-04-21T08:42:17.550015+0000","last_became_peered":"2024-04-21T08:42:17.550015+0000","last_unstale":"2024-04-21T08:42:22.768416+0000","last_undegraded":"2024-04-21T08:42:22.768416+0000","last_fullsized":"2024-04-21T08:42:22.768416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:00:29.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951618+0000","last_change":"2024-04-21T08:42:21.970498+0000","last_active":"2024-04-21T08:42:22.951618+0000","last_peered":"2024-04-21T08:42:22.951618+0000","last_clean":"2024-04-21T08:42:22.951618+0000","last_became_active":"2024-04-21T08:42:21.969819+0000","last_became_peered":"2024-04-21T08:42:21.969819+0000","last_unstale":"2024-04-21T08:42:22.951618+0000","last_undegraded":"2024-04-21T08:42:22.951618+0000","last_fullsized":"2024-04-21T08:42:22.951618+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:13:16.552293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680361+0000","last_change":"2024-04-21T08:42:16.637146+0000","last_active":"2024-04-21T08:42:22.680361+0000","last_peered":"2024-04-21T08:42:22.680361+0000","last_clean":"2024-04-21T08:42:22.680361+0000","last_became_active":"2024-04-21T08:42:16.636781+0000","last_became_peered":"2024-04-21T08:42:16.636781+0000","last_unstale":"2024-04-21T08:42:22.680361+0000","last_undegraded":"2024-04-21T08:42:22.680361+0000","last_fullsized":"2024-04-21T08:42:22.680361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:39:27.641236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771645+0000","last_change":"2024-04-21T08:42:17.542325+0000","last_active":"2024-04-21T08:42:22.771645+0000","last_peered":"2024-04-21T08:42:22.771645+0000","last_clean":"2024-04-21T08:42:22.771645+0000","last_became_active":"2024-04-21T08:42:17.541103+0000","last_became_peered":"2024-04-21T08:42:17.541103+0000","last_unstale":"2024-04-21T08:42:22.771645+0000","last_undegraded":"2024-04-21T08:42:22.771645+0000","last_fullsized":"2024-04-21T08:42:22.771645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:46.903250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664900+0000","last_change":"2024-04-21T08:42:21.671146+0000","last_active":"2024-04-21T08:42:22.664900+0000","last_peered":"2024-04-21T08:42:22.664900+0000","last_clean":"2024-04-21T08:42:22.664900+0000","last_became_active":"2024-04-21T08:42:21.670561+0000","last_became_peered":"2024-04-21T08:42:21.670561+0000","last_unstale":"2024-04-21T08:42:22.664900+0000","last_undegraded":"2024-04-21T08:42:22.664900+0000","last_fullsized":"2024-04-21T08:42:22.664900+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:38:28.028976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091471+0000","last_change":"2024-04-21T08:42:16.645132+0000","last_active":"2024-04-21T08:42:23.091471+0000","last_peered":"2024-04-21T08:42:23.091471+0000","last_clean":"2024-04-21T08:42:23.091471+0000","last_became_active":"2024-04-21T08:42:16.644786+0000","last_became_peered":"2024-04-21T08:42:16.644786+0000","last_unstale":"2024-04-21T08:42:23.091471+0000","last_undegraded":"2024-04-21T08:42:23.091471+0000","last_fullsized":"2024-04-21T08:42:23.091471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:08.298498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771569+0000","last_change":"2024-04-21T08:42:17.550424+0000","last_active":"2024-04-21T08:42:22.771569+0000","last_peered":"2024-04-21T08:42:22.771569+0000","last_clean":"2024-04-21T08:42:22.771569+0000","last_became_active":"2024-04-21T08:42:17.550077+0000","last_became_peered":"2024-04-21T08:42:17.550077+0000","last_unstale":"2024-04-21T08:42:22.771569+0000","last_undegraded":"2024-04-21T08:42:22.771569+0000","last_fullsized":"2024-04-21T08:42:22.771569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:17:07.016204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.665144+0000","last_change":"2024-04-21T08:42:21.970652+0000","last_active":"2024-04-21T08:42:22.665144+0000","last_peered":"2024-04-21T08:42:22.665144+0000","last_clean":"2024-04-21T08:42:22.665144+0000","last_became_active":"2024-04-21T08:42:21.970251+0000","last_became_peered":"2024-04-21T08:42:21.970251+0000","last_unstale":"2024-04-21T08:42:22.665144+0000","last_undegraded":"2024-04-21T08:42:22.665144+0000","last_fullsized":"2024-04-21T08:42:22.665144+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:45.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664385+0000","last_change":"2024-04-21T08:42:17.542820+0000","last_active":"2024-04-21T08:42:22.664385+0000","last_peered":"2024-04-21T08:42:22.664385+0000","last_clean":"2024-04-21T08:42:22.664385+0000","last_became_active":"2024-04-21T08:42:17.541320+0000","last_became_peered":"2024-04-21T08:42:17.541320+0000","last_unstale":"2024-04-21T08:42:22.664385+0000","last_undegraded":"2024-04-21T08:42:22.664385+0000","last_fullsized":"2024-04-21T08:42:22.664385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:23:06.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664297+0000","last_change":"2024-04-21T08:42:16.636950+0000","last_active":"2024-04-21T08:42:22.664297+0000","last_peered":"2024-04-21T08:42:22.664297+0000","last_clean":"2024-04-21T08:42:22.664297+0000","last_became_active":"2024-04-21T08:42:16.636590+0000","last_became_peered":"2024-04-21T08:42:16.636590+0000","last_unstale":"2024-04-21T08:42:22.664297+0000","last_undegraded":"2024-04-21T08:42:22.664297+0000","last_fullsized":"2024-04-21T08:42:22.664297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:55.693681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091648+0000","last_change":"2024-04-21T08:42:21.660404+0000","last_active":"2024-04-21T08:42:23.091648+0000","last_peered":"2024-04-21T08:42:23.091648+0000","last_clean":"2024-04-21T08:42:23.091648+0000","last_became_active":"2024-04-21T08:42:21.659999+0000","last_became_peered":"2024-04-21T08:42:21.659999+0000","last_unstale":"2024-04-21T08:42:23.091648+0000","last_undegraded":"2024-04-21T08:42:23.091648+0000","last_fullsized":"2024-04-21T08:42:23.091648+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:53:32.328873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091550+0000","last_change":"2024-04-21T08:42:17.548059+0000","last_active":"2024-04-21T08:42:23.091550+0000","last_peered":"2024-04-21T08:42:23.091550+0000","last_clean":"2024-04-21T08:42:23.091550+0000","last_became_active":"2024-04-21T08:42:17.547532+0000","last_became_peered":"2024-04-21T08:42:17.547532+0000","last_unstale":"2024-04-21T08:42:23.091550+0000","last_undegraded":"2024-04-21T08:42:23.091550+0000","last_fullsized":"2024-04-21T08:42:23.091550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:08.459103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091980+0000","last_change":"2024-04-21T08:42:16.638993+0000","last_active":"2024-04-21T08:42:23.091980+0000","last_peered":"2024-04-21T08:42:23.091980+0000","last_clean":"2024-04-21T08:42:23.091980+0000","last_became_active":"2024-04-21T08:42:16.638617+0000","last_became_peered":"2024-04-21T08:42:16.638617+0000","last_unstale":"2024-04-21T08:42:23.091980+0000","last_undegraded":"2024-04-21T08:42:23.091980+0000","last_fullsized":"2024-04-21T08:42:23.091980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:14:28.271433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":355,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090851+0000","last_change":"2024-04-21T08:41:00.044984+0000","last_active":"2024-04-21T08:42:23.090851+0000","last_peered":"2024-04-21T08:42:23.090851+0000","last_clean":"2024-04-21T08:42:23.090851+0000","last_became_active":"2024-04-21T08:41:00.044485+0000","last_became_peered":"2024-04-21T08:41:00.044485+0000","last_unstale":"2024-04-21T08:42:23.090851+0000","last_undegraded":"2024-04-21T08:42:23.090851+0000","last_fullsized":"2024-04-21T08:42:23.090851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.370031+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:01.031209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664715+0000","last_change":"2024-04-21T08:42:21.971045+0000","last_active":"2024-04-21T08:42:22.664715+0000","last_peered":"2024-04-21T08:42:22.664715+0000","last_clean":"2024-04-21T08:42:22.664715+0000","last_became_active":"2024-04-21T08:42:21.970577+0000","last_became_peered":"2024-04-21T08:42:21.970577+0000","last_unstale":"2024-04-21T08:42:22.664715+0000","last_undegraded":"2024-04-21T08:42:22.664715+0000","last_fullsized":"2024-04-21T08:42:22.664715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:55.681662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664297+0000","last_change":"2024-04-21T08:42:17.535570+0000","last_active":"2024-04-21T08:42:22.664297+0000","last_peered":"2024-04-21T08:42:22.664297+0000","last_clean":"2024-04-21T08:42:22.664297+0000","last_became_active":"2024-04-21T08:42:17.535217+0000","last_became_peered":"2024-04-21T08:42:17.535217+0000","last_unstale":"2024-04-21T08:42:22.664297+0000","last_undegraded":"2024-04-21T08:42:22.664297+0000","last_fullsized":"2024-04-21T08:42:22.664297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:49.510750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091021+0000","last_change":"2024-04-21T08:42:16.643794+0000","last_active":"2024-04-21T08:42:23.091021+0000","last_peered":"2024-04-21T08:42:23.091021+0000","last_clean":"2024-04-21T08:42:23.091021+0000","last_became_active":"2024-04-21T08:42:16.643357+0000","last_became_peered":"2024-04-21T08:42:16.643357+0000","last_unstale":"2024-04-21T08:42:23.091021+0000","last_undegraded":"2024-04-21T08:42:23.091021+0000","last_fullsized":"2024-04-21T08:42:23.091021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:12.778950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768274+0000","last_change":"2024-04-21T08:42:21.661393+0000","last_active":"2024-04-21T08:42:22.768274+0000","last_peered":"2024-04-21T08:42:22.768274+0000","last_clean":"2024-04-21T08:42:22.768274+0000","last_became_active":"2024-04-21T08:42:21.660201+0000","last_became_peered":"2024-04-21T08:42:21.660201+0000","last_unstale":"2024-04-21T08:42:22.768274+0000","last_undegraded":"2024-04-21T08:42:22.768274+0000","last_fullsized":"2024-04-21T08:42:22.768274+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:13.393389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951753+0000","last_change":"2024-04-21T08:42:16.650507+0000","last_active":"2024-04-21T08:42:22.951753+0000","last_peered":"2024-04-21T08:42:22.951753+0000","last_clean":"2024-04-21T08:42:22.951753+0000","last_became_active":"2024-04-21T08:42:16.650217+0000","last_became_peered":"2024-04-21T08:42:16.650217+0000","last_unstale":"2024-04-21T08:42:22.951753+0000","last_undegraded":"2024-04-21T08:42:22.951753+0000","last_fullsized":"2024-04-21T08:42:22.951753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:13:26.785092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664946+0000","last_change":"2024-04-21T08:42:17.530420+0000","last_active":"2024-04-21T08:42:22.664946+0000","last_peered":"2024-04-21T08:42:22.664946+0000","last_clean":"2024-04-21T08:42:22.664946+0000","last_became_active":"2024-04-21T08:42:17.529920+0000","last_became_peered":"2024-04-21T08:42:17.529920+0000","last_unstale":"2024-04-21T08:42:22.664946+0000","last_undegraded":"2024-04-21T08:42:22.664946+0000","last_fullsized":"2024-04-21T08:42:22.664946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:24.250197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090926+0000","last_change":"2024-04-21T08:42:21.970586+0000","last_active":"2024-04-21T08:42:23.090926+0000","last_peered":"2024-04-21T08:42:23.090926+0000","last_clean":"2024-04-21T08:42:23.090926+0000","last_became_active":"2024-04-21T08:42:21.970132+0000","last_became_peered":"2024-04-21T08:42:21.970132+0000","last_unstale":"2024-04-21T08:42:23.090926+0000","last_undegraded":"2024-04-21T08:42:23.090926+0000","last_fullsized":"2024-04-21T08:42:23.090926+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:11.074370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680514+0000","last_change":"2024-04-21T08:42:16.647916+0000","last_active":"2024-04-21T08:42:22.680514+0000","last_peered":"2024-04-21T08:42:22.680514+0000","last_clean":"2024-04-21T08:42:22.680514+0000","last_became_active":"2024-04-21T08:42:16.647476+0000","last_became_peered":"2024-04-21T08:42:16.647476+0000","last_unstale":"2024-04-21T08:42:22.680514+0000","last_undegraded":"2024-04-21T08:42:22.680514+0000","last_fullsized":"2024-04-21T08:42:22.680514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:27.242153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092078+0000","last_change":"2024-04-21T08:42:17.546144+0000","last_active":"2024-04-21T08:42:23.092078+0000","last_peered":"2024-04-21T08:42:23.092078+0000","last_clean":"2024-04-21T08:42:23.092078+0000","last_became_active":"2024-04-21T08:42:17.545730+0000","last_became_peered":"2024-04-21T08:42:17.545730+0000","last_unstale":"2024-04-21T08:42:23.092078+0000","last_undegraded":"2024-04-21T08:42:23.092078+0000","last_fullsized":"2024-04-21T08:42:23.092078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:42.802309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090217+0000","last_change":"2024-04-21T08:42:21.655802+0000","last_active":"2024-04-21T08:42:23.090217+0000","last_peered":"2024-04-21T08:42:23.090217+0000","last_clean":"2024-04-21T08:42:23.090217+0000","last_became_active":"2024-04-21T08:42:21.655379+0000","last_became_peered":"2024-04-21T08:42:21.655379+0000","last_unstale":"2024-04-21T08:42:23.090217+0000","last_undegraded":"2024-04-21T08:42:23.090217+0000","last_fullsized":"2024-04-21T08:42:23.090217+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:49.486057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091831+0000","last_change":"2024-04-21T08:42:16.642517+0000","last_active":"2024-04-21T08:42:23.091831+0000","last_peered":"2024-04-21T08:42:23.091831+0000","last_clean":"2024-04-21T08:42:23.091831+0000","last_became_active":"2024-04-21T08:42:16.641539+0000","last_became_peered":"2024-04-21T08:42:16.641539+0000","last_unstale":"2024-04-21T08:42:23.091831+0000","last_undegraded":"2024-04-21T08:42:23.091831+0000","last_fullsized":"2024-04-21T08:42:23.091831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:53.912199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091797+0000","last_change":"2024-04-21T08:42:17.550172+0000","last_active":"2024-04-21T08:42:23.091797+0000","last_peered":"2024-04-21T08:42:23.091797+0000","last_clean":"2024-04-21T08:42:23.091797+0000","last_became_active":"2024-04-21T08:42:17.549824+0000","last_became_peered":"2024-04-21T08:42:17.549824+0000","last_unstale":"2024-04-21T08:42:23.091797+0000","last_undegraded":"2024-04-21T08:42:23.091797+0000","last_fullsized":"2024-04-21T08:42:23.091797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:59.553645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092068+0000","last_change":"2024-04-21T08:42:21.970654+0000","last_active":"2024-04-21T08:42:23.092068+0000","last_peered":"2024-04-21T08:42:23.092068+0000","last_clean":"2024-04-21T08:42:23.092068+0000","last_became_active":"2024-04-21T08:42:21.970211+0000","last_became_peered":"2024-04-21T08:42:21.970211+0000","last_unstale":"2024-04-21T08:42:23.092068+0000","last_undegraded":"2024-04-21T08:42:23.092068+0000","last_fullsized":"2024-04-21T08:42:23.092068+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:24:51.406125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664251+0000","last_change":"2024-04-21T08:42:16.647985+0000","last_active":"2024-04-21T08:42:22.664251+0000","last_peered":"2024-04-21T08:42:22.664251+0000","last_clean":"2024-04-21T08:42:22.664251+0000","last_became_active":"2024-04-21T08:42:16.647577+0000","last_became_peered":"2024-04-21T08:42:16.647577+0000","last_unstale":"2024-04-21T08:42:22.664251+0000","last_undegraded":"2024-04-21T08:42:22.664251+0000","last_fullsized":"2024-04-21T08:42:22.664251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:00.731708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951434+0000","last_change":"2024-04-21T08:42:17.541757+0000","last_active":"2024-04-21T08:42:22.951434+0000","last_peered":"2024-04-21T08:42:22.951434+0000","last_clean":"2024-04-21T08:42:22.951434+0000","last_became_active":"2024-04-21T08:42:17.539280+0000","last_became_peered":"2024-04-21T08:42:17.539280+0000","last_unstale":"2024-04-21T08:42:22.951434+0000","last_undegraded":"2024-04-21T08:42:22.951434+0000","last_fullsized":"2024-04-21T08:42:22.951434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:45:54.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680519+0000","last_change":"2024-04-21T08:42:21.664566+0000","last_active":"2024-04-21T08:42:22.680519+0000","last_peered":"2024-04-21T08:42:22.680519+0000","last_clean":"2024-04-21T08:42:22.680519+0000","last_became_active":"2024-04-21T08:42:21.664136+0000","last_became_peered":"2024-04-21T08:42:21.664136+0000","last_unstale":"2024-04-21T08:42:22.680519+0000","last_undegraded":"2024-04-21T08:42:22.680519+0000","last_fullsized":"2024-04-21T08:42:22.680519+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:19.856577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091318+0000","last_change":"2024-04-21T08:42:16.650664+0000","last_active":"2024-04-21T08:42:23.091318+0000","last_peered":"2024-04-21T08:42:23.091318+0000","last_clean":"2024-04-21T08:42:23.091318+0000","last_became_active":"2024-04-21T08:42:16.650339+0000","last_became_peered":"2024-04-21T08:42:16.650339+0000","last_unstale":"2024-04-21T08:42:23.091318+0000","last_undegraded":"2024-04-21T08:42:23.091318+0000","last_fullsized":"2024-04-21T08:42:23.091318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:59.565455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768020+0000","last_change":"2024-04-21T08:42:17.538142+0000","last_active":"2024-04-21T08:42:22.768020+0000","last_peered":"2024-04-21T08:42:22.768020+0000","last_clean":"2024-04-21T08:42:22.768020+0000","last_became_active":"2024-04-21T08:42:17.537793+0000","last_became_peered":"2024-04-21T08:42:17.537793+0000","last_unstale":"2024-04-21T08:42:22.768020+0000","last_undegraded":"2024-04-21T08:42:22.768020+0000","last_fullsized":"2024-04-21T08:42:22.768020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:11.059294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092051+0000","last_change":"2024-04-21T08:42:21.670642+0000","last_active":"2024-04-21T08:42:23.092051+0000","last_peered":"2024-04-21T08:42:23.092051+0000","last_clean":"2024-04-21T08:42:23.092051+0000","last_became_active":"2024-04-21T08:42:21.670128+0000","last_became_peered":"2024-04-21T08:42:21.670128+0000","last_unstale":"2024-04-21T08:42:23.092051+0000","last_undegraded":"2024-04-21T08:42:23.092051+0000","last_fullsized":"2024-04-21T08:42:23.092051+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:51:28.604381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090228+0000","last_change":"2024-04-21T08:42:16.642471+0000","last_active":"2024-04-21T08:42:23.090228+0000","last_peered":"2024-04-21T08:42:23.090228+0000","last_clean":"2024-04-21T08:42:23.090228+0000","last_became_active":"2024-04-21T08:42:16.642142+0000","last_became_peered":"2024-04-21T08:42:16.642142+0000","last_unstale":"2024-04-21T08:42:23.090228+0000","last_undegraded":"2024-04-21T08:42:23.090228+0000","last_fullsized":"2024-04-21T08:42:23.090228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:17.847374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091690+0000","last_change":"2024-04-21T08:42:17.547326+0000","last_active":"2024-04-21T08:42:23.091690+0000","last_peered":"2024-04-21T08:42:23.091690+0000","last_clean":"2024-04-21T08:42:23.091690+0000","last_became_active":"2024-04-21T08:42:17.546920+0000","last_became_peered":"2024-04-21T08:42:17.546920+0000","last_unstale":"2024-04-21T08:42:23.091690+0000","last_undegraded":"2024-04-21T08:42:23.091690+0000","last_fullsized":"2024-04-21T08:42:23.091690+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:43.350877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092100+0000","last_change":"2024-04-21T08:42:21.971147+0000","last_active":"2024-04-21T08:42:23.092100+0000","last_peered":"2024-04-21T08:42:23.092100+0000","last_clean":"2024-04-21T08:42:23.092100+0000","last_became_active":"2024-04-21T08:42:21.970001+0000","last_became_peered":"2024-04-21T08:42:21.970001+0000","last_unstale":"2024-04-21T08:42:23.092100+0000","last_undegraded":"2024-04-21T08:42:23.092100+0000","last_fullsized":"2024-04-21T08:42:23.092100+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:01:44.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951533+0000","last_change":"2024-04-21T08:42:16.643371+0000","last_active":"2024-04-21T08:42:22.951533+0000","last_peered":"2024-04-21T08:42:22.951533+0000","last_clean":"2024-04-21T08:42:22.951533+0000","last_became_active":"2024-04-21T08:42:16.642906+0000","last_became_peered":"2024-04-21T08:42:16.642906+0000","last_unstale":"2024-04-21T08:42:22.951533+0000","last_undegraded":"2024-04-21T08:42:22.951533+0000","last_fullsized":"2024-04-21T08:42:22.951533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:58.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091533+0000","last_change":"2024-04-21T08:42:17.545185+0000","last_active":"2024-04-21T08:42:23.091533+0000","last_peered":"2024-04-21T08:42:23.091533+0000","last_clean":"2024-04-21T08:42:23.091533+0000","last_became_active":"2024-04-21T08:42:17.544757+0000","last_became_peered":"2024-04-21T08:42:17.544757+0000","last_unstale":"2024-04-21T08:42:23.091533+0000","last_undegraded":"2024-04-21T08:42:23.091533+0000","last_fullsized":"2024-04-21T08:42:23.091533+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:36:18.968041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664466+0000","last_change":"2024-04-21T08:42:21.662541+0000","last_active":"2024-04-21T08:42:22.664466+0000","last_peered":"2024-04-21T08:42:22.664466+0000","last_clean":"2024-04-21T08:42:22.664466+0000","last_became_active":"2024-04-21T08:42:21.662144+0000","last_became_peered":"2024-04-21T08:42:21.662144+0000","last_unstale":"2024-04-21T08:42:22.664466+0000","last_undegraded":"2024-04-21T08:42:22.664466+0000","last_fullsized":"2024-04-21T08:42:22.664466+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:04.347722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090684+0000","last_change":"2024-04-21T08:42:16.638601+0000","last_active":"2024-04-21T08:42:23.090684+0000","last_peered":"2024-04-21T08:42:23.090684+0000","last_clean":"2024-04-21T08:42:23.090684+0000","last_became_active":"2024-04-21T08:42:16.638211+0000","last_became_peered":"2024-04-21T08:42:16.638211+0000","last_unstale":"2024-04-21T08:42:23.090684+0000","last_undegraded":"2024-04-21T08:42:23.090684+0000","last_fullsized":"2024-04-21T08:42:23.090684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:02.633086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091786+0000","last_change":"2024-04-21T08:42:17.545495+0000","last_active":"2024-04-21T08:42:23.091786+0000","last_peered":"2024-04-21T08:42:23.091786+0000","last_clean":"2024-04-21T08:42:23.091786+0000","last_became_active":"2024-04-21T08:42:17.545159+0000","last_became_peered":"2024-04-21T08:42:17.545159+0000","last_unstale":"2024-04-21T08:42:23.091786+0000","last_undegraded":"2024-04-21T08:42:23.091786+0000","last_fullsized":"2024-04-21T08:42:23.091786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:27.150277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091243+0000","last_change":"2024-04-21T08:42:21.656890+0000","last_active":"2024-04-21T08:42:23.091243+0000","last_peered":"2024-04-21T08:42:23.091243+0000","last_clean":"2024-04-21T08:42:23.091243+0000","last_became_active":"2024-04-21T08:42:21.656475+0000","last_became_peered":"2024-04-21T08:42:21.656475+0000","last_unstale":"2024-04-21T08:42:23.091243+0000","last_undegraded":"2024-04-21T08:42:23.091243+0000","last_fullsized":"2024-04-21T08:42:23.091243+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:46.138591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664449+0000","last_change":"2024-04-21T08:42:16.638244+0000","last_active":"2024-04-21T08:42:22.664449+0000","last_peered":"2024-04-21T08:42:22.664449+0000","last_clean":"2024-04-21T08:42:22.664449+0000","last_became_active":"2024-04-21T08:42:16.637883+0000","last_became_peered":"2024-04-21T08:42:16.637883+0000","last_unstale":"2024-04-21T08:42:22.664449+0000","last_undegraded":"2024-04-21T08:42:22.664449+0000","last_fullsized":"2024-04-21T08:42:22.664449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:20.888301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091372+0000","last_change":"2024-04-21T08:42:17.548032+0000","last_active":"2024-04-21T08:42:23.091372+0000","last_peered":"2024-04-21T08:42:23.091372+0000","last_clean":"2024-04-21T08:42:23.091372+0000","last_became_active":"2024-04-21T08:42:17.547660+0000","last_became_peered":"2024-04-21T08:42:17.547660+0000","last_unstale":"2024-04-21T08:42:23.091372+0000","last_undegraded":"2024-04-21T08:42:23.091372+0000","last_fullsized":"2024-04-21T08:42:23.091372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:14.710759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.663796+0000","last_change":"2024-04-21T08:42:21.659286+0000","last_active":"2024-04-21T08:42:22.663796+0000","last_peered":"2024-04-21T08:42:22.663796+0000","last_clean":"2024-04-21T08:42:22.663796+0000","last_became_active":"2024-04-21T08:42:21.658844+0000","last_became_peered":"2024-04-21T08:42:21.658844+0000","last_unstale":"2024-04-21T08:42:22.663796+0000","last_undegraded":"2024-04-21T08:42:22.663796+0000","last_fullsized":"2024-04-21T08:42:22.663796+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:28.742646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.665047+0000","last_change":"2024-04-21T08:42:16.643354+0000","last_active":"2024-04-21T08:42:22.665047+0000","last_peered":"2024-04-21T08:42:22.665047+0000","last_clean":"2024-04-21T08:42:22.665047+0000","last_became_active":"2024-04-21T08:42:16.642967+0000","last_became_peered":"2024-04-21T08:42:16.642967+0000","last_unstale":"2024-04-21T08:42:22.665047+0000","last_undegraded":"2024-04-21T08:42:22.665047+0000","last_fullsized":"2024-04-21T08:42:22.665047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:06:03.495181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091938+0000","last_change":"2024-04-21T08:42:17.539551+0000","last_active":"2024-04-21T08:42:23.091938+0000","last_peered":"2024-04-21T08:42:23.091938+0000","last_clean":"2024-04-21T08:42:23.091938+0000","last_became_active":"2024-04-21T08:42:17.539202+0000","last_became_peered":"2024-04-21T08:42:17.539202+0000","last_unstale":"2024-04-21T08:42:23.091938+0000","last_undegraded":"2024-04-21T08:42:23.091938+0000","last_fullsized":"2024-04-21T08:42:23.091938+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:37.109421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768071+0000","last_change":"2024-04-21T08:42:17.544582+0000","last_active":"2024-04-21T08:42:22.768071+0000","last_peered":"2024-04-21T08:42:22.768071+0000","last_clean":"2024-04-21T08:42:22.768071+0000","last_became_active":"2024-04-21T08:42:17.543800+0000","last_became_peered":"2024-04-21T08:42:17.543800+0000","last_unstale":"2024-04-21T08:42:22.768071+0000","last_undegraded":"2024-04-21T08:42:22.768071+0000","last_fullsized":"2024-04-21T08:42:22.768071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:44:04.107207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090298+0000","last_change":"2024-04-21T08:42:17.543788+0000","last_active":"2024-04-21T08:42:23.090298+0000","last_peered":"2024-04-21T08:42:23.090298+0000","last_clean":"2024-04-21T08:42:23.090298+0000","last_became_active":"2024-04-21T08:42:17.540939+0000","last_became_peered":"2024-04-21T08:42:17.540939+0000","last_unstale":"2024-04-21T08:42:23.090298+0000","last_undegraded":"2024-04-21T08:42:23.090298+0000","last_fullsized":"2024-04-21T08:42:23.090298+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:59.275098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090192+0000","last_change":"2024-04-21T08:42:17.540605+0000","last_active":"2024-04-21T08:42:23.090192+0000","last_peered":"2024-04-21T08:42:23.090192+0000","last_clean":"2024-04-21T08:42:23.090192+0000","last_became_active":"2024-04-21T08:42:17.540073+0000","last_became_peered":"2024-04-21T08:42:17.540073+0000","last_unstale":"2024-04-21T08:42:23.090192+0000","last_undegraded":"2024-04-21T08:42:23.090192+0000","last_fullsized":"2024-04-21T08:42:23.090192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:02:29.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091491+0000","last_change":"2024-04-21T08:42:17.551608+0000","last_active":"2024-04-21T08:42:23.091491+0000","last_peered":"2024-04-21T08:42:23.091491+0000","last_clean":"2024-04-21T08:42:23.091491+0000","last_became_active":"2024-04-21T08:42:17.551117+0000","last_became_peered":"2024-04-21T08:42:17.551117+0000","last_unstale":"2024-04-21T08:42:23.091491+0000","last_undegraded":"2024-04-21T08:42:23.091491+0000","last_fullsized":"2024-04-21T08:42:23.091491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:18.391928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664818+0000","last_change":"2024-04-21T08:42:17.536509+0000","last_active":"2024-04-21T08:42:22.664818+0000","last_peered":"2024-04-21T08:42:22.664818+0000","last_clean":"2024-04-21T08:42:22.664818+0000","last_became_active":"2024-04-21T08:42:17.536146+0000","last_became_peered":"2024-04-21T08:42:17.536146+0000","last_unstale":"2024-04-21T08:42:22.664818+0000","last_undegraded":"2024-04-21T08:42:22.664818+0000","last_fullsized":"2024-04-21T08:42:22.664818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:29.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768483+0000","last_change":"2024-04-21T08:42:17.543786+0000","last_active":"2024-04-21T08:42:22.768483+0000","last_peered":"2024-04-21T08:42:22.768483+0000","last_clean":"2024-04-21T08:42:22.768483+0000","last_became_active":"2024-04-21T08:42:17.543435+0000","last_became_peered":"2024-04-21T08:42:17.543435+0000","last_unstale":"2024-04-21T08:42:22.768483+0000","last_undegraded":"2024-04-21T08:42:22.768483+0000","last_fullsized":"2024-04-21T08:42:22.768483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090632+0000","last_change":"2024-04-21T08:42:17.541611+0000","last_active":"2024-04-21T08:42:23.090632+0000","last_peered":"2024-04-21T08:42:23.090632+0000","last_clean":"2024-04-21T08:42:23.090632+0000","last_became_active":"2024-04-21T08:42:17.541259+0000","last_became_peered":"2024-04-21T08:42:17.541259+0000","last_unstale":"2024-04-21T08:42:23.090632+0000","last_undegraded":"2024-04-21T08:42:23.090632+0000","last_fullsized":"2024-04-21T08:42:23.090632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:20.900629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092259+0000","last_change":"2024-04-21T08:42:17.541964+0000","last_active":"2024-04-21T08:42:23.092259+0000","last_peered":"2024-04-21T08:42:23.092259+0000","last_clean":"2024-04-21T08:42:23.092259+0000","last_became_active":"2024-04-21T08:42:17.541555+0000","last_became_peered":"2024-04-21T08:42:17.541555+0000","last_unstale":"2024-04-21T08:42:23.092259+0000","last_undegraded":"2024-04-21T08:42:23.092259+0000","last_fullsized":"2024-04-21T08:42:23.092259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:32:47.421452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664282+0000","last_change":"2024-04-21T08:42:17.545526+0000","last_active":"2024-04-21T08:42:22.664282+0000","last_peered":"2024-04-21T08:42:22.664282+0000","last_clean":"2024-04-21T08:42:22.664282+0000","last_became_active":"2024-04-21T08:42:17.545135+0000","last_became_peered":"2024-04-21T08:42:17.545135+0000","last_unstale":"2024-04-21T08:42:22.664282+0000","last_undegraded":"2024-04-21T08:42:22.664282+0000","last_fullsized":"2024-04-21T08:42:22.664282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:37:58.038459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091426+0000","last_change":"2024-04-21T08:42:17.547063+0000","last_active":"2024-04-21T08:42:23.091426+0000","last_peered":"2024-04-21T08:42:23.091426+0000","last_clean":"2024-04-21T08:42:23.091426+0000","last_became_active":"2024-04-21T08:42:17.546689+0000","last_became_peered":"2024-04-21T08:42:17.546689+0000","last_unstale":"2024-04-21T08:42:23.091426+0000","last_undegraded":"2024-04-21T08:42:23.091426+0000","last_fullsized":"2024-04-21T08:42:23.091426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:26.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664551+0000","last_change":"2024-04-21T08:42:17.542545+0000","last_active":"2024-04-21T08:42:22.664551+0000","last_peered":"2024-04-21T08:42:22.664551+0000","last_clean":"2024-04-21T08:42:22.664551+0000","last_became_active":"2024-04-21T08:42:17.542147+0000","last_became_peered":"2024-04-21T08:42:17.542147+0000","last_unstale":"2024-04-21T08:42:22.664551+0000","last_undegraded":"2024-04-21T08:42:22.664551+0000","last_fullsized":"2024-04-21T08:42:22.664551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:15.064110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664687+0000","last_change":"2024-04-21T08:42:17.541262+0000","last_active":"2024-04-21T08:42:22.664687+0000","last_peered":"2024-04-21T08:42:22.664687+0000","last_clean":"2024-04-21T08:42:22.664687+0000","last_became_active":"2024-04-21T08:42:17.540817+0000","last_became_peered":"2024-04-21T08:42:17.540817+0000","last_unstale":"2024-04-21T08:42:22.664687+0000","last_undegraded":"2024-04-21T08:42:22.664687+0000","last_fullsized":"2024-04-21T08:42:22.664687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:22:59.597680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768318+0000","last_change":"2024-04-21T08:42:17.551109+0000","last_active":"2024-04-21T08:42:22.768318+0000","last_peered":"2024-04-21T08:42:22.768318+0000","last_clean":"2024-04-21T08:42:22.768318+0000","last_became_active":"2024-04-21T08:42:17.550104+0000","last_became_peered":"2024-04-21T08:42:17.550104+0000","last_unstale":"2024-04-21T08:42:22.768318+0000","last_undegraded":"2024-04-21T08:42:22.768318+0000","last_fullsized":"2024-04-21T08:42:22.768318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:22:19.361752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281211,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":13920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":970752,"data_stored":480266,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612631,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":1074670,"data_compressed":50511,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776157,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":974848,"data_stored":480848,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037794,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":978944,"data_stored":480870,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299430,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":1071216,"data_compressed":50511,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462957,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":970752,"data_stored":480266,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":484108,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":974848,"data_stored":480356,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":484142,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":480370,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888119,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":974848,"data_stored":480288,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724593,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":1070816,"data_compressed":50511,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:42:45.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:42:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.705+0000 7fcb38a3a640 1 -- 172.21.15.7:0/4181521260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb34105330 msgr2=0x7fcb34107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.705+0000 7fcb38a3a640 1 --2- 172.21.15.7:0/4181521260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb34105330 0x7fcb34107720 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcb28009900 tx=0x7fcb2802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.706+0000 7fcb38a3a640 1 -- 172.21.15.7:0/4181521260 shutdown_connections 2024-04-21T08:42:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.706+0000 7fcb38a3a640 1 --2- 172.21.15.7:0/4181521260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb34107c60 0x7fcb3410a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.706+0000 7fcb38a3a640 1 --2- 172.21.15.7:0/4181521260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb34105330 0x7fcb34107720 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.706+0000 7fcb38a3a640 1 --2- 172.21.15.7:0/4181521260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb34102a00 0x7fcb34104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.706+0000 7fcb38a3a640 1 -- 172.21.15.7:0/4181521260 >> 172.21.15.7:0/4181521260 conn(0x7fcb340fc3f0 msgr2=0x7fcb340fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.706+0000 7fcb38a3a640 1 -- 172.21.15.7:0/4181521260 shutdown_connections 2024-04-21T08:42:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.706+0000 7fcb38a3a640 1 -- 172.21.15.7:0/4181521260 wait complete. 2024-04-21T08:42:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.706+0000 7fcb38a3a640 1 Processor -- start 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.706+0000 7fcb38a3a640 1 -- start start 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb38a3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb34102a00 0x7fcb34102370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb38a3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb34105330 0x7fcb341009b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb38a3a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb34107c60 0x7fcb34100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb38a3a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3410daa0 con 0x7fcb34107c60 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb38a3a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3410d920 con 0x7fcb34102a00 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb38a3a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3410dc20 con 0x7fcb34105330 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb32d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb34107c60 0x7fcb34100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb32d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb34107c60 0x7fcb34100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50290/0 (socket says 172.21.15.7:50290) 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb32d9d640 1 -- 172.21.15.7:0/3518531820 learned_addr learned my addr 172.21.15.7:0/3518531820 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb3259c640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb34102a00 0x7fcb34102370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb31d9b640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb34105330 0x7fcb341009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb32d9d640 1 -- 172.21.15.7:0/3518531820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb34105330 msgr2=0x7fcb341009b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb32d9d640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb34105330 0x7fcb341009b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb32d9d640 1 -- 172.21.15.7:0/3518531820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb34102a00 msgr2=0x7fcb34102370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb32d9d640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb34102a00 0x7fcb34102370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb32d9d640 1 -- 172.21.15.7:0/3518531820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb34101780 con 0x7fcb34107c60 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb32d9d640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb34107c60 0x7fcb34100ef0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c00eaa0 tx=0x7fcb2c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb237fe640 1 -- 172.21.15.7:0/3518531820 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c00cdb0 con 0x7fcb34107c60 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb237fe640 1 -- 172.21.15.7:0/3518531820 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb2c005020 con 0x7fcb34107c60 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb38a3a640 1 -- 172.21.15.7:0/3518531820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb341a22a0 con 0x7fcb34107c60 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb237fe640 1 -- 172.21.15.7:0/3518531820 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c0116d0 con 0x7fcb34107c60 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.707+0000 7fcb38a3a640 1 -- 172.21.15.7:0/3518531820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb341a27c0 con 0x7fcb34107c60 2024-04-21T08:42:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.708+0000 7fcb38a3a640 1 -- 172.21.15.7:0/3518531820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb34068e00 con 0x7fcb34107c60 2024-04-21T08:42:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.709+0000 7fcb237fe640 1 -- 172.21.15.7:0/3518531820 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fcb2c0051d0 con 0x7fcb34107c60 2024-04-21T08:42:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.710+0000 7fcb237fe640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb1c0a3c40 0x7fcb1c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.710+0000 7fcb3259c640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb1c0a3c40 0x7fcb1c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:45.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.710+0000 7fcb237fe640 1 -- 172.21.15.7:0/3518531820 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcb2c0d09b0 con 0x7fcb34107c60 2024-04-21T08:42:45.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.710+0000 7fcb3259c640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb1c0a3c40 0x7fcb1c0a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcb240058c0 tx=0x7fcb2400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:45.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.715+0000 7fcb237fe640 1 -- 172.21.15.7:0/3518531820 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb2c016030 con 0x7fcb34107c60 2024-04-21T08:42:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.898+0000 7fcb38a3a640 1 -- 172.21.15.7:0/3518531820 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcb34062390 con 0x7fcb1c0a3c40 2024-04-21T08:42:45.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:45 smithi007 ceph-mon[28477]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:45.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:45 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/345078780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:45.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:45 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/247816535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.907+0000 7fcb237fe640 1 -- 172.21.15.7:0/3518531820 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340498 (secure 0 0 0) 0x7fcb34062390 con 0x7fcb1c0a3c40 2024-04-21T08:42:45.910 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-21T08:42:45.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 -- 172.21.15.7:0/3518531820 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb1c0a3c40 msgr2=0x7fcb1c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb1c0a3c40 0x7fcb1c0a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcb240058c0 tx=0x7fcb2400a2d0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 -- 172.21.15.7:0/3518531820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb34107c60 msgr2=0x7fcb34100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb34107c60 0x7fcb34100ef0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c00eaa0 tx=0x7fcb2c00ef60 comp rx=0 tx=0).stop 2024-04-21T08:42:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 -- 172.21.15.7:0/3518531820 shutdown_connections 2024-04-21T08:42:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fcb1c0a3c40 0x7fcb1c0a6100 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb34107c60 0x7fcb34100ef0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb34105330 0x7fcb341009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 --2- 172.21.15.7:0/3518531820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb34102a00 0x7fcb34102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 -- 172.21.15.7:0/3518531820 >> 172.21.15.7:0/3518531820 conn(0x7fcb340fc3f0 msgr2=0x7fcb341034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 -- 172.21.15.7:0/3518531820 shutdown_connections 2024-04-21T08:42:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:45.911+0000 7fcb38a3a640 1 -- 172.21.15.7:0/3518531820 wait complete. 2024-04-21T08:42:45.921 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-04-21T08:42:45.634271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595125,"num_objects":25,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":141,"num_read_kb":124,"num_write":231,"num_write_kb":2958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":402436,"kb_used_data":11636,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124539388,"statfs":{"total":1151940427776,"available":1151528333312,"internally_reserved":0,"allocated":11915264,"data_stored":7567104,"data_compressed":284238,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":258,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004151"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768636+0000","last_change":"2024-04-21T08:42:17.545701+0000","last_active":"2024-04-21T08:42:22.768636+0000","last_peered":"2024-04-21T08:42:22.768636+0000","last_clean":"2024-04-21T08:42:22.768636+0000","last_became_active":"2024-04-21T08:42:17.545157+0000","last_became_peered":"2024-04-21T08:42:17.545157+0000","last_unstale":"2024-04-21T08:42:22.768636+0000","last_undegraded":"2024-04-21T08:42:22.768636+0000","last_fullsized":"2024-04-21T08:42:22.768636+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:05:41.952879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091903+0000","last_change":"2024-04-21T08:42:17.541153+0000","last_active":"2024-04-21T08:42:23.091903+0000","last_peered":"2024-04-21T08:42:23.091903+0000","last_clean":"2024-04-21T08:42:23.091903+0000","last_became_active":"2024-04-21T08:42:17.540771+0000","last_became_peered":"2024-04-21T08:42:17.540771+0000","last_unstale":"2024-04-21T08:42:23.091903+0000","last_undegraded":"2024-04-21T08:42:23.091903+0000","last_fullsized":"2024-04-21T08:42:23.091903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:35:06.077367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091371+0000","last_change":"2024-04-21T08:42:17.545525+0000","last_active":"2024-04-21T08:42:23.091371+0000","last_peered":"2024-04-21T08:42:23.091371+0000","last_clean":"2024-04-21T08:42:23.091371+0000","last_became_active":"2024-04-21T08:42:17.545168+0000","last_became_peered":"2024-04-21T08:42:17.545168+0000","last_unstale":"2024-04-21T08:42:23.091371+0000","last_undegraded":"2024-04-21T08:42:23.091371+0000","last_fullsized":"2024-04-21T08:42:23.091371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:51.080385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091605+0000","last_change":"2024-04-21T08:42:17.543917+0000","last_active":"2024-04-21T08:42:23.091605+0000","last_peered":"2024-04-21T08:42:23.091605+0000","last_clean":"2024-04-21T08:42:23.091605+0000","last_became_active":"2024-04-21T08:42:17.543513+0000","last_became_peered":"2024-04-21T08:42:17.543513+0000","last_unstale":"2024-04-21T08:42:23.091605+0000","last_undegraded":"2024-04-21T08:42:23.091605+0000","last_fullsized":"2024-04-21T08:42:23.091605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:23.058600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951634+0000","last_change":"2024-04-21T08:42:17.549737+0000","last_active":"2024-04-21T08:42:22.951634+0000","last_peered":"2024-04-21T08:42:22.951634+0000","last_clean":"2024-04-21T08:42:22.951634+0000","last_became_active":"2024-04-21T08:42:17.549396+0000","last_became_peered":"2024-04-21T08:42:17.549396+0000","last_unstale":"2024-04-21T08:42:22.951634+0000","last_undegraded":"2024-04-21T08:42:22.951634+0000","last_fullsized":"2024-04-21T08:42:22.951634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:01:51.879960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771397+0000","last_change":"2024-04-21T08:42:17.538676+0000","last_active":"2024-04-21T08:42:22.771397+0000","last_peered":"2024-04-21T08:42:22.771397+0000","last_clean":"2024-04-21T08:42:22.771397+0000","last_became_active":"2024-04-21T08:42:17.538252+0000","last_became_peered":"2024-04-21T08:42:17.538252+0000","last_unstale":"2024-04-21T08:42:22.771397+0000","last_undegraded":"2024-04-21T08:42:22.771397+0000","last_fullsized":"2024-04-21T08:42:22.771397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:10:24.081715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091199+0000","last_change":"2024-04-21T08:42:17.541172+0000","last_active":"2024-04-21T08:42:23.091199+0000","last_peered":"2024-04-21T08:42:23.091199+0000","last_clean":"2024-04-21T08:42:23.091199+0000","last_became_active":"2024-04-21T08:42:17.540829+0000","last_became_peered":"2024-04-21T08:42:17.540829+0000","last_unstale":"2024-04-21T08:42:23.091199+0000","last_undegraded":"2024-04-21T08:42:23.091199+0000","last_fullsized":"2024-04-21T08:42:23.091199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:16:12.633980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090834+0000","last_change":"2024-04-21T08:42:17.549744+0000","last_active":"2024-04-21T08:42:23.090834+0000","last_peered":"2024-04-21T08:42:23.090834+0000","last_clean":"2024-04-21T08:42:23.090834+0000","last_became_active":"2024-04-21T08:42:17.549378+0000","last_became_peered":"2024-04-21T08:42:17.549378+0000","last_unstale":"2024-04-21T08:42:23.090834+0000","last_undegraded":"2024-04-21T08:42:23.090834+0000","last_fullsized":"2024-04-21T08:42:23.090834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:03:07.702143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664941+0000","last_change":"2024-04-21T08:42:17.548675+0000","last_active":"2024-04-21T08:42:22.664941+0000","last_peered":"2024-04-21T08:42:22.664941+0000","last_clean":"2024-04-21T08:42:22.664941+0000","last_became_active":"2024-04-21T08:42:17.547809+0000","last_became_peered":"2024-04-21T08:42:17.547809+0000","last_unstale":"2024-04-21T08:42:22.664941+0000","last_undegraded":"2024-04-21T08:42:22.664941+0000","last_fullsized":"2024-04-21T08:42:22.664941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:23.343473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771859+0000","last_change":"2024-04-21T08:42:17.551012+0000","last_active":"2024-04-21T08:42:22.771859+0000","last_peered":"2024-04-21T08:42:22.771859+0000","last_clean":"2024-04-21T08:42:22.771859+0000","last_became_active":"2024-04-21T08:42:17.550627+0000","last_became_peered":"2024-04-21T08:42:17.550627+0000","last_unstale":"2024-04-21T08:42:22.771859+0000","last_undegraded":"2024-04-21T08:42:22.771859+0000","last_fullsized":"2024-04-21T08:42:22.771859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:57.501588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091724+0000","last_change":"2024-04-21T08:42:17.546192+0000","last_active":"2024-04-21T08:42:23.091724+0000","last_peered":"2024-04-21T08:42:23.091724+0000","last_clean":"2024-04-21T08:42:23.091724+0000","last_became_active":"2024-04-21T08:42:17.545808+0000","last_became_peered":"2024-04-21T08:42:17.545808+0000","last_unstale":"2024-04-21T08:42:23.091724+0000","last_undegraded":"2024-04-21T08:42:23.091724+0000","last_fullsized":"2024-04-21T08:42:23.091724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:54.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091491+0000","last_change":"2024-04-21T08:42:17.553351+0000","last_active":"2024-04-21T08:42:23.091491+0000","last_peered":"2024-04-21T08:42:23.091491+0000","last_clean":"2024-04-21T08:42:23.091491+0000","last_became_active":"2024-04-21T08:42:17.550507+0000","last_became_peered":"2024-04-21T08:42:17.550507+0000","last_unstale":"2024-04-21T08:42:23.091491+0000","last_undegraded":"2024-04-21T08:42:23.091491+0000","last_fullsized":"2024-04-21T08:42:23.091491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:25.561375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664867+0000","last_change":"2024-04-21T08:42:17.540533+0000","last_active":"2024-04-21T08:42:22.664867+0000","last_peered":"2024-04-21T08:42:22.664867+0000","last_clean":"2024-04-21T08:42:22.664867+0000","last_became_active":"2024-04-21T08:42:17.539940+0000","last_became_peered":"2024-04-21T08:42:17.539940+0000","last_unstale":"2024-04-21T08:42:22.664867+0000","last_undegraded":"2024-04-21T08:42:22.664867+0000","last_fullsized":"2024-04-21T08:42:22.664867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:45:19.102899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091016+0000","last_change":"2024-04-21T08:42:17.537794+0000","last_active":"2024-04-21T08:42:23.091016+0000","last_peered":"2024-04-21T08:42:23.091016+0000","last_clean":"2024-04-21T08:42:23.091016+0000","last_became_active":"2024-04-21T08:42:17.537376+0000","last_became_peered":"2024-04-21T08:42:17.537376+0000","last_unstale":"2024-04-21T08:42:23.091016+0000","last_undegraded":"2024-04-21T08:42:23.091016+0000","last_fullsized":"2024-04-21T08:42:23.091016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:57.430036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091693+0000","last_change":"2024-04-21T08:42:17.550574+0000","last_active":"2024-04-21T08:42:23.091693+0000","last_peered":"2024-04-21T08:42:23.091693+0000","last_clean":"2024-04-21T08:42:23.091693+0000","last_became_active":"2024-04-21T08:42:17.550205+0000","last_became_peered":"2024-04-21T08:42:17.550205+0000","last_unstale":"2024-04-21T08:42:23.091693+0000","last_undegraded":"2024-04-21T08:42:23.091693+0000","last_fullsized":"2024-04-21T08:42:23.091693+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:41:08.226477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664131+0000","last_change":"2024-04-21T08:42:17.542239+0000","last_active":"2024-04-21T08:42:22.664131+0000","last_peered":"2024-04-21T08:42:22.664131+0000","last_clean":"2024-04-21T08:42:22.664131+0000","last_became_active":"2024-04-21T08:42:17.541848+0000","last_became_peered":"2024-04-21T08:42:17.541848+0000","last_unstale":"2024-04-21T08:42:22.664131+0000","last_undegraded":"2024-04-21T08:42:22.664131+0000","last_fullsized":"2024-04-21T08:42:22.664131+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:47.717451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768476+0000","last_change":"2024-04-21T08:42:17.553200+0000","last_active":"2024-04-21T08:42:22.768476+0000","last_peered":"2024-04-21T08:42:22.768476+0000","last_clean":"2024-04-21T08:42:22.768476+0000","last_became_active":"2024-04-21T08:42:17.552913+0000","last_became_peered":"2024-04-21T08:42:17.552913+0000","last_unstale":"2024-04-21T08:42:22.768476+0000","last_undegraded":"2024-04-21T08:42:22.768476+0000","last_fullsized":"2024-04-21T08:42:22.768476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:39:04.186928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680572+0000","last_change":"2024-04-21T08:42:17.534881+0000","last_active":"2024-04-21T08:42:22.680572+0000","last_peered":"2024-04-21T08:42:22.680572+0000","last_clean":"2024-04-21T08:42:22.680572+0000","last_became_active":"2024-04-21T08:42:17.534496+0000","last_became_peered":"2024-04-21T08:42:17.534496+0000","last_unstale":"2024-04-21T08:42:22.680572+0000","last_undegraded":"2024-04-21T08:42:22.680572+0000","last_fullsized":"2024-04-21T08:42:22.680572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:50:19.409391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090574+0000","last_change":"2024-04-21T08:42:17.540480+0000","last_active":"2024-04-21T08:42:23.090574+0000","last_peered":"2024-04-21T08:42:23.090574+0000","last_clean":"2024-04-21T08:42:23.090574+0000","last_became_active":"2024-04-21T08:42:17.539742+0000","last_became_peered":"2024-04-21T08:42:17.539742+0000","last_unstale":"2024-04-21T08:42:23.090574+0000","last_undegraded":"2024-04-21T08:42:23.090574+0000","last_fullsized":"2024-04-21T08:42:23.090574+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:00.238976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090525+0000","last_change":"2024-04-21T08:42:21.661514+0000","last_active":"2024-04-21T08:42:23.090525+0000","last_peered":"2024-04-21T08:42:23.090525+0000","last_clean":"2024-04-21T08:42:23.090525+0000","last_became_active":"2024-04-21T08:42:21.661095+0000","last_became_peered":"2024-04-21T08:42:21.661095+0000","last_unstale":"2024-04-21T08:42:23.090525+0000","last_undegraded":"2024-04-21T08:42:23.090525+0000","last_fullsized":"2024-04-21T08:42:23.090525+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:27:28.933446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680638+0000","last_change":"2024-04-21T08:42:16.639888+0000","last_active":"2024-04-21T08:42:22.680638+0000","last_peered":"2024-04-21T08:42:22.680638+0000","last_clean":"2024-04-21T08:42:22.680638+0000","last_became_active":"2024-04-21T08:42:16.639403+0000","last_became_peered":"2024-04-21T08:42:16.639403+0000","last_unstale":"2024-04-21T08:42:22.680638+0000","last_undegraded":"2024-04-21T08:42:22.680638+0000","last_fullsized":"2024-04-21T08:42:22.680638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:08:17.164924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091798+0000","last_change":"2024-04-21T08:42:17.542557+0000","last_active":"2024-04-21T08:42:23.091798+0000","last_peered":"2024-04-21T08:42:23.091798+0000","last_clean":"2024-04-21T08:42:23.091798+0000","last_became_active":"2024-04-21T08:42:17.542196+0000","last_became_peered":"2024-04-21T08:42:17.542196+0000","last_unstale":"2024-04-21T08:42:23.091798+0000","last_undegraded":"2024-04-21T08:42:23.091798+0000","last_fullsized":"2024-04-21T08:42:23.091798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:26:33.153918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091987+0000","last_change":"2024-04-21T08:42:21.673197+0000","last_active":"2024-04-21T08:42:23.091987+0000","last_peered":"2024-04-21T08:42:23.091987+0000","last_clean":"2024-04-21T08:42:23.091987+0000","last_became_active":"2024-04-21T08:42:21.672793+0000","last_became_peered":"2024-04-21T08:42:21.672793+0000","last_unstale":"2024-04-21T08:42:23.091987+0000","last_undegraded":"2024-04-21T08:42:23.091987+0000","last_fullsized":"2024-04-21T08:42:23.091987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:44:29.788459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092057+0000","last_change":"2024-04-21T08:42:16.648577+0000","last_active":"2024-04-21T08:42:23.092057+0000","last_peered":"2024-04-21T08:42:23.092057+0000","last_clean":"2024-04-21T08:42:23.092057+0000","last_became_active":"2024-04-21T08:42:16.646716+0000","last_became_peered":"2024-04-21T08:42:16.646716+0000","last_unstale":"2024-04-21T08:42:23.092057+0000","last_undegraded":"2024-04-21T08:42:23.092057+0000","last_fullsized":"2024-04-21T08:42:23.092057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:10.327078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092011+0000","last_change":"2024-04-21T08:42:17.545430+0000","last_active":"2024-04-21T08:42:23.092011+0000","last_peered":"2024-04-21T08:42:23.092011+0000","last_clean":"2024-04-21T08:42:23.092011+0000","last_became_active":"2024-04-21T08:42:17.545057+0000","last_became_peered":"2024-04-21T08:42:17.545057+0000","last_unstale":"2024-04-21T08:42:23.092011+0000","last_undegraded":"2024-04-21T08:42:23.092011+0000","last_fullsized":"2024-04-21T08:42:23.092011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:44:59.030830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091987+0000","last_change":"2024-04-21T08:42:21.673196+0000","last_active":"2024-04-21T08:42:23.091987+0000","last_peered":"2024-04-21T08:42:23.091987+0000","last_clean":"2024-04-21T08:42:23.091987+0000","last_became_active":"2024-04-21T08:42:21.672808+0000","last_became_peered":"2024-04-21T08:42:21.672808+0000","last_unstale":"2024-04-21T08:42:23.091987+0000","last_undegraded":"2024-04-21T08:42:23.091987+0000","last_fullsized":"2024-04-21T08:42:23.091987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:39.482353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951824+0000","last_change":"2024-04-21T08:42:16.641611+0000","last_active":"2024-04-21T08:42:22.951824+0000","last_peered":"2024-04-21T08:42:22.951824+0000","last_clean":"2024-04-21T08:42:22.951824+0000","last_became_active":"2024-04-21T08:42:16.641221+0000","last_became_peered":"2024-04-21T08:42:16.641221+0000","last_unstale":"2024-04-21T08:42:22.951824+0000","last_undegraded":"2024-04-21T08:42:22.951824+0000","last_fullsized":"2024-04-21T08:42:22.951824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:45.496532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664491+0000","last_change":"2024-04-21T08:42:17.532986+0000","last_active":"2024-04-21T08:42:22.664491+0000","last_peered":"2024-04-21T08:42:22.664491+0000","last_clean":"2024-04-21T08:42:22.664491+0000","last_became_active":"2024-04-21T08:42:17.532583+0000","last_became_peered":"2024-04-21T08:42:17.532583+0000","last_unstale":"2024-04-21T08:42:22.664491+0000","last_undegraded":"2024-04-21T08:42:22.664491+0000","last_fullsized":"2024-04-21T08:42:22.664491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:43.780935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092145+0000","last_change":"2024-04-21T08:42:21.667459+0000","last_active":"2024-04-21T08:42:23.092145+0000","last_peered":"2024-04-21T08:42:23.092145+0000","last_clean":"2024-04-21T08:42:23.092145+0000","last_became_active":"2024-04-21T08:42:21.667183+0000","last_became_peered":"2024-04-21T08:42:21.667183+0000","last_unstale":"2024-04-21T08:42:23.092145+0000","last_undegraded":"2024-04-21T08:42:23.092145+0000","last_fullsized":"2024-04-21T08:42:23.092145+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:32:23.626267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951843+0000","last_change":"2024-04-21T08:42:16.645210+0000","last_active":"2024-04-21T08:42:22.951843+0000","last_peered":"2024-04-21T08:42:22.951843+0000","last_clean":"2024-04-21T08:42:22.951843+0000","last_became_active":"2024-04-21T08:42:16.644747+0000","last_became_peered":"2024-04-21T08:42:16.644747+0000","last_unstale":"2024-04-21T08:42:22.951843+0000","last_undegraded":"2024-04-21T08:42:22.951843+0000","last_fullsized":"2024-04-21T08:42:22.951843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:34:45.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664370+0000","last_change":"2024-04-21T08:42:17.547110+0000","last_active":"2024-04-21T08:42:22.664370+0000","last_peered":"2024-04-21T08:42:22.664370+0000","last_clean":"2024-04-21T08:42:22.664370+0000","last_became_active":"2024-04-21T08:42:17.546715+0000","last_became_peered":"2024-04-21T08:42:17.546715+0000","last_unstale":"2024-04-21T08:42:22.664370+0000","last_undegraded":"2024-04-21T08:42:22.664370+0000","last_fullsized":"2024-04-21T08:42:22.664370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:26:51.780113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090612+0000","last_change":"2024-04-21T08:42:21.660739+0000","last_active":"2024-04-21T08:42:23.090612+0000","last_peered":"2024-04-21T08:42:23.090612+0000","last_clean":"2024-04-21T08:42:23.090612+0000","last_became_active":"2024-04-21T08:42:21.660307+0000","last_became_peered":"2024-04-21T08:42:21.660307+0000","last_unstale":"2024-04-21T08:42:23.090612+0000","last_undegraded":"2024-04-21T08:42:23.090612+0000","last_fullsized":"2024-04-21T08:42:23.090612+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:26.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664111+0000","last_change":"2024-04-21T08:42:16.639865+0000","last_active":"2024-04-21T08:42:22.664111+0000","last_peered":"2024-04-21T08:42:22.664111+0000","last_clean":"2024-04-21T08:42:22.664111+0000","last_became_active":"2024-04-21T08:42:16.639539+0000","last_became_peered":"2024-04-21T08:42:16.639539+0000","last_unstale":"2024-04-21T08:42:22.664111+0000","last_undegraded":"2024-04-21T08:42:22.664111+0000","last_fullsized":"2024-04-21T08:42:22.664111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:52:33.645493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951535+0000","last_change":"2024-04-21T08:42:17.542315+0000","last_active":"2024-04-21T08:42:22.951535+0000","last_peered":"2024-04-21T08:42:22.951535+0000","last_clean":"2024-04-21T08:42:22.951535+0000","last_became_active":"2024-04-21T08:42:17.541895+0000","last_became_peered":"2024-04-21T08:42:17.541895+0000","last_unstale":"2024-04-21T08:42:22.951535+0000","last_undegraded":"2024-04-21T08:42:22.951535+0000","last_fullsized":"2024-04-21T08:42:22.951535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:23:33.909499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090736+0000","last_change":"2024-04-21T08:42:21.661541+0000","last_active":"2024-04-21T08:42:23.090736+0000","last_peered":"2024-04-21T08:42:23.090736+0000","last_clean":"2024-04-21T08:42:23.090736+0000","last_became_active":"2024-04-21T08:42:21.661121+0000","last_became_peered":"2024-04-21T08:42:21.661121+0000","last_unstale":"2024-04-21T08:42:23.090736+0000","last_undegraded":"2024-04-21T08:42:23.090736+0000","last_fullsized":"2024-04-21T08:42:23.090736+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:28.624203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771355+0000","last_change":"2024-04-21T08:42:16.644402+0000","last_active":"2024-04-21T08:42:22.771355+0000","last_peered":"2024-04-21T08:42:22.771355+0000","last_clean":"2024-04-21T08:42:22.771355+0000","last_became_active":"2024-04-21T08:42:16.643928+0000","last_became_peered":"2024-04-21T08:42:16.643928+0000","last_unstale":"2024-04-21T08:42:22.771355+0000","last_undegraded":"2024-04-21T08:42:22.771355+0000","last_fullsized":"2024-04-21T08:42:22.771355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:54:14.154892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664231+0000","last_change":"2024-04-21T08:42:17.547770+0000","last_active":"2024-04-21T08:42:22.664231+0000","last_peered":"2024-04-21T08:42:22.664231+0000","last_clean":"2024-04-21T08:42:22.664231+0000","last_became_active":"2024-04-21T08:42:17.547440+0000","last_became_peered":"2024-04-21T08:42:17.547440+0000","last_unstale":"2024-04-21T08:42:22.664231+0000","last_undegraded":"2024-04-21T08:42:22.664231+0000","last_fullsized":"2024-04-21T08:42:22.664231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:37:35.555737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768567+0000","last_change":"2024-04-21T08:42:21.670684+0000","last_active":"2024-04-21T08:42:22.768567+0000","last_peered":"2024-04-21T08:42:22.768567+0000","last_clean":"2024-04-21T08:42:22.768567+0000","last_became_active":"2024-04-21T08:42:21.670260+0000","last_became_peered":"2024-04-21T08:42:21.670260+0000","last_unstale":"2024-04-21T08:42:22.768567+0000","last_undegraded":"2024-04-21T08:42:22.768567+0000","last_fullsized":"2024-04-21T08:42:22.768567+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:45:06.192040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664440+0000","last_change":"2024-04-21T08:42:16.639632+0000","last_active":"2024-04-21T08:42:22.664440+0000","last_peered":"2024-04-21T08:42:22.664440+0000","last_clean":"2024-04-21T08:42:22.664440+0000","last_became_active":"2024-04-21T08:42:16.639301+0000","last_became_peered":"2024-04-21T08:42:16.639301+0000","last_unstale":"2024-04-21T08:42:22.664440+0000","last_undegraded":"2024-04-21T08:42:22.664440+0000","last_fullsized":"2024-04-21T08:42:22.664440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:58.838667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090950+0000","last_change":"2024-04-21T08:42:17.546109+0000","last_active":"2024-04-21T08:42:23.090950+0000","last_peered":"2024-04-21T08:42:23.090950+0000","last_clean":"2024-04-21T08:42:23.090950+0000","last_became_active":"2024-04-21T08:42:17.545673+0000","last_became_peered":"2024-04-21T08:42:17.545673+0000","last_unstale":"2024-04-21T08:42:23.090950+0000","last_undegraded":"2024-04-21T08:42:23.090950+0000","last_fullsized":"2024-04-21T08:42:23.090950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:18.947848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091744+0000","last_change":"2024-04-21T08:42:21.666114+0000","last_active":"2024-04-21T08:42:23.091744+0000","last_peered":"2024-04-21T08:42:23.091744+0000","last_clean":"2024-04-21T08:42:23.091744+0000","last_became_active":"2024-04-21T08:42:21.665020+0000","last_became_peered":"2024-04-21T08:42:21.665020+0000","last_unstale":"2024-04-21T08:42:23.091744+0000","last_undegraded":"2024-04-21T08:42:23.091744+0000","last_fullsized":"2024-04-21T08:42:23.091744+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:33.644178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664153+0000","last_change":"2024-04-21T08:42:16.639118+0000","last_active":"2024-04-21T08:42:22.664153+0000","last_peered":"2024-04-21T08:42:22.664153+0000","last_clean":"2024-04-21T08:42:22.664153+0000","last_became_active":"2024-04-21T08:42:16.638792+0000","last_became_peered":"2024-04-21T08:42:16.638792+0000","last_unstale":"2024-04-21T08:42:22.664153+0000","last_undegraded":"2024-04-21T08:42:22.664153+0000","last_fullsized":"2024-04-21T08:42:22.664153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:55.568712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680193+0000","last_change":"2024-04-21T08:42:17.539688+0000","last_active":"2024-04-21T08:42:22.680193+0000","last_peered":"2024-04-21T08:42:22.680193+0000","last_clean":"2024-04-21T08:42:22.680193+0000","last_became_active":"2024-04-21T08:42:17.539335+0000","last_became_peered":"2024-04-21T08:42:17.539335+0000","last_unstale":"2024-04-21T08:42:22.680193+0000","last_undegraded":"2024-04-21T08:42:22.680193+0000","last_fullsized":"2024-04-21T08:42:22.680193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:27:51.520232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091240+0000","last_change":"2024-04-21T08:42:21.660570+0000","last_active":"2024-04-21T08:42:23.091240+0000","last_peered":"2024-04-21T08:42:23.091240+0000","last_clean":"2024-04-21T08:42:23.091240+0000","last_became_active":"2024-04-21T08:42:21.659398+0000","last_became_peered":"2024-04-21T08:42:21.659398+0000","last_unstale":"2024-04-21T08:42:23.091240+0000","last_undegraded":"2024-04-21T08:42:23.091240+0000","last_fullsized":"2024-04-21T08:42:23.091240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:55.840334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771677+0000","last_change":"2024-04-21T08:42:16.648192+0000","last_active":"2024-04-21T08:42:22.771677+0000","last_peered":"2024-04-21T08:42:22.771677+0000","last_clean":"2024-04-21T08:42:22.771677+0000","last_became_active":"2024-04-21T08:42:16.647666+0000","last_became_peered":"2024-04-21T08:42:16.647666+0000","last_unstale":"2024-04-21T08:42:22.771677+0000","last_undegraded":"2024-04-21T08:42:22.771677+0000","last_fullsized":"2024-04-21T08:42:22.771677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:33.344680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664948+0000","last_change":"2024-04-21T08:42:17.545925+0000","last_active":"2024-04-21T08:42:22.664948+0000","last_peered":"2024-04-21T08:42:22.664948+0000","last_clean":"2024-04-21T08:42:22.664948+0000","last_became_active":"2024-04-21T08:42:17.545555+0000","last_became_peered":"2024-04-21T08:42:17.545555+0000","last_unstale":"2024-04-21T08:42:22.664948+0000","last_undegraded":"2024-04-21T08:42:22.664948+0000","last_fullsized":"2024-04-21T08:42:22.664948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:13:07.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680076+0000","last_change":"2024-04-21T08:42:21.662757+0000","last_active":"2024-04-21T08:42:22.680076+0000","last_peered":"2024-04-21T08:42:22.680076+0000","last_clean":"2024-04-21T08:42:22.680076+0000","last_became_active":"2024-04-21T08:42:21.662306+0000","last_became_peered":"2024-04-21T08:42:21.662306+0000","last_unstale":"2024-04-21T08:42:22.680076+0000","last_undegraded":"2024-04-21T08:42:22.680076+0000","last_fullsized":"2024-04-21T08:42:22.680076+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:26.856147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091702+0000","last_change":"2024-04-21T08:42:16.644191+0000","last_active":"2024-04-21T08:42:23.091702+0000","last_peered":"2024-04-21T08:42:23.091702+0000","last_clean":"2024-04-21T08:42:23.091702+0000","last_became_active":"2024-04-21T08:42:16.643697+0000","last_became_peered":"2024-04-21T08:42:16.643697+0000","last_unstale":"2024-04-21T08:42:23.091702+0000","last_undegraded":"2024-04-21T08:42:23.091702+0000","last_fullsized":"2024-04-21T08:42:23.091702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:48.680680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092262+0000","last_change":"2024-04-21T08:42:17.551026+0000","last_active":"2024-04-21T08:42:23.092262+0000","last_peered":"2024-04-21T08:42:23.092262+0000","last_clean":"2024-04-21T08:42:23.092262+0000","last_became_active":"2024-04-21T08:42:17.550461+0000","last_became_peered":"2024-04-21T08:42:17.550461+0000","last_unstale":"2024-04-21T08:42:23.092262+0000","last_undegraded":"2024-04-21T08:42:23.092262+0000","last_fullsized":"2024-04-21T08:42:23.092262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:09.736721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092346+0000","last_change":"2024-04-21T08:42:21.670561+0000","last_active":"2024-04-21T08:42:23.092346+0000","last_peered":"2024-04-21T08:42:23.092346+0000","last_clean":"2024-04-21T08:42:23.092346+0000","last_became_active":"2024-04-21T08:42:21.670170+0000","last_became_peered":"2024-04-21T08:42:21.670170+0000","last_unstale":"2024-04-21T08:42:23.092346+0000","last_undegraded":"2024-04-21T08:42:23.092346+0000","last_fullsized":"2024-04-21T08:42:23.092346+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:35:54.538013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664307+0000","last_change":"2024-04-21T08:42:16.637319+0000","last_active":"2024-04-21T08:42:22.664307+0000","last_peered":"2024-04-21T08:42:22.664307+0000","last_clean":"2024-04-21T08:42:22.664307+0000","last_became_active":"2024-04-21T08:42:16.636940+0000","last_became_peered":"2024-04-21T08:42:16.636940+0000","last_unstale":"2024-04-21T08:42:22.664307+0000","last_undegraded":"2024-04-21T08:42:22.664307+0000","last_fullsized":"2024-04-21T08:42:22.664307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:59:09.826786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664745+0000","last_change":"2024-04-21T08:42:17.546202+0000","last_active":"2024-04-21T08:42:22.664745+0000","last_peered":"2024-04-21T08:42:22.664745+0000","last_clean":"2024-04-21T08:42:22.664745+0000","last_became_active":"2024-04-21T08:42:17.545781+0000","last_became_peered":"2024-04-21T08:42:17.545781+0000","last_unstale":"2024-04-21T08:42:22.664745+0000","last_undegraded":"2024-04-21T08:42:22.664745+0000","last_fullsized":"2024-04-21T08:42:22.664745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:23.990951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680338+0000","last_change":"2024-04-21T08:42:21.662822+0000","last_active":"2024-04-21T08:42:22.680338+0000","last_peered":"2024-04-21T08:42:22.680338+0000","last_clean":"2024-04-21T08:42:22.680338+0000","last_became_active":"2024-04-21T08:42:21.662398+0000","last_became_peered":"2024-04-21T08:42:21.662398+0000","last_unstale":"2024-04-21T08:42:22.680338+0000","last_undegraded":"2024-04-21T08:42:22.680338+0000","last_fullsized":"2024-04-21T08:42:22.680338+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:38:40.499294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'10","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:33.258766+0000","last_change":"2024-04-21T08:42:16.636330+0000","last_active":"2024-04-21T08:42:33.258766+0000","last_peered":"2024-04-21T08:42:33.258766+0000","last_clean":"2024-04-21T08:42:33.258766+0000","last_became_active":"2024-04-21T08:42:16.635762+0000","last_became_peered":"2024-04-21T08:42:16.635762+0000","last_unstale":"2024-04-21T08:42:33.258766+0000","last_undegraded":"2024-04-21T08:42:33.258766+0000","last_fullsized":"2024-04-21T08:42:33.258766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:31.137923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951660+0000","last_change":"2024-04-21T08:42:17.543033+0000","last_active":"2024-04-21T08:42:22.951660+0000","last_peered":"2024-04-21T08:42:22.951660+0000","last_clean":"2024-04-21T08:42:22.951660+0000","last_became_active":"2024-04-21T08:42:17.542692+0000","last_became_peered":"2024-04-21T08:42:17.542692+0000","last_unstale":"2024-04-21T08:42:22.951660+0000","last_undegraded":"2024-04-21T08:42:22.951660+0000","last_fullsized":"2024-04-21T08:42:22.951660+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:33:32.111999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664742+0000","last_change":"2024-04-21T08:42:21.668831+0000","last_active":"2024-04-21T08:42:22.664742+0000","last_peered":"2024-04-21T08:42:22.664742+0000","last_clean":"2024-04-21T08:42:22.664742+0000","last_became_active":"2024-04-21T08:42:21.668485+0000","last_became_peered":"2024-04-21T08:42:21.668485+0000","last_unstale":"2024-04-21T08:42:22.664742+0000","last_undegraded":"2024-04-21T08:42:22.664742+0000","last_fullsized":"2024-04-21T08:42:22.664742+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:49:29.049707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090476+0000","last_change":"2024-04-21T08:42:16.640288+0000","last_active":"2024-04-21T08:42:23.090476+0000","last_peered":"2024-04-21T08:42:23.090476+0000","last_clean":"2024-04-21T08:42:23.090476+0000","last_became_active":"2024-04-21T08:42:16.639909+0000","last_became_peered":"2024-04-21T08:42:16.639909+0000","last_unstale":"2024-04-21T08:42:23.090476+0000","last_undegraded":"2024-04-21T08:42:23.090476+0000","last_fullsized":"2024-04-21T08:42:23.090476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:33.839442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664531+0000","last_change":"2024-04-21T08:42:17.541154+0000","last_active":"2024-04-21T08:42:22.664531+0000","last_peered":"2024-04-21T08:42:22.664531+0000","last_clean":"2024-04-21T08:42:22.664531+0000","last_became_active":"2024-04-21T08:42:17.540747+0000","last_became_peered":"2024-04-21T08:42:17.540747+0000","last_unstale":"2024-04-21T08:42:22.664531+0000","last_undegraded":"2024-04-21T08:42:22.664531+0000","last_fullsized":"2024-04-21T08:42:22.664531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:14:50.641144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768232+0000","last_change":"2024-04-21T08:42:21.670767+0000","last_active":"2024-04-21T08:42:22.768232+0000","last_peered":"2024-04-21T08:42:22.768232+0000","last_clean":"2024-04-21T08:42:22.768232+0000","last_became_active":"2024-04-21T08:42:21.670338+0000","last_became_peered":"2024-04-21T08:42:21.670338+0000","last_unstale":"2024-04-21T08:42:22.768232+0000","last_undegraded":"2024-04-21T08:42:22.768232+0000","last_fullsized":"2024-04-21T08:42:22.768232+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:38:55.275603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664133+0000","last_change":"2024-04-21T08:42:16.640144+0000","last_active":"2024-04-21T08:42:22.664133+0000","last_peered":"2024-04-21T08:42:22.664133+0000","last_clean":"2024-04-21T08:42:22.664133+0000","last_became_active":"2024-04-21T08:42:16.639707+0000","last_became_peered":"2024-04-21T08:42:16.639707+0000","last_unstale":"2024-04-21T08:42:22.664133+0000","last_undegraded":"2024-04-21T08:42:22.664133+0000","last_fullsized":"2024-04-21T08:42:22.664133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:11:32.785519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091827+0000","last_change":"2024-04-21T08:42:17.555019+0000","last_active":"2024-04-21T08:42:23.091827+0000","last_peered":"2024-04-21T08:42:23.091827+0000","last_clean":"2024-04-21T08:42:23.091827+0000","last_became_active":"2024-04-21T08:42:17.554645+0000","last_became_peered":"2024-04-21T08:42:17.554645+0000","last_unstale":"2024-04-21T08:42:23.091827+0000","last_undegraded":"2024-04-21T08:42:23.091827+0000","last_fullsized":"2024-04-21T08:42:23.091827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:42:30.353945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680387+0000","last_change":"2024-04-21T08:42:21.663567+0000","last_active":"2024-04-21T08:42:22.680387+0000","last_peered":"2024-04-21T08:42:22.680387+0000","last_clean":"2024-04-21T08:42:22.680387+0000","last_became_active":"2024-04-21T08:42:21.663203+0000","last_became_peered":"2024-04-21T08:42:21.663203+0000","last_unstale":"2024-04-21T08:42:22.680387+0000","last_undegraded":"2024-04-21T08:42:22.680387+0000","last_fullsized":"2024-04-21T08:42:22.680387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:50.760336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091780+0000","last_change":"2024-04-21T08:42:16.640407+0000","last_active":"2024-04-21T08:42:23.091780+0000","last_peered":"2024-04-21T08:42:23.091780+0000","last_clean":"2024-04-21T08:42:23.091780+0000","last_became_active":"2024-04-21T08:42:16.640015+0000","last_became_peered":"2024-04-21T08:42:16.640015+0000","last_unstale":"2024-04-21T08:42:23.091780+0000","last_undegraded":"2024-04-21T08:42:23.091780+0000","last_fullsized":"2024-04-21T08:42:23.091780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:23:10.647184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091167+0000","last_change":"2024-04-21T08:42:17.545438+0000","last_active":"2024-04-21T08:42:23.091167+0000","last_peered":"2024-04-21T08:42:23.091167+0000","last_clean":"2024-04-21T08:42:23.091167+0000","last_became_active":"2024-04-21T08:42:17.544258+0000","last_became_peered":"2024-04-21T08:42:17.544258+0000","last_unstale":"2024-04-21T08:42:23.091167+0000","last_undegraded":"2024-04-21T08:42:23.091167+0000","last_fullsized":"2024-04-21T08:42:23.091167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:43.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.665171+0000","last_change":"2024-04-21T08:42:21.671000+0000","last_active":"2024-04-21T08:42:22.665171+0000","last_peered":"2024-04-21T08:42:22.665171+0000","last_clean":"2024-04-21T08:42:22.665171+0000","last_became_active":"2024-04-21T08:42:21.670380+0000","last_became_peered":"2024-04-21T08:42:21.670380+0000","last_unstale":"2024-04-21T08:42:22.665171+0000","last_undegraded":"2024-04-21T08:42:22.665171+0000","last_fullsized":"2024-04-21T08:42:22.665171+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:57.075686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951408+0000","last_change":"2024-04-21T08:42:16.649599+0000","last_active":"2024-04-21T08:42:22.951408+0000","last_peered":"2024-04-21T08:42:22.951408+0000","last_clean":"2024-04-21T08:42:22.951408+0000","last_became_active":"2024-04-21T08:42:16.649232+0000","last_became_peered":"2024-04-21T08:42:16.649232+0000","last_unstale":"2024-04-21T08:42:22.951408+0000","last_undegraded":"2024-04-21T08:42:22.951408+0000","last_fullsized":"2024-04-21T08:42:22.951408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:21.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664102+0000","last_change":"2024-04-21T08:42:17.545910+0000","last_active":"2024-04-21T08:42:22.664102+0000","last_peered":"2024-04-21T08:42:22.664102+0000","last_clean":"2024-04-21T08:42:22.664102+0000","last_became_active":"2024-04-21T08:42:17.545582+0000","last_became_peered":"2024-04-21T08:42:17.545582+0000","last_unstale":"2024-04-21T08:42:22.664102+0000","last_undegraded":"2024-04-21T08:42:22.664102+0000","last_fullsized":"2024-04-21T08:42:22.664102+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:25.466146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.663988+0000","last_change":"2024-04-21T08:42:21.664811+0000","last_active":"2024-04-21T08:42:22.663988+0000","last_peered":"2024-04-21T08:42:22.663988+0000","last_clean":"2024-04-21T08:42:22.663988+0000","last_became_active":"2024-04-21T08:42:21.664332+0000","last_became_peered":"2024-04-21T08:42:21.664332+0000","last_unstale":"2024-04-21T08:42:22.663988+0000","last_undegraded":"2024-04-21T08:42:22.663988+0000","last_fullsized":"2024-04-21T08:42:22.663988+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:40.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951969+0000","last_change":"2024-04-21T08:42:16.641233+0000","last_active":"2024-04-21T08:42:22.951969+0000","last_peered":"2024-04-21T08:42:22.951969+0000","last_clean":"2024-04-21T08:42:22.951969+0000","last_became_active":"2024-04-21T08:42:16.640873+0000","last_became_peered":"2024-04-21T08:42:16.640873+0000","last_unstale":"2024-04-21T08:42:22.951969+0000","last_undegraded":"2024-04-21T08:42:22.951969+0000","last_fullsized":"2024-04-21T08:42:22.951969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:00.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768416+0000","last_change":"2024-04-21T08:42:17.550438+0000","last_active":"2024-04-21T08:42:22.768416+0000","last_peered":"2024-04-21T08:42:22.768416+0000","last_clean":"2024-04-21T08:42:22.768416+0000","last_became_active":"2024-04-21T08:42:17.550015+0000","last_became_peered":"2024-04-21T08:42:17.550015+0000","last_unstale":"2024-04-21T08:42:22.768416+0000","last_undegraded":"2024-04-21T08:42:22.768416+0000","last_fullsized":"2024-04-21T08:42:22.768416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:00:29.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951618+0000","last_change":"2024-04-21T08:42:21.970498+0000","last_active":"2024-04-21T08:42:22.951618+0000","last_peered":"2024-04-21T08:42:22.951618+0000","last_clean":"2024-04-21T08:42:22.951618+0000","last_became_active":"2024-04-21T08:42:21.969819+0000","last_became_peered":"2024-04-21T08:42:21.969819+0000","last_unstale":"2024-04-21T08:42:22.951618+0000","last_undegraded":"2024-04-21T08:42:22.951618+0000","last_fullsized":"2024-04-21T08:42:22.951618+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:13:16.552293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680361+0000","last_change":"2024-04-21T08:42:16.637146+0000","last_active":"2024-04-21T08:42:22.680361+0000","last_peered":"2024-04-21T08:42:22.680361+0000","last_clean":"2024-04-21T08:42:22.680361+0000","last_became_active":"2024-04-21T08:42:16.636781+0000","last_became_peered":"2024-04-21T08:42:16.636781+0000","last_unstale":"2024-04-21T08:42:22.680361+0000","last_undegraded":"2024-04-21T08:42:22.680361+0000","last_fullsized":"2024-04-21T08:42:22.680361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:39:27.641236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771645+0000","last_change":"2024-04-21T08:42:17.542325+0000","last_active":"2024-04-21T08:42:22.771645+0000","last_peered":"2024-04-21T08:42:22.771645+0000","last_clean":"2024-04-21T08:42:22.771645+0000","last_became_active":"2024-04-21T08:42:17.541103+0000","last_became_peered":"2024-04-21T08:42:17.541103+0000","last_unstale":"2024-04-21T08:42:22.771645+0000","last_undegraded":"2024-04-21T08:42:22.771645+0000","last_fullsized":"2024-04-21T08:42:22.771645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:46.903250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664900+0000","last_change":"2024-04-21T08:42:21.671146+0000","last_active":"2024-04-21T08:42:22.664900+0000","last_peered":"2024-04-21T08:42:22.664900+0000","last_clean":"2024-04-21T08:42:22.664900+0000","last_became_active":"2024-04-21T08:42:21.670561+0000","last_became_peered":"2024-04-21T08:42:21.670561+0000","last_unstale":"2024-04-21T08:42:22.664900+0000","last_undegraded":"2024-04-21T08:42:22.664900+0000","last_fullsized":"2024-04-21T08:42:22.664900+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:38:28.028976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091471+0000","last_change":"2024-04-21T08:42:16.645132+0000","last_active":"2024-04-21T08:42:23.091471+0000","last_peered":"2024-04-21T08:42:23.091471+0000","last_clean":"2024-04-21T08:42:23.091471+0000","last_became_active":"2024-04-21T08:42:16.644786+0000","last_became_peered":"2024-04-21T08:42:16.644786+0000","last_unstale":"2024-04-21T08:42:23.091471+0000","last_undegraded":"2024-04-21T08:42:23.091471+0000","last_fullsized":"2024-04-21T08:42:23.091471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:08.298498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.771569+0000","last_change":"2024-04-21T08:42:17.550424+0000","last_active":"2024-04-21T08:42:22.771569+0000","last_peered":"2024-04-21T08:42:22.771569+0000","last_clean":"2024-04-21T08:42:22.771569+0000","last_became_active":"2024-04-21T08:42:17.550077+0000","last_became_peered":"2024-04-21T08:42:17.550077+0000","last_unstale":"2024-04-21T08:42:22.771569+0000","last_undegraded":"2024-04-21T08:42:22.771569+0000","last_fullsized":"2024-04-21T08:42:22.771569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:17:07.016204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.665144+0000","last_change":"2024-04-21T08:42:21.970652+0000","last_active":"2024-04-21T08:42:22.665144+0000","last_peered":"2024-04-21T08:42:22.665144+0000","last_clean":"2024-04-21T08:42:22.665144+0000","last_became_active":"2024-04-21T08:42:21.970251+0000","last_became_peered":"2024-04-21T08:42:21.970251+0000","last_unstale":"2024-04-21T08:42:22.665144+0000","last_undegraded":"2024-04-21T08:42:22.665144+0000","last_fullsized":"2024-04-21T08:42:22.665144+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:45.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664385+0000","last_change":"2024-04-21T08:42:17.542820+0000","last_active":"2024-04-21T08:42:22.664385+0000","last_peered":"2024-04-21T08:42:22.664385+0000","last_clean":"2024-04-21T08:42:22.664385+0000","last_became_active":"2024-04-21T08:42:17.541320+0000","last_became_peered":"2024-04-21T08:42:17.541320+0000","last_unstale":"2024-04-21T08:42:22.664385+0000","last_undegraded":"2024-04-21T08:42:22.664385+0000","last_fullsized":"2024-04-21T08:42:22.664385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:23:06.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664297+0000","last_change":"2024-04-21T08:42:16.636950+0000","last_active":"2024-04-21T08:42:22.664297+0000","last_peered":"2024-04-21T08:42:22.664297+0000","last_clean":"2024-04-21T08:42:22.664297+0000","last_became_active":"2024-04-21T08:42:16.636590+0000","last_became_peered":"2024-04-21T08:42:16.636590+0000","last_unstale":"2024-04-21T08:42:22.664297+0000","last_undegraded":"2024-04-21T08:42:22.664297+0000","last_fullsized":"2024-04-21T08:42:22.664297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:55.693681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.918266+0000","last_change":"2024-04-21T08:42:21.660404+0000","last_active":"2024-04-21T08:42:23.918266+0000","last_peered":"2024-04-21T08:42:23.918266+0000","last_clean":"2024-04-21T08:42:23.918266+0000","last_became_active":"2024-04-21T08:42:21.659999+0000","last_became_peered":"2024-04-21T08:42:21.659999+0000","last_unstale":"2024-04-21T08:42:23.918266+0000","last_undegraded":"2024-04-21T08:42:23.918266+0000","last_fullsized":"2024-04-21T08:42:23.918266+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:53:32.328873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091550+0000","last_change":"2024-04-21T08:42:17.548059+0000","last_active":"2024-04-21T08:42:23.091550+0000","last_peered":"2024-04-21T08:42:23.091550+0000","last_clean":"2024-04-21T08:42:23.091550+0000","last_became_active":"2024-04-21T08:42:17.547532+0000","last_became_peered":"2024-04-21T08:42:17.547532+0000","last_unstale":"2024-04-21T08:42:23.091550+0000","last_undegraded":"2024-04-21T08:42:23.091550+0000","last_fullsized":"2024-04-21T08:42:23.091550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:08.459103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091980+0000","last_change":"2024-04-21T08:42:16.638993+0000","last_active":"2024-04-21T08:42:23.091980+0000","last_peered":"2024-04-21T08:42:23.091980+0000","last_clean":"2024-04-21T08:42:23.091980+0000","last_became_active":"2024-04-21T08:42:16.638617+0000","last_became_peered":"2024-04-21T08:42:16.638617+0000","last_unstale":"2024-04-21T08:42:23.091980+0000","last_undegraded":"2024-04-21T08:42:23.091980+0000","last_fullsized":"2024-04-21T08:42:23.091980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:14:28.271433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":355,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090851+0000","last_change":"2024-04-21T08:41:00.044984+0000","last_active":"2024-04-21T08:42:23.090851+0000","last_peered":"2024-04-21T08:42:23.090851+0000","last_clean":"2024-04-21T08:42:23.090851+0000","last_became_active":"2024-04-21T08:41:00.044485+0000","last_became_peered":"2024-04-21T08:41:00.044485+0000","last_unstale":"2024-04-21T08:42:23.090851+0000","last_undegraded":"2024-04-21T08:42:23.090851+0000","last_fullsized":"2024-04-21T08:42:23.090851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.370031+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:01.031209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664715+0000","last_change":"2024-04-21T08:42:21.971045+0000","last_active":"2024-04-21T08:42:22.664715+0000","last_peered":"2024-04-21T08:42:22.664715+0000","last_clean":"2024-04-21T08:42:22.664715+0000","last_became_active":"2024-04-21T08:42:21.970577+0000","last_became_peered":"2024-04-21T08:42:21.970577+0000","last_unstale":"2024-04-21T08:42:22.664715+0000","last_undegraded":"2024-04-21T08:42:22.664715+0000","last_fullsized":"2024-04-21T08:42:22.664715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:55.681662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664297+0000","last_change":"2024-04-21T08:42:17.535570+0000","last_active":"2024-04-21T08:42:22.664297+0000","last_peered":"2024-04-21T08:42:22.664297+0000","last_clean":"2024-04-21T08:42:22.664297+0000","last_became_active":"2024-04-21T08:42:17.535217+0000","last_became_peered":"2024-04-21T08:42:17.535217+0000","last_unstale":"2024-04-21T08:42:22.664297+0000","last_undegraded":"2024-04-21T08:42:22.664297+0000","last_fullsized":"2024-04-21T08:42:22.664297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:49.510750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091021+0000","last_change":"2024-04-21T08:42:16.643794+0000","last_active":"2024-04-21T08:42:23.091021+0000","last_peered":"2024-04-21T08:42:23.091021+0000","last_clean":"2024-04-21T08:42:23.091021+0000","last_became_active":"2024-04-21T08:42:16.643357+0000","last_became_peered":"2024-04-21T08:42:16.643357+0000","last_unstale":"2024-04-21T08:42:23.091021+0000","last_undegraded":"2024-04-21T08:42:23.091021+0000","last_fullsized":"2024-04-21T08:42:23.091021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:12.778950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768274+0000","last_change":"2024-04-21T08:42:21.661393+0000","last_active":"2024-04-21T08:42:22.768274+0000","last_peered":"2024-04-21T08:42:22.768274+0000","last_clean":"2024-04-21T08:42:22.768274+0000","last_became_active":"2024-04-21T08:42:21.660201+0000","last_became_peered":"2024-04-21T08:42:21.660201+0000","last_unstale":"2024-04-21T08:42:22.768274+0000","last_undegraded":"2024-04-21T08:42:22.768274+0000","last_fullsized":"2024-04-21T08:42:22.768274+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:13.393389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951753+0000","last_change":"2024-04-21T08:42:16.650507+0000","last_active":"2024-04-21T08:42:22.951753+0000","last_peered":"2024-04-21T08:42:22.951753+0000","last_clean":"2024-04-21T08:42:22.951753+0000","last_became_active":"2024-04-21T08:42:16.650217+0000","last_became_peered":"2024-04-21T08:42:16.650217+0000","last_unstale":"2024-04-21T08:42:22.951753+0000","last_undegraded":"2024-04-21T08:42:22.951753+0000","last_fullsized":"2024-04-21T08:42:22.951753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:13:26.785092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664946+0000","last_change":"2024-04-21T08:42:17.530420+0000","last_active":"2024-04-21T08:42:22.664946+0000","last_peered":"2024-04-21T08:42:22.664946+0000","last_clean":"2024-04-21T08:42:22.664946+0000","last_became_active":"2024-04-21T08:42:17.529920+0000","last_became_peered":"2024-04-21T08:42:17.529920+0000","last_unstale":"2024-04-21T08:42:22.664946+0000","last_undegraded":"2024-04-21T08:42:22.664946+0000","last_fullsized":"2024-04-21T08:42:22.664946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:24.250197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090926+0000","last_change":"2024-04-21T08:42:21.970586+0000","last_active":"2024-04-21T08:42:23.090926+0000","last_peered":"2024-04-21T08:42:23.090926+0000","last_clean":"2024-04-21T08:42:23.090926+0000","last_became_active":"2024-04-21T08:42:21.970132+0000","last_became_peered":"2024-04-21T08:42:21.970132+0000","last_unstale":"2024-04-21T08:42:23.090926+0000","last_undegraded":"2024-04-21T08:42:23.090926+0000","last_fullsized":"2024-04-21T08:42:23.090926+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:11.074370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680514+0000","last_change":"2024-04-21T08:42:16.647916+0000","last_active":"2024-04-21T08:42:22.680514+0000","last_peered":"2024-04-21T08:42:22.680514+0000","last_clean":"2024-04-21T08:42:22.680514+0000","last_became_active":"2024-04-21T08:42:16.647476+0000","last_became_peered":"2024-04-21T08:42:16.647476+0000","last_unstale":"2024-04-21T08:42:22.680514+0000","last_undegraded":"2024-04-21T08:42:22.680514+0000","last_fullsized":"2024-04-21T08:42:22.680514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:27.242153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092078+0000","last_change":"2024-04-21T08:42:17.546144+0000","last_active":"2024-04-21T08:42:23.092078+0000","last_peered":"2024-04-21T08:42:23.092078+0000","last_clean":"2024-04-21T08:42:23.092078+0000","last_became_active":"2024-04-21T08:42:17.545730+0000","last_became_peered":"2024-04-21T08:42:17.545730+0000","last_unstale":"2024-04-21T08:42:23.092078+0000","last_undegraded":"2024-04-21T08:42:23.092078+0000","last_fullsized":"2024-04-21T08:42:23.092078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:42.802309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090217+0000","last_change":"2024-04-21T08:42:21.655802+0000","last_active":"2024-04-21T08:42:23.090217+0000","last_peered":"2024-04-21T08:42:23.090217+0000","last_clean":"2024-04-21T08:42:23.090217+0000","last_became_active":"2024-04-21T08:42:21.655379+0000","last_became_peered":"2024-04-21T08:42:21.655379+0000","last_unstale":"2024-04-21T08:42:23.090217+0000","last_undegraded":"2024-04-21T08:42:23.090217+0000","last_fullsized":"2024-04-21T08:42:23.090217+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:49.486057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091831+0000","last_change":"2024-04-21T08:42:16.642517+0000","last_active":"2024-04-21T08:42:23.091831+0000","last_peered":"2024-04-21T08:42:23.091831+0000","last_clean":"2024-04-21T08:42:23.091831+0000","last_became_active":"2024-04-21T08:42:16.641539+0000","last_became_peered":"2024-04-21T08:42:16.641539+0000","last_unstale":"2024-04-21T08:42:23.091831+0000","last_undegraded":"2024-04-21T08:42:23.091831+0000","last_fullsized":"2024-04-21T08:42:23.091831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:53.912199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091797+0000","last_change":"2024-04-21T08:42:17.550172+0000","last_active":"2024-04-21T08:42:23.091797+0000","last_peered":"2024-04-21T08:42:23.091797+0000","last_clean":"2024-04-21T08:42:23.091797+0000","last_became_active":"2024-04-21T08:42:17.549824+0000","last_became_peered":"2024-04-21T08:42:17.549824+0000","last_unstale":"2024-04-21T08:42:23.091797+0000","last_undegraded":"2024-04-21T08:42:23.091797+0000","last_fullsized":"2024-04-21T08:42:23.091797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:59.553645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092068+0000","last_change":"2024-04-21T08:42:21.970654+0000","last_active":"2024-04-21T08:42:23.092068+0000","last_peered":"2024-04-21T08:42:23.092068+0000","last_clean":"2024-04-21T08:42:23.092068+0000","last_became_active":"2024-04-21T08:42:21.970211+0000","last_became_peered":"2024-04-21T08:42:21.970211+0000","last_unstale":"2024-04-21T08:42:23.092068+0000","last_undegraded":"2024-04-21T08:42:23.092068+0000","last_fullsized":"2024-04-21T08:42:23.092068+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:24:51.406125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664251+0000","last_change":"2024-04-21T08:42:16.647985+0000","last_active":"2024-04-21T08:42:22.664251+0000","last_peered":"2024-04-21T08:42:22.664251+0000","last_clean":"2024-04-21T08:42:22.664251+0000","last_became_active":"2024-04-21T08:42:16.647577+0000","last_became_peered":"2024-04-21T08:42:16.647577+0000","last_unstale":"2024-04-21T08:42:22.664251+0000","last_undegraded":"2024-04-21T08:42:22.664251+0000","last_fullsized":"2024-04-21T08:42:22.664251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:00.731708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951434+0000","last_change":"2024-04-21T08:42:17.541757+0000","last_active":"2024-04-21T08:42:22.951434+0000","last_peered":"2024-04-21T08:42:22.951434+0000","last_clean":"2024-04-21T08:42:22.951434+0000","last_became_active":"2024-04-21T08:42:17.539280+0000","last_became_peered":"2024-04-21T08:42:17.539280+0000","last_unstale":"2024-04-21T08:42:22.951434+0000","last_undegraded":"2024-04-21T08:42:22.951434+0000","last_fullsized":"2024-04-21T08:42:22.951434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:45:54.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.680519+0000","last_change":"2024-04-21T08:42:21.664566+0000","last_active":"2024-04-21T08:42:22.680519+0000","last_peered":"2024-04-21T08:42:22.680519+0000","last_clean":"2024-04-21T08:42:22.680519+0000","last_became_active":"2024-04-21T08:42:21.664136+0000","last_became_peered":"2024-04-21T08:42:21.664136+0000","last_unstale":"2024-04-21T08:42:22.680519+0000","last_undegraded":"2024-04-21T08:42:22.680519+0000","last_fullsized":"2024-04-21T08:42:22.680519+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:19.856577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091318+0000","last_change":"2024-04-21T08:42:16.650664+0000","last_active":"2024-04-21T08:42:23.091318+0000","last_peered":"2024-04-21T08:42:23.091318+0000","last_clean":"2024-04-21T08:42:23.091318+0000","last_became_active":"2024-04-21T08:42:16.650339+0000","last_became_peered":"2024-04-21T08:42:16.650339+0000","last_unstale":"2024-04-21T08:42:23.091318+0000","last_undegraded":"2024-04-21T08:42:23.091318+0000","last_fullsized":"2024-04-21T08:42:23.091318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:59.565455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768020+0000","last_change":"2024-04-21T08:42:17.538142+0000","last_active":"2024-04-21T08:42:22.768020+0000","last_peered":"2024-04-21T08:42:22.768020+0000","last_clean":"2024-04-21T08:42:22.768020+0000","last_became_active":"2024-04-21T08:42:17.537793+0000","last_became_peered":"2024-04-21T08:42:17.537793+0000","last_unstale":"2024-04-21T08:42:22.768020+0000","last_undegraded":"2024-04-21T08:42:22.768020+0000","last_fullsized":"2024-04-21T08:42:22.768020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:11.059294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092051+0000","last_change":"2024-04-21T08:42:21.670642+0000","last_active":"2024-04-21T08:42:23.092051+0000","last_peered":"2024-04-21T08:42:23.092051+0000","last_clean":"2024-04-21T08:42:23.092051+0000","last_became_active":"2024-04-21T08:42:21.670128+0000","last_became_peered":"2024-04-21T08:42:21.670128+0000","last_unstale":"2024-04-21T08:42:23.092051+0000","last_undegraded":"2024-04-21T08:42:23.092051+0000","last_fullsized":"2024-04-21T08:42:23.092051+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:51:28.604381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090228+0000","last_change":"2024-04-21T08:42:16.642471+0000","last_active":"2024-04-21T08:42:23.090228+0000","last_peered":"2024-04-21T08:42:23.090228+0000","last_clean":"2024-04-21T08:42:23.090228+0000","last_became_active":"2024-04-21T08:42:16.642142+0000","last_became_peered":"2024-04-21T08:42:16.642142+0000","last_unstale":"2024-04-21T08:42:23.090228+0000","last_undegraded":"2024-04-21T08:42:23.090228+0000","last_fullsized":"2024-04-21T08:42:23.090228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:17.847374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091690+0000","last_change":"2024-04-21T08:42:17.547326+0000","last_active":"2024-04-21T08:42:23.091690+0000","last_peered":"2024-04-21T08:42:23.091690+0000","last_clean":"2024-04-21T08:42:23.091690+0000","last_became_active":"2024-04-21T08:42:17.546920+0000","last_became_peered":"2024-04-21T08:42:17.546920+0000","last_unstale":"2024-04-21T08:42:23.091690+0000","last_undegraded":"2024-04-21T08:42:23.091690+0000","last_fullsized":"2024-04-21T08:42:23.091690+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:43.350877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092100+0000","last_change":"2024-04-21T08:42:21.971147+0000","last_active":"2024-04-21T08:42:23.092100+0000","last_peered":"2024-04-21T08:42:23.092100+0000","last_clean":"2024-04-21T08:42:23.092100+0000","last_became_active":"2024-04-21T08:42:21.970001+0000","last_became_peered":"2024-04-21T08:42:21.970001+0000","last_unstale":"2024-04-21T08:42:23.092100+0000","last_undegraded":"2024-04-21T08:42:23.092100+0000","last_fullsized":"2024-04-21T08:42:23.092100+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:01:44.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.951582+0000","last_change":"2024-04-21T08:42:16.643371+0000","last_active":"2024-04-21T08:42:22.951582+0000","last_peered":"2024-04-21T08:42:22.951582+0000","last_clean":"2024-04-21T08:42:22.951582+0000","last_became_active":"2024-04-21T08:42:16.642906+0000","last_became_peered":"2024-04-21T08:42:16.642906+0000","last_unstale":"2024-04-21T08:42:22.951582+0000","last_undegraded":"2024-04-21T08:42:22.951582+0000","last_fullsized":"2024-04-21T08:42:22.951582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:58.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091533+0000","last_change":"2024-04-21T08:42:17.545185+0000","last_active":"2024-04-21T08:42:23.091533+0000","last_peered":"2024-04-21T08:42:23.091533+0000","last_clean":"2024-04-21T08:42:23.091533+0000","last_became_active":"2024-04-21T08:42:17.544757+0000","last_became_peered":"2024-04-21T08:42:17.544757+0000","last_unstale":"2024-04-21T08:42:23.091533+0000","last_undegraded":"2024-04-21T08:42:23.091533+0000","last_fullsized":"2024-04-21T08:42:23.091533+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:36:18.968041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664466+0000","last_change":"2024-04-21T08:42:21.662541+0000","last_active":"2024-04-21T08:42:22.664466+0000","last_peered":"2024-04-21T08:42:22.664466+0000","last_clean":"2024-04-21T08:42:22.664466+0000","last_became_active":"2024-04-21T08:42:21.662144+0000","last_became_peered":"2024-04-21T08:42:21.662144+0000","last_unstale":"2024-04-21T08:42:22.664466+0000","last_undegraded":"2024-04-21T08:42:22.664466+0000","last_fullsized":"2024-04-21T08:42:22.664466+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:04.347722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090684+0000","last_change":"2024-04-21T08:42:16.638601+0000","last_active":"2024-04-21T08:42:23.090684+0000","last_peered":"2024-04-21T08:42:23.090684+0000","last_clean":"2024-04-21T08:42:23.090684+0000","last_became_active":"2024-04-21T08:42:16.638211+0000","last_became_peered":"2024-04-21T08:42:16.638211+0000","last_unstale":"2024-04-21T08:42:23.090684+0000","last_undegraded":"2024-04-21T08:42:23.090684+0000","last_fullsized":"2024-04-21T08:42:23.090684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:02.633086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091786+0000","last_change":"2024-04-21T08:42:17.545495+0000","last_active":"2024-04-21T08:42:23.091786+0000","last_peered":"2024-04-21T08:42:23.091786+0000","last_clean":"2024-04-21T08:42:23.091786+0000","last_became_active":"2024-04-21T08:42:17.545159+0000","last_became_peered":"2024-04-21T08:42:17.545159+0000","last_unstale":"2024-04-21T08:42:23.091786+0000","last_undegraded":"2024-04-21T08:42:23.091786+0000","last_fullsized":"2024-04-21T08:42:23.091786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:27.150277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091243+0000","last_change":"2024-04-21T08:42:21.656890+0000","last_active":"2024-04-21T08:42:23.091243+0000","last_peered":"2024-04-21T08:42:23.091243+0000","last_clean":"2024-04-21T08:42:23.091243+0000","last_became_active":"2024-04-21T08:42:21.656475+0000","last_became_peered":"2024-04-21T08:42:21.656475+0000","last_unstale":"2024-04-21T08:42:23.091243+0000","last_undegraded":"2024-04-21T08:42:23.091243+0000","last_fullsized":"2024-04-21T08:42:23.091243+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:46.138591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664449+0000","last_change":"2024-04-21T08:42:16.638244+0000","last_active":"2024-04-21T08:42:22.664449+0000","last_peered":"2024-04-21T08:42:22.664449+0000","last_clean":"2024-04-21T08:42:22.664449+0000","last_became_active":"2024-04-21T08:42:16.637883+0000","last_became_peered":"2024-04-21T08:42:16.637883+0000","last_unstale":"2024-04-21T08:42:22.664449+0000","last_undegraded":"2024-04-21T08:42:22.664449+0000","last_fullsized":"2024-04-21T08:42:22.664449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:20.888301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091372+0000","last_change":"2024-04-21T08:42:17.548032+0000","last_active":"2024-04-21T08:42:23.091372+0000","last_peered":"2024-04-21T08:42:23.091372+0000","last_clean":"2024-04-21T08:42:23.091372+0000","last_became_active":"2024-04-21T08:42:17.547660+0000","last_became_peered":"2024-04-21T08:42:17.547660+0000","last_unstale":"2024-04-21T08:42:23.091372+0000","last_undegraded":"2024-04-21T08:42:23.091372+0000","last_fullsized":"2024-04-21T08:42:23.091372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:14.710759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.663796+0000","last_change":"2024-04-21T08:42:21.659286+0000","last_active":"2024-04-21T08:42:22.663796+0000","last_peered":"2024-04-21T08:42:22.663796+0000","last_clean":"2024-04-21T08:42:22.663796+0000","last_became_active":"2024-04-21T08:42:21.658844+0000","last_became_peered":"2024-04-21T08:42:21.658844+0000","last_unstale":"2024-04-21T08:42:22.663796+0000","last_undegraded":"2024-04-21T08:42:22.663796+0000","last_fullsized":"2024-04-21T08:42:22.663796+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:28.742646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.665047+0000","last_change":"2024-04-21T08:42:16.643354+0000","last_active":"2024-04-21T08:42:22.665047+0000","last_peered":"2024-04-21T08:42:22.665047+0000","last_clean":"2024-04-21T08:42:22.665047+0000","last_became_active":"2024-04-21T08:42:16.642967+0000","last_became_peered":"2024-04-21T08:42:16.642967+0000","last_unstale":"2024-04-21T08:42:22.665047+0000","last_undegraded":"2024-04-21T08:42:22.665047+0000","last_fullsized":"2024-04-21T08:42:22.665047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:06:03.495181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091938+0000","last_change":"2024-04-21T08:42:17.539551+0000","last_active":"2024-04-21T08:42:23.091938+0000","last_peered":"2024-04-21T08:42:23.091938+0000","last_clean":"2024-04-21T08:42:23.091938+0000","last_became_active":"2024-04-21T08:42:17.539202+0000","last_became_peered":"2024-04-21T08:42:17.539202+0000","last_unstale":"2024-04-21T08:42:23.091938+0000","last_undegraded":"2024-04-21T08:42:23.091938+0000","last_fullsized":"2024-04-21T08:42:23.091938+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:37.109421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768071+0000","last_change":"2024-04-21T08:42:17.544582+0000","last_active":"2024-04-21T08:42:22.768071+0000","last_peered":"2024-04-21T08:42:22.768071+0000","last_clean":"2024-04-21T08:42:22.768071+0000","last_became_active":"2024-04-21T08:42:17.543800+0000","last_became_peered":"2024-04-21T08:42:17.543800+0000","last_unstale":"2024-04-21T08:42:22.768071+0000","last_undegraded":"2024-04-21T08:42:22.768071+0000","last_fullsized":"2024-04-21T08:42:22.768071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:44:04.107207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090298+0000","last_change":"2024-04-21T08:42:17.543788+0000","last_active":"2024-04-21T08:42:23.090298+0000","last_peered":"2024-04-21T08:42:23.090298+0000","last_clean":"2024-04-21T08:42:23.090298+0000","last_became_active":"2024-04-21T08:42:17.540939+0000","last_became_peered":"2024-04-21T08:42:17.540939+0000","last_unstale":"2024-04-21T08:42:23.090298+0000","last_undegraded":"2024-04-21T08:42:23.090298+0000","last_fullsized":"2024-04-21T08:42:23.090298+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:59.275098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090192+0000","last_change":"2024-04-21T08:42:17.540605+0000","last_active":"2024-04-21T08:42:23.090192+0000","last_peered":"2024-04-21T08:42:23.090192+0000","last_clean":"2024-04-21T08:42:23.090192+0000","last_became_active":"2024-04-21T08:42:17.540073+0000","last_became_peered":"2024-04-21T08:42:17.540073+0000","last_unstale":"2024-04-21T08:42:23.090192+0000","last_undegraded":"2024-04-21T08:42:23.090192+0000","last_fullsized":"2024-04-21T08:42:23.090192+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:02:29.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091491+0000","last_change":"2024-04-21T08:42:17.551608+0000","last_active":"2024-04-21T08:42:23.091491+0000","last_peered":"2024-04-21T08:42:23.091491+0000","last_clean":"2024-04-21T08:42:23.091491+0000","last_became_active":"2024-04-21T08:42:17.551117+0000","last_became_peered":"2024-04-21T08:42:17.551117+0000","last_unstale":"2024-04-21T08:42:23.091491+0000","last_undegraded":"2024-04-21T08:42:23.091491+0000","last_fullsized":"2024-04-21T08:42:23.091491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:18.391928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664818+0000","last_change":"2024-04-21T08:42:17.536509+0000","last_active":"2024-04-21T08:42:22.664818+0000","last_peered":"2024-04-21T08:42:22.664818+0000","last_clean":"2024-04-21T08:42:22.664818+0000","last_became_active":"2024-04-21T08:42:17.536146+0000","last_became_peered":"2024-04-21T08:42:17.536146+0000","last_unstale":"2024-04-21T08:42:22.664818+0000","last_undegraded":"2024-04-21T08:42:22.664818+0000","last_fullsized":"2024-04-21T08:42:22.664818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:29.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768483+0000","last_change":"2024-04-21T08:42:17.543786+0000","last_active":"2024-04-21T08:42:22.768483+0000","last_peered":"2024-04-21T08:42:22.768483+0000","last_clean":"2024-04-21T08:42:22.768483+0000","last_became_active":"2024-04-21T08:42:17.543435+0000","last_became_peered":"2024-04-21T08:42:17.543435+0000","last_unstale":"2024-04-21T08:42:22.768483+0000","last_undegraded":"2024-04-21T08:42:22.768483+0000","last_fullsized":"2024-04-21T08:42:22.768483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.090632+0000","last_change":"2024-04-21T08:42:17.541611+0000","last_active":"2024-04-21T08:42:23.090632+0000","last_peered":"2024-04-21T08:42:23.090632+0000","last_clean":"2024-04-21T08:42:23.090632+0000","last_became_active":"2024-04-21T08:42:17.541259+0000","last_became_peered":"2024-04-21T08:42:17.541259+0000","last_unstale":"2024-04-21T08:42:23.090632+0000","last_undegraded":"2024-04-21T08:42:23.090632+0000","last_fullsized":"2024-04-21T08:42:23.090632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:20.900629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.092259+0000","last_change":"2024-04-21T08:42:17.541964+0000","last_active":"2024-04-21T08:42:23.092259+0000","last_peered":"2024-04-21T08:42:23.092259+0000","last_clean":"2024-04-21T08:42:23.092259+0000","last_became_active":"2024-04-21T08:42:17.541555+0000","last_became_peered":"2024-04-21T08:42:17.541555+0000","last_unstale":"2024-04-21T08:42:23.092259+0000","last_undegraded":"2024-04-21T08:42:23.092259+0000","last_fullsized":"2024-04-21T08:42:23.092259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:32:47.421452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664282+0000","last_change":"2024-04-21T08:42:17.545526+0000","last_active":"2024-04-21T08:42:22.664282+0000","last_peered":"2024-04-21T08:42:22.664282+0000","last_clean":"2024-04-21T08:42:22.664282+0000","last_became_active":"2024-04-21T08:42:17.545135+0000","last_became_peered":"2024-04-21T08:42:17.545135+0000","last_unstale":"2024-04-21T08:42:22.664282+0000","last_undegraded":"2024-04-21T08:42:22.664282+0000","last_fullsized":"2024-04-21T08:42:22.664282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:37:58.038459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:23.091426+0000","last_change":"2024-04-21T08:42:17.547063+0000","last_active":"2024-04-21T08:42:23.091426+0000","last_peered":"2024-04-21T08:42:23.091426+0000","last_clean":"2024-04-21T08:42:23.091426+0000","last_became_active":"2024-04-21T08:42:17.546689+0000","last_became_peered":"2024-04-21T08:42:17.546689+0000","last_unstale":"2024-04-21T08:42:23.091426+0000","last_undegraded":"2024-04-21T08:42:23.091426+0000","last_fullsized":"2024-04-21T08:42:23.091426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:26.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664551+0000","last_change":"2024-04-21T08:42:17.542545+0000","last_active":"2024-04-21T08:42:22.664551+0000","last_peered":"2024-04-21T08:42:22.664551+0000","last_clean":"2024-04-21T08:42:22.664551+0000","last_became_active":"2024-04-21T08:42:17.542147+0000","last_became_peered":"2024-04-21T08:42:17.542147+0000","last_unstale":"2024-04-21T08:42:22.664551+0000","last_undegraded":"2024-04-21T08:42:22.664551+0000","last_fullsized":"2024-04-21T08:42:22.664551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:15.064110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.664687+0000","last_change":"2024-04-21T08:42:17.541262+0000","last_active":"2024-04-21T08:42:22.664687+0000","last_peered":"2024-04-21T08:42:22.664687+0000","last_clean":"2024-04-21T08:42:22.664687+0000","last_became_active":"2024-04-21T08:42:17.540817+0000","last_became_peered":"2024-04-21T08:42:17.540817+0000","last_unstale":"2024-04-21T08:42:22.664687+0000","last_undegraded":"2024-04-21T08:42:22.664687+0000","last_fullsized":"2024-04-21T08:42:22.664687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:22:59.597680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T08:42:22.768318+0000","last_change":"2024-04-21T08:42:17.551109+0000","last_active":"2024-04-21T08:42:22.768318+0000","last_peered":"2024-04-21T08:42:22.768318+0000","last_clean":"2024-04-21T08:42:22.768318+0000","last_became_active":"2024-04-21T08:42:17.550104+0000","last_became_peered":"2024-04-21T08:42:17.550104+0000","last_unstale":"2024-04-21T08:42:22.768318+0000","last_undegraded":"2024-04-21T08:42:22.768318+0000","last_fullsized":"2024-04-21T08:42:22.768318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:22:19.361752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281211,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":33219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350995,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":484362,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612631,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":1074670,"data_compressed":50511,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776157,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":974848,"data_stored":480848,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037795,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":484966,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299431,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1028096,"data_stored":1075312,"data_compressed":50511,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462957,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":970752,"data_stored":480266,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":490708,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":974848,"data_stored":480356,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":484142,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":480370,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888119,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":974848,"data_stored":480288,"data_compressed":14745,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724593,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":1070816,"data_compressed":50511,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:42:45.927 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-21T08:42:45.927 INFO:tasks.cephadm:Setting up client nodes... 2024-04-21T08:42:45.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-21T08:42:46.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:45 smithi016 ceph-mon[31259]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:46.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:45 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/345078780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:46.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:45 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/247816535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:46.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:45 smithi077 ceph-mon[31615]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:45 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/345078780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:45 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/247816535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:46.208 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:46.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:46 smithi007 ceph-mon[28477]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:46.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:46 smithi007 ceph-mon[28477]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 255 B/s wr, 0 op/s 2024-04-21T08:42:46.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:46 smithi007 ceph-mon[28477]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:46 smithi016 ceph-mon[31259]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:46 smithi016 ceph-mon[31259]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 255 B/s wr, 0 op/s 2024-04-21T08:42:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:46 smithi016 ceph-mon[31259]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:47.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:46 smithi077 ceph-mon[31615]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:46 smithi077 ceph-mon[31615]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 255 B/s wr, 0 op/s 2024-04-21T08:42:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:46 smithi077 ceph-mon[31615]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.000+0000 7f9bb64ba700 1 -- 172.21.15.7:0/3409066784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bb0104690 msgr2=0x7f9bb010af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.000+0000 7f9bb64ba700 1 --2- 172.21.15.7:0/3409066784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bb0104690 0x7f9bb010af40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800b0d0 tx=0x7f9ba800b3e0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.000+0000 7f9bb64ba700 1 -- 172.21.15.7:0/3409066784 shutdown_connections 2024-04-21T08:42:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.000+0000 7f9bb64ba700 1 --2- 172.21.15.7:0/3409066784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bb0102af0 0x7f9bb0102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.000+0000 7f9bb64ba700 1 --2- 172.21.15.7:0/3409066784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bb0104690 0x7f9bb010af40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.000+0000 7f9bb64ba700 1 --2- 172.21.15.7:0/3409066784 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0103cf0 0x7f9bb0104150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.000+0000 7f9bb64ba700 1 -- 172.21.15.7:0/3409066784 >> 172.21.15.7:0/3409066784 conn(0x7f9bb00fe090 msgr2=0x7f9bb01004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.000+0000 7f9bb64ba700 1 -- 172.21.15.7:0/3409066784 shutdown_connections 2024-04-21T08:42:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.001+0000 7f9bb64ba700 1 -- 172.21.15.7:0/3409066784 wait complete. 2024-04-21T08:42:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.001+0000 7f9bb64ba700 1 Processor -- start 2024-04-21T08:42:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.001+0000 7f9bb64ba700 1 -- start start 2024-04-21T08:42:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.001+0000 7f9bb64ba700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bb0102af0 0x7f9bb00767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.001+0000 7f9bb64ba700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0103cf0 0x7f9bb0074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.001+0000 7f9bb64ba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bb0104690 0x7f9bb0075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.001+0000 7f9bb64ba700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb010e600 con 0x7f9bb0104690 2024-04-21T08:42:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.001+0000 7f9bb64ba700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb010e480 con 0x7f9bb0103cf0 2024-04-21T08:42:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.001+0000 7f9bb64ba700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb010e780 con 0x7f9bb0102af0 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bb4a57700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bb0104690 0x7f9bb0075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bb4a57700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bb0104690 0x7f9bb0075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50302/0 (socket says 172.21.15.7:50302) 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bb4a57700 1 -- 172.21.15.7:0/1425635302 learned_addr learned my addr 172.21.15.7:0/1425635302 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9baf7fe700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0103cf0 0x7f9bb0074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9baffff700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bb0102af0 0x7f9bb00767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bb4a57700 1 -- 172.21.15.7:0/1425635302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bb0102af0 msgr2=0x7f9bb00767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bb4a57700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bb0102af0 0x7f9bb00767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bb4a57700 1 -- 172.21.15.7:0/1425635302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0103cf0 msgr2=0x7f9bb0074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bb4a57700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0103cf0 0x7f9bb0074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bb4a57700 1 -- 172.21.15.7:0/1425635302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb0075c20 con 0x7f9bb0104690 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9baffff700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bb0102af0 0x7f9bb00767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bb4a57700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bb0104690 0x7f9bb0075370 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009040 tx=0x7f9ba8004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bad7fa700 1 -- 172.21.15.7:0/1425635302 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba801f030 con 0x7f9bb0104690 2024-04-21T08:42:48.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bb64ba700 1 -- 172.21.15.7:0/1425635302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb01a2c80 con 0x7f9bb0104690 2024-04-21T08:42:48.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bad7fa700 1 -- 172.21.15.7:0/1425635302 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ba8003a00 con 0x7f9bb0104690 2024-04-21T08:42:48.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bad7fa700 1 -- 172.21.15.7:0/1425635302 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba80047c0 con 0x7f9bb0104690 2024-04-21T08:42:48.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.002+0000 7f9bb64ba700 1 -- 172.21.15.7:0/1425635302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb01a30d0 con 0x7f9bb0104690 2024-04-21T08:42:48.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.004+0000 7f9bb64ba700 1 -- 172.21.15.7:0/1425635302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bb002df80 con 0x7f9bb0104690 2024-04-21T08:42:48.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.005+0000 7f9bad7fa700 1 -- 172.21.15.7:0/1425635302 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9ba8003bb0 con 0x7f9bb0104690 2024-04-21T08:42:48.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.005+0000 7f9bad7fa700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b980a3d80 0x7f9b980a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.005+0000 7f9baffff700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b980a3d80 0x7f9b980a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.005+0000 7f9bad7fa700 1 -- 172.21.15.7:0/1425635302 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f9ba800f020 con 0x7f9bb0104690 2024-04-21T08:42:48.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.006+0000 7f9baffff700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b980a3d80 0x7f9b980a6240 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0103b50 tx=0x7f9ba0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.010+0000 7f9bad7fa700 1 -- 172.21.15.7:0/1425635302 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ba8029020 con 0x7f9bb0104690 2024-04-21T08:42:48.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.309+0000 7f9bb64ba700 1 -- 172.21.15.7:0/1425635302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f9bb002e150 con 0x7f9bb0104690 2024-04-21T08:42:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.311+0000 7f9bad7fa700 1 -- 172.21.15.7:0/1425635302 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f9ba809e140 con 0x7f9bb0104690 2024-04-21T08:42:48.311 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-04-21T08:42:48.311 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQCI0SRm+LKEEhAAWmALtIdMjQYqhHmJa36K3g== 2024-04-21T08:42:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.313+0000 7f9bb64ba700 1 -- 172.21.15.7:0/1425635302 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b980a3d80 msgr2=0x7f9b980a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.313+0000 7f9bb64ba700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b980a3d80 0x7f9b980a6240 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0103b50 tx=0x7f9ba0008040 comp rx=0 tx=0).stop 2024-04-21T08:42:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.313+0000 7f9bb64ba700 1 -- 172.21.15.7:0/1425635302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bb0104690 msgr2=0x7f9bb0075370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.313+0000 7f9bb64ba700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bb0104690 0x7f9bb0075370 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9ba8009040 tx=0x7f9ba8004d50 comp rx=0 tx=0).stop 2024-04-21T08:42:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.314+0000 7f9bb64ba700 1 -- 172.21.15.7:0/1425635302 shutdown_connections 2024-04-21T08:42:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.314+0000 7f9bb64ba700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b980a3d80 0x7f9b980a6240 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.314+0000 7f9bb64ba700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bb0102af0 0x7f9bb00767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.314+0000 7f9bb64ba700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bb0104690 0x7f9bb0075370 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.314+0000 7f9bb64ba700 1 --2- 172.21.15.7:0/1425635302 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0103cf0 0x7f9bb0074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.314+0000 7f9bb64ba700 1 -- 172.21.15.7:0/1425635302 >> 172.21.15.7:0/1425635302 conn(0x7f9bb00fe090 msgr2=0x7f9bb0106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.314+0000 7f9bb64ba700 1 -- 172.21.15.7:0/1425635302 shutdown_connections 2024-04-21T08:42:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:48.314+0000 7f9bb64ba700 1 -- 172.21.15.7:0/1425635302 wait complete. 2024-04-21T08:42:49.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:48 smithi016 ceph-mon[31259]: pgmap v309: 129 pgs: 129 active+clean; 622 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-04-21T08:42:49.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:48 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1425635302' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:42:49.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:48 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1425635302' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:42:49.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:48 smithi077 ceph-mon[31615]: pgmap v309: 129 pgs: 129 active+clean; 622 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-04-21T08:42:49.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:48 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1425635302' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:42:49.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:48 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1425635302' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:42:49.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:48 smithi007 ceph-mon[28477]: pgmap v309: 129 pgs: 129 active+clean; 622 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-04-21T08:42:49.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:48 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1425635302' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:42:49.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:48 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1425635302' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:42:49.324 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:42:49.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-21T08:42:49.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-21T08:42:49.366 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-21T08:42:49.366 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-21T08:42:49.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-21T08:42:49.661 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:51.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:50 smithi007 ceph-mon[28477]: pgmap v310: 129 pgs: 129 active+clean; 622 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-04-21T08:42:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:50 smithi016 ceph-mon[31259]: pgmap v310: 129 pgs: 129 active+clean; 622 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-04-21T08:42:51.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:50 smithi077 ceph-mon[31615]: pgmap v310: 129 pgs: 129 active+clean; 622 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-04-21T08:42:51.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.443+0000 7f2628eec700 1 -- 172.21.15.7:0/1282693733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2624108a30 msgr2=0x7f2624108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.443+0000 7f2628eec700 1 --2- 172.21.15.7:0/1282693733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2624108a30 0x7f2624108e10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2614009990 tx=0x7f2614009ca0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.443+0000 7f2628eec700 1 -- 172.21.15.7:0/1282693733 shutdown_connections 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.443+0000 7f2628eec700 1 --2- 172.21.15.7:0/1282693733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f262406b0f0 0x7f262410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.443+0000 7f2628eec700 1 --2- 172.21.15.7:0/1282693733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2624108a30 0x7f2624108e10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.443+0000 7f2628eec700 1 --2- 172.21.15.7:0/1282693733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f262406a750 0x7f262406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.443+0000 7f2628eec700 1 -- 172.21.15.7:0/1282693733 >> 172.21.15.7:0/1282693733 conn(0x7f26240788f0 msgr2=0x7f2624078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.443+0000 7f2628eec700 1 -- 172.21.15.7:0/1282693733 shutdown_connections 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.443+0000 7f2628eec700 1 -- 172.21.15.7:0/1282693733 wait complete. 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2628eec700 1 Processor -- start 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2628eec700 1 -- start start 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2628eec700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f262406a750 0x7f2624101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2628eec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f262406b0f0 0x7f2624101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2628eec700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2624108a30 0x7f2624102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2628eec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2624112bb0 con 0x7f262406b0f0 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2628eec700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2624112a30 con 0x7f2624108a30 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2628eec700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2624112d30 con 0x7f262406a750 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2621d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f262406b0f0 0x7f2624101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2621d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f262406b0f0 0x7f2624101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50322/0 (socket says 172.21.15.7:50322) 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2621d9b700 1 -- 172.21.15.7:0/2387524366 learned_addr learned my addr 172.21.15.7:0/2387524366 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f262259c700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f262406a750 0x7f2624101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.444+0000 7f2622d9d700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2624108a30 0x7f2624102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.445+0000 7f2621d9b700 1 -- 172.21.15.7:0/2387524366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f262406a750 msgr2=0x7f2624101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.445+0000 7f2621d9b700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f262406a750 0x7f2624101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.445+0000 7f2621d9b700 1 -- 172.21.15.7:0/2387524366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2624108a30 msgr2=0x7f2624102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.445+0000 7f2621d9b700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2624108a30 0x7f2624102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.445+0000 7f2621d9b700 1 -- 172.21.15.7:0/2387524366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26241a7630 con 0x7f262406b0f0 2024-04-21T08:42:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.445+0000 7f2621d9b700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f262406b0f0 0x7f2624101f50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f261800c920 tx=0x7f261800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.445+0000 7f26137fe700 1 -- 172.21.15.7:0/2387524366 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2618004a30 con 0x7f262406b0f0 2024-04-21T08:42:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.445+0000 7f26137fe700 1 -- 172.21.15.7:0/2387524366 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26180054d0 con 0x7f262406b0f0 2024-04-21T08:42:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.445+0000 7f2628eec700 1 -- 172.21.15.7:0/2387524366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26241a7920 con 0x7f262406b0f0 2024-04-21T08:42:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.445+0000 7f26137fe700 1 -- 172.21.15.7:0/2387524366 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2618007c70 con 0x7f262406b0f0 2024-04-21T08:42:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.445+0000 7f2628eec700 1 -- 172.21.15.7:0/2387524366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26241a7e60 con 0x7f262406b0f0 2024-04-21T08:42:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.446+0000 7f2628eec700 1 -- 172.21.15.7:0/2387524366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2624069730 con 0x7f262406b0f0 2024-04-21T08:42:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.447+0000 7f26137fe700 1 -- 172.21.15.7:0/2387524366 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f2618004c60 con 0x7f262406b0f0 2024-04-21T08:42:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.448+0000 7f26137fe700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f260c0a3cd0 0x7f260c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.448+0000 7f262259c700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f260c0a3cd0 0x7f260c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.448+0000 7f26137fe700 1 -- 172.21.15.7:0/2387524366 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f26180d0300 con 0x7f262406b0f0 2024-04-21T08:42:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.448+0000 7f262259c700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f260c0a3cd0 0x7f260c0a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f261400b5c0 tx=0x7f2614005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:51.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.453+0000 7f26137fe700 1 -- 172.21.15.7:0/2387524366 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2618014030 con 0x7f262406b0f0 2024-04-21T08:42:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.721+0000 7f2628eec700 1 -- 172.21.15.7:0/2387524366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f262402df80 con 0x7f262406b0f0 2024-04-21T08:42:51.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.725+0000 7f26137fe700 1 -- 172.21.15.7:0/2387524366 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264987 (secure 0 0 0) 0x7f261809c940 con 0x7f262406b0f0 2024-04-21T08:42:51.728 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 -- 172.21.15.7:0/2387524366 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f260c0a3cd0 msgr2=0x7f260c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f260c0a3cd0 0x7f260c0a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f261400b5c0 tx=0x7f2614005960 comp rx=0 tx=0).stop 2024-04-21T08:42:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 -- 172.21.15.7:0/2387524366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f262406b0f0 msgr2=0x7f2624101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f262406b0f0 0x7f2624101f50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f261800c920 tx=0x7f261800cc30 comp rx=0 tx=0).stop 2024-04-21T08:42:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 -- 172.21.15.7:0/2387524366 shutdown_connections 2024-04-21T08:42:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f260c0a3cd0 0x7f260c0a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f262406a750 0x7f2624101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2624108a30 0x7f2624102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 --2- 172.21.15.7:0/2387524366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f262406b0f0 0x7f2624101f50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 -- 172.21.15.7:0/2387524366 >> 172.21.15.7:0/2387524366 conn(0x7f26240788f0 msgr2=0x7f26240ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 -- 172.21.15.7:0/2387524366 shutdown_connections 2024-04-21T08:42:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:51.728+0000 7f2628eec700 1 -- 172.21.15.7:0/2387524366 wait complete. 2024-04-21T08:42:52.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:51 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2387524366' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:42:52.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:51 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2387524366' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:42:52.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:51 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2387524366' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:42:52.440 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2555078032},{"type":"v1","addr":"172.21.15.7:6801","nonce":2555078032}]},"active_addr":"172.21.15.7:6801/2555078032","active_change":"2024-04-21T08:35:09.542049+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.7:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":117789107}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3451243741}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":4167672076}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3630897301}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":833982125}]}]} 2024-04-21T08:42:52.446 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-21T08:42:52.446 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-21T08:42:52.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:42:52.699 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:52 smithi016 ceph-mon[31259]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-04-21T08:42:53.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:52 smithi007 ceph-mon[28477]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-04-21T08:42:53.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:52 smithi077 ceph-mon[31615]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-04-21T08:42:54.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.510+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1326422197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0b0102ac0 msgr2=0x7fd0b0102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.510+0000 7fd0b4f43700 1 --2- 172.21.15.7:0/1326422197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0b0102ac0 0x7fd0b0102f40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd098009990 tx=0x7fd098009ca0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.511+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1326422197 shutdown_connections 2024-04-21T08:42:54.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.511+0000 7fd0b4f43700 1 --2- 172.21.15.7:0/1326422197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0b0108a70 0x7fd0b0108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.511+0000 7fd0b4f43700 1 --2- 172.21.15.7:0/1326422197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0103480 0x7fd0b010f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.511+0000 7fd0b4f43700 1 --2- 172.21.15.7:0/1326422197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0b0102ac0 0x7fd0b0102f40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.511+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1326422197 >> 172.21.15.7:0/1326422197 conn(0x7fd0b0079bb0 msgr2=0x7fd0b0079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.511+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1326422197 shutdown_connections 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.511+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1326422197 wait complete. 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.511+0000 7fd0b4f43700 1 Processor -- start 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.511+0000 7fd0b4f43700 1 -- start start 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0b4f43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0b0102ac0 0x7fd0b0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0b4f43700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0b0103480 0x7fd0b0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0b4f43700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0108a70 0x7fd0b00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0b4f43700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b0112c20 con 0x7fd0b0102ac0 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0b4f43700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b0112aa0 con 0x7fd0b0108a70 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0b4f43700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b0112da0 con 0x7fd0b0103480 2024-04-21T08:42:54.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0ae59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0b0102ac0 0x7fd0b0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0ae59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0b0102ac0 0x7fd0b0074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50340/0 (socket says 172.21.15.7:50340) 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0ae59c700 1 -- 172.21.15.7:0/1664779821 learned_addr learned my addr 172.21.15.7:0/1664779821 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0add9b700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0b0103480 0x7fd0b0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0aed9d700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0108a70 0x7fd0b00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0ae59c700 1 -- 172.21.15.7:0/1664779821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0b0103480 msgr2=0x7fd0b0075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0ae59c700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0b0103480 0x7fd0b0075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0ae59c700 1 -- 172.21.15.7:0/1664779821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0108a70 msgr2=0x7fd0b00757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0ae59c700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0108a70 0x7fd0b00757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0ae59c700 1 -- 172.21.15.7:0/1664779821 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b01a75c0 con 0x7fd0b0102ac0 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.512+0000 7fd0ae59c700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0b0102ac0 0x7fd0b0074d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800d980 tx=0x7fd0a800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.513+0000 7fd0a77fe700 1 -- 172.21.15.7:0/1664779821 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8017070 con 0x7fd0b0102ac0 2024-04-21T08:42:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.513+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1664779821 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b01a78b0 con 0x7fd0b0102ac0 2024-04-21T08:42:54.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.513+0000 7fd0a77fe700 1 -- 172.21.15.7:0/1664779821 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0a8005020 con 0x7fd0b0102ac0 2024-04-21T08:42:54.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.513+0000 7fd0a77fe700 1 -- 172.21.15.7:0/1664779821 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a800fa10 con 0x7fd0b0102ac0 2024-04-21T08:42:54.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.513+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1664779821 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b01a7dc0 con 0x7fd0b0102ac0 2024-04-21T08:42:54.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.514+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1664779821 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b0069730 con 0x7fd0b0102ac0 2024-04-21T08:42:54.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.515+0000 7fd0a77fe700 1 -- 172.21.15.7:0/1664779821 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fd0a8004a30 con 0x7fd0b0102ac0 2024-04-21T08:42:54.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.516+0000 7fd0a77fe700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd09c0a3cd0 0x7fd09c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.516+0000 7fd0add9b700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd09c0a3cd0 0x7fd09c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.516+0000 7fd0a77fe700 1 -- 172.21.15.7:0/1664779821 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd0a80d0a20 con 0x7fd0b0102ac0 2024-04-21T08:42:54.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.516+0000 7fd0add9b700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd09c0a3cd0 0x7fd09c0a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd09800b5c0 tx=0x7fd098005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:54.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.519+0000 7fd0a77fe700 1 -- 172.21.15.7:0/1664779821 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0a8020020 con 0x7fd0b0102ac0 2024-04-21T08:42:54.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.730+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1664779821 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0b002df80 con 0x7fd0b0102ac0 2024-04-21T08:42:54.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.732+0000 7fd0a77fe700 1 -- 172.21.15.7:0/1664779821 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23961 (secure 0 0 0) 0x7fd0a809d000 con 0x7fd0b0102ac0 2024-04-21T08:42:54.731 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:54.732 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":90,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:42:22.023111+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1664779821 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd09c0a3cd0 msgr2=0x7fd09c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd09c0a3cd0 0x7fd09c0a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd09800b5c0 tx=0x7fd098005960 comp rx=0 tx=0).stop 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1664779821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0b0102ac0 msgr2=0x7fd0b0074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0b0102ac0 0x7fd0b0074d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800d980 tx=0x7fd0a800dc90 comp rx=0 tx=0).stop 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1664779821 shutdown_connections 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd09c0a3cd0 0x7fd09c0a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0108a70 0x7fd0b00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0b0102ac0 0x7fd0b0074d40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 --2- 172.21.15.7:0/1664779821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0b0103480 0x7fd0b0075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1664779821 >> 172.21.15.7:0/1664779821 conn(0x7fd0b0079bb0 msgr2=0x7fd0b01016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1664779821 shutdown_connections 2024-04-21T08:42:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:54.734+0000 7fd0b4f43700 1 -- 172.21.15.7:0/1664779821 wait complete. 2024-04-21T08:42:54.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:54 smithi007 ceph-mon[28477]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-04-21T08:42:54.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:54 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1664779821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:54 smithi016 ceph-mon[31259]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-04-21T08:42:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:54 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1664779821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:54 smithi077 ceph-mon[31615]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-04-21T08:42:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:54 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1664779821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:55.449 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-21T08:42:55.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:42:55.701 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:57.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:56 smithi016 ceph-mon[31259]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-04-21T08:42:57.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:56 smithi007 ceph-mon[28477]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-04-21T08:42:57.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:56 smithi077 ceph-mon[31615]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-04-21T08:42:57.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.319+0000 7f900ca7e700 1 -- 172.21.15.7:0/232796041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9008108a30 msgr2=0x7f9008108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:57.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.319+0000 7f900ca7e700 1 --2- 172.21.15.7:0/232796041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9008108a30 0x7f9008108e10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009990 tx=0x7f8ff8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.320+0000 7f900ca7e700 1 -- 172.21.15.7:0/232796041 shutdown_connections 2024-04-21T08:42:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.320+0000 7f900ca7e700 1 --2- 172.21.15.7:0/232796041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900806b0f0 0x7f900810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.320+0000 7f900ca7e700 1 --2- 172.21.15.7:0/232796041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9008108a30 0x7f9008108e10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.320+0000 7f900ca7e700 1 --2- 172.21.15.7:0/232796041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900806a750 0x7f900806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.320+0000 7f900ca7e700 1 -- 172.21.15.7:0/232796041 >> 172.21.15.7:0/232796041 conn(0x7f90080788f0 msgr2=0x7f9008078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.320+0000 7f900ca7e700 1 -- 172.21.15.7:0/232796041 shutdown_connections 2024-04-21T08:42:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.320+0000 7f900ca7e700 1 -- 172.21.15.7:0/232796041 wait complete. 2024-04-21T08:42:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.320+0000 7f900ca7e700 1 Processor -- start 2024-04-21T08:42:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.320+0000 7f900ca7e700 1 -- start start 2024-04-21T08:42:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f900ca7e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900806a750 0x7f9008101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f900ca7e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900806b0f0 0x7f9008101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f900ca7e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9008108a30 0x7f9008102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f900ca7e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9008112bb0 con 0x7f900806b0f0 2024-04-21T08:42:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f900ca7e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9008112a30 con 0x7f900806a750 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f900ca7e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9008112d30 con 0x7f9008108a30 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f9005d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900806b0f0 0x7f9008101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f9005d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900806b0f0 0x7f9008101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40364/0 (socket says 172.21.15.7:40364) 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f9005d9b700 1 -- 172.21.15.7:0/324975597 learned_addr learned my addr 172.21.15.7:0/324975597 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f900659c700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900806a750 0x7f9008101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f9006d9d700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9008108a30 0x7f9008102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f9005d9b700 1 -- 172.21.15.7:0/324975597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9008108a30 msgr2=0x7f9008102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f9005d9b700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9008108a30 0x7f9008102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f9005d9b700 1 -- 172.21.15.7:0/324975597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900806a750 msgr2=0x7f9008101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f9005d9b700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900806a750 0x7f9008101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.321+0000 7f9005d9b700 1 -- 172.21.15.7:0/324975597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90081a75f0 con 0x7f900806b0f0 2024-04-21T08:42:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.322+0000 7f9005d9b700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900806b0f0 0x7f9008101f50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00d950 tx=0x7f8ffc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:57.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.322+0000 7f8ff77fe700 1 -- 172.21.15.7:0/324975597 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc016070 con 0x7f900806b0f0 2024-04-21T08:42:57.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.322+0000 7f8ff77fe700 1 -- 172.21.15.7:0/324975597 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ffc009df0 con 0x7f900806b0f0 2024-04-21T08:42:57.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.322+0000 7f900ca7e700 1 -- 172.21.15.7:0/324975597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90081a78e0 con 0x7f900806b0f0 2024-04-21T08:42:57.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.322+0000 7f8ff77fe700 1 -- 172.21.15.7:0/324975597 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc00f3c0 con 0x7f900806b0f0 2024-04-21T08:42:57.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.322+0000 7f900ca7e700 1 -- 172.21.15.7:0/324975597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90081a7e70 con 0x7f900806b0f0 2024-04-21T08:42:57.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.323+0000 7f900ca7e700 1 -- 172.21.15.7:0/324975597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9008069730 con 0x7f900806b0f0 2024-04-21T08:42:57.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.324+0000 7f8ff77fe700 1 -- 172.21.15.7:0/324975597 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8ffc019070 con 0x7f900806b0f0 2024-04-21T08:42:57.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.324+0000 7f8ff77fe700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8ff00a3d50 0x7f8ff00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:57.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.325+0000 7f8ff77fe700 1 -- 172.21.15.7:0/324975597 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8ffc0d0300 con 0x7f900806b0f0 2024-04-21T08:42:57.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.325+0000 7f900659c700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8ff00a3d50 0x7f8ff00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:57.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.325+0000 7f900659c700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8ff00a3d50 0x7f8ff00a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8ff80098e0 tx=0x7f8ff8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:57.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.328+0000 7f8ff77fe700 1 -- 172.21.15.7:0/324975597 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ffc09c940 con 0x7f900806b0f0 2024-04-21T08:42:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.537+0000 7f900ca7e700 1 -- 172.21.15.7:0/324975597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f900802df80 con 0x7f900806b0f0 2024-04-21T08:42:57.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.539+0000 7f8ff77fe700 1 -- 172.21.15.7:0/324975597 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23961 (secure 0 0 0) 0x7f8ffc09c240 con 0x7f900806b0f0 2024-04-21T08:42:57.539 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:42:57.539 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":90,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:42:22.023111+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 -- 172.21.15.7:0/324975597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8ff00a3d50 msgr2=0x7f8ff00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8ff00a3d50 0x7f8ff00a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8ff80098e0 tx=0x7f8ff8005960 comp rx=0 tx=0).stop 2024-04-21T08:42:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 -- 172.21.15.7:0/324975597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900806b0f0 msgr2=0x7f9008101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900806b0f0 0x7f9008101f50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00d950 tx=0x7f8ffc00dc60 comp rx=0 tx=0).stop 2024-04-21T08:42:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 -- 172.21.15.7:0/324975597 shutdown_connections 2024-04-21T08:42:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8ff00a3d50 0x7f8ff00a6210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9008108a30 0x7f9008102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900806a750 0x7f9008101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 --2- 172.21.15.7:0/324975597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900806b0f0 0x7f9008101f50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 -- 172.21.15.7:0/324975597 >> 172.21.15.7:0/324975597 conn(0x7f90080788f0 msgr2=0x7f90080ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 -- 172.21.15.7:0/324975597 shutdown_connections 2024-04-21T08:42:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:42:57.542+0000 7f900ca7e700 1 -- 172.21.15.7:0/324975597 wait complete. 2024-04-21T08:42:58.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:57 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/324975597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:58.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:57 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/324975597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:58.111 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:57 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/324975597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:58.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-21T08:42:58.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-21T08:42:58.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-21T08:42:58.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-21T08:42:58.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-21T08:42:58.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-21T08:42:58.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-21T08:42:58.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-21T08:42:58.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-21T08:42:58.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-21T08:42:58.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-21T08:42:58.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-21T08:42:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:42:58 smithi016 ceph-mon[31259]: pgmap v314: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 5.8 KiB/s wr, 5 op/s 2024-04-21T08:42:59.090 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:59.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:42:58 smithi077 ceph-mon[31615]: pgmap v314: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 5.8 KiB/s wr, 5 op/s 2024-04-21T08:42:59.137 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:59.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:42:58 smithi007 ceph-mon[28477]: pgmap v314: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 5.8 KiB/s wr, 5 op/s 2024-04-21T08:43:00.529 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:00.547 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:00.548 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:00.549 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:00.549 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:00.553 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:00.553 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:00.555 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:00.604 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:01.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:00 smithi007 ceph-mon[28477]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-21T08:43:01.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:00 smithi016 ceph-mon[31259]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-21T08:43:01.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:00 smithi077 ceph-mon[31615]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:02.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.814+0000 7f43cb9fc700 1 -- 172.21.15.7:0/1438648008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43c406a750 msgr2=0x7f43c406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:02.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.814+0000 7f43cb9fc700 1 --2- 172.21.15.7:0/1438648008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43c406a750 0x7f43c406abb0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f43b4009990 tx=0x7f43b4009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.814+0000 7f43cb9fc700 1 -- 172.21.15.7:0/1438648008 shutdown_connections 2024-04-21T08:43:02.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.814+0000 7f43cb9fc700 1 --2- 172.21.15.7:0/1438648008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c406b0f0 0x7f43c410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.814+0000 7f43cb9fc700 1 --2- 172.21.15.7:0/1438648008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c4108a30 0x7f43c4108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.814+0000 7f43cb9fc700 1 --2- 172.21.15.7:0/1438648008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43c406a750 0x7f43c406abb0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.814+0000 7f43cb9fc700 1 -- 172.21.15.7:0/1438648008 >> 172.21.15.7:0/1438648008 conn(0x7f43c40788f0 msgr2=0x7f43c4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:02.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.814+0000 7f43cb9fc700 1 -- 172.21.15.7:0/1438648008 shutdown_connections 2024-04-21T08:43:02.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.814+0000 7f43cb9fc700 1 -- 172.21.15.7:0/1438648008 wait complete. 2024-04-21T08:43:02.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43cb9fc700 1 Processor -- start 2024-04-21T08:43:02.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43cb9fc700 1 -- start start 2024-04-21T08:43:02.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43cb9fc700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c406a750 0x7f43c4101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43cb9fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43c406b0f0 0x7f43c4101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43cb9fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c4108a30 0x7f43c4102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43cb9fc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f43c4112c10 con 0x7f43c406b0f0 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43cb9fc700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f43c4112a90 con 0x7f43c406a750 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43cb9fc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f43c4112d90 con 0x7f43c4108a30 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43c8f97700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43c406b0f0 0x7f43c4101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43c8f97700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43c406b0f0 0x7f43c4101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40392/0 (socket says 172.21.15.7:40392) 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43c8f97700 1 -- 172.21.15.7:0/212326210 learned_addr learned my addr 172.21.15.7:0/212326210 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.815+0000 7f43c9f99700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c4108a30 0x7f43c4102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43c9798700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c406a750 0x7f43c4101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43c8f97700 1 -- 172.21.15.7:0/212326210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c4108a30 msgr2=0x7f43c4102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43c8f97700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43c4108a30 0x7f43c4102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43c8f97700 1 -- 172.21.15.7:0/212326210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c406a750 msgr2=0x7f43c4101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43c8f97700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c406a750 0x7f43c4101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43c8f97700 1 -- 172.21.15.7:0/212326210 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c41a7690 con 0x7f43c406b0f0 2024-04-21T08:43:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43c9798700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c406a750 0x7f43c4101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43c8f97700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43c406b0f0 0x7f43c4101f50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f43b4000c00 tx=0x7f43b40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43ba7fc700 1 -- 172.21.15.7:0/212326210 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b4027070 con 0x7f43c406b0f0 2024-04-21T08:43:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43cb9fc700 1 -- 172.21.15.7:0/212326210 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c41a7920 con 0x7f43c406b0f0 2024-04-21T08:43:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43ba7fc700 1 -- 172.21.15.7:0/212326210 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43b4003f20 con 0x7f43c406b0f0 2024-04-21T08:43:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43ba7fc700 1 -- 172.21.15.7:0/212326210 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b4022cf0 con 0x7f43c406b0f0 2024-04-21T08:43:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.816+0000 7f43cb9fc700 1 -- 172.21.15.7:0/212326210 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c41a7dd0 con 0x7f43c406b0f0 2024-04-21T08:43:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.817+0000 7f43cb9fc700 1 -- 172.21.15.7:0/212326210 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f43c40630d0 con 0x7f43c406b0f0 2024-04-21T08:43:02.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.818+0000 7f43ba7fc700 1 -- 172.21.15.7:0/212326210 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f43b402a070 con 0x7f43c406b0f0 2024-04-21T08:43:02.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.819+0000 7f43ba7fc700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f43b00a3cd0 0x7f43b00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.819+0000 7f43ba7fc700 1 -- 172.21.15.7:0/212326210 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f43b40d1b40 con 0x7f43c406b0f0 2024-04-21T08:43:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.819+0000 7f43c9798700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f43b00a3cd0 0x7f43b00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.819+0000 7f43ba7fc700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881] conn(0x7f43b00ae990 0x7f43b00b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.819+0000 7f43ba7fc700 1 -- 172.21.15.7:0/212326210 --> [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43b402ce50 con 0x7f43b00ae990 2024-04-21T08:43:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.819+0000 7f43ba7fc700 1 -- 172.21.15.7:0/212326210 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f43b40d1f90 con 0x7f43c406b0f0 2024-04-21T08:43:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.820+0000 7f43c9f99700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881] conn(0x7f43b00ae990 0x7f43b00b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.820+0000 7f43c9798700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f43b00a3cd0 0x7f43b00a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f43c0005fd0 tx=0x7f43c0005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.820+0000 7f43c9f99700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881] conn(0x7f43b00ae990 0x7f43b00b0dd0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.821+0000 7f43ba7fc700 1 -- 172.21.15.7:0/212326210 <== osd.3 v2:172.21.15.7:6810/3942680881 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f43b402ce50 con 0x7f43b00ae990 2024-04-21T08:43:02.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.838+0000 7f43cb9fc700 1 -- 172.21.15.7:0/212326210 --> [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f43c40634d0 con 0x7f43b00ae990 2024-04-21T08:43:02.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.839+0000 7f43ba7fc700 1 -- 172.21.15.7:0/212326210 <== osd.3 v2:172.21.15.7:6810/3942680881 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f43c40634d0 con 0x7f43b00ae990 2024-04-21T08:43:02.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.839+0000 7f43cb9fc700 1 -- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881] conn(0x7f43b00ae990 msgr2=0x7f43b00b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:02.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.839+0000 7f43cb9fc700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881] conn(0x7f43b00ae990 0x7f43b00b0dd0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.839+0000 7f43cb9fc700 1 -- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f43b00a3cd0 msgr2=0x7f43b00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:02.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.839+0000 7f43cb9fc700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f43b00a3cd0 0x7f43b00a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f43c0005fd0 tx=0x7f43c0005db0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.839+0000 7f43cb9fc700 1 -- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43c406b0f0 msgr2=0x7f43c4101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:02.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.839+0000 7f43cb9fc700 1 --2- 172.21.15.7:0/212326210 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43c406b0f0 0x7f43c4101f50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f43b4000c00 tx=0x7f43b40047b0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.839+0000 7f43c9f99700 1 -- 172.21.15.7:0/212326210 reap_dead start 2024-04-21T08:43:02.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.839+0000 7f43cb9fc700 1 -- 172.21.15.7:0/212326210 shutdown_connections 2024-04-21T08:43:02.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.839+0000 7f43cb9fc700 1 -- 172.21.15.7:0/212326210 >> 172.21.15.7:0/212326210 conn(0x7f43c40788f0 msgr2=0x7f43c40ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:02.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.840+0000 7f43cb9fc700 1 -- 172.21.15.7:0/212326210 shutdown_connections 2024-04-21T08:43:02.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:02.840+0000 7f43cb9fc700 1 -- 172.21.15.7:0/212326210 wait complete. 2024-04-21T08:43:03.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:02 smithi007 ceph-mon[28477]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-21T08:43:03.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:02 smithi016 ceph-mon[31259]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-21T08:43:03.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:02 smithi077 ceph-mon[31615]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-21T08:43:04.169 INFO:teuthology.orchestra.run.smithi007.stdout:111669149760 2024-04-21T08:43:04.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-21T08:43:05.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:04 smithi007 ceph-mon[28477]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T08:43:05.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:04 smithi016 ceph-mon[31259]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T08:43:05.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:04 smithi077 ceph-mon[31615]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T08:43:05.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.249+0000 7f6f25f6c700 1 -- 172.21.15.7:0/3117562676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f20108a70 msgr2=0x7f6f20108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.249+0000 7f6f25f6c700 1 --2- 172.21.15.7:0/3117562676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f20108a70 0x7f6f20108e50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009990 tx=0x7f6f10009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.249+0000 7f6f25f6c700 1 -- 172.21.15.7:0/3117562676 shutdown_connections 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.249+0000 7f6f25f6c700 1 --2- 172.21.15.7:0/3117562676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f20103480 0x7f6f2010f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.249+0000 7f6f25f6c700 1 --2- 172.21.15.7:0/3117562676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f20108a70 0x7f6f20108e50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.249+0000 7f6f25f6c700 1 --2- 172.21.15.7:0/3117562676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f20102ac0 0x7f6f20102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.249+0000 7f6f25f6c700 1 -- 172.21.15.7:0/3117562676 >> 172.21.15.7:0/3117562676 conn(0x7f6f20079bb0 msgr2=0x7f6f20079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.249+0000 7f6f25f6c700 1 -- 172.21.15.7:0/3117562676 shutdown_connections 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.249+0000 7f6f25f6c700 1 -- 172.21.15.7:0/3117562676 wait complete. 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.249+0000 7f6f25f6c700 1 Processor -- start 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f25f6c700 1 -- start start 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f25f6c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f20102ac0 0x7f6f20074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f25f6c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f20103480 0x7f6f20075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f25f6c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f20108a70 0x7f6f200757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f25f6c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6f20112c20 con 0x7f6f20102ac0 2024-04-21T08:43:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f25f6c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f20112aa0 con 0x7f6f20103480 2024-04-21T08:43:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f25f6c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6f20112da0 con 0x7f6f20108a70 2024-04-21T08:43:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f1f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f20102ac0 0x7f6f20074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f1f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f20102ac0 0x7f6f20074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53830/0 (socket says 172.21.15.7:53830) 2024-04-21T08:43:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f1f7fe700 1 -- 172.21.15.7:0/1612715861 learned_addr learned my addr 172.21.15.7:0/1612715861 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f1effd700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f20103480 0x7f6f20075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f1ffff700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f20108a70 0x7f6f200757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f1f7fe700 1 -- 172.21.15.7:0/1612715861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f20108a70 msgr2=0x7f6f200757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f1f7fe700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f20108a70 0x7f6f200757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f1f7fe700 1 -- 172.21.15.7:0/1612715861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f20103480 msgr2=0x7f6f20075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f1f7fe700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f20103480 0x7f6f20075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f1f7fe700 1 -- 172.21.15.7:0/1612715861 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f201a7620 con 0x7f6f20102ac0 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.250+0000 7f6f1ffff700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f20108a70 0x7f6f200757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.251+0000 7f6f1f7fe700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f20102ac0 0x7f6f20074d40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6f10000c00 tx=0x7f6f100047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.251+0000 7f6f1cff9700 1 -- 172.21.15.7:0/1612715861 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f10027070 con 0x7f6f20102ac0 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.251+0000 7f6f25f6c700 1 -- 172.21.15.7:0/1612715861 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f201a78b0 con 0x7f6f20102ac0 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.251+0000 7f6f1cff9700 1 -- 172.21.15.7:0/1612715861 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f10003f20 con 0x7f6f20102ac0 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.251+0000 7f6f1cff9700 1 -- 172.21.15.7:0/1612715861 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f10022cf0 con 0x7f6f20102ac0 2024-04-21T08:43:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.251+0000 7f6f25f6c700 1 -- 172.21.15.7:0/1612715861 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f201a7de0 con 0x7f6f20102ac0 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.252+0000 7f6f25f6c700 1 -- 172.21.15.7:0/1612715861 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6f200630d0 con 0x7f6f20102ac0 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.253+0000 7f6f1cff9700 1 -- 172.21.15.7:0/1612715861 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f6f1002a070 con 0x7f6f20102ac0 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.254+0000 7f6f1cff9700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6f080a3d50 0x7f6f080a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.254+0000 7f6f1effd700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6f080a3d50 0x7f6f080a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.254+0000 7f6f1cff9700 1 -- 172.21.15.7:0/1612715861 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6f100d1b40 con 0x7f6f20102ac0 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.254+0000 7f6f1cff9700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831] conn(0x7f6f080aea10 0x7f6f080b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.254+0000 7f6f1cff9700 1 -- 172.21.15.7:0/1612715861 --> [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f100095d0 con 0x7f6f080aea10 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.254+0000 7f6f1cff9700 1 -- 172.21.15.7:0/1612715861 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f6f100d1f90 con 0x7f6f20102ac0 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.254+0000 7f6f1ffff700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831] conn(0x7f6f080aea10 0x7f6f080b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.255+0000 7f6f1effd700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6f080a3d50 0x7f6f080a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6f2007a220 tx=0x7f6f1400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.255+0000 7f6f1ffff700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831] conn(0x7f6f080aea10 0x7f6f080b0e50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.256+0000 7f6f1cff9700 1 -- 172.21.15.7:0/1612715861 <== osd.9 v2:172.21.15.7:6826/485389831 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f100095d0 con 0x7f6f080aea10 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.283+0000 7f6f25f6c700 1 -- 172.21.15.7:0/1612715861 --> [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6f200634d0 con 0x7f6f080aea10 2024-04-21T08:43:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.284+0000 7f6f1cff9700 1 -- 172.21.15.7:0/1612715861 <== osd.9 v2:172.21.15.7:6826/485389831 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6f200634d0 con 0x7f6f080aea10 2024-04-21T08:43:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.284+0000 7f6f25f6c700 1 -- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831] conn(0x7f6f080aea10 msgr2=0x7f6f080b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.284+0000 7f6f25f6c700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831] conn(0x7f6f080aea10 0x7f6f080b0e50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.284+0000 7f6f25f6c700 1 -- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6f080a3d50 msgr2=0x7f6f080a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.284+0000 7f6f25f6c700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f6f080a3d50 0x7f6f080a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6f2007a220 tx=0x7f6f1400b3c0 comp rx=0 tx=0).stop 2024-04-21T08:43:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.284+0000 7f6f25f6c700 1 -- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f20102ac0 msgr2=0x7f6f20074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.284+0000 7f6f25f6c700 1 --2- 172.21.15.7:0/1612715861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f20102ac0 0x7f6f20074d40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6f10000c00 tx=0x7f6f100047b0 comp rx=0 tx=0).stop 2024-04-21T08:43:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.285+0000 7f6f1ffff700 1 -- 172.21.15.7:0/1612715861 reap_dead start 2024-04-21T08:43:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.285+0000 7f6f25f6c700 1 -- 172.21.15.7:0/1612715861 shutdown_connections 2024-04-21T08:43:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.285+0000 7f6f25f6c700 1 -- 172.21.15.7:0/1612715861 >> 172.21.15.7:0/1612715861 conn(0x7f6f20079bb0 msgr2=0x7f6f201017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.285+0000 7f6f25f6c700 1 -- 172.21.15.7:0/1612715861 shutdown_connections 2024-04-21T08:43:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:05.285+0000 7f6f25f6c700 1 -- 172.21.15.7:0/1612715861 wait complete. 2024-04-21T08:43:07.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:06 smithi016 ceph-mon[31259]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-21T08:43:07.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:06 smithi077 ceph-mon[31615]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-21T08:43:07.106 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:06 smithi007 ceph-mon[28477]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-21T08:43:09.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:08 smithi016 ceph-mon[31259]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T08:43:09.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:08 smithi077 ceph-mon[31615]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T08:43:09.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:08 smithi007 ceph-mon[28477]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T08:43:09.707 INFO:teuthology.orchestra.run.smithi007.stdout:292057776162 2024-04-21T08:43:09.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-21T08:43:09.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-21T08:43:09.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-21T08:43:09.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-21T08:43:09.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-21T08:43:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-21T08:43:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-21T08:43:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-21T08:43:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-21T08:43:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 7]}]: dispatch 2024-04-21T08:43:09.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:09 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-04-21T08:43:10.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-21T08:43:10.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-21T08:43:10.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-21T08:43:10.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-21T08:43:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-21T08:43:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-21T08:43:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-21T08:43:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-21T08:43:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 7]}]: dispatch 2024-04-21T08:43:10.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:09 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-04-21T08:43:10.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-21T08:43:10.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-21T08:43:10.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-21T08:43:10.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-21T08:43:10.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-21T08:43:10.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-21T08:43:10.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-21T08:43:10.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-21T08:43:10.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 7]}]: dispatch 2024-04-21T08:43:10.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:09 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-04-21T08:43:10.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: pgmap v320: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T08:43:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-21T08:43:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-21T08:43:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-21T08:43:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-21T08:43:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-21T08:43:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-21T08:43:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-21T08:43:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-21T08:43:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 7]}]': finished 2024-04-21T08:43:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-04-21T08:43:10.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:10 smithi007 ceph-mon[28477]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:43:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: pgmap v320: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T08:43:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-21T08:43:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-21T08:43:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-21T08:43:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-21T08:43:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-21T08:43:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-21T08:43:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-21T08:43:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-21T08:43:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 7]}]': finished 2024-04-21T08:43:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-04-21T08:43:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:10 smithi016 ceph-mon[31259]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:43:11.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: pgmap v320: 129 pgs: 129 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T08:43:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-21T08:43:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-21T08:43:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-21T08:43:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-21T08:43:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-21T08:43:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-21T08:43:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-21T08:43:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-21T08:43:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 7]}]': finished 2024-04-21T08:43:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-04-21T08:43:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:10 smithi077 ceph-mon[31615]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:43:11.997 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:11 smithi007 ceph-mon[28477]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:43:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:11 smithi016 ceph-mon[31259]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:43:12.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:11 smithi077 ceph-mon[31615]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:43:13.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:12 smithi016 ceph-mon[31259]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-21T08:43:13.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:12 smithi077 ceph-mon[31615]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-21T08:43:13.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:12 smithi007 ceph-mon[28477]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-21T08:43:13.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.861+0000 7fb407de8700 1 -- 172.21.15.7:0/1760480380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb40006a750 msgr2=0x7fb40006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.861+0000 7fb407de8700 1 --2- 172.21.15.7:0/1760480380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb40006a750 0x7fb40006abb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009990 tx=0x7fb3f0009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.861+0000 7fb407de8700 1 -- 172.21.15.7:0/1760480380 shutdown_connections 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.861+0000 7fb407de8700 1 --2- 172.21.15.7:0/1760480380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb400108a30 0x7fb400108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.861+0000 7fb407de8700 1 --2- 172.21.15.7:0/1760480380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40006b0f0 0x7fb40010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.861+0000 7fb407de8700 1 --2- 172.21.15.7:0/1760480380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb40006a750 0x7fb40006abb0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.861+0000 7fb407de8700 1 -- 172.21.15.7:0/1760480380 >> 172.21.15.7:0/1760480380 conn(0x7fb4000788f0 msgr2=0x7fb400078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.861+0000 7fb407de8700 1 -- 172.21.15.7:0/1760480380 shutdown_connections 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.861+0000 7fb407de8700 1 -- 172.21.15.7:0/1760480380 wait complete. 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.862+0000 7fb407de8700 1 Processor -- start 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.862+0000 7fb407de8700 1 -- start start 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.862+0000 7fb407de8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb40006a750 0x7fb400199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.862+0000 7fb407de8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40006b0f0 0x7fb4001996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.862+0000 7fb407de8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb400108a30 0x7fb40019d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb405b84700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb40006a750 0x7fb400199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:13.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb405b84700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb40006a750 0x7fb400199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53856/0 (socket says 172.21.15.7:53856) 2024-04-21T08:43:13.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb405383700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40006b0f0 0x7fb4001996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:13.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb400112bd0 con 0x7fb40006a750 2024-04-21T08:43:13.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb405b84700 1 -- 172.21.15.7:0/985044903 learned_addr learned my addr 172.21.15.7:0/985044903 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:13.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb400112a50 con 0x7fb40006b0f0 2024-04-21T08:43:13.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb400112d50 con 0x7fb400108a30 2024-04-21T08:43:13.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb406385700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb400108a30 0x7fb40019d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:13.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb405383700 1 -- 172.21.15.7:0/985044903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb400108a30 msgr2=0x7fb40019d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb405383700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb400108a30 0x7fb40019d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb405383700 1 -- 172.21.15.7:0/985044903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb40006a750 msgr2=0x7fb400199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb405383700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb40006a750 0x7fb400199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb405383700 1 -- 172.21.15.7:0/985044903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40019e120 con 0x7fb40006b0f0 2024-04-21T08:43:13.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb406385700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb400108a30 0x7fb40019d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:13.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.863+0000 7fb405383700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40006b0f0 0x7fb4001996a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00059d0 tx=0x7fb3f00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:13.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.864+0000 7fb3f6ffd700 1 -- 172.21.15.7:0/985044903 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0027070 con 0x7fb40006b0f0 2024-04-21T08:43:13.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.864+0000 7fb3f6ffd700 1 -- 172.21.15.7:0/985044903 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3f0003fe0 con 0x7fb40006b0f0 2024-04-21T08:43:13.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.864+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40019e3b0 con 0x7fb40006b0f0 2024-04-21T08:43:13.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.864+0000 7fb3f6ffd700 1 -- 172.21.15.7:0/985044903 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0022b20 con 0x7fb40006b0f0 2024-04-21T08:43:13.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.864+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4001a5c90 con 0x7fb40006b0f0 2024-04-21T08:43:13.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.865+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb4000631d0 con 0x7fb40006b0f0 2024-04-21T08:43:13.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.866+0000 7fb3f6ffd700 1 -- 172.21.15.7:0/985044903 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fb3f002a070 con 0x7fb40006b0f0 2024-04-21T08:43:13.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.867+0000 7fb3f6ffd700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb3ec0a3d50 0x7fb3ec0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:13.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.867+0000 7fb405b84700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb3ec0a3d50 0x7fb3ec0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:13.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.867+0000 7fb3f6ffd700 1 -- 172.21.15.7:0/985044903 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fb3f00d1ad0 con 0x7fb40006b0f0 2024-04-21T08:43:13.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.867+0000 7fb3f6ffd700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994] conn(0x7fb3ec0aeeb0 0x7fb3ec0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:13.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.867+0000 7fb3f6ffd700 1 -- 172.21.15.7:0/985044903 --> [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3f0010ea0 con 0x7fb3ec0aeeb0 2024-04-21T08:43:13.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.867+0000 7fb3f6ffd700 1 -- 172.21.15.7:0/985044903 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fb3f00d1f20 con 0x7fb40006b0f0 2024-04-21T08:43:13.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.868+0000 7fb405b84700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb3ec0a3d50 0x7fb3ec0a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00bcb0 tx=0x7fb3fc00b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:13.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.868+0000 7fb406385700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994] conn(0x7fb3ec0aeeb0 0x7fb3ec0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:13.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.868+0000 7fb406385700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994] conn(0x7fb3ec0aeeb0 0x7fb3ec0b1350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:13.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.869+0000 7fb3f6ffd700 1 -- 172.21.15.7:0/985044903 <== osd.5 v2:172.21.15.77:6808/286186994 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3f0010ea0 con 0x7fb3ec0aeeb0 2024-04-21T08:43:13.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.886+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 --> [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb400063650 con 0x7fb3ec0aeeb0 2024-04-21T08:43:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.886+0000 7fb3f6ffd700 1 -- 172.21.15.7:0/985044903 <== osd.5 v2:172.21.15.77:6808/286186994 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb400063650 con 0x7fb3ec0aeeb0 2024-04-21T08:43:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.887+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 >> [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994] conn(0x7fb3ec0aeeb0 msgr2=0x7fb3ec0b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.887+0000 7fb407de8700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994] conn(0x7fb3ec0aeeb0 0x7fb3ec0b1350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.887+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb3ec0a3d50 msgr2=0x7fb3ec0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.887+0000 7fb407de8700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fb3ec0a3d50 0x7fb3ec0a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00bcb0 tx=0x7fb3fc00b480 comp rx=0 tx=0).stop 2024-04-21T08:43:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.887+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40006b0f0 msgr2=0x7fb4001996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.887+0000 7fb407de8700 1 --2- 172.21.15.7:0/985044903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb40006b0f0 0x7fb4001996a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00059d0 tx=0x7fb3f00047b0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.887+0000 7fb406385700 1 -- 172.21.15.7:0/985044903 reap_dead start 2024-04-21T08:43:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.887+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 shutdown_connections 2024-04-21T08:43:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.887+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 >> 172.21.15.7:0/985044903 conn(0x7fb4000788f0 msgr2=0x7fb4000ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.887+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 shutdown_connections 2024-04-21T08:43:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:13.887+0000 7fb407de8700 1 -- 172.21.15.7:0/985044903 wait complete. 2024-04-21T08:43:14.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:14 smithi007 ceph-mon[28477]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-21T08:43:15.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:14 smithi016 ceph-mon[31259]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-21T08:43:15.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:14 smithi077 ceph-mon[31615]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 639 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2 op/s 2024-04-21T08:43:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.320+0000 7fdc19866700 1 -- 172.21.15.7:0/3900896489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc14107980 msgr2=0x7fdc14107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.320+0000 7fdc19866700 1 --2- 172.21.15.7:0/3900896489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc14107980 0x7fdc14107d60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00b0a0 tx=0x7fdc0c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.320+0000 7fdc19866700 1 -- 172.21.15.7:0/3900896489 shutdown_connections 2024-04-21T08:43:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.320+0000 7fdc19866700 1 --2- 172.21.15.7:0/3900896489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc141022f0 0x7fdc1410e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.320+0000 7fdc19866700 1 --2- 172.21.15.7:0/3900896489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc14107980 0x7fdc14107d60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.320+0000 7fdc19866700 1 --2- 172.21.15.7:0/3900896489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc14101930 0x7fdc14101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.320+0000 7fdc19866700 1 -- 172.21.15.7:0/3900896489 >> 172.21.15.7:0/3900896489 conn(0x7fdc140788b0 msgr2=0x7fdc14078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.320+0000 7fdc19866700 1 -- 172.21.15.7:0/3900896489 shutdown_connections 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.320+0000 7fdc19866700 1 -- 172.21.15.7:0/3900896489 wait complete. 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc19866700 1 Processor -- start 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc19866700 1 -- start start 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc19866700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc14101930 0x7fdc14198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc19866700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc141022f0 0x7fdc141985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc19866700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc14107980 0x7fdc1419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc19866700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdc14111b60 con 0x7fdc14107980 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc19866700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141119e0 con 0x7fdc14101930 2024-04-21T08:43:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc19866700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdc14111ce0 con 0x7fdc141022f0 2024-04-21T08:43:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc14107980 0x7fdc1419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc14107980 0x7fdc1419c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45022/0 (socket says 172.21.15.7:45022) 2024-04-21T08:43:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc137fe700 1 -- 172.21.15.7:0/2953812888 learned_addr learned my addr 172.21.15.7:0/2953812888 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc127fc700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc141022f0 0x7fdc141985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.321+0000 7fdc12ffd700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc14101930 0x7fdc14198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.322+0000 7fdc137fe700 1 -- 172.21.15.7:0/2953812888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc141022f0 msgr2=0x7fdc141985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.322+0000 7fdc137fe700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdc141022f0 0x7fdc141985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.322+0000 7fdc137fe700 1 -- 172.21.15.7:0/2953812888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc14101930 msgr2=0x7fdc14198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.322+0000 7fdc137fe700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc14101930 0x7fdc14198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.322+0000 7fdc137fe700 1 -- 172.21.15.7:0/2953812888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1419d080 con 0x7fdc14107980 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.322+0000 7fdc137fe700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc14107980 0x7fdc1419c940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400b550 tx=0x7fdc0400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.322+0000 7fdc18864700 1 -- 172.21.15.7:0/2953812888 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04015020 con 0x7fdc14107980 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.322+0000 7fdc19866700 1 -- 172.21.15.7:0/2953812888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1419d370 con 0x7fdc14107980 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.322+0000 7fdc18864700 1 -- 172.21.15.7:0/2953812888 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc04011ce0 con 0x7fdc14107980 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.322+0000 7fdc19866700 1 -- 172.21.15.7:0/2953812888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc141a4bf0 con 0x7fdc14107980 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.322+0000 7fdc18864700 1 -- 172.21.15.7:0/2953812888 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04010c70 con 0x7fdc14107980 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.323+0000 7fdc19866700 1 -- 172.21.15.7:0/2953812888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdc140631d0 con 0x7fdc14107980 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.324+0000 7fdc18864700 1 -- 172.21.15.7:0/2953812888 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fdc04018020 con 0x7fdc14107980 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.325+0000 7fdc18864700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdc000a3cd0 0x7fdc000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.325+0000 7fdc18864700 1 -- 172.21.15.7:0/2953812888 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fdc040d04f0 con 0x7fdc14107980 2024-04-21T08:43:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.325+0000 7fdc12ffd700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdc000a3cd0 0x7fdc000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.325+0000 7fdc18864700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241] conn(0x7fdc000aee30 0x7fdc000b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.325+0000 7fdc18864700 1 -- 172.21.15.7:0/2953812888 --> [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc0401ce70 con 0x7fdc000aee30 2024-04-21T08:43:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.325+0000 7fdc18864700 1 -- 172.21.15.7:0/2953812888 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fdc040d0940 con 0x7fdc14107980 2024-04-21T08:43:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.325+0000 7fdc127fc700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241] conn(0x7fdc000aee30 0x7fdc000b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.325+0000 7fdc12ffd700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdc000a3cd0 0x7fdc000a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009fd0 tx=0x7fdc0c00b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.325+0000 7fdc127fc700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241] conn(0x7fdc000aee30 0x7fdc000b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.326+0000 7fdc18864700 1 -- 172.21.15.7:0/2953812888 <== osd.0 v2:172.21.15.7:6802/3999850241 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc0401ce70 con 0x7fdc000aee30 2024-04-21T08:43:15.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.343+0000 7fdc19866700 1 -- 172.21.15.7:0/2953812888 --> [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdc14063650 con 0x7fdc000aee30 2024-04-21T08:43:15.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.343+0000 7fdc18864700 1 -- 172.21.15.7:0/2953812888 <== osd.0 v2:172.21.15.7:6802/3999850241 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdc14063650 con 0x7fdc000aee30 2024-04-21T08:43:15.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.343+0000 7fdc19866700 1 -- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241] conn(0x7fdc000aee30 msgr2=0x7fdc000b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.343+0000 7fdc19866700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241] conn(0x7fdc000aee30 0x7fdc000b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.343+0000 7fdc19866700 1 -- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdc000a3cd0 msgr2=0x7fdc000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.343+0000 7fdc19866700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdc000a3cd0 0x7fdc000a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009fd0 tx=0x7fdc0c00b7b0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.343+0000 7fdc19866700 1 -- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc14107980 msgr2=0x7fdc1419c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.343+0000 7fdc19866700 1 --2- 172.21.15.7:0/2953812888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc14107980 0x7fdc1419c940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400b550 tx=0x7fdc0400b860 comp rx=0 tx=0).stop 2024-04-21T08:43:15.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.343+0000 7fdc137fe700 1 -- 172.21.15.7:0/2953812888 reap_dead start 2024-04-21T08:43:15.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.344+0000 7fdc19866700 1 -- 172.21.15.7:0/2953812888 shutdown_connections 2024-04-21T08:43:15.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.344+0000 7fdc19866700 1 -- 172.21.15.7:0/2953812888 >> 172.21.15.7:0/2953812888 conn(0x7fdc140788b0 msgr2=0x7fdc14100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:15.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.344+0000 7fdc19866700 1 -- 172.21.15.7:0/2953812888 shutdown_connections 2024-04-21T08:43:15.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:15.344+0000 7fdc19866700 1 -- 172.21.15.7:0/2953812888 wait complete. 2024-04-21T08:43:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.520+0000 7f807474b700 1 -- 172.21.15.7:0/691038253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f806c06a750 msgr2=0x7f806c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.520+0000 7f807474b700 1 --2- 172.21.15.7:0/691038253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f806c06a750 0x7f806c06abb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8068009990 tx=0x7f8068009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.521+0000 7f807474b700 1 -- 172.21.15.7:0/691038253 shutdown_connections 2024-04-21T08:43:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.521+0000 7f807474b700 1 --2- 172.21.15.7:0/691038253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f806c06b0f0 0x7f806c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.521+0000 7f807474b700 1 --2- 172.21.15.7:0/691038253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806c108a30 0x7f806c108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.521+0000 7f807474b700 1 --2- 172.21.15.7:0/691038253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f806c06a750 0x7f806c06abb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.521+0000 7f807474b700 1 -- 172.21.15.7:0/691038253 >> 172.21.15.7:0/691038253 conn(0x7f806c0788f0 msgr2=0x7f806c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.521+0000 7f807474b700 1 -- 172.21.15.7:0/691038253 shutdown_connections 2024-04-21T08:43:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.521+0000 7f807474b700 1 -- 172.21.15.7:0/691038253 wait complete. 2024-04-21T08:43:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.521+0000 7f807474b700 1 Processor -- start 2024-04-21T08:43:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.521+0000 7f807474b700 1 -- start start 2024-04-21T08:43:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f807474b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f806c06a750 0x7f806c199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f807474b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806c06b0f0 0x7f806c199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f807474b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f806c108a30 0x7f806c19d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f807474b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f806c112c30 con 0x7f806c06a750 2024-04-21T08:43:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f807474b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f806c112ab0 con 0x7f806c06b0f0 2024-04-21T08:43:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f807474b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f806c112db0 con 0x7f806c108a30 2024-04-21T08:43:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f80724e7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f806c06a750 0x7f806c199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f80724e7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f806c06a750 0x7f806c199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53884/0 (socket says 172.21.15.7:53884) 2024-04-21T08:43:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f80724e7700 1 -- 172.21.15.7:0/2783025522 learned_addr learned my addr 172.21.15.7:0/2783025522 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f8071ce6700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806c06b0f0 0x7f806c199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f8072ce8700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f806c108a30 0x7f806c19d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:15.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f80724e7700 1 -- 172.21.15.7:0/2783025522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f806c108a30 msgr2=0x7f806c19d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f80724e7700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f806c108a30 0x7f806c19d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f80724e7700 1 -- 172.21.15.7:0/2783025522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806c06b0f0 msgr2=0x7f806c199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f80724e7700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806c06b0f0 0x7f806c199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f80724e7700 1 -- 172.21.15.7:0/2783025522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c19e130 con 0x7f806c06a750 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.522+0000 7f80724e7700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f806c06a750 0x7f806c199120 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f806000d950 tx=0x7f806000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.523+0000 7f805f7fe700 1 -- 172.21.15.7:0/2783025522 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8060016070 con 0x7f806c06a750 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.523+0000 7f807474b700 1 -- 172.21.15.7:0/2783025522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806c19e420 con 0x7f806c06a750 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.523+0000 7f805f7fe700 1 -- 172.21.15.7:0/2783025522 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8060009df0 con 0x7f806c06a750 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.523+0000 7f805f7fe700 1 -- 172.21.15.7:0/2783025522 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806000f3c0 con 0x7f806c06a750 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.523+0000 7f807474b700 1 -- 172.21.15.7:0/2783025522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f806c1a5ca0 con 0x7f806c06a750 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.523+0000 7f807474b700 1 -- 172.21.15.7:0/2783025522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f806c0631d0 con 0x7f806c06a750 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.525+0000 7f805f7fe700 1 -- 172.21.15.7:0/2783025522 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8060019070 con 0x7f806c06a750 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.526+0000 7f805f7fe700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f80580a3cd0 0x7f80580a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.526+0000 7f8071ce6700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f80580a3cd0 0x7f80580a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.526+0000 7f805f7fe700 1 -- 172.21.15.7:0/2783025522 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f80600d09a0 con 0x7f806c06a750 2024-04-21T08:43:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.526+0000 7f805f7fe700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687] conn(0x7f80580aee30 0x7f80580b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.526+0000 7f805f7fe700 1 -- 172.21.15.7:0/2783025522 --> [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80580b19a0 con 0x7f80580aee30 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.526+0000 7f805f7fe700 1 -- 172.21.15.7:0/2783025522 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f80600d0d50 con 0x7f806c06a750 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.526+0000 7f8072ce8700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687] conn(0x7f80580aee30 0x7f80580b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.526+0000 7f8071ce6700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f80580a3cd0 0x7f80580a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f806800b2e0 tx=0x7f8068005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.527+0000 7f8072ce8700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687] conn(0x7f80580aee30 0x7f80580b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.527+0000 7f805f7fe700 1 -- 172.21.15.7:0/2783025522 <== osd.6 v2:172.21.15.7:6818/3776602687 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80580b19a0 con 0x7f80580aee30 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.544+0000 7f807474b700 1 -- 172.21.15.7:0/2783025522 --> [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f806c0635d0 con 0x7f80580aee30 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.544+0000 7f805f7fe700 1 -- 172.21.15.7:0/2783025522 <== osd.6 v2:172.21.15.7:6818/3776602687 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f806c0635d0 con 0x7f80580aee30 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.544+0000 7f807474b700 1 -- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687] conn(0x7f80580aee30 msgr2=0x7f80580b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.544+0000 7f807474b700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687] conn(0x7f80580aee30 0x7f80580b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.545+0000 7f807474b700 1 -- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f80580a3cd0 msgr2=0x7f80580a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.545+0000 7f807474b700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f80580a3cd0 0x7f80580a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f806800b2e0 tx=0x7f8068005960 comp rx=0 tx=0).stop 2024-04-21T08:43:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.545+0000 7f807474b700 1 -- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f806c06a750 msgr2=0x7f806c199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.545+0000 7f807474b700 1 --2- 172.21.15.7:0/2783025522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f806c06a750 0x7f806c199120 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f806000d950 tx=0x7f806000dc60 comp rx=0 tx=0).stop 2024-04-21T08:43:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.545+0000 7f8072ce8700 1 -- 172.21.15.7:0/2783025522 reap_dead start 2024-04-21T08:43:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.545+0000 7f807474b700 1 -- 172.21.15.7:0/2783025522 shutdown_connections 2024-04-21T08:43:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.545+0000 7f807474b700 1 -- 172.21.15.7:0/2783025522 >> 172.21.15.7:0/2783025522 conn(0x7f806c0788f0 msgr2=0x7f806c0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.545+0000 7f807474b700 1 -- 172.21.15.7:0/2783025522 shutdown_connections 2024-04-21T08:43:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.545+0000 7f807474b700 1 -- 172.21.15.7:0/2783025522 wait complete. 2024-04-21T08:43:16.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:15 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:16.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:15 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:16.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:15 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:16.234 INFO:teuthology.orchestra.run.smithi007.stdout:201863462964 2024-04-21T08:43:16.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-21T08:43:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:16 smithi016 ceph-mon[31259]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-21T08:43:17.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:16 smithi007 ceph-mon[28477]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-21T08:43:17.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:16 smithi077 ceph-mon[31615]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-21T08:43:17.370 INFO:teuthology.orchestra.run.smithi007.stdout:34359738449 2024-04-21T08:43:17.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-21T08:43:17.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.614+0000 7f77f3571700 1 -- 172.21.15.7:0/1632086309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77ec102a90 msgr2=0x7f77ec102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:17.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.614+0000 7f77f3571700 1 --2- 172.21.15.7:0/1632086309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77ec102a90 0x7f77ec102f10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009990 tx=0x7f77dc009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.614+0000 7f77f3571700 1 -- 172.21.15.7:0/1632086309 shutdown_connections 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.614+0000 7f77f3571700 1 --2- 172.21.15.7:0/1632086309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77ec103450 0x7f77ec10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.614+0000 7f77f3571700 1 --2- 172.21.15.7:0/1632086309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ec108a80 0x7f77ec108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.614+0000 7f77f3571700 1 --2- 172.21.15.7:0/1632086309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77ec102a90 0x7f77ec102f10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.614+0000 7f77f3571700 1 -- 172.21.15.7:0/1632086309 >> 172.21.15.7:0/1632086309 conn(0x7f77ec079bf0 msgr2=0x7f77ec07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.614+0000 7f77f3571700 1 -- 172.21.15.7:0/1632086309 shutdown_connections 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.614+0000 7f77f3571700 1 -- 172.21.15.7:0/1632086309 wait complete. 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f3571700 1 Processor -- start 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f3571700 1 -- start start 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f3571700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ec102a90 0x7f77ec074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f3571700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77ec103450 0x7f77ec075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f3571700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77ec108a80 0x7f77ec0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:17.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f3571700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f77ec112c50 con 0x7f77ec108a80 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f3571700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77ec112ad0 con 0x7f77ec102a90 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f3571700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f77ec112dd0 con 0x7f77ec103450 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f1b0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77ec108a80 0x7f77ec0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f1b0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77ec108a80 0x7f77ec0757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53900/0 (socket says 172.21.15.7:53900) 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f1b0e700 1 -- 172.21.15.7:0/2207543886 learned_addr learned my addr 172.21.15.7:0/2207543886 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f0b0c700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77ec103450 0x7f77ec075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.615+0000 7f77f130d700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ec102a90 0x7f77ec074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77f1b0e700 1 -- 172.21.15.7:0/2207543886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77ec103450 msgr2=0x7f77ec075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77f1b0e700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77ec103450 0x7f77ec075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77f1b0e700 1 -- 172.21.15.7:0/2207543886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ec102a90 msgr2=0x7f77ec074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77f1b0e700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ec102a90 0x7f77ec074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77f1b0e700 1 -- 172.21.15.7:0/2207543886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77ec1a7650 con 0x7f77ec108a80 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77f130d700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77ec102a90 0x7f77ec074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:43:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77f1b0e700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77ec108a80 0x7f77ec0757c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f77e4007f10 tx=0x7f77e400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77e27fc700 1 -- 172.21.15.7:0/2207543886 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e4009e20 con 0x7f77ec108a80 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77f3571700 1 -- 172.21.15.7:0/2207543886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77ec1a7940 con 0x7f77ec108a80 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77e27fc700 1 -- 172.21.15.7:0/2207543886 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77e4004bb0 con 0x7f77ec108a80 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77e27fc700 1 -- 172.21.15.7:0/2207543886 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e40058e0 con 0x7f77ec108a80 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.616+0000 7f77f3571700 1 -- 172.21.15.7:0/2207543886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77ec1a7ea0 con 0x7f77ec108a80 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.617+0000 7f77f3571700 1 -- 172.21.15.7:0/2207543886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f77ec0630d0 con 0x7f77ec108a80 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.618+0000 7f77e27fc700 1 -- 172.21.15.7:0/2207543886 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f77e401a020 con 0x7f77ec108a80 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.619+0000 7f77e27fc700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f77d80a3cd0 0x7f77d80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.619+0000 7f77f130d700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f77d80a3cd0 0x7f77d80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.619+0000 7f77e27fc700 1 -- 172.21.15.7:0/2207543886 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f77e40d0660 con 0x7f77ec108a80 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.619+0000 7f77e27fc700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848] conn(0x7f77d80aee30 0x7f77d80b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.619+0000 7f77e27fc700 1 -- 172.21.15.7:0/2207543886 --> [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77d80b19a0 con 0x7f77d80aee30 2024-04-21T08:43:17.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.619+0000 7f77e27fc700 1 -- 172.21.15.7:0/2207543886 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f77e40d0a10 con 0x7f77ec108a80 2024-04-21T08:43:17.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.619+0000 7f77f130d700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f77d80a3cd0 0x7f77d80a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f77e8007f50 tx=0x7f77e80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:17.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.619+0000 7f77f0b0c700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848] conn(0x7f77d80aee30 0x7f77d80b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:17.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.620+0000 7f77f0b0c700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848] conn(0x7f77d80aee30 0x7f77d80b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:17.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.621+0000 7f77e27fc700 1 -- 172.21.15.7:0/2207543886 <== osd.4 v2:172.21.15.16:6808/849126848 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77d80b19a0 con 0x7f77d80aee30 2024-04-21T08:43:17.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.640+0000 7f77f3571700 1 -- 172.21.15.7:0/2207543886 --> [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f77ec063550 con 0x7f77d80aee30 2024-04-21T08:43:17.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77e27fc700 1 -- 172.21.15.7:0/2207543886 <== osd.4 v2:172.21.15.16:6808/849126848 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f77ec063550 con 0x7f77d80aee30 2024-04-21T08:43:17.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77f3571700 1 -- 172.21.15.7:0/2207543886 >> [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848] conn(0x7f77d80aee30 msgr2=0x7f77d80b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:17.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77f3571700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848] conn(0x7f77d80aee30 0x7f77d80b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77f3571700 1 -- 172.21.15.7:0/2207543886 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f77d80a3cd0 msgr2=0x7f77d80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:17.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77f3571700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f77d80a3cd0 0x7f77d80a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f77e8007f50 tx=0x7f77e80073d0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77f3571700 1 -- 172.21.15.7:0/2207543886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77ec108a80 msgr2=0x7f77ec0757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:17.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77f3571700 1 --2- 172.21.15.7:0/2207543886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77ec108a80 0x7f77ec0757c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f77e4007f10 tx=0x7f77e400d3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77f1b0e700 1 -- 172.21.15.7:0/2207543886 reap_dead start 2024-04-21T08:43:17.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77f3571700 1 -- 172.21.15.7:0/2207543886 shutdown_connections 2024-04-21T08:43:17.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77f3571700 1 -- 172.21.15.7:0/2207543886 >> 172.21.15.7:0/2207543886 conn(0x7f77ec079bf0 msgr2=0x7f77ec101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:17.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77f3571700 1 -- 172.21.15.7:0/2207543886 shutdown_connections 2024-04-21T08:43:17.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:14.641+0000 7f77f3571700 1 -- 172.21.15.7:0/2207543886 wait complete. 2024-04-21T08:43:18.043 INFO:teuthology.orchestra.run.smithi007.stdout:146028888126 2024-04-21T08:43:18.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-21T08:43:18.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.283+0000 7fbf9a326700 1 -- 172.21.15.7:0/975872904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf94102300 msgr2=0x7fbf9410e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.283+0000 7fbf9a326700 1 --2- 172.21.15.7:0/975872904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf94102300 0x7fbf9410e7f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00b0a0 tx=0x7fbf8c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.283+0000 7fbf9a326700 1 -- 172.21.15.7:0/975872904 shutdown_connections 2024-04-21T08:43:18.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.283+0000 7fbf9a326700 1 --2- 172.21.15.7:0/975872904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf94102300 0x7fbf9410e7f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.283+0000 7fbf9a326700 1 --2- 172.21.15.7:0/975872904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf94107990 0x7fbf94107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.283+0000 7fbf9a326700 1 --2- 172.21.15.7:0/975872904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94101940 0x7fbf94101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.283+0000 7fbf9a326700 1 -- 172.21.15.7:0/975872904 >> 172.21.15.7:0/975872904 conn(0x7fbf940788f0 msgr2=0x7fbf94078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.283+0000 7fbf9a326700 1 -- 172.21.15.7:0/975872904 shutdown_connections 2024-04-21T08:43:18.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.283+0000 7fbf9a326700 1 -- 172.21.15.7:0/975872904 wait complete. 2024-04-21T08:43:18.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.283+0000 7fbf9a326700 1 Processor -- start 2024-04-21T08:43:18.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.283+0000 7fbf9a326700 1 -- start start 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf9a326700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf94101940 0x7fbf941034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf9a326700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94102300 0x7fbf94103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf9a326700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf94107990 0x7fbf94103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf9a326700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbf94111ae0 con 0x7fbf94107990 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf9a326700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf94111960 con 0x7fbf94101940 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf9a326700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbf94111c60 con 0x7fbf94102300 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf937fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94102300 0x7fbf94103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf93fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf94101940 0x7fbf941034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf988c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf94107990 0x7fbf94103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf93fff700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf94101940 0x7fbf941034f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59266/0 (socket says 172.21.15.7:59266) 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf93fff700 1 -- 172.21.15.7:0/1008991374 learned_addr learned my addr 172.21.15.7:0/1008991374 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf937fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94102300 0x7fbf94103a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:37398/0 (socket says 172.21.15.7:37398) 2024-04-21T08:43:18.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf93fff700 1 -- 172.21.15.7:0/1008991374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94102300 msgr2=0x7fbf94103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf93fff700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf94102300 0x7fbf94103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf93fff700 1 -- 172.21.15.7:0/1008991374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf94107990 msgr2=0x7fbf94103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf93fff700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf94107990 0x7fbf94103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf93fff700 1 -- 172.21.15.7:0/1008991374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf941a75b0 con 0x7fbf94101940 2024-04-21T08:43:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf988c3700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf94107990 0x7fbf94103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:43:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.284+0000 7fbf93fff700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf94101940 0x7fbf941034f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00d950 tx=0x7fbf7c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.285+0000 7fbf917fa700 1 -- 172.21.15.7:0/1008991374 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c016070 con 0x7fbf94101940 2024-04-21T08:43:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.285+0000 7fbf917fa700 1 -- 172.21.15.7:0/1008991374 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf7c00fd40 con 0x7fbf94101940 2024-04-21T08:43:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.285+0000 7fbf9a326700 1 -- 172.21.15.7:0/1008991374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf941a78a0 con 0x7fbf94101940 2024-04-21T08:43:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.285+0000 7fbf917fa700 1 -- 172.21.15.7:0/1008991374 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c00e3b0 con 0x7fbf94101940 2024-04-21T08:43:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.285+0000 7fbf9a326700 1 -- 172.21.15.7:0/1008991374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf941a7de0 con 0x7fbf94101940 2024-04-21T08:43:18.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.286+0000 7fbf9a326700 1 -- 172.21.15.7:0/1008991374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbf940630d0 con 0x7fbf94101940 2024-04-21T08:43:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.287+0000 7fbf917fa700 1 -- 172.21.15.7:0/1008991374 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fbf7c019070 con 0x7fbf94101940 2024-04-21T08:43:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.287+0000 7fbf917fa700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbf800a3cd0 0x7fbf800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.287+0000 7fbf917fa700 1 -- 172.21.15.7:0/1008991374 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbf7c0d0840 con 0x7fbf94101940 2024-04-21T08:43:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.287+0000 7fbf937fe700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbf800a3cd0 0x7fbf800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.288+0000 7fbf917fa700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303] conn(0x7fbf800aee30 0x7fbf800b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.288+0000 7fbf917fa700 1 -- 172.21.15.7:0/1008991374 --> [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf7c013ec0 con 0x7fbf800aee30 2024-04-21T08:43:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.288+0000 7fbf917fa700 1 -- 172.21.15.7:0/1008991374 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fbf7c0d0c90 con 0x7fbf94101940 2024-04-21T08:43:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.288+0000 7fbf988c3700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303] conn(0x7fbf800aee30 0x7fbf800b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.288+0000 7fbf937fe700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbf800a3cd0 0x7fbf800a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbf94079030 tx=0x7fbf8400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.288+0000 7fbf988c3700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303] conn(0x7fbf800aee30 0x7fbf800b12d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:18.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.289+0000 7fbf917fa700 1 -- 172.21.15.7:0/1008991374 <== osd.11 v2:172.21.15.77:6824/3994048303 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf7c013ec0 con 0x7fbf800aee30 2024-04-21T08:43:18.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.305+0000 7fbf9a326700 1 -- 172.21.15.7:0/1008991374 --> [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbf94063550 con 0x7fbf800aee30 2024-04-21T08:43:18.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.306+0000 7fbf917fa700 1 -- 172.21.15.7:0/1008991374 <== osd.11 v2:172.21.15.77:6824/3994048303 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbf94063550 con 0x7fbf800aee30 2024-04-21T08:43:18.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.306+0000 7fbf9a326700 1 -- 172.21.15.7:0/1008991374 >> [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303] conn(0x7fbf800aee30 msgr2=0x7fbf800b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.306+0000 7fbf9a326700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303] conn(0x7fbf800aee30 0x7fbf800b12d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:18.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.307+0000 7fbf9a326700 1 -- 172.21.15.7:0/1008991374 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbf800a3cd0 msgr2=0x7fbf800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.307+0000 7fbf9a326700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbf800a3cd0 0x7fbf800a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbf94079030 tx=0x7fbf8400b410 comp rx=0 tx=0).stop 2024-04-21T08:43:18.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.307+0000 7fbf9a326700 1 -- 172.21.15.7:0/1008991374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf94101940 msgr2=0x7fbf941034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:18.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.307+0000 7fbf9a326700 1 --2- 172.21.15.7:0/1008991374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf94101940 0x7fbf941034f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00d950 tx=0x7fbf7c00dc60 comp rx=0 tx=0).stop 2024-04-21T08:43:18.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.307+0000 7fbf988c3700 1 -- 172.21.15.7:0/1008991374 reap_dead start 2024-04-21T08:43:18.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.307+0000 7fbf9a326700 1 -- 172.21.15.7:0/1008991374 shutdown_connections 2024-04-21T08:43:18.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.307+0000 7fbf9a326700 1 -- 172.21.15.7:0/1008991374 >> 172.21.15.7:0/1008991374 conn(0x7fbf940788f0 msgr2=0x7fbf9406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:18.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.307+0000 7fbf9a326700 1 -- 172.21.15.7:0/1008991374 shutdown_connections 2024-04-21T08:43:18.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:18.307+0000 7fbf9a326700 1 -- 172.21.15.7:0/1008991374 wait complete. 2024-04-21T08:43:18.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:18.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:18.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:18.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:18 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:18 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:18.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:18.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:18.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:18.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:18 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:19.406 INFO:teuthology.orchestra.run.smithi007.stdout:347892351002 2024-04-21T08:43:19.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-21T08:43:19.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:19 smithi007 ceph-mon[28477]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:19.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:19 smithi016 ceph-mon[31259]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:19.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:19 smithi077 ceph-mon[31615]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:19.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.896+0000 7f9f37841700 1 -- 172.21.15.7:0/4044865760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f30101dc0 msgr2=0x7f9f3010f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.896+0000 7f9f37841700 1 --2- 172.21.15.7:0/4044865760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f30101dc0 0x7f9f3010f830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9f2800b0a0 tx=0x7f9f2800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.896+0000 7f9f37841700 1 -- 172.21.15.7:0/4044865760 shutdown_connections 2024-04-21T08:43:19.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.896+0000 7f9f37841700 1 --2- 172.21.15.7:0/4044865760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f30107b20 0x7f9f30107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.896+0000 7f9f37841700 1 --2- 172.21.15.7:0/4044865760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f30101dc0 0x7f9f3010f830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.896+0000 7f9f37841700 1 --2- 172.21.15.7:0/4044865760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f30101400 0x7f9f30101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.896+0000 7f9f37841700 1 -- 172.21.15.7:0/4044865760 >> 172.21.15.7:0/4044865760 conn(0x7f9f30076c60 msgr2=0x7f9f30077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.896+0000 7f9f37841700 1 -- 172.21.15.7:0/4044865760 shutdown_connections 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.896+0000 7f9f37841700 1 -- 172.21.15.7:0/4044865760 wait complete. 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f37841700 1 Processor -- start 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f37841700 1 -- start start 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f37841700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f30101400 0x7f9f301990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f37841700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f30101dc0 0x7f9f301995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f37841700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f30107b20 0x7f9f3019d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f37841700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9f30112cb0 con 0x7f9f30101dc0 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f37841700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f30112b30 con 0x7f9f30101400 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f37841700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9f30112e30 con 0x7f9f30107b20 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f34ddc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f30101dc0 0x7f9f301995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f34ddc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f30101dc0 0x7f9f301995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45054/0 (socket says 172.21.15.7:45054) 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f34ddc700 1 -- 172.21.15.7:0/3240122042 learned_addr learned my addr 172.21.15.7:0/3240122042 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f355dd700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f30101400 0x7f9f301990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f35dde700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f30107b20 0x7f9f3019d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f34ddc700 1 -- 172.21.15.7:0/3240122042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f30107b20 msgr2=0x7f9f3019d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f34ddc700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f30107b20 0x7f9f3019d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f34ddc700 1 -- 172.21.15.7:0/3240122042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f30101400 msgr2=0x7f9f301990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f34ddc700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f30101400 0x7f9f301990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.897+0000 7f9f34ddc700 1 -- 172.21.15.7:0/3240122042 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3019e0b0 con 0x7f9f30101dc0 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.898+0000 7f9f34ddc700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f30101dc0 0x7f9f301995e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000b530 tx=0x7f9f2000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.898+0000 7f9f267fc700 1 -- 172.21.15.7:0/3240122042 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f20015020 con 0x7f9f30101dc0 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.898+0000 7f9f37841700 1 -- 172.21.15.7:0/3240122042 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f3019e3a0 con 0x7f9f30101dc0 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.898+0000 7f9f267fc700 1 -- 172.21.15.7:0/3240122042 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f20011ce0 con 0x7f9f30101dc0 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.898+0000 7f9f37841700 1 -- 172.21.15.7:0/3240122042 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f301a5ca0 con 0x7f9f30101dc0 2024-04-21T08:43:19.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.898+0000 7f9f267fc700 1 -- 172.21.15.7:0/3240122042 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f20010dd0 con 0x7f9f30101dc0 2024-04-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.899+0000 7f9f37841700 1 -- 172.21.15.7:0/3240122042 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f14001030 con 0x7f9f30101dc0 2024-04-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.900+0000 7f9f267fc700 1 -- 172.21.15.7:0/3240122042 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9f20018020 con 0x7f9f30101dc0 2024-04-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.900+0000 7f9f267fc700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9f1c0a3d50 0x7f9f1c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.901+0000 7f9f355dd700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9f1c0a3d50 0x7f9f1c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.901+0000 7f9f267fc700 1 -- 172.21.15.7:0/3240122042 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f9f200d0480 con 0x7f9f30101dc0 2024-04-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.901+0000 7f9f267fc700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202] conn(0x7f9f1c0aeeb0 0x7f9f1c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.901+0000 7f9f267fc700 1 -- 172.21.15.7:0/3240122042 --> [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f2001ce70 con 0x7f9f1c0aeeb0 2024-04-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.901+0000 7f9f267fc700 1 -- 172.21.15.7:0/3240122042 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f9f200d0830 con 0x7f9f30101dc0 2024-04-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.901+0000 7f9f355dd700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9f1c0a3d50 0x7f9f1c0a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9f300781e0 tx=0x7f9f2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.901+0000 7f9f35dde700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202] conn(0x7f9f1c0aeeb0 0x7f9f1c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.902+0000 7f9f35dde700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202] conn(0x7f9f1c0aeeb0 0x7f9f1c0b1350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.902+0000 7f9f267fc700 1 -- 172.21.15.7:0/3240122042 <== osd.1 v2:172.21.15.16:6800/1911240202 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9f2001ce70 con 0x7f9f1c0aeeb0 2024-04-21T08:43:19.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.919+0000 7f9f37841700 1 -- 172.21.15.7:0/3240122042 --> [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9f14002e60 con 0x7f9f1c0aeeb0 2024-04-21T08:43:19.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.920+0000 7f9f267fc700 1 -- 172.21.15.7:0/3240122042 <== osd.1 v2:172.21.15.16:6800/1911240202 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9f14002e60 con 0x7f9f1c0aeeb0 2024-04-21T08:43:19.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.920+0000 7f9f37841700 1 -- 172.21.15.7:0/3240122042 >> [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202] conn(0x7f9f1c0aeeb0 msgr2=0x7f9f1c0b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.920+0000 7f9f37841700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202] conn(0x7f9f1c0aeeb0 0x7f9f1c0b1350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.920+0000 7f9f37841700 1 -- 172.21.15.7:0/3240122042 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9f1c0a3d50 msgr2=0x7f9f1c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.920+0000 7f9f37841700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9f1c0a3d50 0x7f9f1c0a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9f300781e0 tx=0x7f9f2c00b410 comp rx=0 tx=0).stop 2024-04-21T08:43:19.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.920+0000 7f9f37841700 1 -- 172.21.15.7:0/3240122042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f30101dc0 msgr2=0x7f9f301995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.920+0000 7f9f37841700 1 --2- 172.21.15.7:0/3240122042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f30101dc0 0x7f9f301995e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000b530 tx=0x7f9f2000b840 comp rx=0 tx=0).stop 2024-04-21T08:43:19.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.920+0000 7f9f35dde700 1 -- 172.21.15.7:0/3240122042 reap_dead start 2024-04-21T08:43:19.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.920+0000 7f9f37841700 1 -- 172.21.15.7:0/3240122042 shutdown_connections 2024-04-21T08:43:19.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.920+0000 7f9f37841700 1 -- 172.21.15.7:0/3240122042 >> 172.21.15.7:0/3240122042 conn(0x7f9f30076c60 msgr2=0x7f9f300ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.921+0000 7f9f37841700 1 -- 172.21.15.7:0/3240122042 shutdown_connections 2024-04-21T08:43:19.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:19.921+0000 7f9f37841700 1 -- 172.21.15.7:0/3240122042 wait complete. 2024-04-21T08:43:20.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.147+0000 7fa6fb547700 1 -- 172.21.15.7:0/2381198676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f4107a60 msgr2=0x7fa6f4107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.147+0000 7fa6fb547700 1 --2- 172.21.15.7:0/2381198676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f4107a60 0x7fa6f4107e40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa6f00099c0 tx=0x7fa6f0009cd0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.147+0000 7fa6fb547700 1 -- 172.21.15.7:0/2381198676 shutdown_connections 2024-04-21T08:43:20.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.147+0000 7fa6fb547700 1 --2- 172.21.15.7:0/2381198676 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6f4101950 0x7fa6f410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.147+0000 7fa6fb547700 1 --2- 172.21.15.7:0/2381198676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f4107a60 0x7fa6f4107e40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.147+0000 7fa6fb547700 1 --2- 172.21.15.7:0/2381198676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f4100f90 0x7fa6f4101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.147+0000 7fa6fb547700 1 -- 172.21.15.7:0/2381198676 >> 172.21.15.7:0/2381198676 conn(0x7fa6f4077f40 msgr2=0x7fa6f40ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.147+0000 7fa6fb547700 1 -- 172.21.15.7:0/2381198676 shutdown_connections 2024-04-21T08:43:20.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.147+0000 7fa6fb547700 1 -- 172.21.15.7:0/2381198676 wait complete. 2024-04-21T08:43:20.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6fb547700 1 Processor -- start 2024-04-21T08:43:20.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6fb547700 1 -- start start 2024-04-21T08:43:20.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6fb547700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f4100f90 0x7fa6f4199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6fb547700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6f4101950 0x7fa6f4199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6fb547700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f4107a60 0x7fa6f419d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6fb547700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f4112d30 con 0x7fa6f4107a60 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6fb547700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f4112bb0 con 0x7fa6f4100f90 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6fb547700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f4112eb0 con 0x7fa6f4101950 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6f9ae4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f4107a60 0x7fa6f419d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6f9ae4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f4107a60 0x7fa6f419d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45072/0 (socket says 172.21.15.7:45072) 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6f9ae4700 1 -- 172.21.15.7:0/3269652462 learned_addr learned my addr 172.21.15.7:0/3269652462 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6f92e3700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f4100f90 0x7fa6f4199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6f8ae2700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6f4101950 0x7fa6f4199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6f9ae4700 1 -- 172.21.15.7:0/3269652462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6f4101950 msgr2=0x7fa6f4199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6f9ae4700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6f4101950 0x7fa6f4199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6f9ae4700 1 -- 172.21.15.7:0/3269652462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f4100f90 msgr2=0x7fa6f4199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6f9ae4700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6f4100f90 0x7fa6f4199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6f9ae4700 1 -- 172.21.15.7:0/3269652462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f419e110 con 0x7fa6f4107a60 2024-04-21T08:43:20.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.148+0000 7fa6f8ae2700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6f4101950 0x7fa6f4199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:20.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.149+0000 7fa6f9ae4700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f4107a60 0x7fa6f419d9d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec007f10 tx=0x7fa6ec00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.149+0000 7fa6ea7fc700 1 -- 172.21.15.7:0/3269652462 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec00ddd0 con 0x7fa6f4107a60 2024-04-21T08:43:20.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.149+0000 7fa6fb547700 1 -- 172.21.15.7:0/3269652462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f419e400 con 0x7fa6f4107a60 2024-04-21T08:43:20.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.149+0000 7fa6ea7fc700 1 -- 172.21.15.7:0/3269652462 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6ec004020 con 0x7fa6f4107a60 2024-04-21T08:43:20.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.149+0000 7fa6ea7fc700 1 -- 172.21.15.7:0/3269652462 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec015660 con 0x7fa6f4107a60 2024-04-21T08:43:20.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.149+0000 7fa6fb547700 1 -- 172.21.15.7:0/3269652462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f41a5c80 con 0x7fa6f4107a60 2024-04-21T08:43:20.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.150+0000 7fa6fb547700 1 -- 172.21.15.7:0/3269652462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa6f4063150 con 0x7fa6f4107a60 2024-04-21T08:43:20.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.151+0000 7fa6ea7fc700 1 -- 172.21.15.7:0/3269652462 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fa6ec0041d0 con 0x7fa6f4107a60 2024-04-21T08:43:20.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.152+0000 7fa6ea7fc700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa6e00a3cd0 0x7fa6e00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.152+0000 7fa6f92e3700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa6e00a3cd0 0x7fa6e00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.152+0000 7fa6ea7fc700 1 -- 172.21.15.7:0/3269652462 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fa6ec0d0e70 con 0x7fa6f4107a60 2024-04-21T08:43:20.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.152+0000 7fa6ea7fc700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881] conn(0x7fa6e00aee30 0x7fa6e00b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.152+0000 7fa6ea7fc700 1 -- 172.21.15.7:0/3269652462 --> [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6ec00be50 con 0x7fa6e00aee30 2024-04-21T08:43:20.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.152+0000 7fa6ea7fc700 1 -- 172.21.15.7:0/3269652462 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fa6ec0d1220 con 0x7fa6f4107a60 2024-04-21T08:43:20.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.152+0000 7fa6f92e3700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa6e00a3cd0 0x7fa6e00a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009990 tx=0x7fa6f000b790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.152+0000 7fa6f8ae2700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881] conn(0x7fa6e00aee30 0x7fa6e00b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.153+0000 7fa6f8ae2700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881] conn(0x7fa6e00aee30 0x7fa6e00b12d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.153+0000 7fa6ea7fc700 1 -- 172.21.15.7:0/3269652462 <== osd.7 v2:172.21.15.16:6816/2331386881 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa6ec00be50 con 0x7fa6e00aee30 2024-04-21T08:43:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.170+0000 7fa6fb547700 1 -- 172.21.15.7:0/3269652462 --> [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa6f4063550 con 0x7fa6e00aee30 2024-04-21T08:43:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.170+0000 7fa6ea7fc700 1 -- 172.21.15.7:0/3269652462 <== osd.7 v2:172.21.15.16:6816/2331386881 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa6f4063550 con 0x7fa6e00aee30 2024-04-21T08:43:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.171+0000 7fa6fb547700 1 -- 172.21.15.7:0/3269652462 >> [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881] conn(0x7fa6e00aee30 msgr2=0x7fa6e00b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.171+0000 7fa6fb547700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881] conn(0x7fa6e00aee30 0x7fa6e00b12d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.171+0000 7fa6fb547700 1 -- 172.21.15.7:0/3269652462 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa6e00a3cd0 msgr2=0x7fa6e00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.171+0000 7fa6fb547700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa6e00a3cd0 0x7fa6e00a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009990 tx=0x7fa6f000b790 comp rx=0 tx=0).stop 2024-04-21T08:43:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.171+0000 7fa6fb547700 1 -- 172.21.15.7:0/3269652462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f4107a60 msgr2=0x7fa6f419d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.171+0000 7fa6fb547700 1 --2- 172.21.15.7:0/3269652462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f4107a60 0x7fa6f419d9d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec007f10 tx=0x7fa6ec00d3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.171+0000 7fa6f9ae4700 1 -- 172.21.15.7:0/3269652462 reap_dead start 2024-04-21T08:43:20.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.171+0000 7fa6fb547700 1 -- 172.21.15.7:0/3269652462 shutdown_connections 2024-04-21T08:43:20.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.171+0000 7fa6fb547700 1 -- 172.21.15.7:0/3269652462 >> 172.21.15.7:0/3269652462 conn(0x7fa6f4077f40 msgr2=0x7fa6f4105b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.171+0000 7fa6fb547700 1 -- 172.21.15.7:0/3269652462 shutdown_connections 2024-04-21T08:43:20.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:20.171+0000 7fa6fb547700 1 -- 172.21.15.7:0/3269652462 wait complete. 2024-04-21T08:43:20.838 INFO:teuthology.orchestra.run.smithi007.stdout:55834574926 2024-04-21T08:43:20.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-21T08:43:21.756 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:21 smithi007 ceph-mon[28477]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-21T08:43:21.787 INFO:teuthology.orchestra.run.smithi007.stdout:167503724600 2024-04-21T08:43:21.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-21T08:43:21.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:21 smithi016 ceph-mon[31259]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-21T08:43:21.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:21 smithi077 ceph-mon[31615]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-21T08:43:21.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.985+0000 7f13e7555700 1 -- 172.21.15.7:0/3852394660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e006b980 msgr2=0x7f13e010f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.985+0000 7f13e7555700 1 --2- 172.21.15.7:0/3852394660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e006b980 0x7f13e010f8f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f13d800b0a0 tx=0x7f13d800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.985+0000 7f13e7555700 1 -- 172.21.15.7:0/3852394660 shutdown_connections 2024-04-21T08:43:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.985+0000 7f13e7555700 1 --2- 172.21.15.7:0/3852394660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e006b980 0x7f13e010f8f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.985+0000 7f13e7555700 1 --2- 172.21.15.7:0/3852394660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e0108a20 0x7f13e0108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.985+0000 7f13e7555700 1 --2- 172.21.15.7:0/3852394660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e006afe0 0x7f13e006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.985+0000 7f13e7555700 1 -- 172.21.15.7:0/3852394660 >> 172.21.15.7:0/3852394660 conn(0x7f13e00788b0 msgr2=0x7f13e0078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.985+0000 7f13e7555700 1 -- 172.21.15.7:0/3852394660 shutdown_connections 2024-04-21T08:43:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.985+0000 7f13e7555700 1 -- 172.21.15.7:0/3852394660 wait complete. 2024-04-21T08:43:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.986+0000 7f13e7555700 1 Processor -- start 2024-04-21T08:43:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.986+0000 7f13e7555700 1 -- start start 2024-04-21T08:43:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.987+0000 7f13e7555700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e006afe0 0x7f13e0101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.987+0000 7f13e7555700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e006b980 0x7f13e0101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.987+0000 7f13e7555700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e0108a20 0x7f13e0102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.987+0000 7f13e7555700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f13e0112be0 con 0x7f13e006afe0 2024-04-21T08:43:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.987+0000 7f13e7555700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13e0112a60 con 0x7f13e006b980 2024-04-21T08:43:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.987+0000 7f13e52f1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e006afe0 0x7f13e0101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.987+0000 7f13e4af0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e006b980 0x7f13e0101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.987+0000 7f13e4af0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e006b980 0x7f13e0101f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59312/0 (socket says 172.21.15.7:59312) 2024-04-21T08:43:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.987+0000 7f13e4af0700 1 -- 172.21.15.7:0/1627993507 learned_addr learned my addr 172.21.15.7:0/1627993507 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.987+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f13e0112d60 con 0x7f13e0108a20 2024-04-21T08:43:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.987+0000 7f13e5af2700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e0108a20 0x7f13e0102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.988+0000 7f13e52f1700 1 -- 172.21.15.7:0/1627993507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e0108a20 msgr2=0x7f13e0102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.988+0000 7f13e52f1700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13e0108a20 0x7f13e0102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.988+0000 7f13e52f1700 1 -- 172.21.15.7:0/1627993507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e006b980 msgr2=0x7f13e0101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.988+0000 7f13e52f1700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13e006b980 0x7f13e0101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.988+0000 7f13e52f1700 1 -- 172.21.15.7:0/1627993507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e01a7600 con 0x7f13e006afe0 2024-04-21T08:43:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.988+0000 7f13e52f1700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e006afe0 0x7f13e0101a00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00e9f0 tx=0x7f13dc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.988+0000 7f13d67fc700 1 -- 172.21.15.7:0/1627993507 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc00c9c0 con 0x7f13e006afe0 2024-04-21T08:43:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.988+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e01a78f0 con 0x7f13e006afe0 2024-04-21T08:43:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.988+0000 7f13d67fc700 1 -- 172.21.15.7:0/1627993507 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13dc00cb60 con 0x7f13e006afe0 2024-04-21T08:43:21.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.988+0000 7f13d67fc700 1 -- 172.21.15.7:0/1627993507 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc018680 con 0x7f13e006afe0 2024-04-21T08:43:21.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.988+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e01a7e80 con 0x7f13e006afe0 2024-04-21T08:43:21.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.989+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f13e0063110 con 0x7f13e006afe0 2024-04-21T08:43:21.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.991+0000 7f13d67fc700 1 -- 172.21.15.7:0/1627993507 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f13dc010430 con 0x7f13e006afe0 2024-04-21T08:43:21.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.992+0000 7f13d67fc700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f13cc0a3cd0 0x7f13cc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:21.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.992+0000 7f13e4af0700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f13cc0a3cd0 0x7f13cc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:21.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.992+0000 7f13d67fc700 1 -- 172.21.15.7:0/1627993507 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f13dc0d1510 con 0x7f13e006afe0 2024-04-21T08:43:21.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.992+0000 7f13d67fc700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333] conn(0x7f13cc0aee30 0x7f13cc0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:21.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.992+0000 7f13d67fc700 1 -- 172.21.15.7:0/1627993507 --> [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13dc016ec0 con 0x7f13cc0aee30 2024-04-21T08:43:21.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.992+0000 7f13d67fc700 1 -- 172.21.15.7:0/1627993507 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f13dc0d1960 con 0x7f13e006afe0 2024-04-21T08:43:21.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.993+0000 7f13e5af2700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333] conn(0x7f13cc0aee30 0x7f13cc0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:21.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.993+0000 7f13e4af0700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f13cc0a3cd0 0x7f13cc0a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f13e0078e90 tx=0x7f13d000b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:21.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.993+0000 7f13e5af2700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333] conn(0x7f13cc0aee30 0x7f13cc0b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:21.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:21.994+0000 7f13d67fc700 1 -- 172.21.15.7:0/1627993507 <== osd.10 v2:172.21.15.16:6824/368172333 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f13dc016ec0 con 0x7f13cc0aee30 2024-04-21T08:43:22.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.013+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 --> [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f13e0063540 con 0x7f13cc0aee30 2024-04-21T08:43:22.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.013+0000 7f13d67fc700 1 -- 172.21.15.7:0/1627993507 <== osd.10 v2:172.21.15.16:6824/368172333 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f13e0063540 con 0x7f13cc0aee30 2024-04-21T08:43:22.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.014+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 >> [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333] conn(0x7f13cc0aee30 msgr2=0x7f13cc0b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:22.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.014+0000 7f13e7555700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333] conn(0x7f13cc0aee30 0x7f13cc0b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:22.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.014+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f13cc0a3cd0 msgr2=0x7f13cc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:22.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.014+0000 7f13e7555700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f13cc0a3cd0 0x7f13cc0a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f13e0078e90 tx=0x7f13d000b480 comp rx=0 tx=0).stop 2024-04-21T08:43:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.014+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e006afe0 msgr2=0x7f13e0101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.014+0000 7f13e7555700 1 --2- 172.21.15.7:0/1627993507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e006afe0 0x7f13e0101a00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00e9f0 tx=0x7f13dc00ed00 comp rx=0 tx=0).stop 2024-04-21T08:43:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.014+0000 7f13e5af2700 1 -- 172.21.15.7:0/1627993507 reap_dead start 2024-04-21T08:43:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.014+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 shutdown_connections 2024-04-21T08:43:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.014+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 >> 172.21.15.7:0/1627993507 conn(0x7f13e00788b0 msgr2=0x7f13e00ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.014+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 shutdown_connections 2024-04-21T08:43:22.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:22.014+0000 7f13e7555700 1 -- 172.21.15.7:0/1627993507 wait complete. 2024-04-21T08:43:22.783 INFO:teuthology.orchestra.run.smithi007.stdout:223338299439 2024-04-21T08:43:22.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-21T08:43:23.347 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:23.347 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:23.351 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:23.770 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:23 smithi007 ceph-mon[28477]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-21T08:43:23.802 INFO:teuthology.orchestra.run.smithi007.stdout:313532612640 2024-04-21T08:43:23.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-21T08:43:23.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:23 smithi016 ceph-mon[31259]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-21T08:43:23.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:23 smithi077 ceph-mon[31615]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-21T08:43:24.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.314+0000 7f39a71e5700 1 -- 172.21.15.7:0/3093069394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39a0108a50 msgr2=0x7f39a0108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.314+0000 7f39a71e5700 1 --2- 172.21.15.7:0/3093069394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39a0108a50 0x7f39a0108e30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3994009990 tx=0x7f3994009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.314+0000 7f39a71e5700 1 -- 172.21.15.7:0/3093069394 shutdown_connections 2024-04-21T08:43:24.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.314+0000 7f39a71e5700 1 --2- 172.21.15.7:0/3093069394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39a0103450 0x7f39a010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.314+0000 7f39a71e5700 1 --2- 172.21.15.7:0/3093069394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39a0108a50 0x7f39a0108e30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.314+0000 7f39a71e5700 1 --2- 172.21.15.7:0/3093069394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39a0102a90 0x7f39a0102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.314+0000 7f39a71e5700 1 -- 172.21.15.7:0/3093069394 >> 172.21.15.7:0/3093069394 conn(0x7f39a0079ba0 msgr2=0x7f39a0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:24.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.314+0000 7f39a71e5700 1 -- 172.21.15.7:0/3093069394 shutdown_connections 2024-04-21T08:43:24.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.314+0000 7f39a71e5700 1 -- 172.21.15.7:0/3093069394 wait complete. 2024-04-21T08:43:24.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.314+0000 7f39a71e5700 1 Processor -- start 2024-04-21T08:43:24.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.314+0000 7f39a71e5700 1 -- start start 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a71e5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39a0102a90 0x7f39a0199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a71e5700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39a0103450 0x7f39a0199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a71e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39a0108a50 0x7f39a019d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a71e5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f39a0112c00 con 0x7f39a0108a50 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a71e5700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f39a0112a80 con 0x7f39a0103450 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a71e5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f39a0112d80 con 0x7f39a0102a90 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a5782700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39a0108a50 0x7f39a019d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a5782700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39a0108a50 0x7f39a019d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45100/0 (socket says 172.21.15.7:45100) 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a5782700 1 -- 172.21.15.7:0/3524528733 learned_addr learned my addr 172.21.15.7:0/3524528733 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a4f81700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39a0102a90 0x7f39a0199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f399ffff700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39a0103450 0x7f39a0199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a5782700 1 -- 172.21.15.7:0/3524528733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39a0102a90 msgr2=0x7f39a0199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a5782700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39a0102a90 0x7f39a0199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a5782700 1 -- 172.21.15.7:0/3524528733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39a0103450 msgr2=0x7f39a0199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a5782700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39a0103450 0x7f39a0199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a5782700 1 -- 172.21.15.7:0/3524528733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a019e120 con 0x7f39a0108a50 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.315+0000 7f39a5782700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39a0108a50 0x7f39a019d9e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3998007ee0 tx=0x7f399800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.316+0000 7f399dffb700 1 -- 172.21.15.7:0/3524528733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3998004bb0 con 0x7f39a0108a50 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.316+0000 7f399dffb700 1 -- 172.21.15.7:0/3524528733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3998004d50 con 0x7f39a0108a50 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.316+0000 7f39a71e5700 1 -- 172.21.15.7:0/3524528733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a019e410 con 0x7f39a0108a50 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.316+0000 7f399dffb700 1 -- 172.21.15.7:0/3524528733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3998005920 con 0x7f39a0108a50 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.316+0000 7f39a71e5700 1 -- 172.21.15.7:0/3524528733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a01a5c90 con 0x7f39a0108a50 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.317+0000 7f39a71e5700 1 -- 172.21.15.7:0/3524528733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f39a00631d0 con 0x7f39a0108a50 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.318+0000 7f399dffb700 1 -- 172.21.15.7:0/3524528733 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f399801a020 con 0x7f39a0108a50 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.318+0000 7f399dffb700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f39900a3d50 0x7f39900a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.319+0000 7f399dffb700 1 -- 172.21.15.7:0/3524528733 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f39980d0560 con 0x7f39a0108a50 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.319+0000 7f39a4f81700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f39900a3d50 0x7f39900a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.319+0000 7f399dffb700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717] conn(0x7f39900aeeb0 0x7f39900b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.319+0000 7f399dffb700 1 -- 172.21.15.7:0/3524528733 --> [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39900b1a20 con 0x7f39900aeeb0 2024-04-21T08:43:24.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.319+0000 7f399dffb700 1 -- 172.21.15.7:0/3524528733 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f39980d09b0 con 0x7f39a0108a50 2024-04-21T08:43:24.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.319+0000 7f39a4f81700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f39900a3d50 0x7f39900a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f39940098e0 tx=0x7f3994005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:24.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.319+0000 7f399ffff700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717] conn(0x7f39900aeeb0 0x7f39900b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.319+0000 7f399ffff700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717] conn(0x7f39900aeeb0 0x7f39900b1350 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:24.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.320+0000 7f399dffb700 1 -- 172.21.15.7:0/3524528733 <== osd.8 v2:172.21.15.77:6816/3684606717 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39900b1a20 con 0x7f39900aeeb0 2024-04-21T08:43:24.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.338+0000 7f39a71e5700 1 -- 172.21.15.7:0/3524528733 --> [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f39a0063650 con 0x7f39900aeeb0 2024-04-21T08:43:24.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.338+0000 7f399dffb700 1 -- 172.21.15.7:0/3524528733 <== osd.8 v2:172.21.15.77:6816/3684606717 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f39a0063650 con 0x7f39900aeeb0 2024-04-21T08:43:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.339+0000 7f39a71e5700 1 -- 172.21.15.7:0/3524528733 >> [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717] conn(0x7f39900aeeb0 msgr2=0x7f39900b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.339+0000 7f39a71e5700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717] conn(0x7f39900aeeb0 0x7f39900b1350 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.339+0000 7f39a71e5700 1 -- 172.21.15.7:0/3524528733 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f39900a3d50 msgr2=0x7f39900a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.339+0000 7f39a71e5700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f39900a3d50 0x7f39900a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f39940098e0 tx=0x7f3994005960 comp rx=0 tx=0).stop 2024-04-21T08:43:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.339+0000 7f39a71e5700 1 -- 172.21.15.7:0/3524528733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39a0108a50 msgr2=0x7f39a019d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.339+0000 7f39a71e5700 1 --2- 172.21.15.7:0/3524528733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39a0108a50 0x7f39a019d9e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3998007ee0 tx=0x7f399800d3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.339+0000 7f39a5782700 1 -- 172.21.15.7:0/3524528733 reap_dead start 2024-04-21T08:43:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.339+0000 7f39a71e5700 1 -- 172.21.15.7:0/3524528733 shutdown_connections 2024-04-21T08:43:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.339+0000 7f39a71e5700 1 -- 172.21.15.7:0/3524528733 >> 172.21.15.7:0/3524528733 conn(0x7f39a0079ba0 msgr2=0x7f39a0101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.339+0000 7f39a71e5700 1 -- 172.21.15.7:0/3524528733 shutdown_connections 2024-04-21T08:43:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:24.339+0000 7f39a71e5700 1 -- 172.21.15.7:0/3524528733 wait complete. 2024-04-21T08:43:25.036 INFO:teuthology.orchestra.run.smithi007.stdout:257698037803 2024-04-21T08:43:25.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-21T08:43:25.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:25 smithi007 ceph-mon[28477]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-04-21T08:43:25.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.691+0000 7f79c2a8a700 1 -- 172.21.15.7:0/416547394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79bc103450 msgr2=0x7f79bc10f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.691+0000 7f79c2a8a700 1 --2- 172.21.15.7:0/416547394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79bc103450 0x7f79bc10f8b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f79b400b0a0 tx=0x7f79b400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 -- 172.21.15.7:0/416547394 shutdown_connections 2024-04-21T08:43:25.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 --2- 172.21.15.7:0/416547394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79bc103450 0x7f79bc10f8b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 --2- 172.21.15.7:0/416547394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79bc108a50 0x7f79bc108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 --2- 172.21.15.7:0/416547394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79bc102a90 0x7f79bc102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 -- 172.21.15.7:0/416547394 >> 172.21.15.7:0/416547394 conn(0x7f79bc079ba0 msgr2=0x7f79bc079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:25.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 -- 172.21.15.7:0/416547394 shutdown_connections 2024-04-21T08:43:25.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 -- 172.21.15.7:0/416547394 wait complete. 2024-04-21T08:43:25.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 Processor -- start 2024-04-21T08:43:25.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 -- start start 2024-04-21T08:43:25.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79bc102a90 0x7f79bc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79bc103450 0x7f79bc075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79bc108a50 0x7f79bc0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.692+0000 7f79c2a8a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc112c00 con 0x7f79bc103450 2024-04-21T08:43:25.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79c2a8a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc112a80 con 0x7f79bc102a90 2024-04-21T08:43:25.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79c2a8a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc112d80 con 0x7f79bc108a50 2024-04-21T08:43:25.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79bc103450 0x7f79bc075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79bc103450 0x7f79bc075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38466/0 (socket says 172.21.15.7:38466) 2024-04-21T08:43:25.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79bbfff700 1 -- 172.21.15.7:0/1210395218 learned_addr learned my addr 172.21.15.7:0/1210395218 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:25.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79c1027700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79bc108a50 0x7f79bc0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79c0826700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79bc102a90 0x7f79bc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79bbfff700 1 -- 172.21.15.7:0/1210395218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79bc108a50 msgr2=0x7f79bc0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79bbfff700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79bc108a50 0x7f79bc0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79bbfff700 1 -- 172.21.15.7:0/1210395218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79bc102a90 msgr2=0x7f79bc074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79bbfff700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79bc102a90 0x7f79bc074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79bbfff700 1 -- 172.21.15.7:0/1210395218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79bc1a7570 con 0x7f79bc103450 2024-04-21T08:43:25.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79c1027700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79bc108a50 0x7f79bc0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:43:25.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.693+0000 7f79bbfff700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79bc103450 0x7f79bc075280 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f79a800c9b0 tx=0x7f79a800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:25.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.694+0000 7f79b9ffb700 1 -- 172.21.15.7:0/1210395218 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a8007b00 con 0x7f79bc103450 2024-04-21T08:43:25.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.694+0000 7f79b9ffb700 1 -- 172.21.15.7:0/1210395218 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79a800ce50 con 0x7f79bc103450 2024-04-21T08:43:25.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.694+0000 7f79b9ffb700 1 -- 172.21.15.7:0/1210395218 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a8018660 con 0x7f79bc103450 2024-04-21T08:43:25.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.694+0000 7f79c2a8a700 1 -- 172.21.15.7:0/1210395218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79bc1a7860 con 0x7f79bc103450 2024-04-21T08:43:25.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.694+0000 7f79c2a8a700 1 -- 172.21.15.7:0/1210395218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79bc1a7d50 con 0x7f79bc103450 2024-04-21T08:43:25.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.696+0000 7f79c2a8a700 1 -- 172.21.15.7:0/1210395218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f79bc0630d0 con 0x7f79bc103450 2024-04-21T08:43:25.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.698+0000 7f79b9ffb700 1 -- 172.21.15.7:0/1210395218 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f79a800f450 con 0x7f79bc103450 2024-04-21T08:43:25.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.699+0000 7f79b9ffb700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79ac0a3ea0 0x7f79ac0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.699+0000 7f79b9ffb700 1 -- 172.21.15.7:0/1210395218 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f79a80d0b80 con 0x7f79bc103450 2024-04-21T08:43:25.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.699+0000 7f79c0826700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79ac0a3ea0 0x7f79ac0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.699+0000 7f79b9ffb700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623] conn(0x7f79ac0af000 0x7f79ac0b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.699+0000 7f79b9ffb700 1 -- 172.21.15.7:0/1210395218 --> [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79a800c120 con 0x7f79ac0af000 2024-04-21T08:43:25.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.699+0000 7f79b9ffb700 1 -- 172.21.15.7:0/1210395218 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f79a80d0f70 con 0x7f79bc103450 2024-04-21T08:43:25.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.700+0000 7f79c1027700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623] conn(0x7f79ac0af000 0x7f79ac0b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.700+0000 7f79c0826700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79ac0a3ea0 0x7f79ac0a6360 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009be0 tx=0x7f79b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:25.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.700+0000 7f79c1027700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623] conn(0x7f79ac0af000 0x7f79ac0b14a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:25.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.701+0000 7f79b9ffb700 1 -- 172.21.15.7:0/1210395218 <== osd.2 v2:172.21.15.77:6800/1494490623 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f79a800c120 con 0x7f79ac0af000 2024-04-21T08:43:25.706 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:25.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.718+0000 7f79c2a8a700 1 -- 172.21.15.7:0/1210395218 --> [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f79bc0635a0 con 0x7f79ac0af000 2024-04-21T08:43:25.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.719+0000 7f79b9ffb700 1 -- 172.21.15.7:0/1210395218 <== osd.2 v2:172.21.15.77:6800/1494490623 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f79bc0635a0 con 0x7f79ac0af000 2024-04-21T08:43:25.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.719+0000 7f79c2a8a700 1 -- 172.21.15.7:0/1210395218 >> [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623] conn(0x7f79ac0af000 msgr2=0x7f79ac0b14a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.719+0000 7f79c2a8a700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623] conn(0x7f79ac0af000 0x7f79ac0b14a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.720+0000 7f79c2a8a700 1 -- 172.21.15.7:0/1210395218 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79ac0a3ea0 msgr2=0x7f79ac0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.720+0000 7f79c2a8a700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79ac0a3ea0 0x7f79ac0a6360 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009be0 tx=0x7f79b0009260 comp rx=0 tx=0).stop 2024-04-21T08:43:25.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.720+0000 7f79c2a8a700 1 -- 172.21.15.7:0/1210395218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79bc103450 msgr2=0x7f79bc075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.720+0000 7f79c2a8a700 1 --2- 172.21.15.7:0/1210395218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79bc103450 0x7f79bc075280 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f79a800c9b0 tx=0x7f79a800ccc0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.720+0000 7f79c1027700 1 -- 172.21.15.7:0/1210395218 reap_dead start 2024-04-21T08:43:25.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.720+0000 7f79c2a8a700 1 -- 172.21.15.7:0/1210395218 shutdown_connections 2024-04-21T08:43:25.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.720+0000 7f79c2a8a700 1 -- 172.21.15.7:0/1210395218 >> 172.21.15.7:0/1210395218 conn(0x7f79bc079ba0 msgr2=0x7f79bc101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:25.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.720+0000 7f79c2a8a700 1 -- 172.21.15.7:0/1210395218 shutdown_connections 2024-04-21T08:43:25.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:25.720+0000 7f79c2a8a700 1 -- 172.21.15.7:0/1210395218 wait complete. 2024-04-21T08:43:25.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:25 smithi016 ceph-mon[31259]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-04-21T08:43:25.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:25 smithi077 ceph-mon[31615]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-04-21T08:43:26.360 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:26.360 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:26.360 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:27.705 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:27 smithi007 ceph-mon[28477]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-04-21T08:43:27.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:27 smithi016 ceph-mon[31259]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-04-21T08:43:27.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:27 smithi077 ceph-mon[31615]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-04-21T08:43:28.600 INFO:teuthology.orchestra.run.smithi007.stdout:77309411402 2024-04-21T08:43:28.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-21T08:43:29.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:29 smithi007 ceph-mon[28477]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:29 smithi016 ceph-mon[31259]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:29.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:29 smithi077 ceph-mon[31615]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:30.554 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:31.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:31 smithi007 ceph-mon[28477]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:31.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:31 smithi016 ceph-mon[31259]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:31.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:31 smithi077 ceph-mon[31615]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:33.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.232+0000 7f8c308eb700 1 -- 172.21.15.7:0/1277961412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c2c108c20 msgr2=0x7f8c2c109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:33.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.232+0000 7f8c308eb700 1 --2- 172.21.15.7:0/1277961412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c2c108c20 0x7f8c2c109000 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c00b0a0 tx=0x7f8c1c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.232+0000 7f8c308eb700 1 -- 172.21.15.7:0/1277961412 shutdown_connections 2024-04-21T08:43:33.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.232+0000 7f8c308eb700 1 --2- 172.21.15.7:0/1277961412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c2c101e30 0x7f8c2c10f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.232+0000 7f8c308eb700 1 --2- 172.21.15.7:0/1277961412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c2c108c20 0x7f8c2c109000 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.232+0000 7f8c308eb700 1 --2- 172.21.15.7:0/1277961412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c2c101470 0x7f8c2c1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.232+0000 7f8c308eb700 1 -- 172.21.15.7:0/1277961412 >> 172.21.15.7:0/1277961412 conn(0x7f8c2c076bb0 msgr2=0x7f8c2c076fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:33.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.232+0000 7f8c308eb700 1 -- 172.21.15.7:0/1277961412 shutdown_connections 2024-04-21T08:43:33.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.232+0000 7f8c308eb700 1 -- 172.21.15.7:0/1277961412 wait complete. 2024-04-21T08:43:33.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c308eb700 1 Processor -- start 2024-04-21T08:43:33.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c308eb700 1 -- start start 2024-04-21T08:43:33.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c308eb700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c2c101470 0x7f8c2c1990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c308eb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c2c101e30 0x7f8c2c199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c308eb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c2c108c20 0x7f8c2c19d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c308eb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c112c20 con 0x7f8c2c108c20 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c308eb700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c112aa0 con 0x7f8c2c101470 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c308eb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c112da0 con 0x7f8c2c101e30 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c2ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c2c108c20 0x7f8c2c19d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c2ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c2c108c20 0x7f8c2c19d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38484/0 (socket says 172.21.15.7:38484) 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c2a59c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c2c101470 0x7f8c2c1990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c2ad9d700 1 -- 172.21.15.7:0/3627213693 learned_addr learned my addr 172.21.15.7:0/3627213693 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.233+0000 7f8c29d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c2c101e30 0x7f8c2c199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.234+0000 7f8c2ad9d700 1 -- 172.21.15.7:0/3627213693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c2c101e30 msgr2=0x7f8c2c199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.234+0000 7f8c2ad9d700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c2c101e30 0x7f8c2c199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.234+0000 7f8c2ad9d700 1 -- 172.21.15.7:0/3627213693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c2c101470 msgr2=0x7f8c2c1990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.234+0000 7f8c2ad9d700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c2c101470 0x7f8c2c1990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.234+0000 7f8c2ad9d700 1 -- 172.21.15.7:0/3627213693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2c19e100 con 0x7f8c2c108c20 2024-04-21T08:43:33.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.234+0000 7f8c2ad9d700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c2c108c20 0x7f8c2c19d9c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400c970 tx=0x7f8c2400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:33.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.234+0000 7f8c237fe700 1 -- 172.21.15.7:0/3627213693 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c24004a30 con 0x7f8c2c108c20 2024-04-21T08:43:33.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.234+0000 7f8c237fe700 1 -- 172.21.15.7:0/3627213693 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c240054d0 con 0x7f8c2c108c20 2024-04-21T08:43:33.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.234+0000 7f8c308eb700 1 -- 172.21.15.7:0/3627213693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c2c19e3f0 con 0x7f8c2c108c20 2024-04-21T08:43:33.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.234+0000 7f8c237fe700 1 -- 172.21.15.7:0/3627213693 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2400ec20 con 0x7f8c2c108c20 2024-04-21T08:43:33.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.234+0000 7f8c308eb700 1 -- 172.21.15.7:0/3627213693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c2c1a5c70 con 0x7f8c2c108c20 2024-04-21T08:43:33.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.235+0000 7f8c308eb700 1 -- 172.21.15.7:0/3627213693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c2c10b060 con 0x7f8c2c108c20 2024-04-21T08:43:33.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.236+0000 7f8c237fe700 1 -- 172.21.15.7:0/3627213693 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8c24004fc0 con 0x7f8c2c108c20 2024-04-21T08:43:33.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.237+0000 7f8c237fe700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8c180a3cd0 0x7f8c180a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:33.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.237+0000 7f8c2a59c700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8c180a3cd0 0x7f8c180a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:33.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.237+0000 7f8c237fe700 1 -- 172.21.15.7:0/3627213693 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f8c240d0770 con 0x7f8c2c108c20 2024-04-21T08:43:33.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.238+0000 7f8c2a59c700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8c180a3cd0 0x7f8c180a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c009e60 tx=0x7f8c1c00b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:33.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.241+0000 7f8c237fe700 1 -- 172.21.15.7:0/3627213693 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c24014030 con 0x7f8c2c108c20 2024-04-21T08:43:33.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.450+0000 7f8c308eb700 1 -- 172.21.15.7:0/3627213693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f8c2c069730 con 0x7f8c2c108c20 2024-04-21T08:43:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.453+0000 7f8c237fe700 1 -- 172.21.15.7:0/3627213693 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8c2c069730 con 0x7f8c2c108c20 2024-04-21T08:43:33.453 INFO:teuthology.orchestra.run.smithi007.stdout:111669149765 2024-04-21T08:43:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 -- 172.21.15.7:0/3627213693 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8c180a3cd0 msgr2=0x7f8c180a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8c180a3cd0 0x7f8c180a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c009e60 tx=0x7f8c1c00b7b0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 -- 172.21.15.7:0/3627213693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c2c108c20 msgr2=0x7f8c2c19d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c2c108c20 0x7f8c2c19d9c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400c970 tx=0x7f8c2400cc80 comp rx=0 tx=0).stop 2024-04-21T08:43:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 -- 172.21.15.7:0/3627213693 shutdown_connections 2024-04-21T08:43:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f8c180a3cd0 0x7f8c180a6190 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c2c108c20 0x7f8c2c19d9c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c2c101470 0x7f8c2c1990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 --2- 172.21.15.7:0/3627213693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c2c101e30 0x7f8c2c199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 -- 172.21.15.7:0/3627213693 >> 172.21.15.7:0/3627213693 conn(0x7f8c2c076bb0 msgr2=0x7f8c2c0fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:33.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 -- 172.21.15.7:0/3627213693 shutdown_connections 2024-04-21T08:43:33.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.455+0000 7f8c308eb700 1 -- 172.21.15.7:0/3627213693 wait complete. 2024-04-21T08:43:33.751 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:33 smithi007 ceph-mon[28477]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:33 smithi016 ceph-mon[31259]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:33.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:33 smithi077 ceph-mon[31615]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:33.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.973+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1259337508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd98107940 msgr2=0x7fdd98107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:33.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.973+0000 7fdd9ee5d700 1 --2- 172.21.15.7:0/1259337508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd98107940 0x7fdd98107d20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0099c0 tx=0x7fdd8c009cd0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.973+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1259337508 shutdown_connections 2024-04-21T08:43:33.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.973+0000 7fdd9ee5d700 1 --2- 172.21.15.7:0/1259337508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd981034b0 0x7fdd9810f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.973+0000 7fdd9ee5d700 1 --2- 172.21.15.7:0/1259337508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd98107940 0x7fdd98107d20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.973+0000 7fdd9ee5d700 1 --2- 172.21.15.7:0/1259337508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd98102af0 0x7fdd98102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.973+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1259337508 >> 172.21.15.7:0/1259337508 conn(0x7fdd98079ba0 msgr2=0x7fdd98079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:33.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.973+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1259337508 shutdown_connections 2024-04-21T08:43:33.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.973+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1259337508 wait complete. 2024-04-21T08:43:33.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.973+0000 7fdd9ee5d700 1 Processor -- start 2024-04-21T08:43:33.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9ee5d700 1 -- start start 2024-04-21T08:43:33.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9ee5d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd98102af0 0x7fdd98199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:33.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9ee5d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd981034b0 0x7fdd98199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:33.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9ee5d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd98107940 0x7fdd9819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:33.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9ee5d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98112c60 con 0x7fdd98102af0 2024-04-21T08:43:33.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9ee5d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98112ae0 con 0x7fdd981034b0 2024-04-21T08:43:33.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9ee5d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98112de0 con 0x7fdd98107940 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9cbf9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd98102af0 0x7fdd98199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9cbf9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd98102af0 0x7fdd98199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38500/0 (socket says 172.21.15.7:38500) 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9cbf9700 1 -- 172.21.15.7:0/1421121030 learned_addr learned my addr 172.21.15.7:0/1421121030 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd97fff700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd981034b0 0x7fdd98199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9d3fa700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd98107940 0x7fdd9819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9cbf9700 1 -- 172.21.15.7:0/1421121030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd98107940 msgr2=0x7fdd9819da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9cbf9700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd98107940 0x7fdd9819da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9cbf9700 1 -- 172.21.15.7:0/1421121030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd981034b0 msgr2=0x7fdd98199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9cbf9700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd981034b0 0x7fdd98199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9cbf9700 1 -- 172.21.15.7:0/1421121030 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9819e140 con 0x7fdd98102af0 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.974+0000 7fdd9cbf9700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd98102af0 0x7fdd98199130 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c006010 tx=0x7fdd8c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.975+0000 7fdd95ffb700 1 -- 172.21.15.7:0/1421121030 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c026070 con 0x7fdd98102af0 2024-04-21T08:43:33.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.975+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1421121030 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9819e3d0 con 0x7fdd98102af0 2024-04-21T08:43:33.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.975+0000 7fdd95ffb700 1 -- 172.21.15.7:0/1421121030 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd8c003c70 con 0x7fdd98102af0 2024-04-21T08:43:33.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.975+0000 7fdd95ffb700 1 -- 172.21.15.7:0/1421121030 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c00fbe0 con 0x7fdd98102af0 2024-04-21T08:43:33.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.975+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1421121030 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd981a5cb0 con 0x7fdd98102af0 2024-04-21T08:43:33.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.976+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1421121030 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd98069730 con 0x7fdd98102af0 2024-04-21T08:43:33.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.977+0000 7fdd95ffb700 1 -- 172.21.15.7:0/1421121030 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fdd8c029070 con 0x7fdd98102af0 2024-04-21T08:43:33.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.978+0000 7fdd95ffb700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdd880a3cd0 0x7fdd880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:33.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.978+0000 7fdd97fff700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdd880a3cd0 0x7fdd880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:33.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.978+0000 7fdd95ffb700 1 -- 172.21.15.7:0/1421121030 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fdd8c0d1520 con 0x7fdd98102af0 2024-04-21T08:43:33.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.979+0000 7fdd97fff700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdd880a3cd0 0x7fdd880a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdd9807a210 tx=0x7fdd84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:33.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:33.982+0000 7fdd95ffb700 1 -- 172.21.15.7:0/1421121030 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd8c030080 con 0x7fdd98102af0 2024-04-21T08:43:34.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.192+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1421121030 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fdd9802df80 con 0x7fdd98102af0 2024-04-21T08:43:34.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.192+0000 7fdd95ffb700 1 -- 172.21.15.7:0/1421121030 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdd8c013070 con 0x7fdd98102af0 2024-04-21T08:43:34.193 INFO:teuthology.orchestra.run.smithi007.stdout:55834574928 2024-04-21T08:43:34.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1421121030 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdd880a3cd0 msgr2=0x7fdd880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdd880a3cd0 0x7fdd880a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdd9807a210 tx=0x7fdd84009260 comp rx=0 tx=0).stop 2024-04-21T08:43:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1421121030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd98102af0 msgr2=0x7fdd98199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd98102af0 0x7fdd98199130 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c006010 tx=0x7fdd8c004270 comp rx=0 tx=0).stop 2024-04-21T08:43:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1421121030 shutdown_connections 2024-04-21T08:43:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fdd880a3cd0 0x7fdd880a6190 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd98107940 0x7fdd9819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd98102af0 0x7fdd98199130 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 --2- 172.21.15.7:0/1421121030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd981034b0 0x7fdd98199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1421121030 >> 172.21.15.7:0/1421121030 conn(0x7fdd98079ba0 msgr2=0x7fdd98100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1421121030 shutdown_connections 2024-04-21T08:43:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:34.194+0000 7fdd9ee5d700 1 -- 172.21.15.7:0/1421121030 wait complete. 2024-04-21T08:43:34.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:34 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3627213693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:43:34.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:34 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1421121030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:43:34.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:34 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3627213693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:43:34.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:34 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1421121030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:43:34.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:34 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3627213693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:43:34.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:34 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1421121030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:43:34.884 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574926 got 55834574928 for osd.1 2024-04-21T08:43:34.885 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:35.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.325+0000 7f823a61d700 1 -- 172.21.15.7:0/3164255217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8234108a70 msgr2=0x7f8234108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.325+0000 7f823a61d700 1 --2- 172.21.15.7:0/3164255217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8234108a70 0x7f8234108e50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8224009990 tx=0x7f8224009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.325+0000 7f823a61d700 1 -- 172.21.15.7:0/3164255217 shutdown_connections 2024-04-21T08:43:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.325+0000 7f823a61d700 1 --2- 172.21.15.7:0/3164255217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8234103480 0x7f823410f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.325+0000 7f823a61d700 1 --2- 172.21.15.7:0/3164255217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8234108a70 0x7f8234108e50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.325+0000 7f823a61d700 1 --2- 172.21.15.7:0/3164255217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8234102ac0 0x7f8234102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.325+0000 7f823a61d700 1 -- 172.21.15.7:0/3164255217 >> 172.21.15.7:0/3164255217 conn(0x7f8234079bb0 msgr2=0x7f8234079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.325+0000 7f823a61d700 1 -- 172.21.15.7:0/3164255217 shutdown_connections 2024-04-21T08:43:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.325+0000 7f823a61d700 1 -- 172.21.15.7:0/3164255217 wait complete. 2024-04-21T08:43:35.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f823a61d700 1 Processor -- start 2024-04-21T08:43:35.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f823a61d700 1 -- start start 2024-04-21T08:43:35.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f823a61d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8234102ac0 0x7f8234199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:35.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f823a61d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8234103480 0x7f8234199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:35.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f823a61d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8234108a70 0x7f823419da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:35.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f823a61d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8234112c20 con 0x7f8234102ac0 2024-04-21T08:43:35.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f823a61d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8234112aa0 con 0x7f8234103480 2024-04-21T08:43:35.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f823a61d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8234112da0 con 0x7f8234108a70 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f8233fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8234102ac0 0x7f8234199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f8233fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8234102ac0 0x7f8234199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53938/0 (socket says 172.21.15.7:53938) 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f8233fff700 1 -- 172.21.15.7:0/413490592 learned_addr learned my addr 172.21.15.7:0/413490592 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f8238bba700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8234108a70 0x7f823419da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.326+0000 7f82337fe700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8234103480 0x7f8234199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f8233fff700 1 -- 172.21.15.7:0/413490592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8234108a70 msgr2=0x7f823419da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f8233fff700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8234108a70 0x7f823419da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f8233fff700 1 -- 172.21.15.7:0/413490592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8234103480 msgr2=0x7f8234199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f8233fff700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8234103480 0x7f8234199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f8233fff700 1 -- 172.21.15.7:0/413490592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823419e140 con 0x7f8234102ac0 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f82337fe700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8234103480 0x7f8234199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f8233fff700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8234102ac0 0x7f8234199130 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8224000c00 tx=0x7f82240047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f82317fa700 1 -- 172.21.15.7:0/413490592 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8224027070 con 0x7f8234102ac0 2024-04-21T08:43:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f82317fa700 1 -- 172.21.15.7:0/413490592 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82240040e0 con 0x7f8234102ac0 2024-04-21T08:43:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f823a61d700 1 -- 172.21.15.7:0/413490592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f823419e3d0 con 0x7f8234102ac0 2024-04-21T08:43:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f82317fa700 1 -- 172.21.15.7:0/413490592 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822400f460 con 0x7f8234102ac0 2024-04-21T08:43:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.327+0000 7f823a61d700 1 -- 172.21.15.7:0/413490592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82341a5cb0 con 0x7f8234102ac0 2024-04-21T08:43:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.328+0000 7f823a61d700 1 -- 172.21.15.7:0/413490592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8234069730 con 0x7f8234102ac0 2024-04-21T08:43:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.329+0000 7f82317fa700 1 -- 172.21.15.7:0/413490592 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f822402a070 con 0x7f8234102ac0 2024-04-21T08:43:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.330+0000 7f82317fa700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f821c0a3d50 0x7f821c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.330+0000 7f82337fe700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f821c0a3d50 0x7f821c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.330+0000 7f82317fa700 1 -- 172.21.15.7:0/413490592 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f82240d1830 con 0x7f8234102ac0 2024-04-21T08:43:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.330+0000 7f82337fe700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f821c0a3d50 0x7f821c0a6210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f823407a290 tx=0x7f8228005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:35.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.334+0000 7f82317fa700 1 -- 172.21.15.7:0/413490592 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f822405eff0 con 0x7f8234102ac0 2024-04-21T08:43:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.544+0000 7f823a61d700 1 -- 172.21.15.7:0/413490592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f823402df80 con 0x7f8234102ac0 2024-04-21T08:43:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.545+0000 7f82317fa700 1 -- 172.21.15.7:0/413490592 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8224013070 con 0x7f8234102ac0 2024-04-21T08:43:35.546 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:35 smithi007 ceph-mon[28477]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:35.546 INFO:teuthology.orchestra.run.smithi007.stdout:201863462968 2024-04-21T08:43:35.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 -- 172.21.15.7:0/413490592 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f821c0a3d50 msgr2=0x7f821c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:35.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f821c0a3d50 0x7f821c0a6210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f823407a290 tx=0x7f8228005f30 comp rx=0 tx=0).stop 2024-04-21T08:43:35.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 -- 172.21.15.7:0/413490592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8234102ac0 msgr2=0x7f8234199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:35.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8234102ac0 0x7f8234199130 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8224000c00 tx=0x7f82240047b0 comp rx=0 tx=0).stop 2024-04-21T08:43:35.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 -- 172.21.15.7:0/413490592 shutdown_connections 2024-04-21T08:43:35.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f821c0a3d50 0x7f821c0a6210 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:35.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8234108a70 0x7f823419da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:35.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8234102ac0 0x7f8234199130 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:35.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 --2- 172.21.15.7:0/413490592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8234103480 0x7f8234199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:35.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 -- 172.21.15.7:0/413490592 >> 172.21.15.7:0/413490592 conn(0x7f8234079bb0 msgr2=0x7f82341018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:35.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 -- 172.21.15.7:0/413490592 shutdown_connections 2024-04-21T08:43:35.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:35.548+0000 7f823a61d700 1 -- 172.21.15.7:0/413490592 wait complete. 2024-04-21T08:43:35.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:35 smithi016 ceph-mon[31259]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:35.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:35 smithi077 ceph-mon[31615]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:36.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:36 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/413490592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:43:36.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:36 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/413490592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:43:36.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:36 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/413490592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:43:37.270 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149760 got 111669149765 for osd.3 2024-04-21T08:43:37.270 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:37.561 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:37 smithi007 ceph-mon[28477]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-04-21T08:43:37.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:37 smithi016 ceph-mon[31259]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-04-21T08:43:37.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:37 smithi077 ceph-mon[31615]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-04-21T08:43:38.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.282+0000 7f79b7c3a700 1 -- 172.21.15.7:0/1966265087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b006b980 msgr2=0x7f79b010f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:38.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.282+0000 7f79b7c3a700 1 --2- 172.21.15.7:0/1966265087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b006b980 0x7f79b010f8f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f79a800b0a0 tx=0x7f79a800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:38.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.282+0000 7f79b7c3a700 1 -- 172.21.15.7:0/1966265087 shutdown_connections 2024-04-21T08:43:38.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.282+0000 7f79b7c3a700 1 --2- 172.21.15.7:0/1966265087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b0108a20 0x7f79b0108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:38.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.282+0000 7f79b7c3a700 1 --2- 172.21.15.7:0/1966265087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b006b980 0x7f79b010f8f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:38.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.282+0000 7f79b7c3a700 1 --2- 172.21.15.7:0/1966265087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b006afe0 0x7f79b006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:38.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.282+0000 7f79b7c3a700 1 -- 172.21.15.7:0/1966265087 >> 172.21.15.7:0/1966265087 conn(0x7f79b00788b0 msgr2=0x7f79b0078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.283+0000 7f79b7c3a700 1 -- 172.21.15.7:0/1966265087 shutdown_connections 2024-04-21T08:43:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.283+0000 7f79b7c3a700 1 -- 172.21.15.7:0/1966265087 wait complete. 2024-04-21T08:43:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.283+0000 7f79b7c3a700 1 Processor -- start 2024-04-21T08:43:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.283+0000 7f79b7c3a700 1 -- start start 2024-04-21T08:43:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.283+0000 7f79b7c3a700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b006afe0 0x7f79b01990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.283+0000 7f79b7c3a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b006b980 0x7f79b01995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.283+0000 7f79b7c3a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b0108a20 0x7f79b019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.283+0000 7f79b7c3a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f79b0112bc0 con 0x7f79b006b980 2024-04-21T08:43:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.283+0000 7f79b7c3a700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79b0112a40 con 0x7f79b006afe0 2024-04-21T08:43:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.283+0000 7f79b7c3a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f79b0112d40 con 0x7f79b0108a20 2024-04-21T08:43:38.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b51d5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b006b980 0x7f79b01995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b51d5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b006b980 0x7f79b01995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53980/0 (socket says 172.21.15.7:53980) 2024-04-21T08:43:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b51d5700 1 -- 172.21.15.7:0/631881577 learned_addr learned my addr 172.21.15.7:0/631881577 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b59d6700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b006afe0 0x7f79b01990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b61d7700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b0108a20 0x7f79b019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b51d5700 1 -- 172.21.15.7:0/631881577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b0108a20 msgr2=0x7f79b019d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b51d5700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b0108a20 0x7f79b019d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b51d5700 1 -- 172.21.15.7:0/631881577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b006afe0 msgr2=0x7f79b01990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b51d5700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b006afe0 0x7f79b01990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b51d5700 1 -- 172.21.15.7:0/631881577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b019e070 con 0x7f79b006b980 2024-04-21T08:43:38.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b61d7700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b0108a20 0x7f79b019d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b51d5700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b006b980 0x7f79b01995f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f79a000c9b0 tx=0x7f79a000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79a6ffd700 1 -- 172.21.15.7:0/631881577 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0007b00 con 0x7f79b006b980 2024-04-21T08:43:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79a6ffd700 1 -- 172.21.15.7:0/631881577 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79a0007ca0 con 0x7f79b006b980 2024-04-21T08:43:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b7c3a700 1 -- 172.21.15.7:0/631881577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b019e360 con 0x7f79b006b980 2024-04-21T08:43:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79a6ffd700 1 -- 172.21.15.7:0/631881577 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0018660 con 0x7f79b006b980 2024-04-21T08:43:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.284+0000 7f79b7c3a700 1 -- 172.21.15.7:0/631881577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b01a5be0 con 0x7f79b006b980 2024-04-21T08:43:38.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.286+0000 7f79b7c3a700 1 -- 172.21.15.7:0/631881577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b0069730 con 0x7f79b006b980 2024-04-21T08:43:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.287+0000 7f79a6ffd700 1 -- 172.21.15.7:0/631881577 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f79a000f450 con 0x7f79b006b980 2024-04-21T08:43:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.287+0000 7f79a6ffd700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f799c0a3d50 0x7f799c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.287+0000 7f79b59d6700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f799c0a3d50 0x7f799c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.287+0000 7f79a6ffd700 1 -- 172.21.15.7:0/631881577 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f79a00d0850 con 0x7f79b006b980 2024-04-21T08:43:38.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.288+0000 7f79b59d6700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f799c0a3d50 0x7f799c0a6210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f79ac005950 tx=0x7f79ac00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:38.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.292+0000 7f79a6ffd700 1 -- 172.21.15.7:0/631881577 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79a0014030 con 0x7f79b006b980 2024-04-21T08:43:38.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.518+0000 7f79b7c3a700 1 -- 172.21.15.7:0/631881577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f79b002df80 con 0x7f79b006b980 2024-04-21T08:43:38.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.518+0000 7f79a6ffd700 1 -- 172.21.15.7:0/631881577 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f79a009cd70 con 0x7f79b006b980 2024-04-21T08:43:38.518 INFO:teuthology.orchestra.run.smithi007.stdout:347892351006 2024-04-21T08:43:38.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 -- 172.21.15.7:0/631881577 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f799c0a3d50 msgr2=0x7f799c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:38.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f799c0a3d50 0x7f799c0a6210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f79ac005950 tx=0x7f79ac00a300 comp rx=0 tx=0).stop 2024-04-21T08:43:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 -- 172.21.15.7:0/631881577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b006b980 msgr2=0x7f79b01995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b006b980 0x7f79b01995f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f79a000c9b0 tx=0x7f79a000ccc0 comp rx=0 tx=0).stop 2024-04-21T08:43:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 -- 172.21.15.7:0/631881577 shutdown_connections 2024-04-21T08:43:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f799c0a3d50 0x7f799c0a6210 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79b0108a20 0x7f79b019d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b006afe0 0x7f79b01990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 --2- 172.21.15.7:0/631881577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b006b980 0x7f79b01995f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 -- 172.21.15.7:0/631881577 >> 172.21.15.7:0/631881577 conn(0x7f79b00788b0 msgr2=0x7f79b00ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 -- 172.21.15.7:0/631881577 shutdown_connections 2024-04-21T08:43:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.520+0000 7f79b7c3a700 1 -- 172.21.15.7:0/631881577 wait complete. 2024-04-21T08:43:38.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:38 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/377013703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:43:38.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:38 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/377013703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:43:38.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:38 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/377013703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:43:39.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.055+0000 7fa240df8700 1 -- 172.21.15.7:0/2826850473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa23c102a90 msgr2=0x7fa23c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.055+0000 7fa240df8700 1 --2- 172.21.15.7:0/2826850473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa23c102a90 0x7fa23c102f10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa230009990 tx=0x7fa230009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.055+0000 7fa240df8700 1 -- 172.21.15.7:0/2826850473 shutdown_connections 2024-04-21T08:43:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.055+0000 7fa240df8700 1 --2- 172.21.15.7:0/2826850473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23c108a80 0x7fa23c108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.055+0000 7fa240df8700 1 --2- 172.21.15.7:0/2826850473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c103450 0x7fa23c10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.055+0000 7fa240df8700 1 --2- 172.21.15.7:0/2826850473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa23c102a90 0x7fa23c102f10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.055+0000 7fa240df8700 1 -- 172.21.15.7:0/2826850473 >> 172.21.15.7:0/2826850473 conn(0x7fa23c079bf0 msgr2=0x7fa23c07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.055+0000 7fa240df8700 1 -- 172.21.15.7:0/2826850473 shutdown_connections 2024-04-21T08:43:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.055+0000 7fa240df8700 1 -- 172.21.15.7:0/2826850473 wait complete. 2024-04-21T08:43:39.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa240df8700 1 Processor -- start 2024-04-21T08:43:39.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa240df8700 1 -- start start 2024-04-21T08:43:39.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa240df8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa23c102a90 0x7fa23c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:39.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa240df8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23c103450 0x7fa23c075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:39.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa240df8700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c108a80 0x7fa23c0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:39.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa240df8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa23c112c80 con 0x7fa23c102a90 2024-04-21T08:43:39.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa240df8700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa23c112b00 con 0x7fa23c108a80 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa240df8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa23c112e00 con 0x7fa23c103450 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa23a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa23c102a90 0x7fa23c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa23a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa23c102a90 0x7fa23c074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54002/0 (socket says 172.21.15.7:54002) 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa23a59c700 1 -- 172.21.15.7:0/1964335826 learned_addr learned my addr 172.21.15.7:0/1964335826 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa23ad9d700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c108a80 0x7fa23c0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.056+0000 7fa239d9b700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23c103450 0x7fa23c075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa23ad9d700 1 -- 172.21.15.7:0/1964335826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23c103450 msgr2=0x7fa23c075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa23ad9d700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23c103450 0x7fa23c075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa23ad9d700 1 -- 172.21.15.7:0/1964335826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa23c102a90 msgr2=0x7fa23c074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa23ad9d700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa23c102a90 0x7fa23c074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa23ad9d700 1 -- 172.21.15.7:0/1964335826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa23c1a7760 con 0x7fa23c108a80 2024-04-21T08:43:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa239d9b700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23c103450 0x7fa23c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa23a59c700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa23c102a90 0x7fa23c074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:43:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa23ad9d700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c108a80 0x7fa23c0757c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa234007f10 tx=0x7fa23400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa22b7fe700 1 -- 172.21.15.7:0/1964335826 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23400ddd0 con 0x7fa23c108a80 2024-04-21T08:43:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa22b7fe700 1 -- 172.21.15.7:0/1964335826 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa234009e20 con 0x7fa23c108a80 2024-04-21T08:43:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa240df8700 1 -- 172.21.15.7:0/1964335826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa23c1a7a50 con 0x7fa23c108a80 2024-04-21T08:43:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa22b7fe700 1 -- 172.21.15.7:0/1964335826 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234015400 con 0x7fa23c108a80 2024-04-21T08:43:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.057+0000 7fa240df8700 1 -- 172.21.15.7:0/1964335826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa23c1a7f40 con 0x7fa23c108a80 2024-04-21T08:43:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.058+0000 7fa240df8700 1 -- 172.21.15.7:0/1964335826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa23c069730 con 0x7fa23c108a80 2024-04-21T08:43:39.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.060+0000 7fa22b7fe700 1 -- 172.21.15.7:0/1964335826 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fa234005020 con 0x7fa23c108a80 2024-04-21T08:43:39.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.061+0000 7fa22b7fe700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2240a3cd0 0x7fa2240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:39.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.061+0000 7fa23a59c700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2240a3cd0 0x7fa2240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:39.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.061+0000 7fa22b7fe700 1 -- 172.21.15.7:0/1964335826 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fa2340d02b0 con 0x7fa23c108a80 2024-04-21T08:43:39.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.062+0000 7fa23a59c700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2240a3cd0 0x7fa2240a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa22c005950 tx=0x7fa22c00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:39.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.066+0000 7fa22b7fe700 1 -- 172.21.15.7:0/1964335826 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa234010070 con 0x7fa23c108a80 2024-04-21T08:43:39.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.275+0000 7fa240df8700 1 -- 172.21.15.7:0/1964335826 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa23c02df80 con 0x7fa23c108a80 2024-04-21T08:43:39.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.276+0000 7fa22b7fe700 1 -- 172.21.15.7:0/1964335826 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa23409c7d0 con 0x7fa23c108a80 2024-04-21T08:43:39.276 INFO:teuthology.orchestra.run.smithi007.stdout:167503724605 2024-04-21T08:43:39.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.278+0000 7fa240df8700 1 -- 172.21.15.7:0/1964335826 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2240a3cd0 msgr2=0x7fa2240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:39.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.278+0000 7fa240df8700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2240a3cd0 0x7fa2240a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa22c005950 tx=0x7fa22c00a3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.278+0000 7fa240df8700 1 -- 172.21.15.7:0/1964335826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c108a80 msgr2=0x7fa23c0757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:39.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.278+0000 7fa240df8700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c108a80 0x7fa23c0757c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa234007f10 tx=0x7fa23400d3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.279+0000 7fa240df8700 1 -- 172.21.15.7:0/1964335826 shutdown_connections 2024-04-21T08:43:39.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.279+0000 7fa240df8700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2240a3cd0 0x7fa2240a6190 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.279+0000 7fa240df8700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa23c108a80 0x7fa23c0757c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.279+0000 7fa240df8700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa23c102a90 0x7fa23c074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.279+0000 7fa240df8700 1 --2- 172.21.15.7:0/1964335826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa23c103450 0x7fa23c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:39.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.279+0000 7fa240df8700 1 -- 172.21.15.7:0/1964335826 >> 172.21.15.7:0/1964335826 conn(0x7fa23c079bf0 msgr2=0x7fa23c101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:39.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.279+0000 7fa240df8700 1 -- 172.21.15.7:0/1964335826 shutdown_connections 2024-04-21T08:43:39.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:39.279+0000 7fa240df8700 1 -- 172.21.15.7:0/1964335826 wait complete. 2024-04-21T08:43:39.514 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462964 got 201863462968 for osd.6 2024-04-21T08:43:39.514 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:39.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:39 smithi007 ceph-mon[28477]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:39.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:39 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/631881577' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:43:39.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:39 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1964335826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:43:39.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:39 smithi016 ceph-mon[31259]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:39.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:39 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/631881577' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:43:39.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:39 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1964335826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:43:39.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:39 smithi077 ceph-mon[31615]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:39.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:39 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/631881577' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:43:39.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:39 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1964335826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:43:40.508 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724600 got 167503724605 for osd.5 2024-04-21T08:43:40.508 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.149+0000 7fba9335b700 1 -- 172.21.15.7:0/4230661805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c107990 msgr2=0x7fba8c107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.149+0000 7fba9335b700 1 --2- 172.21.15.7:0/4230661805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c107990 0x7fba8c107d70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fba88009990 tx=0x7fba88009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.149+0000 7fba9335b700 1 -- 172.21.15.7:0/4230661805 shutdown_connections 2024-04-21T08:43:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.149+0000 7fba9335b700 1 --2- 172.21.15.7:0/4230661805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c102300 0x7fba8c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.149+0000 7fba9335b700 1 --2- 172.21.15.7:0/4230661805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c107990 0x7fba8c107d70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.149+0000 7fba9335b700 1 --2- 172.21.15.7:0/4230661805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba8c101940 0x7fba8c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.149+0000 7fba9335b700 1 -- 172.21.15.7:0/4230661805 >> 172.21.15.7:0/4230661805 conn(0x7fba8c0788f0 msgr2=0x7fba8c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.149+0000 7fba9335b700 1 -- 172.21.15.7:0/4230661805 shutdown_connections 2024-04-21T08:43:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.149+0000 7fba9335b700 1 -- 172.21.15.7:0/4230661805 wait complete. 2024-04-21T08:43:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.149+0000 7fba9335b700 1 Processor -- start 2024-04-21T08:43:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.149+0000 7fba9335b700 1 -- start start 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba9335b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c101940 0x7fba8c197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba9335b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba8c102300 0x7fba8c1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba9335b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c107990 0x7fba8c198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba9335b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c111ae0 con 0x7fba8c101940 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba9335b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c111960 con 0x7fba8c107990 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba9335b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c111c60 con 0x7fba8c102300 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba910f7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c101940 0x7fba8c197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba910f7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c101940 0x7fba8c197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53964/0 (socket says 172.21.15.7:53964) 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba910f7700 1 -- 172.21.15.7:0/377013703 learned_addr learned my addr 172.21.15.7:0/377013703 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba908f6700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba8c102300 0x7fba8c1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba918f8700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c107990 0x7fba8c198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba910f7700 1 -- 172.21.15.7:0/377013703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba8c102300 msgr2=0x7fba8c1982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba910f7700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba8c102300 0x7fba8c1982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:40.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba910f7700 1 -- 172.21.15.7:0/377013703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c107990 msgr2=0x7fba8c198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:40.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba910f7700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c107990 0x7fba8c198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:40.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba910f7700 1 -- 172.21.15.7:0/377013703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8c1a7600 con 0x7fba8c101940 2024-04-21T08:43:40.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba910f7700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c101940 0x7fba8c197d90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fba880057c0 tx=0x7fba880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:40.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba827fc700 1 -- 172.21.15.7:0/377013703 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba88027070 con 0x7fba8c101940 2024-04-21T08:43:40.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.150+0000 7fba9335b700 1 -- 172.21.15.7:0/377013703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba8c1a7890 con 0x7fba8c101940 2024-04-21T08:43:40.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.151+0000 7fba827fc700 1 -- 172.21.15.7:0/377013703 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba88003f20 con 0x7fba8c101940 2024-04-21T08:43:40.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.151+0000 7fba827fc700 1 -- 172.21.15.7:0/377013703 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba88022b30 con 0x7fba8c101940 2024-04-21T08:43:40.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.151+0000 7fba9335b700 1 -- 172.21.15.7:0/377013703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba8c1a7d70 con 0x7fba8c101940 2024-04-21T08:43:40.577 INFO:teuthology.orchestra.run.smithi007.stdout:34359738453 2024-04-21T08:43:40.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.152+0000 7fba9335b700 1 -- 172.21.15.7:0/377013703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba8c109eb0 con 0x7fba8c101940 2024-04-21T08:43:40.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.153+0000 7fba827fc700 1 -- 172.21.15.7:0/377013703 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fba8802a070 con 0x7fba8c101940 2024-04-21T08:43:40.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.153+0000 7fba827fc700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba780a3cd0 0x7fba780a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:40.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.153+0000 7fba908f6700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba780a3cd0 0x7fba780a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:40.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.153+0000 7fba827fc700 1 -- 172.21.15.7:0/377013703 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fba880d1820 con 0x7fba8c101940 2024-04-21T08:43:40.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.154+0000 7fba908f6700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba780a3cd0 0x7fba780a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fba8c199ef0 tx=0x7fba7c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:40.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.158+0000 7fba827fc700 1 -- 172.21.15.7:0/377013703 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba8805eff0 con 0x7fba8c101940 2024-04-21T08:43:40.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.370+0000 7fba9335b700 1 -- 172.21.15.7:0/377013703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fba8c109890 con 0x7fba8c101940 2024-04-21T08:43:40.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.370+0000 7fba827fc700 1 -- 172.21.15.7:0/377013703 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fba88013070 con 0x7fba8c101940 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 -- 172.21.15.7:0/377013703 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba780a3cd0 msgr2=0x7fba780a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba780a3cd0 0x7fba780a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fba8c199ef0 tx=0x7fba7c00a300 comp rx=0 tx=0).stop 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 -- 172.21.15.7:0/377013703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c101940 msgr2=0x7fba8c197d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c101940 0x7fba8c197d90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fba880057c0 tx=0x7fba880057f0 comp rx=0 tx=0).stop 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 -- 172.21.15.7:0/377013703 shutdown_connections 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fba780a3cd0 0x7fba780a6190 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba8c107990 0x7fba8c198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c101940 0x7fba8c197d90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 --2- 172.21.15.7:0/377013703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba8c102300 0x7fba8c1982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 -- 172.21.15.7:0/377013703 >> 172.21.15.7:0/377013703 conn(0x7fba8c0788f0 msgr2=0x7fba8c100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 -- 172.21.15.7:0/377013703 shutdown_connections 2024-04-21T08:43:40.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:38.373+0000 7fba9335b700 1 -- 172.21.15.7:0/377013703 wait complete. 2024-04-21T08:43:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:41.188 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738449 got 34359738453 for osd.0 2024-04-21T08:43:41.189 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:41.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:41 smithi016 ceph-mon[31259]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:41.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:41 smithi007 ceph-mon[28477]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:41.837 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:41.837 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:41.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:41 smithi077 ceph-mon[31615]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:41.862 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351002 got 347892351006 for osd.11 2024-04-21T08:43:41.862 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:42.269 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:43.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.295+0000 7f7335a84700 1 -- 172.21.15.7:0/1463322024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7330108a20 msgr2=0x7f7330108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:43.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.295+0000 7f7335a84700 1 --2- 172.21.15.7:0/1463322024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7330108a20 0x7f7330108e00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7320009990 tx=0x7f7320009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:43.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.295+0000 7f7335a84700 1 -- 172.21.15.7:0/1463322024 shutdown_connections 2024-04-21T08:43:43.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.295+0000 7f7335a84700 1 --2- 172.21.15.7:0/1463322024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f733006b980 0x7f733010f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:43.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.295+0000 7f7335a84700 1 --2- 172.21.15.7:0/1463322024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7330108a20 0x7f7330108e00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:43.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.295+0000 7f7335a84700 1 --2- 172.21.15.7:0/1463322024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733006afe0 0x7f733006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:43.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.295+0000 7f7335a84700 1 -- 172.21.15.7:0/1463322024 >> 172.21.15.7:0/1463322024 conn(0x7f73300788b0 msgr2=0x7f7330078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:43.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.295+0000 7f7335a84700 1 -- 172.21.15.7:0/1463322024 shutdown_connections 2024-04-21T08:43:43.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.295+0000 7f7335a84700 1 -- 172.21.15.7:0/1463322024 wait complete. 2024-04-21T08:43:43.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.295+0000 7f7335a84700 1 Processor -- start 2024-04-21T08:43:43.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f7335a84700 1 -- start start 2024-04-21T08:43:43.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f7335a84700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f733006afe0 0x7f73301990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f7335a84700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733006b980 0x7f7330199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f7335a84700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7330108a20 0x7f733019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f7335a84700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7330112ba0 con 0x7f7330108a20 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f7335a84700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7330112a20 con 0x7f733006b980 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f7335a84700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7330112d20 con 0x7f733006afe0 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f732effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733006b980 0x7f7330199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f732ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7330108a20 0x7f733019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f732effd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733006b980 0x7f7330199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:50668/0 (socket says 172.21.15.7:50668) 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f732effd700 1 -- 172.21.15.7:0/231820015 learned_addr learned my addr 172.21.15.7:0/231820015 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.296+0000 7f732f7fe700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f733006afe0 0x7f73301990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f732effd700 1 -- 172.21.15.7:0/231820015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f733006afe0 msgr2=0x7f73301990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:43.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f732effd700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f733006afe0 0x7f73301990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:43.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f732effd700 1 -- 172.21.15.7:0/231820015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7330108a20 msgr2=0x7f733019d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:43.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f732effd700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7330108a20 0x7f733019d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:43.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f732effd700 1 -- 172.21.15.7:0/231820015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733019e0e0 con 0x7f733006b980 2024-04-21T08:43:43.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f732ffff700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7330108a20 0x7f733019d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:43:43.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f732effd700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733006b980 0x7f7330199610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f732400d980 tx=0x7f732400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:43.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f732cff9700 1 -- 172.21.15.7:0/231820015 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7324016070 con 0x7f733006b980 2024-04-21T08:43:43.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f7335a84700 1 -- 172.21.15.7:0/231820015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f733019e3d0 con 0x7f733006b980 2024-04-21T08:43:43.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f732cff9700 1 -- 172.21.15.7:0/231820015 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7324004830 con 0x7f733006b980 2024-04-21T08:43:43.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f732cff9700 1 -- 172.21.15.7:0/231820015 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7324005540 con 0x7f733006b980 2024-04-21T08:43:43.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.297+0000 7f7335a84700 1 -- 172.21.15.7:0/231820015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7330101930 con 0x7f733006b980 2024-04-21T08:43:43.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.298+0000 7f7335a84700 1 -- 172.21.15.7:0/231820015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7330069730 con 0x7f733006b980 2024-04-21T08:43:43.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.300+0000 7f732cff9700 1 -- 172.21.15.7:0/231820015 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f7324004220 con 0x7f733006b980 2024-04-21T08:43:43.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.300+0000 7f732cff9700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f73180a3d50 0x7f73180a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:43.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.300+0000 7f732f7fe700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f73180a3d50 0x7f73180a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:43.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.300+0000 7f732cff9700 1 -- 172.21.15.7:0/231820015 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f73240d0b30 con 0x7f733006b980 2024-04-21T08:43:43.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.301+0000 7f732f7fe700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f73180a3d50 0x7f73180a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f732000b5c0 tx=0x7f7320005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:43.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.304+0000 7f732cff9700 1 -- 172.21.15.7:0/231820015 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f732401f020 con 0x7f733006b980 2024-04-21T08:43:43.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.521+0000 7f7335a84700 1 -- 172.21.15.7:0/231820015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f733002df80 con 0x7f733006b980 2024-04-21T08:43:43.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.522+0000 7f732cff9700 1 -- 172.21.15.7:0/231820015 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f732409d050 con 0x7f733006b980 2024-04-21T08:43:43.522 INFO:teuthology.orchestra.run.smithi007.stdout:146028888131 2024-04-21T08:43:43.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 -- 172.21.15.7:0/231820015 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f73180a3d50 msgr2=0x7f73180a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:43.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f73180a3d50 0x7f73180a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f732000b5c0 tx=0x7f7320005960 comp rx=0 tx=0).stop 2024-04-21T08:43:43.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 -- 172.21.15.7:0/231820015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733006b980 msgr2=0x7f7330199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:43.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733006b980 0x7f7330199610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f732400d980 tx=0x7f732400dc90 comp rx=0 tx=0).stop 2024-04-21T08:43:43.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 -- 172.21.15.7:0/231820015 shutdown_connections 2024-04-21T08:43:43.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f73180a3d50 0x7f73180a6210 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:43.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f733006afe0 0x7f73301990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:43.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7330108a20 0x7f733019d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:43.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 --2- 172.21.15.7:0/231820015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f733006b980 0x7f7330199610 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:43.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 -- 172.21.15.7:0/231820015 >> 172.21.15.7:0/231820015 conn(0x7f73300788b0 msgr2=0x7f73300ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:43.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 -- 172.21.15.7:0/231820015 shutdown_connections 2024-04-21T08:43:43.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:43.524+0000 7f7335a84700 1 -- 172.21.15.7:0/231820015 wait complete. 2024-04-21T08:43:43.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:43 smithi007 ceph-mon[28477]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:43.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:43 smithi016 ceph-mon[31259]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:43.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:43 smithi077 ceph-mon[31615]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:44.315 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888126 got 146028888131 for osd.4 2024-04-21T08:43:44.315 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:44.547 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:44 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/231820015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:43:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:44 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/231820015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:43:44.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:44 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/231820015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:43:45.797 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:45 smithi007 ceph-mon[28477]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:45.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:45 smithi016 ceph-mon[31259]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:45.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:45 smithi077 ceph-mon[31615]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.486+0000 7f13525a0700 1 -- 172.21.15.7:0/1066991503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c107880 msgr2=0x7f134c107c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.486+0000 7f13525a0700 1 --2- 172.21.15.7:0/1066991503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c107880 0x7f134c107c60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f134400b0a0 tx=0x7f134400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.486+0000 7f13525a0700 1 -- 172.21.15.7:0/1066991503 shutdown_connections 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.486+0000 7f13525a0700 1 --2- 172.21.15.7:0/1066991503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f134c1024a0 0x7f134c10e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.486+0000 7f13525a0700 1 --2- 172.21.15.7:0/1066991503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c107880 0x7f134c107c60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.486+0000 7f13525a0700 1 --2- 172.21.15.7:0/1066991503 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c101b00 0x7f134c101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.486+0000 7f13525a0700 1 -- 172.21.15.7:0/1066991503 >> 172.21.15.7:0/1066991503 conn(0x7f134c078780 msgr2=0x7f134c078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.486+0000 7f13525a0700 1 -- 172.21.15.7:0/1066991503 shutdown_connections 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.486+0000 7f13525a0700 1 -- 172.21.15.7:0/1066991503 wait complete. 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f13525a0700 1 Processor -- start 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f13525a0700 1 -- start start 2024-04-21T08:43:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f13525a0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f134c101b00 0x7f134c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:46.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f13525a0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c1024a0 0x7f134c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:46.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f13525a0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c107880 0x7f134c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:46.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f13525a0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f134c111a50 con 0x7f134c1024a0 2024-04-21T08:43:46.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f13525a0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f134c1118d0 con 0x7f134c107880 2024-04-21T08:43:46.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f13525a0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f134c111bd0 con 0x7f134c101b00 2024-04-21T08:43:46.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f1350d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c1024a0 0x7f134c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:46.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f1350d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c1024a0 0x7f134c1984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44036/0 (socket says 172.21.15.7:44036) 2024-04-21T08:43:46.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f1350d9d700 1 -- 172.21.15.7:0/867711612 learned_addr learned my addr 172.21.15.7:0/867711612 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:46.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f135159e700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f134c101b00 0x7f134c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:46.401 INFO:teuthology.orchestra.run.smithi007.stdout:292057776170 2024-04-21T08:43:46.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.487+0000 7f1351d9f700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c107880 0x7f134c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:46.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.488+0000 7f1350d9d700 1 -- 172.21.15.7:0/867711612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f134c101b00 msgr2=0x7f134c197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:46.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.488+0000 7f1350d9d700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f134c101b00 0x7f134c197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:46.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.488+0000 7f1350d9d700 1 -- 172.21.15.7:0/867711612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c107880 msgr2=0x7f134c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:46.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.488+0000 7f1350d9d700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c107880 0x7f134c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:46.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.488+0000 7f1350d9d700 1 -- 172.21.15.7:0/867711612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134c19cfb0 con 0x7f134c1024a0 2024-04-21T08:43:46.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.488+0000 7f1350d9d700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c1024a0 0x7f134c1984e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f134000d980 tx=0x7f134000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:46.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.488+0000 7f134a7fc700 1 -- 172.21.15.7:0/867711612 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1340016070 con 0x7f134c1024a0 2024-04-21T08:43:46.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.488+0000 7f134a7fc700 1 -- 172.21.15.7:0/867711612 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1340004830 con 0x7f134c1024a0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.488+0000 7f134a7fc700 1 -- 172.21.15.7:0/867711612 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1340005540 con 0x7f134c1024a0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.488+0000 7f13525a0700 1 -- 172.21.15.7:0/867711612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134c19d2a0 con 0x7f134c1024a0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.488+0000 7f13525a0700 1 -- 172.21.15.7:0/867711612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134c1a4b20 con 0x7f134c1024a0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.489+0000 7f13525a0700 1 -- 172.21.15.7:0/867711612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134c069730 con 0x7f134c1024a0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.490+0000 7f134a7fc700 1 -- 172.21.15.7:0/867711612 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f1340004220 con 0x7f134c1024a0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.491+0000 7f134a7fc700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f13340a3cd0 0x7f13340a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.491+0000 7f135159e700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f13340a3cd0 0x7f13340a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.491+0000 7f134a7fc700 1 -- 172.21.15.7:0/867711612 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f13400d0bb0 con 0x7f134c1024a0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.492+0000 7f135159e700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f13340a3cd0 0x7f13340a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1344009fd0 tx=0x7f1344006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.495+0000 7f134a7fc700 1 -- 172.21.15.7:0/867711612 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f134001f020 con 0x7f134c1024a0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.707+0000 7f13525a0700 1 -- 172.21.15.7:0/867711612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f134c02df80 con 0x7f134c1024a0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.708+0000 7f134a7fc700 1 -- 172.21.15.7:0/867711612 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f134009d0d0 con 0x7f134c1024a0 2024-04-21T08:43:46.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 -- 172.21.15.7:0/867711612 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f13340a3cd0 msgr2=0x7f13340a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f13340a3cd0 0x7f13340a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1344009fd0 tx=0x7f1344006210 comp rx=0 tx=0).stop 2024-04-21T08:43:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 -- 172.21.15.7:0/867711612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c1024a0 msgr2=0x7f134c1984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c1024a0 0x7f134c1984e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f134000d980 tx=0x7f134000dc90 comp rx=0 tx=0).stop 2024-04-21T08:43:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 -- 172.21.15.7:0/867711612 shutdown_connections 2024-04-21T08:43:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f13340a3cd0 0x7f13340a6190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f134c101b00 0x7f134c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c107880 0x7f134c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 --2- 172.21.15.7:0/867711612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c1024a0 0x7f134c1984e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 -- 172.21.15.7:0/867711612 >> 172.21.15.7:0/867711612 conn(0x7f134c078780 msgr2=0x7f134c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 -- 172.21.15.7:0/867711612 shutdown_connections 2024-04-21T08:43:46.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:45.710+0000 7f13525a0700 1 -- 172.21.15.7:0/867711612 wait complete. 2024-04-21T08:43:46.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:46 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/867711612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:43:46.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:46 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/867711612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:43:46.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:46 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/867711612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:43:47.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:47 smithi016 ceph-mon[31259]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:47.827 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:47 smithi007 ceph-mon[28477]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:47.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:47 smithi077 ceph-mon[31615]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:48.146 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776162 got 292057776170 for osd.9 2024-04-21T08:43:48.147 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:49.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:49 smithi007 ceph-mon[28477]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:49.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:49 smithi016 ceph-mon[31259]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:49.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:49 smithi077 ceph-mon[31615]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:50.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.080+0000 7fd0e1f40700 1 -- 172.21.15.7:0/1537853474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0dc102300 msgr2=0x7fd0dc10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.080+0000 7fd0e1f40700 1 --2- 172.21.15.7:0/1537853474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0dc102300 0x7fd0dc10e7f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400b0a0 tx=0x7fd0d400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.080+0000 7fd0e1f40700 1 -- 172.21.15.7:0/1537853474 shutdown_connections 2024-04-21T08:43:50.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.080+0000 7fd0e1f40700 1 --2- 172.21.15.7:0/1537853474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0dc102300 0x7fd0dc10e7f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.080+0000 7fd0e1f40700 1 --2- 172.21.15.7:0/1537853474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0dc107990 0x7fd0dc107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.080+0000 7fd0e1f40700 1 --2- 172.21.15.7:0/1537853474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc101940 0x7fd0dc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.080+0000 7fd0e1f40700 1 -- 172.21.15.7:0/1537853474 >> 172.21.15.7:0/1537853474 conn(0x7fd0dc0788f0 msgr2=0x7fd0dc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.080+0000 7fd0e1f40700 1 -- 172.21.15.7:0/1537853474 shutdown_connections 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.080+0000 7fd0e1f40700 1 -- 172.21.15.7:0/1537853474 wait complete. 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.080+0000 7fd0e1f40700 1 Processor -- start 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.080+0000 7fd0e1f40700 1 -- start start 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0e1f40700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0dc101940 0x7fd0dc199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0e1f40700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0dc102300 0x7fd0dc1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0e1f40700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc107990 0x7fd0dc19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0e1f40700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc111ae0 con 0x7fd0dc102300 2024-04-21T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0e1f40700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc111960 con 0x7fd0dc101940 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0e1f40700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc111c60 con 0x7fd0dc107990 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0db7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0dc101940 0x7fd0dc199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0daffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0dc102300 0x7fd0dc1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0db7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0dc101940 0x7fd0dc199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38202/0 (socket says 172.21.15.7:38202) 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0db7fe700 1 -- 172.21.15.7:0/2955885231 learned_addr learned my addr 172.21.15.7:0/2955885231 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0dbfff700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc107990 0x7fd0dc19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0daffd700 1 -- 172.21.15.7:0/2955885231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc107990 msgr2=0x7fd0dc19d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0daffd700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc107990 0x7fd0dc19d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0daffd700 1 -- 172.21.15.7:0/2955885231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0dc101940 msgr2=0x7fd0dc199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0daffd700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0dc101940 0x7fd0dc199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0daffd700 1 -- 172.21.15.7:0/2955885231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0dc19e070 con 0x7fd0dc102300 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.081+0000 7fd0db7fe700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0dc101940 0x7fd0dc199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.082+0000 7fd0daffd700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0dc102300 0x7fd0dc1995a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000d950 tx=0x7fd0d000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.082+0000 7fd0d8ff9700 1 -- 172.21.15.7:0/2955885231 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0016070 con 0x7fd0dc102300 2024-04-21T08:43:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.082+0000 7fd0e1f40700 1 -- 172.21.15.7:0/2955885231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0dc19e360 con 0x7fd0dc102300 2024-04-21T08:43:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.082+0000 7fd0d8ff9700 1 -- 172.21.15.7:0/2955885231 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0d000fd40 con 0x7fd0dc102300 2024-04-21T08:43:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.082+0000 7fd0d8ff9700 1 -- 172.21.15.7:0/2955885231 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d000e3b0 con 0x7fd0dc102300 2024-04-21T08:43:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.082+0000 7fd0e1f40700 1 -- 172.21.15.7:0/2955885231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0dc1a5be0 con 0x7fd0dc102300 2024-04-21T08:43:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.083+0000 7fd0e1f40700 1 -- 172.21.15.7:0/2955885231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0dc069730 con 0x7fd0dc102300 2024-04-21T08:43:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.084+0000 7fd0d8ff9700 1 -- 172.21.15.7:0/2955885231 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fd0d0019070 con 0x7fd0dc102300 2024-04-21T08:43:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.085+0000 7fd0d8ff9700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd0c40a3d50 0x7fd0c40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.085+0000 7fd0db7fe700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd0c40a3d50 0x7fd0c40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.085+0000 7fd0d8ff9700 1 -- 172.21.15.7:0/2955885231 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fd0d00d07b0 con 0x7fd0dc102300 2024-04-21T08:43:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.085+0000 7fd0db7fe700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd0c40a3d50 0x7fd0c40a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc011550 tx=0x7fd0cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.093+0000 7fd0d8ff9700 1 -- 172.21.15.7:0/2955885231 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0d009cc70 con 0x7fd0dc102300 2024-04-21T08:43:50.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.312+0000 7fd0e1f40700 1 -- 172.21.15.7:0/2955885231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fd0dc02df80 con 0x7fd0dc102300 2024-04-21T08:43:50.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.312+0000 7fd0d8ff9700 1 -- 172.21.15.7:0/2955885231 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd0d009c570 con 0x7fd0dc102300 2024-04-21T08:43:50.312 INFO:teuthology.orchestra.run.smithi007.stdout:257698037808 2024-04-21T08:43:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 -- 172.21.15.7:0/2955885231 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd0c40a3d50 msgr2=0x7fd0c40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd0c40a3d50 0x7fd0c40a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc011550 tx=0x7fd0cc009260 comp rx=0 tx=0).stop 2024-04-21T08:43:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 -- 172.21.15.7:0/2955885231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0dc102300 msgr2=0x7fd0dc1995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0dc102300 0x7fd0dc1995a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000d950 tx=0x7fd0d000dc60 comp rx=0 tx=0).stop 2024-04-21T08:43:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 -- 172.21.15.7:0/2955885231 shutdown_connections 2024-04-21T08:43:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd0c40a3d50 0x7fd0c40a6210 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc107990 0x7fd0dc19d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0dc101940 0x7fd0dc199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 --2- 172.21.15.7:0/2955885231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0dc102300 0x7fd0dc1995a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 -- 172.21.15.7:0/2955885231 >> 172.21.15.7:0/2955885231 conn(0x7fd0dc0788f0 msgr2=0x7fd0dc06aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 -- 172.21.15.7:0/2955885231 shutdown_connections 2024-04-21T08:43:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.315+0000 7fd0e1f40700 1 -- 172.21.15.7:0/2955885231 wait complete. 2024-04-21T08:43:50.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:50 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:50.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:50 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:50.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:50 smithi007 ceph-mon[28477]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:50.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:50 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:50.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:50 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:50.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:50 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:50.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:50 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2955885231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:43:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.807+0000 7f5634a9e700 1 -- 172.21.15.7:0/2511142535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5630107990 msgr2=0x7f5630107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.807+0000 7f5634a9e700 1 --2- 172.21.15.7:0/2511142535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5630107990 0x7f5630107d70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f562000b0a0 tx=0x7f562000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.807+0000 7f5634a9e700 1 -- 172.21.15.7:0/2511142535 shutdown_connections 2024-04-21T08:43:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.807+0000 7f5634a9e700 1 --2- 172.21.15.7:0/2511142535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5630102300 0x7f563010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.807+0000 7f5634a9e700 1 --2- 172.21.15.7:0/2511142535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5630107990 0x7f5630107d70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.807+0000 7f5634a9e700 1 --2- 172.21.15.7:0/2511142535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5630101940 0x7f5630101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.807+0000 7f5634a9e700 1 -- 172.21.15.7:0/2511142535 >> 172.21.15.7:0/2511142535 conn(0x7f56300788f0 msgr2=0x7f5630078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.807+0000 7f5634a9e700 1 -- 172.21.15.7:0/2511142535 shutdown_connections 2024-04-21T08:43:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.807+0000 7f5634a9e700 1 -- 172.21.15.7:0/2511142535 wait complete. 2024-04-21T08:43:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f5634a9e700 1 Processor -- start 2024-04-21T08:43:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f5634a9e700 1 -- start start 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f5634a9e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5630101940 0x7f56301034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f5634a9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5630102300 0x7f5630103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f5634a9e700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5630107990 0x7f5630103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f5634a9e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5630111c20 con 0x7f5630102300 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f5634a9e700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5630111aa0 con 0x7f5630107990 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f5634a9e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5630111da0 con 0x7f5630101940 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f562dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5630102300 0x7f5630103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f562dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5630102300 0x7f5630103a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44078/0 (socket says 172.21.15.7:44078) 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f562dd9b700 1 -- 172.21.15.7:0/243261689 learned_addr learned my addr 172.21.15.7:0/243261689 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f562ed9d700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5630107990 0x7f5630103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.808+0000 7f562e59c700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5630101940 0x7f56301034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f562dd9b700 1 -- 172.21.15.7:0/243261689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5630101940 msgr2=0x7f56301034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f562dd9b700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5630101940 0x7f56301034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f562dd9b700 1 -- 172.21.15.7:0/243261689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5630107990 msgr2=0x7f5630103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f562dd9b700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5630107990 0x7f5630103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f562dd9b700 1 -- 172.21.15.7:0/243261689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56301a7870 con 0x7f5630102300 2024-04-21T08:43:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f562ed9d700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5630107990 0x7f5630103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f562dd9b700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5630102300 0x7f5630103a30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f561800b530 tx=0x7f561800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f56277fe700 1 -- 172.21.15.7:0/243261689 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618015020 con 0x7f5630102300 2024-04-21T08:43:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f56277fe700 1 -- 172.21.15.7:0/243261689 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5618011ce0 con 0x7f5630102300 2024-04-21T08:43:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f56277fe700 1 -- 172.21.15.7:0/243261689 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618010d20 con 0x7f5630102300 2024-04-21T08:43:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f5634a9e700 1 -- 172.21.15.7:0/243261689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56301a7b60 con 0x7f5630102300 2024-04-21T08:43:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.809+0000 7f5634a9e700 1 -- 172.21.15.7:0/243261689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56301a8050 con 0x7f5630102300 2024-04-21T08:43:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.810+0000 7f5634a9e700 1 -- 172.21.15.7:0/243261689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5630069730 con 0x7f5630102300 2024-04-21T08:43:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.811+0000 7f56277fe700 1 -- 172.21.15.7:0/243261689 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f5618018020 con 0x7f5630102300 2024-04-21T08:43:50.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.812+0000 7f56277fe700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f561c0a3cd0 0x7f561c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:50.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.812+0000 7f562e59c700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f561c0a3cd0 0x7f561c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:50.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.812+0000 7f56277fe700 1 -- 172.21.15.7:0/243261689 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f56180d0070 con 0x7f5630102300 2024-04-21T08:43:50.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.813+0000 7f562e59c700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f561c0a3cd0 0x7f561c0a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5620009e60 tx=0x7f562000b7d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:50.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:50.817+0000 7f56277fe700 1 -- 172.21.15.7:0/243261689 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f561801f020 con 0x7f5630102300 2024-04-21T08:43:50.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:50 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:50.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:50 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:50.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:50 smithi077 ceph-mon[31615]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:50.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:50 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:50.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:50 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:50.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:50 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:50.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:50 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2955885231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:43:51.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.028+0000 7f5634a9e700 1 -- 172.21.15.7:0/243261689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f56301a8280 con 0x7f5630102300 2024-04-21T08:43:51.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.028+0000 7f56277fe700 1 -- 172.21.15.7:0/243261689 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f561809c640 con 0x7f5630102300 2024-04-21T08:43:51.028 INFO:teuthology.orchestra.run.smithi007.stdout:223338299445 2024-04-21T08:43:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.030+0000 7f5634a9e700 1 -- 172.21.15.7:0/243261689 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f561c0a3cd0 msgr2=0x7f561c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.030+0000 7f5634a9e700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f561c0a3cd0 0x7f561c0a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5620009e60 tx=0x7f562000b7d0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.030+0000 7f5634a9e700 1 -- 172.21.15.7:0/243261689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5630102300 msgr2=0x7f5630103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.030+0000 7f5634a9e700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5630102300 0x7f5630103a30 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f561800b530 tx=0x7f561800b8f0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.030+0000 7f5634a9e700 1 -- 172.21.15.7:0/243261689 shutdown_connections 2024-04-21T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.030+0000 7f5634a9e700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f561c0a3cd0 0x7f561c0a6190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.030+0000 7f5634a9e700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5630101940 0x7f56301034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.030+0000 7f5634a9e700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5630107990 0x7f5630103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.030+0000 7f5634a9e700 1 --2- 172.21.15.7:0/243261689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5630102300 0x7f5630103a30 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.030+0000 7f5634a9e700 1 -- 172.21.15.7:0/243261689 >> 172.21.15.7:0/243261689 conn(0x7f56300788f0 msgr2=0x7f563006a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.031+0000 7f5634a9e700 1 -- 172.21.15.7:0/243261689 shutdown_connections 2024-04-21T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.031+0000 7f5634a9e700 1 -- 172.21.15.7:0/243261689 wait complete. 2024-04-21T08:43:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:50 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:50 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:50 smithi016 ceph-mon[31259]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:50 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:50 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:50 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' 2024-04-21T08:43:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:50 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2955885231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:43:51.392 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037803 got 257698037808 for osd.8 2024-04-21T08:43:51.392 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:51.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.568+0000 7f41c903b700 1 -- 172.21.15.7:0/1893756238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c4103450 msgr2=0x7f41c410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.568+0000 7f41c903b700 1 --2- 172.21.15.7:0/1893756238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c4103450 0x7f41c410f8b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00b0a0 tx=0x7f41bc00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.568+0000 7f41c903b700 1 -- 172.21.15.7:0/1893756238 shutdown_connections 2024-04-21T08:43:51.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.568+0000 7f41c903b700 1 --2- 172.21.15.7:0/1893756238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c4103450 0x7f41c410f8b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.568+0000 7f41c903b700 1 --2- 172.21.15.7:0/1893756238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4108a50 0x7f41c4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.568+0000 7f41c903b700 1 --2- 172.21.15.7:0/1893756238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4102a90 0x7f41c4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.568+0000 7f41c903b700 1 -- 172.21.15.7:0/1893756238 >> 172.21.15.7:0/1893756238 conn(0x7f41c4079ba0 msgr2=0x7f41c4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.568+0000 7f41c903b700 1 -- 172.21.15.7:0/1893756238 shutdown_connections 2024-04-21T08:43:51.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.568+0000 7f41c903b700 1 -- 172.21.15.7:0/1893756238 wait complete. 2024-04-21T08:43:51.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.569+0000 7f41c903b700 1 Processor -- start 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.569+0000 7f41c903b700 1 -- start start 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.569+0000 7f41c903b700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4102a90 0x7f41c41990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.569+0000 7f41c903b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c4103450 0x7f41c4199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.569+0000 7f41c903b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4108a50 0x7f41c419d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.569+0000 7f41c903b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4112c10 con 0x7f41c4103450 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.569+0000 7f41c903b700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4112a90 con 0x7f41c4102a90 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.569+0000 7f41c903b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4112d90 con 0x7f41c4108a50 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c4103450 0x7f41c4199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c4103450 0x7f41c4199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44100/0 (socket says 172.21.15.7:44100) 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41bbfff700 1 -- 172.21.15.7:0/3901528559 learned_addr learned my addr 172.21.15.7:0/3901528559 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41c2d9d700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4102a90 0x7f41c41990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41c359e700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4108a50 0x7f41c419d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41bbfff700 1 -- 172.21.15.7:0/3901528559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4108a50 msgr2=0x7f41c419d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41bbfff700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4108a50 0x7f41c419d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41bbfff700 1 -- 172.21.15.7:0/3901528559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4102a90 msgr2=0x7f41c41990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41bbfff700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4102a90 0x7f41c41990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41bbfff700 1 -- 172.21.15.7:0/3901528559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c419e0f0 con 0x7f41c4103450 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41c359e700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4108a50 0x7f41c419d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41c2d9d700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4102a90 0x7f41c41990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41bbfff700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c4103450 0x7f41c4199620 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00d980 tx=0x7f41ac00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41c0d99700 1 -- 172.21.15.7:0/3901528559 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac017070 con 0x7f41c4103450 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41c903b700 1 -- 172.21.15.7:0/3901528559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c419e3e0 con 0x7f41c4103450 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41c0d99700 1 -- 172.21.15.7:0/3901528559 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41ac004830 con 0x7f41c4103450 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.570+0000 7f41c0d99700 1 -- 172.21.15.7:0/3901528559 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac005440 con 0x7f41c4103450 2024-04-21T08:43:51.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.571+0000 7f41c903b700 1 -- 172.21.15.7:0/3901528559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c41a5c60 con 0x7f41c4103450 2024-04-21T08:43:51.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.571+0000 7f41c903b700 1 -- 172.21.15.7:0/3901528559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c4069730 con 0x7f41c4103450 2024-04-21T08:43:51.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.573+0000 7f41c0d99700 1 -- 172.21.15.7:0/3901528559 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f41ac0055e0 con 0x7f41c4103450 2024-04-21T08:43:51.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.573+0000 7f41c0d99700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f41b00a3d50 0x7f41b00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.573+0000 7f41c2d9d700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f41b00a3d50 0x7f41b00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.573+0000 7f41c0d99700 1 -- 172.21.15.7:0/3901528559 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f41ac0d0bc0 con 0x7f41c4103450 2024-04-21T08:43:51.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.574+0000 7f41c2d9d700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f41b00a3d50 0x7f41b00a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f41b400bb90 tx=0x7f41b4005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.577+0000 7f41c0d99700 1 -- 172.21.15.7:0/3901528559 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41ac020020 con 0x7f41c4103450 2024-04-21T08:43:51.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:51 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/243261689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:43:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.785+0000 7f41c903b700 1 -- 172.21.15.7:0/3901528559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f41c402df80 con 0x7f41c4103450 2024-04-21T08:43:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.786+0000 7f41c0d99700 1 -- 172.21.15.7:0/3901528559 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f41ac09d160 con 0x7f41c4103450 2024-04-21T08:43:51.786 INFO:teuthology.orchestra.run.smithi007.stdout:77309411407 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 -- 172.21.15.7:0/3901528559 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f41b00a3d50 msgr2=0x7f41b00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f41b00a3d50 0x7f41b00a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f41b400bb90 tx=0x7f41b4005d80 comp rx=0 tx=0).stop 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 -- 172.21.15.7:0/3901528559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c4103450 msgr2=0x7f41c4199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c4103450 0x7f41c4199620 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00d980 tx=0x7f41ac00dd40 comp rx=0 tx=0).stop 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 -- 172.21.15.7:0/3901528559 shutdown_connections 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f41b00a3d50 0x7f41b00a6210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4108a50 0x7f41c419d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41c4102a90 0x7f41c41990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 --2- 172.21.15.7:0/3901528559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c4103450 0x7f41c4199620 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 -- 172.21.15.7:0/3901528559 >> 172.21.15.7:0/3901528559 conn(0x7f41c4079ba0 msgr2=0x7f41c4101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 -- 172.21.15.7:0/3901528559 shutdown_connections 2024-04-21T08:43:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.788+0000 7f41c903b700 1 -- 172.21.15.7:0/3901528559 wait complete. 2024-04-21T08:43:51.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:51 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/243261689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:43:51.955 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:51 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/243261689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:43:52.557 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299439 got 223338299445 for osd.7 2024-04-21T08:43:52.557 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:52.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:52 smithi077 ceph-mon[31615]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:52.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:52 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3901528559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:43:52.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:52 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3928987550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:43:52.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:52 smithi007 ceph-mon[28477]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:52.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:52 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3901528559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:43:52.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:52 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3928987550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:43:53.002 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411402 got 77309411407 for osd.2 2024-04-21T08:43:53.003 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:52 smithi016 ceph-mon[31259]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:52 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3901528559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:43:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:52 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3928987550' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:43:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.979+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/982369516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e9406bbb0 msgr2=0x7f5e9410f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.979+0000 7f5e9bd7c700 1 --2- 172.21.15.7:0/982369516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e9406bbb0 0x7f5e9410f930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00b0a0 tx=0x7f5e8c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/982369516 shutdown_connections 2024-04-21T08:43:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 --2- 172.21.15.7:0/982369516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e9406bbb0 0x7f5e9410f930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 --2- 172.21.15.7:0/982369516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e94108a60 0x7f5e94108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 --2- 172.21.15.7:0/982369516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e9406b1f0 0x7f5e9406b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/982369516 >> 172.21.15.7:0/982369516 conn(0x7f5e940789e0 msgr2=0x7f5e94078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/982369516 shutdown_connections 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/982369516 wait complete. 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 Processor -- start 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 -- start start 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9406b1f0 0x7f5e94199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e9406bbb0 0x7f5e941997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e94108a60 0x7f5e9419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5e94112d00 con 0x7f5e9406bbb0 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5e94112b80 con 0x7f5e9406b1f0 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.980+0000 7f5e9bd7c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5e94112e80 con 0x7f5e94108a60 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e99317700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e9406bbb0 0x7f5e941997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e99317700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e9406bbb0 0x7f5e941997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44128/0 (socket says 172.21.15.7:44128) 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e99317700 1 -- 172.21.15.7:0/3928987550 learned_addr learned my addr 172.21.15.7:0/3928987550 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e9a319700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e94108a60 0x7f5e9419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e99b18700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9406b1f0 0x7f5e94199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e99317700 1 -- 172.21.15.7:0/3928987550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e94108a60 msgr2=0x7f5e9419daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e99317700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e94108a60 0x7f5e9419daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e99317700 1 -- 172.21.15.7:0/3928987550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9406b1f0 msgr2=0x7f5e94199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e99317700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9406b1f0 0x7f5e94199270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e99317700 1 -- 172.21.15.7:0/3928987550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9419e230 con 0x7f5e9406bbb0 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e9a319700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e94108a60 0x7f5e9419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e99317700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e9406bbb0 0x7f5e941997b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800b530 tx=0x7f5e8800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e86ffd700 1 -- 172.21.15.7:0/3928987550 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e88015020 con 0x7f5e9406bbb0 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/3928987550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9419e520 con 0x7f5e9406bbb0 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e86ffd700 1 -- 172.21.15.7:0/3928987550 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e88011ce0 con 0x7f5e9406bbb0 2024-04-21T08:43:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e86ffd700 1 -- 172.21.15.7:0/3928987550 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e88010d20 con 0x7f5e9406bbb0 2024-04-21T08:43:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.981+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/3928987550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e941a5da0 con 0x7f5e9406bbb0 2024-04-21T08:43:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.982+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/3928987550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e94069730 con 0x7f5e9406bbb0 2024-04-21T08:43:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.983+0000 7f5e86ffd700 1 -- 172.21.15.7:0/3928987550 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f5e88018020 con 0x7f5e9406bbb0 2024-04-21T08:43:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.984+0000 7f5e86ffd700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5e800a3cd0 0x7f5e800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.984+0000 7f5e86ffd700 1 -- 172.21.15.7:0/3928987550 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f5e880d0560 con 0x7f5e9406bbb0 2024-04-21T08:43:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.984+0000 7f5e99b18700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5e800a3cd0 0x7f5e800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.984+0000 7f5e99b18700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5e800a3cd0 0x7f5e800a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5e90006fd0 tx=0x7f5e90008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:51.988+0000 7f5e86ffd700 1 -- 172.21.15.7:0/3928987550 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5e8801f020 con 0x7f5e9406bbb0 2024-04-21T08:43:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.197+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/3928987550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f5e9410b060 con 0x7f5e9406bbb0 2024-04-21T08:43:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.197+0000 7f5e86ffd700 1 -- 172.21.15.7:0/3928987550 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f5e8809cb90 con 0x7f5e9406bbb0 2024-04-21T08:43:53.112 INFO:teuthology.orchestra.run.smithi007.stdout:313532612645 2024-04-21T08:43:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/3928987550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5e800a3cd0 msgr2=0x7f5e800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5e800a3cd0 0x7f5e800a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5e90006fd0 tx=0x7f5e90008040 comp rx=0 tx=0).stop 2024-04-21T08:43:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/3928987550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e9406bbb0 msgr2=0x7f5e941997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e9406bbb0 0x7f5e941997b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800b530 tx=0x7f5e8800b8f0 comp rx=0 tx=0).stop 2024-04-21T08:43:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/3928987550 shutdown_connections 2024-04-21T08:43:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f5e800a3cd0 0x7f5e800a6190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5e94108a60 0x7f5e9419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9406b1f0 0x7f5e94199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 --2- 172.21.15.7:0/3928987550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e9406bbb0 0x7f5e941997b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/3928987550 >> 172.21.15.7:0/3928987550 conn(0x7f5e940789e0 msgr2=0x7f5e940ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/3928987550 shutdown_connections 2024-04-21T08:43:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:52.199+0000 7f5e9bd7c700 1 -- 172.21.15.7:0/3928987550 wait complete. 2024-04-21T08:43:53.785 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612640 got 313532612645 for osd.10 2024-04-21T08:43:53.785 DEBUG:teuthology.parallel:result is None 2024-04-21T08:43:53.785 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-21T08:43:53.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T08:43:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:54 smithi016 ceph-mon[31259]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:55.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:54 smithi077 ceph-mon[31615]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:55.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:54 smithi007 ceph-mon[28477]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.799+0000 7f03aad7d700 1 -- 172.21.15.7:0/2282306544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03a406bb10 msgr2=0x7f03a410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.799+0000 7f03aad7d700 1 --2- 172.21.15.7:0/2282306544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03a406bb10 0x7f03a410f900 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b0a0 tx=0x7f039c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.799+0000 7f03aad7d700 1 -- 172.21.15.7:0/2282306544 shutdown_connections 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.799+0000 7f03aad7d700 1 --2- 172.21.15.7:0/2282306544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03a4108a30 0x7f03a4108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.799+0000 7f03aad7d700 1 --2- 172.21.15.7:0/2282306544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03a406bb10 0x7f03a410f900 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.799+0000 7f03aad7d700 1 --2- 172.21.15.7:0/2282306544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03a406b170 0x7f03a406b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.799+0000 7f03aad7d700 1 -- 172.21.15.7:0/2282306544 >> 172.21.15.7:0/2282306544 conn(0x7f03a40788f0 msgr2=0x7f03a4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.799+0000 7f03aad7d700 1 -- 172.21.15.7:0/2282306544 shutdown_connections 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.799+0000 7f03aad7d700 1 -- 172.21.15.7:0/2282306544 wait complete. 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03aad7d700 1 Processor -- start 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03aad7d700 1 -- start start 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03aad7d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03a406b170 0x7f03a4101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03aad7d700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03a406bb10 0x7f03a4101f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03aad7d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03a4108a30 0x7f03a41024c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03aad7d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f03a4112bd0 con 0x7f03a4108a30 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03aad7d700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03a4112a50 con 0x7f03a406bb10 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03aad7d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f03a4112d50 con 0x7f03a406b170 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03a931a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03a4108a30 0x7f03a41024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03a931a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03a4108a30 0x7f03a41024c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36266/0 (socket says 172.21.15.7:36266) 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03a931a700 1 -- 172.21.15.7:0/33805985 learned_addr learned my addr 172.21.15.7:0/33805985 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03a3fff700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03a406bb10 0x7f03a4101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03a8b19700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03a406b170 0x7f03a4101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03a931a700 1 -- 172.21.15.7:0/33805985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03a406b170 msgr2=0x7f03a4101a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03a931a700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03a406b170 0x7f03a4101a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03a931a700 1 -- 172.21.15.7:0/33805985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03a406bb10 msgr2=0x7f03a4101f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03a931a700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03a406bb10 0x7f03a4101f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.800+0000 7f03a931a700 1 -- 172.21.15.7:0/33805985 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a41a7570 con 0x7f03a4108a30 2024-04-21T08:43:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.801+0000 7f03a931a700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03a4108a30 0x7f03a41024c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b830 tx=0x7f039c004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.801+0000 7f03a1ffb700 1 -- 172.21.15.7:0/33805985 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c021070 con 0x7f03a4108a30 2024-04-21T08:43:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.801+0000 7f03a1ffb700 1 -- 172.21.15.7:0/33805985 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f039c007e30 con 0x7f03a4108a30 2024-04-21T08:43:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.801+0000 7f03aad7d700 1 -- 172.21.15.7:0/33805985 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a41a7800 con 0x7f03a4108a30 2024-04-21T08:43:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.801+0000 7f03a1ffb700 1 -- 172.21.15.7:0/33805985 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c004700 con 0x7f03a4108a30 2024-04-21T08:43:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.801+0000 7f03aad7d700 1 -- 172.21.15.7:0/33805985 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a41a7cb0 con 0x7f03a4108a30 2024-04-21T08:43:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.802+0000 7f03aad7d700 1 -- 172.21.15.7:0/33805985 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03a4069730 con 0x7f03a4108a30 2024-04-21T08:43:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.803+0000 7f03a1ffb700 1 -- 172.21.15.7:0/33805985 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f039c003d70 con 0x7f03a4108a30 2024-04-21T08:43:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.804+0000 7f03a1ffb700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f038c0a3cd0 0x7f038c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.804+0000 7f03a8b19700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f038c0a3cd0 0x7f038c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.804+0000 7f03a1ffb700 1 -- 172.21.15.7:0/33805985 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f039c00f020 con 0x7f03a4108a30 2024-04-21T08:43:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.804+0000 7f03a8b19700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f038c0a3cd0 0x7f038c0a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f03940045b0 tx=0x7f039400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:55.807+0000 7f03a1ffb700 1 -- 172.21.15.7:0/33805985 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f039c02a020 con 0x7f03a4108a30 2024-04-21T08:43:56.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.011+0000 7f03aad7d700 1 -- 172.21.15.7:0/33805985 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f03a4062e80 con 0x7f038c0a3cd0 2024-04-21T08:43:56.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.021+0000 7f03a1ffb700 1 -- 172.21.15.7:0/33805985 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340591 (secure 0 0 0) 0x7f03a4062e80 con 0x7f038c0a3cd0 2024-04-21T08:43:56.025 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:43:56.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 -- 172.21.15.7:0/33805985 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f038c0a3cd0 msgr2=0x7f038c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:56.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f038c0a3cd0 0x7f038c0a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f03940045b0 tx=0x7f039400b3c0 comp rx=0 tx=0).stop 2024-04-21T08:43:56.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 -- 172.21.15.7:0/33805985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03a4108a30 msgr2=0x7f03a41024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03a4108a30 0x7f03a41024c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b830 tx=0x7f039c004ce0 comp rx=0 tx=0).stop 2024-04-21T08:43:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 -- 172.21.15.7:0/33805985 shutdown_connections 2024-04-21T08:43:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f038c0a3cd0 0x7f038c0a6190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03a406b170 0x7f03a4101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03a4108a30 0x7f03a41024c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 --2- 172.21.15.7:0/33805985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03a406bb10 0x7f03a4101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 -- 172.21.15.7:0/33805985 >> 172.21.15.7:0/33805985 conn(0x7f03a40788f0 msgr2=0x7f03a40ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 -- 172.21.15.7:0/33805985 shutdown_connections 2024-04-21T08:43:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:56.025+0000 7f03aad7d700 1 -- 172.21.15.7:0/33805985 wait complete. 2024-04-21T08:43:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-21T08:43:56.774 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-04-21T08:43:55.653571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654756,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":289,"num_read_kb":260,"num_write":262,"num_write_kb":3030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":402944,"kb_used_data":12104,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124538880,"statfs":{"total":1151940427776,"available":1151527813120,"internally_reserved":0,"allocated":12394496,"data_stored":7986686,"data_compressed":360942,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003479"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788844+0000","last_change":"2024-04-21T08:42:17.545701+0000","last_active":"2024-04-21T08:43:10.788844+0000","last_peered":"2024-04-21T08:43:10.788844+0000","last_clean":"2024-04-21T08:43:10.788844+0000","last_became_active":"2024-04-21T08:42:17.545157+0000","last_became_peered":"2024-04-21T08:42:17.545157+0000","last_unstale":"2024-04-21T08:43:10.788844+0000","last_undegraded":"2024-04-21T08:43:10.788844+0000","last_fullsized":"2024-04-21T08:43:10.788844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:05:41.952879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790294+0000","last_change":"2024-04-21T08:42:17.541153+0000","last_active":"2024-04-21T08:43:10.790294+0000","last_peered":"2024-04-21T08:43:10.790294+0000","last_clean":"2024-04-21T08:43:10.790294+0000","last_became_active":"2024-04-21T08:42:17.540771+0000","last_became_peered":"2024-04-21T08:42:17.540771+0000","last_unstale":"2024-04-21T08:43:10.790294+0000","last_undegraded":"2024-04-21T08:43:10.790294+0000","last_fullsized":"2024-04-21T08:43:10.790294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:35:06.077367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790689+0000","last_change":"2024-04-21T08:42:17.545525+0000","last_active":"2024-04-21T08:43:10.790689+0000","last_peered":"2024-04-21T08:43:10.790689+0000","last_clean":"2024-04-21T08:43:10.790689+0000","last_became_active":"2024-04-21T08:42:17.545168+0000","last_became_peered":"2024-04-21T08:42:17.545168+0000","last_unstale":"2024-04-21T08:43:10.790689+0000","last_undegraded":"2024-04-21T08:43:10.790689+0000","last_fullsized":"2024-04-21T08:43:10.790689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:51.080385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984411+0000","last_change":"2024-04-21T08:42:17.543917+0000","last_active":"2024-04-21T08:43:10.984411+0000","last_peered":"2024-04-21T08:43:10.984411+0000","last_clean":"2024-04-21T08:43:10.984411+0000","last_became_active":"2024-04-21T08:42:17.543513+0000","last_became_peered":"2024-04-21T08:42:17.543513+0000","last_unstale":"2024-04-21T08:43:10.984411+0000","last_undegraded":"2024-04-21T08:43:10.984411+0000","last_fullsized":"2024-04-21T08:43:10.984411+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:23.058600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786620+0000","last_change":"2024-04-21T08:42:17.549737+0000","last_active":"2024-04-21T08:43:10.786620+0000","last_peered":"2024-04-21T08:43:10.786620+0000","last_clean":"2024-04-21T08:43:10.786620+0000","last_became_active":"2024-04-21T08:42:17.549396+0000","last_became_peered":"2024-04-21T08:42:17.549396+0000","last_unstale":"2024-04-21T08:43:10.786620+0000","last_undegraded":"2024-04-21T08:43:10.786620+0000","last_fullsized":"2024-04-21T08:43:10.786620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:01:51.879960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.792321+0000","last_change":"2024-04-21T08:43:10.792321+0000","last_active":"2024-04-21T08:43:10.792321+0000","last_peered":"2024-04-21T08:43:10.792321+0000","last_clean":"2024-04-21T08:43:10.792321+0000","last_became_active":"2024-04-21T08:43:10.791979+0000","last_became_peered":"2024-04-21T08:43:10.791979+0000","last_unstale":"2024-04-21T08:43:10.792321+0000","last_undegraded":"2024-04-21T08:43:10.792321+0000","last_fullsized":"2024-04-21T08:43:10.792321+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:11:16.056778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789571+0000","last_change":"2024-04-21T08:42:17.541172+0000","last_active":"2024-04-21T08:43:10.789571+0000","last_peered":"2024-04-21T08:43:10.789571+0000","last_clean":"2024-04-21T08:43:10.789571+0000","last_became_active":"2024-04-21T08:42:17.540829+0000","last_became_peered":"2024-04-21T08:42:17.540829+0000","last_unstale":"2024-04-21T08:43:10.789571+0000","last_undegraded":"2024-04-21T08:43:10.789571+0000","last_fullsized":"2024-04-21T08:43:10.789571+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:16:12.633980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789003+0000","last_change":"2024-04-21T08:42:17.549744+0000","last_active":"2024-04-21T08:43:10.789003+0000","last_peered":"2024-04-21T08:43:10.789003+0000","last_clean":"2024-04-21T08:43:10.789003+0000","last_became_active":"2024-04-21T08:42:17.549378+0000","last_became_peered":"2024-04-21T08:42:17.549378+0000","last_unstale":"2024-04-21T08:43:10.789003+0000","last_undegraded":"2024-04-21T08:43:10.789003+0000","last_fullsized":"2024-04-21T08:43:10.789003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:03:07.702143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983286+0000","last_change":"2024-04-21T08:42:17.548675+0000","last_active":"2024-04-21T08:43:10.983286+0000","last_peered":"2024-04-21T08:43:10.983286+0000","last_clean":"2024-04-21T08:43:10.983286+0000","last_became_active":"2024-04-21T08:42:17.547809+0000","last_became_peered":"2024-04-21T08:42:17.547809+0000","last_unstale":"2024-04-21T08:43:10.983286+0000","last_undegraded":"2024-04-21T08:43:10.983286+0000","last_fullsized":"2024-04-21T08:43:10.983286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:23.343473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788107+0000","last_change":"2024-04-21T08:42:17.551012+0000","last_active":"2024-04-21T08:43:10.788107+0000","last_peered":"2024-04-21T08:43:10.788107+0000","last_clean":"2024-04-21T08:43:10.788107+0000","last_became_active":"2024-04-21T08:42:17.550627+0000","last_became_peered":"2024-04-21T08:42:17.550627+0000","last_unstale":"2024-04-21T08:43:10.788107+0000","last_undegraded":"2024-04-21T08:43:10.788107+0000","last_fullsized":"2024-04-21T08:43:10.788107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:57.501588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790388+0000","last_change":"2024-04-21T08:42:17.546192+0000","last_active":"2024-04-21T08:43:10.790388+0000","last_peered":"2024-04-21T08:43:10.790388+0000","last_clean":"2024-04-21T08:43:10.790388+0000","last_became_active":"2024-04-21T08:42:17.545808+0000","last_became_peered":"2024-04-21T08:42:17.545808+0000","last_unstale":"2024-04-21T08:43:10.790388+0000","last_undegraded":"2024-04-21T08:43:10.790388+0000","last_fullsized":"2024-04-21T08:43:10.790388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:54.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984291+0000","last_change":"2024-04-21T08:42:17.553351+0000","last_active":"2024-04-21T08:43:10.984291+0000","last_peered":"2024-04-21T08:43:10.984291+0000","last_clean":"2024-04-21T08:43:10.984291+0000","last_became_active":"2024-04-21T08:42:17.550507+0000","last_became_peered":"2024-04-21T08:42:17.550507+0000","last_unstale":"2024-04-21T08:43:10.984291+0000","last_undegraded":"2024-04-21T08:43:10.984291+0000","last_fullsized":"2024-04-21T08:43:10.984291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:25.561375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790352+0000","last_change":"2024-04-21T08:42:17.540533+0000","last_active":"2024-04-21T08:43:10.790352+0000","last_peered":"2024-04-21T08:43:10.790352+0000","last_clean":"2024-04-21T08:43:10.790352+0000","last_became_active":"2024-04-21T08:42:17.539940+0000","last_became_peered":"2024-04-21T08:42:17.539940+0000","last_unstale":"2024-04-21T08:43:10.790352+0000","last_undegraded":"2024-04-21T08:43:10.790352+0000","last_fullsized":"2024-04-21T08:43:10.790352+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:45:19.102899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789978+0000","last_change":"2024-04-21T08:42:17.537794+0000","last_active":"2024-04-21T08:43:10.789978+0000","last_peered":"2024-04-21T08:43:10.789978+0000","last_clean":"2024-04-21T08:43:10.789978+0000","last_became_active":"2024-04-21T08:42:17.537376+0000","last_became_peered":"2024-04-21T08:42:17.537376+0000","last_unstale":"2024-04-21T08:43:10.789978+0000","last_undegraded":"2024-04-21T08:43:10.789978+0000","last_fullsized":"2024-04-21T08:43:10.789978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:57.430036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790075+0000","last_change":"2024-04-21T08:42:17.550574+0000","last_active":"2024-04-21T08:43:10.790075+0000","last_peered":"2024-04-21T08:43:10.790075+0000","last_clean":"2024-04-21T08:43:10.790075+0000","last_became_active":"2024-04-21T08:42:17.550205+0000","last_became_peered":"2024-04-21T08:42:17.550205+0000","last_unstale":"2024-04-21T08:43:10.790075+0000","last_undegraded":"2024-04-21T08:43:10.790075+0000","last_fullsized":"2024-04-21T08:43:10.790075+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:41:08.226477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982659+0000","last_change":"2024-04-21T08:42:17.542239+0000","last_active":"2024-04-21T08:43:10.982659+0000","last_peered":"2024-04-21T08:43:10.982659+0000","last_clean":"2024-04-21T08:43:10.982659+0000","last_became_active":"2024-04-21T08:42:17.541848+0000","last_became_peered":"2024-04-21T08:42:17.541848+0000","last_unstale":"2024-04-21T08:43:10.982659+0000","last_undegraded":"2024-04-21T08:43:10.982659+0000","last_fullsized":"2024-04-21T08:43:10.982659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:47.717451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788708+0000","last_change":"2024-04-21T08:42:17.553200+0000","last_active":"2024-04-21T08:43:10.788708+0000","last_peered":"2024-04-21T08:43:10.788708+0000","last_clean":"2024-04-21T08:43:10.788708+0000","last_became_active":"2024-04-21T08:42:17.552913+0000","last_became_peered":"2024-04-21T08:42:17.552913+0000","last_unstale":"2024-04-21T08:43:10.788708+0000","last_undegraded":"2024-04-21T08:43:10.788708+0000","last_fullsized":"2024-04-21T08:43:10.788708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:39:04.186928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786341+0000","last_change":"2024-04-21T08:42:17.534881+0000","last_active":"2024-04-21T08:43:10.786341+0000","last_peered":"2024-04-21T08:43:10.786341+0000","last_clean":"2024-04-21T08:43:10.786341+0000","last_became_active":"2024-04-21T08:42:17.534496+0000","last_became_peered":"2024-04-21T08:42:17.534496+0000","last_unstale":"2024-04-21T08:43:10.786341+0000","last_undegraded":"2024-04-21T08:43:10.786341+0000","last_fullsized":"2024-04-21T08:43:10.786341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:50:19.409391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981907+0000","last_change":"2024-04-21T08:42:17.540480+0000","last_active":"2024-04-21T08:43:10.981907+0000","last_peered":"2024-04-21T08:43:10.981907+0000","last_clean":"2024-04-21T08:43:10.981907+0000","last_became_active":"2024-04-21T08:42:17.539742+0000","last_became_peered":"2024-04-21T08:42:17.539742+0000","last_unstale":"2024-04-21T08:43:10.981907+0000","last_undegraded":"2024-04-21T08:43:10.981907+0000","last_fullsized":"2024-04-21T08:43:10.981907+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:00.238976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981405+0000","last_change":"2024-04-21T08:42:21.661514+0000","last_active":"2024-04-21T08:43:10.981405+0000","last_peered":"2024-04-21T08:43:10.981405+0000","last_clean":"2024-04-21T08:43:10.981405+0000","last_became_active":"2024-04-21T08:42:21.661095+0000","last_became_peered":"2024-04-21T08:42:21.661095+0000","last_unstale":"2024-04-21T08:43:10.981405+0000","last_undegraded":"2024-04-21T08:43:10.981405+0000","last_fullsized":"2024-04-21T08:43:10.981405+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:27:28.933446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786350+0000","last_change":"2024-04-21T08:42:16.639888+0000","last_active":"2024-04-21T08:43:10.786350+0000","last_peered":"2024-04-21T08:43:10.786350+0000","last_clean":"2024-04-21T08:43:10.786350+0000","last_became_active":"2024-04-21T08:42:16.639403+0000","last_became_peered":"2024-04-21T08:42:16.639403+0000","last_unstale":"2024-04-21T08:43:10.786350+0000","last_undegraded":"2024-04-21T08:43:10.786350+0000","last_fullsized":"2024-04-21T08:43:10.786350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:08:17.164924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983368+0000","last_change":"2024-04-21T08:43:10.983368+0000","last_active":"2024-04-21T08:43:10.983368+0000","last_peered":"2024-04-21T08:43:10.983368+0000","last_clean":"2024-04-21T08:43:10.983368+0000","last_became_active":"2024-04-21T08:43:10.982956+0000","last_became_peered":"2024-04-21T08:43:10.982956+0000","last_unstale":"2024-04-21T08:43:10.983368+0000","last_undegraded":"2024-04-21T08:43:10.983368+0000","last_fullsized":"2024-04-21T08:43:10.983368+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:12:44.919996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982417+0000","last_change":"2024-04-21T08:42:21.673197+0000","last_active":"2024-04-21T08:43:10.982417+0000","last_peered":"2024-04-21T08:43:10.982417+0000","last_clean":"2024-04-21T08:43:10.982417+0000","last_became_active":"2024-04-21T08:42:21.672793+0000","last_became_peered":"2024-04-21T08:42:21.672793+0000","last_unstale":"2024-04-21T08:43:10.982417+0000","last_undegraded":"2024-04-21T08:43:10.982417+0000","last_fullsized":"2024-04-21T08:43:10.982417+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:44:29.788459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790068+0000","last_change":"2024-04-21T08:42:16.648577+0000","last_active":"2024-04-21T08:43:10.790068+0000","last_peered":"2024-04-21T08:43:10.790068+0000","last_clean":"2024-04-21T08:43:10.790068+0000","last_became_active":"2024-04-21T08:42:16.646716+0000","last_became_peered":"2024-04-21T08:42:16.646716+0000","last_unstale":"2024-04-21T08:43:10.790068+0000","last_undegraded":"2024-04-21T08:43:10.790068+0000","last_fullsized":"2024-04-21T08:43:10.790068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:10.327078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790487+0000","last_change":"2024-04-21T08:42:17.545430+0000","last_active":"2024-04-21T08:43:10.790487+0000","last_peered":"2024-04-21T08:43:10.790487+0000","last_clean":"2024-04-21T08:43:10.790487+0000","last_became_active":"2024-04-21T08:42:17.545057+0000","last_became_peered":"2024-04-21T08:42:17.545057+0000","last_unstale":"2024-04-21T08:43:10.790487+0000","last_undegraded":"2024-04-21T08:43:10.790487+0000","last_fullsized":"2024-04-21T08:43:10.790487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:44:59.030830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982777+0000","last_change":"2024-04-21T08:42:21.673196+0000","last_active":"2024-04-21T08:43:10.982777+0000","last_peered":"2024-04-21T08:43:10.982777+0000","last_clean":"2024-04-21T08:43:10.982777+0000","last_became_active":"2024-04-21T08:42:21.672808+0000","last_became_peered":"2024-04-21T08:42:21.672808+0000","last_unstale":"2024-04-21T08:43:10.982777+0000","last_undegraded":"2024-04-21T08:43:10.982777+0000","last_fullsized":"2024-04-21T08:43:10.982777+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:39.482353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787192+0000","last_change":"2024-04-21T08:42:16.641611+0000","last_active":"2024-04-21T08:43:10.787192+0000","last_peered":"2024-04-21T08:43:10.787192+0000","last_clean":"2024-04-21T08:43:10.787192+0000","last_became_active":"2024-04-21T08:42:16.641221+0000","last_became_peered":"2024-04-21T08:42:16.641221+0000","last_unstale":"2024-04-21T08:43:10.787192+0000","last_undegraded":"2024-04-21T08:43:10.787192+0000","last_fullsized":"2024-04-21T08:43:10.787192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:45.496532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982089+0000","last_change":"2024-04-21T08:42:17.532986+0000","last_active":"2024-04-21T08:43:10.982089+0000","last_peered":"2024-04-21T08:43:10.982089+0000","last_clean":"2024-04-21T08:43:10.982089+0000","last_became_active":"2024-04-21T08:42:17.532583+0000","last_became_peered":"2024-04-21T08:42:17.532583+0000","last_unstale":"2024-04-21T08:43:10.982089+0000","last_undegraded":"2024-04-21T08:43:10.982089+0000","last_fullsized":"2024-04-21T08:43:10.982089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:43.780935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982579+0000","last_change":"2024-04-21T08:42:21.667459+0000","last_active":"2024-04-21T08:43:10.982579+0000","last_peered":"2024-04-21T08:43:10.982579+0000","last_clean":"2024-04-21T08:43:10.982579+0000","last_became_active":"2024-04-21T08:42:21.667183+0000","last_became_peered":"2024-04-21T08:42:21.667183+0000","last_unstale":"2024-04-21T08:43:10.982579+0000","last_undegraded":"2024-04-21T08:43:10.982579+0000","last_fullsized":"2024-04-21T08:43:10.982579+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:32:23.626267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787200+0000","last_change":"2024-04-21T08:42:16.645210+0000","last_active":"2024-04-21T08:43:10.787200+0000","last_peered":"2024-04-21T08:43:10.787200+0000","last_clean":"2024-04-21T08:43:10.787200+0000","last_became_active":"2024-04-21T08:42:16.644747+0000","last_became_peered":"2024-04-21T08:42:16.644747+0000","last_unstale":"2024-04-21T08:43:10.787200+0000","last_undegraded":"2024-04-21T08:43:10.787200+0000","last_fullsized":"2024-04-21T08:43:10.787200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:34:45.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984152+0000","last_change":"2024-04-21T08:43:10.984152+0000","last_active":"2024-04-21T08:43:10.984152+0000","last_peered":"2024-04-21T08:43:10.984152+0000","last_clean":"2024-04-21T08:43:10.984152+0000","last_became_active":"2024-04-21T08:43:10.983736+0000","last_became_peered":"2024-04-21T08:43:10.983736+0000","last_unstale":"2024-04-21T08:43:10.984152+0000","last_undegraded":"2024-04-21T08:43:10.984152+0000","last_fullsized":"2024-04-21T08:43:10.984152+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:10.182667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981762+0000","last_change":"2024-04-21T08:42:21.660739+0000","last_active":"2024-04-21T08:43:10.981762+0000","last_peered":"2024-04-21T08:43:10.981762+0000","last_clean":"2024-04-21T08:43:10.981762+0000","last_became_active":"2024-04-21T08:42:21.660307+0000","last_became_peered":"2024-04-21T08:42:21.660307+0000","last_unstale":"2024-04-21T08:43:10.981762+0000","last_undegraded":"2024-04-21T08:43:10.981762+0000","last_fullsized":"2024-04-21T08:43:10.981762+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:26.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.986184+0000","last_change":"2024-04-21T08:43:10.986184+0000","last_active":"2024-04-21T08:43:10.986184+0000","last_peered":"2024-04-21T08:43:10.986184+0000","last_clean":"2024-04-21T08:43:10.986184+0000","last_became_active":"2024-04-21T08:43:10.985533+0000","last_became_peered":"2024-04-21T08:43:10.985533+0000","last_unstale":"2024-04-21T08:43:10.986184+0000","last_undegraded":"2024-04-21T08:43:10.986184+0000","last_fullsized":"2024-04-21T08:43:10.986184+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:26.023306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786815+0000","last_change":"2024-04-21T08:42:17.542315+0000","last_active":"2024-04-21T08:43:10.786815+0000","last_peered":"2024-04-21T08:43:10.786815+0000","last_clean":"2024-04-21T08:43:10.786815+0000","last_became_active":"2024-04-21T08:42:17.541895+0000","last_became_peered":"2024-04-21T08:42:17.541895+0000","last_unstale":"2024-04-21T08:43:10.786815+0000","last_undegraded":"2024-04-21T08:43:10.786815+0000","last_fullsized":"2024-04-21T08:43:10.786815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:23:33.909499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981737+0000","last_change":"2024-04-21T08:42:21.661541+0000","last_active":"2024-04-21T08:43:10.981737+0000","last_peered":"2024-04-21T08:43:10.981737+0000","last_clean":"2024-04-21T08:43:10.981737+0000","last_became_active":"2024-04-21T08:42:21.661121+0000","last_became_peered":"2024-04-21T08:42:21.661121+0000","last_unstale":"2024-04-21T08:43:10.981737+0000","last_undegraded":"2024-04-21T08:43:10.981737+0000","last_fullsized":"2024-04-21T08:43:10.981737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:28.624203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984839+0000","last_change":"2024-04-21T08:43:10.984839+0000","last_active":"2024-04-21T08:43:10.984839+0000","last_peered":"2024-04-21T08:43:10.984839+0000","last_clean":"2024-04-21T08:43:10.984839+0000","last_became_active":"2024-04-21T08:43:10.984382+0000","last_became_peered":"2024-04-21T08:43:10.984382+0000","last_unstale":"2024-04-21T08:43:10.984839+0000","last_undegraded":"2024-04-21T08:43:10.984839+0000","last_fullsized":"2024-04-21T08:43:10.984839+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:47:20.774102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791380+0000","last_change":"2024-04-21T08:42:17.547770+0000","last_active":"2024-04-21T08:43:10.791380+0000","last_peered":"2024-04-21T08:43:10.791380+0000","last_clean":"2024-04-21T08:43:10.791380+0000","last_became_active":"2024-04-21T08:42:17.547440+0000","last_became_peered":"2024-04-21T08:42:17.547440+0000","last_unstale":"2024-04-21T08:43:10.791380+0000","last_undegraded":"2024-04-21T08:43:10.791380+0000","last_fullsized":"2024-04-21T08:43:10.791380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:37:35.555737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788605+0000","last_change":"2024-04-21T08:42:21.670684+0000","last_active":"2024-04-21T08:43:10.788605+0000","last_peered":"2024-04-21T08:43:10.788605+0000","last_clean":"2024-04-21T08:43:10.788605+0000","last_became_active":"2024-04-21T08:42:21.670260+0000","last_became_peered":"2024-04-21T08:42:21.670260+0000","last_unstale":"2024-04-21T08:43:10.788605+0000","last_undegraded":"2024-04-21T08:43:10.788605+0000","last_fullsized":"2024-04-21T08:43:10.788605+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:45:06.192040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790350+0000","last_change":"2024-04-21T08:42:16.639632+0000","last_active":"2024-04-21T08:43:10.790350+0000","last_peered":"2024-04-21T08:43:10.790350+0000","last_clean":"2024-04-21T08:43:10.790350+0000","last_became_active":"2024-04-21T08:42:16.639301+0000","last_became_peered":"2024-04-21T08:42:16.639301+0000","last_unstale":"2024-04-21T08:43:10.790350+0000","last_undegraded":"2024-04-21T08:43:10.790350+0000","last_fullsized":"2024-04-21T08:43:10.790350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:58.838667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789111+0000","last_change":"2024-04-21T08:42:17.546109+0000","last_active":"2024-04-21T08:43:10.789111+0000","last_peered":"2024-04-21T08:43:10.789111+0000","last_clean":"2024-04-21T08:43:10.789111+0000","last_became_active":"2024-04-21T08:42:17.545673+0000","last_became_peered":"2024-04-21T08:42:17.545673+0000","last_unstale":"2024-04-21T08:43:10.789111+0000","last_undegraded":"2024-04-21T08:43:10.789111+0000","last_fullsized":"2024-04-21T08:43:10.789111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:18.947848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790068+0000","last_change":"2024-04-21T08:42:21.666114+0000","last_active":"2024-04-21T08:43:10.790068+0000","last_peered":"2024-04-21T08:43:10.790068+0000","last_clean":"2024-04-21T08:43:10.790068+0000","last_became_active":"2024-04-21T08:42:21.665020+0000","last_became_peered":"2024-04-21T08:42:21.665020+0000","last_unstale":"2024-04-21T08:43:10.790068+0000","last_undegraded":"2024-04-21T08:43:10.790068+0000","last_fullsized":"2024-04-21T08:43:10.790068+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:33.644178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790644+0000","last_change":"2024-04-21T08:42:16.639118+0000","last_active":"2024-04-21T08:43:10.790644+0000","last_peered":"2024-04-21T08:43:10.790644+0000","last_clean":"2024-04-21T08:43:10.790644+0000","last_became_active":"2024-04-21T08:42:16.638792+0000","last_became_peered":"2024-04-21T08:42:16.638792+0000","last_unstale":"2024-04-21T08:43:10.790644+0000","last_undegraded":"2024-04-21T08:43:10.790644+0000","last_fullsized":"2024-04-21T08:43:10.790644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:55.568712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.785869+0000","last_change":"2024-04-21T08:42:17.539688+0000","last_active":"2024-04-21T08:43:10.785869+0000","last_peered":"2024-04-21T08:43:10.785869+0000","last_clean":"2024-04-21T08:43:10.785869+0000","last_became_active":"2024-04-21T08:42:17.539335+0000","last_became_peered":"2024-04-21T08:42:17.539335+0000","last_unstale":"2024-04-21T08:43:10.785869+0000","last_undegraded":"2024-04-21T08:43:10.785869+0000","last_fullsized":"2024-04-21T08:43:10.785869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:27:51.520232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789278+0000","last_change":"2024-04-21T08:42:21.660570+0000","last_active":"2024-04-21T08:43:10.789278+0000","last_peered":"2024-04-21T08:43:10.789278+0000","last_clean":"2024-04-21T08:43:10.789278+0000","last_became_active":"2024-04-21T08:42:21.659398+0000","last_became_peered":"2024-04-21T08:42:21.659398+0000","last_unstale":"2024-04-21T08:43:10.789278+0000","last_undegraded":"2024-04-21T08:43:10.789278+0000","last_fullsized":"2024-04-21T08:43:10.789278+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:55.840334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787693+0000","last_change":"2024-04-21T08:42:16.648192+0000","last_active":"2024-04-21T08:43:10.787693+0000","last_peered":"2024-04-21T08:43:10.787693+0000","last_clean":"2024-04-21T08:43:10.787693+0000","last_became_active":"2024-04-21T08:42:16.647666+0000","last_became_peered":"2024-04-21T08:42:16.647666+0000","last_unstale":"2024-04-21T08:43:10.787693+0000","last_undegraded":"2024-04-21T08:43:10.787693+0000","last_fullsized":"2024-04-21T08:43:10.787693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:33.344680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983282+0000","last_change":"2024-04-21T08:42:17.545925+0000","last_active":"2024-04-21T08:43:10.983282+0000","last_peered":"2024-04-21T08:43:10.983282+0000","last_clean":"2024-04-21T08:43:10.983282+0000","last_became_active":"2024-04-21T08:42:17.545555+0000","last_became_peered":"2024-04-21T08:42:17.545555+0000","last_unstale":"2024-04-21T08:43:10.983282+0000","last_undegraded":"2024-04-21T08:43:10.983282+0000","last_fullsized":"2024-04-21T08:43:10.983282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:13:07.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.785669+0000","last_change":"2024-04-21T08:42:21.662757+0000","last_active":"2024-04-21T08:43:10.785669+0000","last_peered":"2024-04-21T08:43:10.785669+0000","last_clean":"2024-04-21T08:43:10.785669+0000","last_became_active":"2024-04-21T08:42:21.662306+0000","last_became_peered":"2024-04-21T08:42:21.662306+0000","last_unstale":"2024-04-21T08:43:10.785669+0000","last_undegraded":"2024-04-21T08:43:10.785669+0000","last_fullsized":"2024-04-21T08:43:10.785669+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:26.856147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984690+0000","last_change":"2024-04-21T08:43:10.984690+0000","last_active":"2024-04-21T08:43:10.984690+0000","last_peered":"2024-04-21T08:43:10.984690+0000","last_clean":"2024-04-21T08:43:10.984690+0000","last_became_active":"2024-04-21T08:43:10.984290+0000","last_became_peered":"2024-04-21T08:43:10.984290+0000","last_unstale":"2024-04-21T08:43:10.984690+0000","last_undegraded":"2024-04-21T08:43:10.984690+0000","last_fullsized":"2024-04-21T08:43:10.984690+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:37:36.400946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.793316+0000","last_change":"2024-04-21T08:43:10.793316+0000","last_active":"2024-04-21T08:43:10.793316+0000","last_peered":"2024-04-21T08:43:10.793316+0000","last_clean":"2024-04-21T08:43:10.793316+0000","last_became_active":"2024-04-21T08:43:10.793035+0000","last_became_peered":"2024-04-21T08:43:10.793035+0000","last_unstale":"2024-04-21T08:43:10.793316+0000","last_undegraded":"2024-04-21T08:43:10.793316+0000","last_fullsized":"2024-04-21T08:43:10.793316+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:41:04.247385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983457+0000","last_change":"2024-04-21T08:42:21.670561+0000","last_active":"2024-04-21T08:43:10.983457+0000","last_peered":"2024-04-21T08:43:10.983457+0000","last_clean":"2024-04-21T08:43:10.983457+0000","last_became_active":"2024-04-21T08:42:21.670170+0000","last_became_peered":"2024-04-21T08:42:21.670170+0000","last_unstale":"2024-04-21T08:43:10.983457+0000","last_undegraded":"2024-04-21T08:43:10.983457+0000","last_fullsized":"2024-04-21T08:43:10.983457+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:35:54.538013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790743+0000","last_change":"2024-04-21T08:42:16.637319+0000","last_active":"2024-04-21T08:43:10.790743+0000","last_peered":"2024-04-21T08:43:10.790743+0000","last_clean":"2024-04-21T08:43:10.790743+0000","last_became_active":"2024-04-21T08:42:16.636940+0000","last_became_peered":"2024-04-21T08:42:16.636940+0000","last_unstale":"2024-04-21T08:43:10.790743+0000","last_undegraded":"2024-04-21T08:43:10.790743+0000","last_fullsized":"2024-04-21T08:43:10.790743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:59:09.826786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982487+0000","last_change":"2024-04-21T08:42:17.546202+0000","last_active":"2024-04-21T08:43:10.982487+0000","last_peered":"2024-04-21T08:43:10.982487+0000","last_clean":"2024-04-21T08:43:10.982487+0000","last_became_active":"2024-04-21T08:42:17.545781+0000","last_became_peered":"2024-04-21T08:42:17.545781+0000","last_unstale":"2024-04-21T08:43:10.982487+0000","last_undegraded":"2024-04-21T08:43:10.982487+0000","last_fullsized":"2024-04-21T08:43:10.982487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:23.990951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.785762+0000","last_change":"2024-04-21T08:42:21.662822+0000","last_active":"2024-04-21T08:43:10.785762+0000","last_peered":"2024-04-21T08:43:10.785762+0000","last_clean":"2024-04-21T08:43:10.785762+0000","last_became_active":"2024-04-21T08:42:21.662398+0000","last_became_peered":"2024-04-21T08:42:21.662398+0000","last_unstale":"2024-04-21T08:43:10.785762+0000","last_undegraded":"2024-04-21T08:43:10.785762+0000","last_fullsized":"2024-04-21T08:43:10.785762+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:38:40.499294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":204,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:51.050320+0000","last_change":"2024-04-21T08:42:16.636330+0000","last_active":"2024-04-21T08:43:51.050320+0000","last_peered":"2024-04-21T08:43:51.050320+0000","last_clean":"2024-04-21T08:43:51.050320+0000","last_became_active":"2024-04-21T08:42:16.635762+0000","last_became_peered":"2024-04-21T08:42:16.635762+0000","last_unstale":"2024-04-21T08:43:51.050320+0000","last_undegraded":"2024-04-21T08:43:51.050320+0000","last_fullsized":"2024-04-21T08:43:51.050320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:31.137923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":147,"num_read_kb":135,"num_write":30,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787464+0000","last_change":"2024-04-21T08:42:17.543033+0000","last_active":"2024-04-21T08:43:10.787464+0000","last_peered":"2024-04-21T08:43:10.787464+0000","last_clean":"2024-04-21T08:43:10.787464+0000","last_became_active":"2024-04-21T08:42:17.542692+0000","last_became_peered":"2024-04-21T08:42:17.542692+0000","last_unstale":"2024-04-21T08:43:10.787464+0000","last_undegraded":"2024-04-21T08:43:10.787464+0000","last_fullsized":"2024-04-21T08:43:10.787464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:33:32.111999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791344+0000","last_change":"2024-04-21T08:42:21.668831+0000","last_active":"2024-04-21T08:43:10.791344+0000","last_peered":"2024-04-21T08:43:10.791344+0000","last_clean":"2024-04-21T08:43:10.791344+0000","last_became_active":"2024-04-21T08:42:21.668485+0000","last_became_peered":"2024-04-21T08:42:21.668485+0000","last_unstale":"2024-04-21T08:43:10.791344+0000","last_undegraded":"2024-04-21T08:43:10.791344+0000","last_fullsized":"2024-04-21T08:43:10.791344+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:49:29.049707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981747+0000","last_change":"2024-04-21T08:42:16.640288+0000","last_active":"2024-04-21T08:43:10.981747+0000","last_peered":"2024-04-21T08:43:10.981747+0000","last_clean":"2024-04-21T08:43:10.981747+0000","last_became_active":"2024-04-21T08:42:16.639909+0000","last_became_peered":"2024-04-21T08:42:16.639909+0000","last_unstale":"2024-04-21T08:43:10.981747+0000","last_undegraded":"2024-04-21T08:43:10.981747+0000","last_fullsized":"2024-04-21T08:43:10.981747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:33.839442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790747+0000","last_change":"2024-04-21T08:42:17.541154+0000","last_active":"2024-04-21T08:43:10.790747+0000","last_peered":"2024-04-21T08:43:10.790747+0000","last_clean":"2024-04-21T08:43:10.790747+0000","last_became_active":"2024-04-21T08:42:17.540747+0000","last_became_peered":"2024-04-21T08:42:17.540747+0000","last_unstale":"2024-04-21T08:43:10.790747+0000","last_undegraded":"2024-04-21T08:43:10.790747+0000","last_fullsized":"2024-04-21T08:43:10.790747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:14:50.641144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788325+0000","last_change":"2024-04-21T08:42:21.670767+0000","last_active":"2024-04-21T08:43:10.788325+0000","last_peered":"2024-04-21T08:43:10.788325+0000","last_clean":"2024-04-21T08:43:10.788325+0000","last_became_active":"2024-04-21T08:42:21.670338+0000","last_became_peered":"2024-04-21T08:42:21.670338+0000","last_unstale":"2024-04-21T08:43:10.788325+0000","last_undegraded":"2024-04-21T08:43:10.788325+0000","last_fullsized":"2024-04-21T08:43:10.788325+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:38:55.275603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790663+0000","last_change":"2024-04-21T08:42:16.640144+0000","last_active":"2024-04-21T08:43:10.790663+0000","last_peered":"2024-04-21T08:43:10.790663+0000","last_clean":"2024-04-21T08:43:10.790663+0000","last_became_active":"2024-04-21T08:42:16.639707+0000","last_became_peered":"2024-04-21T08:42:16.639707+0000","last_unstale":"2024-04-21T08:43:10.790663+0000","last_undegraded":"2024-04-21T08:43:10.790663+0000","last_fullsized":"2024-04-21T08:43:10.790663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:11:32.785519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790033+0000","last_change":"2024-04-21T08:42:17.555019+0000","last_active":"2024-04-21T08:43:10.790033+0000","last_peered":"2024-04-21T08:43:10.790033+0000","last_clean":"2024-04-21T08:43:10.790033+0000","last_became_active":"2024-04-21T08:42:17.554645+0000","last_became_peered":"2024-04-21T08:42:17.554645+0000","last_unstale":"2024-04-21T08:43:10.790033+0000","last_undegraded":"2024-04-21T08:43:10.790033+0000","last_fullsized":"2024-04-21T08:43:10.790033+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:42:30.353945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786133+0000","last_change":"2024-04-21T08:42:21.663567+0000","last_active":"2024-04-21T08:43:10.786133+0000","last_peered":"2024-04-21T08:43:10.786133+0000","last_clean":"2024-04-21T08:43:10.786133+0000","last_became_active":"2024-04-21T08:42:21.663203+0000","last_became_peered":"2024-04-21T08:42:21.663203+0000","last_unstale":"2024-04-21T08:43:10.786133+0000","last_undegraded":"2024-04-21T08:43:10.786133+0000","last_fullsized":"2024-04-21T08:43:10.786133+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:50.760336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982329+0000","last_change":"2024-04-21T08:42:16.640407+0000","last_active":"2024-04-21T08:43:10.982329+0000","last_peered":"2024-04-21T08:43:10.982329+0000","last_clean":"2024-04-21T08:43:10.982329+0000","last_became_active":"2024-04-21T08:42:16.640015+0000","last_became_peered":"2024-04-21T08:42:16.640015+0000","last_unstale":"2024-04-21T08:43:10.982329+0000","last_undegraded":"2024-04-21T08:43:10.982329+0000","last_fullsized":"2024-04-21T08:43:10.982329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:23:10.647184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789722+0000","last_change":"2024-04-21T08:42:17.545438+0000","last_active":"2024-04-21T08:43:10.789722+0000","last_peered":"2024-04-21T08:43:10.789722+0000","last_clean":"2024-04-21T08:43:10.789722+0000","last_became_active":"2024-04-21T08:42:17.544258+0000","last_became_peered":"2024-04-21T08:42:17.544258+0000","last_unstale":"2024-04-21T08:43:10.789722+0000","last_undegraded":"2024-04-21T08:43:10.789722+0000","last_fullsized":"2024-04-21T08:43:10.789722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:43.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791292+0000","last_change":"2024-04-21T08:42:21.671000+0000","last_active":"2024-04-21T08:43:10.791292+0000","last_peered":"2024-04-21T08:43:10.791292+0000","last_clean":"2024-04-21T08:43:10.791292+0000","last_became_active":"2024-04-21T08:42:21.670380+0000","last_became_peered":"2024-04-21T08:42:21.670380+0000","last_unstale":"2024-04-21T08:43:10.791292+0000","last_undegraded":"2024-04-21T08:43:10.791292+0000","last_fullsized":"2024-04-21T08:43:10.791292+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:57.075686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786538+0000","last_change":"2024-04-21T08:42:16.649599+0000","last_active":"2024-04-21T08:43:10.786538+0000","last_peered":"2024-04-21T08:43:10.786538+0000","last_clean":"2024-04-21T08:43:10.786538+0000","last_became_active":"2024-04-21T08:42:16.649232+0000","last_became_peered":"2024-04-21T08:42:16.649232+0000","last_unstale":"2024-04-21T08:43:10.786538+0000","last_undegraded":"2024-04-21T08:43:10.786538+0000","last_fullsized":"2024-04-21T08:43:10.786538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:21.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790390+0000","last_change":"2024-04-21T08:42:17.545910+0000","last_active":"2024-04-21T08:43:10.790390+0000","last_peered":"2024-04-21T08:43:10.790390+0000","last_clean":"2024-04-21T08:43:10.790390+0000","last_became_active":"2024-04-21T08:42:17.545582+0000","last_became_peered":"2024-04-21T08:42:17.545582+0000","last_unstale":"2024-04-21T08:43:10.790390+0000","last_undegraded":"2024-04-21T08:43:10.790390+0000","last_fullsized":"2024-04-21T08:43:10.790390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:25.466146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790378+0000","last_change":"2024-04-21T08:42:21.664811+0000","last_active":"2024-04-21T08:43:10.790378+0000","last_peered":"2024-04-21T08:43:10.790378+0000","last_clean":"2024-04-21T08:43:10.790378+0000","last_became_active":"2024-04-21T08:42:21.664332+0000","last_became_peered":"2024-04-21T08:42:21.664332+0000","last_unstale":"2024-04-21T08:43:10.790378+0000","last_undegraded":"2024-04-21T08:43:10.790378+0000","last_fullsized":"2024-04-21T08:43:10.790378+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:40.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787316+0000","last_change":"2024-04-21T08:42:16.641233+0000","last_active":"2024-04-21T08:43:10.787316+0000","last_peered":"2024-04-21T08:43:10.787316+0000","last_clean":"2024-04-21T08:43:10.787316+0000","last_became_active":"2024-04-21T08:42:16.640873+0000","last_became_peered":"2024-04-21T08:42:16.640873+0000","last_unstale":"2024-04-21T08:43:10.787316+0000","last_undegraded":"2024-04-21T08:43:10.787316+0000","last_fullsized":"2024-04-21T08:43:10.787316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:00.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788866+0000","last_change":"2024-04-21T08:42:17.550438+0000","last_active":"2024-04-21T08:43:10.788866+0000","last_peered":"2024-04-21T08:43:10.788866+0000","last_clean":"2024-04-21T08:43:10.788866+0000","last_became_active":"2024-04-21T08:42:17.550015+0000","last_became_peered":"2024-04-21T08:42:17.550015+0000","last_unstale":"2024-04-21T08:43:10.788866+0000","last_undegraded":"2024-04-21T08:43:10.788866+0000","last_fullsized":"2024-04-21T08:43:10.788866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:00:29.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786730+0000","last_change":"2024-04-21T08:42:21.970498+0000","last_active":"2024-04-21T08:43:10.786730+0000","last_peered":"2024-04-21T08:43:10.786730+0000","last_clean":"2024-04-21T08:43:10.786730+0000","last_became_active":"2024-04-21T08:42:21.969819+0000","last_became_peered":"2024-04-21T08:42:21.969819+0000","last_unstale":"2024-04-21T08:43:10.786730+0000","last_undegraded":"2024-04-21T08:43:10.786730+0000","last_fullsized":"2024-04-21T08:43:10.786730+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:13:16.552293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786143+0000","last_change":"2024-04-21T08:42:16.637146+0000","last_active":"2024-04-21T08:43:10.786143+0000","last_peered":"2024-04-21T08:43:10.786143+0000","last_clean":"2024-04-21T08:43:10.786143+0000","last_became_active":"2024-04-21T08:42:16.636781+0000","last_became_peered":"2024-04-21T08:42:16.636781+0000","last_unstale":"2024-04-21T08:43:10.786143+0000","last_undegraded":"2024-04-21T08:43:10.786143+0000","last_fullsized":"2024-04-21T08:43:10.786143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:39:27.641236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787514+0000","last_change":"2024-04-21T08:42:17.542325+0000","last_active":"2024-04-21T08:43:10.787514+0000","last_peered":"2024-04-21T08:43:10.787514+0000","last_clean":"2024-04-21T08:43:10.787514+0000","last_became_active":"2024-04-21T08:42:17.541103+0000","last_became_peered":"2024-04-21T08:42:17.541103+0000","last_unstale":"2024-04-21T08:43:10.787514+0000","last_undegraded":"2024-04-21T08:43:10.787514+0000","last_fullsized":"2024-04-21T08:43:10.787514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:46.903250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790533+0000","last_change":"2024-04-21T08:42:21.671146+0000","last_active":"2024-04-21T08:43:10.790533+0000","last_peered":"2024-04-21T08:43:10.790533+0000","last_clean":"2024-04-21T08:43:10.790533+0000","last_became_active":"2024-04-21T08:42:21.670561+0000","last_became_peered":"2024-04-21T08:42:21.670561+0000","last_unstale":"2024-04-21T08:43:10.790533+0000","last_undegraded":"2024-04-21T08:43:10.790533+0000","last_fullsized":"2024-04-21T08:43:10.790533+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:38:28.028976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789729+0000","last_change":"2024-04-21T08:42:16.645132+0000","last_active":"2024-04-21T08:43:10.789729+0000","last_peered":"2024-04-21T08:43:10.789729+0000","last_clean":"2024-04-21T08:43:10.789729+0000","last_became_active":"2024-04-21T08:42:16.644786+0000","last_became_peered":"2024-04-21T08:42:16.644786+0000","last_unstale":"2024-04-21T08:43:10.789729+0000","last_undegraded":"2024-04-21T08:43:10.789729+0000","last_fullsized":"2024-04-21T08:43:10.789729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:08.298498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787281+0000","last_change":"2024-04-21T08:42:17.550424+0000","last_active":"2024-04-21T08:43:10.787281+0000","last_peered":"2024-04-21T08:43:10.787281+0000","last_clean":"2024-04-21T08:43:10.787281+0000","last_became_active":"2024-04-21T08:42:17.550077+0000","last_became_peered":"2024-04-21T08:42:17.550077+0000","last_unstale":"2024-04-21T08:43:10.787281+0000","last_undegraded":"2024-04-21T08:43:10.787281+0000","last_fullsized":"2024-04-21T08:43:10.787281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:17:07.016204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789770+0000","last_change":"2024-04-21T08:42:21.970652+0000","last_active":"2024-04-21T08:43:10.789770+0000","last_peered":"2024-04-21T08:43:10.789770+0000","last_clean":"2024-04-21T08:43:10.789770+0000","last_became_active":"2024-04-21T08:42:21.970251+0000","last_became_peered":"2024-04-21T08:42:21.970251+0000","last_unstale":"2024-04-21T08:43:10.789770+0000","last_undegraded":"2024-04-21T08:43:10.789770+0000","last_fullsized":"2024-04-21T08:43:10.789770+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:45.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790879+0000","last_change":"2024-04-21T08:42:17.542820+0000","last_active":"2024-04-21T08:43:10.790879+0000","last_peered":"2024-04-21T08:43:10.790879+0000","last_clean":"2024-04-21T08:43:10.790879+0000","last_became_active":"2024-04-21T08:42:17.541320+0000","last_became_peered":"2024-04-21T08:42:17.541320+0000","last_unstale":"2024-04-21T08:43:10.790879+0000","last_undegraded":"2024-04-21T08:43:10.790879+0000","last_fullsized":"2024-04-21T08:43:10.790879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:23:06.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790253+0000","last_change":"2024-04-21T08:42:16.636950+0000","last_active":"2024-04-21T08:43:10.790253+0000","last_peered":"2024-04-21T08:43:10.790253+0000","last_clean":"2024-04-21T08:43:10.790253+0000","last_became_active":"2024-04-21T08:42:16.636590+0000","last_became_peered":"2024-04-21T08:42:16.636590+0000","last_unstale":"2024-04-21T08:43:10.790253+0000","last_undegraded":"2024-04-21T08:43:10.790253+0000","last_fullsized":"2024-04-21T08:43:10.790253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:55.693681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790154+0000","last_change":"2024-04-21T08:42:21.660404+0000","last_active":"2024-04-21T08:43:10.790154+0000","last_peered":"2024-04-21T08:43:10.790154+0000","last_clean":"2024-04-21T08:43:10.790154+0000","last_became_active":"2024-04-21T08:42:21.659999+0000","last_became_peered":"2024-04-21T08:42:21.659999+0000","last_unstale":"2024-04-21T08:43:10.790154+0000","last_undegraded":"2024-04-21T08:43:10.790154+0000","last_fullsized":"2024-04-21T08:43:10.790154+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:53:32.328873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984671+0000","last_change":"2024-04-21T08:43:10.984361+0000","last_active":"2024-04-21T08:43:10.984671+0000","last_peered":"2024-04-21T08:43:10.984671+0000","last_clean":"2024-04-21T08:43:10.984671+0000","last_became_active":"2024-04-21T08:43:10.984046+0000","last_became_peered":"2024-04-21T08:43:10.984046+0000","last_unstale":"2024-04-21T08:43:10.984671+0000","last_undegraded":"2024-04-21T08:43:10.984671+0000","last_fullsized":"2024-04-21T08:43:10.984671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:33:23.154630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984798+0000","last_change":"2024-04-21T08:42:16.638993+0000","last_active":"2024-04-21T08:43:10.984798+0000","last_peered":"2024-04-21T08:43:10.984798+0000","last_clean":"2024-04-21T08:43:10.984798+0000","last_became_active":"2024-04-21T08:42:16.638617+0000","last_became_peered":"2024-04-21T08:42:16.638617+0000","last_unstale":"2024-04-21T08:43:10.984798+0000","last_undegraded":"2024-04-21T08:43:10.984798+0000","last_fullsized":"2024-04-21T08:43:10.984798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:14:28.271433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":360,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789557+0000","last_change":"2024-04-21T08:41:00.044984+0000","last_active":"2024-04-21T08:43:10.789557+0000","last_peered":"2024-04-21T08:43:10.789557+0000","last_clean":"2024-04-21T08:43:10.789557+0000","last_became_active":"2024-04-21T08:41:00.044485+0000","last_became_peered":"2024-04-21T08:41:00.044485+0000","last_unstale":"2024-04-21T08:43:10.789557+0000","last_undegraded":"2024-04-21T08:43:10.789557+0000","last_fullsized":"2024-04-21T08:43:10.789557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.370031+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:01.031209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791249+0000","last_change":"2024-04-21T08:42:21.971045+0000","last_active":"2024-04-21T08:43:10.791249+0000","last_peered":"2024-04-21T08:43:10.791249+0000","last_clean":"2024-04-21T08:43:10.791249+0000","last_became_active":"2024-04-21T08:42:21.970577+0000","last_became_peered":"2024-04-21T08:42:21.970577+0000","last_unstale":"2024-04-21T08:43:10.791249+0000","last_undegraded":"2024-04-21T08:43:10.791249+0000","last_fullsized":"2024-04-21T08:43:10.791249+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:55.681662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790149+0000","last_change":"2024-04-21T08:42:17.535570+0000","last_active":"2024-04-21T08:43:10.790149+0000","last_peered":"2024-04-21T08:43:10.790149+0000","last_clean":"2024-04-21T08:43:10.790149+0000","last_became_active":"2024-04-21T08:42:17.535217+0000","last_became_peered":"2024-04-21T08:42:17.535217+0000","last_unstale":"2024-04-21T08:43:10.790149+0000","last_undegraded":"2024-04-21T08:43:10.790149+0000","last_fullsized":"2024-04-21T08:43:10.790149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:49.510750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.792548+0000","last_change":"2024-04-21T08:43:10.792548+0000","last_active":"2024-04-21T08:43:10.792548+0000","last_peered":"2024-04-21T08:43:10.792548+0000","last_clean":"2024-04-21T08:43:10.792548+0000","last_became_active":"2024-04-21T08:43:10.792165+0000","last_became_peered":"2024-04-21T08:43:10.792165+0000","last_unstale":"2024-04-21T08:43:10.792548+0000","last_undegraded":"2024-04-21T08:43:10.792548+0000","last_fullsized":"2024-04-21T08:43:10.792548+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:35.980536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788172+0000","last_change":"2024-04-21T08:42:21.661393+0000","last_active":"2024-04-21T08:43:10.788172+0000","last_peered":"2024-04-21T08:43:10.788172+0000","last_clean":"2024-04-21T08:43:10.788172+0000","last_became_active":"2024-04-21T08:42:21.660201+0000","last_became_peered":"2024-04-21T08:42:21.660201+0000","last_unstale":"2024-04-21T08:43:10.788172+0000","last_undegraded":"2024-04-21T08:43:10.788172+0000","last_fullsized":"2024-04-21T08:43:10.788172+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:13.393389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787392+0000","last_change":"2024-04-21T08:42:16.650507+0000","last_active":"2024-04-21T08:43:10.787392+0000","last_peered":"2024-04-21T08:43:10.787392+0000","last_clean":"2024-04-21T08:43:10.787392+0000","last_became_active":"2024-04-21T08:42:16.650217+0000","last_became_peered":"2024-04-21T08:42:16.650217+0000","last_unstale":"2024-04-21T08:43:10.787392+0000","last_undegraded":"2024-04-21T08:43:10.787392+0000","last_fullsized":"2024-04-21T08:43:10.787392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:13:26.785092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790643+0000","last_change":"2024-04-21T08:42:17.530420+0000","last_active":"2024-04-21T08:43:10.790643+0000","last_peered":"2024-04-21T08:43:10.790643+0000","last_clean":"2024-04-21T08:43:10.790643+0000","last_became_active":"2024-04-21T08:42:17.529920+0000","last_became_peered":"2024-04-21T08:42:17.529920+0000","last_unstale":"2024-04-21T08:43:10.790643+0000","last_undegraded":"2024-04-21T08:43:10.790643+0000","last_fullsized":"2024-04-21T08:43:10.790643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:24.250197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789027+0000","last_change":"2024-04-21T08:42:21.970586+0000","last_active":"2024-04-21T08:43:10.789027+0000","last_peered":"2024-04-21T08:43:10.789027+0000","last_clean":"2024-04-21T08:43:10.789027+0000","last_became_active":"2024-04-21T08:42:21.970132+0000","last_became_peered":"2024-04-21T08:42:21.970132+0000","last_unstale":"2024-04-21T08:43:10.789027+0000","last_undegraded":"2024-04-21T08:43:10.789027+0000","last_fullsized":"2024-04-21T08:43:10.789027+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:11.074370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786265+0000","last_change":"2024-04-21T08:42:16.647916+0000","last_active":"2024-04-21T08:43:10.786265+0000","last_peered":"2024-04-21T08:43:10.786265+0000","last_clean":"2024-04-21T08:43:10.786265+0000","last_became_active":"2024-04-21T08:42:16.647476+0000","last_became_peered":"2024-04-21T08:42:16.647476+0000","last_unstale":"2024-04-21T08:43:10.786265+0000","last_undegraded":"2024-04-21T08:43:10.786265+0000","last_fullsized":"2024-04-21T08:43:10.786265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:27.242153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789667+0000","last_change":"2024-04-21T08:42:17.546144+0000","last_active":"2024-04-21T08:43:10.789667+0000","last_peered":"2024-04-21T08:43:10.789667+0000","last_clean":"2024-04-21T08:43:10.789667+0000","last_became_active":"2024-04-21T08:42:17.545730+0000","last_became_peered":"2024-04-21T08:42:17.545730+0000","last_unstale":"2024-04-21T08:43:10.789667+0000","last_undegraded":"2024-04-21T08:43:10.789667+0000","last_fullsized":"2024-04-21T08:43:10.789667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:42.802309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981225+0000","last_change":"2024-04-21T08:42:21.655802+0000","last_active":"2024-04-21T08:43:10.981225+0000","last_peered":"2024-04-21T08:43:10.981225+0000","last_clean":"2024-04-21T08:43:10.981225+0000","last_became_active":"2024-04-21T08:42:21.655379+0000","last_became_peered":"2024-04-21T08:42:21.655379+0000","last_unstale":"2024-04-21T08:43:10.981225+0000","last_undegraded":"2024-04-21T08:43:10.981225+0000","last_fullsized":"2024-04-21T08:43:10.981225+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:49.486057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984198+0000","last_change":"2024-04-21T08:42:16.642517+0000","last_active":"2024-04-21T08:43:10.984198+0000","last_peered":"2024-04-21T08:43:10.984198+0000","last_clean":"2024-04-21T08:43:10.984198+0000","last_became_active":"2024-04-21T08:42:16.641539+0000","last_became_peered":"2024-04-21T08:42:16.641539+0000","last_unstale":"2024-04-21T08:43:10.984198+0000","last_undegraded":"2024-04-21T08:43:10.984198+0000","last_fullsized":"2024-04-21T08:43:10.984198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:53.912199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790179+0000","last_change":"2024-04-21T08:42:17.550172+0000","last_active":"2024-04-21T08:43:10.790179+0000","last_peered":"2024-04-21T08:43:10.790179+0000","last_clean":"2024-04-21T08:43:10.790179+0000","last_became_active":"2024-04-21T08:42:17.549824+0000","last_became_peered":"2024-04-21T08:42:17.549824+0000","last_unstale":"2024-04-21T08:43:10.790179+0000","last_undegraded":"2024-04-21T08:43:10.790179+0000","last_fullsized":"2024-04-21T08:43:10.790179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:59.553645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983731+0000","last_change":"2024-04-21T08:42:21.970654+0000","last_active":"2024-04-21T08:43:10.983731+0000","last_peered":"2024-04-21T08:43:10.983731+0000","last_clean":"2024-04-21T08:43:10.983731+0000","last_became_active":"2024-04-21T08:42:21.970211+0000","last_became_peered":"2024-04-21T08:42:21.970211+0000","last_unstale":"2024-04-21T08:43:10.983731+0000","last_undegraded":"2024-04-21T08:43:10.983731+0000","last_fullsized":"2024-04-21T08:43:10.983731+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:24:51.406125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982736+0000","last_change":"2024-04-21T08:42:16.647985+0000","last_active":"2024-04-21T08:43:10.982736+0000","last_peered":"2024-04-21T08:43:10.982736+0000","last_clean":"2024-04-21T08:43:10.982736+0000","last_became_active":"2024-04-21T08:42:16.647577+0000","last_became_peered":"2024-04-21T08:42:16.647577+0000","last_unstale":"2024-04-21T08:43:10.982736+0000","last_undegraded":"2024-04-21T08:43:10.982736+0000","last_fullsized":"2024-04-21T08:43:10.982736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:00.731708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786532+0000","last_change":"2024-04-21T08:42:17.541757+0000","last_active":"2024-04-21T08:43:10.786532+0000","last_peered":"2024-04-21T08:43:10.786532+0000","last_clean":"2024-04-21T08:43:10.786532+0000","last_became_active":"2024-04-21T08:42:17.539280+0000","last_became_peered":"2024-04-21T08:42:17.539280+0000","last_unstale":"2024-04-21T08:43:10.786532+0000","last_undegraded":"2024-04-21T08:43:10.786532+0000","last_fullsized":"2024-04-21T08:43:10.786532+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:45:54.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786263+0000","last_change":"2024-04-21T08:42:21.664566+0000","last_active":"2024-04-21T08:43:10.786263+0000","last_peered":"2024-04-21T08:43:10.786263+0000","last_clean":"2024-04-21T08:43:10.786263+0000","last_became_active":"2024-04-21T08:42:21.664136+0000","last_became_peered":"2024-04-21T08:42:21.664136+0000","last_unstale":"2024-04-21T08:43:10.786263+0000","last_undegraded":"2024-04-21T08:43:10.786263+0000","last_fullsized":"2024-04-21T08:43:10.786263+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:19.856577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789368+0000","last_change":"2024-04-21T08:42:16.650664+0000","last_active":"2024-04-21T08:43:10.789368+0000","last_peered":"2024-04-21T08:43:10.789368+0000","last_clean":"2024-04-21T08:43:10.789368+0000","last_became_active":"2024-04-21T08:42:16.650339+0000","last_became_peered":"2024-04-21T08:42:16.650339+0000","last_unstale":"2024-04-21T08:43:10.789368+0000","last_undegraded":"2024-04-21T08:43:10.789368+0000","last_fullsized":"2024-04-21T08:43:10.789368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:59.565455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788742+0000","last_change":"2024-04-21T08:42:17.538142+0000","last_active":"2024-04-21T08:43:10.788742+0000","last_peered":"2024-04-21T08:43:10.788742+0000","last_clean":"2024-04-21T08:43:10.788742+0000","last_became_active":"2024-04-21T08:42:17.537793+0000","last_became_peered":"2024-04-21T08:42:17.537793+0000","last_unstale":"2024-04-21T08:43:10.788742+0000","last_undegraded":"2024-04-21T08:43:10.788742+0000","last_fullsized":"2024-04-21T08:43:10.788742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:11.059294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984088+0000","last_change":"2024-04-21T08:42:21.670642+0000","last_active":"2024-04-21T08:43:10.984088+0000","last_peered":"2024-04-21T08:43:10.984088+0000","last_clean":"2024-04-21T08:43:10.984088+0000","last_became_active":"2024-04-21T08:42:21.670128+0000","last_became_peered":"2024-04-21T08:42:21.670128+0000","last_unstale":"2024-04-21T08:43:10.984088+0000","last_undegraded":"2024-04-21T08:43:10.984088+0000","last_fullsized":"2024-04-21T08:43:10.984088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:51:28.604381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.792114+0000","last_change":"2024-04-21T08:43:10.792114+0000","last_active":"2024-04-21T08:43:10.792114+0000","last_peered":"2024-04-21T08:43:10.792114+0000","last_clean":"2024-04-21T08:43:10.792114+0000","last_became_active":"2024-04-21T08:43:10.791664+0000","last_became_peered":"2024-04-21T08:43:10.791664+0000","last_unstale":"2024-04-21T08:43:10.792114+0000","last_undegraded":"2024-04-21T08:43:10.792114+0000","last_fullsized":"2024-04-21T08:43:10.792114+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:07.317804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789375+0000","last_change":"2024-04-21T08:42:17.547326+0000","last_active":"2024-04-21T08:43:10.789375+0000","last_peered":"2024-04-21T08:43:10.789375+0000","last_clean":"2024-04-21T08:43:10.789375+0000","last_became_active":"2024-04-21T08:42:17.546920+0000","last_became_peered":"2024-04-21T08:42:17.546920+0000","last_unstale":"2024-04-21T08:43:10.789375+0000","last_undegraded":"2024-04-21T08:43:10.789375+0000","last_fullsized":"2024-04-21T08:43:10.789375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:43.350877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983874+0000","last_change":"2024-04-21T08:42:21.971147+0000","last_active":"2024-04-21T08:43:10.983874+0000","last_peered":"2024-04-21T08:43:10.983874+0000","last_clean":"2024-04-21T08:43:10.983874+0000","last_became_active":"2024-04-21T08:42:21.970001+0000","last_became_peered":"2024-04-21T08:42:21.970001+0000","last_unstale":"2024-04-21T08:43:10.983874+0000","last_undegraded":"2024-04-21T08:43:10.983874+0000","last_fullsized":"2024-04-21T08:43:10.983874+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:01:44.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786986+0000","last_change":"2024-04-21T08:42:16.643371+0000","last_active":"2024-04-21T08:43:10.786986+0000","last_peered":"2024-04-21T08:43:10.786986+0000","last_clean":"2024-04-21T08:43:10.786986+0000","last_became_active":"2024-04-21T08:42:16.642906+0000","last_became_peered":"2024-04-21T08:42:16.642906+0000","last_unstale":"2024-04-21T08:43:10.786986+0000","last_undegraded":"2024-04-21T08:43:10.786986+0000","last_fullsized":"2024-04-21T08:43:10.786986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:58.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789827+0000","last_change":"2024-04-21T08:42:17.545185+0000","last_active":"2024-04-21T08:43:10.789827+0000","last_peered":"2024-04-21T08:43:10.789827+0000","last_clean":"2024-04-21T08:43:10.789827+0000","last_became_active":"2024-04-21T08:42:17.544757+0000","last_became_peered":"2024-04-21T08:42:17.544757+0000","last_unstale":"2024-04-21T08:43:10.789827+0000","last_undegraded":"2024-04-21T08:43:10.789827+0000","last_fullsized":"2024-04-21T08:43:10.789827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:36:18.968041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982012+0000","last_change":"2024-04-21T08:42:21.662541+0000","last_active":"2024-04-21T08:43:10.982012+0000","last_peered":"2024-04-21T08:43:10.982012+0000","last_clean":"2024-04-21T08:43:10.982012+0000","last_became_active":"2024-04-21T08:42:21.662144+0000","last_became_peered":"2024-04-21T08:42:21.662144+0000","last_unstale":"2024-04-21T08:43:10.982012+0000","last_undegraded":"2024-04-21T08:43:10.982012+0000","last_fullsized":"2024-04-21T08:43:10.982012+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:04.347722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981989+0000","last_change":"2024-04-21T08:42:16.638601+0000","last_active":"2024-04-21T08:43:10.981989+0000","last_peered":"2024-04-21T08:43:10.981989+0000","last_clean":"2024-04-21T08:43:10.981989+0000","last_became_active":"2024-04-21T08:42:16.638211+0000","last_became_peered":"2024-04-21T08:42:16.638211+0000","last_unstale":"2024-04-21T08:43:10.981989+0000","last_undegraded":"2024-04-21T08:43:10.981989+0000","last_fullsized":"2024-04-21T08:43:10.981989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:02.633086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790435+0000","last_change":"2024-04-21T08:42:17.545495+0000","last_active":"2024-04-21T08:43:10.790435+0000","last_peered":"2024-04-21T08:43:10.790435+0000","last_clean":"2024-04-21T08:43:10.790435+0000","last_became_active":"2024-04-21T08:42:17.545159+0000","last_became_peered":"2024-04-21T08:42:17.545159+0000","last_unstale":"2024-04-21T08:43:10.790435+0000","last_undegraded":"2024-04-21T08:43:10.790435+0000","last_fullsized":"2024-04-21T08:43:10.790435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:27.150277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790207+0000","last_change":"2024-04-21T08:42:21.656890+0000","last_active":"2024-04-21T08:43:10.790207+0000","last_peered":"2024-04-21T08:43:10.790207+0000","last_clean":"2024-04-21T08:43:10.790207+0000","last_became_active":"2024-04-21T08:42:21.656475+0000","last_became_peered":"2024-04-21T08:42:21.656475+0000","last_unstale":"2024-04-21T08:43:10.790207+0000","last_undegraded":"2024-04-21T08:43:10.790207+0000","last_fullsized":"2024-04-21T08:43:10.790207+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:46.138591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791054+0000","last_change":"2024-04-21T08:42:16.638244+0000","last_active":"2024-04-21T08:43:10.791054+0000","last_peered":"2024-04-21T08:43:10.791054+0000","last_clean":"2024-04-21T08:43:10.791054+0000","last_became_active":"2024-04-21T08:42:16.637883+0000","last_became_peered":"2024-04-21T08:42:16.637883+0000","last_unstale":"2024-04-21T08:43:10.791054+0000","last_undegraded":"2024-04-21T08:43:10.791054+0000","last_fullsized":"2024-04-21T08:43:10.791054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:20.888301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789878+0000","last_change":"2024-04-21T08:42:17.548032+0000","last_active":"2024-04-21T08:43:10.789878+0000","last_peered":"2024-04-21T08:43:10.789878+0000","last_clean":"2024-04-21T08:43:10.789878+0000","last_became_active":"2024-04-21T08:42:17.547660+0000","last_became_peered":"2024-04-21T08:42:17.547660+0000","last_unstale":"2024-04-21T08:43:10.789878+0000","last_undegraded":"2024-04-21T08:43:10.789878+0000","last_fullsized":"2024-04-21T08:43:10.789878+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:14.710759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790108+0000","last_change":"2024-04-21T08:42:21.659286+0000","last_active":"2024-04-21T08:43:10.790108+0000","last_peered":"2024-04-21T08:43:10.790108+0000","last_clean":"2024-04-21T08:43:10.790108+0000","last_became_active":"2024-04-21T08:42:21.658844+0000","last_became_peered":"2024-04-21T08:42:21.658844+0000","last_unstale":"2024-04-21T08:43:10.790108+0000","last_undegraded":"2024-04-21T08:43:10.790108+0000","last_fullsized":"2024-04-21T08:43:10.790108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:28.742646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791168+0000","last_change":"2024-04-21T08:42:16.643354+0000","last_active":"2024-04-21T08:43:10.791168+0000","last_peered":"2024-04-21T08:43:10.791168+0000","last_clean":"2024-04-21T08:43:10.791168+0000","last_became_active":"2024-04-21T08:42:16.642967+0000","last_became_peered":"2024-04-21T08:42:16.642967+0000","last_unstale":"2024-04-21T08:43:10.791168+0000","last_undegraded":"2024-04-21T08:43:10.791168+0000","last_fullsized":"2024-04-21T08:43:10.791168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:06:03.495181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790284+0000","last_change":"2024-04-21T08:42:17.539551+0000","last_active":"2024-04-21T08:43:10.790284+0000","last_peered":"2024-04-21T08:43:10.790284+0000","last_clean":"2024-04-21T08:43:10.790284+0000","last_became_active":"2024-04-21T08:42:17.539202+0000","last_became_peered":"2024-04-21T08:42:17.539202+0000","last_unstale":"2024-04-21T08:43:10.790284+0000","last_undegraded":"2024-04-21T08:43:10.790284+0000","last_fullsized":"2024-04-21T08:43:10.790284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:37.109421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788958+0000","last_change":"2024-04-21T08:42:17.544582+0000","last_active":"2024-04-21T08:43:10.788958+0000","last_peered":"2024-04-21T08:43:10.788958+0000","last_clean":"2024-04-21T08:43:10.788958+0000","last_became_active":"2024-04-21T08:42:17.543800+0000","last_became_peered":"2024-04-21T08:42:17.543800+0000","last_unstale":"2024-04-21T08:43:10.788958+0000","last_undegraded":"2024-04-21T08:43:10.788958+0000","last_fullsized":"2024-04-21T08:43:10.788958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:44:04.107207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981354+0000","last_change":"2024-04-21T08:42:17.543788+0000","last_active":"2024-04-21T08:43:10.981354+0000","last_peered":"2024-04-21T08:43:10.981354+0000","last_clean":"2024-04-21T08:43:10.981354+0000","last_became_active":"2024-04-21T08:42:17.540939+0000","last_became_peered":"2024-04-21T08:42:17.540939+0000","last_unstale":"2024-04-21T08:43:10.981354+0000","last_undegraded":"2024-04-21T08:43:10.981354+0000","last_fullsized":"2024-04-21T08:43:10.981354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:59.275098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981292+0000","last_change":"2024-04-21T08:42:17.540605+0000","last_active":"2024-04-21T08:43:10.981292+0000","last_peered":"2024-04-21T08:43:10.981292+0000","last_clean":"2024-04-21T08:43:10.981292+0000","last_became_active":"2024-04-21T08:42:17.540073+0000","last_became_peered":"2024-04-21T08:42:17.540073+0000","last_unstale":"2024-04-21T08:43:10.981292+0000","last_undegraded":"2024-04-21T08:43:10.981292+0000","last_fullsized":"2024-04-21T08:43:10.981292+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:02:29.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790444+0000","last_change":"2024-04-21T08:42:17.551608+0000","last_active":"2024-04-21T08:43:10.790444+0000","last_peered":"2024-04-21T08:43:10.790444+0000","last_clean":"2024-04-21T08:43:10.790444+0000","last_became_active":"2024-04-21T08:42:17.551117+0000","last_became_peered":"2024-04-21T08:42:17.551117+0000","last_unstale":"2024-04-21T08:43:10.790444+0000","last_undegraded":"2024-04-21T08:43:10.790444+0000","last_fullsized":"2024-04-21T08:43:10.790444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:18.391928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790597+0000","last_change":"2024-04-21T08:42:17.536509+0000","last_active":"2024-04-21T08:43:10.790597+0000","last_peered":"2024-04-21T08:43:10.790597+0000","last_clean":"2024-04-21T08:43:10.790597+0000","last_became_active":"2024-04-21T08:42:17.536146+0000","last_became_peered":"2024-04-21T08:42:17.536146+0000","last_unstale":"2024-04-21T08:43:10.790597+0000","last_undegraded":"2024-04-21T08:43:10.790597+0000","last_fullsized":"2024-04-21T08:43:10.790597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:29.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788710+0000","last_change":"2024-04-21T08:42:17.543786+0000","last_active":"2024-04-21T08:43:10.788710+0000","last_peered":"2024-04-21T08:43:10.788710+0000","last_clean":"2024-04-21T08:43:10.788710+0000","last_became_active":"2024-04-21T08:42:17.543435+0000","last_became_peered":"2024-04-21T08:42:17.543435+0000","last_unstale":"2024-04-21T08:43:10.788710+0000","last_undegraded":"2024-04-21T08:43:10.788710+0000","last_fullsized":"2024-04-21T08:43:10.788710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981773+0000","last_change":"2024-04-21T08:42:17.541611+0000","last_active":"2024-04-21T08:43:10.981773+0000","last_peered":"2024-04-21T08:43:10.981773+0000","last_clean":"2024-04-21T08:43:10.981773+0000","last_became_active":"2024-04-21T08:42:17.541259+0000","last_became_peered":"2024-04-21T08:42:17.541259+0000","last_unstale":"2024-04-21T08:43:10.981773+0000","last_undegraded":"2024-04-21T08:43:10.981773+0000","last_fullsized":"2024-04-21T08:43:10.981773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:20.900629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789837+0000","last_change":"2024-04-21T08:42:17.541964+0000","last_active":"2024-04-21T08:43:10.789837+0000","last_peered":"2024-04-21T08:43:10.789837+0000","last_clean":"2024-04-21T08:43:10.789837+0000","last_became_active":"2024-04-21T08:42:17.541555+0000","last_became_peered":"2024-04-21T08:42:17.541555+0000","last_unstale":"2024-04-21T08:43:10.789837+0000","last_undegraded":"2024-04-21T08:43:10.789837+0000","last_fullsized":"2024-04-21T08:43:10.789837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:32:47.421452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790774+0000","last_change":"2024-04-21T08:42:17.545526+0000","last_active":"2024-04-21T08:43:10.790774+0000","last_peered":"2024-04-21T08:43:10.790774+0000","last_clean":"2024-04-21T08:43:10.790774+0000","last_became_active":"2024-04-21T08:42:17.545135+0000","last_became_peered":"2024-04-21T08:42:17.545135+0000","last_unstale":"2024-04-21T08:43:10.790774+0000","last_undegraded":"2024-04-21T08:43:10.790774+0000","last_fullsized":"2024-04-21T08:43:10.790774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:37:58.038459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789502+0000","last_change":"2024-04-21T08:42:17.547063+0000","last_active":"2024-04-21T08:43:10.789502+0000","last_peered":"2024-04-21T08:43:10.789502+0000","last_clean":"2024-04-21T08:43:10.789502+0000","last_became_active":"2024-04-21T08:42:17.546689+0000","last_became_peered":"2024-04-21T08:42:17.546689+0000","last_unstale":"2024-04-21T08:43:10.789502+0000","last_undegraded":"2024-04-21T08:43:10.789502+0000","last_fullsized":"2024-04-21T08:43:10.789502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:26.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982688+0000","last_change":"2024-04-21T08:42:17.542545+0000","last_active":"2024-04-21T08:43:10.982688+0000","last_peered":"2024-04-21T08:43:10.982688+0000","last_clean":"2024-04-21T08:43:10.982688+0000","last_became_active":"2024-04-21T08:42:17.542147+0000","last_became_peered":"2024-04-21T08:42:17.542147+0000","last_unstale":"2024-04-21T08:43:10.982688+0000","last_undegraded":"2024-04-21T08:43:10.982688+0000","last_fullsized":"2024-04-21T08:43:10.982688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:15.064110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790177+0000","last_change":"2024-04-21T08:42:17.541262+0000","last_active":"2024-04-21T08:43:10.790177+0000","last_peered":"2024-04-21T08:43:10.790177+0000","last_clean":"2024-04-21T08:43:10.790177+0000","last_became_active":"2024-04-21T08:42:17.540817+0000","last_became_peered":"2024-04-21T08:42:17.540817+0000","last_unstale":"2024-04-21T08:43:10.790177+0000","last_undegraded":"2024-04-21T08:43:10.790177+0000","last_fullsized":"2024-04-21T08:43:10.790177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:22:59.597680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788435+0000","last_change":"2024-04-21T08:42:17.551109+0000","last_active":"2024-04-21T08:43:10.788435+0000","last_peered":"2024-04-21T08:43:10.788435+0000","last_clean":"2024-04-21T08:43:10.788435+0000","last_became_active":"2024-04-21T08:42:17.550104+0000","last_became_peered":"2024-04-21T08:42:17.550104+0000","last_unstale":"2024-04-21T08:43:10.788435+0000","last_undegraded":"2024-04-21T08:43:10.788435+0000","last_fullsized":"2024-04-21T08:43:10.788435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:22:19.361752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281211,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64154,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":153,"num_read_kb":141,"num_write":79,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":192462,"data_compressed":4896,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":503292,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612646,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36444,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708708,"statfs":{"total":95995035648,"available":95957716992,"internally_reserved":0,"allocated":1126400,"data_stored":1153114,"data_compressed":58127,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776172,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":507970,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037810,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":503896,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299446,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":1098338,"data_compressed":56495,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":499196,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1064960,"data_stored":562552,"data_compressed":22361,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":999424,"data_stored":499286,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":562586,"data_compressed":22361,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":499300,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1003520,"data_stored":503314,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":1093842,"data_compressed":56495,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63390,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63550,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:43:56.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T08:43:57.029 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:57.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:56 smithi016 ceph-mon[31259]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:57.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:56 smithi016 ceph-mon[31259]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:43:57.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:56 smithi077 ceph-mon[31615]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:57.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:56 smithi077 ceph-mon[31615]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:43:57.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:56 smithi007 ceph-mon[28477]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:43:57.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:56 smithi007 ceph-mon[28477]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:43:58.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.681+0000 7f2f5384c700 1 -- 172.21.15.7:0/803670191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f4c06bbb0 msgr2=0x7f2f4c10f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:58.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.681+0000 7f2f5384c700 1 --2- 172.21.15.7:0/803670191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f4c06bbb0 0x7f2f4c10f930 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009990 tx=0x7f2f44009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:58.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.681+0000 7f2f5384c700 1 -- 172.21.15.7:0/803670191 shutdown_connections 2024-04-21T08:43:58.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.681+0000 7f2f5384c700 1 --2- 172.21.15.7:0/803670191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f4c108a60 0x7f2f4c108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:58.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.681+0000 7f2f5384c700 1 --2- 172.21.15.7:0/803670191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f4c06bbb0 0x7f2f4c10f930 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:58.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.681+0000 7f2f5384c700 1 --2- 172.21.15.7:0/803670191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4c06b1f0 0x7f2f4c06b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:58.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.681+0000 7f2f5384c700 1 -- 172.21.15.7:0/803670191 >> 172.21.15.7:0/803670191 conn(0x7f2f4c0789e0 msgr2=0x7f2f4c078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:58.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.681+0000 7f2f5384c700 1 -- 172.21.15.7:0/803670191 shutdown_connections 2024-04-21T08:43:58.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.681+0000 7f2f5384c700 1 -- 172.21.15.7:0/803670191 wait complete. 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f5384c700 1 Processor -- start 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f5384c700 1 -- start start 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f5384c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f4c06b1f0 0x7f2f4c196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f5384c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f4c06bbb0 0x7f2f4c1974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f5384c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4c108a60 0x7f2f4c19b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f5384c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4c110a90 con 0x7f2f4c06b1f0 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f5384c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4c110910 con 0x7f2f4c108a60 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f5384c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4c110c10 con 0x7f2f4c06bbb0 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f515e8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f4c06b1f0 0x7f2f4c196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f515e8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f4c06b1f0 0x7f2f4c196fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36284/0 (socket says 172.21.15.7:36284) 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f515e8700 1 -- 172.21.15.7:0/2261398369 learned_addr learned my addr 172.21.15.7:0/2261398369 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f50de7700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f4c06bbb0 0x7f2f4c1974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.682+0000 7f2f51de9700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4c108a60 0x7f2f4c19b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:58.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.683+0000 7f2f515e8700 1 -- 172.21.15.7:0/2261398369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f4c06bbb0 msgr2=0x7f2f4c1974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:58.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.683+0000 7f2f515e8700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f4c06bbb0 0x7f2f4c1974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:58.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.683+0000 7f2f515e8700 1 -- 172.21.15.7:0/2261398369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4c108a60 msgr2=0x7f2f4c19b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:58.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.683+0000 7f2f515e8700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4c108a60 0x7f2f4c19b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:58.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.683+0000 7f2f515e8700 1 -- 172.21.15.7:0/2261398369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4c19bf60 con 0x7f2f4c06b1f0 2024-04-21T08:43:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.683+0000 7f2f515e8700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f4c06b1f0 0x7f2f4c196fa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2f48007f10 tx=0x7f2f4800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.683+0000 7f2f427fc700 1 -- 172.21.15.7:0/2261398369 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f48004bb0 con 0x7f2f4c06b1f0 2024-04-21T08:43:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.683+0000 7f2f427fc700 1 -- 172.21.15.7:0/2261398369 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f48004d50 con 0x7f2f4c06b1f0 2024-04-21T08:43:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.683+0000 7f2f5384c700 1 -- 172.21.15.7:0/2261398369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4c19c250 con 0x7f2f4c06b1f0 2024-04-21T08:43:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.683+0000 7f2f427fc700 1 -- 172.21.15.7:0/2261398369 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f480059b0 con 0x7f2f4c06b1f0 2024-04-21T08:43:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.683+0000 7f2f5384c700 1 -- 172.21.15.7:0/2261398369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f4c1a3ad0 con 0x7f2f4c06b1f0 2024-04-21T08:43:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.685+0000 7f2f5384c700 1 -- 172.21.15.7:0/2261398369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f4c069730 con 0x7f2f4c06b1f0 2024-04-21T08:43:58.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.686+0000 7f2f427fc700 1 -- 172.21.15.7:0/2261398369 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f2f4801a020 con 0x7f2f4c06b1f0 2024-04-21T08:43:58.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.686+0000 7f2f427fc700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2f380a3e20 0x7f2f380a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:58.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.687+0000 7f2f50de7700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2f380a3e20 0x7f2f380a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:58.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.687+0000 7f2f427fc700 1 -- 172.21.15.7:0/2261398369 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f2f480cfeb0 con 0x7f2f4c06b1f0 2024-04-21T08:43:58.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.687+0000 7f2f50de7700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2f380a3e20 0x7f2f380a62e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c078fc0 tx=0x7f2f3c00b700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:58.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.691+0000 7f2f427fc700 1 -- 172.21.15.7:0/2261398369 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f48010070 con 0x7f2f4c06b1f0 2024-04-21T08:43:58.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.895+0000 7f2f5384c700 1 -- 172.21.15.7:0/2261398369 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2f4c1982b0 con 0x7f2f380a3e20 2024-04-21T08:43:58.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.904+0000 7f2f427fc700 1 -- 172.21.15.7:0/2261398369 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340591 (secure 0 0 0) 0x7f2f3c071280 con 0x7f2f380a3e20 2024-04-21T08:43:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:43:58.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.907+0000 7f2f5384c700 1 -- 172.21.15.7:0/2261398369 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2f380a3e20 msgr2=0x7f2f380a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:58.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.907+0000 7f2f5384c700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2f380a3e20 0x7f2f380a62e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c078fc0 tx=0x7f2f3c00b700 comp rx=0 tx=0).stop 2024-04-21T08:43:58.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.907+0000 7f2f5384c700 1 -- 172.21.15.7:0/2261398369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f4c06b1f0 msgr2=0x7f2f4c196fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:58.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.907+0000 7f2f5384c700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f4c06b1f0 0x7f2f4c196fa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2f48007f10 tx=0x7f2f4800d3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:58.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.907+0000 7f2f5384c700 1 -- 172.21.15.7:0/2261398369 shutdown_connections 2024-04-21T08:43:58.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.907+0000 7f2f5384c700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2f380a3e20 0x7f2f380a62e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:58.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.908+0000 7f2f5384c700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f4c108a60 0x7f2f4c19b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:58.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.908+0000 7f2f5384c700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f4c06b1f0 0x7f2f4c196fa0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:58.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.908+0000 7f2f5384c700 1 --2- 172.21.15.7:0/2261398369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f4c06bbb0 0x7f2f4c1974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:58.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.908+0000 7f2f5384c700 1 -- 172.21.15.7:0/2261398369 >> 172.21.15.7:0/2261398369 conn(0x7f2f4c0789e0 msgr2=0x7f2f4c0ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:58.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.908+0000 7f2f5384c700 1 -- 172.21.15.7:0/2261398369 shutdown_connections 2024-04-21T08:43:58.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:43:58.908+0000 7f2f5384c700 1 -- 172.21.15.7:0/2261398369 wait complete. 2024-04-21T08:43:58.912 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-21T08:43:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:58 smithi016 ceph-mon[31259]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:59.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:58 smithi077 ceph-mon[31615]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:59.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:58 smithi007 ceph-mon[28477]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:43:59.742 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-04-21T08:43:57.654355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654756,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":299,"num_read_kb":265,"num_write":262,"num_write_kb":3030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":402944,"kb_used_data":12104,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124538880,"statfs":{"total":1151940427776,"available":1151527813120,"internally_reserved":0,"allocated":12394496,"data_stored":7986686,"data_compressed":360942,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003750"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788844+0000","last_change":"2024-04-21T08:42:17.545701+0000","last_active":"2024-04-21T08:43:10.788844+0000","last_peered":"2024-04-21T08:43:10.788844+0000","last_clean":"2024-04-21T08:43:10.788844+0000","last_became_active":"2024-04-21T08:42:17.545157+0000","last_became_peered":"2024-04-21T08:42:17.545157+0000","last_unstale":"2024-04-21T08:43:10.788844+0000","last_undegraded":"2024-04-21T08:43:10.788844+0000","last_fullsized":"2024-04-21T08:43:10.788844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:05:41.952879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790294+0000","last_change":"2024-04-21T08:42:17.541153+0000","last_active":"2024-04-21T08:43:10.790294+0000","last_peered":"2024-04-21T08:43:10.790294+0000","last_clean":"2024-04-21T08:43:10.790294+0000","last_became_active":"2024-04-21T08:42:17.540771+0000","last_became_peered":"2024-04-21T08:42:17.540771+0000","last_unstale":"2024-04-21T08:43:10.790294+0000","last_undegraded":"2024-04-21T08:43:10.790294+0000","last_fullsized":"2024-04-21T08:43:10.790294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:35:06.077367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790689+0000","last_change":"2024-04-21T08:42:17.545525+0000","last_active":"2024-04-21T08:43:10.790689+0000","last_peered":"2024-04-21T08:43:10.790689+0000","last_clean":"2024-04-21T08:43:10.790689+0000","last_became_active":"2024-04-21T08:42:17.545168+0000","last_became_peered":"2024-04-21T08:42:17.545168+0000","last_unstale":"2024-04-21T08:43:10.790689+0000","last_undegraded":"2024-04-21T08:43:10.790689+0000","last_fullsized":"2024-04-21T08:43:10.790689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:51.080385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984411+0000","last_change":"2024-04-21T08:42:17.543917+0000","last_active":"2024-04-21T08:43:10.984411+0000","last_peered":"2024-04-21T08:43:10.984411+0000","last_clean":"2024-04-21T08:43:10.984411+0000","last_became_active":"2024-04-21T08:42:17.543513+0000","last_became_peered":"2024-04-21T08:42:17.543513+0000","last_unstale":"2024-04-21T08:43:10.984411+0000","last_undegraded":"2024-04-21T08:43:10.984411+0000","last_fullsized":"2024-04-21T08:43:10.984411+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:23.058600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786620+0000","last_change":"2024-04-21T08:42:17.549737+0000","last_active":"2024-04-21T08:43:10.786620+0000","last_peered":"2024-04-21T08:43:10.786620+0000","last_clean":"2024-04-21T08:43:10.786620+0000","last_became_active":"2024-04-21T08:42:17.549396+0000","last_became_peered":"2024-04-21T08:42:17.549396+0000","last_unstale":"2024-04-21T08:43:10.786620+0000","last_undegraded":"2024-04-21T08:43:10.786620+0000","last_fullsized":"2024-04-21T08:43:10.786620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:01:51.879960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.792321+0000","last_change":"2024-04-21T08:43:10.792321+0000","last_active":"2024-04-21T08:43:10.792321+0000","last_peered":"2024-04-21T08:43:10.792321+0000","last_clean":"2024-04-21T08:43:10.792321+0000","last_became_active":"2024-04-21T08:43:10.791979+0000","last_became_peered":"2024-04-21T08:43:10.791979+0000","last_unstale":"2024-04-21T08:43:10.792321+0000","last_undegraded":"2024-04-21T08:43:10.792321+0000","last_fullsized":"2024-04-21T08:43:10.792321+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:11:16.056778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789571+0000","last_change":"2024-04-21T08:42:17.541172+0000","last_active":"2024-04-21T08:43:10.789571+0000","last_peered":"2024-04-21T08:43:10.789571+0000","last_clean":"2024-04-21T08:43:10.789571+0000","last_became_active":"2024-04-21T08:42:17.540829+0000","last_became_peered":"2024-04-21T08:42:17.540829+0000","last_unstale":"2024-04-21T08:43:10.789571+0000","last_undegraded":"2024-04-21T08:43:10.789571+0000","last_fullsized":"2024-04-21T08:43:10.789571+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:16:12.633980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789003+0000","last_change":"2024-04-21T08:42:17.549744+0000","last_active":"2024-04-21T08:43:10.789003+0000","last_peered":"2024-04-21T08:43:10.789003+0000","last_clean":"2024-04-21T08:43:10.789003+0000","last_became_active":"2024-04-21T08:42:17.549378+0000","last_became_peered":"2024-04-21T08:42:17.549378+0000","last_unstale":"2024-04-21T08:43:10.789003+0000","last_undegraded":"2024-04-21T08:43:10.789003+0000","last_fullsized":"2024-04-21T08:43:10.789003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:03:07.702143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983286+0000","last_change":"2024-04-21T08:42:17.548675+0000","last_active":"2024-04-21T08:43:10.983286+0000","last_peered":"2024-04-21T08:43:10.983286+0000","last_clean":"2024-04-21T08:43:10.983286+0000","last_became_active":"2024-04-21T08:42:17.547809+0000","last_became_peered":"2024-04-21T08:42:17.547809+0000","last_unstale":"2024-04-21T08:43:10.983286+0000","last_undegraded":"2024-04-21T08:43:10.983286+0000","last_fullsized":"2024-04-21T08:43:10.983286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:23.343473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788107+0000","last_change":"2024-04-21T08:42:17.551012+0000","last_active":"2024-04-21T08:43:10.788107+0000","last_peered":"2024-04-21T08:43:10.788107+0000","last_clean":"2024-04-21T08:43:10.788107+0000","last_became_active":"2024-04-21T08:42:17.550627+0000","last_became_peered":"2024-04-21T08:42:17.550627+0000","last_unstale":"2024-04-21T08:43:10.788107+0000","last_undegraded":"2024-04-21T08:43:10.788107+0000","last_fullsized":"2024-04-21T08:43:10.788107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:57.501588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790388+0000","last_change":"2024-04-21T08:42:17.546192+0000","last_active":"2024-04-21T08:43:10.790388+0000","last_peered":"2024-04-21T08:43:10.790388+0000","last_clean":"2024-04-21T08:43:10.790388+0000","last_became_active":"2024-04-21T08:42:17.545808+0000","last_became_peered":"2024-04-21T08:42:17.545808+0000","last_unstale":"2024-04-21T08:43:10.790388+0000","last_undegraded":"2024-04-21T08:43:10.790388+0000","last_fullsized":"2024-04-21T08:43:10.790388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:54.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984291+0000","last_change":"2024-04-21T08:42:17.553351+0000","last_active":"2024-04-21T08:43:10.984291+0000","last_peered":"2024-04-21T08:43:10.984291+0000","last_clean":"2024-04-21T08:43:10.984291+0000","last_became_active":"2024-04-21T08:42:17.550507+0000","last_became_peered":"2024-04-21T08:42:17.550507+0000","last_unstale":"2024-04-21T08:43:10.984291+0000","last_undegraded":"2024-04-21T08:43:10.984291+0000","last_fullsized":"2024-04-21T08:43:10.984291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:25.561375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790352+0000","last_change":"2024-04-21T08:42:17.540533+0000","last_active":"2024-04-21T08:43:10.790352+0000","last_peered":"2024-04-21T08:43:10.790352+0000","last_clean":"2024-04-21T08:43:10.790352+0000","last_became_active":"2024-04-21T08:42:17.539940+0000","last_became_peered":"2024-04-21T08:42:17.539940+0000","last_unstale":"2024-04-21T08:43:10.790352+0000","last_undegraded":"2024-04-21T08:43:10.790352+0000","last_fullsized":"2024-04-21T08:43:10.790352+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:45:19.102899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789978+0000","last_change":"2024-04-21T08:42:17.537794+0000","last_active":"2024-04-21T08:43:10.789978+0000","last_peered":"2024-04-21T08:43:10.789978+0000","last_clean":"2024-04-21T08:43:10.789978+0000","last_became_active":"2024-04-21T08:42:17.537376+0000","last_became_peered":"2024-04-21T08:42:17.537376+0000","last_unstale":"2024-04-21T08:43:10.789978+0000","last_undegraded":"2024-04-21T08:43:10.789978+0000","last_fullsized":"2024-04-21T08:43:10.789978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:57.430036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790075+0000","last_change":"2024-04-21T08:42:17.550574+0000","last_active":"2024-04-21T08:43:10.790075+0000","last_peered":"2024-04-21T08:43:10.790075+0000","last_clean":"2024-04-21T08:43:10.790075+0000","last_became_active":"2024-04-21T08:42:17.550205+0000","last_became_peered":"2024-04-21T08:42:17.550205+0000","last_unstale":"2024-04-21T08:43:10.790075+0000","last_undegraded":"2024-04-21T08:43:10.790075+0000","last_fullsized":"2024-04-21T08:43:10.790075+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:41:08.226477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982659+0000","last_change":"2024-04-21T08:42:17.542239+0000","last_active":"2024-04-21T08:43:10.982659+0000","last_peered":"2024-04-21T08:43:10.982659+0000","last_clean":"2024-04-21T08:43:10.982659+0000","last_became_active":"2024-04-21T08:42:17.541848+0000","last_became_peered":"2024-04-21T08:42:17.541848+0000","last_unstale":"2024-04-21T08:43:10.982659+0000","last_undegraded":"2024-04-21T08:43:10.982659+0000","last_fullsized":"2024-04-21T08:43:10.982659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:47.717451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788708+0000","last_change":"2024-04-21T08:42:17.553200+0000","last_active":"2024-04-21T08:43:10.788708+0000","last_peered":"2024-04-21T08:43:10.788708+0000","last_clean":"2024-04-21T08:43:10.788708+0000","last_became_active":"2024-04-21T08:42:17.552913+0000","last_became_peered":"2024-04-21T08:42:17.552913+0000","last_unstale":"2024-04-21T08:43:10.788708+0000","last_undegraded":"2024-04-21T08:43:10.788708+0000","last_fullsized":"2024-04-21T08:43:10.788708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:39:04.186928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786341+0000","last_change":"2024-04-21T08:42:17.534881+0000","last_active":"2024-04-21T08:43:10.786341+0000","last_peered":"2024-04-21T08:43:10.786341+0000","last_clean":"2024-04-21T08:43:10.786341+0000","last_became_active":"2024-04-21T08:42:17.534496+0000","last_became_peered":"2024-04-21T08:42:17.534496+0000","last_unstale":"2024-04-21T08:43:10.786341+0000","last_undegraded":"2024-04-21T08:43:10.786341+0000","last_fullsized":"2024-04-21T08:43:10.786341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:50:19.409391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981907+0000","last_change":"2024-04-21T08:42:17.540480+0000","last_active":"2024-04-21T08:43:10.981907+0000","last_peered":"2024-04-21T08:43:10.981907+0000","last_clean":"2024-04-21T08:43:10.981907+0000","last_became_active":"2024-04-21T08:42:17.539742+0000","last_became_peered":"2024-04-21T08:42:17.539742+0000","last_unstale":"2024-04-21T08:43:10.981907+0000","last_undegraded":"2024-04-21T08:43:10.981907+0000","last_fullsized":"2024-04-21T08:43:10.981907+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:00.238976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981405+0000","last_change":"2024-04-21T08:42:21.661514+0000","last_active":"2024-04-21T08:43:10.981405+0000","last_peered":"2024-04-21T08:43:10.981405+0000","last_clean":"2024-04-21T08:43:10.981405+0000","last_became_active":"2024-04-21T08:42:21.661095+0000","last_became_peered":"2024-04-21T08:42:21.661095+0000","last_unstale":"2024-04-21T08:43:10.981405+0000","last_undegraded":"2024-04-21T08:43:10.981405+0000","last_fullsized":"2024-04-21T08:43:10.981405+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:27:28.933446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786350+0000","last_change":"2024-04-21T08:42:16.639888+0000","last_active":"2024-04-21T08:43:10.786350+0000","last_peered":"2024-04-21T08:43:10.786350+0000","last_clean":"2024-04-21T08:43:10.786350+0000","last_became_active":"2024-04-21T08:42:16.639403+0000","last_became_peered":"2024-04-21T08:42:16.639403+0000","last_unstale":"2024-04-21T08:43:10.786350+0000","last_undegraded":"2024-04-21T08:43:10.786350+0000","last_fullsized":"2024-04-21T08:43:10.786350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:08:17.164924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983368+0000","last_change":"2024-04-21T08:43:10.983368+0000","last_active":"2024-04-21T08:43:10.983368+0000","last_peered":"2024-04-21T08:43:10.983368+0000","last_clean":"2024-04-21T08:43:10.983368+0000","last_became_active":"2024-04-21T08:43:10.982956+0000","last_became_peered":"2024-04-21T08:43:10.982956+0000","last_unstale":"2024-04-21T08:43:10.983368+0000","last_undegraded":"2024-04-21T08:43:10.983368+0000","last_fullsized":"2024-04-21T08:43:10.983368+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:12:44.919996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982417+0000","last_change":"2024-04-21T08:42:21.673197+0000","last_active":"2024-04-21T08:43:10.982417+0000","last_peered":"2024-04-21T08:43:10.982417+0000","last_clean":"2024-04-21T08:43:10.982417+0000","last_became_active":"2024-04-21T08:42:21.672793+0000","last_became_peered":"2024-04-21T08:42:21.672793+0000","last_unstale":"2024-04-21T08:43:10.982417+0000","last_undegraded":"2024-04-21T08:43:10.982417+0000","last_fullsized":"2024-04-21T08:43:10.982417+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:44:29.788459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790068+0000","last_change":"2024-04-21T08:42:16.648577+0000","last_active":"2024-04-21T08:43:10.790068+0000","last_peered":"2024-04-21T08:43:10.790068+0000","last_clean":"2024-04-21T08:43:10.790068+0000","last_became_active":"2024-04-21T08:42:16.646716+0000","last_became_peered":"2024-04-21T08:42:16.646716+0000","last_unstale":"2024-04-21T08:43:10.790068+0000","last_undegraded":"2024-04-21T08:43:10.790068+0000","last_fullsized":"2024-04-21T08:43:10.790068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:10.327078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790487+0000","last_change":"2024-04-21T08:42:17.545430+0000","last_active":"2024-04-21T08:43:10.790487+0000","last_peered":"2024-04-21T08:43:10.790487+0000","last_clean":"2024-04-21T08:43:10.790487+0000","last_became_active":"2024-04-21T08:42:17.545057+0000","last_became_peered":"2024-04-21T08:42:17.545057+0000","last_unstale":"2024-04-21T08:43:10.790487+0000","last_undegraded":"2024-04-21T08:43:10.790487+0000","last_fullsized":"2024-04-21T08:43:10.790487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:44:59.030830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982777+0000","last_change":"2024-04-21T08:42:21.673196+0000","last_active":"2024-04-21T08:43:10.982777+0000","last_peered":"2024-04-21T08:43:10.982777+0000","last_clean":"2024-04-21T08:43:10.982777+0000","last_became_active":"2024-04-21T08:42:21.672808+0000","last_became_peered":"2024-04-21T08:42:21.672808+0000","last_unstale":"2024-04-21T08:43:10.982777+0000","last_undegraded":"2024-04-21T08:43:10.982777+0000","last_fullsized":"2024-04-21T08:43:10.982777+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:39.482353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787192+0000","last_change":"2024-04-21T08:42:16.641611+0000","last_active":"2024-04-21T08:43:10.787192+0000","last_peered":"2024-04-21T08:43:10.787192+0000","last_clean":"2024-04-21T08:43:10.787192+0000","last_became_active":"2024-04-21T08:42:16.641221+0000","last_became_peered":"2024-04-21T08:42:16.641221+0000","last_unstale":"2024-04-21T08:43:10.787192+0000","last_undegraded":"2024-04-21T08:43:10.787192+0000","last_fullsized":"2024-04-21T08:43:10.787192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:45.496532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982089+0000","last_change":"2024-04-21T08:42:17.532986+0000","last_active":"2024-04-21T08:43:10.982089+0000","last_peered":"2024-04-21T08:43:10.982089+0000","last_clean":"2024-04-21T08:43:10.982089+0000","last_became_active":"2024-04-21T08:42:17.532583+0000","last_became_peered":"2024-04-21T08:42:17.532583+0000","last_unstale":"2024-04-21T08:43:10.982089+0000","last_undegraded":"2024-04-21T08:43:10.982089+0000","last_fullsized":"2024-04-21T08:43:10.982089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:43.780935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982579+0000","last_change":"2024-04-21T08:42:21.667459+0000","last_active":"2024-04-21T08:43:10.982579+0000","last_peered":"2024-04-21T08:43:10.982579+0000","last_clean":"2024-04-21T08:43:10.982579+0000","last_became_active":"2024-04-21T08:42:21.667183+0000","last_became_peered":"2024-04-21T08:42:21.667183+0000","last_unstale":"2024-04-21T08:43:10.982579+0000","last_undegraded":"2024-04-21T08:43:10.982579+0000","last_fullsized":"2024-04-21T08:43:10.982579+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:32:23.626267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787200+0000","last_change":"2024-04-21T08:42:16.645210+0000","last_active":"2024-04-21T08:43:10.787200+0000","last_peered":"2024-04-21T08:43:10.787200+0000","last_clean":"2024-04-21T08:43:10.787200+0000","last_became_active":"2024-04-21T08:42:16.644747+0000","last_became_peered":"2024-04-21T08:42:16.644747+0000","last_unstale":"2024-04-21T08:43:10.787200+0000","last_undegraded":"2024-04-21T08:43:10.787200+0000","last_fullsized":"2024-04-21T08:43:10.787200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:34:45.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984152+0000","last_change":"2024-04-21T08:43:10.984152+0000","last_active":"2024-04-21T08:43:10.984152+0000","last_peered":"2024-04-21T08:43:10.984152+0000","last_clean":"2024-04-21T08:43:10.984152+0000","last_became_active":"2024-04-21T08:43:10.983736+0000","last_became_peered":"2024-04-21T08:43:10.983736+0000","last_unstale":"2024-04-21T08:43:10.984152+0000","last_undegraded":"2024-04-21T08:43:10.984152+0000","last_fullsized":"2024-04-21T08:43:10.984152+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:10.182667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981762+0000","last_change":"2024-04-21T08:42:21.660739+0000","last_active":"2024-04-21T08:43:10.981762+0000","last_peered":"2024-04-21T08:43:10.981762+0000","last_clean":"2024-04-21T08:43:10.981762+0000","last_became_active":"2024-04-21T08:42:21.660307+0000","last_became_peered":"2024-04-21T08:42:21.660307+0000","last_unstale":"2024-04-21T08:43:10.981762+0000","last_undegraded":"2024-04-21T08:43:10.981762+0000","last_fullsized":"2024-04-21T08:43:10.981762+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:26.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.986184+0000","last_change":"2024-04-21T08:43:10.986184+0000","last_active":"2024-04-21T08:43:10.986184+0000","last_peered":"2024-04-21T08:43:10.986184+0000","last_clean":"2024-04-21T08:43:10.986184+0000","last_became_active":"2024-04-21T08:43:10.985533+0000","last_became_peered":"2024-04-21T08:43:10.985533+0000","last_unstale":"2024-04-21T08:43:10.986184+0000","last_undegraded":"2024-04-21T08:43:10.986184+0000","last_fullsized":"2024-04-21T08:43:10.986184+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:26.023306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786815+0000","last_change":"2024-04-21T08:42:17.542315+0000","last_active":"2024-04-21T08:43:10.786815+0000","last_peered":"2024-04-21T08:43:10.786815+0000","last_clean":"2024-04-21T08:43:10.786815+0000","last_became_active":"2024-04-21T08:42:17.541895+0000","last_became_peered":"2024-04-21T08:42:17.541895+0000","last_unstale":"2024-04-21T08:43:10.786815+0000","last_undegraded":"2024-04-21T08:43:10.786815+0000","last_fullsized":"2024-04-21T08:43:10.786815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:23:33.909499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981737+0000","last_change":"2024-04-21T08:42:21.661541+0000","last_active":"2024-04-21T08:43:10.981737+0000","last_peered":"2024-04-21T08:43:10.981737+0000","last_clean":"2024-04-21T08:43:10.981737+0000","last_became_active":"2024-04-21T08:42:21.661121+0000","last_became_peered":"2024-04-21T08:42:21.661121+0000","last_unstale":"2024-04-21T08:43:10.981737+0000","last_undegraded":"2024-04-21T08:43:10.981737+0000","last_fullsized":"2024-04-21T08:43:10.981737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:28.624203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984839+0000","last_change":"2024-04-21T08:43:10.984839+0000","last_active":"2024-04-21T08:43:10.984839+0000","last_peered":"2024-04-21T08:43:10.984839+0000","last_clean":"2024-04-21T08:43:10.984839+0000","last_became_active":"2024-04-21T08:43:10.984382+0000","last_became_peered":"2024-04-21T08:43:10.984382+0000","last_unstale":"2024-04-21T08:43:10.984839+0000","last_undegraded":"2024-04-21T08:43:10.984839+0000","last_fullsized":"2024-04-21T08:43:10.984839+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:47:20.774102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791380+0000","last_change":"2024-04-21T08:42:17.547770+0000","last_active":"2024-04-21T08:43:10.791380+0000","last_peered":"2024-04-21T08:43:10.791380+0000","last_clean":"2024-04-21T08:43:10.791380+0000","last_became_active":"2024-04-21T08:42:17.547440+0000","last_became_peered":"2024-04-21T08:42:17.547440+0000","last_unstale":"2024-04-21T08:43:10.791380+0000","last_undegraded":"2024-04-21T08:43:10.791380+0000","last_fullsized":"2024-04-21T08:43:10.791380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:37:35.555737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788605+0000","last_change":"2024-04-21T08:42:21.670684+0000","last_active":"2024-04-21T08:43:10.788605+0000","last_peered":"2024-04-21T08:43:10.788605+0000","last_clean":"2024-04-21T08:43:10.788605+0000","last_became_active":"2024-04-21T08:42:21.670260+0000","last_became_peered":"2024-04-21T08:42:21.670260+0000","last_unstale":"2024-04-21T08:43:10.788605+0000","last_undegraded":"2024-04-21T08:43:10.788605+0000","last_fullsized":"2024-04-21T08:43:10.788605+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:45:06.192040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790350+0000","last_change":"2024-04-21T08:42:16.639632+0000","last_active":"2024-04-21T08:43:10.790350+0000","last_peered":"2024-04-21T08:43:10.790350+0000","last_clean":"2024-04-21T08:43:10.790350+0000","last_became_active":"2024-04-21T08:42:16.639301+0000","last_became_peered":"2024-04-21T08:42:16.639301+0000","last_unstale":"2024-04-21T08:43:10.790350+0000","last_undegraded":"2024-04-21T08:43:10.790350+0000","last_fullsized":"2024-04-21T08:43:10.790350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:58.838667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789111+0000","last_change":"2024-04-21T08:42:17.546109+0000","last_active":"2024-04-21T08:43:10.789111+0000","last_peered":"2024-04-21T08:43:10.789111+0000","last_clean":"2024-04-21T08:43:10.789111+0000","last_became_active":"2024-04-21T08:42:17.545673+0000","last_became_peered":"2024-04-21T08:42:17.545673+0000","last_unstale":"2024-04-21T08:43:10.789111+0000","last_undegraded":"2024-04-21T08:43:10.789111+0000","last_fullsized":"2024-04-21T08:43:10.789111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:18.947848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790068+0000","last_change":"2024-04-21T08:42:21.666114+0000","last_active":"2024-04-21T08:43:10.790068+0000","last_peered":"2024-04-21T08:43:10.790068+0000","last_clean":"2024-04-21T08:43:10.790068+0000","last_became_active":"2024-04-21T08:42:21.665020+0000","last_became_peered":"2024-04-21T08:42:21.665020+0000","last_unstale":"2024-04-21T08:43:10.790068+0000","last_undegraded":"2024-04-21T08:43:10.790068+0000","last_fullsized":"2024-04-21T08:43:10.790068+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:33.644178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790644+0000","last_change":"2024-04-21T08:42:16.639118+0000","last_active":"2024-04-21T08:43:10.790644+0000","last_peered":"2024-04-21T08:43:10.790644+0000","last_clean":"2024-04-21T08:43:10.790644+0000","last_became_active":"2024-04-21T08:42:16.638792+0000","last_became_peered":"2024-04-21T08:42:16.638792+0000","last_unstale":"2024-04-21T08:43:10.790644+0000","last_undegraded":"2024-04-21T08:43:10.790644+0000","last_fullsized":"2024-04-21T08:43:10.790644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:55.568712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.785869+0000","last_change":"2024-04-21T08:42:17.539688+0000","last_active":"2024-04-21T08:43:10.785869+0000","last_peered":"2024-04-21T08:43:10.785869+0000","last_clean":"2024-04-21T08:43:10.785869+0000","last_became_active":"2024-04-21T08:42:17.539335+0000","last_became_peered":"2024-04-21T08:42:17.539335+0000","last_unstale":"2024-04-21T08:43:10.785869+0000","last_undegraded":"2024-04-21T08:43:10.785869+0000","last_fullsized":"2024-04-21T08:43:10.785869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:27:51.520232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789278+0000","last_change":"2024-04-21T08:42:21.660570+0000","last_active":"2024-04-21T08:43:10.789278+0000","last_peered":"2024-04-21T08:43:10.789278+0000","last_clean":"2024-04-21T08:43:10.789278+0000","last_became_active":"2024-04-21T08:42:21.659398+0000","last_became_peered":"2024-04-21T08:42:21.659398+0000","last_unstale":"2024-04-21T08:43:10.789278+0000","last_undegraded":"2024-04-21T08:43:10.789278+0000","last_fullsized":"2024-04-21T08:43:10.789278+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:55.840334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787693+0000","last_change":"2024-04-21T08:42:16.648192+0000","last_active":"2024-04-21T08:43:10.787693+0000","last_peered":"2024-04-21T08:43:10.787693+0000","last_clean":"2024-04-21T08:43:10.787693+0000","last_became_active":"2024-04-21T08:42:16.647666+0000","last_became_peered":"2024-04-21T08:42:16.647666+0000","last_unstale":"2024-04-21T08:43:10.787693+0000","last_undegraded":"2024-04-21T08:43:10.787693+0000","last_fullsized":"2024-04-21T08:43:10.787693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:33.344680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983282+0000","last_change":"2024-04-21T08:42:17.545925+0000","last_active":"2024-04-21T08:43:10.983282+0000","last_peered":"2024-04-21T08:43:10.983282+0000","last_clean":"2024-04-21T08:43:10.983282+0000","last_became_active":"2024-04-21T08:42:17.545555+0000","last_became_peered":"2024-04-21T08:42:17.545555+0000","last_unstale":"2024-04-21T08:43:10.983282+0000","last_undegraded":"2024-04-21T08:43:10.983282+0000","last_fullsized":"2024-04-21T08:43:10.983282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:13:07.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.785669+0000","last_change":"2024-04-21T08:42:21.662757+0000","last_active":"2024-04-21T08:43:10.785669+0000","last_peered":"2024-04-21T08:43:10.785669+0000","last_clean":"2024-04-21T08:43:10.785669+0000","last_became_active":"2024-04-21T08:42:21.662306+0000","last_became_peered":"2024-04-21T08:42:21.662306+0000","last_unstale":"2024-04-21T08:43:10.785669+0000","last_undegraded":"2024-04-21T08:43:10.785669+0000","last_fullsized":"2024-04-21T08:43:10.785669+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:26.856147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984690+0000","last_change":"2024-04-21T08:43:10.984690+0000","last_active":"2024-04-21T08:43:10.984690+0000","last_peered":"2024-04-21T08:43:10.984690+0000","last_clean":"2024-04-21T08:43:10.984690+0000","last_became_active":"2024-04-21T08:43:10.984290+0000","last_became_peered":"2024-04-21T08:43:10.984290+0000","last_unstale":"2024-04-21T08:43:10.984690+0000","last_undegraded":"2024-04-21T08:43:10.984690+0000","last_fullsized":"2024-04-21T08:43:10.984690+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:37:36.400946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.793316+0000","last_change":"2024-04-21T08:43:10.793316+0000","last_active":"2024-04-21T08:43:10.793316+0000","last_peered":"2024-04-21T08:43:10.793316+0000","last_clean":"2024-04-21T08:43:10.793316+0000","last_became_active":"2024-04-21T08:43:10.793035+0000","last_became_peered":"2024-04-21T08:43:10.793035+0000","last_unstale":"2024-04-21T08:43:10.793316+0000","last_undegraded":"2024-04-21T08:43:10.793316+0000","last_fullsized":"2024-04-21T08:43:10.793316+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:41:04.247385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983457+0000","last_change":"2024-04-21T08:42:21.670561+0000","last_active":"2024-04-21T08:43:10.983457+0000","last_peered":"2024-04-21T08:43:10.983457+0000","last_clean":"2024-04-21T08:43:10.983457+0000","last_became_active":"2024-04-21T08:42:21.670170+0000","last_became_peered":"2024-04-21T08:42:21.670170+0000","last_unstale":"2024-04-21T08:43:10.983457+0000","last_undegraded":"2024-04-21T08:43:10.983457+0000","last_fullsized":"2024-04-21T08:43:10.983457+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:35:54.538013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790743+0000","last_change":"2024-04-21T08:42:16.637319+0000","last_active":"2024-04-21T08:43:10.790743+0000","last_peered":"2024-04-21T08:43:10.790743+0000","last_clean":"2024-04-21T08:43:10.790743+0000","last_became_active":"2024-04-21T08:42:16.636940+0000","last_became_peered":"2024-04-21T08:42:16.636940+0000","last_unstale":"2024-04-21T08:43:10.790743+0000","last_undegraded":"2024-04-21T08:43:10.790743+0000","last_fullsized":"2024-04-21T08:43:10.790743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:59:09.826786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982487+0000","last_change":"2024-04-21T08:42:17.546202+0000","last_active":"2024-04-21T08:43:10.982487+0000","last_peered":"2024-04-21T08:43:10.982487+0000","last_clean":"2024-04-21T08:43:10.982487+0000","last_became_active":"2024-04-21T08:42:17.545781+0000","last_became_peered":"2024-04-21T08:42:17.545781+0000","last_unstale":"2024-04-21T08:43:10.982487+0000","last_undegraded":"2024-04-21T08:43:10.982487+0000","last_fullsized":"2024-04-21T08:43:10.982487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:23.990951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.785762+0000","last_change":"2024-04-21T08:42:21.662822+0000","last_active":"2024-04-21T08:43:10.785762+0000","last_peered":"2024-04-21T08:43:10.785762+0000","last_clean":"2024-04-21T08:43:10.785762+0000","last_became_active":"2024-04-21T08:42:21.662398+0000","last_became_peered":"2024-04-21T08:42:21.662398+0000","last_unstale":"2024-04-21T08:43:10.785762+0000","last_undegraded":"2024-04-21T08:43:10.785762+0000","last_fullsized":"2024-04-21T08:43:10.785762+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:38:40.499294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":214,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:56.058633+0000","last_change":"2024-04-21T08:42:16.636330+0000","last_active":"2024-04-21T08:43:56.058633+0000","last_peered":"2024-04-21T08:43:56.058633+0000","last_clean":"2024-04-21T08:43:56.058633+0000","last_became_active":"2024-04-21T08:42:16.635762+0000","last_became_peered":"2024-04-21T08:42:16.635762+0000","last_unstale":"2024-04-21T08:43:56.058633+0000","last_undegraded":"2024-04-21T08:43:56.058633+0000","last_fullsized":"2024-04-21T08:43:56.058633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:31.137923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":157,"num_read_kb":140,"num_write":30,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787464+0000","last_change":"2024-04-21T08:42:17.543033+0000","last_active":"2024-04-21T08:43:10.787464+0000","last_peered":"2024-04-21T08:43:10.787464+0000","last_clean":"2024-04-21T08:43:10.787464+0000","last_became_active":"2024-04-21T08:42:17.542692+0000","last_became_peered":"2024-04-21T08:42:17.542692+0000","last_unstale":"2024-04-21T08:43:10.787464+0000","last_undegraded":"2024-04-21T08:43:10.787464+0000","last_fullsized":"2024-04-21T08:43:10.787464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:33:32.111999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791344+0000","last_change":"2024-04-21T08:42:21.668831+0000","last_active":"2024-04-21T08:43:10.791344+0000","last_peered":"2024-04-21T08:43:10.791344+0000","last_clean":"2024-04-21T08:43:10.791344+0000","last_became_active":"2024-04-21T08:42:21.668485+0000","last_became_peered":"2024-04-21T08:42:21.668485+0000","last_unstale":"2024-04-21T08:43:10.791344+0000","last_undegraded":"2024-04-21T08:43:10.791344+0000","last_fullsized":"2024-04-21T08:43:10.791344+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:49:29.049707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981747+0000","last_change":"2024-04-21T08:42:16.640288+0000","last_active":"2024-04-21T08:43:10.981747+0000","last_peered":"2024-04-21T08:43:10.981747+0000","last_clean":"2024-04-21T08:43:10.981747+0000","last_became_active":"2024-04-21T08:42:16.639909+0000","last_became_peered":"2024-04-21T08:42:16.639909+0000","last_unstale":"2024-04-21T08:43:10.981747+0000","last_undegraded":"2024-04-21T08:43:10.981747+0000","last_fullsized":"2024-04-21T08:43:10.981747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:33.839442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790747+0000","last_change":"2024-04-21T08:42:17.541154+0000","last_active":"2024-04-21T08:43:10.790747+0000","last_peered":"2024-04-21T08:43:10.790747+0000","last_clean":"2024-04-21T08:43:10.790747+0000","last_became_active":"2024-04-21T08:42:17.540747+0000","last_became_peered":"2024-04-21T08:42:17.540747+0000","last_unstale":"2024-04-21T08:43:10.790747+0000","last_undegraded":"2024-04-21T08:43:10.790747+0000","last_fullsized":"2024-04-21T08:43:10.790747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:14:50.641144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788325+0000","last_change":"2024-04-21T08:42:21.670767+0000","last_active":"2024-04-21T08:43:10.788325+0000","last_peered":"2024-04-21T08:43:10.788325+0000","last_clean":"2024-04-21T08:43:10.788325+0000","last_became_active":"2024-04-21T08:42:21.670338+0000","last_became_peered":"2024-04-21T08:42:21.670338+0000","last_unstale":"2024-04-21T08:43:10.788325+0000","last_undegraded":"2024-04-21T08:43:10.788325+0000","last_fullsized":"2024-04-21T08:43:10.788325+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:38:55.275603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790663+0000","last_change":"2024-04-21T08:42:16.640144+0000","last_active":"2024-04-21T08:43:10.790663+0000","last_peered":"2024-04-21T08:43:10.790663+0000","last_clean":"2024-04-21T08:43:10.790663+0000","last_became_active":"2024-04-21T08:42:16.639707+0000","last_became_peered":"2024-04-21T08:42:16.639707+0000","last_unstale":"2024-04-21T08:43:10.790663+0000","last_undegraded":"2024-04-21T08:43:10.790663+0000","last_fullsized":"2024-04-21T08:43:10.790663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:11:32.785519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790033+0000","last_change":"2024-04-21T08:42:17.555019+0000","last_active":"2024-04-21T08:43:10.790033+0000","last_peered":"2024-04-21T08:43:10.790033+0000","last_clean":"2024-04-21T08:43:10.790033+0000","last_became_active":"2024-04-21T08:42:17.554645+0000","last_became_peered":"2024-04-21T08:42:17.554645+0000","last_unstale":"2024-04-21T08:43:10.790033+0000","last_undegraded":"2024-04-21T08:43:10.790033+0000","last_fullsized":"2024-04-21T08:43:10.790033+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:42:30.353945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786133+0000","last_change":"2024-04-21T08:42:21.663567+0000","last_active":"2024-04-21T08:43:10.786133+0000","last_peered":"2024-04-21T08:43:10.786133+0000","last_clean":"2024-04-21T08:43:10.786133+0000","last_became_active":"2024-04-21T08:42:21.663203+0000","last_became_peered":"2024-04-21T08:42:21.663203+0000","last_unstale":"2024-04-21T08:43:10.786133+0000","last_undegraded":"2024-04-21T08:43:10.786133+0000","last_fullsized":"2024-04-21T08:43:10.786133+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:50.760336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982329+0000","last_change":"2024-04-21T08:42:16.640407+0000","last_active":"2024-04-21T08:43:10.982329+0000","last_peered":"2024-04-21T08:43:10.982329+0000","last_clean":"2024-04-21T08:43:10.982329+0000","last_became_active":"2024-04-21T08:42:16.640015+0000","last_became_peered":"2024-04-21T08:42:16.640015+0000","last_unstale":"2024-04-21T08:43:10.982329+0000","last_undegraded":"2024-04-21T08:43:10.982329+0000","last_fullsized":"2024-04-21T08:43:10.982329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:23:10.647184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789722+0000","last_change":"2024-04-21T08:42:17.545438+0000","last_active":"2024-04-21T08:43:10.789722+0000","last_peered":"2024-04-21T08:43:10.789722+0000","last_clean":"2024-04-21T08:43:10.789722+0000","last_became_active":"2024-04-21T08:42:17.544258+0000","last_became_peered":"2024-04-21T08:42:17.544258+0000","last_unstale":"2024-04-21T08:43:10.789722+0000","last_undegraded":"2024-04-21T08:43:10.789722+0000","last_fullsized":"2024-04-21T08:43:10.789722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:43.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791292+0000","last_change":"2024-04-21T08:42:21.671000+0000","last_active":"2024-04-21T08:43:10.791292+0000","last_peered":"2024-04-21T08:43:10.791292+0000","last_clean":"2024-04-21T08:43:10.791292+0000","last_became_active":"2024-04-21T08:42:21.670380+0000","last_became_peered":"2024-04-21T08:42:21.670380+0000","last_unstale":"2024-04-21T08:43:10.791292+0000","last_undegraded":"2024-04-21T08:43:10.791292+0000","last_fullsized":"2024-04-21T08:43:10.791292+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:57.075686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786538+0000","last_change":"2024-04-21T08:42:16.649599+0000","last_active":"2024-04-21T08:43:10.786538+0000","last_peered":"2024-04-21T08:43:10.786538+0000","last_clean":"2024-04-21T08:43:10.786538+0000","last_became_active":"2024-04-21T08:42:16.649232+0000","last_became_peered":"2024-04-21T08:42:16.649232+0000","last_unstale":"2024-04-21T08:43:10.786538+0000","last_undegraded":"2024-04-21T08:43:10.786538+0000","last_fullsized":"2024-04-21T08:43:10.786538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:21.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790390+0000","last_change":"2024-04-21T08:42:17.545910+0000","last_active":"2024-04-21T08:43:10.790390+0000","last_peered":"2024-04-21T08:43:10.790390+0000","last_clean":"2024-04-21T08:43:10.790390+0000","last_became_active":"2024-04-21T08:42:17.545582+0000","last_became_peered":"2024-04-21T08:42:17.545582+0000","last_unstale":"2024-04-21T08:43:10.790390+0000","last_undegraded":"2024-04-21T08:43:10.790390+0000","last_fullsized":"2024-04-21T08:43:10.790390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:22:25.466146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790378+0000","last_change":"2024-04-21T08:42:21.664811+0000","last_active":"2024-04-21T08:43:10.790378+0000","last_peered":"2024-04-21T08:43:10.790378+0000","last_clean":"2024-04-21T08:43:10.790378+0000","last_became_active":"2024-04-21T08:42:21.664332+0000","last_became_peered":"2024-04-21T08:42:21.664332+0000","last_unstale":"2024-04-21T08:43:10.790378+0000","last_undegraded":"2024-04-21T08:43:10.790378+0000","last_fullsized":"2024-04-21T08:43:10.790378+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:40.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787316+0000","last_change":"2024-04-21T08:42:16.641233+0000","last_active":"2024-04-21T08:43:10.787316+0000","last_peered":"2024-04-21T08:43:10.787316+0000","last_clean":"2024-04-21T08:43:10.787316+0000","last_became_active":"2024-04-21T08:42:16.640873+0000","last_became_peered":"2024-04-21T08:42:16.640873+0000","last_unstale":"2024-04-21T08:43:10.787316+0000","last_undegraded":"2024-04-21T08:43:10.787316+0000","last_fullsized":"2024-04-21T08:43:10.787316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:00.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788866+0000","last_change":"2024-04-21T08:42:17.550438+0000","last_active":"2024-04-21T08:43:10.788866+0000","last_peered":"2024-04-21T08:43:10.788866+0000","last_clean":"2024-04-21T08:43:10.788866+0000","last_became_active":"2024-04-21T08:42:17.550015+0000","last_became_peered":"2024-04-21T08:42:17.550015+0000","last_unstale":"2024-04-21T08:43:10.788866+0000","last_undegraded":"2024-04-21T08:43:10.788866+0000","last_fullsized":"2024-04-21T08:43:10.788866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:00:29.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786730+0000","last_change":"2024-04-21T08:42:21.970498+0000","last_active":"2024-04-21T08:43:10.786730+0000","last_peered":"2024-04-21T08:43:10.786730+0000","last_clean":"2024-04-21T08:43:10.786730+0000","last_became_active":"2024-04-21T08:42:21.969819+0000","last_became_peered":"2024-04-21T08:42:21.969819+0000","last_unstale":"2024-04-21T08:43:10.786730+0000","last_undegraded":"2024-04-21T08:43:10.786730+0000","last_fullsized":"2024-04-21T08:43:10.786730+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:13:16.552293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786143+0000","last_change":"2024-04-21T08:42:16.637146+0000","last_active":"2024-04-21T08:43:10.786143+0000","last_peered":"2024-04-21T08:43:10.786143+0000","last_clean":"2024-04-21T08:43:10.786143+0000","last_became_active":"2024-04-21T08:42:16.636781+0000","last_became_peered":"2024-04-21T08:42:16.636781+0000","last_unstale":"2024-04-21T08:43:10.786143+0000","last_undegraded":"2024-04-21T08:43:10.786143+0000","last_fullsized":"2024-04-21T08:43:10.786143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:39:27.641236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787514+0000","last_change":"2024-04-21T08:42:17.542325+0000","last_active":"2024-04-21T08:43:10.787514+0000","last_peered":"2024-04-21T08:43:10.787514+0000","last_clean":"2024-04-21T08:43:10.787514+0000","last_became_active":"2024-04-21T08:42:17.541103+0000","last_became_peered":"2024-04-21T08:42:17.541103+0000","last_unstale":"2024-04-21T08:43:10.787514+0000","last_undegraded":"2024-04-21T08:43:10.787514+0000","last_fullsized":"2024-04-21T08:43:10.787514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:46.903250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790533+0000","last_change":"2024-04-21T08:42:21.671146+0000","last_active":"2024-04-21T08:43:10.790533+0000","last_peered":"2024-04-21T08:43:10.790533+0000","last_clean":"2024-04-21T08:43:10.790533+0000","last_became_active":"2024-04-21T08:42:21.670561+0000","last_became_peered":"2024-04-21T08:42:21.670561+0000","last_unstale":"2024-04-21T08:43:10.790533+0000","last_undegraded":"2024-04-21T08:43:10.790533+0000","last_fullsized":"2024-04-21T08:43:10.790533+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:38:28.028976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789729+0000","last_change":"2024-04-21T08:42:16.645132+0000","last_active":"2024-04-21T08:43:10.789729+0000","last_peered":"2024-04-21T08:43:10.789729+0000","last_clean":"2024-04-21T08:43:10.789729+0000","last_became_active":"2024-04-21T08:42:16.644786+0000","last_became_peered":"2024-04-21T08:42:16.644786+0000","last_unstale":"2024-04-21T08:43:10.789729+0000","last_undegraded":"2024-04-21T08:43:10.789729+0000","last_fullsized":"2024-04-21T08:43:10.789729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:08.298498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787281+0000","last_change":"2024-04-21T08:42:17.550424+0000","last_active":"2024-04-21T08:43:10.787281+0000","last_peered":"2024-04-21T08:43:10.787281+0000","last_clean":"2024-04-21T08:43:10.787281+0000","last_became_active":"2024-04-21T08:42:17.550077+0000","last_became_peered":"2024-04-21T08:42:17.550077+0000","last_unstale":"2024-04-21T08:43:10.787281+0000","last_undegraded":"2024-04-21T08:43:10.787281+0000","last_fullsized":"2024-04-21T08:43:10.787281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:17:07.016204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789770+0000","last_change":"2024-04-21T08:42:21.970652+0000","last_active":"2024-04-21T08:43:10.789770+0000","last_peered":"2024-04-21T08:43:10.789770+0000","last_clean":"2024-04-21T08:43:10.789770+0000","last_became_active":"2024-04-21T08:42:21.970251+0000","last_became_peered":"2024-04-21T08:42:21.970251+0000","last_unstale":"2024-04-21T08:43:10.789770+0000","last_undegraded":"2024-04-21T08:43:10.789770+0000","last_fullsized":"2024-04-21T08:43:10.789770+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:45.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790879+0000","last_change":"2024-04-21T08:42:17.542820+0000","last_active":"2024-04-21T08:43:10.790879+0000","last_peered":"2024-04-21T08:43:10.790879+0000","last_clean":"2024-04-21T08:43:10.790879+0000","last_became_active":"2024-04-21T08:42:17.541320+0000","last_became_peered":"2024-04-21T08:42:17.541320+0000","last_unstale":"2024-04-21T08:43:10.790879+0000","last_undegraded":"2024-04-21T08:43:10.790879+0000","last_fullsized":"2024-04-21T08:43:10.790879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:23:06.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790253+0000","last_change":"2024-04-21T08:42:16.636950+0000","last_active":"2024-04-21T08:43:10.790253+0000","last_peered":"2024-04-21T08:43:10.790253+0000","last_clean":"2024-04-21T08:43:10.790253+0000","last_became_active":"2024-04-21T08:42:16.636590+0000","last_became_peered":"2024-04-21T08:42:16.636590+0000","last_unstale":"2024-04-21T08:43:10.790253+0000","last_undegraded":"2024-04-21T08:43:10.790253+0000","last_fullsized":"2024-04-21T08:43:10.790253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:55.693681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790154+0000","last_change":"2024-04-21T08:42:21.660404+0000","last_active":"2024-04-21T08:43:10.790154+0000","last_peered":"2024-04-21T08:43:10.790154+0000","last_clean":"2024-04-21T08:43:10.790154+0000","last_became_active":"2024-04-21T08:42:21.659999+0000","last_became_peered":"2024-04-21T08:42:21.659999+0000","last_unstale":"2024-04-21T08:43:10.790154+0000","last_undegraded":"2024-04-21T08:43:10.790154+0000","last_fullsized":"2024-04-21T08:43:10.790154+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:53:32.328873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984671+0000","last_change":"2024-04-21T08:43:10.984361+0000","last_active":"2024-04-21T08:43:10.984671+0000","last_peered":"2024-04-21T08:43:10.984671+0000","last_clean":"2024-04-21T08:43:10.984671+0000","last_became_active":"2024-04-21T08:43:10.984046+0000","last_became_peered":"2024-04-21T08:43:10.984046+0000","last_unstale":"2024-04-21T08:43:10.984671+0000","last_undegraded":"2024-04-21T08:43:10.984671+0000","last_fullsized":"2024-04-21T08:43:10.984671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:33:23.154630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984798+0000","last_change":"2024-04-21T08:42:16.638993+0000","last_active":"2024-04-21T08:43:10.984798+0000","last_peered":"2024-04-21T08:43:10.984798+0000","last_clean":"2024-04-21T08:43:10.984798+0000","last_became_active":"2024-04-21T08:42:16.638617+0000","last_became_peered":"2024-04-21T08:42:16.638617+0000","last_unstale":"2024-04-21T08:43:10.984798+0000","last_undegraded":"2024-04-21T08:43:10.984798+0000","last_fullsized":"2024-04-21T08:43:10.984798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:14:28.271433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":360,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789557+0000","last_change":"2024-04-21T08:41:00.044984+0000","last_active":"2024-04-21T08:43:10.789557+0000","last_peered":"2024-04-21T08:43:10.789557+0000","last_clean":"2024-04-21T08:43:10.789557+0000","last_became_active":"2024-04-21T08:41:00.044485+0000","last_became_peered":"2024-04-21T08:41:00.044485+0000","last_unstale":"2024-04-21T08:43:10.789557+0000","last_undegraded":"2024-04-21T08:43:10.789557+0000","last_fullsized":"2024-04-21T08:43:10.789557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.370031+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:01.031209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791249+0000","last_change":"2024-04-21T08:42:21.971045+0000","last_active":"2024-04-21T08:43:10.791249+0000","last_peered":"2024-04-21T08:43:10.791249+0000","last_clean":"2024-04-21T08:43:10.791249+0000","last_became_active":"2024-04-21T08:42:21.970577+0000","last_became_peered":"2024-04-21T08:42:21.970577+0000","last_unstale":"2024-04-21T08:43:10.791249+0000","last_undegraded":"2024-04-21T08:43:10.791249+0000","last_fullsized":"2024-04-21T08:43:10.791249+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:55.681662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790149+0000","last_change":"2024-04-21T08:42:17.535570+0000","last_active":"2024-04-21T08:43:10.790149+0000","last_peered":"2024-04-21T08:43:10.790149+0000","last_clean":"2024-04-21T08:43:10.790149+0000","last_became_active":"2024-04-21T08:42:17.535217+0000","last_became_peered":"2024-04-21T08:42:17.535217+0000","last_unstale":"2024-04-21T08:43:10.790149+0000","last_undegraded":"2024-04-21T08:43:10.790149+0000","last_fullsized":"2024-04-21T08:43:10.790149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:49.510750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.792548+0000","last_change":"2024-04-21T08:43:10.792548+0000","last_active":"2024-04-21T08:43:10.792548+0000","last_peered":"2024-04-21T08:43:10.792548+0000","last_clean":"2024-04-21T08:43:10.792548+0000","last_became_active":"2024-04-21T08:43:10.792165+0000","last_became_peered":"2024-04-21T08:43:10.792165+0000","last_unstale":"2024-04-21T08:43:10.792548+0000","last_undegraded":"2024-04-21T08:43:10.792548+0000","last_fullsized":"2024-04-21T08:43:10.792548+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:35.980536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788172+0000","last_change":"2024-04-21T08:42:21.661393+0000","last_active":"2024-04-21T08:43:10.788172+0000","last_peered":"2024-04-21T08:43:10.788172+0000","last_clean":"2024-04-21T08:43:10.788172+0000","last_became_active":"2024-04-21T08:42:21.660201+0000","last_became_peered":"2024-04-21T08:42:21.660201+0000","last_unstale":"2024-04-21T08:43:10.788172+0000","last_undegraded":"2024-04-21T08:43:10.788172+0000","last_fullsized":"2024-04-21T08:43:10.788172+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:13.393389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.787392+0000","last_change":"2024-04-21T08:42:16.650507+0000","last_active":"2024-04-21T08:43:10.787392+0000","last_peered":"2024-04-21T08:43:10.787392+0000","last_clean":"2024-04-21T08:43:10.787392+0000","last_became_active":"2024-04-21T08:42:16.650217+0000","last_became_peered":"2024-04-21T08:42:16.650217+0000","last_unstale":"2024-04-21T08:43:10.787392+0000","last_undegraded":"2024-04-21T08:43:10.787392+0000","last_fullsized":"2024-04-21T08:43:10.787392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:13:26.785092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790643+0000","last_change":"2024-04-21T08:42:17.530420+0000","last_active":"2024-04-21T08:43:10.790643+0000","last_peered":"2024-04-21T08:43:10.790643+0000","last_clean":"2024-04-21T08:43:10.790643+0000","last_became_active":"2024-04-21T08:42:17.529920+0000","last_became_peered":"2024-04-21T08:42:17.529920+0000","last_unstale":"2024-04-21T08:43:10.790643+0000","last_undegraded":"2024-04-21T08:43:10.790643+0000","last_fullsized":"2024-04-21T08:43:10.790643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:24.250197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789027+0000","last_change":"2024-04-21T08:42:21.970586+0000","last_active":"2024-04-21T08:43:10.789027+0000","last_peered":"2024-04-21T08:43:10.789027+0000","last_clean":"2024-04-21T08:43:10.789027+0000","last_became_active":"2024-04-21T08:42:21.970132+0000","last_became_peered":"2024-04-21T08:42:21.970132+0000","last_unstale":"2024-04-21T08:43:10.789027+0000","last_undegraded":"2024-04-21T08:43:10.789027+0000","last_fullsized":"2024-04-21T08:43:10.789027+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:11.074370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786265+0000","last_change":"2024-04-21T08:42:16.647916+0000","last_active":"2024-04-21T08:43:10.786265+0000","last_peered":"2024-04-21T08:43:10.786265+0000","last_clean":"2024-04-21T08:43:10.786265+0000","last_became_active":"2024-04-21T08:42:16.647476+0000","last_became_peered":"2024-04-21T08:42:16.647476+0000","last_unstale":"2024-04-21T08:43:10.786265+0000","last_undegraded":"2024-04-21T08:43:10.786265+0000","last_fullsized":"2024-04-21T08:43:10.786265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:27.242153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789667+0000","last_change":"2024-04-21T08:42:17.546144+0000","last_active":"2024-04-21T08:43:10.789667+0000","last_peered":"2024-04-21T08:43:10.789667+0000","last_clean":"2024-04-21T08:43:10.789667+0000","last_became_active":"2024-04-21T08:42:17.545730+0000","last_became_peered":"2024-04-21T08:42:17.545730+0000","last_unstale":"2024-04-21T08:43:10.789667+0000","last_undegraded":"2024-04-21T08:43:10.789667+0000","last_fullsized":"2024-04-21T08:43:10.789667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:42.802309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981225+0000","last_change":"2024-04-21T08:42:21.655802+0000","last_active":"2024-04-21T08:43:10.981225+0000","last_peered":"2024-04-21T08:43:10.981225+0000","last_clean":"2024-04-21T08:43:10.981225+0000","last_became_active":"2024-04-21T08:42:21.655379+0000","last_became_peered":"2024-04-21T08:42:21.655379+0000","last_unstale":"2024-04-21T08:43:10.981225+0000","last_undegraded":"2024-04-21T08:43:10.981225+0000","last_fullsized":"2024-04-21T08:43:10.981225+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:49.486057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984198+0000","last_change":"2024-04-21T08:42:16.642517+0000","last_active":"2024-04-21T08:43:10.984198+0000","last_peered":"2024-04-21T08:43:10.984198+0000","last_clean":"2024-04-21T08:43:10.984198+0000","last_became_active":"2024-04-21T08:42:16.641539+0000","last_became_peered":"2024-04-21T08:42:16.641539+0000","last_unstale":"2024-04-21T08:43:10.984198+0000","last_undegraded":"2024-04-21T08:43:10.984198+0000","last_fullsized":"2024-04-21T08:43:10.984198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:53.912199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790179+0000","last_change":"2024-04-21T08:42:17.550172+0000","last_active":"2024-04-21T08:43:10.790179+0000","last_peered":"2024-04-21T08:43:10.790179+0000","last_clean":"2024-04-21T08:43:10.790179+0000","last_became_active":"2024-04-21T08:42:17.549824+0000","last_became_peered":"2024-04-21T08:42:17.549824+0000","last_unstale":"2024-04-21T08:43:10.790179+0000","last_undegraded":"2024-04-21T08:43:10.790179+0000","last_fullsized":"2024-04-21T08:43:10.790179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:59.553645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983731+0000","last_change":"2024-04-21T08:42:21.970654+0000","last_active":"2024-04-21T08:43:10.983731+0000","last_peered":"2024-04-21T08:43:10.983731+0000","last_clean":"2024-04-21T08:43:10.983731+0000","last_became_active":"2024-04-21T08:42:21.970211+0000","last_became_peered":"2024-04-21T08:42:21.970211+0000","last_unstale":"2024-04-21T08:43:10.983731+0000","last_undegraded":"2024-04-21T08:43:10.983731+0000","last_fullsized":"2024-04-21T08:43:10.983731+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:24:51.406125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982736+0000","last_change":"2024-04-21T08:42:16.647985+0000","last_active":"2024-04-21T08:43:10.982736+0000","last_peered":"2024-04-21T08:43:10.982736+0000","last_clean":"2024-04-21T08:43:10.982736+0000","last_became_active":"2024-04-21T08:42:16.647577+0000","last_became_peered":"2024-04-21T08:42:16.647577+0000","last_unstale":"2024-04-21T08:43:10.982736+0000","last_undegraded":"2024-04-21T08:43:10.982736+0000","last_fullsized":"2024-04-21T08:43:10.982736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:00.731708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786532+0000","last_change":"2024-04-21T08:42:17.541757+0000","last_active":"2024-04-21T08:43:10.786532+0000","last_peered":"2024-04-21T08:43:10.786532+0000","last_clean":"2024-04-21T08:43:10.786532+0000","last_became_active":"2024-04-21T08:42:17.539280+0000","last_became_peered":"2024-04-21T08:42:17.539280+0000","last_unstale":"2024-04-21T08:43:10.786532+0000","last_undegraded":"2024-04-21T08:43:10.786532+0000","last_fullsized":"2024-04-21T08:43:10.786532+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:45:54.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786263+0000","last_change":"2024-04-21T08:42:21.664566+0000","last_active":"2024-04-21T08:43:10.786263+0000","last_peered":"2024-04-21T08:43:10.786263+0000","last_clean":"2024-04-21T08:43:10.786263+0000","last_became_active":"2024-04-21T08:42:21.664136+0000","last_became_peered":"2024-04-21T08:42:21.664136+0000","last_unstale":"2024-04-21T08:43:10.786263+0000","last_undegraded":"2024-04-21T08:43:10.786263+0000","last_fullsized":"2024-04-21T08:43:10.786263+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:19.856577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789368+0000","last_change":"2024-04-21T08:42:16.650664+0000","last_active":"2024-04-21T08:43:10.789368+0000","last_peered":"2024-04-21T08:43:10.789368+0000","last_clean":"2024-04-21T08:43:10.789368+0000","last_became_active":"2024-04-21T08:42:16.650339+0000","last_became_peered":"2024-04-21T08:42:16.650339+0000","last_unstale":"2024-04-21T08:43:10.789368+0000","last_undegraded":"2024-04-21T08:43:10.789368+0000","last_fullsized":"2024-04-21T08:43:10.789368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:59.565455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788742+0000","last_change":"2024-04-21T08:42:17.538142+0000","last_active":"2024-04-21T08:43:10.788742+0000","last_peered":"2024-04-21T08:43:10.788742+0000","last_clean":"2024-04-21T08:43:10.788742+0000","last_became_active":"2024-04-21T08:42:17.537793+0000","last_became_peered":"2024-04-21T08:42:17.537793+0000","last_unstale":"2024-04-21T08:43:10.788742+0000","last_undegraded":"2024-04-21T08:43:10.788742+0000","last_fullsized":"2024-04-21T08:43:10.788742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:11.059294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.984088+0000","last_change":"2024-04-21T08:42:21.670642+0000","last_active":"2024-04-21T08:43:10.984088+0000","last_peered":"2024-04-21T08:43:10.984088+0000","last_clean":"2024-04-21T08:43:10.984088+0000","last_became_active":"2024-04-21T08:42:21.670128+0000","last_became_peered":"2024-04-21T08:42:21.670128+0000","last_unstale":"2024-04-21T08:43:10.984088+0000","last_undegraded":"2024-04-21T08:43:10.984088+0000","last_fullsized":"2024-04-21T08:43:10.984088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:51:28.604381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.792114+0000","last_change":"2024-04-21T08:43:10.792114+0000","last_active":"2024-04-21T08:43:10.792114+0000","last_peered":"2024-04-21T08:43:10.792114+0000","last_clean":"2024-04-21T08:43:10.792114+0000","last_became_active":"2024-04-21T08:43:10.791664+0000","last_became_peered":"2024-04-21T08:43:10.791664+0000","last_unstale":"2024-04-21T08:43:10.792114+0000","last_undegraded":"2024-04-21T08:43:10.792114+0000","last_fullsized":"2024-04-21T08:43:10.792114+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:07.317804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789375+0000","last_change":"2024-04-21T08:42:17.547326+0000","last_active":"2024-04-21T08:43:10.789375+0000","last_peered":"2024-04-21T08:43:10.789375+0000","last_clean":"2024-04-21T08:43:10.789375+0000","last_became_active":"2024-04-21T08:42:17.546920+0000","last_became_peered":"2024-04-21T08:42:17.546920+0000","last_unstale":"2024-04-21T08:43:10.789375+0000","last_undegraded":"2024-04-21T08:43:10.789375+0000","last_fullsized":"2024-04-21T08:43:10.789375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:43.350877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.983874+0000","last_change":"2024-04-21T08:42:21.971147+0000","last_active":"2024-04-21T08:43:10.983874+0000","last_peered":"2024-04-21T08:43:10.983874+0000","last_clean":"2024-04-21T08:43:10.983874+0000","last_became_active":"2024-04-21T08:42:21.970001+0000","last_became_peered":"2024-04-21T08:42:21.970001+0000","last_unstale":"2024-04-21T08:43:10.983874+0000","last_undegraded":"2024-04-21T08:43:10.983874+0000","last_fullsized":"2024-04-21T08:43:10.983874+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:01:44.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.786986+0000","last_change":"2024-04-21T08:42:16.643371+0000","last_active":"2024-04-21T08:43:10.786986+0000","last_peered":"2024-04-21T08:43:10.786986+0000","last_clean":"2024-04-21T08:43:10.786986+0000","last_became_active":"2024-04-21T08:42:16.642906+0000","last_became_peered":"2024-04-21T08:42:16.642906+0000","last_unstale":"2024-04-21T08:43:10.786986+0000","last_undegraded":"2024-04-21T08:43:10.786986+0000","last_fullsized":"2024-04-21T08:43:10.786986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:58.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789827+0000","last_change":"2024-04-21T08:42:17.545185+0000","last_active":"2024-04-21T08:43:10.789827+0000","last_peered":"2024-04-21T08:43:10.789827+0000","last_clean":"2024-04-21T08:43:10.789827+0000","last_became_active":"2024-04-21T08:42:17.544757+0000","last_became_peered":"2024-04-21T08:42:17.544757+0000","last_unstale":"2024-04-21T08:43:10.789827+0000","last_undegraded":"2024-04-21T08:43:10.789827+0000","last_fullsized":"2024-04-21T08:43:10.789827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:36:18.968041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982012+0000","last_change":"2024-04-21T08:42:21.662541+0000","last_active":"2024-04-21T08:43:10.982012+0000","last_peered":"2024-04-21T08:43:10.982012+0000","last_clean":"2024-04-21T08:43:10.982012+0000","last_became_active":"2024-04-21T08:42:21.662144+0000","last_became_peered":"2024-04-21T08:42:21.662144+0000","last_unstale":"2024-04-21T08:43:10.982012+0000","last_undegraded":"2024-04-21T08:43:10.982012+0000","last_fullsized":"2024-04-21T08:43:10.982012+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:04.347722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981989+0000","last_change":"2024-04-21T08:42:16.638601+0000","last_active":"2024-04-21T08:43:10.981989+0000","last_peered":"2024-04-21T08:43:10.981989+0000","last_clean":"2024-04-21T08:43:10.981989+0000","last_became_active":"2024-04-21T08:42:16.638211+0000","last_became_peered":"2024-04-21T08:42:16.638211+0000","last_unstale":"2024-04-21T08:43:10.981989+0000","last_undegraded":"2024-04-21T08:43:10.981989+0000","last_fullsized":"2024-04-21T08:43:10.981989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:02.633086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790435+0000","last_change":"2024-04-21T08:42:17.545495+0000","last_active":"2024-04-21T08:43:10.790435+0000","last_peered":"2024-04-21T08:43:10.790435+0000","last_clean":"2024-04-21T08:43:10.790435+0000","last_became_active":"2024-04-21T08:42:17.545159+0000","last_became_peered":"2024-04-21T08:42:17.545159+0000","last_unstale":"2024-04-21T08:43:10.790435+0000","last_undegraded":"2024-04-21T08:43:10.790435+0000","last_fullsized":"2024-04-21T08:43:10.790435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:27.150277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790207+0000","last_change":"2024-04-21T08:42:21.656890+0000","last_active":"2024-04-21T08:43:10.790207+0000","last_peered":"2024-04-21T08:43:10.790207+0000","last_clean":"2024-04-21T08:43:10.790207+0000","last_became_active":"2024-04-21T08:42:21.656475+0000","last_became_peered":"2024-04-21T08:42:21.656475+0000","last_unstale":"2024-04-21T08:43:10.790207+0000","last_undegraded":"2024-04-21T08:43:10.790207+0000","last_fullsized":"2024-04-21T08:43:10.790207+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:46.138591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791054+0000","last_change":"2024-04-21T08:42:16.638244+0000","last_active":"2024-04-21T08:43:10.791054+0000","last_peered":"2024-04-21T08:43:10.791054+0000","last_clean":"2024-04-21T08:43:10.791054+0000","last_became_active":"2024-04-21T08:42:16.637883+0000","last_became_peered":"2024-04-21T08:42:16.637883+0000","last_unstale":"2024-04-21T08:43:10.791054+0000","last_undegraded":"2024-04-21T08:43:10.791054+0000","last_fullsized":"2024-04-21T08:43:10.791054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:20.888301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789878+0000","last_change":"2024-04-21T08:42:17.548032+0000","last_active":"2024-04-21T08:43:10.789878+0000","last_peered":"2024-04-21T08:43:10.789878+0000","last_clean":"2024-04-21T08:43:10.789878+0000","last_became_active":"2024-04-21T08:42:17.547660+0000","last_became_peered":"2024-04-21T08:42:17.547660+0000","last_unstale":"2024-04-21T08:43:10.789878+0000","last_undegraded":"2024-04-21T08:43:10.789878+0000","last_fullsized":"2024-04-21T08:43:10.789878+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:14.710759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790108+0000","last_change":"2024-04-21T08:42:21.659286+0000","last_active":"2024-04-21T08:43:10.790108+0000","last_peered":"2024-04-21T08:43:10.790108+0000","last_clean":"2024-04-21T08:43:10.790108+0000","last_became_active":"2024-04-21T08:42:21.658844+0000","last_became_peered":"2024-04-21T08:42:21.658844+0000","last_unstale":"2024-04-21T08:43:10.790108+0000","last_undegraded":"2024-04-21T08:43:10.790108+0000","last_fullsized":"2024-04-21T08:43:10.790108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:28.742646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.791168+0000","last_change":"2024-04-21T08:42:16.643354+0000","last_active":"2024-04-21T08:43:10.791168+0000","last_peered":"2024-04-21T08:43:10.791168+0000","last_clean":"2024-04-21T08:43:10.791168+0000","last_became_active":"2024-04-21T08:42:16.642967+0000","last_became_peered":"2024-04-21T08:42:16.642967+0000","last_unstale":"2024-04-21T08:43:10.791168+0000","last_undegraded":"2024-04-21T08:43:10.791168+0000","last_fullsized":"2024-04-21T08:43:10.791168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:06:03.495181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790284+0000","last_change":"2024-04-21T08:42:17.539551+0000","last_active":"2024-04-21T08:43:10.790284+0000","last_peered":"2024-04-21T08:43:10.790284+0000","last_clean":"2024-04-21T08:43:10.790284+0000","last_became_active":"2024-04-21T08:42:17.539202+0000","last_became_peered":"2024-04-21T08:42:17.539202+0000","last_unstale":"2024-04-21T08:43:10.790284+0000","last_undegraded":"2024-04-21T08:43:10.790284+0000","last_fullsized":"2024-04-21T08:43:10.790284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:37.109421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788958+0000","last_change":"2024-04-21T08:42:17.544582+0000","last_active":"2024-04-21T08:43:10.788958+0000","last_peered":"2024-04-21T08:43:10.788958+0000","last_clean":"2024-04-21T08:43:10.788958+0000","last_became_active":"2024-04-21T08:42:17.543800+0000","last_became_peered":"2024-04-21T08:42:17.543800+0000","last_unstale":"2024-04-21T08:43:10.788958+0000","last_undegraded":"2024-04-21T08:43:10.788958+0000","last_fullsized":"2024-04-21T08:43:10.788958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:44:04.107207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981354+0000","last_change":"2024-04-21T08:42:17.543788+0000","last_active":"2024-04-21T08:43:10.981354+0000","last_peered":"2024-04-21T08:43:10.981354+0000","last_clean":"2024-04-21T08:43:10.981354+0000","last_became_active":"2024-04-21T08:42:17.540939+0000","last_became_peered":"2024-04-21T08:42:17.540939+0000","last_unstale":"2024-04-21T08:43:10.981354+0000","last_undegraded":"2024-04-21T08:43:10.981354+0000","last_fullsized":"2024-04-21T08:43:10.981354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:59.275098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981292+0000","last_change":"2024-04-21T08:42:17.540605+0000","last_active":"2024-04-21T08:43:10.981292+0000","last_peered":"2024-04-21T08:43:10.981292+0000","last_clean":"2024-04-21T08:43:10.981292+0000","last_became_active":"2024-04-21T08:42:17.540073+0000","last_became_peered":"2024-04-21T08:42:17.540073+0000","last_unstale":"2024-04-21T08:43:10.981292+0000","last_undegraded":"2024-04-21T08:43:10.981292+0000","last_fullsized":"2024-04-21T08:43:10.981292+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:02:29.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790444+0000","last_change":"2024-04-21T08:42:17.551608+0000","last_active":"2024-04-21T08:43:10.790444+0000","last_peered":"2024-04-21T08:43:10.790444+0000","last_clean":"2024-04-21T08:43:10.790444+0000","last_became_active":"2024-04-21T08:42:17.551117+0000","last_became_peered":"2024-04-21T08:42:17.551117+0000","last_unstale":"2024-04-21T08:43:10.790444+0000","last_undegraded":"2024-04-21T08:43:10.790444+0000","last_fullsized":"2024-04-21T08:43:10.790444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:18.391928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790597+0000","last_change":"2024-04-21T08:42:17.536509+0000","last_active":"2024-04-21T08:43:10.790597+0000","last_peered":"2024-04-21T08:43:10.790597+0000","last_clean":"2024-04-21T08:43:10.790597+0000","last_became_active":"2024-04-21T08:42:17.536146+0000","last_became_peered":"2024-04-21T08:42:17.536146+0000","last_unstale":"2024-04-21T08:43:10.790597+0000","last_undegraded":"2024-04-21T08:43:10.790597+0000","last_fullsized":"2024-04-21T08:43:10.790597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:29.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788710+0000","last_change":"2024-04-21T08:42:17.543786+0000","last_active":"2024-04-21T08:43:10.788710+0000","last_peered":"2024-04-21T08:43:10.788710+0000","last_clean":"2024-04-21T08:43:10.788710+0000","last_became_active":"2024-04-21T08:42:17.543435+0000","last_became_peered":"2024-04-21T08:42:17.543435+0000","last_unstale":"2024-04-21T08:43:10.788710+0000","last_undegraded":"2024-04-21T08:43:10.788710+0000","last_fullsized":"2024-04-21T08:43:10.788710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.981773+0000","last_change":"2024-04-21T08:42:17.541611+0000","last_active":"2024-04-21T08:43:10.981773+0000","last_peered":"2024-04-21T08:43:10.981773+0000","last_clean":"2024-04-21T08:43:10.981773+0000","last_became_active":"2024-04-21T08:42:17.541259+0000","last_became_peered":"2024-04-21T08:42:17.541259+0000","last_unstale":"2024-04-21T08:43:10.981773+0000","last_undegraded":"2024-04-21T08:43:10.981773+0000","last_fullsized":"2024-04-21T08:43:10.981773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:20.900629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789837+0000","last_change":"2024-04-21T08:42:17.541964+0000","last_active":"2024-04-21T08:43:10.789837+0000","last_peered":"2024-04-21T08:43:10.789837+0000","last_clean":"2024-04-21T08:43:10.789837+0000","last_became_active":"2024-04-21T08:42:17.541555+0000","last_became_peered":"2024-04-21T08:42:17.541555+0000","last_unstale":"2024-04-21T08:43:10.789837+0000","last_undegraded":"2024-04-21T08:43:10.789837+0000","last_fullsized":"2024-04-21T08:43:10.789837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:32:47.421452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790774+0000","last_change":"2024-04-21T08:42:17.545526+0000","last_active":"2024-04-21T08:43:10.790774+0000","last_peered":"2024-04-21T08:43:10.790774+0000","last_clean":"2024-04-21T08:43:10.790774+0000","last_became_active":"2024-04-21T08:42:17.545135+0000","last_became_peered":"2024-04-21T08:42:17.545135+0000","last_unstale":"2024-04-21T08:43:10.790774+0000","last_undegraded":"2024-04-21T08:43:10.790774+0000","last_fullsized":"2024-04-21T08:43:10.790774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:37:58.038459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.789502+0000","last_change":"2024-04-21T08:42:17.547063+0000","last_active":"2024-04-21T08:43:10.789502+0000","last_peered":"2024-04-21T08:43:10.789502+0000","last_clean":"2024-04-21T08:43:10.789502+0000","last_became_active":"2024-04-21T08:42:17.546689+0000","last_became_peered":"2024-04-21T08:42:17.546689+0000","last_unstale":"2024-04-21T08:43:10.789502+0000","last_undegraded":"2024-04-21T08:43:10.789502+0000","last_fullsized":"2024-04-21T08:43:10.789502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:26.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.982688+0000","last_change":"2024-04-21T08:42:17.542545+0000","last_active":"2024-04-21T08:43:10.982688+0000","last_peered":"2024-04-21T08:43:10.982688+0000","last_clean":"2024-04-21T08:43:10.982688+0000","last_became_active":"2024-04-21T08:42:17.542147+0000","last_became_peered":"2024-04-21T08:42:17.542147+0000","last_unstale":"2024-04-21T08:43:10.982688+0000","last_undegraded":"2024-04-21T08:43:10.982688+0000","last_fullsized":"2024-04-21T08:43:10.982688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:15.064110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.790177+0000","last_change":"2024-04-21T08:42:17.541262+0000","last_active":"2024-04-21T08:43:10.790177+0000","last_peered":"2024-04-21T08:43:10.790177+0000","last_clean":"2024-04-21T08:43:10.790177+0000","last_became_active":"2024-04-21T08:42:17.540817+0000","last_became_peered":"2024-04-21T08:42:17.540817+0000","last_unstale":"2024-04-21T08:43:10.790177+0000","last_undegraded":"2024-04-21T08:43:10.790177+0000","last_fullsized":"2024-04-21T08:43:10.790177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:22:59.597680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T08:43:10.788435+0000","last_change":"2024-04-21T08:42:17.551109+0000","last_active":"2024-04-21T08:43:10.788435+0000","last_peered":"2024-04-21T08:43:10.788435+0000","last_clean":"2024-04-21T08:43:10.788435+0000","last_became_active":"2024-04-21T08:42:17.550104+0000","last_became_peered":"2024-04-21T08:42:17.550104+0000","last_unstale":"2024-04-21T08:43:10.788435+0000","last_undegraded":"2024-04-21T08:43:10.788435+0000","last_fullsized":"2024-04-21T08:43:10.788435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:22:19.361752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281211,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64154,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":163,"num_read_kb":146,"num_write":79,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":192462,"data_compressed":4896,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":503292,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612647,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36444,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708708,"statfs":{"total":95995035648,"available":95957716992,"internally_reserved":0,"allocated":1126400,"data_stored":1153114,"data_compressed":58127,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776173,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":507970,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037810,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":503896,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299446,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":1098338,"data_compressed":56495,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":995328,"data_stored":499196,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1064960,"data_stored":562552,"data_compressed":22361,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":999424,"data_stored":499286,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":562586,"data_compressed":22361,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":499300,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1003520,"data_stored":503314,"data_compressed":20729,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724609,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":1093842,"data_compressed":56495,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34127,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63390,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63550,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:43:59.747 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-21T08:43:59.750 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-21T08:43:59.750 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-21T08:43:59.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph health --format=json 2024-04-21T08:44:00.004 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:00.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:43:59 smithi016 ceph-mon[31259]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:00.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:43:59 smithi077 ceph-mon[31615]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:00.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:43:59 smithi007 ceph-mon[28477]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:01.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:00 smithi016 ceph-mon[31259]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:01.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:00 smithi077 ceph-mon[31615]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:01.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:00 smithi007 ceph-mon[28477]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:01.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.774+0000 7f976e773700 1 -- 172.21.15.7:0/1431142286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9768108a50 msgr2=0x7f9768108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.774+0000 7f976e773700 1 --2- 172.21.15.7:0/1431142286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9768108a50 0x7f9768108e30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9758009990 tx=0x7f9758009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.774+0000 7f976e773700 1 -- 172.21.15.7:0/1431142286 shutdown_connections 2024-04-21T08:44:01.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.774+0000 7f976e773700 1 --2- 172.21.15.7:0/1431142286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9768103450 0x7f976810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.774+0000 7f976e773700 1 --2- 172.21.15.7:0/1431142286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9768108a50 0x7f9768108e30 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.774+0000 7f976e773700 1 --2- 172.21.15.7:0/1431142286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9768102a90 0x7f9768102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.774+0000 7f976e773700 1 -- 172.21.15.7:0/1431142286 >> 172.21.15.7:0/1431142286 conn(0x7f9768079ba0 msgr2=0x7f9768079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.774+0000 7f976e773700 1 -- 172.21.15.7:0/1431142286 shutdown_connections 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.774+0000 7f976e773700 1 -- 172.21.15.7:0/1431142286 wait complete. 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.775+0000 7f976e773700 1 Processor -- start 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.775+0000 7f976e773700 1 -- start start 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.775+0000 7f976e773700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9768102a90 0x7f9768074230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.775+0000 7f976e773700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9768103450 0x7f976806d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.775+0000 7f976e773700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9768108a50 0x7f976806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.775+0000 7f976e773700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9768112c00 con 0x7f9768108a50 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.775+0000 7f976e773700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9768112a80 con 0x7f9768102a90 2024-04-21T08:44:01.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.775+0000 7f976e773700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9768112d80 con 0x7f9768103450 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.775+0000 7f976cd10700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9768108a50 0x7f976806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.775+0000 7f976cd10700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9768108a50 0x7f976806d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36302/0 (socket says 172.21.15.7:36302) 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.775+0000 7f976cd10700 1 -- 172.21.15.7:0/1783977247 learned_addr learned my addr 172.21.15.7:0/1783977247 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f9767fff700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9768102a90 0x7f9768074230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f97677fe700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9768103450 0x7f976806d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f976cd10700 1 -- 172.21.15.7:0/1783977247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9768103450 msgr2=0x7f976806d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f976cd10700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9768103450 0x7f976806d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f976cd10700 1 -- 172.21.15.7:0/1783977247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9768102a90 msgr2=0x7f9768074230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f976cd10700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9768102a90 0x7f9768074230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f976cd10700 1 -- 172.21.15.7:0/1783977247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976806de90 con 0x7f9768108a50 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f976cd10700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9768108a50 0x7f976806d850 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9760007ee0 tx=0x7f976000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f97657fa700 1 -- 172.21.15.7:0/1783977247 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760004bb0 con 0x7f9768108a50 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f97657fa700 1 -- 172.21.15.7:0/1783977247 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9760004d50 con 0x7f9768108a50 2024-04-21T08:44:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f976e773700 1 -- 172.21.15.7:0/1783977247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9768071d60 con 0x7f9768108a50 2024-04-21T08:44:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f97657fa700 1 -- 172.21.15.7:0/1783977247 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760005920 con 0x7f9768108a50 2024-04-21T08:44:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.776+0000 7f976e773700 1 -- 172.21.15.7:0/1783977247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97680722a0 con 0x7f9768108a50 2024-04-21T08:44:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.777+0000 7f976e773700 1 -- 172.21.15.7:0/1783977247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9768104180 con 0x7f9768108a50 2024-04-21T08:44:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.779+0000 7f97657fa700 1 -- 172.21.15.7:0/1783977247 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f976001a020 con 0x7f9768108a50 2024-04-21T08:44:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.780+0000 7f97657fa700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f97500a3cd0 0x7f97500a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.780+0000 7f9767fff700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f97500a3cd0 0x7f97500a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.780+0000 7f97657fa700 1 -- 172.21.15.7:0/1783977247 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f97600cff50 con 0x7f9768108a50 2024-04-21T08:44:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.780+0000 7f9767fff700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f97500a3cd0 0x7f97500a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9758006010 tx=0x7f975800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:01.785+0000 7f97657fa700 1 -- 172.21.15.7:0/1783977247 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9760010070 con 0x7f9768108a50 2024-04-21T08:44:02.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.044+0000 7f976e773700 1 -- 172.21.15.7:0/1783977247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f976806e060 con 0x7f9768108a50 2024-04-21T08:44:02.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.044+0000 7f97657fa700 1 -- 172.21.15.7:0/1783977247 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f976009c410 con 0x7f9768108a50 2024-04-21T08:44:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:02.044 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-21T08:44:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.046+0000 7f976e773700 1 -- 172.21.15.7:0/1783977247 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f97500a3cd0 msgr2=0x7f97500a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.046+0000 7f976e773700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f97500a3cd0 0x7f97500a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9758006010 tx=0x7f975800b280 comp rx=0 tx=0).stop 2024-04-21T08:44:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.046+0000 7f976e773700 1 -- 172.21.15.7:0/1783977247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9768108a50 msgr2=0x7f976806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.046+0000 7f976e773700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9768108a50 0x7f976806d850 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9760007ee0 tx=0x7f976000d3b0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.046+0000 7f976e773700 1 -- 172.21.15.7:0/1783977247 shutdown_connections 2024-04-21T08:44:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.046+0000 7f976e773700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f97500a3cd0 0x7f97500a6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.047+0000 7f976e773700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9768108a50 0x7f976806d850 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.047+0000 7f976e773700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9768102a90 0x7f9768074230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.047+0000 7f976e773700 1 --2- 172.21.15.7:0/1783977247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9768103450 0x7f976806d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.047+0000 7f976e773700 1 -- 172.21.15.7:0/1783977247 >> 172.21.15.7:0/1783977247 conn(0x7f9768079ba0 msgr2=0x7f97681018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:02.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.047+0000 7f976e773700 1 -- 172.21.15.7:0/1783977247 shutdown_connections 2024-04-21T08:44:02.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.047+0000 7f976e773700 1 -- 172.21.15.7:0/1783977247 wait complete. 2024-04-21T08:44:02.804 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-21T08:44:02.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:02.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.912+0000 7fce70231640 1 -- 172.21.15.7:0/1810240536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68107c60 msgr2=0x7fce6810a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:02.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.912+0000 7fce70231640 1 --2- 172.21.15.7:0/1810240536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68107c60 0x7fce6810a0b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fce58009900 tx=0x7fce5802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:02.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.912+0000 7fce70231640 1 -- 172.21.15.7:0/1810240536 shutdown_connections 2024-04-21T08:44:02.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.912+0000 7fce70231640 1 --2- 172.21.15.7:0/1810240536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68107c60 0x7fce6810a0b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.912+0000 7fce70231640 1 --2- 172.21.15.7:0/1810240536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce68105330 0x7fce68107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.912+0000 7fce70231640 1 --2- 172.21.15.7:0/1810240536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce68102a00 0x7fce68104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.912+0000 7fce70231640 1 -- 172.21.15.7:0/1810240536 >> 172.21.15.7:0/1810240536 conn(0x7fce680fc3f0 msgr2=0x7fce680fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:02.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.912+0000 7fce70231640 1 -- 172.21.15.7:0/1810240536 shutdown_connections 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.912+0000 7fce70231640 1 -- 172.21.15.7:0/1810240536 wait complete. 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce70231640 1 Processor -- start 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce70231640 1 -- start start 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce70231640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce68102a00 0x7fce681980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce70231640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce68105330 0x7fce68198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce70231640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68107c60 0x7fce6819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce70231640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fce6810dac0 con 0x7fce68107c60 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce70231640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce6810d940 con 0x7fce68102a00 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce70231640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fce6810dc40 con 0x7fce68105330 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce6e7ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68107c60 0x7fce6819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce6e7ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68107c60 0x7fce6819f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36322/0 (socket says 172.21.15.7:36322) 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce6dfcd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce68102a00 0x7fce681980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:02.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce6e7ce640 1 -- 172.21.15.7:0/355521053 learned_addr learned my addr 172.21.15.7:0/355521053 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce6d7cc640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce68105330 0x7fce68198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce6e7ce640 1 -- 172.21.15.7:0/355521053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce68105330 msgr2=0x7fce68198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce6e7ce640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce68105330 0x7fce68198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce6e7ce640 1 -- 172.21.15.7:0/355521053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce68102a00 msgr2=0x7fce681980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce6e7ce640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce68102a00 0x7fce681980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce6e7ce640 1 -- 172.21.15.7:0/355521053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6819fcf0 con 0x7fce68107c60 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.913+0000 7fce6e7ce640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68107c60 0x7fce6819f680 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fce58009230 tx=0x7fce580040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.914+0000 7fce66ffd640 1 -- 172.21.15.7:0/355521053 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58043020 con 0x7fce68107c60 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.914+0000 7fce66ffd640 1 -- 172.21.15.7:0/355521053 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce580044c0 con 0x7fce68107c60 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.914+0000 7fce70231640 1 -- 172.21.15.7:0/355521053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce6819ff80 con 0x7fce68107c60 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.914+0000 7fce66ffd640 1 -- 172.21.15.7:0/355521053 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58003bc0 con 0x7fce68107c60 2024-04-21T08:44:02.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.914+0000 7fce70231640 1 -- 172.21.15.7:0/355521053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce681a0490 con 0x7fce68107c60 2024-04-21T08:44:02.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.915+0000 7fce70231640 1 -- 172.21.15.7:0/355521053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce680fdee0 con 0x7fce68107c60 2024-04-21T08:44:02.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.916+0000 7fce66ffd640 1 -- 172.21.15.7:0/355521053 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fce58046020 con 0x7fce68107c60 2024-04-21T08:44:02.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.916+0000 7fce66ffd640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fce500a3cc0 0x7fce500a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:02.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.917+0000 7fce6dfcd640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fce500a3cc0 0x7fce500a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:02.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.917+0000 7fce66ffd640 1 -- 172.21.15.7:0/355521053 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fce580f2e70 con 0x7fce68107c60 2024-04-21T08:44:02.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.917+0000 7fce6dfcd640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fce500a3cc0 0x7fce500a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fce60007ec0 tx=0x7fce600075b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:02.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:02.920+0000 7fce66ffd640 1 -- 172.21.15.7:0/355521053 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce5804d020 con 0x7fce68107c60 2024-04-21T08:44:03.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:02 smithi016 ceph-mon[31259]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:03.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:02 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1783977247' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:44:03.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:02 smithi077 ceph-mon[31615]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:03.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:02 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1783977247' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:44:03.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.107+0000 7fce70231640 1 -- 172.21.15.7:0/355521053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce68062bf0 con 0x7fce68107c60 2024-04-21T08:44:03.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.108+0000 7fce66ffd640 1 -- 172.21.15.7:0/355521053 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24432 (secure 0 0 0) 0x7fce580bf4a0 con 0x7fce68107c60 2024-04-21T08:44:03.108 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:03.109 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":92,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:43:10.778941+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:03.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.110+0000 7fce70231640 1 -- 172.21.15.7:0/355521053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fce500a3cc0 msgr2=0x7fce500a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.110+0000 7fce70231640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fce500a3cc0 0x7fce500a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fce60007ec0 tx=0x7fce600075b0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.110+0000 7fce70231640 1 -- 172.21.15.7:0/355521053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68107c60 msgr2=0x7fce6819f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.110+0000 7fce70231640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68107c60 0x7fce6819f680 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fce58009230 tx=0x7fce580040c0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.110+0000 7fce70231640 1 -- 172.21.15.7:0/355521053 shutdown_connections 2024-04-21T08:44:03.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.110+0000 7fce70231640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fce500a3cc0 0x7fce500a6180 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.110+0000 7fce70231640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce68107c60 0x7fce6819f680 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.110+0000 7fce70231640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce68105330 0x7fce68198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.110+0000 7fce70231640 1 --2- 172.21.15.7:0/355521053 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce68102a00 0x7fce681980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.110+0000 7fce70231640 1 -- 172.21.15.7:0/355521053 >> 172.21.15.7:0/355521053 conn(0x7fce680fc3f0 msgr2=0x7fce68103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:03.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.110+0000 7fce70231640 1 -- 172.21.15.7:0/355521053 shutdown_connections 2024-04-21T08:44:03.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.111+0000 7fce70231640 1 -- 172.21.15.7:0/355521053 wait complete. 2024-04-21T08:44:03.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:02 smithi007 ceph-mon[28477]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:03.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:02 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1783977247' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:44:03.122 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:03.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:03.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.228+0000 7f607e2ba640 1 -- 172.21.15.7:0/4044885351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6078102a20 msgr2=0x7f6078104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.228+0000 7f607e2ba640 1 --2- 172.21.15.7:0/4044885351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6078102a20 0x7f6078104e10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f606c009900 tx=0x7f606c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.228+0000 7f607e2ba640 1 -- 172.21.15.7:0/4044885351 shutdown_connections 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.228+0000 7f607e2ba640 1 --2- 172.21.15.7:0/4044885351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6078107c80 0x7f607810a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.228+0000 7f607e2ba640 1 --2- 172.21.15.7:0/4044885351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6078105350 0x7f6078107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.228+0000 7f607e2ba640 1 --2- 172.21.15.7:0/4044885351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6078102a20 0x7f6078104e10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.228+0000 7f607e2ba640 1 -- 172.21.15.7:0/4044885351 >> 172.21.15.7:0/4044885351 conn(0x7f60780fc430 msgr2=0x7f60780fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.228+0000 7f607e2ba640 1 -- 172.21.15.7:0/4044885351 shutdown_connections 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.228+0000 7f607e2ba640 1 -- 172.21.15.7:0/4044885351 wait complete. 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.228+0000 7f607e2ba640 1 Processor -- start 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.228+0000 7f607e2ba640 1 -- start start 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f607e2ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6078102a20 0x7f6078198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f607e2ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6078105350 0x7f6078198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f607e2ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6078107c80 0x7f607819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f607e2ba640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f607810db30 con 0x7f6078102a20 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f607e2ba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f607810d9b0 con 0x7f6078105350 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f607e2ba640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f607810dcb0 con 0x7f6078107c80 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f6077fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6078102a20 0x7f6078198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f6077fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6078102a20 0x7f6078198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36344/0 (socket says 172.21.15.7:36344) 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f6077fff640 1 -- 172.21.15.7:0/3950258573 learned_addr learned my addr 172.21.15.7:0/3950258573 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f60777fe640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6078105350 0x7f6078198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f607c857640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6078107c80 0x7f607819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f6077fff640 1 -- 172.21.15.7:0/3950258573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6078107c80 msgr2=0x7f607819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f6077fff640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6078107c80 0x7f607819f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f6077fff640 1 -- 172.21.15.7:0/3950258573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6078105350 msgr2=0x7f6078198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f6077fff640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6078105350 0x7f6078198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f6077fff640 1 -- 172.21.15.7:0/3950258573 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607819fe50 con 0x7f6078102a20 2024-04-21T08:44:03.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f6077fff640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6078102a20 0x7f6078198230 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f606c000c00 tx=0x7f606c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:03.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f60757fa640 1 -- 172.21.15.7:0/3950258573 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606c043020 con 0x7f6078102a20 2024-04-21T08:44:03.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f60757fa640 1 -- 172.21.15.7:0/3950258573 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f606c004aa0 con 0x7f6078102a20 2024-04-21T08:44:03.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f607e2ba640 1 -- 172.21.15.7:0/3950258573 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60781a00e0 con 0x7f6078102a20 2024-04-21T08:44:03.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.229+0000 7f60757fa640 1 -- 172.21.15.7:0/3950258573 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606c003d70 con 0x7f6078102a20 2024-04-21T08:44:03.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.230+0000 7f607e2ba640 1 -- 172.21.15.7:0/3950258573 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60781a0680 con 0x7f6078102a20 2024-04-21T08:44:03.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.231+0000 7f607e2ba640 1 -- 172.21.15.7:0/3950258573 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60780fdf20 con 0x7f6078102a20 2024-04-21T08:44:03.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.232+0000 7f60757fa640 1 -- 172.21.15.7:0/3950258573 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f606c046020 con 0x7f6078102a20 2024-04-21T08:44:03.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.232+0000 7f60757fa640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60600a3cc0 0x7f60600a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.233+0000 7f60757fa640 1 -- 172.21.15.7:0/3950258573 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f606c0f2ad0 con 0x7f6078102a20 2024-04-21T08:44:03.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.233+0000 7f60777fe640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60600a3cc0 0x7f60600a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.233+0000 7f60777fe640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60600a3cc0 0x7f60600a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6078199740 tx=0x7f606800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:03.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.237+0000 7f60757fa640 1 -- 172.21.15.7:0/3950258573 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f606c04d020 con 0x7f6078102a20 2024-04-21T08:44:03.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.423+0000 7f607e2ba640 1 -- 172.21.15.7:0/3950258573 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6078062b50 con 0x7f6078102a20 2024-04-21T08:44:03.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.423+0000 7f60757fa640 1 -- 172.21.15.7:0/3950258573 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f606c0bf100 con 0x7f6078102a20 2024-04-21T08:44:03.423 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:44:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.425+0000 7f607e2ba640 1 -- 172.21.15.7:0/3950258573 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60600a3cc0 msgr2=0x7f60600a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.425+0000 7f607e2ba640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60600a3cc0 0x7f60600a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6078199740 tx=0x7f606800b390 comp rx=0 tx=0).stop 2024-04-21T08:44:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.425+0000 7f607e2ba640 1 -- 172.21.15.7:0/3950258573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6078102a20 msgr2=0x7f6078198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.425+0000 7f607e2ba640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6078102a20 0x7f6078198230 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f606c000c00 tx=0x7f606c0042f0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.425+0000 7f607e2ba640 1 -- 172.21.15.7:0/3950258573 shutdown_connections 2024-04-21T08:44:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.425+0000 7f607e2ba640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f60600a3cc0 0x7f60600a6180 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.425+0000 7f607e2ba640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6078107c80 0x7f607819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.425+0000 7f607e2ba640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6078105350 0x7f6078198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.425+0000 7f607e2ba640 1 --2- 172.21.15.7:0/3950258573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6078102a20 0x7f6078198230 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.425+0000 7f607e2ba640 1 -- 172.21.15.7:0/3950258573 >> 172.21.15.7:0/3950258573 conn(0x7f60780fc430 msgr2=0x7f60781034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.426+0000 7f607e2ba640 1 -- 172.21.15.7:0/3950258573 shutdown_connections 2024-04-21T08:44:03.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.426+0000 7f607e2ba640 1 -- 172.21.15.7:0/3950258573 wait complete. 2024-04-21T08:44:03.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:03.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.546+0000 7f30eb70d640 1 -- 172.21.15.7:0/2219395730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e4105350 msgr2=0x7f30e4107740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.546+0000 7f30eb70d640 1 --2- 172.21.15.7:0/2219395730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e4105350 0x7f30e4107740 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009900 tx=0x7f30d402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:03.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.546+0000 7f30eb70d640 1 -- 172.21.15.7:0/2219395730 shutdown_connections 2024-04-21T08:44:03.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.546+0000 7f30eb70d640 1 --2- 172.21.15.7:0/2219395730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e4107c80 0x7f30e410a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.546+0000 7f30eb70d640 1 --2- 172.21.15.7:0/2219395730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e4105350 0x7f30e4107740 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.546+0000 7f30eb70d640 1 --2- 172.21.15.7:0/2219395730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e4102a20 0x7f30e4104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.546+0000 7f30eb70d640 1 -- 172.21.15.7:0/2219395730 >> 172.21.15.7:0/2219395730 conn(0x7f30e40fc430 msgr2=0x7f30e40fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.546+0000 7f30eb70d640 1 -- 172.21.15.7:0/2219395730 shutdown_connections 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.546+0000 7f30eb70d640 1 -- 172.21.15.7:0/2219395730 wait complete. 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.546+0000 7f30eb70d640 1 Processor -- start 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30eb70d640 1 -- start start 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30eb70d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e4102a20 0x7f30e4102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30eb70d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e4105350 0x7f30e41009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30eb70d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e4107c80 0x7f30e4100f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30eb70d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f30e410daa0 con 0x7f30e4107c80 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30eb70d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f30e410d920 con 0x7f30e4102a20 2024-04-21T08:44:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30eb70d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f30e410dc20 con 0x7f30e4105350 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30e9caa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e4107c80 0x7f30e4100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30e9caa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e4107c80 0x7f30e4100f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36368/0 (socket says 172.21.15.7:36368) 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30e9caa640 1 -- 172.21.15.7:0/2338347163 learned_addr learned my addr 172.21.15.7:0/2338347163 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30e94a9640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e4102a20 0x7f30e4102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30e8ca8640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e4105350 0x7f30e41009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30e9caa640 1 -- 172.21.15.7:0/2338347163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e4105350 msgr2=0x7f30e41009d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30e9caa640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e4105350 0x7f30e41009d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30e9caa640 1 -- 172.21.15.7:0/2338347163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e4102a20 msgr2=0x7f30e4102390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30e9caa640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e4102a20 0x7f30e4102390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30e9caa640 1 -- 172.21.15.7:0/2338347163 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e41017a0 con 0x7f30e4107c80 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.547+0000 7f30e9caa640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e4107c80 0x7f30e4100f10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00cad0 tx=0x7f30dc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.548+0000 7f30da7fc640 1 -- 172.21.15.7:0/2338347163 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc00ae20 con 0x7f30e4107c80 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.548+0000 7f30da7fc640 1 -- 172.21.15.7:0/2338347163 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30dc005020 con 0x7f30e4107c80 2024-04-21T08:44:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.548+0000 7f30eb70d640 1 -- 172.21.15.7:0/2338347163 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e41a2330 con 0x7f30e4107c80 2024-04-21T08:44:03.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.548+0000 7f30da7fc640 1 -- 172.21.15.7:0/2338347163 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc00f6b0 con 0x7f30e4107c80 2024-04-21T08:44:03.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.548+0000 7f30eb70d640 1 -- 172.21.15.7:0/2338347163 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e41a2800 con 0x7f30e4107c80 2024-04-21T08:44:03.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.549+0000 7f30eb70d640 1 -- 172.21.15.7:0/2338347163 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e4069250 con 0x7f30e4107c80 2024-04-21T08:44:03.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.550+0000 7f30da7fc640 1 -- 172.21.15.7:0/2338347163 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f30dc0051d0 con 0x7f30e4107c80 2024-04-21T08:44:03.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.550+0000 7f30da7fc640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30d00a3c40 0x7f30d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.551+0000 7f30e94a9640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30d00a3c40 0x7f30d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.551+0000 7f30da7fc640 1 -- 172.21.15.7:0/2338347163 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f30dc0d0cd0 con 0x7f30e4107c80 2024-04-21T08:44:03.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.551+0000 7f30e94a9640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30d00a3c40 0x7f30d00a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f30e000bbe0 tx=0x7f30e000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:03.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.554+0000 7f30da7fc640 1 -- 172.21.15.7:0/2338347163 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f30dc014030 con 0x7f30e4107c80 2024-04-21T08:44:03.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.743+0000 7f30eb70d640 1 -- 172.21.15.7:0/2338347163 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f30e4062b50 con 0x7f30e4107c80 2024-04-21T08:44:03.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.744+0000 7f30da7fc640 1 -- 172.21.15.7:0/2338347163 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f30dc09cba0 con 0x7f30e4107c80 2024-04-21T08:44:03.744 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:03.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 -- 172.21.15.7:0/2338347163 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30d00a3c40 msgr2=0x7f30d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30d00a3c40 0x7f30d00a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f30e000bbe0 tx=0x7f30e000b390 comp rx=0 tx=0).stop 2024-04-21T08:44:03.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 -- 172.21.15.7:0/2338347163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e4107c80 msgr2=0x7f30e4100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e4107c80 0x7f30e4100f10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00cad0 tx=0x7f30dc00cf90 comp rx=0 tx=0).stop 2024-04-21T08:44:03.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 -- 172.21.15.7:0/2338347163 shutdown_connections 2024-04-21T08:44:03.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f30d00a3c40 0x7f30d00a6100 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e4107c80 0x7f30e4100f10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e4105350 0x7f30e41009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 --2- 172.21.15.7:0/2338347163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f30e4102a20 0x7f30e4102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 -- 172.21.15.7:0/2338347163 >> 172.21.15.7:0/2338347163 conn(0x7f30e40fc430 msgr2=0x7f30e4103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:03.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 -- 172.21.15.7:0/2338347163 shutdown_connections 2024-04-21T08:44:03.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.746+0000 7f30eb70d640 1 -- 172.21.15.7:0/2338347163 wait complete. 2024-04-21T08:44:03.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:03.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.862+0000 7f897d1ef640 1 -- 172.21.15.7:0/3648683263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8978103b60 msgr2=0x7f897810a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.862+0000 7f897d1ef640 1 --2- 172.21.15.7:0/3648683263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8978103b60 0x7f897810a3f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8970009ca0 tx=0x7f897002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.862+0000 7f897d1ef640 1 -- 172.21.15.7:0/3648683263 shutdown_connections 2024-04-21T08:44:03.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.862+0000 7f897d1ef640 1 --2- 172.21.15.7:0/3648683263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8978103b60 0x7f897810a3f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.862+0000 7f897d1ef640 1 --2- 172.21.15.7:0/3648683263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89781031a0 0x7f8978103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.862+0000 7f897d1ef640 1 --2- 172.21.15.7:0/3648683263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8978101fb0 0x7f89781023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.862+0000 7f897d1ef640 1 -- 172.21.15.7:0/3648683263 >> 172.21.15.7:0/3648683263 conn(0x7f89780fd520 msgr2=0x7f89780ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:03.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.862+0000 7f897d1ef640 1 -- 172.21.15.7:0/3648683263 shutdown_connections 2024-04-21T08:44:03.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.862+0000 7f897d1ef640 1 -- 172.21.15.7:0/3648683263 wait complete. 2024-04-21T08:44:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.863+0000 7f897d1ef640 1 Processor -- start 2024-04-21T08:44:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.863+0000 7f897d1ef640 1 -- start start 2024-04-21T08:44:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.863+0000 7f897d1ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8978101fb0 0x7f8978198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.863+0000 7f897d1ef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89781031a0 0x7f8978198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897d1ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8978103b60 0x7f897819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897d1ef640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f897810db30 con 0x7f8978103b60 2024-04-21T08:44:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897d1ef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f897810d9b0 con 0x7f8978101fb0 2024-04-21T08:44:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897d1ef640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f897810dcb0 con 0x7f89781031a0 2024-04-21T08:44:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8978103b60 0x7f897819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897759e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8978103b60 0x7f897819f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36384/0 (socket says 172.21.15.7:36384) 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897759e640 1 -- 172.21.15.7:0/461884250 learned_addr learned my addr 172.21.15.7:0/461884250 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897659c640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89781031a0 0x7f8978198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f8976d9d640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8978101fb0 0x7f8978198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897759e640 1 -- 172.21.15.7:0/461884250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89781031a0 msgr2=0x7f8978198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897759e640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89781031a0 0x7f8978198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897759e640 1 -- 172.21.15.7:0/461884250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8978101fb0 msgr2=0x7f8978198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897759e640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8978101fb0 0x7f8978198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897759e640 1 -- 172.21.15.7:0/461884250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897819fe20 con 0x7f8978103b60 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897759e640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8978103b60 0x7f897819f7b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8970032380 tx=0x7f8970035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f895ffff640 1 -- 172.21.15.7:0/461884250 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8970039040 con 0x7f8978103b60 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897d1ef640 1 -- 172.21.15.7:0/461884250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89781a00b0 con 0x7f8978103b60 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f895ffff640 1 -- 172.21.15.7:0/461884250 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8970044070 con 0x7f8978103b60 2024-04-21T08:44:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f895ffff640 1 -- 172.21.15.7:0/461884250 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89700042a0 con 0x7f8978103b60 2024-04-21T08:44:03.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.864+0000 7f897d1ef640 1 -- 172.21.15.7:0/461884250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89781a05f0 con 0x7f8978103b60 2024-04-21T08:44:03.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.865+0000 7f897d1ef640 1 -- 172.21.15.7:0/461884250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89781093a0 con 0x7f8978103b60 2024-04-21T08:44:03.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.866+0000 7f895ffff640 1 -- 172.21.15.7:0/461884250 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8970003a00 con 0x7f8978103b60 2024-04-21T08:44:03.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.867+0000 7f895ffff640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89600a3cc0 0x7f89600a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:03.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.867+0000 7f8976d9d640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89600a3cc0 0x7f89600a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:03.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.867+0000 7f895ffff640 1 -- 172.21.15.7:0/461884250 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f89700f2d80 con 0x7f8978103b60 2024-04-21T08:44:03.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.868+0000 7f8976d9d640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89600a3cc0 0x7f89600a6180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8968005db0 tx=0x7f8968005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:03.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:03.872+0000 7f895ffff640 1 -- 172.21.15.7:0/461884250 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f89700bf3b0 con 0x7f8978103b60 2024-04-21T08:44:04.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:03 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/355521053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:04.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:03 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3950258573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:04.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.056+0000 7f897d1ef640 1 -- 172.21.15.7:0/461884250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8978062b50 con 0x7f8978103b60 2024-04-21T08:44:04.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.057+0000 7f895ffff640 1 -- 172.21.15.7:0/461884250 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f89700becb0 con 0x7f8978103b60 2024-04-21T08:44:04.056 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.058+0000 7f897d1ef640 1 -- 172.21.15.7:0/461884250 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89600a3cc0 msgr2=0x7f89600a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.058+0000 7f897d1ef640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89600a3cc0 0x7f89600a6180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8968005db0 tx=0x7f8968005d40 comp rx=0 tx=0).stop 2024-04-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.059+0000 7f897d1ef640 1 -- 172.21.15.7:0/461884250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8978103b60 msgr2=0x7f897819f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.059+0000 7f897d1ef640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8978103b60 0x7f897819f7b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8970032380 tx=0x7f8970035ce0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.059+0000 7f897d1ef640 1 -- 172.21.15.7:0/461884250 shutdown_connections 2024-04-21T08:44:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.059+0000 7f897d1ef640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f89600a3cc0 0x7f89600a6180 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.059+0000 7f897d1ef640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8978103b60 0x7f897819f7b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.059+0000 7f897d1ef640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89781031a0 0x7f8978198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.059+0000 7f897d1ef640 1 --2- 172.21.15.7:0/461884250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8978101fb0 0x7f8978198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.059+0000 7f897d1ef640 1 -- 172.21.15.7:0/461884250 >> 172.21.15.7:0/461884250 conn(0x7f89780fd520 msgr2=0x7f8978106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.059+0000 7f897d1ef640 1 -- 172.21.15.7:0/461884250 shutdown_connections 2024-04-21T08:44:04.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.059+0000 7f897d1ef640 1 -- 172.21.15.7:0/461884250 wait complete. 2024-04-21T08:44:04.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:03 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/355521053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:04.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:03 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3950258573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:04.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:04.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:03 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/355521053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:04.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:03 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3950258573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.175+0000 7f7e7608c640 1 -- 172.21.15.7:0/3403176603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e70100ea0 msgr2=0x7f7e701012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.175+0000 7f7e7608c640 1 --2- 172.21.15.7:0/3403176603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e70100ea0 0x7f7e701012c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7e60009900 tx=0x7f7e6002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.175+0000 7f7e7608c640 1 -- 172.21.15.7:0/3403176603 shutdown_connections 2024-04-21T08:44:04.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.175+0000 7f7e7608c640 1 --2- 172.21.15.7:0/3403176603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e701029b0 0x7f7e701092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.175+0000 7f7e7608c640 1 --2- 172.21.15.7:0/3403176603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e70101ff0 0x7f7e70102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.175+0000 7f7e7608c640 1 --2- 172.21.15.7:0/3403176603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e70100ea0 0x7f7e701012c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.175+0000 7f7e7608c640 1 -- 172.21.15.7:0/3403176603 >> 172.21.15.7:0/3403176603 conn(0x7f7e700fc430 msgr2=0x7f7e700fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.175+0000 7f7e7608c640 1 -- 172.21.15.7:0/3403176603 shutdown_connections 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.175+0000 7f7e7608c640 1 -- 172.21.15.7:0/3403176603 wait complete. 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.175+0000 7f7e7608c640 1 Processor -- start 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.175+0000 7f7e7608c640 1 -- start start 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e7608c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e70100ea0 0x7f7e70198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e7608c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e70101ff0 0x7f7e701986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e7608c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e701029b0 0x7f7e7019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e7608c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7010c9d0 con 0x7f7e701029b0 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e7608c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7010c850 con 0x7f7e70101ff0 2024-04-21T08:44:04.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e7608c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7010cb50 con 0x7f7e70100ea0 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e701029b0 0x7f7e7019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e701029b0 0x7f7e7019f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36408/0 (socket says 172.21.15.7:36408) 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6ffff640 1 -- 172.21.15.7:0/1088191230 learned_addr learned my addr 172.21.15.7:0/1088191230 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6f7fe640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e70100ea0 0x7f7e70198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6effd640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e70101ff0 0x7f7e701986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6ffff640 1 -- 172.21.15.7:0/1088191230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e70100ea0 msgr2=0x7f7e70198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6ffff640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e70100ea0 0x7f7e70198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6ffff640 1 -- 172.21.15.7:0/1088191230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e70101ff0 msgr2=0x7f7e701986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6ffff640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e70101ff0 0x7f7e701986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6ffff640 1 -- 172.21.15.7:0/1088191230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7019fda0 con 0x7f7e701029b0 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6effd640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e70101ff0 0x7f7e701986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6ffff640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e701029b0 0x7f7e7019f730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800cad0 tx=0x7f7e6800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:04.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6cff9640 1 -- 172.21.15.7:0/1088191230 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6800ae20 con 0x7f7e701029b0 2024-04-21T08:44:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e7608c640 1 -- 172.21.15.7:0/1088191230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e701a0090 con 0x7f7e701029b0 2024-04-21T08:44:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6cff9640 1 -- 172.21.15.7:0/1088191230 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e68005020 con 0x7f7e701029b0 2024-04-21T08:44:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e6cff9640 1 -- 172.21.15.7:0/1088191230 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6800f6b0 con 0x7f7e701029b0 2024-04-21T08:44:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.176+0000 7f7e7608c640 1 -- 172.21.15.7:0/1088191230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e701a0590 con 0x7f7e701029b0 2024-04-21T08:44:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.177+0000 7f7e7608c640 1 -- 172.21.15.7:0/1088191230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e701a0850 con 0x7f7e701029b0 2024-04-21T08:44:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.178+0000 7f7e6cff9640 1 -- 172.21.15.7:0/1088191230 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f7e680051d0 con 0x7f7e701029b0 2024-04-21T08:44:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.179+0000 7f7e6cff9640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7e580a3c40 0x7f7e580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.179+0000 7f7e6f7fe640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7e580a3c40 0x7f7e580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.179+0000 7f7e6cff9640 1 -- 172.21.15.7:0/1088191230 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f7e680d0ac0 con 0x7f7e701029b0 2024-04-21T08:44:04.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.180+0000 7f7e6f7fe640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7e580a3c40 0x7f7e580a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7e60000c00 tx=0x7f7e6002c800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.184+0000 7f7e6cff9640 1 -- 172.21.15.7:0/1088191230 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e68014030 con 0x7f7e701029b0 2024-04-21T08:44:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.371+0000 7f7e7608c640 1 -- 172.21.15.7:0/1088191230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7e701a0b40 con 0x7f7e701029b0 2024-04-21T08:44:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.371+0000 7f7e6cff9640 1 -- 172.21.15.7:0/1088191230 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f7e6809cf80 con 0x7f7e701029b0 2024-04-21T08:44:04.371 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:04.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.373+0000 7f7e7608c640 1 -- 172.21.15.7:0/1088191230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7e580a3c40 msgr2=0x7f7e580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.373+0000 7f7e7608c640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7e580a3c40 0x7f7e580a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7e60000c00 tx=0x7f7e6002c800 comp rx=0 tx=0).stop 2024-04-21T08:44:04.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.373+0000 7f7e7608c640 1 -- 172.21.15.7:0/1088191230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e701029b0 msgr2=0x7f7e7019f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.373+0000 7f7e7608c640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e701029b0 0x7f7e7019f730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800cad0 tx=0x7f7e6800cf90 comp rx=0 tx=0).stop 2024-04-21T08:44:04.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.373+0000 7f7e7608c640 1 -- 172.21.15.7:0/1088191230 shutdown_connections 2024-04-21T08:44:04.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.373+0000 7f7e7608c640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7e580a3c40 0x7f7e580a6100 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.373+0000 7f7e7608c640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e701029b0 0x7f7e7019f730 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.373+0000 7f7e7608c640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e70101ff0 0x7f7e701986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.373+0000 7f7e7608c640 1 --2- 172.21.15.7:0/1088191230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e70100ea0 0x7f7e70198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.373+0000 7f7e7608c640 1 -- 172.21.15.7:0/1088191230 >> 172.21.15.7:0/1088191230 conn(0x7f7e700fc430 msgr2=0x7f7e7006a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:04.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.374+0000 7f7e7608c640 1 -- 172.21.15.7:0/1088191230 shutdown_connections 2024-04-21T08:44:04.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.374+0000 7f7e7608c640 1 -- 172.21.15.7:0/1088191230 wait complete. 2024-04-21T08:44:04.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.488+0000 7f82babfd640 1 -- 172.21.15.7:0/2443984014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82b4101fd0 msgr2=0x7f82b4102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.488+0000 7f82babfd640 1 --2- 172.21.15.7:0/2443984014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82b4101fd0 0x7f82b4102450 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009900 tx=0x7f82a402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.488+0000 7f82babfd640 1 -- 172.21.15.7:0/2443984014 shutdown_connections 2024-04-21T08:44:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.488+0000 7f82babfd640 1 --2- 172.21.15.7:0/2443984014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82b4102990 0x7f82b41092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.488+0000 7f82babfd640 1 --2- 172.21.15.7:0/2443984014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82b4101fd0 0x7f82b4102450 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 --2- 172.21.15.7:0/2443984014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b4100e80 0x7f82b41012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 -- 172.21.15.7:0/2443984014 >> 172.21.15.7:0/2443984014 conn(0x7f82b40fc3f0 msgr2=0x7f82b40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 -- 172.21.15.7:0/2443984014 shutdown_connections 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 -- 172.21.15.7:0/2443984014 wait complete. 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 Processor -- start 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 -- start start 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82b4100e80 0x7f82b41981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82b4101fd0 0x7f82b4198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b4102990 0x7f82b419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f82b410c9c0 con 0x7f82b4101fd0 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82b410c840 con 0x7f82b4102990 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82babfd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f82b410cb40 con 0x7f82b4100e80 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82b4101fd0 0x7f82b4198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82b4101fd0 0x7f82b4198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36420/0 (socket says 172.21.15.7:36420) 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.489+0000 7f82b3fff640 1 -- 172.21.15.7:0/3874809261 learned_addr learned my addr 172.21.15.7:0/3874809261 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82b3fff640 1 -- 172.21.15.7:0/3874809261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82b4100e80 msgr2=0x7f82b41981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82b8999640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82b4100e80 0x7f82b41981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82b3fff640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82b4100e80 0x7f82b41981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82b3fff640 1 -- 172.21.15.7:0/3874809261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b4102990 msgr2=0x7f82b419f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82b3fff640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b4102990 0x7f82b419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82b3fff640 1 -- 172.21.15.7:0/3874809261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b419fe00 con 0x7f82b4101fd0 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82b8999640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82b4100e80 0x7f82b41981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82b3fff640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82b4101fd0 0x7f82b4198720 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f82a4036040 tx=0x7f82a40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82b1ffb640 1 -- 172.21.15.7:0/3874809261 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a4043020 con 0x7f82b4101fd0 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82b1ffb640 1 -- 172.21.15.7:0/3874809261 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82a4004aa0 con 0x7f82b4101fd0 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82babfd640 1 -- 172.21.15.7:0/3874809261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b41a0090 con 0x7f82b4101fd0 2024-04-21T08:44:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82b1ffb640 1 -- 172.21.15.7:0/3874809261 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a4003de0 con 0x7f82b4101fd0 2024-04-21T08:44:04.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.490+0000 7f82babfd640 1 -- 172.21.15.7:0/3874809261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b41a05e0 con 0x7f82b4101fd0 2024-04-21T08:44:04.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.491+0000 7f82babfd640 1 -- 172.21.15.7:0/3874809261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82b41a07e0 con 0x7f82b4101fd0 2024-04-21T08:44:04.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.492+0000 7f82b1ffb640 1 -- 172.21.15.7:0/3874809261 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f82a4046020 con 0x7f82b4101fd0 2024-04-21T08:44:04.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.493+0000 7f82b1ffb640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82a00a3c40 0x7f82a00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.493+0000 7f82b8999640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82a00a3c40 0x7f82a00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.493+0000 7f82b1ffb640 1 -- 172.21.15.7:0/3874809261 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f82a40f2d30 con 0x7f82b4101fd0 2024-04-21T08:44:04.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.493+0000 7f82b8999640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82a00a3c40 0x7f82a00a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f82ac005c40 tx=0x7f82ac0082f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:04.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.498+0000 7f82b1ffb640 1 -- 172.21.15.7:0/3874809261 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f82a404d020 con 0x7f82b4101fd0 2024-04-21T08:44:04.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.725+0000 7f82babfd640 1 -- 172.21.15.7:0/3874809261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82b41a0b70 con 0x7f82b4101fd0 2024-04-21T08:44:04.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.725+0000 7f82b1ffb640 1 -- 172.21.15.7:0/3874809261 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7f82a40bf300 con 0x7f82b4101fd0 2024-04-21T08:44:04.725 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:04.725 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:44:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:44:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 -- 172.21.15.7:0/3874809261 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82a00a3c40 msgr2=0x7f82a00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82a00a3c40 0x7f82a00a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f82ac005c40 tx=0x7f82ac0082f0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 -- 172.21.15.7:0/3874809261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82b4101fd0 msgr2=0x7f82b4198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82b4101fd0 0x7f82b4198720 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f82a4036040 tx=0x7f82a40042f0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 -- 172.21.15.7:0/3874809261 shutdown_connections 2024-04-21T08:44:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f82a00a3c40 0x7f82a00a6100 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b4102990 0x7f82b419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82b4101fd0 0x7f82b4198720 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 --2- 172.21.15.7:0/3874809261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82b4100e80 0x7f82b41981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 -- 172.21.15.7:0/3874809261 >> 172.21.15.7:0/3874809261 conn(0x7f82b40fc3f0 msgr2=0x7f82b40fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:04.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 -- 172.21.15.7:0/3874809261 shutdown_connections 2024-04-21T08:44:04.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.727+0000 7f82babfd640 1 -- 172.21.15.7:0/3874809261 wait complete. 2024-04-21T08:44:04.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:04.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.842+0000 7fbe931db640 1 -- 172.21.15.7:0/98498314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe8c102a20 msgr2=0x7fbe8c104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.842+0000 7fbe931db640 1 --2- 172.21.15.7:0/98498314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe8c102a20 0x7fbe8c104e10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009900 tx=0x7fbe8002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:04.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.842+0000 7fbe931db640 1 -- 172.21.15.7:0/98498314 shutdown_connections 2024-04-21T08:44:04.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.842+0000 7fbe931db640 1 --2- 172.21.15.7:0/98498314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe8c107c80 0x7fbe8c10a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.842+0000 7fbe931db640 1 --2- 172.21.15.7:0/98498314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe8c105350 0x7fbe8c107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.842+0000 7fbe931db640 1 --2- 172.21.15.7:0/98498314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe8c102a20 0x7fbe8c104e10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.842+0000 7fbe931db640 1 -- 172.21.15.7:0/98498314 >> 172.21.15.7:0/98498314 conn(0x7fbe8c0fc410 msgr2=0x7fbe8c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.842+0000 7fbe931db640 1 -- 172.21.15.7:0/98498314 shutdown_connections 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.842+0000 7fbe931db640 1 -- 172.21.15.7:0/98498314 wait complete. 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe931db640 1 Processor -- start 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe931db640 1 -- start start 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe931db640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe8c102a20 0x7fbe8c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe931db640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe8c105350 0x7fbe8c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe931db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe8c107c80 0x7fbe8c19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe931db640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c10db40 con 0x7fbe8c105350 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe931db640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c10d9c0 con 0x7fbe8c102a20 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe931db640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c10dcc0 con 0x7fbe8c107c80 2024-04-21T08:44:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe8bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe8c105350 0x7fbe8c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe8bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe8c105350 0x7fbe8c198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40954/0 (socket says 172.21.15.7:40954) 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe8bfff640 1 -- 172.21.15.7:0/3380143388 learned_addr learned my addr 172.21.15.7:0/3380143388 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe90f77640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe8c102a20 0x7fbe8c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe8bfff640 1 -- 172.21.15.7:0/3380143388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe8c107c80 msgr2=0x7fbe8c19f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe91778640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe8c107c80 0x7fbe8c19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe8bfff640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe8c107c80 0x7fbe8c19f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe8bfff640 1 -- 172.21.15.7:0/3380143388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe8c102a20 msgr2=0x7fbe8c1981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.843+0000 7fbe8bfff640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe8c102a20 0x7fbe8c1981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.844+0000 7fbe8bfff640 1 -- 172.21.15.7:0/3380143388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8c19fe00 con 0x7fbe8c105350 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.844+0000 7fbe90f77640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe8c102a20 0x7fbe8c1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.844+0000 7fbe8bfff640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe8c105350 0x7fbe8c198720 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800b450 tx=0x7fbe7800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.844+0000 7fbe89ffb640 1 -- 172.21.15.7:0/3380143388 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78015020 con 0x7fbe8c105350 2024-04-21T08:44:04.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.844+0000 7fbe89ffb640 1 -- 172.21.15.7:0/3380143388 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe78011d40 con 0x7fbe8c105350 2024-04-21T08:44:04.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.844+0000 7fbe931db640 1 -- 172.21.15.7:0/3380143388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe8c1a00f0 con 0x7fbe8c105350 2024-04-21T08:44:04.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.844+0000 7fbe89ffb640 1 -- 172.21.15.7:0/3380143388 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7800f340 con 0x7fbe8c105350 2024-04-21T08:44:04.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.844+0000 7fbe931db640 1 -- 172.21.15.7:0/3380143388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe8c1a06c0 con 0x7fbe8c105350 2024-04-21T08:44:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.845+0000 7fbe931db640 1 -- 172.21.15.7:0/3380143388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe8c0fdf00 con 0x7fbe8c105350 2024-04-21T08:44:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.846+0000 7fbe89ffb640 1 -- 172.21.15.7:0/3380143388 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fbe78018020 con 0x7fbe8c105350 2024-04-21T08:44:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.847+0000 7fbe89ffb640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe7c0a3c40 0x7fbe7c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.847+0000 7fbe90f77640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe7c0a3c40 0x7fbe7c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.847+0000 7fbe89ffb640 1 -- 172.21.15.7:0/3380143388 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fbe780d0320 con 0x7fbe8c105350 2024-04-21T08:44:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.848+0000 7fbe90f77640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe7c0a3c40 0x7fbe7c0a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbe800096c0 tx=0x7fbe80000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:04.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:04.852+0000 7fbe89ffb640 1 -- 172.21.15.7:0/3380143388 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe7801f020 con 0x7fbe8c105350 2024-04-21T08:44:05.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.039+0000 7fbe931db640 1 -- 172.21.15.7:0/3380143388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe8c02d810 con 0x7fbe8c105350 2024-04-21T08:44:05.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.041+0000 7fbe89ffb640 1 -- 172.21.15.7:0/3380143388 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24432 (secure 0 0 0) 0x7fbe7800d040 con 0x7fbe8c105350 2024-04-21T08:44:05.040 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:05.041 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":92,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:43:10.778941+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 -- 172.21.15.7:0/3380143388 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe7c0a3c40 msgr2=0x7fbe7c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe7c0a3c40 0x7fbe7c0a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbe800096c0 tx=0x7fbe80000bc0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 -- 172.21.15.7:0/3380143388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe8c105350 msgr2=0x7fbe8c198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe8c105350 0x7fbe8c198720 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800b450 tx=0x7fbe7800b910 comp rx=0 tx=0).stop 2024-04-21T08:44:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 -- 172.21.15.7:0/3380143388 shutdown_connections 2024-04-21T08:44:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fbe7c0a3c40 0x7fbe7c0a6100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe8c107c80 0x7fbe8c19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe8c105350 0x7fbe8c198720 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 --2- 172.21.15.7:0/3380143388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe8c102a20 0x7fbe8c1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 -- 172.21.15.7:0/3380143388 >> 172.21.15.7:0/3380143388 conn(0x7fbe8c0fc410 msgr2=0x7fbe8c1034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 -- 172.21.15.7:0/3380143388 shutdown_connections 2024-04-21T08:44:05.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.043+0000 7fbe931db640 1 -- 172.21.15.7:0/3380143388 wait complete. 2024-04-21T08:44:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:04 smithi007 ceph-mon[28477]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:04 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2338347163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:04 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/461884250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:04 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1088191230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:04 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3874809261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:05.054 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:05.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:05.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:04 smithi016 ceph-mon[31259]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:05.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:04 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2338347163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:05.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:04 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/461884250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:05.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:04 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1088191230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:05.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:04 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3874809261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:05.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:04 smithi077 ceph-mon[31615]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:05.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:04 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2338347163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:05.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:04 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/461884250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:05.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:04 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1088191230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:05.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:04 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3874809261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.161+0000 7fa7a59b1640 1 -- 172.21.15.7:0/3051778458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a01031a0 msgr2=0x7fa7a0103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.161+0000 7fa7a59b1640 1 --2- 172.21.15.7:0/3051778458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a01031a0 0x7fa7a0103620 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa794009900 tx=0x7fa79402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.161+0000 7fa7a59b1640 1 -- 172.21.15.7:0/3051778458 shutdown_connections 2024-04-21T08:44:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.161+0000 7fa7a59b1640 1 --2- 172.21.15.7:0/3051778458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7a0103b60 0x7fa7a010a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.161+0000 7fa7a59b1640 1 --2- 172.21.15.7:0/3051778458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a01031a0 0x7fa7a0103620 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.161+0000 7fa7a59b1640 1 --2- 172.21.15.7:0/3051778458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a0101fb0 0x7fa7a01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.161+0000 7fa7a59b1640 1 -- 172.21.15.7:0/3051778458 >> 172.21.15.7:0/3051778458 conn(0x7fa7a00fd520 msgr2=0x7fa7a00ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.161+0000 7fa7a59b1640 1 -- 172.21.15.7:0/3051778458 shutdown_connections 2024-04-21T08:44:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.161+0000 7fa7a59b1640 1 -- 172.21.15.7:0/3051778458 wait complete. 2024-04-21T08:44:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.161+0000 7fa7a59b1640 1 Processor -- start 2024-04-21T08:44:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa7a59b1640 1 -- start start 2024-04-21T08:44:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa7a59b1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a0101fb0 0x7fa7a0198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa7a59b1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7a01031a0 0x7fa7a0198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa7a59b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a0103b60 0x7fa7a019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa7a59b1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010daf0 con 0x7fa7a0101fb0 2024-04-21T08:44:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa7a59b1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010d970 con 0x7fa7a0103b60 2024-04-21T08:44:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa7a59b1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010dc70 con 0x7fa7a01031a0 2024-04-21T08:44:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a0101fb0 0x7fa7a0198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a0101fb0 0x7fa7a0198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40978/0 (socket says 172.21.15.7:40978) 2024-04-21T08:44:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79effd640 1 -- 172.21.15.7:0/2104768271 learned_addr learned my addr 172.21.15.7:0/2104768271 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79f7fe640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a0103b60 0x7fa7a019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79e7fc640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7a01031a0 0x7fa7a0198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79effd640 1 -- 172.21.15.7:0/2104768271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7a01031a0 msgr2=0x7fa7a0198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79effd640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7a01031a0 0x7fa7a0198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79effd640 1 -- 172.21.15.7:0/2104768271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a0103b60 msgr2=0x7fa7a019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79effd640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a0103b60 0x7fa7a019f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79effd640 1 -- 172.21.15.7:0/2104768271 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a019fdd0 con 0x7fa7a0101fb0 2024-04-21T08:44:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79f7fe640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a0103b60 0x7fa7a019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79e7fc640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7a01031a0 0x7fa7a0198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.162+0000 7fa79effd640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a0101fb0 0x7fa7a0198250 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa79000e9c0 tx=0x7fa79000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.163+0000 7fa7a49af640 1 -- 172.21.15.7:0/2104768271 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79000cde0 con 0x7fa7a0101fb0 2024-04-21T08:44:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.163+0000 7fa7a49af640 1 -- 172.21.15.7:0/2104768271 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7900108e0 con 0x7fa7a0101fb0 2024-04-21T08:44:05.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.163+0000 7fa7a59b1640 1 -- 172.21.15.7:0/2104768271 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a01a00c0 con 0x7fa7a0101fb0 2024-04-21T08:44:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.163+0000 7fa7a49af640 1 -- 172.21.15.7:0/2104768271 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790018960 con 0x7fa7a0101fb0 2024-04-21T08:44:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.163+0000 7fa7a59b1640 1 -- 172.21.15.7:0/2104768271 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a01a0660 con 0x7fa7a0101fb0 2024-04-21T08:44:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.164+0000 7fa7a59b1640 1 -- 172.21.15.7:0/2104768271 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a01092f0 con 0x7fa7a0101fb0 2024-04-21T08:44:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.165+0000 7fa7a49af640 1 -- 172.21.15.7:0/2104768271 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fa79001f020 con 0x7fa7a0101fb0 2024-04-21T08:44:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.165+0000 7fa7a49af640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa7880a3cc0 0x7fa7880a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.165+0000 7fa79e7fc640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa7880a3cc0 0x7fa7880a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.165+0000 7fa7a49af640 1 -- 172.21.15.7:0/2104768271 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fa7900d0c50 con 0x7fa7a0101fb0 2024-04-21T08:44:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.166+0000 7fa79e7fc640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa7880a3cc0 0x7fa7880a6180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa7940096c0 tx=0x7fa794000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.170+0000 7fa7a49af640 1 -- 172.21.15.7:0/2104768271 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa790053ff0 con 0x7fa7a0101fb0 2024-04-21T08:44:05.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.357+0000 7fa7a59b1640 1 -- 172.21.15.7:0/2104768271 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa7a0062b80 con 0x7fa7a0101fb0 2024-04-21T08:44:05.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.357+0000 7fa7a49af640 1 -- 172.21.15.7:0/2104768271 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7fa79009d170 con 0x7fa7a0101fb0 2024-04-21T08:44:05.357 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:44:05.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 -- 172.21.15.7:0/2104768271 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa7880a3cc0 msgr2=0x7fa7880a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa7880a3cc0 0x7fa7880a6180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa7940096c0 tx=0x7fa794000bc0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 -- 172.21.15.7:0/2104768271 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a0101fb0 msgr2=0x7fa7a0198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a0101fb0 0x7fa7a0198250 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa79000e9c0 tx=0x7fa79000ee80 comp rx=0 tx=0).stop 2024-04-21T08:44:05.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 -- 172.21.15.7:0/2104768271 shutdown_connections 2024-04-21T08:44:05.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa7880a3cc0 0x7fa7880a6180 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a0103b60 0x7fa7a019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7a01031a0 0x7fa7a0198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 --2- 172.21.15.7:0/2104768271 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7a0101fb0 0x7fa7a0198250 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 -- 172.21.15.7:0/2104768271 >> 172.21.15.7:0/2104768271 conn(0x7fa7a00fd520 msgr2=0x7fa7a01064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:05.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 -- 172.21.15.7:0/2104768271 shutdown_connections 2024-04-21T08:44:05.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.359+0000 7fa7a59b1640 1 -- 172.21.15.7:0/2104768271 wait complete. 2024-04-21T08:44:05.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:05.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.476+0000 7f122d7f4640 1 -- 172.21.15.7:0/3680921289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12281029a0 msgr2=0x7f12281092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.476+0000 7f122d7f4640 1 --2- 172.21.15.7:0/3680921289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12281029a0 0x7f12281092c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1220009ca0 tx=0x7f122002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.476+0000 7f122d7f4640 1 -- 172.21.15.7:0/3680921289 shutdown_connections 2024-04-21T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.476+0000 7f122d7f4640 1 --2- 172.21.15.7:0/3680921289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12281029a0 0x7f12281092c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.476+0000 7f122d7f4640 1 --2- 172.21.15.7:0/3680921289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228101fe0 0x7f1228102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.476+0000 7f122d7f4640 1 --2- 172.21.15.7:0/3680921289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1228100e90 0x7f12281012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.476+0000 7f122d7f4640 1 -- 172.21.15.7:0/3680921289 >> 172.21.15.7:0/3680921289 conn(0x7f12280fc420 msgr2=0x7f12280fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.476+0000 7f122d7f4640 1 -- 172.21.15.7:0/3680921289 shutdown_connections 2024-04-21T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.476+0000 7f122d7f4640 1 -- 172.21.15.7:0/3680921289 wait complete. 2024-04-21T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.476+0000 7f122d7f4640 1 Processor -- start 2024-04-21T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f122d7f4640 1 -- start start 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f122d7f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228100e90 0x7f122810b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f122d7f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1228101fe0 0x7f1228109e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f122d7f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12281029a0 0x7f122810a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f122d7f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f122810db30 con 0x7f1228101fe0 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f122d7f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f122810d9b0 con 0x7f1228100e90 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f122d7f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f122810dcb0 con 0x7f12281029a0 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f12267fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1228101fe0 0x7f1228109e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f12267fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1228101fe0 0x7f1228109e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41000/0 (socket says 172.21.15.7:41000) 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f12267fc640 1 -- 172.21.15.7:0/2609884521 learned_addr learned my addr 172.21.15.7:0/2609884521 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f1226ffd640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228100e90 0x7f122810b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f12277fe640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12281029a0 0x7f122810a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f12267fc640 1 -- 172.21.15.7:0/2609884521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12281029a0 msgr2=0x7f122810a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f12267fc640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12281029a0 0x7f122810a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f12267fc640 1 -- 172.21.15.7:0/2609884521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228100e90 msgr2=0x7f122810b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f12267fc640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228100e90 0x7f122810b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f12267fc640 1 -- 172.21.15.7:0/2609884521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122810ac40 con 0x7f1228101fe0 2024-04-21T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f12267fc640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1228101fe0 0x7f1228109e70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f121c00d950 tx=0x7f121c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.477+0000 7f120ffff640 1 -- 172.21.15.7:0/2609884521 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c017020 con 0x7f1228101fe0 2024-04-21T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.478+0000 7f120ffff640 1 -- 172.21.15.7:0/2609884521 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f121c009c30 con 0x7f1228101fe0 2024-04-21T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.478+0000 7f122d7f4640 1 -- 172.21.15.7:0/2609884521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12281a2280 con 0x7f1228101fe0 2024-04-21T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.478+0000 7f120ffff640 1 -- 172.21.15.7:0/2609884521 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c00f6b0 con 0x7f1228101fe0 2024-04-21T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.478+0000 7f122d7f4640 1 -- 172.21.15.7:0/2609884521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12281a27d0 con 0x7f1228101fe0 2024-04-21T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.479+0000 7f122d7f4640 1 -- 172.21.15.7:0/2609884521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1228108240 con 0x7f1228101fe0 2024-04-21T08:44:05.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.480+0000 7f120ffff640 1 -- 172.21.15.7:0/2609884521 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f121c01a020 con 0x7f1228101fe0 2024-04-21T08:44:05.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.480+0000 7f120ffff640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f12100a3c40 0x7f12100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.480+0000 7f1226ffd640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f12100a3c40 0x7f12100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.480+0000 7f120ffff640 1 -- 172.21.15.7:0/2609884521 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f121c0d0550 con 0x7f1228101fe0 2024-04-21T08:44:05.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.481+0000 7f1226ffd640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f12100a3c40 0x7f12100a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f12180058c0 tx=0x7f121800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:05.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.485+0000 7f120ffff640 1 -- 172.21.15.7:0/2609884521 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f121c009de0 con 0x7f1228101fe0 2024-04-21T08:44:05.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.674+0000 7f122d7f4640 1 -- 172.21.15.7:0/2609884521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f122802d850 con 0x7f1228101fe0 2024-04-21T08:44:05.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.674+0000 7f120ffff640 1 -- 172.21.15.7:0/2609884521 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f121c09ca70 con 0x7f1228101fe0 2024-04-21T08:44:05.674 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:05.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 -- 172.21.15.7:0/2609884521 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f12100a3c40 msgr2=0x7f12100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f12100a3c40 0x7f12100a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f12180058c0 tx=0x7f121800a2d0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 -- 172.21.15.7:0/2609884521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1228101fe0 msgr2=0x7f1228109e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1228101fe0 0x7f1228109e70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f121c00d950 tx=0x7f121c00de10 comp rx=0 tx=0).stop 2024-04-21T08:44:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 -- 172.21.15.7:0/2609884521 shutdown_connections 2024-04-21T08:44:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f12100a3c40 0x7f12100a6100 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12281029a0 0x7f122810a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1228101fe0 0x7f1228109e70 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 --2- 172.21.15.7:0/2609884521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228100e90 0x7f122810b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 -- 172.21.15.7:0/2609884521 >> 172.21.15.7:0/2609884521 conn(0x7f12280fc420 msgr2=0x7f1228105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 -- 172.21.15.7:0/2609884521 shutdown_connections 2024-04-21T08:44:05.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.677+0000 7f122d7f4640 1 -- 172.21.15.7:0/2609884521 wait complete. 2024-04-21T08:44:05.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.799+0000 7fe7ccac1640 1 -- 172.21.15.7:0/1754276970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c8103190 msgr2=0x7fe7c8103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.799+0000 7fe7ccac1640 1 --2- 172.21.15.7:0/1754276970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c8103190 0x7fe7c8103610 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009900 tx=0x7fe7bc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.799+0000 7fe7ccac1640 1 -- 172.21.15.7:0/1754276970 shutdown_connections 2024-04-21T08:44:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.799+0000 7fe7ccac1640 1 --2- 172.21.15.7:0/1754276970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c8103b50 0x7fe7c810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.799+0000 7fe7ccac1640 1 --2- 172.21.15.7:0/1754276970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c8103190 0x7fe7c8103610 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.799+0000 7fe7ccac1640 1 --2- 172.21.15.7:0/1754276970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c8101fa0 0x7fe7c81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.799+0000 7fe7ccac1640 1 -- 172.21.15.7:0/1754276970 >> 172.21.15.7:0/1754276970 conn(0x7fe7c80fd510 msgr2=0x7fe7c80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.799+0000 7fe7ccac1640 1 -- 172.21.15.7:0/1754276970 shutdown_connections 2024-04-21T08:44:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.799+0000 7fe7ccac1640 1 -- 172.21.15.7:0/1754276970 wait complete. 2024-04-21T08:44:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7ccac1640 1 Processor -- start 2024-04-21T08:44:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7ccac1640 1 -- start start 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7ccac1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c8101fa0 0x7fe7c81981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7ccac1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c8103190 0x7fe7c8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7ccac1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c8103b50 0x7fe7c819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7ccac1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c810dae0 con 0x7fe7c8103190 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7ccac1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c810d960 con 0x7fe7c8101fa0 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7ccac1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c810dc60 con 0x7fe7c8103b50 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7c5d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c8103190 0x7fe7c8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7c5d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c8103190 0x7fe7c8198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41006/0 (socket says 172.21.15.7:41006) 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7c5d9b640 1 -- 172.21.15.7:0/3393289205 learned_addr learned my addr 172.21.15.7:0/3393289205 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7c659c640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c8101fa0 0x7fe7c81981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7c6d9d640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c8103b50 0x7fe7c819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7c5d9b640 1 -- 172.21.15.7:0/3393289205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c8103b50 msgr2=0x7fe7c819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7c5d9b640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c8103b50 0x7fe7c819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7c5d9b640 1 -- 172.21.15.7:0/3393289205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c8101fa0 msgr2=0x7fe7c81981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.800+0000 7fe7c5d9b640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c8101fa0 0x7fe7c81981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.801+0000 7fe7c5d9b640 1 -- 172.21.15.7:0/3393289205 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c819fdf0 con 0x7fe7c8103190 2024-04-21T08:44:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.801+0000 7fe7c6d9d640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c8103b50 0x7fe7c819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.801+0000 7fe7c659c640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c8101fa0 0x7fe7c81981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.801+0000 7fe7c5d9b640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c8103190 0x7fe7c8198710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc006010 tx=0x7fe7bc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.801+0000 7fe7b77fe640 1 -- 172.21.15.7:0/3393289205 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc043020 con 0x7fe7c8103190 2024-04-21T08:44:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.801+0000 7fe7b77fe640 1 -- 172.21.15.7:0/3393289205 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7bc004aa0 con 0x7fe7c8103190 2024-04-21T08:44:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.801+0000 7fe7ccac1640 1 -- 172.21.15.7:0/3393289205 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c81a0080 con 0x7fe7c8103190 2024-04-21T08:44:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.801+0000 7fe7b77fe640 1 -- 172.21.15.7:0/3393289205 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc003c70 con 0x7fe7c8103190 2024-04-21T08:44:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.801+0000 7fe7ccac1640 1 -- 172.21.15.7:0/3393289205 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c81a0620 con 0x7fe7c8103190 2024-04-21T08:44:05.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.802+0000 7fe7ccac1640 1 -- 172.21.15.7:0/3393289205 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7c81092e0 con 0x7fe7c8103190 2024-04-21T08:44:05.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.803+0000 7fe7b77fe640 1 -- 172.21.15.7:0/3393289205 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fe7bc046020 con 0x7fe7c8103190 2024-04-21T08:44:05.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.804+0000 7fe7b77fe640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe7b00a3cc0 0x7fe7b00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.804+0000 7fe7c659c640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe7b00a3cc0 0x7fe7b00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.804+0000 7fe7b77fe640 1 -- 172.21.15.7:0/3393289205 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fe7bc0f28c0 con 0x7fe7c8103190 2024-04-21T08:44:05.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.804+0000 7fe7c659c640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe7b00a3cc0 0x7fe7b00a6180 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8102ff0 tx=0x7fe7b800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:05.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.808+0000 7fe7b77fe640 1 -- 172.21.15.7:0/3393289205 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7bc04d070 con 0x7fe7c8103190 2024-04-21T08:44:05.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.995+0000 7fe7ccac1640 1 -- 172.21.15.7:0/3393289205 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe7c802d810 con 0x7fe7c8103190 2024-04-21T08:44:05.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.995+0000 7fe7b77fe640 1 -- 172.21.15.7:0/3393289205 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7fe7bc0bee90 con 0x7fe7c8103190 2024-04-21T08:44:05.995 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:44:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 -- 172.21.15.7:0/3393289205 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe7b00a3cc0 msgr2=0x7fe7b00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe7b00a3cc0 0x7fe7b00a6180 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8102ff0 tx=0x7fe7b800a560 comp rx=0 tx=0).stop 2024-04-21T08:44:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 -- 172.21.15.7:0/3393289205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c8103190 msgr2=0x7fe7c8198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c8103190 0x7fe7c8198710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc006010 tx=0x7fe7bc0042f0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 -- 172.21.15.7:0/3393289205 shutdown_connections 2024-04-21T08:44:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe7b00a3cc0 0x7fe7b00a6180 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c8103b50 0x7fe7c819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c8103190 0x7fe7c8198710 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 --2- 172.21.15.7:0/3393289205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7c8101fa0 0x7fe7c81981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 -- 172.21.15.7:0/3393289205 >> 172.21.15.7:0/3393289205 conn(0x7fe7c80fd510 msgr2=0x7fe7c81064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 -- 172.21.15.7:0/3393289205 shutdown_connections 2024-04-21T08:44:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:05.997+0000 7fe7ccac1640 1 -- 172.21.15.7:0/3393289205 wait complete. 2024-04-21T08:44:06.007 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:05 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3380143388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:06.007 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:05 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2104768271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:06.007 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:05 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2609884521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:06.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:06.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:05 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3380143388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:06.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:05 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2104768271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:06.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:05 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2609884521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:06.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:05 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3380143388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:06.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:05 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2104768271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:06.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:05 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2609884521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.114+0000 7ff162e74640 1 -- 172.21.15.7:0/1896965217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff15c102a80 msgr2=0x7ff15c104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.114+0000 7ff162e74640 1 --2- 172.21.15.7:0/1896965217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff15c102a80 0x7ff15c104e70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff150009900 tx=0x7ff15002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.114+0000 7ff162e74640 1 -- 172.21.15.7:0/1896965217 shutdown_connections 2024-04-21T08:44:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.114+0000 7ff162e74640 1 --2- 172.21.15.7:0/1896965217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15c107ce0 0x7ff15c10a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.114+0000 7ff162e74640 1 --2- 172.21.15.7:0/1896965217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff15c1053b0 0x7ff15c1077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.114+0000 7ff162e74640 1 --2- 172.21.15.7:0/1896965217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff15c102a80 0x7ff15c104e70 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.114+0000 7ff162e74640 1 -- 172.21.15.7:0/1896965217 >> 172.21.15.7:0/1896965217 conn(0x7ff15c0fc430 msgr2=0x7ff15c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.114+0000 7ff162e74640 1 -- 172.21.15.7:0/1896965217 shutdown_connections 2024-04-21T08:44:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.114+0000 7ff162e74640 1 -- 172.21.15.7:0/1896965217 wait complete. 2024-04-21T08:44:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff162e74640 1 Processor -- start 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff162e74640 1 -- start start 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff162e74640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff15c102a80 0x7ff15c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff162e74640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15c1053b0 0x7ff15c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff162e74640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff15c107ce0 0x7ff15c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff162e74640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c10dae0 con 0x7ff15c102a80 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff162e74640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c10d960 con 0x7ff15c1053b0 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff162e74640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c10dc60 con 0x7ff15c107ce0 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff160c10640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff15c102a80 0x7ff15c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff160c10640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff15c102a80 0x7ff15c198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41034/0 (socket says 172.21.15.7:41034) 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff160c10640 1 -- 172.21.15.7:0/3066612891 learned_addr learned my addr 172.21.15.7:0/3066612891 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff15bfff640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15c1053b0 0x7ff15c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff161411640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff15c107ce0 0x7ff15c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff160c10640 1 -- 172.21.15.7:0/3066612891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff15c107ce0 msgr2=0x7ff15c19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff160c10640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff15c107ce0 0x7ff15c19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff160c10640 1 -- 172.21.15.7:0/3066612891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15c1053b0 msgr2=0x7ff15c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff160c10640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15c1053b0 0x7ff15c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff160c10640 1 -- 172.21.15.7:0/3066612891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15c19fe20 con 0x7ff15c102a80 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff161411640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff15c107ce0 0x7ff15c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff15bfff640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15c1053b0 0x7ff15c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.115+0000 7ff160c10640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff15c102a80 0x7ff15c198200 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff150006010 tx=0x7ff150004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.116+0000 7ff159ffb640 1 -- 172.21.15.7:0/3066612891 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff150043020 con 0x7ff15c102a80 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.116+0000 7ff159ffb640 1 -- 172.21.15.7:0/3066612891 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff150004440 con 0x7ff15c102a80 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.116+0000 7ff162e74640 1 -- 172.21.15.7:0/3066612891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff15c1a00b0 con 0x7ff15c102a80 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.116+0000 7ff159ffb640 1 -- 172.21.15.7:0/3066612891 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff150003bf0 con 0x7ff15c102a80 2024-04-21T08:44:06.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.116+0000 7ff162e74640 1 -- 172.21.15.7:0/3066612891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15c1a0650 con 0x7ff15c102a80 2024-04-21T08:44:06.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.117+0000 7ff162e74640 1 -- 172.21.15.7:0/3066612891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff15c0fdf20 con 0x7ff15c102a80 2024-04-21T08:44:06.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.118+0000 7ff159ffb640 1 -- 172.21.15.7:0/3066612891 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7ff150046020 con 0x7ff15c102a80 2024-04-21T08:44:06.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.119+0000 7ff159ffb640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff14c0a3cc0 0x7ff14c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.119+0000 7ff159ffb640 1 -- 172.21.15.7:0/3066612891 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7ff1500f31f0 con 0x7ff15c102a80 2024-04-21T08:44:06.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.119+0000 7ff15bfff640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff14c0a3cc0 0x7ff14c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.119+0000 7ff15bfff640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff14c0a3cc0 0x7ff14c0a6180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff15c199710 tx=0x7ff148005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:06.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.122+0000 7ff159ffb640 1 -- 172.21.15.7:0/3066612891 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff15004d020 con 0x7ff15c102a80 2024-04-21T08:44:06.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.308+0000 7ff162e74640 1 -- 172.21.15.7:0/3066612891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff15c062b50 con 0x7ff15c102a80 2024-04-21T08:44:06.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.309+0000 7ff159ffb640 1 -- 172.21.15.7:0/3066612891 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7ff1500bf820 con 0x7ff15c102a80 2024-04-21T08:44:06.309 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:06.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 -- 172.21.15.7:0/3066612891 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff14c0a3cc0 msgr2=0x7ff14c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff14c0a3cc0 0x7ff14c0a6180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff15c199710 tx=0x7ff148005d30 comp rx=0 tx=0).stop 2024-04-21T08:44:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 -- 172.21.15.7:0/3066612891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff15c102a80 msgr2=0x7ff15c198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff15c102a80 0x7ff15c198200 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff150006010 tx=0x7ff150004040 comp rx=0 tx=0).stop 2024-04-21T08:44:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 -- 172.21.15.7:0/3066612891 shutdown_connections 2024-04-21T08:44:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff14c0a3cc0 0x7ff14c0a6180 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff15c107ce0 0x7ff15c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff15c1053b0 0x7ff15c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 --2- 172.21.15.7:0/3066612891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff15c102a80 0x7ff15c198200 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 -- 172.21.15.7:0/3066612891 >> 172.21.15.7:0/3066612891 conn(0x7ff15c0fc430 msgr2=0x7ff15c103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 -- 172.21.15.7:0/3066612891 shutdown_connections 2024-04-21T08:44:06.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.311+0000 7ff162e74640 1 -- 172.21.15.7:0/3066612891 wait complete. 2024-04-21T08:44:06.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.426+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3733367385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68102980 msgr2=0x7f3a681092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.426+0000 7f3a6fe06640 1 --2- 172.21.15.7:0/3733367385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68102980 0x7f3a681092a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009ca0 tx=0x7f3a6002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.426+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3733367385 shutdown_connections 2024-04-21T08:44:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.426+0000 7f3a6fe06640 1 --2- 172.21.15.7:0/3733367385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68102980 0x7f3a681092a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.426+0000 7f3a6fe06640 1 --2- 172.21.15.7:0/3733367385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a68101fc0 0x7f3a68102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.426+0000 7f3a6fe06640 1 --2- 172.21.15.7:0/3733367385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68100e70 0x7f3a68101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.426+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3733367385 >> 172.21.15.7:0/3733367385 conn(0x7f3a680fc420 msgr2=0x7f3a680fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.426+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3733367385 shutdown_connections 2024-04-21T08:44:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.426+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3733367385 wait complete. 2024-04-21T08:44:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.426+0000 7f3a6fe06640 1 Processor -- start 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6fe06640 1 -- start start 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6fe06640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a68100e70 0x7f3a68104df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6fe06640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68101fc0 0x7f3a68103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6fe06640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68102980 0x7f3a68103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6fe06640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6810c9a0 con 0x7f3a68102980 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6fe06640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6810c820 con 0x7f3a68100e70 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6fe06640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6810cb20 con 0x7f3a68101fc0 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6e3a3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68102980 0x7f3a68103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6e3a3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68102980 0x7f3a68103970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41064/0 (socket says 172.21.15.7:41064) 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6e3a3640 1 -- 172.21.15.7:0/3229160684 learned_addr learned my addr 172.21.15.7:0/3229160684 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6dba2640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a68100e70 0x7f3a68104df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6d3a1640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68101fc0 0x7f3a68103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6e3a3640 1 -- 172.21.15.7:0/3229160684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68101fc0 msgr2=0x7f3a68103430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6e3a3640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68101fc0 0x7f3a68103430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6e3a3640 1 -- 172.21.15.7:0/3229160684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a68100e70 msgr2=0x7f3a68104df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6e3a3640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a68100e70 0x7f3a68104df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6e3a3640 1 -- 172.21.15.7:0/3229160684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a68104200 con 0x7f3a68102980 2024-04-21T08:44:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6d3a1640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68101fc0 0x7f3a68103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6dba2640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a68100e70 0x7f3a68104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6e3a3640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68102980 0x7f3a68103970 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3a6002c700 tx=0x7f3a600093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a5effd640 1 -- 172.21.15.7:0/3229160684 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a60041020 con 0x7f3a68102980 2024-04-21T08:44:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a5effd640 1 -- 172.21.15.7:0/3229160684 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a60034d60 con 0x7f3a68102980 2024-04-21T08:44:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3229160684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a681a21d0 con 0x7f3a68102980 2024-04-21T08:44:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.427+0000 7f3a5effd640 1 -- 172.21.15.7:0/3229160684 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a60039040 con 0x7f3a68102980 2024-04-21T08:44:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.428+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3229160684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a681a2770 con 0x7f3a68102980 2024-04-21T08:44:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.429+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3229160684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a681a29d0 con 0x7f3a68102980 2024-04-21T08:44:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.430+0000 7f3a5effd640 1 -- 172.21.15.7:0/3229160684 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f3a60044020 con 0x7f3a68102980 2024-04-21T08:44:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.430+0000 7f3a5effd640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3a540a3cc0 0x7f3a540a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.430+0000 7f3a6dba2640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3a540a3cc0 0x7f3a540a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.430+0000 7f3a5effd640 1 -- 172.21.15.7:0/3229160684 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f3a600f2760 con 0x7f3a68102980 2024-04-21T08:44:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.431+0000 7f3a6dba2640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3a540a3cc0 0x7f3a540a6180 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3a64005d00 tx=0x7f3a64005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.435+0000 7f3a5effd640 1 -- 172.21.15.7:0/3229160684 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a6004b020 con 0x7f3a68102980 2024-04-21T08:44:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.663+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3229160684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a681a2d60 con 0x7f3a68102980 2024-04-21T08:44:06.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.663+0000 7f3a5effd640 1 -- 172.21.15.7:0/3229160684 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7f3a600bed90 con 0x7f3a68102980 2024-04-21T08:44:06.663 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:06.663 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:44:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:44:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3229160684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3a540a3cc0 msgr2=0x7f3a540a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3a540a3cc0 0x7f3a540a6180 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3a64005d00 tx=0x7f3a64005c70 comp rx=0 tx=0).stop 2024-04-21T08:44:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3229160684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68102980 msgr2=0x7f3a68103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68102980 0x7f3a68103970 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3a6002c700 tx=0x7f3a600093f0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3229160684 shutdown_connections 2024-04-21T08:44:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3a540a3cc0 0x7f3a540a6180 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68102980 0x7f3a68103970 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a68101fc0 0x7f3a68103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 --2- 172.21.15.7:0/3229160684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a68100e70 0x7f3a68104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3229160684 >> 172.21.15.7:0/3229160684 conn(0x7f3a680fc420 msgr2=0x7f3a680fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3229160684 shutdown_connections 2024-04-21T08:44:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.666+0000 7f3a6fe06640 1 -- 172.21.15.7:0/3229160684 wait complete. 2024-04-21T08:44:06.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:06.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.779+0000 7f426198f640 1 -- 172.21.15.7:0/3510162099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f425c1053d0 msgr2=0x7f425c1077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.779+0000 7f426198f640 1 --2- 172.21.15.7:0/3510162099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f425c1053d0 0x7f425c1077c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4244009900 tx=0x7f424402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:06.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 -- 172.21.15.7:0/3510162099 shutdown_connections 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 --2- 172.21.15.7:0/3510162099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f425c107d00 0x7f425c10a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 --2- 172.21.15.7:0/3510162099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f425c1053d0 0x7f425c1077c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 --2- 172.21.15.7:0/3510162099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c102aa0 0x7f425c104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 -- 172.21.15.7:0/3510162099 >> 172.21.15.7:0/3510162099 conn(0x7f425c0fc3f0 msgr2=0x7f425c0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 -- 172.21.15.7:0/3510162099 shutdown_connections 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 -- 172.21.15.7:0/3510162099 wait complete. 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 Processor -- start 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 -- start start 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f425c102aa0 0x7f425c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f425c1053d0 0x7f425c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c107d00 0x7f425c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f425c10dae0 con 0x7f425c102aa0 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f425c10d960 con 0x7f425c107d00 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.780+0000 7f426198f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f425c10dc60 con 0x7f425c1053d0 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f425c102aa0 0x7f425c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f425c102aa0 0x7f425c1981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41086/0 (socket says 172.21.15.7:41086) 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425affd640 1 -- 172.21.15.7:0/1512741719 learned_addr learned my addr 172.21.15.7:0/1512741719 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425a7fc640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f425c1053d0 0x7f425c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425b7fe640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c107d00 0x7f425c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425affd640 1 -- 172.21.15.7:0/1512741719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f425c1053d0 msgr2=0x7f425c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425affd640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f425c1053d0 0x7f425c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425affd640 1 -- 172.21.15.7:0/1512741719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c107d00 msgr2=0x7f425c19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425affd640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c107d00 0x7f425c19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425affd640 1 -- 172.21.15.7:0/1512741719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f425c19fdd0 con 0x7f425c102aa0 2024-04-21T08:44:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425a7fc640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f425c1053d0 0x7f425c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425b7fe640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c107d00 0x7f425c19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f425affd640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f425c102aa0 0x7f425c1981b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f425400d950 tx=0x7f425400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f426098d640 1 -- 172.21.15.7:0/1512741719 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4254017070 con 0x7f425c102aa0 2024-04-21T08:44:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f426098d640 1 -- 172.21.15.7:0/1512741719 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4254009c30 con 0x7f425c102aa0 2024-04-21T08:44:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f426198f640 1 -- 172.21.15.7:0/1512741719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f425c1a00c0 con 0x7f425c102aa0 2024-04-21T08:44:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f426098d640 1 -- 172.21.15.7:0/1512741719 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425400f620 con 0x7f425c102aa0 2024-04-21T08:44:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.781+0000 7f426198f640 1 -- 172.21.15.7:0/1512741719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f425c1a05e0 con 0x7f425c102aa0 2024-04-21T08:44:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.782+0000 7f426198f640 1 -- 172.21.15.7:0/1512741719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f425c0fdee0 con 0x7f425c102aa0 2024-04-21T08:44:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.784+0000 7f426098d640 1 -- 172.21.15.7:0/1512741719 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f4254010460 con 0x7f425c102aa0 2024-04-21T08:44:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.784+0000 7f426098d640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42480a3c40 0x7f42480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.784+0000 7f425a7fc640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42480a3c40 0x7f42480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.784+0000 7f426098d640 1 -- 172.21.15.7:0/1512741719 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f42540d0630 con 0x7f425c102aa0 2024-04-21T08:44:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.785+0000 7f425a7fc640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42480a3c40 0x7f42480a6100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4244006010 tx=0x7f424402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:06.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.789+0000 7f426098d640 1 -- 172.21.15.7:0/1512741719 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f425409caf0 con 0x7f425c102aa0 2024-04-21T08:44:06.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.977+0000 7f426198f640 1 -- 172.21.15.7:0/1512741719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f425c062b50 con 0x7f425c102aa0 2024-04-21T08:44:06.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.978+0000 7f426098d640 1 -- 172.21.15.7:0/1512741719 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24432 (secure 0 0 0) 0x7f4254009de0 con 0x7f425c102aa0 2024-04-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:06.978 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":92,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:43:10.778941+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:06.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 -- 172.21.15.7:0/1512741719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42480a3c40 msgr2=0x7f42480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42480a3c40 0x7f42480a6100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4244006010 tx=0x7f424402c7e0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 -- 172.21.15.7:0/1512741719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f425c102aa0 msgr2=0x7f425c1981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f425c102aa0 0x7f425c1981b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f425400d950 tx=0x7f425400de10 comp rx=0 tx=0).stop 2024-04-21T08:44:06.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 -- 172.21.15.7:0/1512741719 shutdown_connections 2024-04-21T08:44:06.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f42480a3c40 0x7f42480a6100 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f425c107d00 0x7f425c19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f425c1053d0 0x7f425c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 --2- 172.21.15.7:0/1512741719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f425c102aa0 0x7f425c1981b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 -- 172.21.15.7:0/1512741719 >> 172.21.15.7:0/1512741719 conn(0x7f425c0fc3f0 msgr2=0x7f425c103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:06.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 -- 172.21.15.7:0/1512741719 shutdown_connections 2024-04-21T08:44:06.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:06.980+0000 7f426198f640 1 -- 172.21.15.7:0/1512741719 wait complete. 2024-04-21T08:44:06.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:06 smithi007 ceph-mon[28477]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:06.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:06 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3393289205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:06.990 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:06 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3066612891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:06.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:06 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3229160684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:06.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:07.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:06 smithi016 ceph-mon[31259]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:07.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:06 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3393289205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:07.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:06 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3066612891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:07.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:06 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3229160684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.094+0000 7f9b18d67640 1 -- 172.21.15.7:0/3148891425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b14101ff0 msgr2=0x7f9b14102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.094+0000 7f9b18d67640 1 --2- 172.21.15.7:0/3148891425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b14101ff0 0x7f9b14102470 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9afc009900 tx=0x7f9afc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.094+0000 7f9b18d67640 1 -- 172.21.15.7:0/3148891425 shutdown_connections 2024-04-21T08:44:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.094+0000 7f9b18d67640 1 --2- 172.21.15.7:0/3148891425 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b141029b0 0x7f9b141092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.094+0000 7f9b18d67640 1 --2- 172.21.15.7:0/3148891425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b14101ff0 0x7f9b14102470 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.094+0000 7f9b18d67640 1 --2- 172.21.15.7:0/3148891425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b14100ea0 0x7f9b141012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.094+0000 7f9b18d67640 1 -- 172.21.15.7:0/3148891425 >> 172.21.15.7:0/3148891425 conn(0x7f9b140fc410 msgr2=0x7f9b140fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.094+0000 7f9b18d67640 1 -- 172.21.15.7:0/3148891425 shutdown_connections 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.094+0000 7f9b18d67640 1 -- 172.21.15.7:0/3148891425 wait complete. 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.095+0000 7f9b18d67640 1 Processor -- start 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.095+0000 7f9b18d67640 1 -- start start 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.095+0000 7f9b18d67640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b14100ea0 0x7f9b141981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.095+0000 7f9b18d67640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b14101ff0 0x7f9b141986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.095+0000 7f9b18d67640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b141029b0 0x7f9b1419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.095+0000 7f9b18d67640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1410c9e0 con 0x7f9b141029b0 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.095+0000 7f9b18d67640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1410c860 con 0x7f9b14101ff0 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.095+0000 7f9b18d67640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1410cb60 con 0x7f9b14100ea0 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b12d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b141029b0 0x7f9b1419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b12d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b141029b0 0x7f9b1419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41112/0 (socket says 172.21.15.7:41112) 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b12d9d640 1 -- 172.21.15.7:0/1176132222 learned_addr learned my addr 172.21.15.7:0/1176132222 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b1259c640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b14100ea0 0x7f9b141981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b11d9b640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b14101ff0 0x7f9b141986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b12d9d640 1 -- 172.21.15.7:0/1176132222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b14100ea0 msgr2=0x7f9b141981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b12d9d640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b14100ea0 0x7f9b141981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b12d9d640 1 -- 172.21.15.7:0/1176132222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b14101ff0 msgr2=0x7f9b141986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b12d9d640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b14101ff0 0x7f9b141986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b12d9d640 1 -- 172.21.15.7:0/1176132222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1419fdc0 con 0x7f9b141029b0 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b12d9d640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b141029b0 0x7f9b1419f750 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c00eaa0 tx=0x7f9b0c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b0b7fe640 1 -- 172.21.15.7:0/1176132222 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c00cdb0 con 0x7f9b141029b0 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b0b7fe640 1 -- 172.21.15.7:0/1176132222 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b0c005020 con 0x7f9b141029b0 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b18d67640 1 -- 172.21.15.7:0/1176132222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b141a00b0 con 0x7f9b141029b0 2024-04-21T08:44:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b0b7fe640 1 -- 172.21.15.7:0/1176132222 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c0116d0 con 0x7f9b141029b0 2024-04-21T08:44:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.096+0000 7f9b18d67640 1 -- 172.21.15.7:0/1176132222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b141a05d0 con 0x7f9b141029b0 2024-04-21T08:44:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.097+0000 7f9b18d67640 1 -- 172.21.15.7:0/1176132222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b141a0890 con 0x7f9b141029b0 2024-04-21T08:44:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.098+0000 7f9b0b7fe640 1 -- 172.21.15.7:0/1176132222 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9b0c0051d0 con 0x7f9b141029b0 2024-04-21T08:44:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.099+0000 7f9b0b7fe640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b000a3cc0 0x7f9b000a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.099+0000 7f9b1259c640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b000a3cc0 0x7f9b000a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.099+0000 7f9b0b7fe640 1 -- 172.21.15.7:0/1176132222 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f9b0c0d0a00 con 0x7f9b141029b0 2024-04-21T08:44:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.100+0000 7f9b1259c640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b000a3cc0 0x7f9b000a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b04009bd0 tx=0x7f9b04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:07.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.104+0000 7f9b0b7fe640 1 -- 172.21.15.7:0/1176132222 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b0c0d2e20 con 0x7f9b141029b0 2024-04-21T08:44:07.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:06 smithi077 ceph-mon[31615]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:07.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:06 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3393289205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:07.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:06 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3066612891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:07.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:06 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3229160684' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:07.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.329+0000 7f9b18d67640 1 -- 172.21.15.7:0/1176132222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b141a0a60 con 0x7f9b141029b0 2024-04-21T08:44:07.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.330+0000 7f9b0b7fe640 1 -- 172.21.15.7:0/1176132222 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7f9b0c016030 con 0x7f9b141029b0 2024-04-21T08:44:07.329 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:07.329 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:44:07.329 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:44:07.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.331+0000 7f9b18d67640 1 -- 172.21.15.7:0/1176132222 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b000a3cc0 msgr2=0x7f9b000a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.331+0000 7f9b18d67640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b000a3cc0 0x7f9b000a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b04009bd0 tx=0x7f9b04005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:07.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.331+0000 7f9b18d67640 1 -- 172.21.15.7:0/1176132222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b141029b0 msgr2=0x7f9b1419f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.331+0000 7f9b18d67640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b141029b0 0x7f9b1419f750 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c00eaa0 tx=0x7f9b0c00ef60 comp rx=0 tx=0).stop 2024-04-21T08:44:07.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.332+0000 7f9b18d67640 1 -- 172.21.15.7:0/1176132222 shutdown_connections 2024-04-21T08:44:07.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.332+0000 7f9b18d67640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f9b000a3cc0 0x7f9b000a6180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.332+0000 7f9b18d67640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b141029b0 0x7f9b1419f750 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.332+0000 7f9b18d67640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b14101ff0 0x7f9b141986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.332+0000 7f9b18d67640 1 --2- 172.21.15.7:0/1176132222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b14100ea0 0x7f9b141981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.332+0000 7f9b18d67640 1 -- 172.21.15.7:0/1176132222 >> 172.21.15.7:0/1176132222 conn(0x7f9b140fc410 msgr2=0x7f9b1406a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:07.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.332+0000 7f9b18d67640 1 -- 172.21.15.7:0/1176132222 shutdown_connections 2024-04-21T08:44:07.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.332+0000 7f9b18d67640 1 -- 172.21.15.7:0/1176132222 wait complete. 2024-04-21T08:44:07.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:42:16.996517+0000', 'modified': '2024-04-21T08:42:42.080753+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24533}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24532': {'gid': 24532, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6839/516970569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 516970569}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 516970569}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6839/1703654929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 1703654929}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 1703654929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24533, 'qdb_cluster': [24533]} 2024-04-21T08:44:07.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T08:44:07.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:44:07.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.446+0000 7fe4dbe88640 1 -- 172.21.15.7:0/2248517189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4d4100ea0 msgr2=0x7fe4d41012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.446+0000 7fe4dbe88640 1 --2- 172.21.15.7:0/2248517189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4d4100ea0 0x7fe4d41012c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009900 tx=0x7fe4c402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.446+0000 7fe4dbe88640 1 -- 172.21.15.7:0/2248517189 shutdown_connections 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.446+0000 7fe4dbe88640 1 --2- 172.21.15.7:0/2248517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d41029b0 0x7fe4d41092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.446+0000 7fe4dbe88640 1 --2- 172.21.15.7:0/2248517189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d4101ff0 0x7fe4d4102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.446+0000 7fe4dbe88640 1 --2- 172.21.15.7:0/2248517189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4d4100ea0 0x7fe4d41012c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.446+0000 7fe4dbe88640 1 -- 172.21.15.7:0/2248517189 >> 172.21.15.7:0/2248517189 conn(0x7fe4d40fc430 msgr2=0x7fe4d40fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.446+0000 7fe4dbe88640 1 -- 172.21.15.7:0/2248517189 shutdown_connections 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.446+0000 7fe4dbe88640 1 -- 172.21.15.7:0/2248517189 wait complete. 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.446+0000 7fe4dbe88640 1 Processor -- start 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.446+0000 7fe4dbe88640 1 -- start start 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4dbe88640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4d4100ea0 0x7fe4d41981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4dbe88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d4101ff0 0x7fe4d4198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4dbe88640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d41029b0 0x7fe4d419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4dbe88640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d410ca40 con 0x7fe4d4100ea0 2024-04-21T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4dbe88640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d410c8c0 con 0x7fe4d4101ff0 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4dbe88640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d410cbc0 con 0x7fe4d41029b0 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4d9c24640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4d4100ea0 0x7fe4d41981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4d9c24640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4d4100ea0 0x7fe4d41981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41138/0 (socket says 172.21.15.7:41138) 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4d9c24640 1 -- 172.21.15.7:0/3792838734 learned_addr learned my addr 172.21.15.7:0/3792838734 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4da425640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d41029b0 0x7fe4d419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4d9c24640 1 -- 172.21.15.7:0/3792838734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d41029b0 msgr2=0x7fe4d419f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d41029b0 0x7fe4d419f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4d9c24640 1 -- 172.21.15.7:0/3792838734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d4101ff0 msgr2=0x7fe4d4198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4d9423640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d4101ff0 0x7fe4d4198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d4101ff0 0x7fe4d4198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4d9c24640 1 -- 172.21.15.7:0/3792838734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d419fe00 con 0x7fe4d4100ea0 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4da425640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d41029b0 0x7fe4d419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4d4100ea0 0x7fe4d41981e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4000c00 tx=0x7fe4c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4caffd640 1 -- 172.21.15.7:0/3792838734 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4043020 con 0x7fe4d4100ea0 2024-04-21T08:44:07.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4caffd640 1 -- 172.21.15.7:0/3792838734 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c4004aa0 con 0x7fe4d4100ea0 2024-04-21T08:44:07.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4dbe88640 1 -- 172.21.15.7:0/3792838734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d41a0090 con 0x7fe4d4100ea0 2024-04-21T08:44:07.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4caffd640 1 -- 172.21.15.7:0/3792838734 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4003d70 con 0x7fe4d4100ea0 2024-04-21T08:44:07.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.447+0000 7fe4dbe88640 1 -- 172.21.15.7:0/3792838734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d41a05e0 con 0x7fe4d4100ea0 2024-04-21T08:44:07.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.449+0000 7fe4caffd640 1 -- 172.21.15.7:0/3792838734 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fe4c4046020 con 0x7fe4d4100ea0 2024-04-21T08:44:07.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4dea7f740 1 Processor -- start 2024-04-21T08:44:07.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4dea7f740 1 -- start start 2024-04-21T08:44:07.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c189e40 0x55bd9c18a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18a830 0x55bd9c18acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18b2d0 0x55bd9c191eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c0eeb10 con 0x55bd9c18b2d0 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c08c240 con 0x55bd9c189e40 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c0eaad0 con 0x55bd9c18a830 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4d9423640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18b2d0 0x55bd9c191eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4d9423640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18b2d0 0x55bd9c191eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41150/0 (socket says 172.21.15.7:41150) 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4d9423640 1 -- 172.21.15.7:0/3314232912 learned_addr learned my addr 172.21.15.7:0/3314232912 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/3314232912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18a830 0x55bd9c18acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4da425640 1 --2- 172.21.15.7:0/3314232912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c189e40 0x55bd9c18a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4caffd640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4c00a3c40 0x7fe4c00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4d9423640 1 -- 172.21.15.7:0/3314232912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18a830 msgr2=0x55bd9c18acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4d9423640 1 --2- 172.21.15.7:0/3314232912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18a830 0x55bd9c18acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4d9423640 1 -- 172.21.15.7:0/3314232912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c189e40 msgr2=0x55bd9c18a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4d9423640 1 --2- 172.21.15.7:0/3314232912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c189e40 0x55bd9c18a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.450+0000 7fe4d9423640 1 -- 172.21.15.7:0/3314232912 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd9be012d0 con 0x55bd9c18b2d0 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.451+0000 7fe4da425640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4c00a3c40 0x7fe4c00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.451+0000 7fe4caffd640 1 -- 172.21.15.7:0/3792838734 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fe4c40f2bd0 con 0x7fe4d4100ea0 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.451+0000 7fe4d9423640 1 --2- 172.21.15.7:0/3314232912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18b2d0 0x55bd9c191eb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009c00 tx=0x7fe4d0010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb42833faa67845e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.451+0000 7fe4bffff640 1 -- 172.21.15.7:0/3314232912 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0004020 con 0x55bd9c18b2d0 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.451+0000 7fe4bffff640 1 -- 172.21.15.7:0/3314232912 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4d00054d0 con 0x55bd9c18b2d0 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.451+0000 7fe4bffff640 1 -- 172.21.15.7:0/3314232912 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0004b80 con 0x55bd9c18b2d0 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4da425640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4c00a3c40 0x7fe4c00a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc005c40 tx=0x7fe4cc004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4dea7f740 1 -- 172.21.15.7:0/3314232912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18b2d0 msgr2=0x55bd9c191eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/3314232912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18b2d0 0x55bd9c191eb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009c00 tx=0x7fe4d0010740 comp rx=0 tx=0).stop 2024-04-21T08:44:07.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4dea7f740 1 -- 172.21.15.7:0/3314232912 shutdown_connections 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/3314232912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18b2d0 0x55bd9c191eb0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/3314232912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18a830 0x55bd9c18acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/3314232912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c189e40 0x55bd9c18a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4dea7f740 1 -- 172.21.15.7:0/3314232912 >> 172.21.15.7:0/3314232912 conn(0x55bd9c185f40 msgr2=0x55bd9c184490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4dea7f740 1 -- 172.21.15.7:0/3314232912 shutdown_connections 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4dea7f740 1 -- 172.21.15.7:0/3314232912 wait complete. 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4dea7f740 1 -- start start 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.452+0000 7fe4dea7f740 1 Processor -- start 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c189e40 0x55bd9c193e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c194390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18b2d0 0x55bd9c19b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c1924b0 con 0x55bd9c18a830 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c077190 con 0x55bd9c189e40 2024-04-21T08:44:07.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c192630 con 0x55bd9c18b2d0 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4dea7f740 10 client.0 _ll_register_callbacks cb 0x55bd9c0728b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4d9423640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c194390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4d9423640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c194390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41156/0 (socket says 172.21.15.7:41156) 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4d9423640 1 -- 172.21.15.7:0/2294430011 learned_addr learned my addr 172.21.15.7:0/2294430011 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4da425640 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18b2d0 0x55bd9c19b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c189e40 0x55bd9c193e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4d9423640 1 -- 172.21.15.7:0/2294430011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18b2d0 msgr2=0x55bd9c19b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4d9423640 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18b2d0 0x55bd9c19b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4d9423640 1 -- 172.21.15.7:0/2294430011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c189e40 msgr2=0x55bd9c193e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4d9423640 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c189e40 0x55bd9c193e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4d9423640 1 -- 172.21.15.7:0/2294430011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd9becb6b0 con 0x55bd9c18a830 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c189e40 0x55bd9c193e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4da425640 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18b2d0 0x55bd9c19b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4d9423640 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c194390 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00057c0 tx=0x7fe4d0013710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9359794060eb70d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/2294430011 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d002a060 con 0x55bd9c18a830 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/2294430011 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fe4d0013d40 con 0x55bd9c18a830 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4dea7f740 1 -- 172.21.15.7:0/2294430011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4d00054d0 con 0x55bd9c18a830 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/2294430011 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe4d0031050 con 0x55bd9c18a830 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4dea7f740 10 client.15240 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.453+0000 7fe4dea7f740 1 -- 172.21.15.7:0/2294430011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bd9c19e7b0 con 0x55bd9c18a830 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.454+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/2294430011 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4d00054d0 con 0x55bd9c18a830 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.454+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/2294430011 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fe4d000f040 con 0x55bd9c18a830 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.454+0000 7fe4dea7f740 10 client.15240 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.454+0000 7fe4dea7f740 10 client.15240 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.454+0000 7fe4dea7f740 20 client.15240 populate_metadata read hostname 'smithi007' 2024-04-21T08:44:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.454+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bd9c1a0c30 0x55bd9c1a20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.454+0000 7fe4dea7f740 4 client.15240 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.454+0000 7fe4dea7f740 1 -- 172.21.15.7:0/2294430011 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd9c0fc530 con 0x55bd9c1a0c30 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.454+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bd9c1a0c30 0x55bd9c1a20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.455+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bd9c1a0c30 0x55bd9c1a20a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8c42a658a69a0ad server_cookie=6f96c02165e668e4 in_seq=0 out_seq=0 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.455+0000 7fe4bd7fa640 10 client.15240 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.455+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/2294430011 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd9c0fc530 con 0x55bd9c1a0c30 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.455+0000 7fe4bd7fa640 10 client.15240 handle_command_reply: tid=0 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.455+0000 7fe4dea7f740 1 client.15240 shutdown 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.455+0000 7fe4dea7f740 1 -- 172.21.15.7:0/2294430011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 msgr2=0x55bd9c194390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.455+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c194390 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00057c0 tx=0x7fe4d0013710 comp rx=0 tx=0).stop 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.456+0000 7fe4dea7f740 1 -- 172.21.15.7:0/2294430011 shutdown_connections 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.456+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bd9c1a0c30 0x55bd9c1a20a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.456+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18b2d0 0x55bd9c19b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.456+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c194390 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.456+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/2294430011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c189e40 0x55bd9c193e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.456+0000 7fe4bd7fa640 0 client.15240 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.456+0000 7fe4dea7f740 1 -- 172.21.15.7:0/2294430011 >> 172.21.15.7:0/2294430011 conn(0x55bd9c185f40 msgr2=0x55bd9c183a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.456+0000 7fe4dea7f740 1 -- 172.21.15.7:0/2294430011 shutdown_connections 2024-04-21T08:44:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.456+0000 7fe4dea7f740 1 -- 172.21.15.7:0/2294430011 wait complete. 2024-04-21T08:44:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.456+0000 7fe4dea7f740 20 client.15240 trim_cache size 0 max 16384 2024-04-21T08:44:07.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4dea7f740 1 Processor -- start 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4dea7f740 1 -- start start 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c189e40 0x55bd9c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c17f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 0x55bd9c191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c1a03e0 con 0x55bd9c18a830 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c1a0ab0 con 0x55bd9c18b2d0 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c1a1180 con 0x55bd9c189e40 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4d9423640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c17f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4d9423640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c17f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41172/0 (socket says 172.21.15.7:41172) 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4d9423640 1 -- 172.21.15.7:0/867476214 learned_addr learned my addr 172.21.15.7:0/867476214 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/867476214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c189e40 0x55bd9c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4d9423640 1 -- 172.21.15.7:0/867476214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c189e40 msgr2=0x55bd9c19fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4da425640 1 --2- 172.21.15.7:0/867476214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 0x55bd9c191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4d9423640 1 --2- 172.21.15.7:0/867476214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c189e40 0x55bd9c19fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4d9423640 1 -- 172.21.15.7:0/867476214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 msgr2=0x55bd9c191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4d9423640 1 --2- 172.21.15.7:0/867476214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 0x55bd9c191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4d9423640 1 -- 172.21.15.7:0/867476214 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd9c194cf0 con 0x55bd9c18a830 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/867476214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c189e40 0x55bd9c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.474+0000 7fe4da425640 1 --2- 172.21.15.7:0/867476214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 0x55bd9c191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4d9423640 1 --2- 172.21.15.7:0/867476214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c17f800 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0001eb0 tx=0x7fe4d0010cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7061bae07f0d339 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4bf7fe640 1 -- 172.21.15.7:0/867476214 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d002a070 con 0x55bd9c18a830 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4bf7fe640 1 -- 172.21.15.7:0/867476214 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4d00116b0 con 0x55bd9c18a830 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4bf7fe640 1 -- 172.21.15.7:0/867476214 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0005460 con 0x55bd9c18a830 2024-04-21T08:44:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4dea7f740 1 -- 172.21.15.7:0/867476214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 msgr2=0x55bd9c17f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/867476214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c17f800 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0001eb0 tx=0x7fe4d0010cf0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4dea7f740 1 -- 172.21.15.7:0/867476214 shutdown_connections 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/867476214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 0x55bd9c191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/867476214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c18a830 0x55bd9c17f800 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/867476214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c189e40 0x55bd9c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4dea7f740 1 -- 172.21.15.7:0/867476214 >> 172.21.15.7:0/867476214 conn(0x55bd9c185f40 msgr2=0x55bd9c184400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4dea7f740 1 -- 172.21.15.7:0/867476214 shutdown_connections 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.475+0000 7fe4dea7f740 1 -- 172.21.15.7:0/867476214 wait complete. 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4dea7f740 1 -- start start 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4dea7f740 1 Processor -- start 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c189e40 0x55bd9c196560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18a830 0x55bd9c196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4dea7f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 0x55bd9c199410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c1a1f20 con 0x55bd9c189e40 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c1a1850 con 0x55bd9c18b2d0 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4dea7f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd9c1a25f0 con 0x55bd9c18a830 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4dea7f740 10 client.0 _ll_register_callbacks cb 0x7fe4d0009a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4d9c24640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c189e40 0x55bd9c196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4d9c24640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c189e40 0x55bd9c196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41176/0 (socket says 172.21.15.7:41176) 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4d9c24640 1 -- 172.21.15.7:0/696164917 learned_addr learned my addr 172.21.15.7:0/696164917 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4da425640 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 0x55bd9c199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4d9c24640 1 -- 172.21.15.7:0/696164917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18a830 msgr2=0x55bd9c196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18a830 0x55bd9c196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4d9c24640 1 -- 172.21.15.7:0/696164917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 msgr2=0x55bd9c199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 0x55bd9c199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4d9c24640 1 -- 172.21.15.7:0/696164917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd9c199a00 con 0x55bd9c189e40 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.476+0000 7fe4da425640 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 0x55bd9c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4d9c24640 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c189e40 0x55bd9c196560 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe4c402c6a0 tx=0x7fe4c4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438c00892abf8e7c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/696164917 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40bf200 con 0x55bd9c189e40 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/696164917 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fe4c40bcba0 con 0x55bd9c189e40 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4dea7f740 1 -- 172.21.15.7:0/696164917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4d00116b0 con 0x55bd9c189e40 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/696164917 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe4c40c7770 con 0x55bd9c189e40 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4dea7f740 10 client.15246 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4dea7f740 1 -- 172.21.15.7:0/696164917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bd9c19ac70 con 0x55bd9c189e40 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/696164917 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4d00116b0 con 0x55bd9c189e40 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/696164917 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fe4c40bb170 con 0x55bd9c189e40 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4dea7f740 10 client.15246 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4dea7f740 10 client.15246 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4dea7f740 20 client.15246 populate_metadata read hostname 'smithi007' 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bd9c20b6e0 0x55bd9c20a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4dea7f740 4 client.15246 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.477+0000 7fe4dea7f740 1 -- 172.21.15.7:0/696164917 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "status"}) -- 0x55bd9c0f63a0 con 0x55bd9c20b6e0 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.478+0000 7fe4d9423640 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bd9c20b6e0 0x55bd9c20a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.478+0000 7fe4bd7fa640 10 client.15246 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.478+0000 7fe4d9423640 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bd9c20b6e0 0x55bd9c20a3f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55524887c4e7ed6e server_cookie=8dee3ed17105ba30 in_seq=0 out_seq=0 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.478+0000 7fe4bd7fa640 1 -- 172.21.15.7:0/696164917 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55bd9c0f63a0 con 0x55bd9c20b6e0 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.478+0000 7fe4bd7fa640 10 client.15246 handle_command_reply: tid=0 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.478+0000 7fe4dea7f740 1 client.15246 shutdown 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.478+0000 7fe4dea7f740 1 -- 172.21.15.7:0/696164917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c189e40 msgr2=0x55bd9c196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.478+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c189e40 0x55bd9c196560 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe4c402c6a0 tx=0x7fe4c4009260 comp rx=0 tx=0).stop 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dea7f740 1 -- 172.21.15.7:0/696164917 shutdown_connections 2024-04-21T08:44:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bd9c20b6e0 0x55bd9c20a3f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bd9c18b2d0 0x55bd9c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd9c18a830 0x55bd9c196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dea7f740 1 --2- 172.21.15.7:0/696164917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd9c189e40 0x55bd9c196560 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4bd7fa640 0 client.15246 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dea7f740 1 -- 172.21.15.7:0/696164917 >> 172.21.15.7:0/696164917 conn(0x55bd9c185f40 msgr2=0x55bd9c19fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dea7f740 1 -- 172.21.15.7:0/696164917 shutdown_connections 2024-04-21T08:44:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dea7f740 1 -- 172.21.15.7:0/696164917 wait complete. 2024-04-21T08:44:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dea7f740 20 client.15246 trim_cache size 0 max 16384 2024-04-21T08:44:07.485 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:44:07.485 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "c9c60282-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:44:07.486 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2024-04-21T08:44:07.486 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24533, 2024-04-21T08:44:07.486 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-04-21T08:44:07.486 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-04-21T08:44:07.486 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-04-21T08:44:07.486 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 109.854463759, 2024-04-21T08:44:07.486 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 21, 2024-04-21T08:44:07.486 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 92, 2024-04-21T08:44:07.486 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T08:44:07.486 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 115.99594786199999 2024-04-21T08:44:07.486 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:44:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 -- 172.21.15.7:0/3792838734 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4c00a3c40 msgr2=0x7fe4c00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4c00a3c40 0x7fe4c00a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc005c40 tx=0x7fe4cc004a40 comp rx=0 tx=0).stop 2024-04-21T08:44:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 -- 172.21.15.7:0/3792838734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4d4100ea0 msgr2=0x7fe4d41981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4d4100ea0 0x7fe4d41981e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4000c00 tx=0x7fe4c40042f0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 -- 172.21.15.7:0/3792838734 shutdown_connections 2024-04-21T08:44:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4c00a3c40 0x7fe4c00a6100 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d41029b0 0x7fe4d419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4d4101ff0 0x7fe4d4198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 --2- 172.21.15.7:0/3792838734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4d4100ea0 0x7fe4d41981e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 -- 172.21.15.7:0/3792838734 >> 172.21.15.7:0/3792838734 conn(0x7fe4d40fc430 msgr2=0x7fe4d40fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 -- 172.21.15.7:0/3792838734 shutdown_connections 2024-04-21T08:44:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:07.479+0000 7fe4dbe88640 1 -- 172.21.15.7:0/3792838734 wait complete. 2024-04-21T08:44:07.490 INFO:tasks.cephadm:Setup complete, yielding 2024-04-21T08:44:07.490 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T08:44:07.500 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-21T08:44:07.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-21T08:44:07.749 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:08.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:07 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1512741719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:08.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:07 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1176132222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:08.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:07 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1512741719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:08.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:07 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1176132222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:08.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:07 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1512741719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:07 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1176132222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:09.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:08 smithi016 ceph-mon[31259]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:09.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:08 smithi007 ceph-mon[28477]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:09.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:08 smithi077 ceph-mon[31615]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:09.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.494+0000 7f523d244700 1 -- 172.21.15.7:0/2434704506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f523810a9a0 msgr2=0x7f523810ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:09.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.494+0000 7f523d244700 1 --2- 172.21.15.7:0/2434704506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f523810a9a0 0x7f523810ce30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5230009990 tx=0x7f5230009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.494+0000 7f523d244700 1 -- 172.21.15.7:0/2434704506 shutdown_connections 2024-04-21T08:44:09.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.494+0000 7f523d244700 1 --2- 172.21.15.7:0/2434704506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f523810a9a0 0x7f523810ce30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.494+0000 7f523d244700 1 --2- 172.21.15.7:0/2434704506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5238076a70 0x7f5238074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.494+0000 7f523d244700 1 --2- 172.21.15.7:0/2434704506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52380754a0 0x7f5238075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.494+0000 7f523d244700 1 -- 172.21.15.7:0/2434704506 >> 172.21.15.7:0/2434704506 conn(0x7f52380fe090 msgr2=0x7f52381004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:09.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.494+0000 7f523d244700 1 -- 172.21.15.7:0/2434704506 shutdown_connections 2024-04-21T08:44:09.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.494+0000 7f523d244700 1 -- 172.21.15.7:0/2434704506 wait complete. 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.494+0000 7f523d244700 1 Processor -- start 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523d244700 1 -- start start 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523d244700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52380754a0 0x7f523819ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523d244700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5238076a70 0x7f523819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523d244700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f523810a9a0 0x7f52381a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523d244700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5238110800 con 0x7f523810a9a0 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523d244700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5238110680 con 0x7f52380754a0 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523d244700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5238110980 con 0x7f5238076a70 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523759e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f523810a9a0 0x7f52381a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523759e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f523810a9a0 0x7f52381a23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41180/0 (socket says 172.21.15.7:41180) 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523759e700 1 -- 172.21.15.7:0/3685383810 learned_addr learned my addr 172.21.15.7:0/3685383810 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f5236d9d700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52380754a0 0x7f523819ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:09.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523659c700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5238076a70 0x7f523819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523759e700 1 -- 172.21.15.7:0/3685383810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5238076a70 msgr2=0x7f523819b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523759e700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5238076a70 0x7f523819b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523759e700 1 -- 172.21.15.7:0/3685383810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52380754a0 msgr2=0x7f523819ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523759e700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52380754a0 0x7f523819ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f523759e700 1 -- 172.21.15.7:0/3685383810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52381a2a60 con 0x7f523810a9a0 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.495+0000 7f5236d9d700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52380754a0 0x7f523819ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.496+0000 7f523759e700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f523810a9a0 0x7f52381a23c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5230006010 tx=0x7f52300047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.496+0000 7f522ffff700 1 -- 172.21.15.7:0/3685383810 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230027070 con 0x7f523810a9a0 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.496+0000 7f523d244700 1 -- 172.21.15.7:0/3685383810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52381a2cf0 con 0x7f523810a9a0 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.496+0000 7f522ffff700 1 -- 172.21.15.7:0/3685383810 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5230004000 con 0x7f523810a9a0 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.496+0000 7f522ffff700 1 -- 172.21.15.7:0/3685383810 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230022d60 con 0x7f523810a9a0 2024-04-21T08:44:09.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.496+0000 7f523d244700 1 -- 172.21.15.7:0/3685383810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52381a31a0 con 0x7f523810a9a0 2024-04-21T08:44:09.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.497+0000 7f523d244700 1 -- 172.21.15.7:0/3685383810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5238069730 con 0x7f523810a9a0 2024-04-21T08:44:09.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.499+0000 7f522ffff700 1 -- 172.21.15.7:0/3685383810 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f523002a070 con 0x7f523810a9a0 2024-04-21T08:44:09.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.500+0000 7f522ffff700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52240a3cd0 0x7f52240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:09.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.500+0000 7f5236d9d700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52240a3cd0 0x7f52240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:09.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.500+0000 7f522ffff700 1 -- 172.21.15.7:0/3685383810 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f52300d16b0 con 0x7f523810a9a0 2024-04-21T08:44:09.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.500+0000 7f5236d9d700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52240a3cd0 0x7f52240a6190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5238076e50 tx=0x7f52280091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:09.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.503+0000 7f522ffff700 1 -- 172.21.15.7:0/3685383810 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5230013070 con 0x7f523810a9a0 2024-04-21T08:44:09.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.756+0000 7f522ffff700 1 -- 172.21.15.7:0/3685383810 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f5230099b70 con 0x7f523810a9a0 2024-04-21T08:44:09.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.769+0000 7f523d244700 1 -- 172.21.15.7:0/3685383810 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f5238062e80 con 0x7f52240a3cd0 2024-04-21T08:44:09.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.770+0000 7f522ffff700 1 -- 172.21.15.7:0/3685383810 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f5238062e80 con 0x7f52240a3cd0 2024-04-21T08:44:09.769 INFO:teuthology.orchestra.run.smithi007.stdout:Backend: cephadm 2024-04-21T08:44:09.769 INFO:teuthology.orchestra.run.smithi007.stdout:Available: Yes 2024-04-21T08:44:09.769 INFO:teuthology.orchestra.run.smithi007.stdout:Paused: No 2024-04-21T08:44:09.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 -- 172.21.15.7:0/3685383810 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52240a3cd0 msgr2=0x7f52240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:09.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52240a3cd0 0x7f52240a6190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5238076e50 tx=0x7f52280091b0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 -- 172.21.15.7:0/3685383810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f523810a9a0 msgr2=0x7f52381a23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:09.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f523810a9a0 0x7f52381a23c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5230006010 tx=0x7f52300047b0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 -- 172.21.15.7:0/3685383810 shutdown_connections 2024-04-21T08:44:09.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f52240a3cd0 0x7f52240a6190 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f523810a9a0 0x7f52381a23c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52380754a0 0x7f523819ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 --2- 172.21.15.7:0/3685383810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5238076a70 0x7f523819b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:09.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 -- 172.21.15.7:0/3685383810 >> 172.21.15.7:0/3685383810 conn(0x7f52380fe090 msgr2=0x7f52381068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:09.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 -- 172.21.15.7:0/3685383810 shutdown_connections 2024-04-21T08:44:09.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:09.772+0000 7f523d244700 1 -- 172.21.15.7:0/3685383810 wait complete. 2024-04-21T08:44:10.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-21T08:44:10.784 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:10 smithi016 ceph-mon[31259]: pgmap v352: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:10 smithi016 ceph-mon[31259]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T08:44:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:10 smithi016 ceph-mon[31259]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-21T08:44:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-04-21T08:44:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-04-21T08:44:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-04-21T08:44:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-21T08:44:11.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:10 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-04-21T08:44:11.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:10 smithi007 ceph-mon[28477]: pgmap v352: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:11.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:10 smithi007 ceph-mon[28477]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T08:44:11.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:10 smithi007 ceph-mon[28477]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:11.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-21T08:44:11.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-04-21T08:44:11.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-04-21T08:44:11.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-04-21T08:44:11.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-21T08:44:11.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:10 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-04-21T08:44:11.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:10 smithi077 ceph-mon[31615]: pgmap v352: 129 pgs: 129 active+clean; 639 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:11.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:10 smithi077 ceph-mon[31615]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T08:44:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:10 smithi077 ceph-mon[31615]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-21T08:44:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-04-21T08:44:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-04-21T08:44:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-04-21T08:44:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-04-21T08:44:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:10 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-04-21T08:44:12.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:11 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-21T08:44:12.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:11 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-04-21T08:44:12.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:11 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-04-21T08:44:12.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:11 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-04-21T08:44:12.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:11 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-21T08:44:12.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:11 smithi007 ceph-mon[28477]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-04-21T08:44:12.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:11 smithi007 ceph-mon[28477]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:44:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:11 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-21T08:44:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:11 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-04-21T08:44:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:11 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-04-21T08:44:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:11 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-04-21T08:44:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:11 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-21T08:44:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:11 smithi016 ceph-mon[31259]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-04-21T08:44:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:11 smithi016 ceph-mon[31259]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:44:12.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:11 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-21T08:44:12.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:11 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-04-21T08:44:12.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:11 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-04-21T08:44:12.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:11 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-04-21T08:44:12.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:11 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-04-21T08:44:12.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:11 smithi077 ceph-mon[31615]: from='mgr.14152 172.21.15.7:0/3674016393' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-04-21T08:44:12.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:11 smithi077 ceph-mon[31615]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:44:12.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.661+0000 7f278d6f4700 1 -- 172.21.15.7:0/1722923108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788102ac0 msgr2=0x7f2788102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:12.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.661+0000 7f278d6f4700 1 --2- 172.21.15.7:0/1722923108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788102ac0 0x7f2788102ee0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f27780099c0 tx=0x7f2778009cd0 comp rx=0 tx=0).stop 2024-04-21T08:44:12.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.661+0000 7f278d6f4700 1 -- 172.21.15.7:0/1722923108 shutdown_connections 2024-04-21T08:44:12.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.661+0000 7f278d6f4700 1 --2- 172.21.15.7:0/1722923108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788104660 0x7f278810af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:12.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.661+0000 7f278d6f4700 1 --2- 172.21.15.7:0/1722923108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788102ac0 0x7f2788102ee0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:12.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.661+0000 7f278d6f4700 1 --2- 172.21.15.7:0/1722923108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2788103cc0 0x7f2788104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:12.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.661+0000 7f278d6f4700 1 -- 172.21.15.7:0/1722923108 >> 172.21.15.7:0/1722923108 conn(0x7f27880fe060 msgr2=0x7f27881004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:12.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.661+0000 7f278d6f4700 1 -- 172.21.15.7:0/1722923108 shutdown_connections 2024-04-21T08:44:12.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.661+0000 7f278d6f4700 1 -- 172.21.15.7:0/1722923108 wait complete. 2024-04-21T08:44:12.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.662+0000 7f278d6f4700 1 Processor -- start 2024-04-21T08:44:12.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.662+0000 7f278d6f4700 1 -- start start 2024-04-21T08:44:12.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f278d6f4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788102ac0 0x7f2788198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:12.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f278d6f4700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788103cc0 0x7f2788199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:12.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f278d6f4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2788104660 0x7f27881a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:12.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f278d6f4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f278810e5b0 con 0x7f2788102ac0 2024-04-21T08:44:12.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f278d6f4700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f278810e430 con 0x7f2788103cc0 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f278d6f4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f278810e730 con 0x7f2788104660 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f2786ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788102ac0 0x7f2788198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f2786ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788102ac0 0x7f2788198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41198/0 (socket says 172.21.15.7:41198) 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f2786ffd700 1 -- 172.21.15.7:0/3803944345 learned_addr learned my addr 172.21.15.7:0/3803944345 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f27877fe700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2788104660 0x7f27881a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f27867fc700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788103cc0 0x7f2788199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f2786ffd700 1 -- 172.21.15.7:0/3803944345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2788104660 msgr2=0x7f27881a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f2786ffd700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2788104660 0x7f27881a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f2786ffd700 1 -- 172.21.15.7:0/3803944345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788103cc0 msgr2=0x7f2788199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f2786ffd700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788103cc0 0x7f2788199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f2786ffd700 1 -- 172.21.15.7:0/3803944345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27881a0830 con 0x7f2788102ac0 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.663+0000 7f2786ffd700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788102ac0 0x7f2788198bd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f277800b580 tx=0x7f27780057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.664+0000 7f276ffff700 1 -- 172.21.15.7:0/3803944345 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778027070 con 0x7f2788102ac0 2024-04-21T08:44:12.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.664+0000 7f278d6f4700 1 -- 172.21.15.7:0/3803944345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27881a0ac0 con 0x7f2788102ac0 2024-04-21T08:44:12.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.664+0000 7f276ffff700 1 -- 172.21.15.7:0/3803944345 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2778004160 con 0x7f2788102ac0 2024-04-21T08:44:12.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.664+0000 7f276ffff700 1 -- 172.21.15.7:0/3803944345 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778022da0 con 0x7f2788102ac0 2024-04-21T08:44:12.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.664+0000 7f278d6f4700 1 -- 172.21.15.7:0/3803944345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27881a0ff0 con 0x7f2788102ac0 2024-04-21T08:44:12.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.665+0000 7f278d6f4700 1 -- 172.21.15.7:0/3803944345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2788069730 con 0x7f2788102ac0 2024-04-21T08:44:12.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.666+0000 7f276ffff700 1 -- 172.21.15.7:0/3803944345 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f277802a070 con 0x7f2788102ac0 2024-04-21T08:44:12.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.666+0000 7f276ffff700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f27700a3c40 0x7f27700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:12.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.667+0000 7f27867fc700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f27700a3c40 0x7f27700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:12.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.667+0000 7f276ffff700 1 -- 172.21.15.7:0/3803944345 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f27780d1580 con 0x7f2788102ac0 2024-04-21T08:44:12.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.667+0000 7f27867fc700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f27700a3c40 0x7f27700a6100 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f277c00ac50 tx=0x7f277c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:12.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.671+0000 7f276ffff700 1 -- 172.21.15.7:0/3803944345 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2778013070 con 0x7f2788102ac0 2024-04-21T08:44:12.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.890+0000 7f278d6f4700 1 -- 172.21.15.7:0/3803944345 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f2788062e80 con 0x7f27700a3c40 2024-04-21T08:44:12.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:12 smithi007 ceph-mon[28477]: pgmap v354: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-04-21T08:44:12.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:12 smithi007 ceph-mon[28477]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:44:12.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.906+0000 7f276ffff700 1 -- 172.21.15.7:0/3803944345 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7f2788062e80 con 0x7f27700a3c40 2024-04-21T08:44:12.906 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-21T08:44:12.906 INFO:teuthology.orchestra.run.smithi007.stdout:mds.a smithi007 running (2m) 23s ago 2m 16.2M - 19.0.0-3154-ga630765c b45a994450e2 7efc3f16e9ed 2024-04-21T08:44:12.906 INFO:teuthology.orchestra.run.smithi007.stdout:mds.b smithi016 running (2m) 55s ago 2m 25.8M - 19.0.0-3154-ga630765c b45a994450e2 e079e6b26092 2024-04-21T08:44:12.906 INFO:teuthology.orchestra.run.smithi007.stdout:mds.c smithi077 running (2m) 55s ago 2m 13.0M - 19.0.0-3154-ga630765c b45a994450e2 73c3870a11e1 2024-04-21T08:44:12.906 INFO:teuthology.orchestra.run.smithi007.stdout:mds.d smithi007 running (2m) 23s ago 2m 16.2M - 19.0.0-3154-ga630765c b45a994450e2 6812fd59cc5d 2024-04-21T08:44:12.906 INFO:teuthology.orchestra.run.smithi007.stdout:mds.e smithi016 running (2m) 55s ago 2m 13.1M - 19.0.0-3154-ga630765c b45a994450e2 01e19dd24fe9 2024-04-21T08:44:12.906 INFO:teuthology.orchestra.run.smithi007.stdout:mds.f smithi077 running (2m) 55s ago 2m 13.7M - 19.0.0-3154-ga630765c b45a994450e2 decaa5941cd0 2024-04-21T08:44:12.906 INFO:teuthology.orchestra.run.smithi007.stdout:mds.g smithi007 running (2m) 23s ago 2m 13.0M - 19.0.0-3154-ga630765c b45a994450e2 0cca774d80e4 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:mds.h smithi016 running (2m) 55s ago 2m 16.2M - 19.0.0-3154-ga630765c b45a994450e2 937012de4432 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:mds.i smithi077 running (2m) 55s ago 2m 19.2M - 19.0.0-3154-ga630765c b45a994450e2 92989efcacbc 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:mds.j smithi007 running (2m) 23s ago 2m 15.4M - 19.0.0-3154-ga630765c b45a994450e2 b54239fdaa87 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:mds.k smithi016 running (2m) 55s ago 2m 13.0M - 19.0.0-3154-ga630765c b45a994450e2 06fcf0ad3eaa 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:mds.l smithi077 running (2m) 55s ago 2m 13.2M - 19.0.0-3154-ga630765c b45a994450e2 ae73de27efd7 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi007 *:9283,8765 running (9m) 23s ago 9m 514M - 19.0.0-3154-ga630765c b45a994450e2 d8b0a57a71a3 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi016 *:8443,8765 running (7m) 55s ago 7m 455M - 19.0.0-3154-ga630765c b45a994450e2 615c421659eb 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.z smithi077 *:8443,8765 running (7m) 55s ago 7m 454M - 19.0.0-3154-ga630765c b45a994450e2 5e0b032c4a9e 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (9m) 23s ago 9m 53.2M 2048M 19.0.0-3154-ga630765c b45a994450e2 1e1e16479767 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi016 running (8m) 55s ago 8m 40.0M 2048M 19.0.0-3154-ga630765c b45a994450e2 b174a6eca6ab 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi077 running (8m) 55s ago 8m 39.7M 2048M 19.0.0-3154-ga630765c b45a994450e2 b616f89a2b7f 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (7m) 23s ago 7m 56.3M 4273M 19.0.0-3154-ga630765c b45a994450e2 8bf7f93ee52e 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi016 running (7m) 55s ago 7m 57.4M 4273M 19.0.0-3154-ga630765c b45a994450e2 23610ec72877 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi077 running (6m) 55s ago 6m 46.6M 4272M 19.0.0-3154-ga630765c b45a994450e2 60ea20e31cd8 2024-04-21T08:44:12.907 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (6m) 23s ago 6m 47.1M 4273M 19.0.0-3154-ga630765c b45a994450e2 72d3379f44a0 2024-04-21T08:44:12.908 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi016 running (5m) 55s ago 5m 55.9M 4273M 19.0.0-3154-ga630765c b45a994450e2 a0452c7b5e5b 2024-04-21T08:44:12.908 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi077 running (5m) 55s ago 5m 45.2M 4272M 19.0.0-3154-ga630765c b45a994450e2 b3032ec765f8 2024-04-21T08:44:12.908 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi007 running (5m) 23s ago 5m 53.2M 4273M 19.0.0-3154-ga630765c b45a994450e2 a44a62def04a 2024-04-21T08:44:12.908 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi016 running (4m) 55s ago 4m 54.1M 4273M 19.0.0-3154-ga630765c b45a994450e2 c1bbed5ce7fd 2024-04-21T08:44:12.908 INFO:teuthology.orchestra.run.smithi007.stdout:osd.8 smithi077 running (4m) 55s ago 4m 55.5M 4272M 19.0.0-3154-ga630765c b45a994450e2 078c4b0dcb59 2024-04-21T08:44:12.908 INFO:teuthology.orchestra.run.smithi007.stdout:osd.9 smithi007 running (3m) 23s ago 3m 52.4M 4273M 19.0.0-3154-ga630765c b45a994450e2 fd7fd4000126 2024-04-21T08:44:12.908 INFO:teuthology.orchestra.run.smithi007.stdout:osd.10 smithi016 running (3m) 55s ago 3m 56.0M 4273M 19.0.0-3154-ga630765c b45a994450e2 fe0c2279ec16 2024-04-21T08:44:12.908 INFO:teuthology.orchestra.run.smithi007.stdout:osd.11 smithi077 running (2m) 55s ago 2m 51.2M 4272M 19.0.0-3154-ga630765c b45a994450e2 f3266d0060c0 2024-04-21T08:44:12.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 -- 172.21.15.7:0/3803944345 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f27700a3c40 msgr2=0x7f27700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:12.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f27700a3c40 0x7f27700a6100 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f277c00ac50 tx=0x7f277c00a380 comp rx=0 tx=0).stop 2024-04-21T08:44:12.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 -- 172.21.15.7:0/3803944345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788102ac0 msgr2=0x7f2788198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:12.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788102ac0 0x7f2788198bd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f277800b580 tx=0x7f27780057a0 comp rx=0 tx=0).stop 2024-04-21T08:44:12.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 -- 172.21.15.7:0/3803944345 shutdown_connections 2024-04-21T08:44:12.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f27700a3c40 0x7f27700a6100 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:12.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2788104660 0x7f27881a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:12.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2788102ac0 0x7f2788198bd0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:12.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 --2- 172.21.15.7:0/3803944345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2788103cc0 0x7f2788199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:12.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 -- 172.21.15.7:0/3803944345 >> 172.21.15.7:0/3803944345 conn(0x7f27880fe060 msgr2=0x7f2788107040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:12.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 -- 172.21.15.7:0/3803944345 shutdown_connections 2024-04-21T08:44:12.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:12.909+0000 7f278d6f4700 1 -- 172.21.15.7:0/3803944345 wait complete. 2024-04-21T08:44:13.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:12 smithi016 ceph-mon[31259]: pgmap v354: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-04-21T08:44:13.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:12 smithi016 ceph-mon[31259]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:44:13.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:12 smithi077 ceph-mon[31615]: pgmap v354: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-04-21T08:44:13.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:12 smithi077 ceph-mon[31615]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:44:13.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-21T08:44:13.890 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:13.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:13 smithi007 ceph-mon[28477]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:14.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:13 smithi016 ceph-mon[31259]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:14.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:13 smithi077 ceph-mon[31615]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:15.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:14 smithi007 ceph-mon[28477]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:15.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:14 smithi016 ceph-mon[31259]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:15.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:14 smithi077 ceph-mon[31615]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.631+0000 7fa254653700 1 -- 172.21.15.7:0/1129041405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c103ca0 msgr2=0x7fa24c104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.631+0000 7fa254653700 1 --2- 172.21.15.7:0/1129041405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c103ca0 0x7fa24c104100 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa23c0099c0 tx=0x7fa23c009cd0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.631+0000 7fa254653700 1 -- 172.21.15.7:0/1129041405 shutdown_connections 2024-04-21T08:44:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.631+0000 7fa254653700 1 --2- 172.21.15.7:0/1129041405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa24c102aa0 0x7fa24c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.631+0000 7fa254653700 1 --2- 172.21.15.7:0/1129041405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa24c104640 0x7fa24c10aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.631+0000 7fa254653700 1 --2- 172.21.15.7:0/1129041405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c103ca0 0x7fa24c104100 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.631+0000 7fa254653700 1 -- 172.21.15.7:0/1129041405 >> 172.21.15.7:0/1129041405 conn(0x7fa24c0fe040 msgr2=0x7fa24c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.631+0000 7fa254653700 1 -- 172.21.15.7:0/1129041405 shutdown_connections 2024-04-21T08:44:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.631+0000 7fa254653700 1 -- 172.21.15.7:0/1129041405 wait complete. 2024-04-21T08:44:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa254653700 1 Processor -- start 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa254653700 1 -- start start 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa254653700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa24c102aa0 0x7fa24c0767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa254653700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c103ca0 0x7fa24c074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa254653700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa24c104640 0x7fa24c075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa254653700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c10e510 con 0x7fa24c103ca0 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa254653700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c10e390 con 0x7fa24c104640 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa254653700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c10e690 con 0x7fa24c102aa0 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa251bee700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c103ca0 0x7fa24c074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa251bee700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c103ca0 0x7fa24c074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55016/0 (socket says 172.21.15.7:55016) 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa251bee700 1 -- 172.21.15.7:0/183282054 learned_addr learned my addr 172.21.15.7:0/183282054 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa252bf0700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa24c104640 0x7fa24c075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.632+0000 7fa2523ef700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa24c102aa0 0x7fa24c0767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa251bee700 1 -- 172.21.15.7:0/183282054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa24c102aa0 msgr2=0x7fa24c0767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa251bee700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa24c102aa0 0x7fa24c0767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa251bee700 1 -- 172.21.15.7:0/183282054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa24c104640 msgr2=0x7fa24c075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa251bee700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa24c104640 0x7fa24c075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa251bee700 1 -- 172.21.15.7:0/183282054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24c075c20 con 0x7fa24c103ca0 2024-04-21T08:44:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa252bf0700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa24c104640 0x7fa24c075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa251bee700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c103ca0 0x7fa24c074e30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa23c003790 tx=0x7fa23c005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa2437fe700 1 -- 172.21.15.7:0/183282054 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23c027070 con 0x7fa24c103ca0 2024-04-21T08:44:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa254653700 1 -- 172.21.15.7:0/183282054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa24c1a2d10 con 0x7fa24c103ca0 2024-04-21T08:44:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa2437fe700 1 -- 172.21.15.7:0/183282054 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa23c0040e0 con 0x7fa24c103ca0 2024-04-21T08:44:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa2437fe700 1 -- 172.21.15.7:0/183282054 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23c022d60 con 0x7fa24c103ca0 2024-04-21T08:44:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.633+0000 7fa254653700 1 -- 172.21.15.7:0/183282054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa24c1a31a0 con 0x7fa24c103ca0 2024-04-21T08:44:15.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.634+0000 7fa254653700 1 -- 172.21.15.7:0/183282054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa24c069730 con 0x7fa24c103ca0 2024-04-21T08:44:15.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.635+0000 7fa2437fe700 1 -- 172.21.15.7:0/183282054 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fa23c02a070 con 0x7fa24c103ca0 2024-04-21T08:44:15.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.636+0000 7fa2437fe700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2380a3c40 0x7fa2380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:15.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.636+0000 7fa2437fe700 1 -- 172.21.15.7:0/183282054 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7fa23c0d1710 con 0x7fa24c103ca0 2024-04-21T08:44:15.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.636+0000 7fa2523ef700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2380a3c40 0x7fa2380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:15.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.636+0000 7fa2523ef700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2380a3c40 0x7fa2380a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa24c103b00 tx=0x7fa24800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:15.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.640+0000 7fa2437fe700 1 -- 172.21.15.7:0/183282054 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa23c013070 con 0x7fa24c103ca0 2024-04-21T08:44:15.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.857+0000 7fa254653700 1 -- 172.21.15.7:0/183282054 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fa24c062e80 con 0x7fa2380a3c40 2024-04-21T08:44:15.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.863+0000 7fa2437fe700 1 -- 172.21.15.7:0/183282054 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fa24c062e80 con 0x7fa2380a3c40 2024-04-21T08:44:15.863 INFO:teuthology.orchestra.run.smithi007.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-21T08:44:15.863 INFO:teuthology.orchestra.run.smithi007.stdout:mds.all 12/12 58s ago 2m smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi016=b;smithi016=e;smithi016=h;smithi016=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l;count:12 2024-04-21T08:44:15.863 INFO:teuthology.orchestra.run.smithi007.stdout:mgr 3/3 58s ago 8m smithi007=x;smithi016=y;smithi077=z;count:3 2024-04-21T08:44:15.863 INFO:teuthology.orchestra.run.smithi007.stdout:mon 3/3 58s ago 8m smithi007:172.21.15.7=a;smithi016:172.21.15.16=b;smithi077:172.21.15.77=c;count:3 2024-04-21T08:44:15.863 INFO:teuthology.orchestra.run.smithi007.stdout:osd 12 58s ago - 2024-04-21T08:44:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.865+0000 7fa254653700 1 -- 172.21.15.7:0/183282054 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2380a3c40 msgr2=0x7fa2380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.865+0000 7fa254653700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2380a3c40 0x7fa2380a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa24c103b00 tx=0x7fa24800a380 comp rx=0 tx=0).stop 2024-04-21T08:44:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.865+0000 7fa254653700 1 -- 172.21.15.7:0/183282054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c103ca0 msgr2=0x7fa24c074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.865+0000 7fa254653700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c103ca0 0x7fa24c074e30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa23c003790 tx=0x7fa23c005b00 comp rx=0 tx=0).stop 2024-04-21T08:44:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.865+0000 7fa254653700 1 -- 172.21.15.7:0/183282054 shutdown_connections 2024-04-21T08:44:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.865+0000 7fa254653700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fa2380a3c40 0x7fa2380a6100 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.865+0000 7fa254653700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa24c102aa0 0x7fa24c0767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.865+0000 7fa254653700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa24c104640 0x7fa24c075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.865+0000 7fa254653700 1 --2- 172.21.15.7:0/183282054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24c103ca0 0x7fa24c074e30 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.865+0000 7fa254653700 1 -- 172.21.15.7:0/183282054 >> 172.21.15.7:0/183282054 conn(0x7fa24c0fe040 msgr2=0x7fa24c107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.866+0000 7fa254653700 1 -- 172.21.15.7:0/183282054 shutdown_connections 2024-04-21T08:44:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:15.866+0000 7fa254653700 1 -- 172.21.15.7:0/183282054 wait complete. 2024-04-21T08:44:16.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-21T08:44:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:16.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:16 smithi007 ceph-mon[28477]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s 2024-04-21T08:44:16.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:16 smithi007 ceph-mon[28477]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:17.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:16 smithi016 ceph-mon[31259]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s 2024-04-21T08:44:17.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:16 smithi016 ceph-mon[31259]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:17.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:16 smithi077 ceph-mon[31615]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s 2024-04-21T08:44:17.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:16 smithi077 ceph-mon[31615]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.761+0000 7fc4164c0700 1 -- 172.21.15.7:0/1098083747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc410108790 msgr2=0x7fc41010ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.761+0000 7fc4164c0700 1 --2- 172.21.15.7:0/1098083747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc410108790 0x7fc41010ac00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc40800b0a0 tx=0x7fc40800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:44:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.761+0000 7fc4164c0700 1 -- 172.21.15.7:0/1098083747 shutdown_connections 2024-04-21T08:44:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.761+0000 7fc4164c0700 1 --2- 172.21.15.7:0/1098083747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc410108790 0x7fc41010ac00 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.761+0000 7fc4164c0700 1 --2- 172.21.15.7:0/1098083747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410103530 0x7fc410105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.761+0000 7fc4164c0700 1 --2- 172.21.15.7:0/1098083747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc410105e60 0x7fc410108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.761+0000 7fc4164c0700 1 -- 172.21.15.7:0/1098083747 >> 172.21.15.7:0/1098083747 conn(0x7fc4100fcf60 msgr2=0x7fc4100ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.761+0000 7fc4164c0700 1 -- 172.21.15.7:0/1098083747 shutdown_connections 2024-04-21T08:44:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.761+0000 7fc4164c0700 1 -- 172.21.15.7:0/1098083747 wait complete. 2024-04-21T08:44:18.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.762+0000 7fc4164c0700 1 Processor -- start 2024-04-21T08:44:18.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.762+0000 7fc4164c0700 1 -- start start 2024-04-21T08:44:18.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.762+0000 7fc4164c0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc410103530 0x7fc410198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.762+0000 7fc4164c0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc410105e60 0x7fc4101990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.762+0000 7fc4164c0700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410108790 0x7fc4101a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.762+0000 7fc4164c0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc41010e530 con 0x7fc410103530 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.762+0000 7fc4164c0700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc41010e3b0 con 0x7fc410108790 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.762+0000 7fc4164c0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc41010e6b0 con 0x7fc410105e60 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc410103530 0x7fc410198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc410103530 0x7fc410198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55034/0 (socket says 172.21.15.7:55034) 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40ffff700 1 -- 172.21.15.7:0/3636675587 learned_addr learned my addr 172.21.15.7:0/3636675587 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40f7fe700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc410105e60 0x7fc4101990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc414a5d700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410108790 0x7fc4101a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40ffff700 1 -- 172.21.15.7:0/3636675587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc410105e60 msgr2=0x7fc4101990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40ffff700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc410105e60 0x7fc4101990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40ffff700 1 -- 172.21.15.7:0/3636675587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410108790 msgr2=0x7fc4101a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:18.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40ffff700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410108790 0x7fc4101a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:18.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40ffff700 1 -- 172.21.15.7:0/3636675587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4101a07e0 con 0x7fc410103530 2024-04-21T08:44:18.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc414a5d700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410108790 0x7fc4101a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:18.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40ffff700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc410103530 0x7fc410198b80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800d980 tx=0x7fc3f800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:18.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40d7fa700 1 -- 172.21.15.7:0/3636675587 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8016070 con 0x7fc410103530 2024-04-21T08:44:18.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40d7fa700 1 -- 172.21.15.7:0/3636675587 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3f8010460 con 0x7fc410103530 2024-04-21T08:44:18.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc4164c0700 1 -- 172.21.15.7:0/3636675587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4101a0ad0 con 0x7fc410103530 2024-04-21T08:44:18.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc40d7fa700 1 -- 172.21.15.7:0/3636675587 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f800f620 con 0x7fc410103530 2024-04-21T08:44:18.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.763+0000 7fc4164c0700 1 -- 172.21.15.7:0/3636675587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4101a1060 con 0x7fc410103530 2024-04-21T08:44:18.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.765+0000 7fc4164c0700 1 -- 172.21.15.7:0/3636675587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc410069730 con 0x7fc410103530 2024-04-21T08:44:18.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.766+0000 7fc40d7fa700 1 -- 172.21.15.7:0/3636675587 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fc3f8019070 con 0x7fc410103530 2024-04-21T08:44:18.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.766+0000 7fc40d7fa700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc3fc0a3cc0 0x7fc3fc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:18.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.766+0000 7fc40d7fa700 1 -- 172.21.15.7:0/3636675587 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7fc3f80d0360 con 0x7fc410103530 2024-04-21T08:44:18.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.766+0000 7fc40f7fe700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc3fc0a3cc0 0x7fc3fc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:18.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.767+0000 7fc40f7fe700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc3fc0a3cc0 0x7fc3fc0a6180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc400005950 tx=0x7fc4000058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:18.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.771+0000 7fc40d7fa700 1 -- 172.21.15.7:0/3636675587 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3f809c930 con 0x7fc410103530 2024-04-21T08:44:18.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:18 smithi007 ceph-mon[28477]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.988+0000 7fc4164c0700 1 -- 172.21.15.7:0/3636675587 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fc410062f40 con 0x7fc3fc0a3cc0 2024-04-21T08:44:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.989+0000 7fc40d7fa700 1 -- 172.21.15.7:0/3636675587 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+187 (secure 0 0 0) 0x7fc410062f40 con 0x7fc3fc0a3cc0 2024-04-21T08:44:18.989 INFO:teuthology.orchestra.run.smithi007.stdout:HOST ADDR LABELS STATUS 2024-04-21T08:44:18.989 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 172.21.15.7 2024-04-21T08:44:18.989 INFO:teuthology.orchestra.run.smithi007.stdout:smithi016 172.21.15.16 2024-04-21T08:44:18.989 INFO:teuthology.orchestra.run.smithi007.stdout:smithi077 172.21.15.77 2024-04-21T08:44:18.989 INFO:teuthology.orchestra.run.smithi007.stdout:3 hosts in cluster 2024-04-21T08:44:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 -- 172.21.15.7:0/3636675587 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc3fc0a3cc0 msgr2=0x7fc3fc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc3fc0a3cc0 0x7fc3fc0a6180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc400005950 tx=0x7fc4000058e0 comp rx=0 tx=0).stop 2024-04-21T08:44:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 -- 172.21.15.7:0/3636675587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc410103530 msgr2=0x7fc410198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc410103530 0x7fc410198b80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800d980 tx=0x7fc3f800dd40 comp rx=0 tx=0).stop 2024-04-21T08:44:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 -- 172.21.15.7:0/3636675587 shutdown_connections 2024-04-21T08:44:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc3fc0a3cc0 0x7fc3fc0a6180 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410108790 0x7fc4101a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc410103530 0x7fc410198b80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 --2- 172.21.15.7:0/3636675587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc410105e60 0x7fc4101990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:18.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 -- 172.21.15.7:0/3636675587 >> 172.21.15.7:0/3636675587 conn(0x7fc4100fcf60 msgr2=0x7fc410103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:18.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 -- 172.21.15.7:0/3636675587 shutdown_connections 2024-04-21T08:44:18.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:18.991+0000 7fc4164c0700 1 -- 172.21.15.7:0/3636675587 wait complete. 2024-04-21T08:44:19.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:18 smithi016 ceph-mon[31259]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:19.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:18 smithi077 ceph-mon[31615]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:19.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-21T08:44:19.945 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:20.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:19 smithi016 ceph-mon[31259]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:20.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:19 smithi007 ceph-mon[28477]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:20.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:19 smithi077 ceph-mon[31615]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:21.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:20 smithi016 ceph-mon[31259]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 345 B/s wr, 2 op/s 2024-04-21T08:44:21.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:20 smithi007 ceph-mon[28477]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 345 B/s wr, 2 op/s 2024-04-21T08:44:21.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:20 smithi077 ceph-mon[31615]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 345 B/s wr, 2 op/s 2024-04-21T08:44:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.635+0000 7f7768751700 1 -- 172.21.15.7:0/3007139027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7760102aa0 msgr2=0x7f7760102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.635+0000 7f7768751700 1 --2- 172.21.15.7:0/3007139027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7760102aa0 0x7f7760102ec0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f775c00b0d0 tx=0x7f775c00b3e0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.635+0000 7f7768751700 1 -- 172.21.15.7:0/3007139027 shutdown_connections 2024-04-21T08:44:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.635+0000 7f7768751700 1 --2- 172.21.15.7:0/3007139027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760104640 0x7f776010aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.635+0000 7f7768751700 1 --2- 172.21.15.7:0/3007139027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7760102aa0 0x7f7760102ec0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.635+0000 7f7768751700 1 --2- 172.21.15.7:0/3007139027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7760103ca0 0x7f7760104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.635+0000 7f7768751700 1 -- 172.21.15.7:0/3007139027 >> 172.21.15.7:0/3007139027 conn(0x7f77600fe040 msgr2=0x7f7760100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.635+0000 7f7768751700 1 -- 172.21.15.7:0/3007139027 shutdown_connections 2024-04-21T08:44:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.635+0000 7f7768751700 1 -- 172.21.15.7:0/3007139027 wait complete. 2024-04-21T08:44:21.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f7768751700 1 Processor -- start 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f7768751700 1 -- start start 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f7768751700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7760102aa0 0x7f7760198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f7768751700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760103ca0 0x7f77601991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f7768751700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7760104640 0x7f77601a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f7768751700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f776010e530 con 0x7f7760102aa0 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f7768751700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f776010e3b0 con 0x7f7760103ca0 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f7768751700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f776010e6b0 con 0x7f7760104640 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f77664ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7760102aa0 0x7f7760198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f77664ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7760102aa0 0x7f7760198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55050/0 (socket says 172.21.15.7:55050) 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f77664ed700 1 -- 172.21.15.7:0/675018230 learned_addr learned my addr 172.21.15.7:0/675018230 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f7766cee700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7760104640 0x7f77601a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.636+0000 7f7765cec700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760103ca0 0x7f77601991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f77664ed700 1 -- 172.21.15.7:0/675018230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7760104640 msgr2=0x7f77601a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f77664ed700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7760104640 0x7f77601a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f77664ed700 1 -- 172.21.15.7:0/675018230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760103ca0 msgr2=0x7f77601991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f77664ed700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760103ca0 0x7f77601991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f77664ed700 1 -- 172.21.15.7:0/675018230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77601a08b0 con 0x7f7760102aa0 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f7766cee700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7760104640 0x7f77601a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f7765cec700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760103ca0 0x7f77601991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f77664ed700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7760102aa0 0x7f7760198c60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f775c009040 tx=0x7f775c004e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f77577fe700 1 -- 172.21.15.7:0/675018230 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c01f030 con 0x7f7760102aa0 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f77577fe700 1 -- 172.21.15.7:0/675018230 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f775c007b40 con 0x7f7760102aa0 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f7768751700 1 -- 172.21.15.7:0/675018230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77601a0b40 con 0x7f7760102aa0 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f77577fe700 1 -- 172.21.15.7:0/675018230 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c004720 con 0x7f7760102aa0 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.637+0000 7f7768751700 1 -- 172.21.15.7:0/675018230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77601a0ff0 con 0x7f7760102aa0 2024-04-21T08:44:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.638+0000 7f7768751700 1 -- 172.21.15.7:0/675018230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7760069730 con 0x7f7760102aa0 2024-04-21T08:44:21.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.639+0000 7f77577fe700 1 -- 172.21.15.7:0/675018230 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f775c003a00 con 0x7f7760102aa0 2024-04-21T08:44:21.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.640+0000 7f77577fe700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f774c0a3cc0 0x7f774c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:21.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.640+0000 7f7765cec700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f774c0a3cc0 0x7f774c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:21.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.640+0000 7f77577fe700 1 -- 172.21.15.7:0/675018230 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f775c00f020 con 0x7f7760102aa0 2024-04-21T08:44:21.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.641+0000 7f7765cec700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f774c0a3cc0 0x7f774c0a6180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f775000bcf0 tx=0x7f775000b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:21.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.644+0000 7f77577fe700 1 -- 172.21.15.7:0/675018230 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f775c029020 con 0x7f7760102aa0 2024-04-21T08:44:21.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.863+0000 7f7768751700 1 -- 172.21.15.7:0/675018230 --> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f7760062e80 con 0x7f774c0a3cc0 2024-04-21T08:44:21.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.864+0000 7f77577fe700 1 -- 172.21.15.7:0/675018230 <== mgr.14152 v2:172.21.15.7:6800/2555078032 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f7760062e80 con 0x7f774c0a3cc0 2024-04-21T08:44:21.864 INFO:teuthology.orchestra.run.smithi007.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-21T08:44:21.864 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300GE480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-21T08:44:21.864 INFO:teuthology.orchestra.run.smithi007.stdout:smithi016 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002C480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-21T08:44:21.864 INFO:teuthology.orchestra.run.smithi007.stdout:smithi077 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-21T08:44:21.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.866+0000 7f7768751700 1 -- 172.21.15.7:0/675018230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f774c0a3cc0 msgr2=0x7f774c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:21.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.866+0000 7f7768751700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f774c0a3cc0 0x7f774c0a6180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f775000bcf0 tx=0x7f775000b2c0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.867+0000 7f7768751700 1 -- 172.21.15.7:0/675018230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7760102aa0 msgr2=0x7f7760198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:21.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.867+0000 7f7768751700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7760102aa0 0x7f7760198c60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f775c009040 tx=0x7f775c004e00 comp rx=0 tx=0).stop 2024-04-21T08:44:21.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.867+0000 7f7768751700 1 -- 172.21.15.7:0/675018230 shutdown_connections 2024-04-21T08:44:21.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.867+0000 7f7768751700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f774c0a3cc0 0x7f774c0a6180 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.867+0000 7f7768751700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7760104640 0x7f77601a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.867+0000 7f7768751700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7760102aa0 0x7f7760198c60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.867+0000 7f7768751700 1 --2- 172.21.15.7:0/675018230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7760103ca0 0x7f77601991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.867+0000 7f7768751700 1 -- 172.21.15.7:0/675018230 >> 172.21.15.7:0/675018230 conn(0x7f77600fe040 msgr2=0x7f77601070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:21.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.867+0000 7f7768751700 1 -- 172.21.15.7:0/675018230 shutdown_connections 2024-04-21T08:44:21.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:21.867+0000 7f7768751700 1 -- 172.21.15.7:0/675018230 wait complete. 2024-04-21T08:44:22.689 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T08:44:22.699 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-21T08:44:22.699 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-21T08:44:22.955 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:22 smithi016 ceph-mon[31259]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 0 B/s wr, 2 op/s 2024-04-21T08:44:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:22 smithi016 ceph-mon[31259]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:23.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:22 smithi007 ceph-mon[28477]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 0 B/s wr, 2 op/s 2024-04-21T08:44:23.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:22 smithi007 ceph-mon[28477]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:23.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:22 smithi077 ceph-mon[31615]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 0 B/s wr, 2 op/s 2024-04-21T08:44:23.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:22 smithi077 ceph-mon[31615]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:24.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.633+0000 7f3160a60700 1 -- 172.21.15.7:0/3825937351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315c103ca0 msgr2=0x7f315c104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.633+0000 7f3160a60700 1 --2- 172.21.15.7:0/3825937351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315c103ca0 0x7f315c104100 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f31500099c0 tx=0x7f3150009cd0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.634+0000 7f3160a60700 1 -- 172.21.15.7:0/3825937351 shutdown_connections 2024-04-21T08:44:24.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.634+0000 7f3160a60700 1 --2- 172.21.15.7:0/3825937351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f315c102aa0 0x7f315c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.634+0000 7f3160a60700 1 --2- 172.21.15.7:0/3825937351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c104640 0x7f315c10aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.634+0000 7f3160a60700 1 --2- 172.21.15.7:0/3825937351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315c103ca0 0x7f315c104100 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.634+0000 7f3160a60700 1 -- 172.21.15.7:0/3825937351 >> 172.21.15.7:0/3825937351 conn(0x7f315c0fe040 msgr2=0x7f315c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:24.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.634+0000 7f3160a60700 1 -- 172.21.15.7:0/3825937351 shutdown_connections 2024-04-21T08:44:24.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.634+0000 7f3160a60700 1 -- 172.21.15.7:0/3825937351 wait complete. 2024-04-21T08:44:24.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f3160a60700 1 Processor -- start 2024-04-21T08:44:24.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f3160a60700 1 -- start start 2024-04-21T08:44:24.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f3160a60700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315c102aa0 0x7f315c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f3160a60700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c103ca0 0x7f315c199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f3160a60700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f315c104640 0x7f315c1a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f3160a60700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f315c10e510 con 0x7f315c102aa0 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f3160a60700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f315c10e390 con 0x7f315c103ca0 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f3160a60700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f315c10e690 con 0x7f315c104640 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f315a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315c102aa0 0x7f315c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f315a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315c102aa0 0x7f315c198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59432/0 (socket says 172.21.15.7:59432) 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f315a59c700 1 -- 172.21.15.7:0/1001343928 learned_addr learned my addr 172.21.15.7:0/1001343928 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f3159d9b700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c103ca0 0x7f315c199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f315ad9d700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f315c104640 0x7f315c1a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f315a59c700 1 -- 172.21.15.7:0/1001343928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f315c104640 msgr2=0x7f315c1a00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f315a59c700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f315c104640 0x7f315c1a00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f315a59c700 1 -- 172.21.15.7:0/1001343928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c103ca0 msgr2=0x7f315c199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f315a59c700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c103ca0 0x7f315c199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f315a59c700 1 -- 172.21.15.7:0/1001343928 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c1a0780 con 0x7f315c102aa0 2024-04-21T08:44:24.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.635+0000 7f3159d9b700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c103ca0 0x7f315c199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:24.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.636+0000 7f315a59c700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315c102aa0 0x7f315c198b20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f314c00c9c0 tx=0x7f314c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.636+0000 7f314b7fe700 1 -- 172.21.15.7:0/1001343928 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c00ce60 con 0x7f315c102aa0 2024-04-21T08:44:24.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.636+0000 7f314b7fe700 1 -- 172.21.15.7:0/1001343928 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f314c00f450 con 0x7f315c102aa0 2024-04-21T08:44:24.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.636+0000 7f314b7fe700 1 -- 172.21.15.7:0/1001343928 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c018660 con 0x7f315c102aa0 2024-04-21T08:44:24.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.636+0000 7f3160a60700 1 -- 172.21.15.7:0/1001343928 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315c1a0a70 con 0x7f315c102aa0 2024-04-21T08:44:24.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.636+0000 7f3160a60700 1 -- 172.21.15.7:0/1001343928 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f315c1a0f60 con 0x7f315c102aa0 2024-04-21T08:44:24.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.637+0000 7f3160a60700 1 -- 172.21.15.7:0/1001343928 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f315c069730 con 0x7f315c102aa0 2024-04-21T08:44:24.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.638+0000 7f314b7fe700 1 -- 172.21.15.7:0/1001343928 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f314c00f920 con 0x7f315c102aa0 2024-04-21T08:44:24.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.639+0000 7f314b7fe700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f31440a3c40 0x7f31440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:24.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.639+0000 7f3159d9b700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f31440a3c40 0x7f31440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:24.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.639+0000 7f314b7fe700 1 -- 172.21.15.7:0/1001343928 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f314c0d07a0 con 0x7f315c102aa0 2024-04-21T08:44:24.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.639+0000 7f3159d9b700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f31440a3c40 0x7f31440a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3150000c00 tx=0x7f3150005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:24.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.643+0000 7f314b7fe700 1 -- 172.21.15.7:0/1001343928 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f314c014030 con 0x7f315c102aa0 2024-04-21T08:44:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.902+0000 7f3160a60700 1 -- 172.21.15.7:0/1001343928 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f315c02df80 con 0x7f315c102aa0 2024-04-21T08:44:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.902+0000 7f314b7fe700 1 -- 172.21.15.7:0/1001343928 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+2992 (secure 0 0 0) 0x7f314c09cda0 con 0x7f315c102aa0 2024-04-21T08:44:24.902 INFO:teuthology.orchestra.run.smithi007.stdout:e21 2024-04-21T08:44:24.902 INFO:teuthology.orchestra.run.smithi007.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:44:24.902 INFO:teuthology.orchestra.run.smithi007.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:legacy client fscid: 1 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:Filesystem 'cephfs' (1) 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:fs_name cephfs 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 21 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-04-21T08:42:16.996517+0000 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:modified 2024-04-21T08:42:42.080753+0000 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:tableserver 0 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:root 0 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:session_timeout 300 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:session_autoclose 300 2024-04-21T08:44:24.903 INFO:teuthology.orchestra.run.smithi007.stdout:max_file_size 1099511627776 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:max_xattr_size 65536 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:required_client_features {} 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:last_failure 0 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:last_failure_osd_epoch 0 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:max_mds 1 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:in 0 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:up {0=24533} 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:failed 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:damaged 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:stopped 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:data_pools [3,4] 2024-04-21T08:44:24.904 INFO:teuthology.orchestra.run.smithi007.stdout:metadata_pool 2 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout:inline_data disabled 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout:balancer 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout:bal_rank_mask -1 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout:standby_count_wanted 0 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout:qdb_cluster leader: 24533 members: 24533 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.b{0:24533} state up:active seq 3 addr [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.i{0:24532} state up:standby-replay seq 1 addr [v2:172.21.15.77:6838/516970569,v1:172.21.15.77:6839/516970569] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout:Standby daemons: 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.j{-1:14562} state up:standby seq 1 addr [v2:172.21.15.7:6834/2001227851,v1:172.21.15.7:6835/2001227851] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.905 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.d{-1:14574} state up:standby seq 1 addr [v2:172.21.15.7:6836/1152814354,v1:172.21.15.7:6837/1152814354] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.906 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.c{-1:14607} state up:standby seq 1 addr [v2:172.21.15.77:6836/674822179,v1:172.21.15.77:6837/674822179] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.906 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.g{-1:14640} state up:standby seq 1 addr [v2:172.21.15.7:6838/2635298189,v1:172.21.15.7:6839/2635298189] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.906 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.a{-1:14652} state up:standby seq 1 addr [v2:172.21.15.7:6840/3576958984,v1:172.21.15.7:6841/3576958984] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.906 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.k{-1:24428} state up:standby seq 1 addr [v2:172.21.15.16:6832/2178684530,v1:172.21.15.16:6833/2178684530] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.906 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.l{-1:24436} state up:standby seq 1 addr [v2:172.21.15.77:6832/1817896021,v1:172.21.15.77:6833/1817896021] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.906 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.e{-1:24461} state up:standby seq 1 addr [v2:172.21.15.16:6834/4020622072,v1:172.21.15.16:6835/4020622072] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.906 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.h{-1:24491} state up:standby seq 1 addr [v2:172.21.15.16:6836/3699203122,v1:172.21.15.16:6837/3699203122] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.906 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.f{-1:24493} state up:standby seq 1 addr [v2:172.21.15.77:6834/2205305943,v1:172.21.15.77:6835/2205305943] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:44:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.904+0000 7f3160a60700 1 -- 172.21.15.7:0/1001343928 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f31440a3c40 msgr2=0x7f31440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.904+0000 7f3160a60700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f31440a3c40 0x7f31440a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3150000c00 tx=0x7f3150005fb0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.904+0000 7f3160a60700 1 -- 172.21.15.7:0/1001343928 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315c102aa0 msgr2=0x7f315c198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.904+0000 7f3160a60700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315c102aa0 0x7f315c198b20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f314c00c9c0 tx=0x7f314c00ccd0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.904+0000 7f3160a60700 1 -- 172.21.15.7:0/1001343928 shutdown_connections 2024-04-21T08:44:24.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.904+0000 7f3160a60700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f31440a3c40 0x7f31440a6100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.904+0000 7f3160a60700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f315c104640 0x7f315c1a00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.904+0000 7f3160a60700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f315c102aa0 0x7f315c198b20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.904+0000 7f3160a60700 1 --2- 172.21.15.7:0/1001343928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f315c103ca0 0x7f315c199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:24.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.904+0000 7f3160a60700 1 -- 172.21.15.7:0/1001343928 >> 172.21.15.7:0/1001343928 conn(0x7f315c0fe040 msgr2=0x7f315c107020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:24.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.905+0000 7f3160a60700 1 -- 172.21.15.7:0/1001343928 shutdown_connections 2024-04-21T08:44:24.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:24.905+0000 7f3160a60700 1 -- 172.21.15.7:0/1001343928 wait complete. 2024-04-21T08:44:24.908 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:44:25.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:24 smithi016 ceph-mon[31259]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 860 B/s rd, 0 B/s wr, 1 op/s 2024-04-21T08:44:25.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:24 smithi077 ceph-mon[31615]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 860 B/s rd, 0 B/s wr, 1 op/s 2024-04-21T08:44:25.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:24 smithi007 ceph-mon[28477]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 860 B/s rd, 0 B/s wr, 1 op/s 2024-04-21T08:44:25.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-21T08:44:25.871 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:25.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:25 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1001343928' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:44:26.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:25 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1001343928' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:44:26.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:25 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1001343928' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:44:26.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:26 smithi007 ceph-mon[28477]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:27.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:26 smithi016 ceph-mon[31259]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:27.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:26 smithi077 ceph-mon[31615]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:27.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.515+0000 7fd88bb48700 1 -- 172.21.15.7:0/1905888481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd884103cc0 msgr2=0x7fd884104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.515+0000 7fd88bb48700 1 --2- 172.21.15.7:0/1905888481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd884103cc0 0x7fd884104120 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd8780099c0 tx=0x7fd878009cd0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.515+0000 7fd88bb48700 1 -- 172.21.15.7:0/1905888481 shutdown_connections 2024-04-21T08:44:27.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.515+0000 7fd88bb48700 1 --2- 172.21.15.7:0/1905888481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd884104660 0x7fd88410af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.515+0000 7fd88bb48700 1 --2- 172.21.15.7:0/1905888481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102ac0 0x7fd884102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.515+0000 7fd88bb48700 1 --2- 172.21.15.7:0/1905888481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd884103cc0 0x7fd884104120 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.515+0000 7fd88bb48700 1 -- 172.21.15.7:0/1905888481 >> 172.21.15.7:0/1905888481 conn(0x7fd8840fe060 msgr2=0x7fd8841004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:27.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.515+0000 7fd88bb48700 1 -- 172.21.15.7:0/1905888481 shutdown_connections 2024-04-21T08:44:27.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.515+0000 7fd88bb48700 1 -- 172.21.15.7:0/1905888481 wait complete. 2024-04-21T08:44:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.516+0000 7fd88bb48700 1 Processor -- start 2024-04-21T08:44:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.516+0000 7fd88bb48700 1 -- start start 2024-04-21T08:44:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.516+0000 7fd88bb48700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102ac0 0x7fd884198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.516+0000 7fd88bb48700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd884103cc0 0x7fd884199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.516+0000 7fd88bb48700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd884104660 0x7fd8841a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.516+0000 7fd88bb48700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd88410e560 con 0x7fd884104660 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.516+0000 7fd88bb48700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd88410e3e0 con 0x7fd884102ac0 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.516+0000 7fd88bb48700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd88410e6e0 con 0x7fd884103cc0 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.516+0000 7fd88a0e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd884104660 0x7fd8841a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.516+0000 7fd88a0e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd884104660 0x7fd8841a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59452/0 (socket says 172.21.15.7:59452) 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.516+0000 7fd88a0e5700 1 -- 172.21.15.7:0/4291106424 learned_addr learned my addr 172.21.15.7:0/4291106424 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd8890e3700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd884103cc0 0x7fd884199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd8898e4700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102ac0 0x7fd884198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd88a0e5700 1 -- 172.21.15.7:0/4291106424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd884103cc0 msgr2=0x7fd884199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd88a0e5700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd884103cc0 0x7fd884199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd88a0e5700 1 -- 172.21.15.7:0/4291106424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102ac0 msgr2=0x7fd884198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd88a0e5700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102ac0 0x7fd884198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd88a0e5700 1 -- 172.21.15.7:0/4291106424 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8841a0840 con 0x7fd884104660 2024-04-21T08:44:27.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd8890e3700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd884103cc0 0x7fd884199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd8898e4700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102ac0 0x7fd884198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd88a0e5700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd884104660 0x7fd8841a01a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd87c007f10 tx=0x7fd87c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd876ffd700 1 -- 172.21.15.7:0/4291106424 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c004bb0 con 0x7fd884104660 2024-04-21T08:44:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd88bb48700 1 -- 172.21.15.7:0/4291106424 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8841a0b30 con 0x7fd884104660 2024-04-21T08:44:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd876ffd700 1 -- 172.21.15.7:0/4291106424 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd87c004d50 con 0x7fd884104660 2024-04-21T08:44:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd876ffd700 1 -- 172.21.15.7:0/4291106424 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c005a30 con 0x7fd884104660 2024-04-21T08:44:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.517+0000 7fd88bb48700 1 -- 172.21.15.7:0/4291106424 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8841a10c0 con 0x7fd884104660 2024-04-21T08:44:27.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.518+0000 7fd88bb48700 1 -- 172.21.15.7:0/4291106424 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd884069730 con 0x7fd884104660 2024-04-21T08:44:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.519+0000 7fd876ffd700 1 -- 172.21.15.7:0/4291106424 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fd87c01a020 con 0x7fd884104660 2024-04-21T08:44:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.520+0000 7fd876ffd700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8700a3c40 0x7fd8700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.520+0000 7fd8898e4700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8700a3c40 0x7fd8700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.520+0000 7fd876ffd700 1 -- 172.21.15.7:0/4291106424 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7fd87c0d0610 con 0x7fd884104660 2024-04-21T08:44:27.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.521+0000 7fd8898e4700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8700a3c40 0x7fd8700a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd884103b20 tx=0x7fd880009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:27.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.524+0000 7fd876ffd700 1 -- 172.21.15.7:0/4291106424 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd87c010070 con 0x7fd884104660 2024-04-21T08:44:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.735+0000 7fd88bb48700 1 -- 172.21.15.7:0/4291106424 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fd884063650 con 0x7fd884104660 2024-04-21T08:44:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.736+0000 7fd876ffd700 1 -- 172.21.15.7:0/4291106424 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) ==== 56+0+5695 (secure 0 0 0) 0x7fd87c09cb80 con 0x7fd884104660 2024-04-21T08:44:27.736 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 94 2024-04-21T08:44:27.736 INFO:teuthology.orchestra.run.smithi007.stdout:fsid c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:44:27.736 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-04-21T08:34:14.986230+0000 2024-04-21T08:44:27.736 INFO:teuthology.orchestra.run.smithi007.stdout:modified 2024-04-21T08:44:11.760990+0000 2024-04-21T08:44:27.736 INFO:teuthology.orchestra.run.smithi007.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-21T08:44:27.736 INFO:teuthology.orchestra.run.smithi007.stdout:crush_version 27 2024-04-21T08:44:27.736 INFO:teuthology.orchestra.run.smithi007.stdout:full_ratio 0.95 2024-04-21T08:44:27.736 INFO:teuthology.orchestra.run.smithi007.stdout:backfillfull_ratio 0.9 2024-04-21T08:44:27.736 INFO:teuthology.orchestra.run.smithi007.stdout:nearfull_ratio 0.85 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:require_min_compat_client luminous 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:min_compat_client luminous 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:require_osd_release squid 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:stretch_mode_enabled false 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:max_osd 12 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 up in weight 1 up_from 8 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.7:6802/3999850241,v1:172.21.15.7:6803/3999850241] [v2:172.21.15.7:6804/3999850241,v1:172.21.15.7:6805/3999850241] exists,up c55caf13-53cc-4a4d-9324-fc3ae9c6f182 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 up in weight 1 up_from 13 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6800/1911240202,v1:172.21.15.16:6801/1911240202] [v2:172.21.15.16:6802/1911240202,v1:172.21.15.16:6803/1911240202] exists,up 3d953784-5cd8-4152-b25e-c1b36ccc8635 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 up in weight 1 up_from 18 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6800/1494490623,v1:172.21.15.77:6801/1494490623] [v2:172.21.15.77:6802/1494490623,v1:172.21.15.77:6803/1494490623] exists,up fb1a5dad-fdc9-4a5b-a41d-de604769e356 2024-04-21T08:44:27.737 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 up in weight 1 up_from 26 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.7:6810/3942680881,v1:172.21.15.7:6811/3942680881] [v2:172.21.15.7:6812/3942680881,v1:172.21.15.7:6813/3942680881] exists,up 4b911243-4599-4632-9186-43b2714c8d39 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 up in weight 1 up_from 34 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/849126848,v1:172.21.15.16:6809/849126848] [v2:172.21.15.16:6810/849126848,v1:172.21.15.16:6811/849126848] exists,up a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 up in weight 1 up_from 39 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6808/286186994,v1:172.21.15.77:6809/286186994] [v2:172.21.15.77:6810/286186994,v1:172.21.15.77:6811/286186994] exists,up 93e0c8bb-dee9-4c3b-83f7-fd536ebdee21 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 up in weight 1 up_from 47 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.7:6818/3776602687,v1:172.21.15.7:6819/3776602687] [v2:172.21.15.7:6820/3776602687,v1:172.21.15.7:6821/3776602687] exists,up e1f722c8-d99f-47c0-8d9c-35d391a82058 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 up in weight 1 up_from 52 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6816/2331386881,v1:172.21.15.16:6817/2331386881] [v2:172.21.15.16:6818/2331386881,v1:172.21.15.16:6819/2331386881] exists,up ce43f302-448d-4896-8a13-3f52413a2252 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:osd.8 up in weight 1 up_from 60 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6816/3684606717,v1:172.21.15.77:6817/3684606717] [v2:172.21.15.77:6818/3684606717,v1:172.21.15.77:6819/3684606717] exists,up 84019e97-520b-4134-93dc-67d0654f29fe 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:osd.9 up in weight 1 up_from 68 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.7:6826/485389831,v1:172.21.15.7:6827/485389831] [v2:172.21.15.7:6828/485389831,v1:172.21.15.7:6829/485389831] exists,up c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:osd.10 up in weight 1 up_from 73 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6824/368172333,v1:172.21.15.16:6825/368172333] [v2:172.21.15.16:6826/368172333,v1:172.21.15.16:6827/368172333] exists,up d4b74336-65ed-4ff2-a634-217a0e7ce75c 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:osd.11 up in weight 1 up_from 81 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6824/3994048303,v1:172.21.15.77:6825/3994048303] [v2:172.21.15.77:6826/3994048303,v1:172.21.15.77:6827/3994048303] exists,up 3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 2.0 [7,11] 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 2.a [2,11] 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 2.16 [2,4] 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 2.1a [8,1] 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 2.1b [2,4] 2024-04-21T08:44:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 3.2 [3,9] 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 3.11 [5,9] 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 3.17 [4,8] 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 3.1d [4,8] 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 3.1e [5,7] 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 3.27 [3,9] 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 3.3a [4,6] 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 4.e [5,8] 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 4.17 [9,8] 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 4.19 [9,8] 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 4.1c [9,10] 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/1764088272 expires 2024-04-22T08:35:09.541897+0000 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/3825837962 expires 2024-04-22T08:35:09.541897+0000 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/2463266160 expires 2024-04-22T08:35:09.541897+0000 2024-04-21T08:44:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/3631314876 expires 2024-04-22T08:34:42.863861+0000 2024-04-21T08:44:27.740 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6801/102274072 expires 2024-04-22T08:35:09.541897+0000 2024-04-21T08:44:27.740 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6801/2031045761 expires 2024-04-22T08:34:42.863861+0000 2024-04-21T08:44:27.740 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6800/2031045761 expires 2024-04-22T08:34:42.863861+0000 2024-04-21T08:44:27.740 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/865111185 expires 2024-04-22T08:34:42.863861+0000 2024-04-21T08:44:27.740 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/2778026231 expires 2024-04-22T08:34:42.863861+0000 2024-04-21T08:44:27.740 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6800/102274072 expires 2024-04-22T08:35:09.541897+0000 2024-04-21T08:44:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.738+0000 7fd88bb48700 1 -- 172.21.15.7:0/4291106424 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8700a3c40 msgr2=0x7fd8700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.738+0000 7fd88bb48700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8700a3c40 0x7fd8700a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd884103b20 tx=0x7fd880009670 comp rx=0 tx=0).stop 2024-04-21T08:44:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.738+0000 7fd88bb48700 1 -- 172.21.15.7:0/4291106424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd884104660 msgr2=0x7fd8841a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.738+0000 7fd88bb48700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd884104660 0x7fd8841a01a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd87c007f10 tx=0x7fd87c00d3b0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.738+0000 7fd88bb48700 1 -- 172.21.15.7:0/4291106424 shutdown_connections 2024-04-21T08:44:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.738+0000 7fd88bb48700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd8700a3c40 0x7fd8700a6100 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.738+0000 7fd88bb48700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd884104660 0x7fd8841a01a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.738+0000 7fd88bb48700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd884102ac0 0x7fd884198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.738+0000 7fd88bb48700 1 --2- 172.21.15.7:0/4291106424 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd884103cc0 0x7fd884199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.738+0000 7fd88bb48700 1 -- 172.21.15.7:0/4291106424 >> 172.21.15.7:0/4291106424 conn(0x7fd8840fe060 msgr2=0x7fd884106fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:27.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.739+0000 7fd88bb48700 1 -- 172.21.15.7:0/4291106424 shutdown_connections 2024-04-21T08:44:27.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:27.739+0000 7fd88bb48700 1 -- 172.21.15.7:0/4291106424 wait complete. 2024-04-21T08:44:28.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:27 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4291106424' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:44:28.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:27 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4291106424' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:44:28.112 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:27 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4291106424' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:44:28.470 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-21T08:44:28.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:28.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.589+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/3217507539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df8101fd0 msgr2=0x7f7df8102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:28.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.589+0000 7f7dfcaa5640 1 --2- 172.21.15.7:0/3217507539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df8101fd0 0x7f7df8102450 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009900 tx=0x7f7dec02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:28.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.589+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/3217507539 shutdown_connections 2024-04-21T08:44:28.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.589+0000 7f7dfcaa5640 1 --2- 172.21.15.7:0/3217507539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df8102990 0x7f7df81092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.589+0000 7f7dfcaa5640 1 --2- 172.21.15.7:0/3217507539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df8101fd0 0x7f7df8102450 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.589+0000 7f7dfcaa5640 1 --2- 172.21.15.7:0/3217507539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df8100e80 0x7f7df81012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.589+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/3217507539 >> 172.21.15.7:0/3217507539 conn(0x7f7df80fc3f0 msgr2=0x7f7df80fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.589+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/3217507539 shutdown_connections 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.589+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/3217507539 wait complete. 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.590+0000 7f7dfcaa5640 1 Processor -- start 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.590+0000 7f7dfcaa5640 1 -- start start 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.590+0000 7f7dfcaa5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df8100e80 0x7f7df8198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.590+0000 7f7dfcaa5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df8101fd0 0x7f7df8198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.590+0000 7f7dfcaa5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df8102990 0x7f7df819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.590+0000 7f7dfcaa5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7df810ca30 con 0x7f7df8102990 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.590+0000 7f7dfcaa5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7df810c8b0 con 0x7f7df8100e80 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.590+0000 7f7dfcaa5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7df810cbb0 con 0x7f7df8101fd0 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.590+0000 7f7df6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df8102990 0x7f7df819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.590+0000 7f7df6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df8102990 0x7f7df819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59468/0 (socket says 172.21.15.7:59468) 2024-04-21T08:44:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.590+0000 7f7df6d9d640 1 -- 172.21.15.7:0/2490721958 learned_addr learned my addr 172.21.15.7:0/2490721958 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7df659c640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df8100e80 0x7f7df8198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7df5d9b640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df8101fd0 0x7f7df8198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7df6d9d640 1 -- 172.21.15.7:0/2490721958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df8101fd0 msgr2=0x7f7df8198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7df6d9d640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df8101fd0 0x7f7df8198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7df6d9d640 1 -- 172.21.15.7:0/2490721958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df8100e80 msgr2=0x7f7df8198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7df6d9d640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df8100e80 0x7f7df8198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7df6d9d640 1 -- 172.21.15.7:0/2490721958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df819fe00 con 0x7f7df8102990 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7df5d9b640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df8101fd0 0x7f7df8198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7df6d9d640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df8102990 0x7f7df819f790 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7df000cce0 tx=0x7f7df000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7de77fe640 1 -- 172.21.15.7:0/2490721958 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df000f450 con 0x7f7df8102990 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7de77fe640 1 -- 172.21.15.7:0/2490721958 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7df000faa0 con 0x7f7df8102990 2024-04-21T08:44:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/2490721958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df81a00f0 con 0x7f7df8102990 2024-04-21T08:44:28.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7de77fe640 1 -- 172.21.15.7:0/2490721958 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0018a80 con 0x7f7df8102990 2024-04-21T08:44:28.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.591+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/2490721958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df81a0670 con 0x7f7df8102990 2024-04-21T08:44:28.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.592+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/2490721958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df81a0930 con 0x7f7df8102990 2024-04-21T08:44:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.593+0000 7f7de77fe640 1 -- 172.21.15.7:0/2490721958 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f7df000fc50 con 0x7f7df8102990 2024-04-21T08:44:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.594+0000 7f7de77fe640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7de00a3bb0 0x7f7de00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.594+0000 7f7df659c640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7de00a3bb0 0x7f7de00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.594+0000 7f7de77fe640 1 -- 172.21.15.7:0/2490721958 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f7df00d0ce0 con 0x7f7df8102990 2024-04-21T08:44:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.594+0000 7f7df659c640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7de00a3bb0 0x7f7de00a6070 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009990 tx=0x7f7de8009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:28.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.598+0000 7f7de77fe640 1 -- 172.21.15.7:0/2490721958 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7df0014030 con 0x7f7df8102990 2024-04-21T08:44:28.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.786+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/2490721958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7df81a0ba0 con 0x7f7df8102990 2024-04-21T08:44:28.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.787+0000 7f7de77fe640 1 -- 172.21.15.7:0/2490721958 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24714 (secure 0 0 0) 0x7f7df009d2b0 con 0x7f7df8102990 2024-04-21T08:44:28.787 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:28.787 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":94,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:44:11.760990+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:28.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.789+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/2490721958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7de00a3bb0 msgr2=0x7f7de00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:28.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.789+0000 7f7dfcaa5640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7de00a3bb0 0x7f7de00a6070 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009990 tx=0x7f7de8009110 comp rx=0 tx=0).stop 2024-04-21T08:44:28.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.789+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/2490721958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df8102990 msgr2=0x7f7df819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:28.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.789+0000 7f7dfcaa5640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df8102990 0x7f7df819f790 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7df000cce0 tx=0x7f7df000a620 comp rx=0 tx=0).stop 2024-04-21T08:44:28.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.789+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/2490721958 shutdown_connections 2024-04-21T08:44:28.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.789+0000 7f7dfcaa5640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f7de00a3bb0 0x7f7de00a6070 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.789+0000 7f7dfcaa5640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df8102990 0x7f7df819f790 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.789+0000 7f7dfcaa5640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7df8101fd0 0x7f7df8198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.789+0000 7f7dfcaa5640 1 --2- 172.21.15.7:0/2490721958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df8100e80 0x7f7df8198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.789+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/2490721958 >> 172.21.15.7:0/2490721958 conn(0x7f7df80fc3f0 msgr2=0x7f7df80fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:28.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.790+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/2490721958 shutdown_connections 2024-04-21T08:44:28.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.790+0000 7f7dfcaa5640 1 -- 172.21.15.7:0/2490721958 wait complete. 2024-04-21T08:44:28.801 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:28.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.908+0000 7f5531843640 1 -- 172.21.15.7:0/1834349587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f552c100ea0 msgr2=0x7f552c1012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.908+0000 7f5531843640 1 --2- 172.21.15.7:0/1834349587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f552c100ea0 0x7f552c1012c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f551c009900 tx=0x7f551c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.908+0000 7f5531843640 1 -- 172.21.15.7:0/1834349587 shutdown_connections 2024-04-21T08:44:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.908+0000 7f5531843640 1 --2- 172.21.15.7:0/1834349587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f552c1029b0 0x7f552c1092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.908+0000 7f5531843640 1 --2- 172.21.15.7:0/1834349587 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f552c101ff0 0x7f552c102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.908+0000 7f5531843640 1 --2- 172.21.15.7:0/1834349587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f552c100ea0 0x7f552c1012c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.908+0000 7f5531843640 1 -- 172.21.15.7:0/1834349587 >> 172.21.15.7:0/1834349587 conn(0x7f552c0fc430 msgr2=0x7f552c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.908+0000 7f5531843640 1 -- 172.21.15.7:0/1834349587 shutdown_connections 2024-04-21T08:44:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.908+0000 7f5531843640 1 -- 172.21.15.7:0/1834349587 wait complete. 2024-04-21T08:44:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f5531843640 1 Processor -- start 2024-04-21T08:44:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f5531843640 1 -- start start 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f5531843640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f552c100ea0 0x7f552c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f5531843640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f552c101ff0 0x7f552c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f5531843640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f552c1029b0 0x7f552c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f5531843640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f552c06b580 con 0x7f552c100ea0 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f5531843640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f552c06b400 con 0x7f552c1029b0 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f5531843640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f552c10dca0 con 0x7f552c101ff0 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f552affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f552c100ea0 0x7f552c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f552affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f552c100ea0 0x7f552c198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59498/0 (socket says 172.21.15.7:59498) 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f552affd640 1 -- 172.21.15.7:0/2512189094 learned_addr learned my addr 172.21.15.7:0/2512189094 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f552b7fe640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f552c1029b0 0x7f552c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f552a7fc640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f552c101ff0 0x7f552c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f552affd640 1 -- 172.21.15.7:0/2512189094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f552c101ff0 msgr2=0x7f552c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f552affd640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f552c101ff0 0x7f552c198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f552affd640 1 -- 172.21.15.7:0/2512189094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f552c1029b0 msgr2=0x7f552c19f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.909+0000 7f552affd640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f552c1029b0 0x7f552c19f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.910+0000 7f552affd640 1 -- 172.21.15.7:0/2512189094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552c19fe40 con 0x7f552c100ea0 2024-04-21T08:44:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.910+0000 7f552affd640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f552c100ea0 0x7f552c198220 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f551c006010 tx=0x7f551c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.910+0000 7f5530841640 1 -- 172.21.15.7:0/2512189094 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c043020 con 0x7f552c100ea0 2024-04-21T08:44:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.910+0000 7f5530841640 1 -- 172.21.15.7:0/2512189094 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f551c004aa0 con 0x7f552c100ea0 2024-04-21T08:44:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.910+0000 7f5531843640 1 -- 172.21.15.7:0/2512189094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f552c1a00d0 con 0x7f552c100ea0 2024-04-21T08:44:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.910+0000 7f5530841640 1 -- 172.21.15.7:0/2512189094 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c003c70 con 0x7f552c100ea0 2024-04-21T08:44:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.910+0000 7f5531843640 1 -- 172.21.15.7:0/2512189094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f552c1a0620 con 0x7f552c100ea0 2024-04-21T08:44:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.911+0000 7f5531843640 1 -- 172.21.15.7:0/2512189094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f552c108320 con 0x7f552c100ea0 2024-04-21T08:44:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.912+0000 7f5530841640 1 -- 172.21.15.7:0/2512189094 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f551c046020 con 0x7f552c100ea0 2024-04-21T08:44:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.913+0000 7f5530841640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f55140a3bb0 0x7f55140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.913+0000 7f552a7fc640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f55140a3bb0 0x7f55140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.913+0000 7f5530841640 1 -- 172.21.15.7:0/2512189094 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f551c0f3260 con 0x7f552c100ea0 2024-04-21T08:44:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.913+0000 7f552a7fc640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f55140a3bb0 0x7f55140a6070 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5520005fd0 tx=0x7f5520005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:28.916+0000 7f5530841640 1 -- 172.21.15.7:0/2512189094 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f551c04d020 con 0x7f552c100ea0 2024-04-21T08:44:29.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:28 smithi016 ceph-mon[31259]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:29.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:28 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2490721958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:29.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.102+0000 7f5531843640 1 -- 172.21.15.7:0/2512189094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f552c062b50 con 0x7f552c100ea0 2024-04-21T08:44:29.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.102+0000 7f5530841640 1 -- 172.21.15.7:0/2512189094 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f551c0bf830 con 0x7f552c100ea0 2024-04-21T08:44:29.102 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:44:29.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:28 smithi077 ceph-mon[31615]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:29.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:28 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2490721958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.104+0000 7f5531843640 1 -- 172.21.15.7:0/2512189094 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f55140a3bb0 msgr2=0x7f55140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.104+0000 7f5531843640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f55140a3bb0 0x7f55140a6070 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5520005fd0 tx=0x7f5520005850 comp rx=0 tx=0).stop 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.104+0000 7f5531843640 1 -- 172.21.15.7:0/2512189094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f552c100ea0 msgr2=0x7f552c198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.104+0000 7f5531843640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f552c100ea0 0x7f552c198220 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f551c006010 tx=0x7f551c0042f0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.104+0000 7f5531843640 1 -- 172.21.15.7:0/2512189094 shutdown_connections 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.104+0000 7f5531843640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f55140a3bb0 0x7f55140a6070 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.104+0000 7f5531843640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f552c1029b0 0x7f552c19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.104+0000 7f5531843640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f552c101ff0 0x7f552c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.104+0000 7f5531843640 1 --2- 172.21.15.7:0/2512189094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f552c100ea0 0x7f552c198220 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.104+0000 7f5531843640 1 -- 172.21.15.7:0/2512189094 >> 172.21.15.7:0/2512189094 conn(0x7f552c0fc430 msgr2=0x7f552c1054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.104+0000 7f5531843640 1 -- 172.21.15.7:0/2512189094 shutdown_connections 2024-04-21T08:44:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.105+0000 7f5531843640 1 -- 172.21.15.7:0/2512189094 wait complete. 2024-04-21T08:44:29.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:28 smithi007 ceph-mon[28477]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:29.114 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:28 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2490721958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:29.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:29.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.221+0000 7f4dd4606640 1 -- 172.21.15.7:0/757132003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dcc101200 msgr2=0x7f4dcc101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.221+0000 7f4dd4606640 1 --2- 172.21.15.7:0/757132003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dcc101200 0x7f4dcc101660 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0009900 tx=0x7f4dc002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 -- 172.21.15.7:0/757132003 shutdown_connections 2024-04-21T08:44:29.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 --2- 172.21.15.7:0/757132003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dcc101ba0 0x7f4dcc102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 --2- 172.21.15.7:0/757132003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dcc101200 0x7f4dcc101660 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 --2- 172.21.15.7:0/757132003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc1008a0 0x7f4dcc100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 -- 172.21.15.7:0/757132003 >> 172.21.15.7:0/757132003 conn(0x7f4dcc0fc420 msgr2=0x7f4dcc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:29.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 -- 172.21.15.7:0/757132003 shutdown_connections 2024-04-21T08:44:29.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 -- 172.21.15.7:0/757132003 wait complete. 2024-04-21T08:44:29.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 Processor -- start 2024-04-21T08:44:29.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 -- start start 2024-04-21T08:44:29.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc1008a0 0x7f4dcc19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dcc101200 0x7f4dcc19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dcc101ba0 0x7f4dcc1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc10fcc0 con 0x7f4dcc101ba0 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc10fb40 con 0x7f4dcc1008a0 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.222+0000 7f4dd4606640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc10fe40 con 0x7f4dcc101200 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd2ba3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dcc101ba0 0x7f4dcc1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd2ba3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dcc101ba0 0x7f4dcc1a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59522/0 (socket says 172.21.15.7:59522) 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd2ba3640 1 -- 172.21.15.7:0/854803291 learned_addr learned my addr 172.21.15.7:0/854803291 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd23a2640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc1008a0 0x7f4dcc19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd1ba1640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dcc101200 0x7f4dcc19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd2ba3640 1 -- 172.21.15.7:0/854803291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dcc101200 msgr2=0x7f4dcc19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd2ba3640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dcc101200 0x7f4dcc19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd2ba3640 1 -- 172.21.15.7:0/854803291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc1008a0 msgr2=0x7f4dcc19a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd2ba3640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc1008a0 0x7f4dcc19a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd2ba3640 1 -- 172.21.15.7:0/854803291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dcc1a2020 con 0x7f4dcc101ba0 2024-04-21T08:44:29.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd23a2640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc1008a0 0x7f4dcc19a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:29.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd2ba3640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dcc101ba0 0x7f4dcc1a19b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00cad0 tx=0x7f4dbc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:29.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dcb7fe640 1 -- 172.21.15.7:0/854803291 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc00ae20 con 0x7f4dcc101ba0 2024-04-21T08:44:29.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dcb7fe640 1 -- 172.21.15.7:0/854803291 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4dbc005020 con 0x7f4dcc101ba0 2024-04-21T08:44:29.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd4606640 1 -- 172.21.15.7:0/854803291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dcc1a2310 con 0x7f4dcc101ba0 2024-04-21T08:44:29.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dcb7fe640 1 -- 172.21.15.7:0/854803291 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc00f6b0 con 0x7f4dcc101ba0 2024-04-21T08:44:29.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.223+0000 7f4dd4606640 1 -- 172.21.15.7:0/854803291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dcc1a27e0 con 0x7f4dcc101ba0 2024-04-21T08:44:29.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.224+0000 7f4dd4606640 1 -- 172.21.15.7:0/854803291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dcc10b880 con 0x7f4dcc101ba0 2024-04-21T08:44:29.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.226+0000 7f4dcb7fe640 1 -- 172.21.15.7:0/854803291 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f4dbc0051d0 con 0x7f4dcc101ba0 2024-04-21T08:44:29.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.226+0000 7f4dcb7fe640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4db40a3bb0 0x7f4db40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.226+0000 7f4dd23a2640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4db40a3bb0 0x7f4db40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:29.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.226+0000 7f4dcb7fe640 1 -- 172.21.15.7:0/854803291 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f4dbc0d08c0 con 0x7f4dcc101ba0 2024-04-21T08:44:29.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.227+0000 7f4dd23a2640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4db40a3bb0 0x7f4db40a6070 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4005d00 tx=0x7f4dc4005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:29.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.231+0000 7f4dcb7fe640 1 -- 172.21.15.7:0/854803291 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4dbc014030 con 0x7f4dcc101ba0 2024-04-21T08:44:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.415+0000 7f4dd4606640 1 -- 172.21.15.7:0/854803291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4dcc062b50 con 0x7f4dcc101ba0 2024-04-21T08:44:29.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.416+0000 7f4dcb7fe640 1 -- 172.21.15.7:0/854803291 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f4dbc09ce30 con 0x7f4dcc101ba0 2024-04-21T08:44:29.530 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:29.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 -- 172.21.15.7:0/854803291 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4db40a3bb0 msgr2=0x7f4db40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4db40a3bb0 0x7f4db40a6070 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4005d00 tx=0x7f4dc4005c70 comp rx=0 tx=0).stop 2024-04-21T08:44:29.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 -- 172.21.15.7:0/854803291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dcc101ba0 msgr2=0x7f4dcc1a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dcc101ba0 0x7f4dcc1a19b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00cad0 tx=0x7f4dbc00cf90 comp rx=0 tx=0).stop 2024-04-21T08:44:29.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 -- 172.21.15.7:0/854803291 shutdown_connections 2024-04-21T08:44:29.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f4db40a3bb0 0x7f4db40a6070 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dcc101ba0 0x7f4dcc1a19b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4dcc101200 0x7f4dcc19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 --2- 172.21.15.7:0/854803291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4dcc1008a0 0x7f4dcc19a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 -- 172.21.15.7:0/854803291 >> 172.21.15.7:0/854803291 conn(0x7f4dcc0fc420 msgr2=0x7f4dcc0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:29.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 -- 172.21.15.7:0/854803291 shutdown_connections 2024-04-21T08:44:29.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.418+0000 7f4dd4606640 1 -- 172.21.15.7:0/854803291 wait complete. 2024-04-21T08:44:29.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:29.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.637+0000 7f50d2959640 1 -- 172.21.15.7:0/2376980060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50cc1029a0 msgr2=0x7f50cc1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.637+0000 7f50d2959640 1 --2- 172.21.15.7:0/2376980060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50cc1029a0 0x7f50cc1092c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009ca0 tx=0x7f50c402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.638+0000 7f50d2959640 1 -- 172.21.15.7:0/2376980060 shutdown_connections 2024-04-21T08:44:29.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.638+0000 7f50d2959640 1 --2- 172.21.15.7:0/2376980060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50cc1029a0 0x7f50cc1092c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.638+0000 7f50d2959640 1 --2- 172.21.15.7:0/2376980060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50cc101fe0 0x7f50cc102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.638+0000 7f50d2959640 1 --2- 172.21.15.7:0/2376980060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc100e90 0x7f50cc1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.638+0000 7f50d2959640 1 -- 172.21.15.7:0/2376980060 >> 172.21.15.7:0/2376980060 conn(0x7f50cc0fc420 msgr2=0x7f50cc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.638+0000 7f50d2959640 1 -- 172.21.15.7:0/2376980060 shutdown_connections 2024-04-21T08:44:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.638+0000 7f50d2959640 1 -- 172.21.15.7:0/2376980060 wait complete. 2024-04-21T08:44:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.638+0000 7f50d2959640 1 Processor -- start 2024-04-21T08:44:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.638+0000 7f50d2959640 1 -- start start 2024-04-21T08:44:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.638+0000 7f50d2959640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50cc100e90 0x7f50cc069de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.638+0000 7f50d2959640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50cc101fe0 0x7f50cc19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50d2959640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc1029a0 0x7f50cc19f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50d2959640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f50cc10c9d0 con 0x7f50cc100e90 2024-04-21T08:44:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50d2959640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f50cc10c850 con 0x7f50cc1029a0 2024-04-21T08:44:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50d2959640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f50cc10cb50 con 0x7f50cc101fe0 2024-04-21T08:44:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50cbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50cc100e90 0x7f50cc069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50cbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50cc100e90 0x7f50cc069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59534/0 (socket says 172.21.15.7:59534) 2024-04-21T08:44:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50cbfff640 1 -- 172.21.15.7:0/2621307832 learned_addr learned my addr 172.21.15.7:0/2621307832 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50d0ef6640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc1029a0 0x7f50cc19f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50cb7fe640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50cc101fe0 0x7f50cc19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50cbfff640 1 -- 172.21.15.7:0/2621307832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50cc101fe0 msgr2=0x7f50cc19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50cbfff640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50cc101fe0 0x7f50cc19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50cbfff640 1 -- 172.21.15.7:0/2621307832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc1029a0 msgr2=0x7f50cc19f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50cbfff640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc1029a0 0x7f50cc19f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50cbfff640 1 -- 172.21.15.7:0/2621307832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50cc19fd70 con 0x7f50cc100e90 2024-04-21T08:44:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50cb7fe640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50cc101fe0 0x7f50cc19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50cbfff640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50cc100e90 0x7f50cc069de0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f50b400e9f0 tx=0x7f50b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50c97fa640 1 -- 172.21.15.7:0/2621307832 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b400cd80 con 0x7f50cc100e90 2024-04-21T08:44:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50c97fa640 1 -- 172.21.15.7:0/2621307832 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50b40108e0 con 0x7f50cc100e90 2024-04-21T08:44:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50d2959640 1 -- 172.21.15.7:0/2621307832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50cc1a0060 con 0x7f50cc100e90 2024-04-21T08:44:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50c97fa640 1 -- 172.21.15.7:0/2621307832 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4018980 con 0x7f50cc100e90 2024-04-21T08:44:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.639+0000 7f50d2959640 1 -- 172.21.15.7:0/2621307832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50cc1a05b0 con 0x7f50cc100e90 2024-04-21T08:44:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.640+0000 7f50d2959640 1 -- 172.21.15.7:0/2621307832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50cc108220 con 0x7f50cc100e90 2024-04-21T08:44:29.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.641+0000 7f50c97fa640 1 -- 172.21.15.7:0/2621307832 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f50b4010430 con 0x7f50cc100e90 2024-04-21T08:44:29.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.642+0000 7f50c97fa640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f50b80a3bb0 0x7f50b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.642+0000 7f50cb7fe640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f50b80a3bb0 0x7f50b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:29.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.642+0000 7f50c97fa640 1 -- 172.21.15.7:0/2621307832 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f50b40d0b90 con 0x7f50cc100e90 2024-04-21T08:44:29.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.642+0000 7f50cb7fe640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f50b80a3bb0 0x7f50b80a6070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f50cc069b50 tx=0x7f50bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:29.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.647+0000 7f50c97fa640 1 -- 172.21.15.7:0/2621307832 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f50b409d160 con 0x7f50cc100e90 2024-04-21T08:44:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.835+0000 7f50d2959640 1 -- 172.21.15.7:0/2621307832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f50cc062b80 con 0x7f50cc100e90 2024-04-21T08:44:29.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.836+0000 7f50c97fa640 1 -- 172.21.15.7:0/2621307832 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f50b409ca60 con 0x7f50cc100e90 2024-04-21T08:44:29.835 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:44:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 -- 172.21.15.7:0/2621307832 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f50b80a3bb0 msgr2=0x7f50b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f50b80a3bb0 0x7f50b80a6070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f50cc069b50 tx=0x7f50bc005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 -- 172.21.15.7:0/2621307832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50cc100e90 msgr2=0x7f50cc069de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50cc100e90 0x7f50cc069de0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f50b400e9f0 tx=0x7f50b400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 -- 172.21.15.7:0/2621307832 shutdown_connections 2024-04-21T08:44:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f50b80a3bb0 0x7f50b80a6070 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc1029a0 0x7f50cc19f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50cc101fe0 0x7f50cc19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 --2- 172.21.15.7:0/2621307832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50cc100e90 0x7f50cc069de0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 -- 172.21.15.7:0/2621307832 >> 172.21.15.7:0/2621307832 conn(0x7f50cc0fc420 msgr2=0x7f50cc1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 -- 172.21.15.7:0/2621307832 shutdown_connections 2024-04-21T08:44:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.838+0000 7f50d2959640 1 -- 172.21.15.7:0/2621307832 wait complete. 2024-04-21T08:44:29.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:29.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.954+0000 7ff4016b3640 1 -- 172.21.15.7:0/1130687579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc102a70 msgr2=0x7ff3fc104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.954+0000 7ff4016b3640 1 --2- 172.21.15.7:0/1130687579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc102a70 0x7ff3fc104e60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009900 tx=0x7ff3f402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:29.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.954+0000 7ff4016b3640 1 -- 172.21.15.7:0/1130687579 shutdown_connections 2024-04-21T08:44:29.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.954+0000 7ff4016b3640 1 --2- 172.21.15.7:0/1130687579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3fc107cd0 0x7ff3fc10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.954+0000 7ff4016b3640 1 --2- 172.21.15.7:0/1130687579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3fc1053a0 0x7ff3fc107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.954+0000 7ff4016b3640 1 --2- 172.21.15.7:0/1130687579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc102a70 0x7ff3fc104e60 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.954+0000 7ff4016b3640 1 -- 172.21.15.7:0/1130687579 >> 172.21.15.7:0/1130687579 conn(0x7ff3fc0fc420 msgr2=0x7ff3fc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:29.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.954+0000 7ff4016b3640 1 -- 172.21.15.7:0/1130687579 shutdown_connections 2024-04-21T08:44:29.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.954+0000 7ff4016b3640 1 -- 172.21.15.7:0/1130687579 wait complete. 2024-04-21T08:44:29.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.955+0000 7ff4016b3640 1 Processor -- start 2024-04-21T08:44:29.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.955+0000 7ff4016b3640 1 -- start start 2024-04-21T08:44:29.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.955+0000 7ff4016b3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc102a70 0x7ff3fc198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.955+0000 7ff4016b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3fc1053a0 0x7ff3fc1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.955+0000 7ff4016b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3fc107cd0 0x7ff3fc19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.955+0000 7ff4016b3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc10dac0 con 0x7ff3fc102a70 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.955+0000 7ff4016b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc10d940 con 0x7ff3fc107cd0 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.955+0000 7ff4016b3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc10dc40 con 0x7ff3fc1053a0 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.955+0000 7ff3faffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc102a70 0x7ff3fc198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.955+0000 7ff3faffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc102a70 0x7ff3fc198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59560/0 (socket says 172.21.15.7:59560) 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.955+0000 7ff3faffd640 1 -- 172.21.15.7:0/4077180583 learned_addr learned my addr 172.21.15.7:0/4077180583 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff3fb7fe640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3fc107cd0 0x7ff3fc19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff3faffd640 1 -- 172.21.15.7:0/4077180583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3fc1053a0 msgr2=0x7ff3fc1987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff3faffd640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3fc1053a0 0x7ff3fc1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff3faffd640 1 -- 172.21.15.7:0/4077180583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3fc107cd0 msgr2=0x7ff3fc19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff3faffd640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3fc107cd0 0x7ff3fc19f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff3faffd640 1 -- 172.21.15.7:0/4077180583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3fc19fea0 con 0x7ff3fc102a70 2024-04-21T08:44:29.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff3faffd640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc102a70 0x7ff3fc198280 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4006010 tx=0x7ff3f40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:29.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff3e3fff640 1 -- 172.21.15.7:0/4077180583 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4043020 con 0x7ff3fc102a70 2024-04-21T08:44:29.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff4016b3640 1 -- 172.21.15.7:0/4077180583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3fc1a0130 con 0x7ff3fc102a70 2024-04-21T08:44:29.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff3e3fff640 1 -- 172.21.15.7:0/4077180583 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3f4004aa0 con 0x7ff3fc102a70 2024-04-21T08:44:29.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff3e3fff640 1 -- 172.21.15.7:0/4077180583 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4003c70 con 0x7ff3fc102a70 2024-04-21T08:44:29.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.956+0000 7ff4016b3640 1 -- 172.21.15.7:0/4077180583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3fc1a0680 con 0x7ff3fc102a70 2024-04-21T08:44:29.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.957+0000 7ff4016b3640 1 -- 172.21.15.7:0/4077180583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3fc0fdf10 con 0x7ff3fc102a70 2024-04-21T08:44:29.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.958+0000 7ff3e3fff640 1 -- 172.21.15.7:0/4077180583 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7ff3f4046020 con 0x7ff3fc102a70 2024-04-21T08:44:29.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.959+0000 7ff3e3fff640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff3e40a3bb0 0x7ff3e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:29.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.959+0000 7ff3fa7fc640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff3e40a3bb0 0x7ff3e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:29.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.959+0000 7ff3e3fff640 1 -- 172.21.15.7:0/4077180583 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7ff3f40f2cf0 con 0x7ff3fc102a70 2024-04-21T08:44:29.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.959+0000 7ff3fa7fc640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff3e40a3bb0 0x7ff3e40a6070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0005fd0 tx=0x7ff3f0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:29.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:29.963+0000 7ff3e3fff640 1 -- 172.21.15.7:0/4077180583 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3f404d020 con 0x7ff3fc102a70 2024-04-21T08:44:30.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:29 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2512189094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:30.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:29 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/854803291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:30.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.150+0000 7ff4016b3640 1 -- 172.21.15.7:0/4077180583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff3fc062b50 con 0x7ff3fc102a70 2024-04-21T08:44:30.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.150+0000 7ff3e3fff640 1 -- 172.21.15.7:0/4077180583 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7ff3f40bf260 con 0x7ff3fc102a70 2024-04-21T08:44:30.151 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:30.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:29 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2512189094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:30.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:29 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/854803291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:30.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.152+0000 7ff4016b3640 1 -- 172.21.15.7:0/4077180583 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff3e40a3bb0 msgr2=0x7ff3e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.152+0000 7ff4016b3640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff3e40a3bb0 0x7ff3e40a6070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0005fd0 tx=0x7ff3f0005850 comp rx=0 tx=0).stop 2024-04-21T08:44:30.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.152+0000 7ff4016b3640 1 -- 172.21.15.7:0/4077180583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc102a70 msgr2=0x7ff3fc198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.152+0000 7ff4016b3640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc102a70 0x7ff3fc198280 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4006010 tx=0x7ff3f40042f0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.153+0000 7ff4016b3640 1 -- 172.21.15.7:0/4077180583 shutdown_connections 2024-04-21T08:44:30.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.153+0000 7ff4016b3640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff3e40a3bb0 0x7ff3e40a6070 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.153+0000 7ff4016b3640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3fc107cd0 0x7ff3fc19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.153+0000 7ff4016b3640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3fc1053a0 0x7ff3fc1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.153+0000 7ff4016b3640 1 --2- 172.21.15.7:0/4077180583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc102a70 0x7ff3fc198280 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.153+0000 7ff4016b3640 1 -- 172.21.15.7:0/4077180583 >> 172.21.15.7:0/4077180583 conn(0x7ff3fc0fc420 msgr2=0x7ff3fc1033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:30.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.153+0000 7ff4016b3640 1 -- 172.21.15.7:0/4077180583 shutdown_connections 2024-04-21T08:44:30.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.153+0000 7ff4016b3640 1 -- 172.21.15.7:0/4077180583 wait complete. 2024-04-21T08:44:30.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.265+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1914007594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd688100870 msgr2=0x7fd688100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.265+0000 7fd68dd2c640 1 --2- 172.21.15.7:0/1914007594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd688100870 0x7fd688100c90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd678009900 tx=0x7fd67802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.265+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1914007594 shutdown_connections 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.265+0000 7fd68dd2c640 1 --2- 172.21.15.7:0/1914007594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd688101b70 0x7fd688101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.265+0000 7fd68dd2c640 1 --2- 172.21.15.7:0/1914007594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6881011d0 0x7fd688101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.265+0000 7fd68dd2c640 1 --2- 172.21.15.7:0/1914007594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd688100870 0x7fd688100c90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.265+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1914007594 >> 172.21.15.7:0/1914007594 conn(0x7fd6880fc3f0 msgr2=0x7fd6880fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.266+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1914007594 shutdown_connections 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.266+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1914007594 wait complete. 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.266+0000 7fd68dd2c640 1 Processor -- start 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.266+0000 7fd68dd2c640 1 -- start start 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.266+0000 7fd68dd2c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd688100870 0x7fd6881063f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.266+0000 7fd68dd2c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6881011d0 0x7fd688106930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.266+0000 7fd68dd2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd688101b70 0x7fd688102f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.266+0000 7fd68dd2c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd68810fd00 con 0x7fd688100870 2024-04-21T08:44:30.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.266+0000 7fd68dd2c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd68810fb80 con 0x7fd688101b70 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.266+0000 7fd68dd2c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd68810fe80 con 0x7fd6881011d0 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.266+0000 7fd6877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd688100870 0x7fd6881063f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd6877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd688100870 0x7fd6881063f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59574/0 (socket says 172.21.15.7:59574) 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd6877fe640 1 -- 172.21.15.7:0/1961984926 learned_addr learned my addr 172.21.15.7:0/1961984926 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd687fff640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd688101b70 0x7fd688102f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd686ffd640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6881011d0 0x7fd688106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd6877fe640 1 -- 172.21.15.7:0/1961984926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6881011d0 msgr2=0x7fd688106930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd6877fe640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6881011d0 0x7fd688106930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd6877fe640 1 -- 172.21.15.7:0/1961984926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd688101b70 msgr2=0x7fd688102f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd6877fe640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd688101b70 0x7fd688102f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd6877fe640 1 -- 172.21.15.7:0/1961984926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd688103820 con 0x7fd688100870 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd6877fe640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd688100870 0x7fd6881063f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd678006010 tx=0x7fd67802d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:30.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd684ff9640 1 -- 172.21.15.7:0/1961984926 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678043020 con 0x7fd688100870 2024-04-21T08:44:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1961984926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6881a6860 con 0x7fd688100870 2024-04-21T08:44:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd684ff9640 1 -- 172.21.15.7:0/1961984926 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd67802dd90 con 0x7fd688100870 2024-04-21T08:44:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd684ff9640 1 -- 172.21.15.7:0/1961984926 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678003bb0 con 0x7fd688100870 2024-04-21T08:44:30.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.267+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1961984926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6881a6ce0 con 0x7fd688100870 2024-04-21T08:44:30.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.268+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1961984926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd688068e00 con 0x7fd688100870 2024-04-21T08:44:30.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.270+0000 7fd684ff9640 1 -- 172.21.15.7:0/1961984926 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fd678046020 con 0x7fd688100870 2024-04-21T08:44:30.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.270+0000 7fd684ff9640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd6700a3bb0 0x7fd6700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.270+0000 7fd686ffd640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd6700a3bb0 0x7fd6700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.270+0000 7fd684ff9640 1 -- 172.21.15.7:0/1961984926 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7fd6780f2b50 con 0x7fd688100870 2024-04-21T08:44:30.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.271+0000 7fd686ffd640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd6700a3bb0 0x7fd6700a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd688104370 tx=0x7fd67c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:30.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.275+0000 7fd684ff9640 1 -- 172.21.15.7:0/1961984926 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd67804d020 con 0x7fd688100870 2024-04-21T08:44:30.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:29 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2512189094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:30.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:29 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/854803291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:30.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.500+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1961984926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd688062b50 con 0x7fd688100870 2024-04-21T08:44:30.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.500+0000 7fd684ff9640 1 -- 172.21.15.7:0/1961984926 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7fd6780bf0c0 con 0x7fd688100870 2024-04-21T08:44:30.501 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:30.501 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:44:30.501 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:44:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.502+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1961984926 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd6700a3bb0 msgr2=0x7fd6700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.502+0000 7fd68dd2c640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd6700a3bb0 0x7fd6700a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd688104370 tx=0x7fd67c008040 comp rx=0 tx=0).stop 2024-04-21T08:44:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.502+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1961984926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd688100870 msgr2=0x7fd6881063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.502+0000 7fd68dd2c640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd688100870 0x7fd6881063f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd678006010 tx=0x7fd67802d990 comp rx=0 tx=0).stop 2024-04-21T08:44:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.502+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1961984926 shutdown_connections 2024-04-21T08:44:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.502+0000 7fd68dd2c640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fd6700a3bb0 0x7fd6700a6070 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.502+0000 7fd68dd2c640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd688101b70 0x7fd688102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.502+0000 7fd68dd2c640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6881011d0 0x7fd688106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.502+0000 7fd68dd2c640 1 --2- 172.21.15.7:0/1961984926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd688100870 0x7fd6881063f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.502+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1961984926 >> 172.21.15.7:0/1961984926 conn(0x7fd6880fc3f0 msgr2=0x7fd6880ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:30.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.503+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1961984926 shutdown_connections 2024-04-21T08:44:30.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.503+0000 7fd68dd2c640 1 -- 172.21.15.7:0/1961984926 wait complete. 2024-04-21T08:44:30.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 -- 172.21.15.7:0/1364007332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f4100030 msgr2=0x7f93f41004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 --2- 172.21.15.7:0/1364007332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f4100030 0x7f93f41004b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009900 tx=0x7f93e802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 -- 172.21.15.7:0/1364007332 shutdown_connections 2024-04-21T08:44:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 --2- 172.21.15.7:0/1364007332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f41009f0 0x7f93f410a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 --2- 172.21.15.7:0/1364007332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f4100030 0x7f93f41004b0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 --2- 172.21.15.7:0/1364007332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f41035c0 0x7f93f41039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 -- 172.21.15.7:0/1364007332 >> 172.21.15.7:0/1364007332 conn(0x7f93f40fbbb0 msgr2=0x7f93f40fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 -- 172.21.15.7:0/1364007332 shutdown_connections 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 -- 172.21.15.7:0/1364007332 wait complete. 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 Processor -- start 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 -- start start 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f4100030 0x7f93f41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f41009f0 0x7f93f41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f41035c0 0x7f93f419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f93f410db50 con 0x7f93f41009f0 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f93f410d9d0 con 0x7f93f4100030 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93faed9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f93f410dcd0 con 0x7f93f41035c0 2024-04-21T08:44:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93f3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f41009f0 0x7f93f41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.617+0000 7f93f3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f41009f0 0x7f93f41986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59588/0 (socket says 172.21.15.7:59588) 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f3fff640 1 -- 172.21.15.7:0/2278559641 learned_addr learned my addr 172.21.15.7:0/2278559641 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f8c75640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f4100030 0x7f93f41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f3fff640 1 -- 172.21.15.7:0/2278559641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f41035c0 msgr2=0x7f93f419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f9476640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f41035c0 0x7f93f419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f3fff640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f41035c0 0x7f93f419f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f3fff640 1 -- 172.21.15.7:0/2278559641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f4100030 msgr2=0x7f93f41981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f3fff640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f4100030 0x7f93f41981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f3fff640 1 -- 172.21.15.7:0/2278559641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f419fdd0 con 0x7f93f41009f0 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f3fff640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f41009f0 0x7f93f41986f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f93e8006010 tx=0x7f93e80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f1ffb640 1 -- 172.21.15.7:0/2278559641 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e8043020 con 0x7f93f41009f0 2024-04-21T08:44:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93faed9640 1 -- 172.21.15.7:0/2278559641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f41a0060 con 0x7f93f41009f0 2024-04-21T08:44:30.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f1ffb640 1 -- 172.21.15.7:0/2278559641 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93e8004aa0 con 0x7f93f41009f0 2024-04-21T08:44:30.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.618+0000 7f93f1ffb640 1 -- 172.21.15.7:0/2278559641 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e8003c70 con 0x7f93f41009f0 2024-04-21T08:44:30.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.619+0000 7f93faed9640 1 -- 172.21.15.7:0/2278559641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f41a05b0 con 0x7f93f41009f0 2024-04-21T08:44:30.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.620+0000 7f93faed9640 1 -- 172.21.15.7:0/2278559641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93f4109480 con 0x7f93f41009f0 2024-04-21T08:44:30.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.620+0000 7f93f1ffb640 1 -- 172.21.15.7:0/2278559641 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f93e8046020 con 0x7f93f41009f0 2024-04-21T08:44:30.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.621+0000 7f93f1ffb640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93dc0a3bb0 0x7f93dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.621+0000 7f93f8c75640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93dc0a3bb0 0x7f93dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.621+0000 7f93f1ffb640 1 -- 172.21.15.7:0/2278559641 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f93e80f2ba0 con 0x7f93f41009f0 2024-04-21T08:44:30.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.622+0000 7f93f8c75640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93dc0a3bb0 0x7f93dc0a6070 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f93e4005fd0 tx=0x7f93e400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.626+0000 7f93f1ffb640 1 -- 172.21.15.7:0/2278559641 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f93e804d020 con 0x7f93f41009f0 2024-04-21T08:44:30.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.816+0000 7f93faed9640 1 -- 172.21.15.7:0/2278559641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93f4062b50 con 0x7f93f41009f0 2024-04-21T08:44:30.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.817+0000 7f93f1ffb640 1 -- 172.21.15.7:0/2278559641 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24714 (secure 0 0 0) 0x7f93e80bf170 con 0x7f93f41009f0 2024-04-21T08:44:30.818 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:30.818 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":94,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:44:11.760990+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.819+0000 7f93faed9640 1 -- 172.21.15.7:0/2278559641 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93dc0a3bb0 msgr2=0x7f93dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.819+0000 7f93faed9640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93dc0a3bb0 0x7f93dc0a6070 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f93e4005fd0 tx=0x7f93e400b410 comp rx=0 tx=0).stop 2024-04-21T08:44:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.819+0000 7f93faed9640 1 -- 172.21.15.7:0/2278559641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f41009f0 msgr2=0x7f93f41986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.819+0000 7f93faed9640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f41009f0 0x7f93f41986f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f93e8006010 tx=0x7f93e80042f0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.820+0000 7f93faed9640 1 -- 172.21.15.7:0/2278559641 shutdown_connections 2024-04-21T08:44:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.820+0000 7f93faed9640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f93dc0a3bb0 0x7f93dc0a6070 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.820+0000 7f93faed9640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93f41035c0 0x7f93f419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.820+0000 7f93faed9640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93f41009f0 0x7f93f41986f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.820+0000 7f93faed9640 1 --2- 172.21.15.7:0/2278559641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93f4100030 0x7f93f41981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.820+0000 7f93faed9640 1 -- 172.21.15.7:0/2278559641 >> 172.21.15.7:0/2278559641 conn(0x7f93f40fbbb0 msgr2=0x7f93f4106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.820+0000 7f93faed9640 1 -- 172.21.15.7:0/2278559641 shutdown_connections 2024-04-21T08:44:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.820+0000 7f93faed9640 1 -- 172.21.15.7:0/2278559641 wait complete. 2024-04-21T08:44:30.832 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:30.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:30.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.938+0000 7fc305206640 1 -- 172.21.15.7:0/3788750329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc300107c80 msgr2=0x7fc30010a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.938+0000 7fc305206640 1 --2- 172.21.15.7:0/3788750329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc300107c80 0x7fc30010a0d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009ca0 tx=0x7fc2f802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.938+0000 7fc305206640 1 -- 172.21.15.7:0/3788750329 shutdown_connections 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.938+0000 7fc305206640 1 --2- 172.21.15.7:0/3788750329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc300107c80 0x7fc30010a0d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.938+0000 7fc305206640 1 --2- 172.21.15.7:0/3788750329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc300105350 0x7fc300107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.938+0000 7fc305206640 1 --2- 172.21.15.7:0/3788750329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102a20 0x7fc300104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.938+0000 7fc305206640 1 -- 172.21.15.7:0/3788750329 >> 172.21.15.7:0/3788750329 conn(0x7fc3000fc430 msgr2=0x7fc3000fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.938+0000 7fc305206640 1 -- 172.21.15.7:0/3788750329 shutdown_connections 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.938+0000 7fc305206640 1 -- 172.21.15.7:0/3788750329 wait complete. 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.938+0000 7fc305206640 1 Processor -- start 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc305206640 1 -- start start 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc305206640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102a20 0x7fc300102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc305206640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc300105350 0x7fc3001009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc305206640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc300107c80 0x7fc300100f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc305206640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc30010dae0 con 0x7fc300105350 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc305206640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc30010d960 con 0x7fc300102a20 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc305206640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc30010dc60 con 0x7fc300107c80 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2fe59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc300105350 0x7fc3001009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2fe59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc300105350 0x7fc3001009d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59604/0 (socket says 172.21.15.7:59604) 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2fe59c640 1 -- 172.21.15.7:0/2973827363 learned_addr learned my addr 172.21.15.7:0/2973827363 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2fed9d640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102a20 0x7fc300102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2ff59e640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc300107c80 0x7fc300100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2fe59c640 1 -- 172.21.15.7:0/2973827363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc300107c80 msgr2=0x7fc300100f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2fe59c640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc300107c80 0x7fc300100f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2fe59c640 1 -- 172.21.15.7:0/2973827363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102a20 msgr2=0x7fc300102390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2fe59c640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102a20 0x7fc300102390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2fe59c640 1 -- 172.21.15.7:0/2973827363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3001017a0 con 0x7fc300105350 2024-04-21T08:44:30.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2fed9d640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102a20 0x7fc300102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:30.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.939+0000 7fc2fe59c640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc300105350 0x7fc3001009d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400d740 tx=0x7fc2f400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:30.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.940+0000 7fc2e7fff640 1 -- 172.21.15.7:0/2973827363 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4017070 con 0x7fc300105350 2024-04-21T08:44:30.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.940+0000 7fc2e7fff640 1 -- 172.21.15.7:0/2973827363 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2f400de50 con 0x7fc300105350 2024-04-21T08:44:30.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.940+0000 7fc305206640 1 -- 172.21.15.7:0/2973827363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3001a22d0 con 0x7fc300105350 2024-04-21T08:44:30.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.940+0000 7fc2e7fff640 1 -- 172.21.15.7:0/2973827363 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4004ce0 con 0x7fc300105350 2024-04-21T08:44:30.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.940+0000 7fc305206640 1 -- 172.21.15.7:0/2973827363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3001a27d0 con 0x7fc300105350 2024-04-21T08:44:30.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.941+0000 7fc305206640 1 -- 172.21.15.7:0/2973827363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3000fdf20 con 0x7fc300105350 2024-04-21T08:44:30.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.942+0000 7fc2e7fff640 1 -- 172.21.15.7:0/2973827363 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fc2f4004e80 con 0x7fc300105350 2024-04-21T08:44:30.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.942+0000 7fc2e7fff640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc2e80a3bb0 0x7fc2e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.942+0000 7fc2fed9d640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc2e80a3bb0 0x7fc2e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.942+0000 7fc2e7fff640 1 -- 172.21.15.7:0/2973827363 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7fc2f40d01f0 con 0x7fc300105350 2024-04-21T08:44:30.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.943+0000 7fc2fed9d640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc2e80a3bb0 0x7fc2e80a6070 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009c20 tx=0x7fc2f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:30.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:30.947+0000 7fc2e7fff640 1 -- 172.21.15.7:0/2973827363 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2f409c7c0 con 0x7fc300105350 2024-04-21T08:44:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:30 smithi077 ceph-mon[31615]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:30 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2621307832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:30 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4077180583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:30 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1961984926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.133+0000 7fc305206640 1 -- 172.21.15.7:0/2973827363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc300062b50 con 0x7fc300105350 2024-04-21T08:44:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.133+0000 7fc2e7fff640 1 -- 172.21.15.7:0/2973827363 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fc2f409c0c0 con 0x7fc300105350 2024-04-21T08:44:31.134 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 -- 172.21.15.7:0/2973827363 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc2e80a3bb0 msgr2=0x7fc2e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc2e80a3bb0 0x7fc2e80a6070 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009c20 tx=0x7fc2f0005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 -- 172.21.15.7:0/2973827363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc300105350 msgr2=0x7fc3001009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc300105350 0x7fc3001009d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400d740 tx=0x7fc2f400dc00 comp rx=0 tx=0).stop 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 -- 172.21.15.7:0/2973827363 shutdown_connections 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc2e80a3bb0 0x7fc2e80a6070 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc300107c80 0x7fc300100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc300105350 0x7fc3001009d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 --2- 172.21.15.7:0/2973827363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc300102a20 0x7fc300102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 -- 172.21.15.7:0/2973827363 >> 172.21.15.7:0/2973827363 conn(0x7fc3000fc430 msgr2=0x7fc300103380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 -- 172.21.15.7:0/2973827363 shutdown_connections 2024-04-21T08:44:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.135+0000 7fc305206640 1 -- 172.21.15.7:0/2973827363 wait complete. 2024-04-21T08:44:31.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:30 smithi007 ceph-mon[28477]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:31.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:30 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2621307832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:31.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:30 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4077180583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:31.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:30 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1961984926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:31.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:31.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.251+0000 7fabf60f1640 1 -- 172.21.15.7:0/3138514542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf0107c70 msgr2=0x7fabf010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.251+0000 7fabf60f1640 1 --2- 172.21.15.7:0/3138514542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf0107c70 0x7fabf010a0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fabe8009900 tx=0x7fabe802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:31.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.251+0000 7fabf60f1640 1 -- 172.21.15.7:0/3138514542 shutdown_connections 2024-04-21T08:44:31.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.252+0000 7fabf60f1640 1 --2- 172.21.15.7:0/3138514542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf0107c70 0x7fabf010a0c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.252+0000 7fabf60f1640 1 --2- 172.21.15.7:0/3138514542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0105340 0x7fabf0107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.252+0000 7fabf60f1640 1 --2- 172.21.15.7:0/3138514542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabf0102a10 0x7fabf0104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.252+0000 7fabf60f1640 1 -- 172.21.15.7:0/3138514542 >> 172.21.15.7:0/3138514542 conn(0x7fabf00fc420 msgr2=0x7fabf00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:31.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.252+0000 7fabf60f1640 1 -- 172.21.15.7:0/3138514542 shutdown_connections 2024-04-21T08:44:31.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.252+0000 7fabf60f1640 1 -- 172.21.15.7:0/3138514542 wait complete. 2024-04-21T08:44:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.252+0000 7fabf60f1640 1 Processor -- start 2024-04-21T08:44:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.252+0000 7fabf60f1640 1 -- start start 2024-04-21T08:44:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.252+0000 7fabf60f1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf0102a10 0x7fabf0102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabf60f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabf0105340 0x7fabf01009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabf60f1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0107c70 0x7fabf0100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabf60f1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fabf010dae0 con 0x7fabf0102a10 2024-04-21T08:44:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabf60f1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabf010d960 con 0x7fabf0107c70 2024-04-21T08:44:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabf60f1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fabf010dc60 con 0x7fabf0105340 2024-04-21T08:44:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabef7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf0102a10 0x7fabf0102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabef7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf0102a10 0x7fabf0102380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59624/0 (socket says 172.21.15.7:59624) 2024-04-21T08:44:31.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabef7fe640 1 -- 172.21.15.7:0/1377669900 learned_addr learned my addr 172.21.15.7:0/1377669900 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabeeffd640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabf0105340 0x7fabf01009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabeffff640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0107c70 0x7fabf0100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabef7fe640 1 -- 172.21.15.7:0/1377669900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabf0105340 msgr2=0x7fabf01009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabef7fe640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabf0105340 0x7fabf01009c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabef7fe640 1 -- 172.21.15.7:0/1377669900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0107c70 msgr2=0x7fabf0100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabef7fe640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0107c70 0x7fabf0100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabef7fe640 1 -- 172.21.15.7:0/1377669900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf0101790 con 0x7fabf0102a10 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabeeffd640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabf0105340 0x7fabf01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabeffff640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0107c70 0x7fabf0100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabef7fe640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf0102a10 0x7fabf0102380 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fabe000cce0 tx=0x7fabe000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabecff9640 1 -- 172.21.15.7:0/1377669900 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe0004b20 con 0x7fabf0102a10 2024-04-21T08:44:31.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabecff9640 1 -- 172.21.15.7:0/1377669900 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabe0004cc0 con 0x7fabf0102a10 2024-04-21T08:44:31.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabecff9640 1 -- 172.21.15.7:0/1377669900 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe000f7c0 con 0x7fabf0102a10 2024-04-21T08:44:31.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabf60f1640 1 -- 172.21.15.7:0/1377669900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf01a22d0 con 0x7fabf0102a10 2024-04-21T08:44:31.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.253+0000 7fabf60f1640 1 -- 172.21.15.7:0/1377669900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf01a2820 con 0x7fabf0102a10 2024-04-21T08:44:31.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.254+0000 7fabf60f1640 1 -- 172.21.15.7:0/1377669900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabf00fdf10 con 0x7fabf0102a10 2024-04-21T08:44:31.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.256+0000 7fabecff9640 1 -- 172.21.15.7:0/1377669900 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fabe0005560 con 0x7fabf0102a10 2024-04-21T08:44:31.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.256+0000 7fabecff9640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fabdc0a3c30 0x7fabdc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.257+0000 7fabeeffd640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fabdc0a3c30 0x7fabdc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.257+0000 7fabecff9640 1 -- 172.21.15.7:0/1377669900 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7fabe00d0960 con 0x7fabf0102a10 2024-04-21T08:44:31.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.257+0000 7fabeeffd640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fabdc0a3c30 0x7fabdc0a60f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fabf01020f0 tx=0x7fabd8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:31.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.261+0000 7fabecff9640 1 -- 172.21.15.7:0/1377669900 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fabe0014030 con 0x7fabf0102a10 2024-04-21T08:44:31.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:30 smithi016 ceph-mon[31259]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:31.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:30 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2621307832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:31.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:30 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4077180583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:31.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:30 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1961984926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:31.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.446+0000 7fabf60f1640 1 -- 172.21.15.7:0/1377669900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fabf0062b50 con 0x7fabf0102a10 2024-04-21T08:44:31.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.447+0000 7fabecff9640 1 -- 172.21.15.7:0/1377669900 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fabe009cf30 con 0x7fabf0102a10 2024-04-21T08:44:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:31.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 -- 172.21.15.7:0/1377669900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fabdc0a3c30 msgr2=0x7fabdc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fabdc0a3c30 0x7fabdc0a60f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fabf01020f0 tx=0x7fabd8005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 -- 172.21.15.7:0/1377669900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf0102a10 msgr2=0x7fabf0102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf0102a10 0x7fabf0102380 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fabe000cce0 tx=0x7fabe000a620 comp rx=0 tx=0).stop 2024-04-21T08:44:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 -- 172.21.15.7:0/1377669900 shutdown_connections 2024-04-21T08:44:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fabdc0a3c30 0x7fabdc0a60f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabf0107c70 0x7fabf0100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabf0105340 0x7fabf01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 --2- 172.21.15.7:0/1377669900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabf0102a10 0x7fabf0102380 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 -- 172.21.15.7:0/1377669900 >> 172.21.15.7:0/1377669900 conn(0x7fabf00fc420 msgr2=0x7fabf0103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 -- 172.21.15.7:0/1377669900 shutdown_connections 2024-04-21T08:44:31.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.449+0000 7fabf60f1640 1 -- 172.21.15.7:0/1377669900 wait complete. 2024-04-21T08:44:31.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:31.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.565+0000 7fe28e2c1640 1 -- 172.21.15.7:0/1784695785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe288103b60 msgr2=0x7fe28810a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.565+0000 7fe28e2c1640 1 --2- 172.21.15.7:0/1784695785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe288103b60 0x7fe28810a3f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe280009ca0 tx=0x7fe28002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.565+0000 7fe28e2c1640 1 -- 172.21.15.7:0/1784695785 shutdown_connections 2024-04-21T08:44:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.565+0000 7fe28e2c1640 1 --2- 172.21.15.7:0/1784695785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe288103b60 0x7fe28810a3f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.565+0000 7fe28e2c1640 1 --2- 172.21.15.7:0/1784695785 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881031a0 0x7fe288103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.565+0000 7fe28e2c1640 1 --2- 172.21.15.7:0/1784695785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe288101fb0 0x7fe2881023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.565+0000 7fe28e2c1640 1 -- 172.21.15.7:0/1784695785 >> 172.21.15.7:0/1784695785 conn(0x7fe2880fd520 msgr2=0x7fe2880ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.565+0000 7fe28e2c1640 1 -- 172.21.15.7:0/1784695785 shutdown_connections 2024-04-21T08:44:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.565+0000 7fe28e2c1640 1 -- 172.21.15.7:0/1784695785 wait complete. 2024-04-21T08:44:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.566+0000 7fe28e2c1640 1 Processor -- start 2024-04-21T08:44:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.566+0000 7fe28e2c1640 1 -- start start 2024-04-21T08:44:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.566+0000 7fe28e2c1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe288101fb0 0x7fe288198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.566+0000 7fe28e2c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881031a0 0x7fe288198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.566+0000 7fe28e2c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe288103b60 0x7fe28819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.566+0000 7fe28e2c1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe28810daf0 con 0x7fe288101fb0 2024-04-21T08:44:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.566+0000 7fe28e2c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe28810d970 con 0x7fe288103b60 2024-04-21T08:44:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.566+0000 7fe28e2c1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe28810dc70 con 0x7fe2881031a0 2024-04-21T08:44:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.566+0000 7fe287fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe288101fb0 0x7fe288198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.566+0000 7fe287fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe288101fb0 0x7fe288198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59644/0 (socket says 172.21.15.7:59644) 2024-04-21T08:44:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe287fff640 1 -- 172.21.15.7:0/142954216 learned_addr learned my addr 172.21.15.7:0/142954216 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe2877fe640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881031a0 0x7fe288198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe28c85e640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe288103b60 0x7fe28819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe287fff640 1 -- 172.21.15.7:0/142954216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881031a0 msgr2=0x7fe288198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe287fff640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881031a0 0x7fe288198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe287fff640 1 -- 172.21.15.7:0/142954216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe288103b60 msgr2=0x7fe28819f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe287fff640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe288103b60 0x7fe28819f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe287fff640 1 -- 172.21.15.7:0/142954216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28819fd60 con 0x7fe288101fb0 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe28c85e640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe288103b60 0x7fe28819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe287fff640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe288101fb0 0x7fe288198140 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe27000c970 tx=0x7fe27000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe2857fa640 1 -- 172.21.15.7:0/142954216 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270007c10 con 0x7fe288101fb0 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe2857fa640 1 -- 172.21.15.7:0/142954216 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe270007db0 con 0x7fe288101fb0 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe2857fa640 1 -- 172.21.15.7:0/142954216 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27000f870 con 0x7fe288101fb0 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe28e2c1640 1 -- 172.21.15.7:0/142954216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28819fff0 con 0x7fe288101fb0 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.567+0000 7fe28e2c1640 1 -- 172.21.15.7:0/142954216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2881a04b0 con 0x7fe288101fb0 2024-04-21T08:44:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.568+0000 7fe28e2c1640 1 -- 172.21.15.7:0/142954216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2881092d0 con 0x7fe288101fb0 2024-04-21T08:44:31.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.569+0000 7fe2857fa640 1 -- 172.21.15.7:0/142954216 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fe27000fa10 con 0x7fe288101fb0 2024-04-21T08:44:31.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.570+0000 7fe2857fa640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2740a3bb0 0x7fe2740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.570+0000 7fe2877fe640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2740a3bb0 0x7fe2740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.570+0000 7fe2857fa640 1 -- 172.21.15.7:0/142954216 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7fe2700d0cb0 con 0x7fe288101fb0 2024-04-21T08:44:31.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.571+0000 7fe2877fe640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2740a3bb0 0x7fe2740a6070 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe278005fd0 tx=0x7fe278005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:31.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.574+0000 7fe2857fa640 1 -- 172.21.15.7:0/142954216 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe27009d2b0 con 0x7fe288101fb0 2024-04-21T08:44:31.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.763+0000 7fe28e2c1640 1 -- 172.21.15.7:0/142954216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe288062bf0 con 0x7fe288101fb0 2024-04-21T08:44:31.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.763+0000 7fe2857fa640 1 -- 172.21.15.7:0/142954216 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fe27009cbb0 con 0x7fe288101fb0 2024-04-21T08:44:31.764 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:44:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 -- 172.21.15.7:0/142954216 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2740a3bb0 msgr2=0x7fe2740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2740a3bb0 0x7fe2740a6070 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe278005fd0 tx=0x7fe278005850 comp rx=0 tx=0).stop 2024-04-21T08:44:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 -- 172.21.15.7:0/142954216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe288101fb0 msgr2=0x7fe288198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe288101fb0 0x7fe288198140 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe27000c970 tx=0x7fe27000ce30 comp rx=0 tx=0).stop 2024-04-21T08:44:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 -- 172.21.15.7:0/142954216 shutdown_connections 2024-04-21T08:44:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe2740a3bb0 0x7fe2740a6070 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe288103b60 0x7fe28819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2881031a0 0x7fe288198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 --2- 172.21.15.7:0/142954216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe288101fb0 0x7fe288198140 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 -- 172.21.15.7:0/142954216 >> 172.21.15.7:0/142954216 conn(0x7fe2880fd520 msgr2=0x7fe2881064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 -- 172.21.15.7:0/142954216 shutdown_connections 2024-04-21T08:44:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.765+0000 7fe28e2c1640 1 -- 172.21.15.7:0/142954216 wait complete. 2024-04-21T08:44:31.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.882+0000 7fc4fcec1640 1 -- 172.21.15.7:0/3164684766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f8101fa0 msgr2=0x7fc4f81023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.882+0000 7fc4fcec1640 1 --2- 172.21.15.7:0/3164684766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f8101fa0 0x7fc4f81023c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8009900 tx=0x7fc4e802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:31.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.882+0000 7fc4fcec1640 1 -- 172.21.15.7:0/3164684766 shutdown_connections 2024-04-21T08:44:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.882+0000 7fc4fcec1640 1 --2- 172.21.15.7:0/3164684766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4f8103b50 0x7fc4f810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.882+0000 7fc4fcec1640 1 --2- 172.21.15.7:0/3164684766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f8103190 0x7fc4f8103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.882+0000 7fc4fcec1640 1 --2- 172.21.15.7:0/3164684766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f8101fa0 0x7fc4f81023c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.882+0000 7fc4fcec1640 1 -- 172.21.15.7:0/3164684766 >> 172.21.15.7:0/3164684766 conn(0x7fc4f80fd510 msgr2=0x7fc4f80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.882+0000 7fc4fcec1640 1 -- 172.21.15.7:0/3164684766 shutdown_connections 2024-04-21T08:44:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.882+0000 7fc4fcec1640 1 -- 172.21.15.7:0/3164684766 wait complete. 2024-04-21T08:44:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4fcec1640 1 Processor -- start 2024-04-21T08:44:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4fcec1640 1 -- start start 2024-04-21T08:44:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4fcec1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4f8101fa0 0x7fc4f81981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4fcec1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f8103190 0x7fc4f8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4fcec1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f8103b50 0x7fc4f819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4fcec1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f810daa0 con 0x7fc4f8103b50 2024-04-21T08:44:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4fcec1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f810d920 con 0x7fc4f8101fa0 2024-04-21T08:44:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4fcec1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f810dc20 con 0x7fc4f8103190 2024-04-21T08:44:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4f6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f8103b50 0x7fc4f819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4f659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4f8101fa0 0x7fc4f81981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4f6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f8103b50 0x7fc4f819f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59668/0 (socket says 172.21.15.7:59668) 2024-04-21T08:44:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4f6d9d640 1 -- 172.21.15.7:0/4230341103 learned_addr learned my addr 172.21.15.7:0/4230341103 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4f5d9b640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f8103190 0x7fc4f8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4f6d9d640 1 -- 172.21.15.7:0/4230341103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f8103190 msgr2=0x7fc4f8198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4f6d9d640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f8103190 0x7fc4f8198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4f6d9d640 1 -- 172.21.15.7:0/4230341103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4f8101fa0 msgr2=0x7fc4f81981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4f6d9d640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4f8101fa0 0x7fc4f81981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.883+0000 7fc4f6d9d640 1 -- 172.21.15.7:0/4230341103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f819fde0 con 0x7fc4f8103b50 2024-04-21T08:44:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.884+0000 7fc4f6d9d640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f8103b50 0x7fc4f819f770 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc4f000cce0 tx=0x7fc4f000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.884+0000 7fc4e77fe640 1 -- 172.21.15.7:0/4230341103 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f000f450 con 0x7fc4f8103b50 2024-04-21T08:44:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.884+0000 7fc4e77fe640 1 -- 172.21.15.7:0/4230341103 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4f000faa0 con 0x7fc4f8103b50 2024-04-21T08:44:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.884+0000 7fc4fcec1640 1 -- 172.21.15.7:0/4230341103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f81a00d0 con 0x7fc4f8103b50 2024-04-21T08:44:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.884+0000 7fc4e77fe640 1 -- 172.21.15.7:0/4230341103 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0018b70 con 0x7fc4f8103b50 2024-04-21T08:44:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.884+0000 7fc4fcec1640 1 -- 172.21.15.7:0/4230341103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f81a05d0 con 0x7fc4f8103b50 2024-04-21T08:44:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.885+0000 7fc4fcec1640 1 -- 172.21.15.7:0/4230341103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f8109390 con 0x7fc4f8103b50 2024-04-21T08:44:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.886+0000 7fc4e77fe640 1 -- 172.21.15.7:0/4230341103 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7fc4f0018d10 con 0x7fc4f8103b50 2024-04-21T08:44:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.887+0000 7fc4e77fe640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4e00a3bb0 0x7fc4e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.887+0000 7fc4f659c640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4e00a3bb0 0x7fc4e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.887+0000 7fc4e77fe640 1 -- 172.21.15.7:0/4230341103 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7fc4f00d0b60 con 0x7fc4f8103b50 2024-04-21T08:44:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.887+0000 7fc4f659c640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4e00a3bb0 0x7fc4e00a6070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8005ab0 tx=0x7fc4e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:31.892+0000 7fc4e77fe640 1 -- 172.21.15.7:0/4230341103 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4f0014030 con 0x7fc4f8103b50 2024-04-21T08:44:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.081+0000 7fc4fcec1640 1 -- 172.21.15.7:0/4230341103 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc4f8062b50 con 0x7fc4f8103b50 2024-04-21T08:44:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.081+0000 7fc4e77fe640 1 -- 172.21.15.7:0/4230341103 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fc4f009d130 con 0x7fc4f8103b50 2024-04-21T08:44:32.082 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.083+0000 7fc4fcec1640 1 -- 172.21.15.7:0/4230341103 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4e00a3bb0 msgr2=0x7fc4e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.083+0000 7fc4fcec1640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4e00a3bb0 0x7fc4e00a6070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8005ab0 tx=0x7fc4e8005fd0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.083+0000 7fc4fcec1640 1 -- 172.21.15.7:0/4230341103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f8103b50 msgr2=0x7fc4f819f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.083+0000 7fc4fcec1640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f8103b50 0x7fc4f819f770 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc4f000cce0 tx=0x7fc4f000a620 comp rx=0 tx=0).stop 2024-04-21T08:44:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.083+0000 7fc4fcec1640 1 -- 172.21.15.7:0/4230341103 shutdown_connections 2024-04-21T08:44:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.083+0000 7fc4fcec1640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fc4e00a3bb0 0x7fc4e00a6070 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.083+0000 7fc4fcec1640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f8103b50 0x7fc4f819f770 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.083+0000 7fc4fcec1640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4f8103190 0x7fc4f8198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.083+0000 7fc4fcec1640 1 --2- 172.21.15.7:0/4230341103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4f8101fa0 0x7fc4f81981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.083+0000 7fc4fcec1640 1 -- 172.21.15.7:0/4230341103 >> 172.21.15.7:0/4230341103 conn(0x7fc4f80fd510 msgr2=0x7fc4f8106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.084+0000 7fc4fcec1640 1 -- 172.21.15.7:0/4230341103 shutdown_connections 2024-04-21T08:44:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.084+0000 7fc4fcec1640 1 -- 172.21.15.7:0/4230341103 wait complete. 2024-04-21T08:44:32.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:31 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2278559641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:32.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:31 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2973827363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:32.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:31 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1377669900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:32.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:31 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/142954216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:32.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:32.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:31 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2278559641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:32.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:31 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2973827363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:32.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:31 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1377669900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:32.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:31 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/142954216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.197+0000 7ff972175640 1 -- 172.21.15.7:0/2504059506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff96c101b50 msgr2=0x7ff96c101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.197+0000 7ff972175640 1 --2- 172.21.15.7:0/2504059506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff96c101b50 0x7ff96c101fd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff964009ca0 tx=0x7ff96402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.197+0000 7ff972175640 1 -- 172.21.15.7:0/2504059506 shutdown_connections 2024-04-21T08:44:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.197+0000 7ff972175640 1 --2- 172.21.15.7:0/2504059506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff96c101b50 0x7ff96c101fd0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.197+0000 7ff972175640 1 --2- 172.21.15.7:0/2504059506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff96c1011b0 0x7ff96c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.197+0000 7ff972175640 1 --2- 172.21.15.7:0/2504059506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff96c100850 0x7ff96c100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.197+0000 7ff972175640 1 -- 172.21.15.7:0/2504059506 >> 172.21.15.7:0/2504059506 conn(0x7ff96c0fc3f0 msgr2=0x7ff96c0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.197+0000 7ff972175640 1 -- 172.21.15.7:0/2504059506 shutdown_connections 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.197+0000 7ff972175640 1 -- 172.21.15.7:0/2504059506 wait complete. 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.198+0000 7ff972175640 1 Processor -- start 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.198+0000 7ff972175640 1 -- start start 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.198+0000 7ff972175640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff96c100850 0x7ff96c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.198+0000 7ff972175640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff96c1011b0 0x7ff96c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.198+0000 7ff972175640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff96c101b50 0x7ff96c1a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.198+0000 7ff972175640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff96c10fcd0 con 0x7ff96c1011b0 2024-04-21T08:44:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.198+0000 7ff972175640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff96c10fb50 con 0x7ff96c101b50 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.198+0000 7ff972175640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff96c10fe50 con 0x7ff96c100850 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.198+0000 7ff96affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff96c1011b0 0x7ff96c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.198+0000 7ff96affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff96c1011b0 0x7ff96c19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59680/0 (socket says 172.21.15.7:59680) 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.198+0000 7ff96affd640 1 -- 172.21.15.7:0/629285314 learned_addr learned my addr 172.21.15.7:0/629285314 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff96bfff640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff96c101b50 0x7ff96c1a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff96b7fe640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff96c100850 0x7ff96c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff96affd640 1 -- 172.21.15.7:0/629285314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff96c100850 msgr2=0x7ff96c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff96affd640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff96c100850 0x7ff96c19a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff96affd640 1 -- 172.21.15.7:0/629285314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff96c101b50 msgr2=0x7ff96c1a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff96affd640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff96c101b50 0x7ff96c1a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff96affd640 1 -- 172.21.15.7:0/629285314 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96c1a1e70 con 0x7ff96c1011b0 2024-04-21T08:44:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff96affd640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff96c1011b0 0x7ff96c19a790 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff96000c920 tx=0x7ff96000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:32.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff968ff9640 1 -- 172.21.15.7:0/629285314 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960007c70 con 0x7ff96c1011b0 2024-04-21T08:44:32.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff968ff9640 1 -- 172.21.15.7:0/629285314 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff960007e10 con 0x7ff96c1011b0 2024-04-21T08:44:32.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff972175640 1 -- 172.21.15.7:0/629285314 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff96c1a2160 con 0x7ff96c1011b0 2024-04-21T08:44:32.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff968ff9640 1 -- 172.21.15.7:0/629285314 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960018820 con 0x7ff96c1011b0 2024-04-21T08:44:32.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.199+0000 7ff972175640 1 -- 172.21.15.7:0/629285314 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff96c1a2700 con 0x7ff96c1011b0 2024-04-21T08:44:32.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.200+0000 7ff972175640 1 -- 172.21.15.7:0/629285314 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff96c10b6b0 con 0x7ff96c1011b0 2024-04-21T08:44:32.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.201+0000 7ff968ff9640 1 -- 172.21.15.7:0/629285314 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7ff96000f450 con 0x7ff96c1011b0 2024-04-21T08:44:32.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.202+0000 7ff968ff9640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff9540a3c30 0x7ff9540a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.202+0000 7ff96b7fe640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff9540a3c30 0x7ff9540a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.202+0000 7ff968ff9640 1 -- 172.21.15.7:0/629285314 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7ff9600d0b80 con 0x7ff96c1011b0 2024-04-21T08:44:32.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.203+0000 7ff96b7fe640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff9540a3c30 0x7ff9540a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009aa0 tx=0x7ff95c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:32.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.206+0000 7ff968ff9640 1 -- 172.21.15.7:0/629285314 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff960014030 con 0x7ff96c1011b0 2024-04-21T08:44:32.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:31 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2278559641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:32.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:31 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2973827363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:32.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:31 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1377669900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:32.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:31 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/142954216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:32.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.433+0000 7ff972175640 1 -- 172.21.15.7:0/629285314 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff96c02d830 con 0x7ff96c1011b0 2024-04-21T08:44:32.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.434+0000 7ff968ff9640 1 -- 172.21.15.7:0/629285314 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7ff96009d260 con 0x7ff96c1011b0 2024-04-21T08:44:32.434 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:32.435 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:44:32.435 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:44:32.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 -- 172.21.15.7:0/629285314 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff9540a3c30 msgr2=0x7ff9540a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff9540a3c30 0x7ff9540a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009aa0 tx=0x7ff95c005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 -- 172.21.15.7:0/629285314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff96c1011b0 msgr2=0x7ff96c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff96c1011b0 0x7ff96c19a790 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff96000c920 tx=0x7ff96000cde0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 -- 172.21.15.7:0/629285314 shutdown_connections 2024-04-21T08:44:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7ff9540a3c30 0x7ff9540a60f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff96c101b50 0x7ff96c1a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff96c1011b0 0x7ff96c19a790 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 --2- 172.21.15.7:0/629285314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff96c100850 0x7ff96c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 -- 172.21.15.7:0/629285314 >> 172.21.15.7:0/629285314 conn(0x7ff96c0fc3f0 msgr2=0x7ff96c0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 -- 172.21.15.7:0/629285314 shutdown_connections 2024-04-21T08:44:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.436+0000 7ff972175640 1 -- 172.21.15.7:0/629285314 wait complete. 2024-04-21T08:44:32.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.549+0000 7f35a07e6640 1 -- 172.21.15.7:0/3175338469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3598100810 msgr2=0x7f3598102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.549+0000 7f35a07e6640 1 --2- 172.21.15.7:0/3175338469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3598100810 0x7f3598102c40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3594009900 tx=0x7f359402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.549+0000 7f35a07e6640 1 -- 172.21.15.7:0/3175338469 shutdown_connections 2024-04-21T08:44:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.549+0000 7f35a07e6640 1 --2- 172.21.15.7:0/3175338469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3598105b30 0x7f3598105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.549+0000 7f35a07e6640 1 --2- 172.21.15.7:0/3175338469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3598103180 0x7f35981055f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.549+0000 7f35a07e6640 1 --2- 172.21.15.7:0/3175338469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3598100810 0x7f3598102c40 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.549+0000 7f35a07e6640 1 -- 172.21.15.7:0/3175338469 >> 172.21.15.7:0/3175338469 conn(0x7f35980fc410 msgr2=0x7f35980fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.549+0000 7f35a07e6640 1 -- 172.21.15.7:0/3175338469 shutdown_connections 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.549+0000 7f35a07e6640 1 -- 172.21.15.7:0/3175338469 wait complete. 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f35a07e6640 1 Processor -- start 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f35a07e6640 1 -- start start 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f35a07e6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3598100810 0x7f359819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f35a07e6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3598103180 0x7f359819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f35a07e6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3598105b30 0x7f35981a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f35a07e6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3598114110 con 0x7f3598100810 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f35a07e6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3598113f90 con 0x7f3598105b30 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f35a07e6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3598114290 con 0x7f3598103180 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f359e582640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3598100810 0x7f359819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f359e582640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3598100810 0x7f359819e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59706/0 (socket says 172.21.15.7:59706) 2024-04-21T08:44:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f359e582640 1 -- 172.21.15.7:0/935911786 learned_addr learned my addr 172.21.15.7:0/935911786 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f359dd81640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3598103180 0x7f359819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f359e582640 1 -- 172.21.15.7:0/935911786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3598103180 msgr2=0x7f359819ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f359e582640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3598103180 0x7f359819ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f359e582640 1 -- 172.21.15.7:0/935911786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3598105b30 msgr2=0x7f35981a5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f359e582640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3598105b30 0x7f35981a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.550+0000 7f359e582640 1 -- 172.21.15.7:0/935911786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35981a6420 con 0x7f3598100810 2024-04-21T08:44:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.551+0000 7f359e582640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3598100810 0x7f359819e850 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3594006010 tx=0x7f35940042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:32.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.551+0000 7f358f7fe640 1 -- 172.21.15.7:0/935911786 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594043020 con 0x7f3598100810 2024-04-21T08:44:32.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.551+0000 7f358f7fe640 1 -- 172.21.15.7:0/935911786 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3594004aa0 con 0x7f3598100810 2024-04-21T08:44:32.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.551+0000 7f35a07e6640 1 -- 172.21.15.7:0/935911786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35981a66b0 con 0x7f3598100810 2024-04-21T08:44:32.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.551+0000 7f358f7fe640 1 -- 172.21.15.7:0/935911786 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594003c70 con 0x7f3598100810 2024-04-21T08:44:32.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.551+0000 7f35a07e6640 1 -- 172.21.15.7:0/935911786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35981a6c80 con 0x7f3598100810 2024-04-21T08:44:32.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.552+0000 7f35a07e6640 1 -- 172.21.15.7:0/935911786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35980fdf00 con 0x7f3598100810 2024-04-21T08:44:32.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.553+0000 7f358f7fe640 1 -- 172.21.15.7:0/935911786 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f3594046020 con 0x7f3598100810 2024-04-21T08:44:32.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.553+0000 7f358f7fe640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f35840a3bb0 0x7f35840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.554+0000 7f359dd81640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f35840a3bb0 0x7f35840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.554+0000 7f358f7fe640 1 -- 172.21.15.7:0/935911786 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f35940f3320 con 0x7f3598100810 2024-04-21T08:44:32.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.554+0000 7f359dd81640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f35840a3bb0 0x7f35840a6070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3588005fd0 tx=0x7f3588005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:32.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.557+0000 7f358f7fe640 1 -- 172.21.15.7:0/935911786 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f359404d020 con 0x7f3598100810 2024-04-21T08:44:32.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.771+0000 7f35a07e6640 1 -- 172.21.15.7:0/935911786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3598062bf0 con 0x7f3598100810 2024-04-21T08:44:32.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.773+0000 7f358f7fe640 1 -- 172.21.15.7:0/935911786 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24714 (secure 0 0 0) 0x7f35940bf8f0 con 0x7f3598100810 2024-04-21T08:44:32.774 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:32.774 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":94,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:44:11.760990+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:32.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 -- 172.21.15.7:0/935911786 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f35840a3bb0 msgr2=0x7f35840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f35840a3bb0 0x7f35840a6070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3588005fd0 tx=0x7f3588005850 comp rx=0 tx=0).stop 2024-04-21T08:44:32.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 -- 172.21.15.7:0/935911786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3598100810 msgr2=0x7f359819e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3598100810 0x7f359819e850 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3594006010 tx=0x7f35940042f0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 -- 172.21.15.7:0/935911786 shutdown_connections 2024-04-21T08:44:32.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f35840a3bb0 0x7f35840a6070 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3598105b30 0x7f35981a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3598103180 0x7f359819ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 --2- 172.21.15.7:0/935911786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3598100810 0x7f359819e850 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 -- 172.21.15.7:0/935911786 >> 172.21.15.7:0/935911786 conn(0x7f35980fc410 msgr2=0x7f3598103b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:32.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 -- 172.21.15.7:0/935911786 shutdown_connections 2024-04-21T08:44:32.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.775+0000 7f35a07e6640 1 -- 172.21.15.7:0/935911786 wait complete. 2024-04-21T08:44:32.788 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:42:16.996517+0000', 'modified': '2024-04-21T08:42:42.080753+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24533}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24532': {'gid': 24532, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6839/516970569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 516970569}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 516970569}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6839/1703654929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 1703654929}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 1703654929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24533, 'qdb_cluster': [24533]} 2024-04-21T08:44:32.788 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T08:44:32.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:44:32.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.891+0000 7f05d398e640 1 -- 172.21.15.7:0/3543318708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05cc100ea0 msgr2=0x7f05cc1012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.891+0000 7f05d398e640 1 --2- 172.21.15.7:0/3543318708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05cc100ea0 0x7f05cc1012c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009900 tx=0x7f05c802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:32.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.891+0000 7f05d398e640 1 -- 172.21.15.7:0/3543318708 shutdown_connections 2024-04-21T08:44:32.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.891+0000 7f05d398e640 1 --2- 172.21.15.7:0/3543318708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05cc1029b0 0x7f05cc1092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.891+0000 7f05d398e640 1 --2- 172.21.15.7:0/3543318708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05cc101ff0 0x7f05cc102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.891+0000 7f05d398e640 1 --2- 172.21.15.7:0/3543318708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05cc100ea0 0x7f05cc1012c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.891+0000 7f05d398e640 1 -- 172.21.15.7:0/3543318708 >> 172.21.15.7:0/3543318708 conn(0x7f05cc0fc430 msgr2=0x7f05cc0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:32.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.891+0000 7f05d398e640 1 -- 172.21.15.7:0/3543318708 shutdown_connections 2024-04-21T08:44:32.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.891+0000 7f05d398e640 1 -- 172.21.15.7:0/3543318708 wait complete. 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d398e640 1 Processor -- start 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d398e640 1 -- start start 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d398e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05cc100ea0 0x7f05cc19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d398e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05cc101ff0 0x7f05cc19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d398e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05cc1029b0 0x7f05cc1a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d398e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f05cc10c9f0 con 0x7f05cc100ea0 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d398e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05cc10c870 con 0x7f05cc1029b0 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d398e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05cc10cb70 con 0x7f05cc101ff0 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d172a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05cc100ea0 0x7f05cc19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d172a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05cc100ea0 0x7f05cc19a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59732/0 (socket says 172.21.15.7:59732) 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d172a640 1 -- 172.21.15.7:0/4146286262 learned_addr learned my addr 172.21.15.7:0/4146286262 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d0f29640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05cc101ff0 0x7f05cc19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d172a640 1 -- 172.21.15.7:0/4146286262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05cc101ff0 msgr2=0x7f05cc19a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d172a640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05cc101ff0 0x7f05cc19a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d172a640 1 -- 172.21.15.7:0/4146286262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05cc1029b0 msgr2=0x7f05cc1a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d172a640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05cc1029b0 0x7f05cc1a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d172a640 1 -- 172.21.15.7:0/4146286262 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05cc1a1fd0 con 0x7f05cc100ea0 2024-04-21T08:44:32.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.892+0000 7f05d172a640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05cc100ea0 0x7f05cc19a3b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f05c8006010 tx=0x7f05c8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:32.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.893+0000 7f05c27fc640 1 -- 172.21.15.7:0/4146286262 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8043020 con 0x7f05cc100ea0 2024-04-21T08:44:32.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.893+0000 7f05c27fc640 1 -- 172.21.15.7:0/4146286262 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c80043d0 con 0x7f05cc100ea0 2024-04-21T08:44:32.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.893+0000 7f05d398e640 1 -- 172.21.15.7:0/4146286262 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05cc1a2260 con 0x7f05cc100ea0 2024-04-21T08:44:32.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.893+0000 7f05c27fc640 1 -- 172.21.15.7:0/4146286262 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8003b80 con 0x7f05cc100ea0 2024-04-21T08:44:32.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.893+0000 7f05d398e640 1 -- 172.21.15.7:0/4146286262 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05cc1a27b0 con 0x7f05cc100ea0 2024-04-21T08:44:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.895+0000 7f05c27fc640 1 -- 172.21.15.7:0/4146286262 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f05c8046020 con 0x7f05cc100ea0 2024-04-21T08:44:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d6576740 1 Processor -- start 2024-04-21T08:44:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d6576740 1 -- start start 2024-04-21T08:44:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d6576740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca280260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d6576740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca280cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d6576740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca287eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d6576740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55cdca1e4b10 con 0x55cdca27fe40 2024-04-21T08:44:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d6576740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cdca182240 con 0x55cdca280830 2024-04-21T08:44:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d6576740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cdca1e0ad0 con 0x55cdca2812d0 2024-04-21T08:44:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d1f2b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca280260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d1f2b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca280260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59736/0 (socket says 172.21.15.7:59736) 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d1f2b640 1 -- 172.21.15.7:0/3238847573 learned_addr learned my addr 172.21.15.7:0/3238847573 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05c27fc640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f05b80a3bb0 0x7f05b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d0f29640 1 --2- 172.21.15.7:0/3238847573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca287eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.896+0000 7f05d172a640 1 --2- 172.21.15.7:0/3238847573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca280cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05d1f2b640 1 -- 172.21.15.7:0/3238847573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 msgr2=0x55cdca287eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05d1f2b640 1 --2- 172.21.15.7:0/3238847573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca287eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05d1f2b640 1 -- 172.21.15.7:0/3238847573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 msgr2=0x55cdca280cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05d1f2b640 1 --2- 172.21.15.7:0/3238847573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca280cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05d1f2b640 1 -- 172.21.15.7:0/3238847573 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdc9ef72d0 con 0x55cdca27fe40 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05d0f29640 1 --2- 172.21.15.7:0/3238847573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca287eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05d172a640 1 --2- 172.21.15.7:0/3238847573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca280cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05c27fc640 1 -- 172.21.15.7:0/4146286262 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f05c80f2b80 con 0x7f05cc100ea0 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05d1f2b640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f05b80a3bb0 0x7f05b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05d1f2b640 1 --2- 172.21.15.7:0/3238847573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca280260 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f05c4009fd0 tx=0x7f05c400e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be317ed04e362a0d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05b77fe640 1 -- 172.21.15.7:0/3238847573 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c401b8e0 con 0x55cdca27fe40 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05b77fe640 1 -- 172.21.15.7:0/3238847573 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c401a410 con 0x55cdca27fe40 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05b77fe640 1 -- 172.21.15.7:0/3238847573 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c400f620 con 0x55cdca27fe40 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.897+0000 7f05d1f2b640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f05b80a3bb0 0x7f05b80a6070 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f05c401f000 tx=0x7f05c4020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.898+0000 7f05d6576740 1 -- 172.21.15.7:0/3238847573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 msgr2=0x55cdca280260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.898+0000 7f05d6576740 1 --2- 172.21.15.7:0/3238847573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca280260 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f05c4009fd0 tx=0x7f05c400e830 comp rx=0 tx=0).stop 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.898+0000 7f05d6576740 1 -- 172.21.15.7:0/3238847573 shutdown_connections 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.898+0000 7f05d6576740 1 --2- 172.21.15.7:0/3238847573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca287eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.898+0000 7f05d6576740 1 --2- 172.21.15.7:0/3238847573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca280cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.898+0000 7f05d6576740 1 --2- 172.21.15.7:0/3238847573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca280260 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.898+0000 7f05d6576740 1 -- 172.21.15.7:0/3238847573 >> 172.21.15.7:0/3238847573 conn(0x55cdca27bf40 msgr2=0x55cdca27a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.898+0000 7f05d6576740 1 -- 172.21.15.7:0/3238847573 shutdown_connections 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.898+0000 7f05d6576740 1 -- 172.21.15.7:0/3238847573 wait complete. 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.898+0000 7f05d6576740 1 -- start start 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.898+0000 7f05d6576740 1 Processor -- start 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d6576740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d6576740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca28a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d6576740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca291380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d6576740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55cdca2884b0 con 0x55cdca27fe40 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d6576740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cdca16d190 con 0x55cdca280830 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d6576740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cdca288630 con 0x55cdca2812d0 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d6576740 10 client.0 _ll_register_callbacks cb 0x55cdca1688b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d172a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d172a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59752/0 (socket says 172.21.15.7:59752) 2024-04-21T08:44:32.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d172a640 1 -- 172.21.15.7:0/3442816939 learned_addr learned my addr 172.21.15.7:0/3442816939 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d0f29640 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca28a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d1f2b640 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca291380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d172a640 1 -- 172.21.15.7:0/3442816939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 msgr2=0x55cdca291380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d172a640 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca291380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d172a640 1 -- 172.21.15.7:0/3442816939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 msgr2=0x55cdca28a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d172a640 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca28a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d172a640 1 -- 172.21.15.7:0/3442816939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdc9fc16b0 con 0x55cdca27fe40 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d172a640 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289e00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f05c80bb420 tx=0x7f05c808e380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b724cd7b45cc6140 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05b4ff9640 1 -- 172.21.15.7:0/3442816939 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c80df0b0 con 0x55cdca27fe40 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05b4ff9640 1 -- 172.21.15.7:0/3442816939 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f05c8001ad0 con 0x55cdca27fe40 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.899+0000 7f05d6576740 1 -- 172.21.15.7:0/3442816939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05c401a410 con 0x55cdca27fe40 2024-04-21T08:44:32.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05b4ff9640 1 -- 172.21.15.7:0/3442816939 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f05c80895b0 con 0x55cdca27fe40 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05d6576740 10 client.15378 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05d6576740 1 -- 172.21.15.7:0/3442816939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55cdca294660 con 0x55cdca27fe40 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05b4ff9640 1 -- 172.21.15.7:0/3442816939 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c401a410 con 0x55cdca27fe40 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05b4ff9640 1 -- 172.21.15.7:0/3442816939 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f05c80becb0 con 0x55cdca27fe40 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05d6576740 10 client.15378 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05d6576740 10 client.15378 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05d6576740 20 client.15378 populate_metadata read hostname 'smithi007' 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05d6576740 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55cdca296d70 0x55cdca2981e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05d6576740 4 client.15378 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05d6576740 1 -- 172.21.15.7:0/3442816939 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdca1f2530 con 0x55cdca296d70 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.900+0000 7f05d0f29640 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55cdca296d70 0x55cdca2981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.901+0000 7f05d0f29640 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55cdca296d70 0x55cdca2981e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30876c1ed379d30a server_cookie=cdfc2a43948825ca in_seq=0 out_seq=0 2024-04-21T08:44:32.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.901+0000 7f05b4ff9640 10 client.15378 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.901+0000 7f05b4ff9640 1 -- 172.21.15.7:0/3442816939 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cdca1f2530 con 0x55cdca296d70 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.901+0000 7f05b4ff9640 10 client.15378 handle_command_reply: tid=0 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.901+0000 7f05d6576740 1 client.15378 shutdown 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05d6576740 1 -- 172.21.15.7:0/3442816939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 msgr2=0x55cdca289e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05d6576740 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289e00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f05c80bb420 tx=0x7f05c808e380 comp rx=0 tx=0).stop 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05d6576740 1 -- 172.21.15.7:0/3442816939 shutdown_connections 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05d6576740 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55cdca296d70 0x55cdca2981e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05d6576740 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca291380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05d6576740 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca28a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05b4ff9640 0 client.15378 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05d6576740 1 --2- 172.21.15.7:0/3442816939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289e00 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05d6576740 1 -- 172.21.15.7:0/3442816939 >> 172.21.15.7:0/3442816939 conn(0x55cdca27bf40 msgr2=0x55cdca279970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05d6576740 1 -- 172.21.15.7:0/3442816939 shutdown_connections 2024-04-21T08:44:32.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05d6576740 1 -- 172.21.15.7:0/3442816939 wait complete. 2024-04-21T08:44:32.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.902+0000 7f05d6576740 20 client.15378 trim_cache size 0 max 16384 2024-04-21T08:44:32.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.920+0000 7f05d6576740 1 Processor -- start 2024-04-21T08:44:32.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.920+0000 7f05d6576740 1 -- start start 2024-04-21T08:44:32.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.920+0000 7f05d6576740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.920+0000 7f05d6576740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca291310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.920+0000 7f05d6576740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca289f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.920+0000 7f05d6576740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55cdca296520 con 0x55cdca27fe40 2024-04-21T08:44:32.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.920+0000 7f05d6576740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cdca296bf0 con 0x55cdca280830 2024-04-21T08:44:32.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.920+0000 7f05d6576740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cdca2972c0 con 0x55cdca2812d0 2024-04-21T08:44:32.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.920+0000 7f05d172a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.920+0000 7f05d172a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59754/0 (socket says 172.21.15.7:59754) 2024-04-21T08:44:32.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.920+0000 7f05d172a640 1 -- 172.21.15.7:0/2019498810 learned_addr learned my addr 172.21.15.7:0/2019498810 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:32.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d172a640 1 -- 172.21.15.7:0/2019498810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 msgr2=0x55cdca289f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:32.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d1f2b640 1 --2- 172.21.15.7:0/2019498810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca289f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d172a640 1 --2- 172.21.15.7:0/2019498810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca289f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d172a640 1 -- 172.21.15.7:0/2019498810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 msgr2=0x55cdca291310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:32.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d172a640 1 --2- 172.21.15.7:0/2019498810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca291310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d172a640 1 -- 172.21.15.7:0/2019498810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdca27b230 con 0x55cdca27fe40 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d1f2b640 1 --2- 172.21.15.7:0/2019498810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca289f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d172a640 1 --2- 172.21.15.7:0/2019498810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289a10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f05c8005ab0 tx=0x7f05c80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d5fa4e18f2aacf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05b6ffd640 1 -- 172.21.15.7:0/2019498810 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c80c7710 con 0x55cdca27fe40 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05b6ffd640 1 -- 172.21.15.7:0/2019498810 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c80bf150 con 0x55cdca27fe40 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05b6ffd640 1 -- 172.21.15.7:0/2019498810 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c80c7230 con 0x55cdca27fe40 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d6576740 1 -- 172.21.15.7:0/2019498810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 msgr2=0x55cdca289a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d6576740 1 --2- 172.21.15.7:0/2019498810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289a10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f05c8005ab0 tx=0x7f05c80035f0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d6576740 1 -- 172.21.15.7:0/2019498810 shutdown_connections 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d6576740 1 --2- 172.21.15.7:0/2019498810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca289f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d6576740 1 --2- 172.21.15.7:0/2019498810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca280830 0x55cdca291310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.921+0000 7f05d6576740 1 --2- 172.21.15.7:0/2019498810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca27fe40 0x55cdca289a10 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 1 -- 172.21.15.7:0/2019498810 >> 172.21.15.7:0/2019498810 conn(0x55cdca27bf40 msgr2=0x55cdca2959b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:32.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 1 -- 172.21.15.7:0/2019498810 shutdown_connections 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 1 -- 172.21.15.7:0/2019498810 wait complete. 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 1 -- start start 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 1 Processor -- start 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca27fe40 0x55cdca28cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca280830 0x55cdca28d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca2904f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55cdca298060 con 0x55cdca280830 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55cdca297990 con 0x55cdca27fe40 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cdca298730 con 0x55cdca2812d0 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d6576740 10 client.0 _ll_register_callbacks cb 0x7f05c80bacf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d0f29640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca280830 0x55cdca28d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d0f29640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca280830 0x55cdca28d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59766/0 (socket says 172.21.15.7:59766) 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.922+0000 7f05d0f29640 1 -- 172.21.15.7:0/433981305 learned_addr learned my addr 172.21.15.7:0/433981305 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05d172a640 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca27fe40 0x55cdca28cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05d0f29640 1 -- 172.21.15.7:0/433981305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 msgr2=0x55cdca2904f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05d0f29640 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca2904f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05d0f29640 1 -- 172.21.15.7:0/433981305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca27fe40 msgr2=0x55cdca28cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05d0f29640 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca27fe40 0x55cdca28cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05d0f29640 1 -- 172.21.15.7:0/433981305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdca290ae0 con 0x55cdca280830 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05d172a640 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca27fe40 0x55cdca28cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05d0f29640 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca280830 0x55cdca28d4d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009ca0 tx=0x7f05bc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2bc3a709c53ce42 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05b4ff9640 1 -- 172.21.15.7:0/433981305 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc004230 con 0x55cdca280830 2024-04-21T08:44:32.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05b4ff9640 1 -- 172.21.15.7:0/433981305 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f05bc0043d0 con 0x55cdca280830 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05d6576740 1 -- 172.21.15.7:0/433981305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05c80bf150 con 0x55cdca280830 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05b4ff9640 1 -- 172.21.15.7:0/433981305 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f05bc02a050 con 0x55cdca280830 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05d6576740 10 client.15384 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05d6576740 1 -- 172.21.15.7:0/433981305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55cdca2fd320 con 0x55cdca280830 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05b4ff9640 1 -- 172.21.15.7:0/433981305 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c80bf150 con 0x55cdca280830 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.923+0000 7f05b4ff9640 1 -- 172.21.15.7:0/433981305 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f05bc003b10 con 0x55cdca280830 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d6576740 10 client.15384 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d6576740 10 client.15384 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d6576740 20 client.15384 populate_metadata read hostname 'smithi007' 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d6576740 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55cdca302020 0x55cdca300d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d6576740 4 client.15384 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d6576740 1 -- 172.21.15.7:0/433981305 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "status"}) -- 0x55cdca1ec3a0 con 0x55cdca302020 2024-04-21T08:44:32.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d172a640 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55cdca302020 0x55cdca300d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d172a640 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55cdca302020 0x55cdca300d30 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91cb5e6a361be9bb server_cookie=ca0c1d7e9a9df0b in_seq=0 out_seq=0 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05b4ff9640 10 client.15384 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05b4ff9640 1 -- 172.21.15.7:0/433981305 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55cdca1ec3a0 con 0x55cdca302020 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05b4ff9640 10 client.15384 handle_command_reply: tid=0 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d6576740 1 client.15384 shutdown 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d6576740 1 -- 172.21.15.7:0/433981305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca280830 msgr2=0x55cdca28d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d6576740 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca280830 0x55cdca28d4d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009ca0 tx=0x7f05bc011810 comp rx=0 tx=0).stop 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.924+0000 7f05d6576740 1 -- 172.21.15.7:0/433981305 shutdown_connections 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d6576740 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55cdca302020 0x55cdca300d30 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d6576740 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cdca2812d0 0x55cdca2904f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05b4ff9640 0 client.15384 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d6576740 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cdca280830 0x55cdca28d4d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d6576740 1 --2- 172.21.15.7:0/433981305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55cdca27fe40 0x55cdca28cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d6576740 1 -- 172.21.15.7:0/433981305 >> 172.21.15.7:0/433981305 conn(0x55cdca27bf40 msgr2=0x55cdca2959b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:32.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d6576740 1 -- 172.21.15.7:0/433981305 shutdown_connections 2024-04-21T08:44:32.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d6576740 1 -- 172.21.15.7:0/433981305 wait complete. 2024-04-21T08:44:32.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d6576740 20 client.15384 trim_cache size 0 max 16384 2024-04-21T08:44:32.933 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:44:32.933 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "c9c60282-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:44:32.933 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2024-04-21T08:44:32.933 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24533, 2024-04-21T08:44:32.933 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-04-21T08:44:32.933 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-04-21T08:44:32.933 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-04-21T08:44:32.934 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 135.30151463000001, 2024-04-21T08:44:32.934 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 21, 2024-04-21T08:44:32.934 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 94, 2024-04-21T08:44:32.934 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T08:44:32.934 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 141.442999308 2024-04-21T08:44:32.934 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:44:32.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 -- 172.21.15.7:0/4146286262 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f05b80a3bb0 msgr2=0x7f05b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f05b80a3bb0 0x7f05b80a6070 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f05c401f000 tx=0x7f05c4020000 comp rx=0 tx=0).stop 2024-04-21T08:44:32.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 -- 172.21.15.7:0/4146286262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05cc100ea0 msgr2=0x7f05cc19a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:32.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05cc100ea0 0x7f05cc19a3b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f05c8006010 tx=0x7f05c8005e40 comp rx=0 tx=0).stop 2024-04-21T08:44:32.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 -- 172.21.15.7:0/4146286262 shutdown_connections 2024-04-21T08:44:32.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f05b80a3bb0 0x7f05b80a6070 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05cc1029b0 0x7f05cc1a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05cc101ff0 0x7f05cc19a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 --2- 172.21.15.7:0/4146286262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05cc100ea0 0x7f05cc19a3b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:32.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 -- 172.21.15.7:0/4146286262 >> 172.21.15.7:0/4146286262 conn(0x7f05cc0fc430 msgr2=0x7f05cc0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:32.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 -- 172.21.15.7:0/4146286262 shutdown_connections 2024-04-21T08:44:32.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:32.925+0000 7f05d398e640 1 -- 172.21.15.7:0/4146286262 wait complete. 2024-04-21T08:44:32.937 INFO:teuthology.run_tasks:Running task sequential... 2024-04-21T08:44:32.947 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-21T08:44:32.947 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-21T08:44:32.947 INFO:teuthology.task.print:Enabling mgr modules 2024-04-21T08:44:32.947 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-21T08:44:32.948 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:44:32.948 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-21T08:44:32.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-21T08:44:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.049+0000 7f7670bef640 1 -- 172.21.15.7:0/1215735347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c107cc0 msgr2=0x7f766c10a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.049+0000 7f7670bef640 1 --2- 172.21.15.7:0/1215735347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c107cc0 0x7f766c10a110 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7664009900 tx=0x7f766402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 -- 172.21.15.7:0/1215735347 shutdown_connections 2024-04-21T08:44:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 --2- 172.21.15.7:0/1215735347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c107cc0 0x7f766c10a110 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 --2- 172.21.15.7:0/1215735347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c105390 0x7f766c107780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 --2- 172.21.15.7:0/1215735347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f766c102a60 0x7f766c104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 -- 172.21.15.7:0/1215735347 >> 172.21.15.7:0/1215735347 conn(0x7f766c0fc470 msgr2=0x7f766c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 -- 172.21.15.7:0/1215735347 shutdown_connections 2024-04-21T08:44:33.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 -- 172.21.15.7:0/1215735347 wait complete. 2024-04-21T08:44:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 Processor -- start 2024-04-21T08:44:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 -- start start 2024-04-21T08:44:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c102a60 0x7f766c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c105390 0x7f766c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f766c107cc0 0x7f766c19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.050+0000 7f7670bef640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f766c10db20 con 0x7f766c105390 2024-04-21T08:44:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7670bef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f766c10d9a0 con 0x7f766c102a60 2024-04-21T08:44:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7670bef640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f766c10dca0 con 0x7f766c107cc0 2024-04-21T08:44:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7669d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c105390 0x7f766c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7669d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c105390 0x7f766c1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59780/0 (socket says 172.21.15.7:59780) 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7669d9b640 1 -- 172.21.15.7:0/2233798733 learned_addr learned my addr 172.21.15.7:0/2233798733 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f766a59c640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c102a60 0x7f766c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f766ad9d640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f766c107cc0 0x7f766c19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7669d9b640 1 -- 172.21.15.7:0/2233798733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f766c107cc0 msgr2=0x7f766c19f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7669d9b640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f766c107cc0 0x7f766c19f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7669d9b640 1 -- 172.21.15.7:0/2233798733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c102a60 msgr2=0x7f766c1981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7669d9b640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c102a60 0x7f766c1981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7669d9b640 1 -- 172.21.15.7:0/2233798733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766c19fdc0 con 0x7f766c105390 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f766ad9d640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f766c107cc0 0x7f766c19f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7669d9b640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c105390 0x7f766c1986e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7654005fd0 tx=0x7f765400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f76637fe640 1 -- 172.21.15.7:0/2233798733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654017070 con 0x7f766c105390 2024-04-21T08:44:33.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f76637fe640 1 -- 172.21.15.7:0/2233798733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7654009c30 con 0x7f766c105390 2024-04-21T08:44:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7670bef640 1 -- 172.21.15.7:0/2233798733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f766c1a00b0 con 0x7f766c105390 2024-04-21T08:44:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f76637fe640 1 -- 172.21.15.7:0/2233798733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654003d30 con 0x7f766c105390 2024-04-21T08:44:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.051+0000 7f7670bef640 1 -- 172.21.15.7:0/2233798733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f766c1a0600 con 0x7f766c105390 2024-04-21T08:44:33.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.052+0000 7f7670bef640 1 -- 172.21.15.7:0/2233798733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f766c0fdf60 con 0x7f766c105390 2024-04-21T08:44:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.053+0000 7f76637fe640 1 -- 172.21.15.7:0/2233798733 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 138645+0+0 (secure 0 0 0) 0x7f7654003ed0 con 0x7f766c105390 2024-04-21T08:44:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.054+0000 7f76637fe640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f76580a3ba0 0x7f76580a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.054+0000 7f766a59c640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f76580a3ba0 0x7f76580a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.054+0000 7f76637fe640 1 -- 172.21.15.7:0/2233798733 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f76540d0510 con 0x7f766c105390 2024-04-21T08:44:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.055+0000 7f766a59c640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f76580a3ba0 0x7f76580a6060 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f765c005c40 tx=0x7f765c00a570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:33.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.059+0000 7f76637fe640 1 -- 172.21.15.7:0/2233798733 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7654020030 con 0x7f766c105390 2024-04-21T08:44:33.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:32 smithi077 ceph-mon[31615]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:32 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4230341103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:32 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/629285314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:32 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/935911786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:33.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:32 smithi007 ceph-mon[28477]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:33.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:32 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4230341103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:33.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:32 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/629285314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:33.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:32 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/935911786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:33.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.300+0000 7f7670bef640 1 -- 172.21.15.7:0/2233798733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f766c062b50 con 0x7f766c105390 2024-04-21T08:44:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:32 smithi016 ceph-mon[31259]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:32 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4230341103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:32 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/629285314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:32 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/935911786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:33.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.821+0000 7f76637fe640 1 -- 172.21.15.7:0/2233798733 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f7654009de0 con 0x7f766c105390 2024-04-21T08:44:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.824+0000 7f7670bef640 1 -- 172.21.15.7:0/2233798733 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f76580a3ba0 msgr2=0x7f76580a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.824+0000 7f7670bef640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f76580a3ba0 0x7f76580a6060 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f765c005c40 tx=0x7f765c00a570 comp rx=0 tx=0).stop 2024-04-21T08:44:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.824+0000 7f7670bef640 1 -- 172.21.15.7:0/2233798733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c105390 msgr2=0x7f766c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.824+0000 7f7670bef640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c105390 0x7f766c1986e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7654005fd0 tx=0x7f765400dde0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.825+0000 7f7670bef640 1 -- 172.21.15.7:0/2233798733 shutdown_connections 2024-04-21T08:44:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.825+0000 7f7670bef640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f76580a3ba0 0x7f76580a6060 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.825+0000 7f7670bef640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f766c107cc0 0x7f766c19f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.825+0000 7f7670bef640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c105390 0x7f766c1986e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.825+0000 7f7670bef640 1 --2- 172.21.15.7:0/2233798733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f766c102a60 0x7f766c1981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.825+0000 7f7670bef640 1 -- 172.21.15.7:0/2233798733 >> 172.21.15.7:0/2233798733 conn(0x7f766c0fc470 msgr2=0x7f766c1033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.825+0000 7f7670bef640 1 -- 172.21.15.7:0/2233798733 shutdown_connections 2024-04-21T08:44:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.825+0000 7f7670bef640 1 -- 172.21.15.7:0/2233798733 wait complete. 2024-04-21T08:44:33.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-21T08:44:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.960+0000 7f1cee3b2640 1 -- 172.21.15.7:0/1398976065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce8107c90 msgr2=0x7f1ce810a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.960+0000 7f1cee3b2640 1 --2- 172.21.15.7:0/1398976065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce8107c90 0x7f1ce810a0e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009ca0 tx=0x7f1ce002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.960+0000 7f1cee3b2640 1 -- 172.21.15.7:0/1398976065 shutdown_connections 2024-04-21T08:44:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.960+0000 7f1cee3b2640 1 --2- 172.21.15.7:0/1398976065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce8107c90 0x7f1ce810a0e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.960+0000 7f1cee3b2640 1 --2- 172.21.15.7:0/1398976065 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce8105360 0x7f1ce8107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.960+0000 7f1cee3b2640 1 --2- 172.21.15.7:0/1398976065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce8102a30 0x7f1ce8104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.960+0000 7f1cee3b2640 1 -- 172.21.15.7:0/1398976065 >> 172.21.15.7:0/1398976065 conn(0x7f1ce80fc420 msgr2=0x7f1ce80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.960+0000 7f1cee3b2640 1 -- 172.21.15.7:0/1398976065 shutdown_connections 2024-04-21T08:44:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.960+0000 7f1cee3b2640 1 -- 172.21.15.7:0/1398976065 wait complete. 2024-04-21T08:44:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1cee3b2640 1 Processor -- start 2024-04-21T08:44:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1cee3b2640 1 -- start start 2024-04-21T08:44:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1cee3b2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce8102a30 0x7f1ce8198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1cee3b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce8105360 0x7f1ce81986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1cee3b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce8107c90 0x7f1ce819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1cee3b2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce810daf0 con 0x7f1ce8102a30 2024-04-21T08:44:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1cee3b2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce810d970 con 0x7f1ce8105360 2024-04-21T08:44:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1cee3b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce810dc70 con 0x7f1ce8107c90 2024-04-21T08:44:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1ce7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce8102a30 0x7f1ce8198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1ce7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce8102a30 0x7f1ce8198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59806/0 (socket says 172.21.15.7:59806) 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1ce7fff640 1 -- 172.21.15.7:0/151717101 learned_addr learned my addr 172.21.15.7:0/151717101 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1cec94f640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce8107c90 0x7f1ce819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.961+0000 7f1ce77fe640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce8105360 0x7f1ce81986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1ce7fff640 1 -- 172.21.15.7:0/151717101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce8107c90 msgr2=0x7f1ce819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1ce7fff640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce8107c90 0x7f1ce819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1ce7fff640 1 -- 172.21.15.7:0/151717101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce8105360 msgr2=0x7f1ce81986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1ce7fff640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce8105360 0x7f1ce81986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1ce7fff640 1 -- 172.21.15.7:0/151717101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce819fd90 con 0x7f1ce8102a30 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1cec94f640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce8107c90 0x7f1ce819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1ce77fe640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce8105360 0x7f1ce81986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1ce7fff640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce8102a30 0x7f1ce8198170 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800e9f0 tx=0x7f1cd800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1ce57fa640 1 -- 172.21.15.7:0/151717101 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd800cd80 con 0x7f1ce8102a30 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1cee3b2640 1 -- 172.21.15.7:0/151717101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce81a0080 con 0x7f1ce8102a30 2024-04-21T08:44:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1ce57fa640 1 -- 172.21.15.7:0/151717101 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cd80108e0 con 0x7f1ce8102a30 2024-04-21T08:44:33.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1ce57fa640 1 -- 172.21.15.7:0/151717101 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8018980 con 0x7f1ce8102a30 2024-04-21T08:44:33.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.962+0000 7f1cee3b2640 1 -- 172.21.15.7:0/151717101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce81a05d0 con 0x7f1ce8102a30 2024-04-21T08:44:33.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.963+0000 7f1cee3b2640 1 -- 172.21.15.7:0/151717101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ce80fdf10 con 0x7f1ce8102a30 2024-04-21T08:44:33.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.964+0000 7f1ce57fa640 1 -- 172.21.15.7:0/151717101 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 19) ==== 138662+0+0 (secure 0 0 0) 0x7f1cd8010430 con 0x7f1ce8102a30 2024-04-21T08:44:33.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.965+0000 7f1ce57fa640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1cd00a3bf0 0x7f1cd00a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:33.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.965+0000 7f1ce77fe640 1 -- 172.21.15.7:0/151717101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1cd00a3bf0 msgr2=0x7f1cd00a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:33.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.965+0000 7f1ce57fa640 1 -- 172.21.15.7:0/151717101 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f1cd80d0b40 con 0x7f1ce8102a30 2024-04-21T08:44:33.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.965+0000 7f1ce77fe640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1cd00a3bf0 0x7f1cd00a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:44:33.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:33.970+0000 7f1ce57fa640 1 -- 172.21.15.7:0/151717101 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1cd809cac0 con 0x7f1ce8102a30 2024-04-21T08:44:34.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:33 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2233798733' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:44:34.104 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:33 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: ignoring --setuser ceph since I am not root 2024-04-21T08:44:34.104 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:33 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: ignoring --setgroup ceph since I am not root 2024-04-21T08:44:34.104 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:33 smithi077 ceph-mgr[33478]: -- 172.21.15.77:0/877390441 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ac8b005a0 con 0x562ac8ad5000 2024-04-21T08:44:34.104 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:34 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:34.005+0000 7f765b8a9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:44:34.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:33 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2233798733' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:44:34.152 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: ignoring --setuser ceph since I am not root 2024-04-21T08:44:34.152 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: ignoring --setgroup ceph since I am not root 2024-04-21T08:44:34.152 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:34.020+0000 7f6e8ced7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:44:34.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.159+0000 7f1cee3b2640 1 -- 172.21.15.7:0/151717101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f1ce802d5c0 con 0x7f1ce8102a30 2024-04-21T08:44:34.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.162+0000 7f1ce57fa640 1 -- 172.21.15.7:0/151717101 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v32)=0 v32) ==== 138+0+0 (secure 0 0 0) 0x7f1cd80a0dc0 con 0x7f1ce8102a30 2024-04-21T08:44:34.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.164+0000 7f1cee3b2640 1 -- 172.21.15.7:0/151717101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1cd00a3bf0 msgr2=0x7f1cd00a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:44:34.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.164+0000 7f1cee3b2640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1cd00a3bf0 0x7f1cd00a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.164+0000 7f1cee3b2640 1 -- 172.21.15.7:0/151717101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce8102a30 msgr2=0x7f1ce8198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:34.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.164+0000 7f1cee3b2640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce8102a30 0x7f1ce8198170 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800e9f0 tx=0x7f1cd800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.165+0000 7f1cee3b2640 1 -- 172.21.15.7:0/151717101 shutdown_connections 2024-04-21T08:44:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.165+0000 7f1cee3b2640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f1cd00a3bf0 0x7f1cd00a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.165+0000 7f1cee3b2640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce8107c90 0x7f1ce819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.165+0000 7f1cee3b2640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce8105360 0x7f1ce81986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.165+0000 7f1cee3b2640 1 --2- 172.21.15.7:0/151717101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce8102a30 0x7f1ce8198170 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.165+0000 7f1cee3b2640 1 -- 172.21.15.7:0/151717101 >> 172.21.15.7:0/151717101 conn(0x7f1ce80fc420 msgr2=0x7f1ce8103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.165+0000 7f1cee3b2640 1 -- 172.21.15.7:0/151717101 shutdown_connections 2024-04-21T08:44:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.165+0000 7f1cee3b2640 1 -- 172.21.15.7:0/151717101 wait complete. 2024-04-21T08:44:34.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-21T08:44:34.201 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:33 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2233798733' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:44:34.202 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:33 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: ignoring --setuser ceph since I am not root 2024-04-21T08:44:34.202 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:33 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: ignoring --setgroup ceph since I am not root 2024-04-21T08:44:34.202 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:33 smithi016 ceph-mgr[32647]: -- 172.21.15.16:0/2407638823 <== mon.2 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e36b7a5a0 con 0x557e36b4e800 2024-04-21T08:44:34.202 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:34 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:34.011+0000 7f13d75ab200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:44:34.202 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:34 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:34.202+0000 7f13d75ab200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:44:34.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.280+0000 7fedb224e640 1 -- 172.21.15.7:0/1193535859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedac102040 msgr2=0x7fedac1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:34.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.280+0000 7fedb224e640 1 --2- 172.21.15.7:0/1193535859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedac102040 0x7fedac1024c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7feda0009900 tx=0x7feda002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:34.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.280+0000 7fedb224e640 1 -- 172.21.15.7:0/1193535859 shutdown_connections 2024-04-21T08:44:34.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.280+0000 7fedb224e640 1 --2- 172.21.15.7:0/1193535859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedac102a00 0x7fedac109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.280+0000 7fedb224e640 1 --2- 172.21.15.7:0/1193535859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedac102040 0x7fedac1024c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.280+0000 7fedb224e640 1 --2- 172.21.15.7:0/1193535859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedac100e60 0x7fedac101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.280+0000 7fedb224e640 1 -- 172.21.15.7:0/1193535859 >> 172.21.15.7:0/1193535859 conn(0x7fedac0fc420 msgr2=0x7fedac0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.280+0000 7fedb224e640 1 -- 172.21.15.7:0/1193535859 shutdown_connections 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.280+0000 7fedb224e640 1 -- 172.21.15.7:0/1193535859 wait complete. 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.281+0000 7fedb224e640 1 Processor -- start 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.281+0000 7fedb224e640 1 -- start start 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.281+0000 7fedb224e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedac100e60 0x7fedac198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.281+0000 7fedb224e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedac102040 0x7fedac198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.281+0000 7fedb224e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedac102a00 0x7fedac19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.281+0000 7fedb224e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fedac10ca00 con 0x7fedac102a00 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.281+0000 7fedb224e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fedac10c880 con 0x7fedac102040 2024-04-21T08:44:34.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.281+0000 7fedb224e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fedac10cb80 con 0x7fedac100e60 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.281+0000 7fedabfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedac102a00 0x7fedac19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedabfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedac102a00 0x7fedac19f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59826/0 (socket says 172.21.15.7:59826) 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedabfff640 1 -- 172.21.15.7:0/3725814402 learned_addr learned my addr 172.21.15.7:0/3725814402 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedab7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedac100e60 0x7fedac198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedaaffd640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedac102040 0x7fedac198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedabfff640 1 -- 172.21.15.7:0/3725814402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedac100e60 msgr2=0x7fedac198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedabfff640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedac100e60 0x7fedac198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedabfff640 1 -- 172.21.15.7:0/3725814402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedac102040 msgr2=0x7fedac198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedabfff640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedac102040 0x7fedac198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedabfff640 1 -- 172.21.15.7:0/3725814402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedac19fe40 con 0x7fedac102a00 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedaaffd640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedac102040 0x7fedac198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedabfff640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedac102a00 0x7fedac19f7d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7feda400eb10 tx=0x7feda400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7feda8ff9640 1 -- 172.21.15.7:0/3725814402 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda4004b20 con 0x7fedac102a00 2024-04-21T08:44:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7feda8ff9640 1 -- 172.21.15.7:0/3725814402 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feda4004cc0 con 0x7fedac102a00 2024-04-21T08:44:34.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7feda8ff9640 1 -- 172.21.15.7:0/3725814402 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda4010670 con 0x7fedac102a00 2024-04-21T08:44:34.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedb224e640 1 -- 172.21.15.7:0/3725814402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedac1a0130 con 0x7fedac102a00 2024-04-21T08:44:34.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.282+0000 7fedb224e640 1 -- 172.21.15.7:0/3725814402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedac1a0650 con 0x7fedac102a00 2024-04-21T08:44:34.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.283+0000 7fedb224e640 1 -- 172.21.15.7:0/3725814402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedac1a0910 con 0x7fedac102a00 2024-04-21T08:44:34.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.285+0000 7feda8ff9640 1 -- 172.21.15.7:0/3725814402 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 19) ==== 138662+0+0 (secure 0 0 0) 0x7feda4005000 con 0x7fedac102a00 2024-04-21T08:44:34.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.285+0000 7feda8ff9640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fed940a3bf0 0x7fed940a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:34.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.285+0000 7fedab7fe640 1 -- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fed940a3bf0 msgr2=0x7fed940a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:34.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.285+0000 7feda8ff9640 1 -- 172.21.15.7:0/3725814402 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7feda40d0fa0 con 0x7fedac102a00 2024-04-21T08:44:34.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.285+0000 7fedab7fe640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fed940a3bf0 0x7fed940a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:44:34.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.293+0000 7feda8ff9640 1 -- 172.21.15.7:0/3725814402 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feda4053ff0 con 0x7fedac102a00 2024-04-21T08:44:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.481+0000 7fedb224e640 1 -- 172.21.15.7:0/3725814402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fedac1a0ae0 con 0x7fedac102a00 2024-04-21T08:44:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.485+0000 7feda8ff9640 1 -- 172.21.15.7:0/3725814402 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v33)=0 v33) ==== 133+0+0 (secure 0 0 0) 0x7feda409d570 con 0x7fedac102a00 2024-04-21T08:44:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.486+0000 7fedab7fe640 1 -- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fed940a3bf0 msgr2=0x7fed940a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.486+0000 7fedab7fe640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fed940a3bf0 0x7fed940a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:44:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 -- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fed940a3bf0 msgr2=0x7fed940a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:44:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fed940a3bf0 0x7fed940a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 -- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedac102a00 msgr2=0x7fedac19f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedac102a00 0x7fedac19f7d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7feda400eb10 tx=0x7feda400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 -- 172.21.15.7:0/3725814402 shutdown_connections 2024-04-21T08:44:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fed940a3bf0 0x7fed940a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedac102a00 0x7fedac19f7d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedac102040 0x7fedac198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 --2- 172.21.15.7:0/3725814402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedac100e60 0x7fedac198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 -- 172.21.15.7:0/3725814402 >> 172.21.15.7:0/3725814402 conn(0x7fedac0fc420 msgr2=0x7fedac0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 -- 172.21.15.7:0/3725814402 shutdown_connections 2024-04-21T08:44:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:34.488+0000 7fedb224e640 1 -- 172.21.15.7:0/3725814402 wait complete. 2024-04-21T08:44:34.499 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:34.214+0000 7f6e8ced7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:44:34.499 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:34.329+0000 7f6e8ced7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:44:34.500 INFO:teuthology.run_tasks:Running task kclient... 2024-04-21T08:44:34.756 INFO:tasks.kclient:Mounting kernel clients... 2024-04-21T08:44:34.756 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-21T08:44:34.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:34.757 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-04-21T08:44:34.758 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:34 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:34.188+0000 7f765b8a9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:44:34.758 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:34 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:34.290+0000 7f765b8a9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:44:34.758 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:34 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:34.308+0000 7f13d75ab200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:44:34.772 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-21T08:44:34.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:34.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:44:34.822 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:34.655+0000 7f6e8ced7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:44:34.822 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:34.743+0000 7f6e8ced7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:44:34.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:34.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link delete ceph-brx 2024-04-21T08:44:34.853 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:34 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:34.594+0000 7f765b8a9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:44:34.854 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:34 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:34.678+0000 7f765b8a9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:44:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:34 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2233798733' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:44:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:34 smithi077 ceph-mon[31615]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:44:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:34 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/151717101' entity='client.admin' 2024-04-21T08:44:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:34 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3725814402' entity='client.admin' 2024-04-21T08:44:34.937 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-21T08:44:34.938 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T08:44:34.938 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T08:44:34.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph fs ls 2024-04-21T08:44:35.055 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:34 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:34.628+0000 7f13d75ab200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:44:35.055 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:34 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:34.716+0000 7f13d75ab200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:44:35.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:34 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2233798733' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:44:35.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:34 smithi016 ceph-mon[31259]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:44:35.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:34 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/151717101' entity='client.admin' 2024-04-21T08:44:35.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:34 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3725814402' entity='client.admin' 2024-04-21T08:44:35.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:34 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2233798733' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:44:35.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:34 smithi007 ceph-mon[28477]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:44:35.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:34 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/151717101' entity='client.admin' 2024-04-21T08:44:35.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:34 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3725814402' entity='client.admin' 2024-04-21T08:44:35.236 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:36.103 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:35 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:35.851+0000 7f765b8a9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:44:36.104 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:36.022+0000 7f765b8a9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:44:36.304 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:35 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:35.957+0000 7f13d75ab200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:44:36.304 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:36 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:36.132+0000 7f13d75ab200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:44:36.381 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:36.032+0000 7f6e8ced7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:44:36.381 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:36.213+0000 7f6e8ced7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:44:36.480 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:36.192+0000 7f765b8a9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:44:36.605 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:36 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:36.308+0000 7f13d75ab200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:44:36.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:36.393+0000 7f6e8ced7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:44:36.773 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:36.480+0000 7f765b8a9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:44:36.773 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:36.573+0000 7f765b8a9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:44:36.910 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:36 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:36.605+0000 7f13d75ab200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:44:36.910 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:36 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:36.701+0000 7f13d75ab200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:44:36.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.960+0000 7f3e2816f700 1 -- 172.21.15.7:0/1656528269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e20101930 msgr2=0x7f3e20101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:36.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.960+0000 7f3e2816f700 1 --2- 172.21.15.7:0/1656528269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e20101930 0x7f3e20101db0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009990 tx=0x7f3e10009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:36.962 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:36.709+0000 7f6e8ced7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:44:36.962 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:36.806+0000 7f6e8ced7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:44:36.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.961+0000 7f3e2816f700 1 -- 172.21.15.7:0/1656528269 shutdown_connections 2024-04-21T08:44:36.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.961+0000 7f3e2816f700 1 --2- 172.21.15.7:0/1656528269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e20107980 0x7f3e20107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:36.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.961+0000 7f3e2816f700 1 --2- 172.21.15.7:0/1656528269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e201022f0 0x7f3e2010e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:36.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.961+0000 7f3e2816f700 1 --2- 172.21.15.7:0/1656528269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e20101930 0x7f3e20101db0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:36.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.961+0000 7f3e2816f700 1 -- 172.21.15.7:0/1656528269 >> 172.21.15.7:0/1656528269 conn(0x7f3e200788b0 msgr2=0x7f3e20078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:36.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.961+0000 7f3e2816f700 1 -- 172.21.15.7:0/1656528269 shutdown_connections 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.961+0000 7f3e2816f700 1 -- 172.21.15.7:0/1656528269 wait complete. 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.961+0000 7f3e2816f700 1 Processor -- start 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2816f700 1 -- start start 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2816f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e20101930 0x7f3e2019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2816f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e201022f0 0x7f3e2019b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2816f700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e20107980 0x7f3e2019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2816f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3e20111ad0 con 0x7f3e20101930 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2816f700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3e20111950 con 0x7f3e20107980 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2816f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3e20111c50 con 0x7f3e201022f0 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2570a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e201022f0 0x7f3e2019b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e25f0b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e20101930 0x7f3e2019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2570a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e201022f0 0x7f3e2019b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:42336/0 (socket says 172.21.15.7:42336) 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e25f0b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e20101930 0x7f3e2019b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53370/0 (socket says 172.21.15.7:53370) 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2570a700 1 -- 172.21.15.7:0/238944339 learned_addr learned my addr 172.21.15.7:0/238944339 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:36.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2670c700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e20107980 0x7f3e2019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:36.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2570a700 1 -- 172.21.15.7:0/238944339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e20107980 msgr2=0x7f3e2019fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:36.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2570a700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e20107980 0x7f3e2019fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:36.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2570a700 1 -- 172.21.15.7:0/238944339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e20101930 msgr2=0x7f3e2019b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:36.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2570a700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e20101930 0x7f3e2019b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:36.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.962+0000 7f3e2570a700 1 -- 172.21.15.7:0/238944339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e201a0370 con 0x7f3e201022f0 2024-04-21T08:44:36.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.963+0000 7f3e2570a700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e201022f0 0x7f3e2019b8a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3e10000c00 tx=0x7f3e10009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:36.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.963+0000 7f3e16ffd700 1 -- 172.21.15.7:0/238944339 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10027070 con 0x7f3e201022f0 2024-04-21T08:44:36.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.963+0000 7f3e16ffd700 1 -- 172.21.15.7:0/238944339 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e10022470 con 0x7f3e201022f0 2024-04-21T08:44:36.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.963+0000 7f3e2816f700 1 -- 172.21.15.7:0/238944339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e201a0600 con 0x7f3e201022f0 2024-04-21T08:44:36.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.963+0000 7f3e16ffd700 1 -- 172.21.15.7:0/238944339 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10017650 con 0x7f3e201022f0 2024-04-21T08:44:36.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.963+0000 7f3e2816f700 1 -- 172.21.15.7:0/238944339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e201a7ee0 con 0x7f3e201022f0 2024-04-21T08:44:36.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.964+0000 7f3e2816f700 1 -- 172.21.15.7:0/238944339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e20103050 con 0x7f3e201022f0 2024-04-21T08:44:36.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.965+0000 7f3e16ffd700 1 -- 172.21.15.7:0/238944339 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 138662+0+0 (secure 0 0 0) 0x7f3e1002a070 con 0x7f3e201022f0 2024-04-21T08:44:36.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.966+0000 7f3e16ffd700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3e0c0a3c90 0x7f3e0c0a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:36.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.966+0000 7f3e25f0b700 1 -- 172.21.15.7:0/238944339 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3e0c0a3c90 msgr2=0x7f3e0c0a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:36.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.966+0000 7f3e16ffd700 1 -- 172.21.15.7:0/238944339 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f3e100d15a0 con 0x7f3e201022f0 2024-04-21T08:44:36.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.966+0000 7f3e25f0b700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3e0c0a3c90 0x7f3e0c0a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:44:36.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:36.970+0000 7f3e16ffd700 1 -- 172.21.15.7:0/238944339 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e10013070 con 0x7f3e201022f0 2024-04-21T08:44:37.103 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:36.773+0000 7f765b8a9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:44:37.104 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:36.968+0000 7f765b8a9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:44:37.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.167+0000 7f3e25f0b700 1 -- 172.21.15.7:0/238944339 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3e0c0a3c90 msgr2=0x7f3e0c0a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:37.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.167+0000 7f3e25f0b700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3e0c0a3c90 0x7f3e0c0a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:44:37.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.227+0000 7f3e2816f700 1 -- 172.21.15.7:0/238944339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3e2019c6c0 con 0x7f3e201022f0 2024-04-21T08:44:37.228 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:37 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:37.024+0000 7f6e8ced7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:44:37.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.228+0000 7f3e16ffd700 1 -- 172.21.15.7:0/238944339 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+88 (secure 0 0 0) 0x7f3e1009dba0 con 0x7f3e201022f0 2024-04-21T08:44:37.229 INFO:teuthology.orchestra.run.smithi007.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T08:44:37.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 -- 172.21.15.7:0/238944339 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3e0c0a3c90 msgr2=0x7f3e0c0a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:44:37.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3e0c0a3c90 0x7f3e0c0a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:37.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 -- 172.21.15.7:0/238944339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e201022f0 msgr2=0x7f3e2019b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:37.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e201022f0 0x7f3e2019b8a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3e10000c00 tx=0x7f3e10009580 comp rx=0 tx=0).stop 2024-04-21T08:44:37.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 -- 172.21.15.7:0/238944339 shutdown_connections 2024-04-21T08:44:37.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f3e0c0a3c90 0x7f3e0c0a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:37.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e20107980 0x7f3e2019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:37.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e20101930 0x7f3e2019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:37.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 --2- 172.21.15.7:0/238944339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e201022f0 0x7f3e2019b8a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:37.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 -- 172.21.15.7:0/238944339 >> 172.21.15.7:0/238944339 conn(0x7f3e200788b0 msgr2=0x7f3e200ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:37.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 -- 172.21.15.7:0/238944339 shutdown_connections 2024-04-21T08:44:37.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:37.230+0000 7f3e2816f700 1 -- 172.21.15.7:0/238944339 wait complete. 2024-04-21T08:44:37.280 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:36 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:36.910+0000 7f13d75ab200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:44:37.280 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:37 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:37.109+0000 7f13d75ab200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:44:37.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:37 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/238944339' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:44:37.560 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:37 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/238944339' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:44:37.614 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:37 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/238944339' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:44:37.614 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:37 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:37.230+0000 7f6e8ced7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:44:37.853 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:37 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:37.560+0000 7f765b8a9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:44:37.918 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-21T08:44:37.918 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T08:44:37.918 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi007.front.sepia.ceph.com 2024-04-21T08:44:37.918 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-21T08:44:37.918 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T08:44:37.918 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-21T08:44:37.918 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T08:44:37.918 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T08:44:37.918 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-21T08:44:37.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:37.919 DEBUG:teuthology.orchestra.run.smithi007:> ip addr 2024-04-21T08:44:37.935 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T08:44:37.935 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T08:44:37.935 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T08:44:37.935 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:44:37.935 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2024-04-21T08:44:37.935 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:44:37.935 INFO:teuthology.orchestra.run.smithi007.stdout:2: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T08:44:37.935 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2024-04-21T08:44:37.935 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f0 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 42046sec preferred_lft 42046sec 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link noprefixroute 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout:3: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f0 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f1 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2024-04-21T08:44:37.936 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f1 2024-04-21T08:44:37.937 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T08:44:37.937 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:44:37.937 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-04-21T08:44:37.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add name ceph-brx type bridge 2024-04-21T08:44:37.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr flush dev ceph-brx 2024-04-21T08:44:37.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set ceph-brx up 2024-04-21T08:44:37.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T08:44:37.938 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-04-21T08:44:38.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:44:38.054 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:37 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:37.732+0000 7f13d75ab200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:44:38.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:44:38.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:38.079 DEBUG:teuthology.orchestra.run.smithi007:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T08:44:38.103 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-04-21T08:44:38.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:38.105 DEBUG:teuthology.orchestra.run.smithi007:> route 2024-04-21T08:44:38.151 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:37 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:37.867+0000 7f6e8ced7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:44:38.162 INFO:teuthology.orchestra.run.smithi007.stdout:Kernel IP routing table 2024-04-21T08:44:38.162 INFO:teuthology.orchestra.run.smithi007.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T08:44:38.162 INFO:teuthology.orchestra.run.smithi007.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-21T08:44:38.162 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-21T08:44:38.163 INFO:teuthology.orchestra.run.smithi007.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T08:44:38.163 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:44:38.163 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-04-21T08:44:38.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-21T08:44:38.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-21T08:44:38.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-21T08:44:38.164 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-04-21T08:44:38.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:44:38.281 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:37 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:37.982+0000 7f765b8a9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:44:38.281 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:38 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:38.079+0000 7f765b8a9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:44:38.281 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:38 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:38.281+0000 7f765b8a9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:44:38.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:44:38.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:38.291 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-04-21T08:44:38.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:38.308 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list-id 2024-04-21T08:44:38.363 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:44:38.364 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-04-21T08:44:38.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T08:44:38.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-21T08:44:38.364 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-04-21T08:44:38.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:44:38.473 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:38.168+0000 7f13d75ab200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:44:38.474 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:38.266+0000 7f13d75ab200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:44:38.474 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:38.473+0000 7f13d75ab200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:44:38.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:44:38.483 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-21T08:44:38.483 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:44:38.483 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-04-21T08:44:38.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-21T08:44:38.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T08:44:38.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-21T08:44:38.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-21T08:44:38.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-21T08:44:38.484 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-04-21T08:44:38.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:44:38.542 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:38.332+0000 7f6e8ced7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:44:38.542 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:38.434+0000 7f6e8ced7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:44:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:44:38.600 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:44:38.601 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-04-21T08:44:38.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set brx.0 up 2024-04-21T08:44:38.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T08:44:38.601 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-04-21T08:44:38.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:44:38.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:44:38.681 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:44:38.681 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-21T08:44:38.696 INFO:teuthology.orchestra.run.smithi007.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-21T08:44:38.697 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:44:38.697 DEBUG:teuthology.orchestra.run.smithi007:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-21T08:44:38.752 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-21T08:44:38.752 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-21T08:44:38.752 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/38a4aff8-de86-46c2-80e8-f71cbab9c8ff 2024-04-21T08:44:38.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:38.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/38a4aff8-de86-46c2-80e8-f71cbab9c8ff /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-04-21T08:44:38.813 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:38.661+0000 7f6e8ced7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:44:38.836 INFO:teuthology.orchestra.run.smithi007.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-04-21T08:44:38.836 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-04-21T08:44:38.955 INFO:teuthology.orchestra.run.smithi007.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-04-21T08:44:38.955 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-04-21T08:44:38.955 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: resolved to: "172.21.15.7:3300,172.21.15.16:3300,172.21.15.77:3300" 2024-04-21T08:44:38.955 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: trying mount with new device syntax: 0@c9c60282-ffb9-11ee-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/38a4aff8-de86-46c2-80e8-f71cbab9c8ff 2024-04-21T08:44:38.955 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.7:3300/172.21.15.16:3300/172.21.15.77:3300" will pass to kernel 2024-04-21T08:44:38.956 INFO:teuthology.orchestra.run.smithi007.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-21T08:44:38.956 INFO:teuthology.orchestra.run.smithi007.stdout: You just mounted a file system that supports labels which does not 2024-04-21T08:44:38.956 INFO:teuthology.orchestra.run.smithi007.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-21T08:44:38.956 INFO:teuthology.orchestra.run.smithi007.stdout: applications will generate AVC messages and not be allowed access to 2024-04-21T08:44:38.956 INFO:teuthology.orchestra.run.smithi007.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-21T08:44:38.958 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-21T08:44:38.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:38.958 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-21T08:44:39.024 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:44:39.103 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:38 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:38.800+0000 7f765b8a9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:44:39.104 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:38 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:38.892+0000 7f765b8a9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:44:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:44:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:44:39.123 INFO:teuthology.orchestra.run.smithi007.stdout:client25108 2024-04-21T08:44:39.123 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:44:39.124 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:44:39.139 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:44:39.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:44:39.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:44:39.238 INFO:teuthology.orchestra.run.smithi007.stdout:client25108 2024-04-21T08:44:39.239 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:44:39.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:44:39.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:44:39.296 INFO:teuthology.orchestra.run.smithi007.stdout:client25108 2024-04-21T08:44:39.297 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968.client25108/status') 2024-04-21T08:44:39.304 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:39.018+0000 7f13d75ab200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:44:39.304 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:39.113+0000 7f13d75ab200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:44:39.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:44:39.357 INFO:teuthology.orchestra.run.smithi007.stdout:instance: client.25108 (3)172.21.15.7:0/1486911481 2024-04-21T08:44:39.357 INFO:teuthology.orchestra.run.smithi007.stdout:blocklisted: false 2024-04-21T08:44:39.357 INFO:teuthology.orchestra.run.smithi007.stderr:0+1 records in 2024-04-21T08:44:39.357 INFO:teuthology.orchestra.run.smithi007.stderr:0+1 records out 2024-04-21T08:44:39.358 INFO:teuthology.orchestra.run.smithi007.stderr:70 bytes copied, 2.4148e-05 s, 2.9 MB/s 2024-04-21T08:44:39.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:44:39.366 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-21T08:44:39.377 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-21T08:44:39.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:39.472 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:39 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:39.135+0000 7f765b8a9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:44:39.472 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:39 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:39.265+0000 7f765b8a9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:44:39.472 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:39 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:39.358+0000 7f765b8a9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:44:39.472 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:39 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:39.472+0000 7f765b8a9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:44:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.526+0000 7f5a14506640 1 -- 172.21.15.7:0/1825592458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c1007f0 msgr2=0x7f5a0c100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.526+0000 7f5a14506640 1 --2- 172.21.15.7:0/1825592458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c1007f0 0x7f5a0c100c10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f59fc009900 tx=0x7f59fc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.526+0000 7f5a14506640 1 -- 172.21.15.7:0/1825592458 shutdown_connections 2024-04-21T08:44:39.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.526+0000 7f5a14506640 1 --2- 172.21.15.7:0/1825592458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0c101af0 0x7f5a0c101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.526+0000 7f5a14506640 1 --2- 172.21.15.7:0/1825592458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0c101150 0x7f5a0c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.526+0000 7f5a14506640 1 --2- 172.21.15.7:0/1825592458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c1007f0 0x7f5a0c100c10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.526+0000 7f5a14506640 1 -- 172.21.15.7:0/1825592458 >> 172.21.15.7:0/1825592458 conn(0x7f5a0c0fc3f0 msgr2=0x7f5a0c0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.526+0000 7f5a14506640 1 -- 172.21.15.7:0/1825592458 shutdown_connections 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.526+0000 7f5a14506640 1 -- 172.21.15.7:0/1825592458 wait complete. 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a14506640 1 Processor -- start 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a14506640 1 -- start start 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a14506640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0c1007f0 0x7f5a0c19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a14506640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0c101150 0x7f5a0c19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a14506640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c101af0 0x7f5a0c1a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a14506640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c10fcd0 con 0x7f5a0c101af0 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a14506640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c10fb50 con 0x7f5a0c1007f0 2024-04-21T08:44:39.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a14506640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c10fe50 con 0x7f5a0c101150 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a12aa3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c101af0 0x7f5a0c1a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a122a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0c1007f0 0x7f5a0c19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a12aa3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c101af0 0x7f5a0c1a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53398/0 (socket says 172.21.15.7:53398) 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a11aa1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0c101150 0x7f5a0c19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a122a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0c1007f0 0x7f5a0c19a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:36682/0 (socket says 172.21.15.7:36682) 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a12aa3640 1 -- 172.21.15.7:0/4107496325 learned_addr learned my addr 172.21.15.7:0/4107496325 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a12aa3640 1 -- 172.21.15.7:0/4107496325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0c101150 msgr2=0x7f5a0c19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a12aa3640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0c101150 0x7f5a0c19a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a12aa3640 1 -- 172.21.15.7:0/4107496325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0c1007f0 msgr2=0x7f5a0c19a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a12aa3640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0c1007f0 0x7f5a0c19a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a12aa3640 1 -- 172.21.15.7:0/4107496325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0c1a1fa0 con 0x7f5a0c101af0 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a122a2640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0c1007f0 0x7f5a0c19a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:39.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.527+0000 7f5a12aa3640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c101af0 0x7f5a0c1a1930 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400eb10 tx=0x7f5a0400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.528+0000 7f5a037fe640 1 -- 172.21.15.7:0/4107496325 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04004b20 con 0x7f5a0c101af0 2024-04-21T08:44:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.528+0000 7f5a037fe640 1 -- 172.21.15.7:0/4107496325 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a04004cc0 con 0x7f5a0c101af0 2024-04-21T08:44:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.528+0000 7f5a14506640 1 -- 172.21.15.7:0/4107496325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0c1a2290 con 0x7f5a0c101af0 2024-04-21T08:44:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.528+0000 7f5a037fe640 1 -- 172.21.15.7:0/4107496325 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04010670 con 0x7f5a0c101af0 2024-04-21T08:44:39.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.528+0000 7f5a14506640 1 -- 172.21.15.7:0/4107496325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a0c1a27e0 con 0x7f5a0c101af0 2024-04-21T08:44:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.529+0000 7f5a14506640 1 -- 172.21.15.7:0/4107496325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a0c10b540 con 0x7f5a0c101af0 2024-04-21T08:44:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.530+0000 7f5a037fe640 1 -- 172.21.15.7:0/4107496325 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 19) ==== 138662+0+0 (secure 0 0 0) 0x7f5a04005020 con 0x7f5a0c101af0 2024-04-21T08:44:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.530+0000 7f5a037fe640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f59f80a3c00 0x7f59f80a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.530+0000 7f5a122a2640 1 -- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f59f80a3c00 msgr2=0x7f59f80a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.530+0000 7f5a037fe640 1 -- 172.21.15.7:0/4107496325 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f5a040d0d90 con 0x7f5a0c101af0 2024-04-21T08:44:39.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.530+0000 7f5a122a2640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f59f80a3c00 0x7f59f80a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:44:39.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.535+0000 7f5a037fe640 1 -- 172.21.15.7:0/4107496325 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a0409d360 con 0x7f5a0c101af0 2024-04-21T08:44:39.596 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:39.228+0000 7f6e8ced7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:44:39.596 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:39.330+0000 7f6e8ced7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:44:39.720 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:39.365+0000 7f13d75ab200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:44:39.720 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:39.502+0000 7f13d75ab200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:44:39.720 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:39.600+0000 7f13d75ab200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:44:39.720 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:39.720+0000 7f13d75ab200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:44:39.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.726+0000 7f5a14506640 1 -- 172.21.15.7:0/4107496325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a0c062b50 con 0x7f5a0c101af0 2024-04-21T08:44:39.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.728+0000 7f5a037fe640 1 -- 172.21.15.7:0/4107496325 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24714 (secure 0 0 0) 0x7f5a040bbe20 con 0x7f5a0c101af0 2024-04-21T08:44:39.728 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:39.729 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":94,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:44:11.760990+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a122a2640 1 -- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f59f80a3c00 msgr2=0x7f59f80a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a122a2640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f59f80a3c00 0x7f59f80a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:44:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 -- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f59f80a3c00 msgr2=0x7f59f80a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:44:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f59f80a3c00 0x7f59f80a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 -- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c101af0 msgr2=0x7f5a0c1a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c101af0 0x7f5a0c1a1930 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400eb10 tx=0x7f5a0400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 -- 172.21.15.7:0/4107496325 shutdown_connections 2024-04-21T08:44:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f59f80a3c00 0x7f59f80a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c101af0 0x7f5a0c1a1930 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0c101150 0x7f5a0c19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 --2- 172.21.15.7:0/4107496325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a0c1007f0 0x7f5a0c19a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 -- 172.21.15.7:0/4107496325 >> 172.21.15.7:0/4107496325 conn(0x7f5a0c0fc3f0 msgr2=0x7f5a0c0ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 -- 172.21.15.7:0/4107496325 shutdown_connections 2024-04-21T08:44:39.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.731+0000 7f5a14506640 1 -- 172.21.15.7:0/4107496325 wait complete. 2024-04-21T08:44:39.747 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:39.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:39.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:39 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4107496325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:39.867 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:39.596+0000 7f6e8ced7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:44:39.868 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:39.746+0000 7f6e8ced7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:44:39.868 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:39.867+0000 7f6e8ced7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:44:39.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.872+0000 7f2da7f7e640 1 -- 172.21.15.7:0/1832532472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2da01029a0 msgr2=0x7f2da01092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:39.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.872+0000 7f2da7f7e640 1 --2- 172.21.15.7:0/1832532472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2da01029a0 0x7f2da01092c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009ca0 tx=0x7f2d9002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.872+0000 7f2da7f7e640 1 -- 172.21.15.7:0/1832532472 shutdown_connections 2024-04-21T08:44:39.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.872+0000 7f2da7f7e640 1 --2- 172.21.15.7:0/1832532472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2da01029a0 0x7f2da01092c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.872+0000 7f2da7f7e640 1 --2- 172.21.15.7:0/1832532472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da0101fe0 0x7f2da0102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.872+0000 7f2da7f7e640 1 --2- 172.21.15.7:0/1832532472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2da0100e90 0x7f2da01012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.872+0000 7f2da7f7e640 1 -- 172.21.15.7:0/1832532472 >> 172.21.15.7:0/1832532472 conn(0x7f2da00fc420 msgr2=0x7f2da00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.872+0000 7f2da7f7e640 1 -- 172.21.15.7:0/1832532472 shutdown_connections 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.872+0000 7f2da7f7e640 1 -- 172.21.15.7:0/1832532472 wait complete. 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da7f7e640 1 Processor -- start 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da7f7e640 1 -- start start 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da7f7e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2da0100e90 0x7f2da0069de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da7f7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2da0101fe0 0x7f2da019ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da7f7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da01029a0 0x7f2da019f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da7f7e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2da010c9f0 con 0x7f2da0100e90 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da7f7e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2da010c870 con 0x7f2da01029a0 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da7f7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2da010cb70 con 0x7f2da0101fe0 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da5d1a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2da0100e90 0x7f2da0069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:39.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da5d1a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2da0100e90 0x7f2da0069de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53408/0 (socket says 172.21.15.7:53408) 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da5d1a640 1 -- 172.21.15.7:0/917426624 learned_addr learned my addr 172.21.15.7:0/917426624 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da5519640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2da0101fe0 0x7f2da019ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da651b640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da01029a0 0x7f2da019f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da5d1a640 1 -- 172.21.15.7:0/917426624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2da0101fe0 msgr2=0x7f2da019ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da5d1a640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2da0101fe0 0x7f2da019ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da5d1a640 1 -- 172.21.15.7:0/917426624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da01029a0 msgr2=0x7f2da019f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da5d1a640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da01029a0 0x7f2da019f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da5d1a640 1 -- 172.21.15.7:0/917426624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da019fdd0 con 0x7f2da0100e90 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.873+0000 7f2da651b640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da01029a0 0x7f2da019f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.874+0000 7f2da5d1a640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2da0100e90 0x7f2da0069de0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800e9f0 tx=0x7f2d9800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.874+0000 7f2d9effd640 1 -- 172.21.15.7:0/917426624 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9800cd80 con 0x7f2da0100e90 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.874+0000 7f2d9effd640 1 -- 172.21.15.7:0/917426624 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d980054d0 con 0x7f2da0100e90 2024-04-21T08:44:39.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.874+0000 7f2da7f7e640 1 -- 172.21.15.7:0/917426624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da01a00c0 con 0x7f2da0100e90 2024-04-21T08:44:39.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.874+0000 7f2d9effd640 1 -- 172.21.15.7:0/917426624 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98010690 con 0x7f2da0100e90 2024-04-21T08:44:39.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.874+0000 7f2da7f7e640 1 -- 172.21.15.7:0/917426624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da01a0610 con 0x7f2da0100e90 2024-04-21T08:44:39.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.875+0000 7f2da7f7e640 1 -- 172.21.15.7:0/917426624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da0108210 con 0x7f2da0100e90 2024-04-21T08:44:39.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.876+0000 7f2d9effd640 1 -- 172.21.15.7:0/917426624 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 19) ==== 138662+0+0 (secure 0 0 0) 0x7f2d98004e90 con 0x7f2da0100e90 2024-04-21T08:44:39.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.877+0000 7f2d9effd640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2d880a3c80 0x7f2d880a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:39.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.877+0000 7f2da5519640 1 -- 172.21.15.7:0/917426624 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2d880a3c80 msgr2=0x7f2d880a6140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.877+0000 7f2d9effd640 1 -- 172.21.15.7:0/917426624 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f2d980d0cb0 con 0x7f2da0100e90 2024-04-21T08:44:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.877+0000 7f2da5519640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2d880a3c80 0x7f2d880a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:44:39.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:39.881+0000 7f2d9effd640 1 -- 172.21.15.7:0/917426624 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d9809d280 con 0x7f2da0100e90 2024-04-21T08:44:40.033 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:39 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4107496325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:40.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.070+0000 7f2da7f7e640 1 -- 172.21.15.7:0/917426624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2da0062b50 con 0x7f2da0100e90 2024-04-21T08:44:40.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.071+0000 7f2d9effd640 1 -- 172.21.15.7:0/917426624 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f2d9809cb80 con 0x7f2da0100e90 2024-04-21T08:44:40.072 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:44:40.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 -- 172.21.15.7:0/917426624 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2d880a3c80 msgr2=0x7f2d880a6140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:44:40.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2d880a3c80 0x7f2d880a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 -- 172.21.15.7:0/917426624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2da0100e90 msgr2=0x7f2da0069de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2da0100e90 0x7f2da0069de0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800e9f0 tx=0x7f2d9800edf0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 -- 172.21.15.7:0/917426624 shutdown_connections 2024-04-21T08:44:40.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f2d880a3c80 0x7f2d880a6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2da01029a0 0x7f2da019f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2da0101fe0 0x7f2da019ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 --2- 172.21.15.7:0/917426624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2da0100e90 0x7f2da0069de0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 -- 172.21.15.7:0/917426624 >> 172.21.15.7:0/917426624 conn(0x7f2da00fc420 msgr2=0x7f2da01053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:40.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 -- 172.21.15.7:0/917426624 shutdown_connections 2024-04-21T08:44:40.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.074+0000 7f2da7f7e640 1 -- 172.21.15.7:0/917426624 wait complete. 2024-04-21T08:44:40.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:40.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:39 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4107496325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:40.104 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:39 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:39.770+0000 7f765b8a9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:44:40.151 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:39.994+0000 7f6e8ced7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:44:40.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.198+0000 7f62fb352640 1 -- 172.21.15.7:0/240028130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62f4102a20 msgr2=0x7f62f4109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.198+0000 7f62fb352640 1 --2- 172.21.15.7:0/240028130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62f4102a20 0x7f62f4109340 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009ca0 tx=0x7f62ec02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.198+0000 7f62fb352640 1 -- 172.21.15.7:0/240028130 shutdown_connections 2024-04-21T08:44:40.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.198+0000 7f62fb352640 1 --2- 172.21.15.7:0/240028130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62f4102a20 0x7f62f4109340 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.198+0000 7f62fb352640 1 --2- 172.21.15.7:0/240028130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f4102060 0x7f62f41024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.198+0000 7f62fb352640 1 --2- 172.21.15.7:0/240028130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100e80 0x7f62f41012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.198+0000 7f62fb352640 1 -- 172.21.15.7:0/240028130 >> 172.21.15.7:0/240028130 conn(0x7f62f40fc440 msgr2=0x7f62f40fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:40.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.198+0000 7f62fb352640 1 -- 172.21.15.7:0/240028130 shutdown_connections 2024-04-21T08:44:40.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.198+0000 7f62fb352640 1 -- 172.21.15.7:0/240028130 wait complete. 2024-04-21T08:44:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62fb352640 1 Processor -- start 2024-04-21T08:44:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62fb352640 1 -- start start 2024-04-21T08:44:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62fb352640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100e80 0x7f62f41970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62fb352640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f4102060 0x7f62f4197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62fb352640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62f4102a20 0x7f62f419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62fb352640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f62f410ca20 con 0x7f62f4102a20 2024-04-21T08:44:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62fb352640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62f410c8a0 con 0x7f62f4102060 2024-04-21T08:44:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62fb352640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f62f410cba0 con 0x7f62f4100e80 2024-04-21T08:44:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62f98ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62f4102a20 0x7f62f419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62f98ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62f4102a20 0x7f62f419e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53426/0 (socket says 172.21.15.7:53426) 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62f98ef640 1 -- 172.21.15.7:0/1608284862 learned_addr learned my addr 172.21.15.7:0/1608284862 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62f88ed640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f4102060 0x7f62f4197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62f90ee640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100e80 0x7f62f41970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62f98ef640 1 -- 172.21.15.7:0/1608284862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100e80 msgr2=0x7f62f41970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62f98ef640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100e80 0x7f62f41970c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62f98ef640 1 -- 172.21.15.7:0/1608284862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f4102060 msgr2=0x7f62f4197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.199+0000 7f62f98ef640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f4102060 0x7f62f4197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.200+0000 7f62f98ef640 1 -- 172.21.15.7:0/1608284862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f419ece0 con 0x7f62f4102a20 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.200+0000 7f62f98ef640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62f4102a20 0x7f62f419e670 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f62ec02c700 tx=0x7f62ec0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.200+0000 7f62ea7fc640 1 -- 172.21.15.7:0/1608284862 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec041020 con 0x7f62f4102a20 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.200+0000 7f62fb352640 1 -- 172.21.15.7:0/1608284862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62f419ef70 con 0x7f62f4102a20 2024-04-21T08:44:40.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.200+0000 7f62ea7fc640 1 -- 172.21.15.7:0/1608284862 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62ec034ca0 con 0x7f62f4102a20 2024-04-21T08:44:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.200+0000 7f62ea7fc640 1 -- 172.21.15.7:0/1608284862 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec039040 con 0x7f62f4102a20 2024-04-21T08:44:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.200+0000 7f62fb352640 1 -- 172.21.15.7:0/1608284862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f419f430 con 0x7f62f4102a20 2024-04-21T08:44:40.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.201+0000 7f62fb352640 1 -- 172.21.15.7:0/1608284862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f4108240 con 0x7f62f4102a20 2024-04-21T08:44:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.202+0000 7f62ea7fc640 1 -- 172.21.15.7:0/1608284862 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 19) ==== 138662+0+0 (secure 0 0 0) 0x7f62ec044020 con 0x7f62f4102a20 2024-04-21T08:44:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.203+0000 7f62ea7fc640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62e00a3c00 0x7f62e00a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.203+0000 7f62f90ee640 1 -- 172.21.15.7:0/1608284862 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62e00a3c00 msgr2=0x7f62e00a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.203+0000 7f62ea7fc640 1 -- 172.21.15.7:0/1608284862 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f62ec0f2960 con 0x7f62f4102a20 2024-04-21T08:44:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.203+0000 7f62f90ee640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62e00a3c00 0x7f62e00a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:44:40.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.210+0000 7f62ea7fc640 1 -- 172.21.15.7:0/1608284862 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f62ec04b020 con 0x7f62f4102a20 2024-04-21T08:44:40.304 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:40.033+0000 7f13d75ab200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:44:40.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.396+0000 7f62fb352640 1 -- 172.21.15.7:0/1608284862 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f62f402d5c0 con 0x7f62f4102a20 2024-04-21T08:44:40.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.397+0000 7f62ea7fc640 1 -- 172.21.15.7:0/1608284862 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f62ec0bef30 con 0x7f62f4102a20 2024-04-21T08:44:40.397 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 -- 172.21.15.7:0/1608284862 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62e00a3c00 msgr2=0x7f62e00a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62e00a3c00 0x7f62e00a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 -- 172.21.15.7:0/1608284862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62f4102a20 msgr2=0x7f62f419e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62f4102a20 0x7f62f419e670 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f62ec02c700 tx=0x7f62ec0346d0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 -- 172.21.15.7:0/1608284862 shutdown_connections 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f62e00a3c00 0x7f62e00a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62f4102a20 0x7f62f419e670 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62f4102060 0x7f62f4197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 --2- 172.21.15.7:0/1608284862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62f4100e80 0x7f62f41970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 -- 172.21.15.7:0/1608284862 >> 172.21.15.7:0/1608284862 conn(0x7f62f40fc440 msgr2=0x7f62f4105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 -- 172.21.15.7:0/1608284862 shutdown_connections 2024-04-21T08:44:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.399+0000 7f62fb352640 1 -- 172.21.15.7:0/1608284862 wait complete. 2024-04-21T08:44:40.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:40.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.516+0000 7f28edc14640 1 -- 172.21.15.7:0/2385903364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28e8101fe0 msgr2=0x7f28e8102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.516+0000 7f28edc14640 1 --2- 172.21.15.7:0/2385903364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28e8101fe0 0x7f28e8102460 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009900 tx=0x7f28d002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.516+0000 7f28edc14640 1 -- 172.21.15.7:0/2385903364 shutdown_connections 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.516+0000 7f28edc14640 1 --2- 172.21.15.7:0/2385903364 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28e81029a0 0x7f28e81092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.516+0000 7f28edc14640 1 --2- 172.21.15.7:0/2385903364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28e8101fe0 0x7f28e8102460 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.516+0000 7f28edc14640 1 --2- 172.21.15.7:0/2385903364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e8100e90 0x7f28e81012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.516+0000 7f28edc14640 1 -- 172.21.15.7:0/2385903364 >> 172.21.15.7:0/2385903364 conn(0x7f28e80fc420 msgr2=0x7f28e80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.516+0000 7f28edc14640 1 -- 172.21.15.7:0/2385903364 shutdown_connections 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.516+0000 7f28edc14640 1 -- 172.21.15.7:0/2385903364 wait complete. 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28edc14640 1 Processor -- start 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28edc14640 1 -- start start 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28edc14640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28e8100e90 0x7f28e8198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28edc14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28e8101fe0 0x7f28e8198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28edc14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e81029a0 0x7f28e819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28edc14640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f28e810db30 con 0x7f28e8100e90 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28edc14640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28e810d9b0 con 0x7f28e81029a0 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28edc14640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f28e810dcb0 con 0x7f28e8101fe0 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28e77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28e8100e90 0x7f28e8198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28e77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28e8100e90 0x7f28e8198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53442/0 (socket says 172.21.15.7:53442) 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28e77fe640 1 -- 172.21.15.7:0/3275729507 learned_addr learned my addr 172.21.15.7:0/3275729507 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28e7fff640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e81029a0 0x7f28e819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.517+0000 7f28e6ffd640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28e8101fe0 0x7f28e8198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.518+0000 7f28e77fe640 1 -- 172.21.15.7:0/3275729507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28e8101fe0 msgr2=0x7f28e8198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.518+0000 7f28e77fe640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28e8101fe0 0x7f28e8198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.518+0000 7f28e77fe640 1 -- 172.21.15.7:0/3275729507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e81029a0 msgr2=0x7f28e819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.518+0000 7f28e77fe640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e81029a0 0x7f28e819f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.518+0000 7f28e77fe640 1 -- 172.21.15.7:0/3275729507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e819fe20 con 0x7f28e8100e90 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.518+0000 7f28e77fe640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28e8100e90 0x7f28e8198250 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f28d800e9f0 tx=0x7f28d800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.518+0000 7f28e4ff9640 1 -- 172.21.15.7:0/3275729507 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d800cd80 con 0x7f28e8100e90 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.518+0000 7f28edc14640 1 -- 172.21.15.7:0/3275729507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e81a0110 con 0x7f28e8100e90 2024-04-21T08:44:40.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.518+0000 7f28e4ff9640 1 -- 172.21.15.7:0/3275729507 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28d80108e0 con 0x7f28e8100e90 2024-04-21T08:44:40.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.518+0000 7f28edc14640 1 -- 172.21.15.7:0/3275729507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e81a0680 con 0x7f28e8100e90 2024-04-21T08:44:40.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.518+0000 7f28e4ff9640 1 -- 172.21.15.7:0/3275729507 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8018980 con 0x7f28e8100e90 2024-04-21T08:44:40.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.519+0000 7f28edc14640 1 -- 172.21.15.7:0/3275729507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e8068e00 con 0x7f28e8100e90 2024-04-21T08:44:40.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.520+0000 7f28e4ff9640 1 -- 172.21.15.7:0/3275729507 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 19) ==== 138662+0+0 (secure 0 0 0) 0x7f28d8010430 con 0x7f28e8100e90 2024-04-21T08:44:40.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.521+0000 7f28e4ff9640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28d40a3c00 0x7f28d40a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.521+0000 7f28e6ffd640 1 -- 172.21.15.7:0/3275729507 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28d40a3c00 msgr2=0x7f28d40a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:40.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.521+0000 7f28e4ff9640 1 -- 172.21.15.7:0/3275729507 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f28d80d0b40 con 0x7f28e8100e90 2024-04-21T08:44:40.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.521+0000 7f28e6ffd640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28d40a3c00 0x7f28d40a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:44:40.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.526+0000 7f28e4ff9640 1 -- 172.21.15.7:0/3275729507 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28d809cac0 con 0x7f28e8100e90 2024-04-21T08:44:40.651 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:40 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:40.322+0000 7f6e8ced7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:44:40.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.715+0000 7f28edc14640 1 -- 172.21.15.7:0/3275729507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f28e8108310 con 0x7f28e8100e90 2024-04-21T08:44:40.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.716+0000 7f28e4ff9640 1 -- 172.21.15.7:0/3275729507 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f28e8108310 con 0x7f28e8100e90 2024-04-21T08:44:40.716 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:44:40.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 -- 172.21.15.7:0/3275729507 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28d40a3c00 msgr2=0x7f28d40a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:44:40.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28d40a3c00 0x7f28d40a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 -- 172.21.15.7:0/3275729507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28e8100e90 msgr2=0x7f28e8198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28e8100e90 0x7f28e8198250 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f28d800e9f0 tx=0x7f28d800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 -- 172.21.15.7:0/3275729507 shutdown_connections 2024-04-21T08:44:40.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f28d40a3c00 0x7f28d40a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28e81029a0 0x7f28e819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28e8101fe0 0x7f28e8198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 --2- 172.21.15.7:0/3275729507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28e8100e90 0x7f28e8198250 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 -- 172.21.15.7:0/3275729507 >> 172.21.15.7:0/3275729507 conn(0x7f28e80fc420 msgr2=0x7f28e81054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:40.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 -- 172.21.15.7:0/3275729507 shutdown_connections 2024-04-21T08:44:40.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.718+0000 7f28edc14640 1 -- 172.21.15.7:0/3275729507 wait complete. 2024-04-21T08:44:40.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:40.795 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:40 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:40.366+0000 7f765b8a9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:44:40.795 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:44:40 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z[33474]: 2024-04-21T08:44:40.581+0000 7f765b8a9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:44:40.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.840+0000 7fe45c6c8640 1 -- 172.21.15.7:0/565046549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe454105340 msgr2=0x7fe454107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.840+0000 7fe45c6c8640 1 --2- 172.21.15.7:0/565046549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe454105340 0x7fe454107730 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe450009860 tx=0x7fe45002bb70 comp rx=0 tx=0).stop 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.840+0000 7fe45c6c8640 1 -- 172.21.15.7:0/565046549 shutdown_connections 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.840+0000 7fe45c6c8640 1 --2- 172.21.15.7:0/565046549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454107c70 0x7fe45410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.840+0000 7fe45c6c8640 1 --2- 172.21.15.7:0/565046549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe454105340 0x7fe454107730 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.840+0000 7fe45c6c8640 1 --2- 172.21.15.7:0/565046549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe454102a10 0x7fe454104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.840+0000 7fe45c6c8640 1 -- 172.21.15.7:0/565046549 >> 172.21.15.7:0/565046549 conn(0x7fe4540fc420 msgr2=0x7fe4540fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.840+0000 7fe45c6c8640 1 -- 172.21.15.7:0/565046549 shutdown_connections 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.840+0000 7fe45c6c8640 1 -- 172.21.15.7:0/565046549 wait complete. 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.840+0000 7fe45c6c8640 1 Processor -- start 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.840+0000 7fe45c6c8640 1 -- start start 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe45c6c8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454102a10 0x7fe454102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe45c6c8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe454105340 0x7fe4541009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe45c6c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe454107c70 0x7fe454100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe45c6c8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe45410da70 con 0x7fe454102a10 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe45c6c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe45410d8f0 con 0x7fe454107c70 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe45c6c8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe45410dbf0 con 0x7fe454105340 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe45a464640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454102a10 0x7fe454102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe459c63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe454105340 0x7fe4541009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe459c63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe454105340 0x7fe4541009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:42420/0 (socket says 172.21.15.7:42420) 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe459c63640 1 -- 172.21.15.7:0/3982455684 learned_addr learned my addr 172.21.15.7:0/3982455684 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe45ac65640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe454107c70 0x7fe454100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe459c63640 1 -- 172.21.15.7:0/3982455684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe454107c70 msgr2=0x7fe454100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe459c63640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe454107c70 0x7fe454100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe459c63640 1 -- 172.21.15.7:0/3982455684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454102a10 msgr2=0x7fe454102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe459c63640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454102a10 0x7fe454102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:40.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe459c63640 1 -- 172.21.15.7:0/3982455684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe454101790 con 0x7fe454105340 2024-04-21T08:44:40.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe45a464640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454102a10 0x7fe454102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:40.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe459c63640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe454105340 0x7fe4541009c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe45002c0b0 tx=0x7fe4500042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:40.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.841+0000 7fe44b7fe640 1 -- 172.21.15.7:0/3982455684 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450041020 con 0x7fe454105340 2024-04-21T08:44:40.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.842+0000 7fe44b7fe640 1 -- 172.21.15.7:0/3982455684 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4500035f0 con 0x7fe454105340 2024-04-21T08:44:40.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.842+0000 7fe44b7fe640 1 -- 172.21.15.7:0/3982455684 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450039040 con 0x7fe454105340 2024-04-21T08:44:40.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.842+0000 7fe45c6c8640 1 -- 172.21.15.7:0/3982455684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4541a2340 con 0x7fe454105340 2024-04-21T08:44:40.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.842+0000 7fe45c6c8640 1 -- 172.21.15.7:0/3982455684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4541a2890 con 0x7fe454105340 2024-04-21T08:44:40.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.843+0000 7fe45c6c8640 1 -- 172.21.15.7:0/3982455684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4540fdf10 con 0x7fe454105340 2024-04-21T08:44:40.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.844+0000 7fe44b7fe640 1 -- 172.21.15.7:0/3982455684 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 138662+0+0 (secure 0 0 0) 0x7fe450044020 con 0x7fe454105340 2024-04-21T08:44:40.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.844+0000 7fe44b7fe640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4400a3c00 0x7fe4400a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:40.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.844+0000 7fe45a464640 1 -- 172.21.15.7:0/3982455684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4400a3c00 msgr2=0x7fe4400a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:40.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.844+0000 7fe44b7fe640 1 -- 172.21.15.7:0/3982455684 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7fe4500f2280 con 0x7fe454105340 2024-04-21T08:44:40.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.844+0000 7fe45a464640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4400a3c00 0x7fe4400a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:44:40.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:40.849+0000 7fe44b7fe640 1 -- 172.21.15.7:0/3982455684 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe45004b020 con 0x7fe454105340 2024-04-21T08:44:40.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:40 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/917426624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:40.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:40 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1608284862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:40.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:40 smithi007 ceph-mon[28477]: Standby manager daemon z restarted 2024-04-21T08:44:40.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:40 smithi007 ceph-mon[28477]: Standby manager daemon z started 2024-04-21T08:44:40.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:40 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:44:40.991 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:40 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:44:40.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:40 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:44:40.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:40 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:44:40.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:40 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3275729507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:41.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.038+0000 7fe45c6c8640 1 -- 172.21.15.7:0/3982455684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe45402d5c0 con 0x7fe454105340 2024-04-21T08:44:41.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.038+0000 7fe44b7fe640 1 -- 172.21.15.7:0/3982455684 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fe4500be7f0 con 0x7fe454105340 2024-04-21T08:44:41.039 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:41.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 -- 172.21.15.7:0/3982455684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4400a3c00 msgr2=0x7fe4400a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:44:41.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4400a3c00 0x7fe4400a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 -- 172.21.15.7:0/3982455684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe454105340 msgr2=0x7fe4541009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe454105340 0x7fe4541009c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe45002c0b0 tx=0x7fe4500042f0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 -- 172.21.15.7:0/3982455684 shutdown_connections 2024-04-21T08:44:41.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7fe4400a3c00 0x7fe4400a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe454107c70 0x7fe454100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe454105340 0x7fe4541009c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 --2- 172.21.15.7:0/3982455684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe454102a10 0x7fe454102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 -- 172.21.15.7:0/3982455684 >> 172.21.15.7:0/3982455684 conn(0x7fe4540fc420 msgr2=0x7fe454103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:41.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 -- 172.21.15.7:0/3982455684 shutdown_connections 2024-04-21T08:44:41.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.041+0000 7fe45c6c8640 1 -- 172.21.15.7:0/3982455684 wait complete. 2024-04-21T08:44:41.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:41.054 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:40.661+0000 7f13d75ab200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:44:41.054 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y[32643]: 2024-04-21T08:44:40.886+0000 7f13d75ab200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:44:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/917426624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1608284862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-mon[31259]: Standby manager daemon z restarted 2024-04-21T08:44:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-mon[31259]: Standby manager daemon z started 2024-04-21T08:44:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:44:41.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:44:41.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:44:41.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:44:41.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:40 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3275729507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:41.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:40 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/917426624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:40 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1608284862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:40 smithi077 ceph-mon[31615]: Standby manager daemon z restarted 2024-04-21T08:44:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:40 smithi077 ceph-mon[31615]: Standby manager daemon z started 2024-04-21T08:44:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:40 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:44:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:40 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:44:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:40 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:44:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:40 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.77:0/1135089975' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:44:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:40 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3275729507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.155+0000 7f797c86a640 1 -- 172.21.15.7:0/3174881375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7978107c60 msgr2=0x7f797810a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.155+0000 7f797c86a640 1 --2- 172.21.15.7:0/3174881375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7978107c60 0x7f797810a0b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7970009ca0 tx=0x7f797002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.155+0000 7f797c86a640 1 -- 172.21.15.7:0/3174881375 shutdown_connections 2024-04-21T08:44:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.155+0000 7f797c86a640 1 --2- 172.21.15.7:0/3174881375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7978107c60 0x7f797810a0b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.155+0000 7f797c86a640 1 --2- 172.21.15.7:0/3174881375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7978105330 0x7f7978107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.155+0000 7f797c86a640 1 --2- 172.21.15.7:0/3174881375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7978102a00 0x7f7978104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.155+0000 7f797c86a640 1 -- 172.21.15.7:0/3174881375 >> 172.21.15.7:0/3174881375 conn(0x7f79780fc3f0 msgr2=0x7f79780fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.155+0000 7f797c86a640 1 -- 172.21.15.7:0/3174881375 shutdown_connections 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.155+0000 7f797c86a640 1 -- 172.21.15.7:0/3174881375 wait complete. 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f797c86a640 1 Processor -- start 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f797c86a640 1 -- start start 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f797c86a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7978102a00 0x7f7978102370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f797c86a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7978105330 0x7f79781009b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f797c86a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7978107c60 0x7f7978100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f797c86a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f797810dad0 con 0x7f7978107c60 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f797c86a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f797810d950 con 0x7f7978105330 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f797c86a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f797810dc50 con 0x7f7978102a00 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f7976d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7978107c60 0x7f7978100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f7976d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7978107c60 0x7f7978100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53482/0 (socket says 172.21.15.7:53482) 2024-04-21T08:44:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f7976d9d640 1 -- 172.21.15.7:0/4195764895 learned_addr learned my addr 172.21.15.7:0/4195764895 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f797659c640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7978102a00 0x7f7978102370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.156+0000 7f7975d9b640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7978105330 0x7f79781009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f7976d9d640 1 -- 172.21.15.7:0/4195764895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7978102a00 msgr2=0x7f7978102370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f7976d9d640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7978102a00 0x7f7978102370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f7976d9d640 1 -- 172.21.15.7:0/4195764895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7978105330 msgr2=0x7f79781009b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f7976d9d640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7978105330 0x7f79781009b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f7976d9d640 1 -- 172.21.15.7:0/4195764895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7978101780 con 0x7f7978107c60 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f7975d9b640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7978105330 0x7f79781009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f797659c640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7978102a00 0x7f7978102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f7976d9d640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7978107c60 0x7f7978100ef0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f797002c700 tx=0x7f79700346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f79677fe640 1 -- 172.21.15.7:0/4195764895 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7970041020 con 0x7f7978107c60 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f797c86a640 1 -- 172.21.15.7:0/4195764895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79781a2230 con 0x7f7978107c60 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f79677fe640 1 -- 172.21.15.7:0/4195764895 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7970034ca0 con 0x7f7978107c60 2024-04-21T08:44:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f79677fe640 1 -- 172.21.15.7:0/4195764895 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7970039040 con 0x7f7978107c60 2024-04-21T08:44:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.157+0000 7f797c86a640 1 -- 172.21.15.7:0/4195764895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79781a27d0 con 0x7f7978107c60 2024-04-21T08:44:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.158+0000 7f797c86a640 1 -- 172.21.15.7:0/4195764895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79780fdee0 con 0x7f7978107c60 2024-04-21T08:44:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.159+0000 7f79677fe640 1 -- 172.21.15.7:0/4195764895 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 20) ==== 138662+0+0 (secure 0 0 0) 0x7f7970044020 con 0x7f7978107c60 2024-04-21T08:44:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.160+0000 7f79677fe640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79600a3c00 0x7f79600a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.160+0000 7f797659c640 1 -- 172.21.15.7:0/4195764895 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79600a3c00 msgr2=0x7f79600a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/2555078032 2024-04-21T08:44:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.160+0000 7f79677fe640 1 -- 172.21.15.7:0/4195764895 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9266+0+0 (secure 0 0 0) 0x7f79700f2ac0 con 0x7f7978107c60 2024-04-21T08:44:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.160+0000 7f797659c640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79600a3c00 0x7f79600a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:44:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.164+0000 7f79677fe640 1 -- 172.21.15.7:0/4195764895 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f797004b020 con 0x7f7978107c60 2024-04-21T08:44:41.242 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:40 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:40.990+0000 7f6e8ced7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:44:41.242 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:41.241+0000 7f6e8ced7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:44:41.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.254+0000 7f79677fe640 1 -- 172.21.15.7:0/4195764895 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f79700bafa0 con 0x7f7978107c60 2024-04-21T08:44:41.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.255+0000 7f79677fe640 1 -- 172.21.15.7:0/4195764895 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79600a3c00 msgr2=0x7f79600a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:44:41.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.255+0000 7f79677fe640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79600a3c00 0x7f79600a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.397+0000 7f797c86a640 1 -- 172.21.15.7:0/4195764895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7978062b50 con 0x7f7978107c60 2024-04-21T08:44:41.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.400+0000 7f79677fe640 1 -- 172.21.15.7:0/4195764895 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25085 (secure 0 0 0) 0x7f79700bf030 con 0x7f7978107c60 2024-04-21T08:44:41.401 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:41.401 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":95,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:44:41.244735+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3451243741":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/117789107":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/4167672076":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/3630897301":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:41.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.403+0000 7f797c86a640 1 -- 172.21.15.7:0/4195764895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7978107c60 msgr2=0x7f7978100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.403+0000 7f797c86a640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7978107c60 0x7f7978100ef0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f797002c700 tx=0x7f79700346d0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.403+0000 7f797c86a640 1 -- 172.21.15.7:0/4195764895 shutdown_connections 2024-04-21T08:44:41.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.403+0000 7f797c86a640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.7:6800/2555078032,v1:172.21.15.7:6801/2555078032] conn(0x7f79600a3c00 0x7f79600a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.403+0000 7f797c86a640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7978107c60 0x7f7978100ef0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.403+0000 7f797c86a640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7978105330 0x7f79781009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.403+0000 7f797c86a640 1 --2- 172.21.15.7:0/4195764895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7978102a00 0x7f7978102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.403+0000 7f797c86a640 1 -- 172.21.15.7:0/4195764895 >> 172.21.15.7:0/4195764895 conn(0x7f79780fc3f0 msgr2=0x7f7978103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:41.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.404+0000 7f797c86a640 1 -- 172.21.15.7:0/4195764895 shutdown_connections 2024-04-21T08:44:41.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.404+0000 7f797c86a640 1 -- 172.21.15.7:0/4195764895 wait complete. 2024-04-21T08:44:41.420 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:41.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.554+0000 7f6c43cd2640 1 -- 172.21.15.7:0/3316951307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c105400 msgr2=0x7f6c3c1077f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.554+0000 7f6c43cd2640 1 --2- 172.21.15.7:0/3316951307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c105400 0x7f6c3c1077f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009900 tx=0x7f6c2c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.554+0000 7f6c43cd2640 1 -- 172.21.15.7:0/3316951307 shutdown_connections 2024-04-21T08:44:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.554+0000 7f6c43cd2640 1 --2- 172.21.15.7:0/3316951307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c3c107d30 0x7f6c3c10a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.554+0000 7f6c43cd2640 1 --2- 172.21.15.7:0/3316951307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c105400 0x7f6c3c1077f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.554+0000 7f6c43cd2640 1 --2- 172.21.15.7:0/3316951307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c3c102ad0 0x7f6c3c104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.554+0000 7f6c43cd2640 1 -- 172.21.15.7:0/3316951307 >> 172.21.15.7:0/3316951307 conn(0x7f6c3c0fc440 msgr2=0x7f6c3c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.554+0000 7f6c43cd2640 1 -- 172.21.15.7:0/3316951307 shutdown_connections 2024-04-21T08:44:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.554+0000 7f6c43cd2640 1 -- 172.21.15.7:0/3316951307 wait complete. 2024-04-21T08:44:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.554+0000 7f6c43cd2640 1 Processor -- start 2024-04-21T08:44:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.554+0000 7f6c43cd2640 1 -- start start 2024-04-21T08:44:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c43cd2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c3c102ad0 0x7f6c3c198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c43cd2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c3c105400 0x7f6c3c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c43cd2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c107d30 0x7f6c3c19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c43cd2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c10db60 con 0x7f6c3c107d30 2024-04-21T08:44:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c43cd2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c10d9e0 con 0x7f6c3c105400 2024-04-21T08:44:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c43cd2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c10dce0 con 0x7f6c3c102ad0 2024-04-21T08:44:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c4226f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c107d30 0x7f6c3c19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c4226f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c107d30 0x7f6c3c19f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53588/0 (socket says 172.21.15.7:53588) 2024-04-21T08:44:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c4226f640 1 -- 172.21.15.7:0/708680441 learned_addr learned my addr 172.21.15.7:0/708680441 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c4126d640 1 --2- 172.21.15.7:0/708680441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c3c105400 0x7f6c3c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c41a6e640 1 --2- 172.21.15.7:0/708680441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c3c102ad0 0x7f6c3c198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c4226f640 1 -- 172.21.15.7:0/708680441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c3c102ad0 msgr2=0x7f6c3c198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c4226f640 1 --2- 172.21.15.7:0/708680441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c3c102ad0 0x7f6c3c198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c4226f640 1 -- 172.21.15.7:0/708680441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c3c105400 msgr2=0x7f6c3c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c4226f640 1 --2- 172.21.15.7:0/708680441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c3c105400 0x7f6c3c1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c4226f640 1 -- 172.21.15.7:0/708680441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3c19fe90 con 0x7f6c3c107d30 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c41a6e640 1 --2- 172.21.15.7:0/708680441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c3c102ad0 0x7f6c3c198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.555+0000 7f6c4126d640 1 --2- 172.21.15.7:0/708680441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c3c105400 0x7f6c3c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.556+0000 7f6c4226f640 1 --2- 172.21.15.7:0/708680441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c107d30 0x7f6c3c19f820 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400eb10 tx=0x7f6c3400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.556+0000 7f6c32ffd640 1 -- 172.21.15.7:0/708680441 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34004b20 con 0x7f6c3c107d30 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.556+0000 7f6c32ffd640 1 -- 172.21.15.7:0/708680441 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c34004cc0 con 0x7f6c3c107d30 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.556+0000 7f6c43cd2640 1 -- 172.21.15.7:0/708680441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3c100920 con 0x7f6c3c107d30 2024-04-21T08:44:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.556+0000 7f6c43cd2640 1 -- 172.21.15.7:0/708680441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c3c100e40 con 0x7f6c3c107d30 2024-04-21T08:44:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.556+0000 7f6c32ffd640 1 -- 172.21.15.7:0/708680441 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34010690 con 0x7f6c3c107d30 2024-04-21T08:44:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.557+0000 7f6c43cd2640 1 -- 172.21.15.7:0/708680441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c200052c0 con 0x7f6c3c107d30 2024-04-21T08:44:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.559+0000 7f6c32ffd640 1 -- 172.21.15.7:0/708680441 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f6c34005000 con 0x7f6c3c107d30 2024-04-21T08:44:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.560+0000 7f6c32ffd640 1 -- 172.21.15.7:0/708680441 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f6c340d09e0 con 0x7f6c3c107d30 2024-04-21T08:44:41.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.562+0000 7f6c32ffd640 1 -- 172.21.15.7:0/708680441 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c34053ff0 con 0x7f6c3c107d30 2024-04-21T08:44:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.784+0000 7f6c43cd2640 1 -- 172.21.15.7:0/708680441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6c20005880 con 0x7f6c3c107d30 2024-04-21T08:44:41.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.784+0000 7f6c32ffd640 1 -- 172.21.15.7:0/708680441 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f6c3409d250 con 0x7f6c3c107d30 2024-04-21T08:44:41.785 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:44:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.786+0000 7f6c43cd2640 1 -- 172.21.15.7:0/708680441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c107d30 msgr2=0x7f6c3c19f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.786+0000 7f6c43cd2640 1 --2- 172.21.15.7:0/708680441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c107d30 0x7f6c3c19f820 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400eb10 tx=0x7f6c3400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.786+0000 7f6c43cd2640 1 -- 172.21.15.7:0/708680441 shutdown_connections 2024-04-21T08:44:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.786+0000 7f6c43cd2640 1 --2- 172.21.15.7:0/708680441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c107d30 0x7f6c3c19f820 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.786+0000 7f6c43cd2640 1 --2- 172.21.15.7:0/708680441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c3c105400 0x7f6c3c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.786+0000 7f6c43cd2640 1 --2- 172.21.15.7:0/708680441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c3c102ad0 0x7f6c3c198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.786+0000 7f6c43cd2640 1 -- 172.21.15.7:0/708680441 >> 172.21.15.7:0/708680441 conn(0x7f6c3c0fc440 msgr2=0x7f6c3c103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.786+0000 7f6c43cd2640 1 -- 172.21.15.7:0/708680441 shutdown_connections 2024-04-21T08:44:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.786+0000 7f6c43cd2640 1 -- 172.21.15.7:0/708680441 wait complete. 2024-04-21T08:44:41.794 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T08:44:41.795 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: Standby manager daemon y restarted 2024-04-21T08:44:41.795 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: Standby manager daemon y started 2024-04-21T08:44:41.795 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:44:41.796 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:41.515+0000 7f6deb008700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:44:41.796 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:41.515+0000 7f6deb008700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:44:41.796 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:41.515+0000 7f6deb008700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:44:41.796 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:41.515+0000 7f6deb008700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:44:41.796 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:41.515+0000 7f6deb008700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:44:41.796 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:41.516+0000 7f6dee04e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:44:41.796 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:41.516+0000 7f6dee04e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:44:41.796 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:41.516+0000 7f6dee04e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:44:41.796 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:41.516+0000 7f6dee04e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:44:41.796 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:44:41.516+0000 7f6dee04e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:44:41.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:41.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.909+0000 7f1fb7c81640 1 -- 172.21.15.7:0/4173273120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fb0101fe0 msgr2=0x7f1fb0102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.909+0000 7f1fb7c81640 1 --2- 172.21.15.7:0/4173273120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fb0101fe0 0x7f1fb0102460 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009900 tx=0x7f1fa002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.909+0000 7f1fb7c81640 1 -- 172.21.15.7:0/4173273120 shutdown_connections 2024-04-21T08:44:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.909+0000 7f1fb7c81640 1 --2- 172.21.15.7:0/4173273120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fb01029a0 0x7f1fb01092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.909+0000 7f1fb7c81640 1 --2- 172.21.15.7:0/4173273120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fb0101fe0 0x7f1fb0102460 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.909+0000 7f1fb7c81640 1 --2- 172.21.15.7:0/4173273120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb0100e90 0x7f1fb01012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.909+0000 7f1fb7c81640 1 -- 172.21.15.7:0/4173273120 >> 172.21.15.7:0/4173273120 conn(0x7f1fb00fc420 msgr2=0x7f1fb00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.909+0000 7f1fb7c81640 1 -- 172.21.15.7:0/4173273120 shutdown_connections 2024-04-21T08:44:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.909+0000 7f1fb7c81640 1 -- 172.21.15.7:0/4173273120 wait complete. 2024-04-21T08:44:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.909+0000 7f1fb7c81640 1 Processor -- start 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb7c81640 1 -- start start 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb7c81640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fb0100e90 0x7f1fb01981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb7c81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fb0101fe0 0x7f1fb0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb7c81640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01029a0 0x7f1fb019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb7c81640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb010c9e0 con 0x7f1fb0100e90 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb7c81640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb010c860 con 0x7f1fb01029a0 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb7c81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb010cb60 con 0x7f1fb0101fe0 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb5a1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fb0100e90 0x7f1fb01981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb5a1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fb0100e90 0x7f1fb01981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53610/0 (socket says 172.21.15.7:53610) 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb5a1d640 1 -- 172.21.15.7:0/2680971337 learned_addr learned my addr 172.21.15.7:0/2680971337 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb621e640 1 --2- 172.21.15.7:0/2680971337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01029a0 0x7f1fb019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb521c640 1 --2- 172.21.15.7:0/2680971337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fb0101fe0 0x7f1fb0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb621e640 1 -- 172.21.15.7:0/2680971337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fb0101fe0 msgr2=0x7f1fb0198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb621e640 1 --2- 172.21.15.7:0/2680971337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fb0101fe0 0x7f1fb0198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb621e640 1 -- 172.21.15.7:0/2680971337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fb0100e90 msgr2=0x7f1fb01981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb621e640 1 --2- 172.21.15.7:0/2680971337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fb0100e90 0x7f1fb01981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb621e640 1 -- 172.21.15.7:0/2680971337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb019fdf0 con 0x7f1fb01029a0 2024-04-21T08:44:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb5a1d640 1 --2- 172.21.15.7:0/2680971337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fb0100e90 0x7f1fb01981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.910+0000 7f1fb621e640 1 --2- 172.21.15.7:0/2680971337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01029a0 0x7f1fb019f780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009fd0 tx=0x7f1fa800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.911+0000 7f1fa6ffd640 1 -- 172.21.15.7:0/2680971337 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa800ce20 con 0x7f1fb01029a0 2024-04-21T08:44:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.911+0000 7f1fb7c81640 1 -- 172.21.15.7:0/2680971337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb01a00e0 con 0x7f1fb01029a0 2024-04-21T08:44:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.911+0000 7f1fa6ffd640 1 -- 172.21.15.7:0/2680971337 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fa8005020 con 0x7f1fb01029a0 2024-04-21T08:44:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.911+0000 7f1fa6ffd640 1 -- 172.21.15.7:0/2680971337 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8010690 con 0x7f1fb01029a0 2024-04-21T08:44:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.911+0000 7f1fb7c81640 1 -- 172.21.15.7:0/2680971337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb01a05b0 con 0x7f1fb01029a0 2024-04-21T08:44:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.912+0000 7f1fb7c81640 1 -- 172.21.15.7:0/2680971337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fb01a0870 con 0x7f1fb01029a0 2024-04-21T08:44:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.913+0000 7f1fa6ffd640 1 -- 172.21.15.7:0/2680971337 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f1fa80054d0 con 0x7f1fb01029a0 2024-04-21T08:44:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.914+0000 7f1fa6ffd640 1 -- 172.21.15.7:0/2680971337 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f1fa80d0ad0 con 0x7f1fb01029a0 2024-04-21T08:44:41.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:41.918+0000 7f1fa6ffd640 1 -- 172.21.15.7:0/2680971337 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1fa809d340 con 0x7f1fb01029a0 2024-04-21T08:44:42.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: Standby manager daemon y restarted 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: Standby manager daemon y started 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3982455684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: Active manager daemon x restarted 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: Activating manager daemon x 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: mgrmap e21: x(active, starting, since 0.0135502s), standbys: y, z 2024-04-21T08:44:42.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:44:42.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:44:42.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: Manager daemon x is now available 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4195764895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:42.058 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:41 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/708680441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:44:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:44:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:44:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3982455684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: Active manager daemon x restarted 2024-04-21T08:44:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: Activating manager daemon x 2024-04-21T08:44:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:44:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: mgrmap e21: x(active, starting, since 0.0135502s), standbys: y, z 2024-04-21T08:44:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:42.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:44:42.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:44:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:44:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:44:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:44:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:44:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: Manager daemon x is now available 2024-04-21T08:44:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:44:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:44:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:44:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:44:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4195764895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:41 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/708680441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:42.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.105+0000 7f1fb7c81640 1 -- 172.21.15.7:0/2680971337 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1fb01a0a40 con 0x7f1fb01029a0 2024-04-21T08:44:42.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.105+0000 7f1fa6ffd640 1 -- 172.21.15.7:0/2680971337 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f1fa809cc40 con 0x7f1fb01029a0 2024-04-21T08:44:42.108 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:42.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.108+0000 7f1fb7c81640 1 -- 172.21.15.7:0/2680971337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01029a0 msgr2=0x7f1fb019f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.108+0000 7f1fb7c81640 1 --2- 172.21.15.7:0/2680971337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01029a0 0x7f1fb019f780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009fd0 tx=0x7f1fa800ef10 comp rx=0 tx=0).stop 2024-04-21T08:44:42.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.109+0000 7f1fb7c81640 1 -- 172.21.15.7:0/2680971337 shutdown_connections 2024-04-21T08:44:42.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.109+0000 7f1fb7c81640 1 --2- 172.21.15.7:0/2680971337 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fb01029a0 0x7f1fb019f780 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.109+0000 7f1fb7c81640 1 --2- 172.21.15.7:0/2680971337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fb0101fe0 0x7f1fb0198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.109+0000 7f1fb7c81640 1 --2- 172.21.15.7:0/2680971337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fb0100e90 0x7f1fb01981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.109+0000 7f1fb7c81640 1 -- 172.21.15.7:0/2680971337 >> 172.21.15.7:0/2680971337 conn(0x7f1fb00fc420 msgr2=0x7f1fb00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:42.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.109+0000 7f1fb7c81640 1 -- 172.21.15.7:0/2680971337 shutdown_connections 2024-04-21T08:44:42.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.109+0000 7f1fb7c81640 1 -- 172.21.15.7:0/2680971337 wait complete. 2024-04-21T08:44:42.123 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: Standby manager daemon y restarted 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: Standby manager daemon y started 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.? 172.21.15.16:0/1193049695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3982455684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: Active manager daemon x restarted 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: Activating manager daemon x 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: mgrmap e21: x(active, starting, since 0.0135502s), standbys: y, z 2024-04-21T08:44:42.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:44:42.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:44:42.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:44:42.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:42.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:44:42.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:44:42.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:44:42.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:44:42.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:42.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: Manager daemon x is now available 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:42.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:44:42.129 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:44:42.129 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:44:42.129 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:44:42.129 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4195764895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:42.129 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:41 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/708680441' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:42.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.238+0000 7fb0e7881640 1 -- 172.21.15.7:0/1494789957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e0100e90 msgr2=0x7fb0e01012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.238+0000 7fb0e7881640 1 --2- 172.21.15.7:0/1494789957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e0100e90 0x7fb0e01012b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009900 tx=0x7fb0dc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:42.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.238+0000 7fb0e7881640 1 -- 172.21.15.7:0/1494789957 shutdown_connections 2024-04-21T08:44:42.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.238+0000 7fb0e7881640 1 --2- 172.21.15.7:0/1494789957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e01029a0 0x7fb0e01092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.238+0000 7fb0e7881640 1 --2- 172.21.15.7:0/1494789957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e0101fe0 0x7fb0e0102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.238+0000 7fb0e7881640 1 --2- 172.21.15.7:0/1494789957 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e0100e90 0x7fb0e01012b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.238+0000 7fb0e7881640 1 -- 172.21.15.7:0/1494789957 >> 172.21.15.7:0/1494789957 conn(0x7fb0e00fc420 msgr2=0x7fb0e00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:42.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.238+0000 7fb0e7881640 1 -- 172.21.15.7:0/1494789957 shutdown_connections 2024-04-21T08:44:42.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.238+0000 7fb0e7881640 1 -- 172.21.15.7:0/1494789957 wait complete. 2024-04-21T08:44:42.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.238+0000 7fb0e7881640 1 Processor -- start 2024-04-21T08:44:42.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.238+0000 7fb0e7881640 1 -- start start 2024-04-21T08:44:42.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e7881640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e0100e90 0x7fb0e0104dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e7881640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e0101fe0 0x7fb0e0103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e7881640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e01029a0 0x7fb0e0103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e7881640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e010c9c0 con 0x7fb0e01029a0 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e7881640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e010c840 con 0x7fb0e0101fe0 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e7881640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e010cb40 con 0x7fb0e0100e90 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e5e1e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e01029a0 0x7fb0e0103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e4e1c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e0101fe0 0x7fb0e0103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e561d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e0100e90 0x7fb0e0104dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e4e1c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e0101fe0 0x7fb0e0103450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:36912/0 (socket says 172.21.15.7:36912) 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e5e1e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e01029a0 0x7fb0e0103990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53628/0 (socket says 172.21.15.7:53628) 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e561d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e0100e90 0x7fb0e0104dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:42578/0 (socket says 172.21.15.7:42578) 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e4e1c640 1 -- 172.21.15.7:0/3353669853 learned_addr learned my addr 172.21.15.7:0/3353669853 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e4e1c640 1 -- 172.21.15.7:0/3353669853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e0100e90 msgr2=0x7fb0e0104dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e4e1c640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e0100e90 0x7fb0e0104dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e4e1c640 1 -- 172.21.15.7:0/3353669853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e01029a0 msgr2=0x7fb0e0103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e4e1c640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e01029a0 0x7fb0e0103990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e4e1c640 1 -- 172.21.15.7:0/3353669853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e0103ed0 con 0x7fb0e0101fe0 2024-04-21T08:44:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e561d640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e0100e90 0x7fb0e0104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.239+0000 7fb0e5e1e640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e01029a0 0x7fb0e0103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.240+0000 7fb0e4e1c640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e0101fe0 0x7fb0e0103450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000e9c0 tx=0x7fb0d000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.240+0000 7fb0d67fc640 1 -- 172.21.15.7:0/3353669853 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d000cde0 con 0x7fb0e0101fe0 2024-04-21T08:44:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.240+0000 7fb0d67fc640 1 -- 172.21.15.7:0/3353669853 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0d00108e0 con 0x7fb0e0101fe0 2024-04-21T08:44:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.240+0000 7fb0e7881640 1 -- 172.21.15.7:0/3353669853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e01041c0 con 0x7fb0e0101fe0 2024-04-21T08:44:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.240+0000 7fb0d67fc640 1 -- 172.21.15.7:0/3353669853 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0018960 con 0x7fb0e0101fe0 2024-04-21T08:44:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.240+0000 7fb0e7881640 1 -- 172.21.15.7:0/3353669853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e01a6cf0 con 0x7fb0e0101fe0 2024-04-21T08:44:42.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.241+0000 7fb0e7881640 1 -- 172.21.15.7:0/3353669853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e0068e00 con 0x7fb0e0101fe0 2024-04-21T08:44:42.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.242+0000 7fb0d67fc640 1 -- 172.21.15.7:0/3353669853 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7fb0d001f020 con 0x7fb0e0101fe0 2024-04-21T08:44:42.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.243+0000 7fb0d67fc640 1 -- 172.21.15.7:0/3353669853 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fb0d00d0850 con 0x7fb0e0101fe0 2024-04-21T08:44:42.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.247+0000 7fb0d67fc640 1 -- 172.21.15.7:0/3353669853 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb0d0046e20 con 0x7fb0e0101fe0 2024-04-21T08:44:42.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.262+0000 7fb0d67fc640 1 -- 172.21.15.7:0/3353669853 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fb0d0010a90 con 0x7fb0e0101fe0 2024-04-21T08:44:42.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.262+0000 7fb0d67fc640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0cc0ac1c0 0x7fb0cc0ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.264+0000 7fb0e561d640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0cc0ac1c0 0x7fb0cc0ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.267+0000 7fb0e561d640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0cc0ac1c0 0x7fb0cc0ae5b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009230 tx=0x7fb0dc005fd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:42.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.457+0000 7fb0e7881640 1 -- 172.21.15.7:0/3353669853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb0e01a70e0 con 0x7fb0e0101fe0 2024-04-21T08:44:42.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.459+0000 7fb0d67fc640 1 -- 172.21.15.7:0/3353669853 <== mon.2 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fb0d00a35c0 con 0x7fb0e0101fe0 2024-04-21T08:44:42.458 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:44:42.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 -- 172.21.15.7:0/3353669853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0cc0ac1c0 msgr2=0x7fb0cc0ae5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0cc0ac1c0 0x7fb0cc0ae5b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009230 tx=0x7fb0dc005fd0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 -- 172.21.15.7:0/3353669853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e0101fe0 msgr2=0x7fb0e0103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e0101fe0 0x7fb0e0103450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000e9c0 tx=0x7fb0d000ee80 comp rx=0 tx=0).stop 2024-04-21T08:44:42.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 -- 172.21.15.7:0/3353669853 shutdown_connections 2024-04-21T08:44:42.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e01029a0 0x7fb0e0103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0e0101fe0 0x7fb0e0103450 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0cc0ac1c0 0x7fb0cc0ae5b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 --2- 172.21.15.7:0/3353669853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0e0100e90 0x7fb0e0104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 -- 172.21.15.7:0/3353669853 >> 172.21.15.7:0/3353669853 conn(0x7fb0e00fc420 msgr2=0x7fb0e00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:42.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 -- 172.21.15.7:0/3353669853 shutdown_connections 2024-04-21T08:44:42.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.461+0000 7fb0e7881640 1 -- 172.21.15.7:0/3353669853 wait complete. 2024-04-21T08:44:42.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:42.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.576+0000 7efe7f6a5640 1 -- 172.21.15.7:0/2847002797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe78103190 msgr2=0x7efe78103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.576+0000 7efe7f6a5640 1 --2- 172.21.15.7:0/2847002797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe78103190 0x7efe78103610 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7efe68009900 tx=0x7efe6802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:42.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.576+0000 7efe7f6a5640 1 -- 172.21.15.7:0/2847002797 shutdown_connections 2024-04-21T08:44:42.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.576+0000 7efe7f6a5640 1 --2- 172.21.15.7:0/2847002797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe78103b50 0x7efe7810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.576+0000 7efe7f6a5640 1 --2- 172.21.15.7:0/2847002797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe78103190 0x7efe78103610 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.576+0000 7efe7f6a5640 1 --2- 172.21.15.7:0/2847002797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe78101fa0 0x7efe781023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.576+0000 7efe7f6a5640 1 -- 172.21.15.7:0/2847002797 >> 172.21.15.7:0/2847002797 conn(0x7efe780fd510 msgr2=0x7efe780ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:42.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.576+0000 7efe7f6a5640 1 -- 172.21.15.7:0/2847002797 shutdown_connections 2024-04-21T08:44:42.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.576+0000 7efe7f6a5640 1 -- 172.21.15.7:0/2847002797 wait complete. 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7f6a5640 1 Processor -- start 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7f6a5640 1 -- start start 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7f6a5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe78101fa0 0x7efe781981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7f6a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe78103190 0x7efe78198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7f6a5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe78103b50 0x7efe7819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7f6a5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efe7810dae0 con 0x7efe78101fa0 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7f6a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe7810d960 con 0x7efe78103190 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7f6a5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efe7810dc60 con 0x7efe78103b50 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7d441640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe78101fa0 0x7efe781981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7d441640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe78101fa0 0x7efe781981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53648/0 (socket says 172.21.15.7:53648) 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7d441640 1 -- 172.21.15.7:0/346373134 learned_addr learned my addr 172.21.15.7:0/346373134 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7dc42640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe78103b50 0x7efe7819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7cc40640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe78103190 0x7efe78198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7d441640 1 -- 172.21.15.7:0/346373134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe78103b50 msgr2=0x7efe7819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7d441640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe78103b50 0x7efe7819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7d441640 1 -- 172.21.15.7:0/346373134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe78103190 msgr2=0x7efe78198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7d441640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe78103190 0x7efe78198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.577+0000 7efe7d441640 1 -- 172.21.15.7:0/346373134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe7819fde0 con 0x7efe78101fa0 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.578+0000 7efe7d441640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe78101fa0 0x7efe781981c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7efe7400e9c0 tx=0x7efe7400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.578+0000 7efe6e7fc640 1 -- 172.21.15.7:0/346373134 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7400cd80 con 0x7efe78101fa0 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.578+0000 7efe7f6a5640 1 -- 172.21.15.7:0/346373134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe781a00d0 con 0x7efe78101fa0 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.578+0000 7efe6e7fc640 1 -- 172.21.15.7:0/346373134 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe740035f0 con 0x7efe78101fa0 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.578+0000 7efe6e7fc640 1 -- 172.21.15.7:0/346373134 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe74010690 con 0x7efe78101fa0 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.578+0000 7efe7f6a5640 1 -- 172.21.15.7:0/346373134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe781a05f0 con 0x7efe78101fa0 2024-04-21T08:44:42.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.579+0000 7efe7f6a5640 1 -- 172.21.15.7:0/346373134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe78109340 con 0x7efe78101fa0 2024-04-21T08:44:42.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.580+0000 7efe6e7fc640 1 -- 172.21.15.7:0/346373134 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7efe740037a0 con 0x7efe78101fa0 2024-04-21T08:44:42.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.580+0000 7efe6e7fc640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe640a39b0 0x7efe640a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.581+0000 7efe7cc40640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe640a39b0 0x7efe640a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.581+0000 7efe6e7fc640 1 -- 172.21.15.7:0/346373134 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7efe740d0870 con 0x7efe78101fa0 2024-04-21T08:44:42.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.581+0000 7efe7cc40640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe640a39b0 0x7efe640a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efe68009230 tx=0x7efe68005fd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:42.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.585+0000 7efe6e7fc640 1 -- 172.21.15.7:0/346373134 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe7409d030 con 0x7efe78101fa0 2024-04-21T08:44:42.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.778+0000 7efe7f6a5640 1 -- 172.21.15.7:0/346373134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efe78062b50 con 0x7efe78101fa0 2024-04-21T08:44:42.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.779+0000 7efe6e7fc640 1 -- 172.21.15.7:0/346373134 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7efe7409c930 con 0x7efe78101fa0 2024-04-21T08:44:42.780 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 -- 172.21.15.7:0/346373134 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe640a39b0 msgr2=0x7efe640a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe640a39b0 0x7efe640a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efe68009230 tx=0x7efe68005fd0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 -- 172.21.15.7:0/346373134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe78101fa0 msgr2=0x7efe781981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe78101fa0 0x7efe781981c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7efe7400e9c0 tx=0x7efe7400ee80 comp rx=0 tx=0).stop 2024-04-21T08:44:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 -- 172.21.15.7:0/346373134 shutdown_connections 2024-04-21T08:44:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe640a39b0 0x7efe640a5e70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe78103b50 0x7efe7819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe78103190 0x7efe78198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 --2- 172.21.15.7:0/346373134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe78101fa0 0x7efe781981c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 -- 172.21.15.7:0/346373134 >> 172.21.15.7:0/346373134 conn(0x7efe780fd510 msgr2=0x7efe78106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 -- 172.21.15.7:0/346373134 shutdown_connections 2024-04-21T08:44:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.781+0000 7efe7f6a5640 1 -- 172.21.15.7:0/346373134 wait complete. 2024-04-21T08:44:42.793 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-21T08:44:42.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:42.859 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:42 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2680971337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:42.859 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:42 smithi016 ceph-mon[31259]: mgrmap e22: x(active, since 1.01699s), standbys: y, z 2024-04-21T08:44:42.859 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:42 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3353669853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:42.859 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:42 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/346373134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:42.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.897+0000 7fb93728c640 1 -- 172.21.15.7:0/1673248975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb930102a00 msgr2=0x7fb930104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.897+0000 7fb93728c640 1 --2- 172.21.15.7:0/1673248975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb930102a00 0x7fb930104df0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb92c009900 tx=0x7fb92c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:42.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.897+0000 7fb93728c640 1 -- 172.21.15.7:0/1673248975 shutdown_connections 2024-04-21T08:44:42.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.897+0000 7fb93728c640 1 --2- 172.21.15.7:0/1673248975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb930107c60 0x7fb93010a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 --2- 172.21.15.7:0/1673248975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb930105330 0x7fb930107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 --2- 172.21.15.7:0/1673248975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb930102a00 0x7fb930104df0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 -- 172.21.15.7:0/1673248975 >> 172.21.15.7:0/1673248975 conn(0x7fb9300fc3f0 msgr2=0x7fb9300fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 -- 172.21.15.7:0/1673248975 shutdown_connections 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 -- 172.21.15.7:0/1673248975 wait complete. 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 Processor -- start 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 -- start start 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb930102a00 0x7fb9301981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb930105330 0x7fb930198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb930107c60 0x7fb93019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb93010db00 con 0x7fb930107c60 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb93010d980 con 0x7fb930105330 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb93728c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb93010dc80 con 0x7fb930102a00 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb935829640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb930107c60 0x7fb93019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb935829640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb930107c60 0x7fb93019f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53666/0 (socket says 172.21.15.7:53666) 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.898+0000 7fb935829640 1 -- 172.21.15.7:0/1193411548 learned_addr learned my addr 172.21.15.7:0/1193411548 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:42.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb934827640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb930105330 0x7fb930198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb935028640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb930102a00 0x7fb9301981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb935829640 1 -- 172.21.15.7:0/1193411548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb930102a00 msgr2=0x7fb9301981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb935829640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb930102a00 0x7fb9301981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb935829640 1 -- 172.21.15.7:0/1193411548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb930105330 msgr2=0x7fb930198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb935829640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb930105330 0x7fb930198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb935829640 1 -- 172.21.15.7:0/1193411548 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb93019fdf0 con 0x7fb930107c60 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb935028640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb930102a00 0x7fb9301981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb935829640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb930107c60 0x7fb93019f780 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb92800eb10 tx=0x7fb92800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb9267fc640 1 -- 172.21.15.7:0/1193411548 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb928004b20 con 0x7fb930107c60 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb9267fc640 1 -- 172.21.15.7:0/1193411548 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb928004cc0 con 0x7fb930107c60 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb93728c640 1 -- 172.21.15.7:0/1193411548 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9301a00e0 con 0x7fb930107c60 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb9267fc640 1 -- 172.21.15.7:0/1193411548 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb928010670 con 0x7fb930107c60 2024-04-21T08:44:42.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.899+0000 7fb93728c640 1 -- 172.21.15.7:0/1193411548 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9301a05e0 con 0x7fb930107c60 2024-04-21T08:44:42.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.900+0000 7fb93728c640 1 -- 172.21.15.7:0/1193411548 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9300fdee0 con 0x7fb930107c60 2024-04-21T08:44:42.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.901+0000 7fb9267fc640 1 -- 172.21.15.7:0/1193411548 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fb928005020 con 0x7fb930107c60 2024-04-21T08:44:42.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.903+0000 7fb9267fc640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb91c0a39b0 0x7fb91c0a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:42.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.903+0000 7fb9267fc640 1 -- 172.21.15.7:0/1193411548 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fb9280d0b70 con 0x7fb930107c60 2024-04-21T08:44:42.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.903+0000 7fb935028640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb91c0a39b0 0x7fb91c0a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:42.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.903+0000 7fb935028640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb91c0a39b0 0x7fb91c0a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb92c006010 tx=0x7fb92c02c7e0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:42.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:42.906+0000 7fb9267fc640 1 -- 172.21.15.7:0/1193411548 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb92809d330 con 0x7fb930107c60 2024-04-21T08:44:43.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:42 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2680971337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:43.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:42 smithi077 ceph-mon[31615]: mgrmap e22: x(active, since 1.01699s), standbys: y, z 2024-04-21T08:44:43.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:42 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3353669853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:43.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:42 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/346373134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:43.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:42 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2680971337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:43.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:42 smithi007 ceph-mon[28477]: mgrmap e22: x(active, since 1.01699s), standbys: y, z 2024-04-21T08:44:43.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:42 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3353669853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:43.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:42 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/346373134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:43.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.161+0000 7fb93728c640 1 -- 172.21.15.7:0/1193411548 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb930062b50 con 0x7fb930107c60 2024-04-21T08:44:43.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.162+0000 7fb9267fc640 1 -- 172.21.15.7:0/1193411548 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7fb92809cc30 con 0x7fb930107c60 2024-04-21T08:44:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:43.163 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:44:43.163 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:44:43.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.164+0000 7fb93728c640 1 -- 172.21.15.7:0/1193411548 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb91c0a39b0 msgr2=0x7fb91c0a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.164+0000 7fb93728c640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb91c0a39b0 0x7fb91c0a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb92c006010 tx=0x7fb92c02c7e0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.164+0000 7fb93728c640 1 -- 172.21.15.7:0/1193411548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb930107c60 msgr2=0x7fb93019f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.164+0000 7fb93728c640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb930107c60 0x7fb93019f780 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb92800eb10 tx=0x7fb92800c2d0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.165+0000 7fb93728c640 1 -- 172.21.15.7:0/1193411548 shutdown_connections 2024-04-21T08:44:43.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.165+0000 7fb93728c640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb930107c60 0x7fb93019f780 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.165+0000 7fb93728c640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb930105330 0x7fb930198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.165+0000 7fb93728c640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb91c0a39b0 0x7fb91c0a5e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.165+0000 7fb93728c640 1 --2- 172.21.15.7:0/1193411548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb930102a00 0x7fb9301981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.165+0000 7fb93728c640 1 -- 172.21.15.7:0/1193411548 >> 172.21.15.7:0/1193411548 conn(0x7fb9300fc3f0 msgr2=0x7fb9301034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:43.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.165+0000 7fb93728c640 1 -- 172.21.15.7:0/1193411548 shutdown_connections 2024-04-21T08:44:43.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.165+0000 7fb93728c640 1 -- 172.21.15.7:0/1193411548 wait complete. 2024-04-21T08:44:43.177 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-21T08:44:43.177 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-21T08:44:43.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:44:43.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.288+0000 7fbf7abdb640 1 -- 172.21.15.7:0/3894651671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf74103180 msgr2=0x7fbf74103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.288+0000 7fbf7abdb640 1 --2- 172.21.15.7:0/3894651671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf74103180 0x7fbf74103600 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009900 tx=0x7fbf6002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:43.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.288+0000 7fbf7abdb640 1 -- 172.21.15.7:0/3894651671 shutdown_connections 2024-04-21T08:44:43.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.288+0000 7fbf7abdb640 1 --2- 172.21.15.7:0/3894651671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf74103b40 0x7fbf7410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.288+0000 7fbf7abdb640 1 --2- 172.21.15.7:0/3894651671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf74103180 0x7fbf74103600 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.288+0000 7fbf7abdb640 1 --2- 172.21.15.7:0/3894651671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf74101f90 0x7fbf741023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.288+0000 7fbf7abdb640 1 -- 172.21.15.7:0/3894651671 >> 172.21.15.7:0/3894651671 conn(0x7fbf740fd520 msgr2=0x7fbf740ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:43.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.288+0000 7fbf7abdb640 1 -- 172.21.15.7:0/3894651671 shutdown_connections 2024-04-21T08:44:43.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.288+0000 7fbf7abdb640 1 -- 172.21.15.7:0/3894651671 wait complete. 2024-04-21T08:44:43.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf7abdb640 1 Processor -- start 2024-04-21T08:44:43.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf7abdb640 1 -- start start 2024-04-21T08:44:43.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf7abdb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf74101f90 0x7fbf7419e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:43.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf7abdb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf74103180 0x7fbf7419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:43.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf7abdb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf74103b40 0x7fbf741a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:43.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf7abdb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7410dad0 con 0x7fbf74103180 2024-04-21T08:44:43.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf7abdb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7410d950 con 0x7fbf74101f90 2024-04-21T08:44:43.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf7abdb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7410dc50 con 0x7fbf74103b40 2024-04-21T08:44:43.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf73fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf74103180 0x7fbf7419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:43.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf73fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf74103180 0x7fbf7419ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53680/0 (socket says 172.21.15.7:53680) 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf79178640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf74103b40 0x7fbf741a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf73fff640 1 -- 172.21.15.7:0/2988698100 learned_addr learned my addr 172.21.15.7:0/2988698100 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.289+0000 7fbf78977640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf74101f90 0x7fbf7419e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf73fff640 1 -- 172.21.15.7:0/2988698100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf74103b40 msgr2=0x7fbf741a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf73fff640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf74103b40 0x7fbf741a5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf73fff640 1 -- 172.21.15.7:0/2988698100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf74101f90 msgr2=0x7fbf7419e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf73fff640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf74101f90 0x7fbf7419e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf73fff640 1 -- 172.21.15.7:0/2988698100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf741a63d0 con 0x7fbf74103180 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf79178640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf74103b40 0x7fbf741a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf78977640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf74101f90 0x7fbf7419e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf73fff640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf74103180 0x7fbf7419ecf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbf60006010 tx=0x7fbf60005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:43.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf71ffb640 1 -- 172.21.15.7:0/2988698100 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf60043020 con 0x7fbf74103180 2024-04-21T08:44:43.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf71ffb640 1 -- 172.21.15.7:0/2988698100 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf600043d0 con 0x7fbf74103180 2024-04-21T08:44:43.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf7abdb640 1 -- 172.21.15.7:0/2988698100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf741a6660 con 0x7fbf74103180 2024-04-21T08:44:43.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf71ffb640 1 -- 172.21.15.7:0/2988698100 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf60003b80 con 0x7fbf74103180 2024-04-21T08:44:43.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.290+0000 7fbf7abdb640 1 -- 172.21.15.7:0/2988698100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf741a6bb0 con 0x7fbf74103180 2024-04-21T08:44:43.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.292+0000 7fbf7abdb640 1 -- 172.21.15.7:0/2988698100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf741a6ed0 con 0x7fbf74103180 2024-04-21T08:44:43.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.293+0000 7fbf71ffb640 1 -- 172.21.15.7:0/2988698100 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fbf60046020 con 0x7fbf74103180 2024-04-21T08:44:43.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.295+0000 7fbf71ffb640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbf640a39b0 0x7fbf640a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:43.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.295+0000 7fbf78977640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbf640a39b0 0x7fbf640a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:43.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.295+0000 7fbf71ffb640 1 -- 172.21.15.7:0/2988698100 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fbf600f26d0 con 0x7fbf74103180 2024-04-21T08:44:43.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.295+0000 7fbf78977640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbf640a39b0 0x7fbf640a5e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbf74102fe0 tx=0x7fbf68005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:43.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.300+0000 7fbf71ffb640 1 -- 172.21.15.7:0/2988698100 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf6004d020 con 0x7fbf74103180 2024-04-21T08:44:43.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.521+0000 7fbf7abdb640 1 -- 172.21.15.7:0/2988698100 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbf740758a0 con 0x7fbf640a39b0 2024-04-21T08:44:43.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.535+0000 7fbf71ffb640 1 -- 172.21.15.7:0/2988698100 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339635 (secure 0 0 0) 0x7fbf740758a0 con 0x7fbf640a39b0 2024-04-21T08:44:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:43.542 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 -- 172.21.15.7:0/2988698100 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbf640a39b0 msgr2=0x7fbf640a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbf640a39b0 0x7fbf640a5e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbf74102fe0 tx=0x7fbf68005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 -- 172.21.15.7:0/2988698100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf74103180 msgr2=0x7fbf7419ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf74103180 0x7fbf7419ecf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbf60006010 tx=0x7fbf60005e40 comp rx=0 tx=0).stop 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 -- 172.21.15.7:0/2988698100 shutdown_connections 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbf640a39b0 0x7fbf640a5e70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf74103b40 0x7fbf741a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf74103180 0x7fbf7419ecf0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 --2- 172.21.15.7:0/2988698100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf74101f90 0x7fbf7419e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 -- 172.21.15.7:0/2988698100 >> 172.21.15.7:0/2988698100 conn(0x7fbf740fd520 msgr2=0x7fbf74074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 -- 172.21.15.7:0/2988698100 shutdown_connections 2024-04-21T08:44:43.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.541+0000 7fbf7abdb640 1 -- 172.21.15.7:0/2988698100 wait complete. 2024-04-21T08:44:43.558 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-21T08:44:43.262682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186284,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":486,"num_read_kb":399,"num_write":384,"num_write_kb":4351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":403472,"kb_used_data":12676,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124538352,"statfs":{"total":1151940427776,"available":1151527272448,"internally_reserved":0,"allocated":12980224,"data_stored":9924555,"data_compressed":544711,"data_compressed_allocated":737280,"data_compressed_original":4325376,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.989634"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375541+0000","last_change":"2024-04-21T08:42:17.545701+0000","last_active":"2024-04-21T08:44:41.375541+0000","last_peered":"2024-04-21T08:44:41.375541+0000","last_clean":"2024-04-21T08:44:41.375541+0000","last_became_active":"2024-04-21T08:42:17.545157+0000","last_became_peered":"2024-04-21T08:42:17.545157+0000","last_unstale":"2024-04-21T08:44:41.375541+0000","last_undegraded":"2024-04-21T08:44:41.375541+0000","last_fullsized":"2024-04-21T08:44:41.375541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:05:41.952879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418165+0000","last_change":"2024-04-21T08:42:17.541153+0000","last_active":"2024-04-21T08:44:41.418165+0000","last_peered":"2024-04-21T08:44:41.418165+0000","last_clean":"2024-04-21T08:44:41.418165+0000","last_became_active":"2024-04-21T08:42:17.540771+0000","last_became_peered":"2024-04-21T08:42:17.540771+0000","last_unstale":"2024-04-21T08:44:41.418165+0000","last_undegraded":"2024-04-21T08:44:41.418165+0000","last_fullsized":"2024-04-21T08:44:41.418165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:35:06.077367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417993+0000","last_change":"2024-04-21T08:42:17.545525+0000","last_active":"2024-04-21T08:44:41.417993+0000","last_peered":"2024-04-21T08:44:41.417993+0000","last_clean":"2024-04-21T08:44:41.417993+0000","last_became_active":"2024-04-21T08:42:17.545168+0000","last_became_peered":"2024-04-21T08:42:17.545168+0000","last_unstale":"2024-04-21T08:44:41.417993+0000","last_undegraded":"2024-04-21T08:44:41.417993+0000","last_fullsized":"2024-04-21T08:44:41.417993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:51.080385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251479+0000","last_change":"2024-04-21T08:42:17.543917+0000","last_active":"2024-04-21T08:44:41.251479+0000","last_peered":"2024-04-21T08:44:41.251479+0000","last_clean":"2024-04-21T08:44:41.251479+0000","last_became_active":"2024-04-21T08:42:17.543513+0000","last_became_peered":"2024-04-21T08:42:17.543513+0000","last_unstale":"2024-04-21T08:44:41.251479+0000","last_undegraded":"2024-04-21T08:44:41.251479+0000","last_fullsized":"2024-04-21T08:44:41.251479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:23.058600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250648+0000","last_change":"2024-04-21T08:42:17.549737+0000","last_active":"2024-04-21T08:44:41.250648+0000","last_peered":"2024-04-21T08:44:41.250648+0000","last_clean":"2024-04-21T08:44:41.250648+0000","last_became_active":"2024-04-21T08:42:17.549396+0000","last_became_peered":"2024-04-21T08:42:17.549396+0000","last_unstale":"2024-04-21T08:44:41.250648+0000","last_undegraded":"2024-04-21T08:44:41.250648+0000","last_fullsized":"2024-04-21T08:44:41.250648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:01:51.879960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368365+0000","last_change":"2024-04-21T08:43:10.792321+0000","last_active":"2024-04-21T08:44:41.368365+0000","last_peered":"2024-04-21T08:44:41.368365+0000","last_clean":"2024-04-21T08:44:41.368365+0000","last_became_active":"2024-04-21T08:43:10.791979+0000","last_became_peered":"2024-04-21T08:43:10.791979+0000","last_unstale":"2024-04-21T08:44:41.368365+0000","last_undegraded":"2024-04-21T08:44:41.368365+0000","last_fullsized":"2024-04-21T08:44:41.368365+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:11:16.056778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374889+0000","last_change":"2024-04-21T08:42:17.541172+0000","last_active":"2024-04-21T08:44:41.374889+0000","last_peered":"2024-04-21T08:44:41.374889+0000","last_clean":"2024-04-21T08:44:41.374889+0000","last_became_active":"2024-04-21T08:42:17.540829+0000","last_became_peered":"2024-04-21T08:42:17.540829+0000","last_unstale":"2024-04-21T08:44:41.374889+0000","last_undegraded":"2024-04-21T08:44:41.374889+0000","last_fullsized":"2024-04-21T08:44:41.374889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:16:12.633980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249753+0000","last_change":"2024-04-21T08:42:17.549744+0000","last_active":"2024-04-21T08:44:41.249753+0000","last_peered":"2024-04-21T08:44:41.249753+0000","last_clean":"2024-04-21T08:44:41.249753+0000","last_became_active":"2024-04-21T08:42:17.549378+0000","last_became_peered":"2024-04-21T08:42:17.549378+0000","last_unstale":"2024-04-21T08:44:41.249753+0000","last_undegraded":"2024-04-21T08:44:41.249753+0000","last_fullsized":"2024-04-21T08:44:41.249753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:03:07.702143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375768+0000","last_change":"2024-04-21T08:42:17.548675+0000","last_active":"2024-04-21T08:44:41.375768+0000","last_peered":"2024-04-21T08:44:41.375768+0000","last_clean":"2024-04-21T08:44:41.375768+0000","last_became_active":"2024-04-21T08:42:17.547809+0000","last_became_peered":"2024-04-21T08:42:17.547809+0000","last_unstale":"2024-04-21T08:44:41.375768+0000","last_undegraded":"2024-04-21T08:44:41.375768+0000","last_fullsized":"2024-04-21T08:44:41.375768+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:23.343473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368771+0000","last_change":"2024-04-21T08:42:17.551012+0000","last_active":"2024-04-21T08:44:41.368771+0000","last_peered":"2024-04-21T08:44:41.368771+0000","last_clean":"2024-04-21T08:44:41.368771+0000","last_became_active":"2024-04-21T08:42:17.550627+0000","last_became_peered":"2024-04-21T08:42:17.550627+0000","last_unstale":"2024-04-21T08:44:41.368771+0000","last_undegraded":"2024-04-21T08:44:41.368771+0000","last_fullsized":"2024-04-21T08:44:41.368771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:57.501588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375100+0000","last_change":"2024-04-21T08:42:17.546192+0000","last_active":"2024-04-21T08:44:41.375100+0000","last_peered":"2024-04-21T08:44:41.375100+0000","last_clean":"2024-04-21T08:44:41.375100+0000","last_became_active":"2024-04-21T08:42:17.545808+0000","last_became_peered":"2024-04-21T08:42:17.545808+0000","last_unstale":"2024-04-21T08:44:41.375100+0000","last_undegraded":"2024-04-21T08:44:41.375100+0000","last_fullsized":"2024-04-21T08:44:41.375100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:54.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251368+0000","last_change":"2024-04-21T08:42:17.553351+0000","last_active":"2024-04-21T08:44:41.251368+0000","last_peered":"2024-04-21T08:44:41.251368+0000","last_clean":"2024-04-21T08:44:41.251368+0000","last_became_active":"2024-04-21T08:42:17.550507+0000","last_became_peered":"2024-04-21T08:42:17.550507+0000","last_unstale":"2024-04-21T08:44:41.251368+0000","last_undegraded":"2024-04-21T08:44:41.251368+0000","last_fullsized":"2024-04-21T08:44:41.251368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:25.561375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376794+0000","last_change":"2024-04-21T08:42:17.540533+0000","last_active":"2024-04-21T08:44:41.376794+0000","last_peered":"2024-04-21T08:44:41.376794+0000","last_clean":"2024-04-21T08:44:41.376794+0000","last_became_active":"2024-04-21T08:42:17.539940+0000","last_became_peered":"2024-04-21T08:42:17.539940+0000","last_unstale":"2024-04-21T08:44:41.376794+0000","last_undegraded":"2024-04-21T08:44:41.376794+0000","last_fullsized":"2024-04-21T08:44:41.376794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:45:19.102899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417676+0000","last_change":"2024-04-21T08:42:17.537794+0000","last_active":"2024-04-21T08:44:41.417676+0000","last_peered":"2024-04-21T08:44:41.417676+0000","last_clean":"2024-04-21T08:44:41.417676+0000","last_became_active":"2024-04-21T08:42:17.537376+0000","last_became_peered":"2024-04-21T08:42:17.537376+0000","last_unstale":"2024-04-21T08:44:41.417676+0000","last_undegraded":"2024-04-21T08:44:41.417676+0000","last_fullsized":"2024-04-21T08:44:41.417676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:57.430036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417515+0000","last_change":"2024-04-21T08:42:17.550574+0000","last_active":"2024-04-21T08:44:41.417515+0000","last_peered":"2024-04-21T08:44:41.417515+0000","last_clean":"2024-04-21T08:44:41.417515+0000","last_became_active":"2024-04-21T08:42:17.550205+0000","last_became_peered":"2024-04-21T08:42:17.550205+0000","last_unstale":"2024-04-21T08:44:41.417515+0000","last_undegraded":"2024-04-21T08:44:41.417515+0000","last_fullsized":"2024-04-21T08:44:41.417515+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:41:08.226477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374544+0000","last_change":"2024-04-21T08:42:17.542239+0000","last_active":"2024-04-21T08:44:41.374544+0000","last_peered":"2024-04-21T08:44:41.374544+0000","last_clean":"2024-04-21T08:44:41.374544+0000","last_became_active":"2024-04-21T08:42:17.541848+0000","last_became_peered":"2024-04-21T08:42:17.541848+0000","last_unstale":"2024-04-21T08:44:41.374544+0000","last_undegraded":"2024-04-21T08:44:41.374544+0000","last_fullsized":"2024-04-21T08:44:41.374544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:47.717451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375417+0000","last_change":"2024-04-21T08:42:17.553200+0000","last_active":"2024-04-21T08:44:41.375417+0000","last_peered":"2024-04-21T08:44:41.375417+0000","last_clean":"2024-04-21T08:44:41.375417+0000","last_became_active":"2024-04-21T08:42:17.552913+0000","last_became_peered":"2024-04-21T08:42:17.552913+0000","last_unstale":"2024-04-21T08:44:41.375417+0000","last_undegraded":"2024-04-21T08:44:41.375417+0000","last_fullsized":"2024-04-21T08:44:41.375417+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:39:04.186928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356904+0000","last_change":"2024-04-21T08:42:17.534881+0000","last_active":"2024-04-21T08:44:41.356904+0000","last_peered":"2024-04-21T08:44:41.356904+0000","last_clean":"2024-04-21T08:44:41.356904+0000","last_became_active":"2024-04-21T08:42:17.534496+0000","last_became_peered":"2024-04-21T08:42:17.534496+0000","last_unstale":"2024-04-21T08:44:41.356904+0000","last_undegraded":"2024-04-21T08:44:41.356904+0000","last_fullsized":"2024-04-21T08:44:41.356904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:50:19.409391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375734+0000","last_change":"2024-04-21T08:42:17.540480+0000","last_active":"2024-04-21T08:44:41.375734+0000","last_peered":"2024-04-21T08:44:41.375734+0000","last_clean":"2024-04-21T08:44:41.375734+0000","last_became_active":"2024-04-21T08:42:17.539742+0000","last_became_peered":"2024-04-21T08:42:17.539742+0000","last_unstale":"2024-04-21T08:44:41.375734+0000","last_undegraded":"2024-04-21T08:44:41.375734+0000","last_fullsized":"2024-04-21T08:44:41.375734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:00.238976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374953+0000","last_change":"2024-04-21T08:44:11.876826+0000","last_active":"2024-04-21T08:44:41.374953+0000","last_peered":"2024-04-21T08:44:41.374953+0000","last_clean":"2024-04-21T08:44:41.374953+0000","last_became_active":"2024-04-21T08:44:11.876354+0000","last_became_peered":"2024-04-21T08:44:11.876354+0000","last_unstale":"2024-04-21T08:44:41.374953+0000","last_undegraded":"2024-04-21T08:44:41.374953+0000","last_fullsized":"2024-04-21T08:44:41.374953+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:00:05.574008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356782+0000","last_change":"2024-04-21T08:42:16.639888+0000","last_active":"2024-04-21T08:44:41.356782+0000","last_peered":"2024-04-21T08:44:41.356782+0000","last_clean":"2024-04-21T08:44:41.356782+0000","last_became_active":"2024-04-21T08:42:16.639403+0000","last_became_peered":"2024-04-21T08:42:16.639403+0000","last_unstale":"2024-04-21T08:44:41.356782+0000","last_undegraded":"2024-04-21T08:44:41.356782+0000","last_fullsized":"2024-04-21T08:44:41.356782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:08:17.164924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249935+0000","last_change":"2024-04-21T08:43:10.983368+0000","last_active":"2024-04-21T08:44:41.249935+0000","last_peered":"2024-04-21T08:44:41.249935+0000","last_clean":"2024-04-21T08:44:41.249935+0000","last_became_active":"2024-04-21T08:43:10.982956+0000","last_became_peered":"2024-04-21T08:43:10.982956+0000","last_unstale":"2024-04-21T08:44:41.249935+0000","last_undegraded":"2024-04-21T08:44:41.249935+0000","last_fullsized":"2024-04-21T08:44:41.249935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:12:44.919996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250725+0000","last_change":"2024-04-21T08:42:21.673197+0000","last_active":"2024-04-21T08:44:41.250725+0000","last_peered":"2024-04-21T08:44:41.250725+0000","last_clean":"2024-04-21T08:44:41.250725+0000","last_became_active":"2024-04-21T08:42:21.672793+0000","last_became_peered":"2024-04-21T08:42:21.672793+0000","last_unstale":"2024-04-21T08:44:41.250725+0000","last_undegraded":"2024-04-21T08:44:41.250725+0000","last_fullsized":"2024-04-21T08:44:41.250725+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:44:29.788459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250990+0000","last_change":"2024-04-21T08:42:16.648577+0000","last_active":"2024-04-21T08:44:41.250990+0000","last_peered":"2024-04-21T08:44:41.250990+0000","last_clean":"2024-04-21T08:44:41.250990+0000","last_became_active":"2024-04-21T08:42:16.646716+0000","last_became_peered":"2024-04-21T08:42:16.646716+0000","last_unstale":"2024-04-21T08:44:41.250990+0000","last_undegraded":"2024-04-21T08:44:41.250990+0000","last_fullsized":"2024-04-21T08:44:41.250990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:10.327078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417936+0000","last_change":"2024-04-21T08:42:17.545430+0000","last_active":"2024-04-21T08:44:41.417936+0000","last_peered":"2024-04-21T08:44:41.417936+0000","last_clean":"2024-04-21T08:44:41.417936+0000","last_became_active":"2024-04-21T08:42:17.545057+0000","last_became_peered":"2024-04-21T08:42:17.545057+0000","last_unstale":"2024-04-21T08:44:41.417936+0000","last_undegraded":"2024-04-21T08:44:41.417936+0000","last_fullsized":"2024-04-21T08:44:41.417936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:44:59.030830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250899+0000","last_change":"2024-04-21T08:42:21.673196+0000","last_active":"2024-04-21T08:44:41.250899+0000","last_peered":"2024-04-21T08:44:41.250899+0000","last_clean":"2024-04-21T08:44:41.250899+0000","last_became_active":"2024-04-21T08:42:21.672808+0000","last_became_peered":"2024-04-21T08:42:21.672808+0000","last_unstale":"2024-04-21T08:44:41.250899+0000","last_undegraded":"2024-04-21T08:44:41.250899+0000","last_fullsized":"2024-04-21T08:44:41.250899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:39.482353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250844+0000","last_change":"2024-04-21T08:42:16.641611+0000","last_active":"2024-04-21T08:44:41.250844+0000","last_peered":"2024-04-21T08:44:41.250844+0000","last_clean":"2024-04-21T08:44:41.250844+0000","last_became_active":"2024-04-21T08:42:16.641221+0000","last_became_peered":"2024-04-21T08:42:16.641221+0000","last_unstale":"2024-04-21T08:44:41.250844+0000","last_undegraded":"2024-04-21T08:44:41.250844+0000","last_fullsized":"2024-04-21T08:44:41.250844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:45.496532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375395+0000","last_change":"2024-04-21T08:42:17.532986+0000","last_active":"2024-04-21T08:44:41.375395+0000","last_peered":"2024-04-21T08:44:41.375395+0000","last_clean":"2024-04-21T08:44:41.375395+0000","last_became_active":"2024-04-21T08:42:17.532583+0000","last_became_peered":"2024-04-21T08:42:17.532583+0000","last_unstale":"2024-04-21T08:44:41.375395+0000","last_undegraded":"2024-04-21T08:44:41.375395+0000","last_fullsized":"2024-04-21T08:44:41.375395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:43.780935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250791+0000","last_change":"2024-04-21T08:42:21.667459+0000","last_active":"2024-04-21T08:44:41.250791+0000","last_peered":"2024-04-21T08:44:41.250791+0000","last_clean":"2024-04-21T08:44:41.250791+0000","last_became_active":"2024-04-21T08:42:21.667183+0000","last_became_peered":"2024-04-21T08:42:21.667183+0000","last_unstale":"2024-04-21T08:44:41.250791+0000","last_undegraded":"2024-04-21T08:44:41.250791+0000","last_fullsized":"2024-04-21T08:44:41.250791+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:32:23.626267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250969+0000","last_change":"2024-04-21T08:42:16.645210+0000","last_active":"2024-04-21T08:44:41.250969+0000","last_peered":"2024-04-21T08:44:41.250969+0000","last_clean":"2024-04-21T08:44:41.250969+0000","last_became_active":"2024-04-21T08:42:16.644747+0000","last_became_peered":"2024-04-21T08:42:16.644747+0000","last_unstale":"2024-04-21T08:44:41.250969+0000","last_undegraded":"2024-04-21T08:44:41.250969+0000","last_fullsized":"2024-04-21T08:44:41.250969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:34:45.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365569+0000","last_change":"2024-04-21T08:43:10.984152+0000","last_active":"2024-04-21T08:44:41.365569+0000","last_peered":"2024-04-21T08:44:41.365569+0000","last_clean":"2024-04-21T08:44:41.365569+0000","last_became_active":"2024-04-21T08:43:10.983736+0000","last_became_peered":"2024-04-21T08:43:10.983736+0000","last_unstale":"2024-04-21T08:44:41.365569+0000","last_undegraded":"2024-04-21T08:44:41.365569+0000","last_fullsized":"2024-04-21T08:44:41.365569+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:10.182667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375391+0000","last_change":"2024-04-21T08:42:21.660739+0000","last_active":"2024-04-21T08:44:41.375391+0000","last_peered":"2024-04-21T08:44:41.375391+0000","last_clean":"2024-04-21T08:44:41.375391+0000","last_became_active":"2024-04-21T08:42:21.660307+0000","last_became_peered":"2024-04-21T08:42:21.660307+0000","last_unstale":"2024-04-21T08:44:41.375391+0000","last_undegraded":"2024-04-21T08:44:41.375391+0000","last_fullsized":"2024-04-21T08:44:41.375391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:26.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365416+0000","last_change":"2024-04-21T08:43:10.986184+0000","last_active":"2024-04-21T08:44:41.365416+0000","last_peered":"2024-04-21T08:44:41.365416+0000","last_clean":"2024-04-21T08:44:41.365416+0000","last_became_active":"2024-04-21T08:43:10.985533+0000","last_became_peered":"2024-04-21T08:43:10.985533+0000","last_unstale":"2024-04-21T08:44:41.365416+0000","last_undegraded":"2024-04-21T08:44:41.365416+0000","last_fullsized":"2024-04-21T08:44:41.365416+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:26.023306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250682+0000","last_change":"2024-04-21T08:42:17.542315+0000","last_active":"2024-04-21T08:44:41.250682+0000","last_peered":"2024-04-21T08:44:41.250682+0000","last_clean":"2024-04-21T08:44:41.250682+0000","last_became_active":"2024-04-21T08:42:17.541895+0000","last_became_peered":"2024-04-21T08:42:17.541895+0000","last_unstale":"2024-04-21T08:44:41.250682+0000","last_undegraded":"2024-04-21T08:44:41.250682+0000","last_fullsized":"2024-04-21T08:44:41.250682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:23:33.909499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375270+0000","last_change":"2024-04-21T08:44:11.878135+0000","last_active":"2024-04-21T08:44:41.375270+0000","last_peered":"2024-04-21T08:44:41.375270+0000","last_clean":"2024-04-21T08:44:41.375270+0000","last_became_active":"2024-04-21T08:44:11.877271+0000","last_became_peered":"2024-04-21T08:44:11.877271+0000","last_unstale":"2024-04-21T08:44:41.375270+0000","last_undegraded":"2024-04-21T08:44:41.375270+0000","last_fullsized":"2024-04-21T08:44:41.375270+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:31.345519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368361+0000","last_change":"2024-04-21T08:43:10.984839+0000","last_active":"2024-04-21T08:44:41.368361+0000","last_peered":"2024-04-21T08:44:41.368361+0000","last_clean":"2024-04-21T08:44:41.368361+0000","last_became_active":"2024-04-21T08:43:10.984382+0000","last_became_peered":"2024-04-21T08:43:10.984382+0000","last_unstale":"2024-04-21T08:44:41.368361+0000","last_undegraded":"2024-04-21T08:44:41.368361+0000","last_fullsized":"2024-04-21T08:44:41.368361+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:47:20.774102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365597+0000","last_change":"2024-04-21T08:42:17.547770+0000","last_active":"2024-04-21T08:44:41.365597+0000","last_peered":"2024-04-21T08:44:41.365597+0000","last_clean":"2024-04-21T08:44:41.365597+0000","last_became_active":"2024-04-21T08:42:17.547440+0000","last_became_peered":"2024-04-21T08:42:17.547440+0000","last_unstale":"2024-04-21T08:44:41.365597+0000","last_undegraded":"2024-04-21T08:44:41.365597+0000","last_fullsized":"2024-04-21T08:44:41.365597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:37:35.555737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375243+0000","last_change":"2024-04-21T08:42:21.670684+0000","last_active":"2024-04-21T08:44:41.375243+0000","last_peered":"2024-04-21T08:44:41.375243+0000","last_clean":"2024-04-21T08:44:41.375243+0000","last_became_active":"2024-04-21T08:42:21.670260+0000","last_became_peered":"2024-04-21T08:42:21.670260+0000","last_unstale":"2024-04-21T08:44:41.375243+0000","last_undegraded":"2024-04-21T08:44:41.375243+0000","last_fullsized":"2024-04-21T08:44:41.375243+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:45:06.192040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376102+0000","last_change":"2024-04-21T08:42:16.639632+0000","last_active":"2024-04-21T08:44:41.376102+0000","last_peered":"2024-04-21T08:44:41.376102+0000","last_clean":"2024-04-21T08:44:41.376102+0000","last_became_active":"2024-04-21T08:42:16.639301+0000","last_became_peered":"2024-04-21T08:42:16.639301+0000","last_unstale":"2024-04-21T08:44:41.376102+0000","last_undegraded":"2024-04-21T08:44:41.376102+0000","last_fullsized":"2024-04-21T08:44:41.376102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:58.838667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249940+0000","last_change":"2024-04-21T08:42:17.546109+0000","last_active":"2024-04-21T08:44:41.249940+0000","last_peered":"2024-04-21T08:44:41.249940+0000","last_clean":"2024-04-21T08:44:41.249940+0000","last_became_active":"2024-04-21T08:42:17.545673+0000","last_became_peered":"2024-04-21T08:42:17.545673+0000","last_unstale":"2024-04-21T08:44:41.249940+0000","last_undegraded":"2024-04-21T08:44:41.249940+0000","last_fullsized":"2024-04-21T08:44:41.249940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:18.947848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417985+0000","last_change":"2024-04-21T08:42:21.666114+0000","last_active":"2024-04-21T08:44:41.417985+0000","last_peered":"2024-04-21T08:44:41.417985+0000","last_clean":"2024-04-21T08:44:41.417985+0000","last_became_active":"2024-04-21T08:42:21.665020+0000","last_became_peered":"2024-04-21T08:42:21.665020+0000","last_unstale":"2024-04-21T08:44:41.417985+0000","last_undegraded":"2024-04-21T08:44:41.417985+0000","last_fullsized":"2024-04-21T08:44:41.417985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:33.644178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365224+0000","last_change":"2024-04-21T08:42:16.639118+0000","last_active":"2024-04-21T08:44:41.365224+0000","last_peered":"2024-04-21T08:44:41.365224+0000","last_clean":"2024-04-21T08:44:41.365224+0000","last_became_active":"2024-04-21T08:42:16.638792+0000","last_became_peered":"2024-04-21T08:42:16.638792+0000","last_unstale":"2024-04-21T08:44:41.365224+0000","last_undegraded":"2024-04-21T08:44:41.365224+0000","last_fullsized":"2024-04-21T08:44:41.365224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:55.568712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356215+0000","last_change":"2024-04-21T08:42:17.539688+0000","last_active":"2024-04-21T08:44:41.356215+0000","last_peered":"2024-04-21T08:44:41.356215+0000","last_clean":"2024-04-21T08:44:41.356215+0000","last_became_active":"2024-04-21T08:42:17.539335+0000","last_became_peered":"2024-04-21T08:42:17.539335+0000","last_unstale":"2024-04-21T08:44:41.356215+0000","last_undegraded":"2024-04-21T08:44:41.356215+0000","last_fullsized":"2024-04-21T08:44:41.356215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:27:51.520232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249882+0000","last_change":"2024-04-21T08:42:21.660570+0000","last_active":"2024-04-21T08:44:41.249882+0000","last_peered":"2024-04-21T08:44:41.249882+0000","last_clean":"2024-04-21T08:44:41.249882+0000","last_became_active":"2024-04-21T08:42:21.659398+0000","last_became_peered":"2024-04-21T08:42:21.659398+0000","last_unstale":"2024-04-21T08:44:41.249882+0000","last_undegraded":"2024-04-21T08:44:41.249882+0000","last_fullsized":"2024-04-21T08:44:41.249882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:55.840334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368786+0000","last_change":"2024-04-21T08:42:16.648192+0000","last_active":"2024-04-21T08:44:41.368786+0000","last_peered":"2024-04-21T08:44:41.368786+0000","last_clean":"2024-04-21T08:44:41.368786+0000","last_became_active":"2024-04-21T08:42:16.647666+0000","last_became_peered":"2024-04-21T08:42:16.647666+0000","last_unstale":"2024-04-21T08:44:41.368786+0000","last_undegraded":"2024-04-21T08:44:41.368786+0000","last_fullsized":"2024-04-21T08:44:41.368786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:33.344680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375708+0000","last_change":"2024-04-21T08:42:17.545925+0000","last_active":"2024-04-21T08:44:41.375708+0000","last_peered":"2024-04-21T08:44:41.375708+0000","last_clean":"2024-04-21T08:44:41.375708+0000","last_became_active":"2024-04-21T08:42:17.545555+0000","last_became_peered":"2024-04-21T08:42:17.545555+0000","last_unstale":"2024-04-21T08:44:41.375708+0000","last_undegraded":"2024-04-21T08:44:41.375708+0000","last_fullsized":"2024-04-21T08:44:41.375708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:13:07.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356087+0000","last_change":"2024-04-21T08:42:21.662757+0000","last_active":"2024-04-21T08:44:41.356087+0000","last_peered":"2024-04-21T08:44:41.356087+0000","last_clean":"2024-04-21T08:44:41.356087+0000","last_became_active":"2024-04-21T08:42:21.662306+0000","last_became_peered":"2024-04-21T08:42:21.662306+0000","last_unstale":"2024-04-21T08:44:41.356087+0000","last_undegraded":"2024-04-21T08:44:41.356087+0000","last_fullsized":"2024-04-21T08:44:41.356087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:26.856147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375135+0000","last_change":"2024-04-21T08:43:10.984690+0000","last_active":"2024-04-21T08:44:41.375135+0000","last_peered":"2024-04-21T08:44:41.375135+0000","last_clean":"2024-04-21T08:44:41.375135+0000","last_became_active":"2024-04-21T08:43:10.984290+0000","last_became_peered":"2024-04-21T08:43:10.984290+0000","last_unstale":"2024-04-21T08:44:41.375135+0000","last_undegraded":"2024-04-21T08:44:41.375135+0000","last_fullsized":"2024-04-21T08:44:41.375135+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:37:36.400946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418211+0000","last_change":"2024-04-21T08:43:10.793316+0000","last_active":"2024-04-21T08:44:41.418211+0000","last_peered":"2024-04-21T08:44:41.418211+0000","last_clean":"2024-04-21T08:44:41.418211+0000","last_became_active":"2024-04-21T08:43:10.793035+0000","last_became_peered":"2024-04-21T08:43:10.793035+0000","last_unstale":"2024-04-21T08:44:41.418211+0000","last_undegraded":"2024-04-21T08:44:41.418211+0000","last_fullsized":"2024-04-21T08:44:41.418211+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:41:04.247385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251083+0000","last_change":"2024-04-21T08:42:21.670561+0000","last_active":"2024-04-21T08:44:41.251083+0000","last_peered":"2024-04-21T08:44:41.251083+0000","last_clean":"2024-04-21T08:44:41.251083+0000","last_became_active":"2024-04-21T08:42:21.670170+0000","last_became_peered":"2024-04-21T08:42:21.670170+0000","last_unstale":"2024-04-21T08:44:41.251083+0000","last_undegraded":"2024-04-21T08:44:41.251083+0000","last_fullsized":"2024-04-21T08:44:41.251083+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:35:54.538013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365460+0000","last_change":"2024-04-21T08:42:16.637319+0000","last_active":"2024-04-21T08:44:41.365460+0000","last_peered":"2024-04-21T08:44:41.365460+0000","last_clean":"2024-04-21T08:44:41.365460+0000","last_became_active":"2024-04-21T08:42:16.636940+0000","last_became_peered":"2024-04-21T08:42:16.636940+0000","last_unstale":"2024-04-21T08:44:41.365460+0000","last_undegraded":"2024-04-21T08:44:41.365460+0000","last_fullsized":"2024-04-21T08:44:41.365460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:59:09.826786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375697+0000","last_change":"2024-04-21T08:42:17.546202+0000","last_active":"2024-04-21T08:44:41.375697+0000","last_peered":"2024-04-21T08:44:41.375697+0000","last_clean":"2024-04-21T08:44:41.375697+0000","last_became_active":"2024-04-21T08:42:17.545781+0000","last_became_peered":"2024-04-21T08:42:17.545781+0000","last_unstale":"2024-04-21T08:44:41.375697+0000","last_undegraded":"2024-04-21T08:44:41.375697+0000","last_fullsized":"2024-04-21T08:44:41.375697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:23.990951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356160+0000","last_change":"2024-04-21T08:42:21.662822+0000","last_active":"2024-04-21T08:44:41.356160+0000","last_peered":"2024-04-21T08:44:41.356160+0000","last_clean":"2024-04-21T08:44:41.356160+0000","last_became_active":"2024-04-21T08:42:21.662398+0000","last_became_peered":"2024-04-21T08:42:21.662398+0000","last_unstale":"2024-04-21T08:44:41.356160+0000","last_undegraded":"2024-04-21T08:44:41.356160+0000","last_fullsized":"2024-04-21T08:44:41.356160+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:38:40.499294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":324,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:42.133017+0000","last_change":"2024-04-21T08:42:16.636330+0000","last_active":"2024-04-21T08:44:42.133017+0000","last_peered":"2024-04-21T08:44:42.133017+0000","last_clean":"2024-04-21T08:44:42.133017+0000","last_became_active":"2024-04-21T08:42:16.635762+0000","last_became_peered":"2024-04-21T08:42:16.635762+0000","last_unstale":"2024-04-21T08:44:42.133017+0000","last_undegraded":"2024-04-21T08:44:42.133017+0000","last_fullsized":"2024-04-21T08:44:42.133017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:31.137923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":195,"num_write":38,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251043+0000","last_change":"2024-04-21T08:42:17.543033+0000","last_active":"2024-04-21T08:44:41.251043+0000","last_peered":"2024-04-21T08:44:41.251043+0000","last_clean":"2024-04-21T08:44:41.251043+0000","last_became_active":"2024-04-21T08:42:17.542692+0000","last_became_peered":"2024-04-21T08:42:17.542692+0000","last_unstale":"2024-04-21T08:44:41.251043+0000","last_undegraded":"2024-04-21T08:44:41.251043+0000","last_fullsized":"2024-04-21T08:44:41.251043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:33:32.111999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365780+0000","last_change":"2024-04-21T08:42:21.668831+0000","last_active":"2024-04-21T08:44:41.365780+0000","last_peered":"2024-04-21T08:44:41.365780+0000","last_clean":"2024-04-21T08:44:41.365780+0000","last_became_active":"2024-04-21T08:42:21.668485+0000","last_became_peered":"2024-04-21T08:42:21.668485+0000","last_unstale":"2024-04-21T08:44:41.365780+0000","last_undegraded":"2024-04-21T08:44:41.365780+0000","last_fullsized":"2024-04-21T08:44:41.365780+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:49:29.049707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375241+0000","last_change":"2024-04-21T08:42:16.640288+0000","last_active":"2024-04-21T08:44:41.375241+0000","last_peered":"2024-04-21T08:44:41.375241+0000","last_clean":"2024-04-21T08:44:41.375241+0000","last_became_active":"2024-04-21T08:42:16.639909+0000","last_became_peered":"2024-04-21T08:42:16.639909+0000","last_unstale":"2024-04-21T08:44:41.375241+0000","last_undegraded":"2024-04-21T08:44:41.375241+0000","last_fullsized":"2024-04-21T08:44:41.375241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:33.839442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375479+0000","last_change":"2024-04-21T08:42:17.541154+0000","last_active":"2024-04-21T08:44:41.375479+0000","last_peered":"2024-04-21T08:44:41.375479+0000","last_clean":"2024-04-21T08:44:41.375479+0000","last_became_active":"2024-04-21T08:42:17.540747+0000","last_became_peered":"2024-04-21T08:42:17.540747+0000","last_unstale":"2024-04-21T08:44:41.375479+0000","last_undegraded":"2024-04-21T08:44:41.375479+0000","last_fullsized":"2024-04-21T08:44:41.375479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:14:50.641144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374983+0000","last_change":"2024-04-21T08:42:21.670767+0000","last_active":"2024-04-21T08:44:41.374983+0000","last_peered":"2024-04-21T08:44:41.374983+0000","last_clean":"2024-04-21T08:44:41.374983+0000","last_became_active":"2024-04-21T08:42:21.670338+0000","last_became_peered":"2024-04-21T08:42:21.670338+0000","last_unstale":"2024-04-21T08:44:41.374983+0000","last_undegraded":"2024-04-21T08:44:41.374983+0000","last_fullsized":"2024-04-21T08:44:41.374983+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:38:55.275603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365468+0000","last_change":"2024-04-21T08:42:16.640144+0000","last_active":"2024-04-21T08:44:41.365468+0000","last_peered":"2024-04-21T08:44:41.365468+0000","last_clean":"2024-04-21T08:44:41.365468+0000","last_became_active":"2024-04-21T08:42:16.639707+0000","last_became_peered":"2024-04-21T08:42:16.639707+0000","last_unstale":"2024-04-21T08:44:41.365468+0000","last_undegraded":"2024-04-21T08:44:41.365468+0000","last_fullsized":"2024-04-21T08:44:41.365468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:11:32.785519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375240+0000","last_change":"2024-04-21T08:42:17.555019+0000","last_active":"2024-04-21T08:44:41.375240+0000","last_peered":"2024-04-21T08:44:41.375240+0000","last_clean":"2024-04-21T08:44:41.375240+0000","last_became_active":"2024-04-21T08:42:17.554645+0000","last_became_peered":"2024-04-21T08:42:17.554645+0000","last_unstale":"2024-04-21T08:44:41.375240+0000","last_undegraded":"2024-04-21T08:44:41.375240+0000","last_fullsized":"2024-04-21T08:44:41.375240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:42:30.353945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251223+0000","last_change":"2024-04-21T08:44:11.876740+0000","last_active":"2024-04-21T08:44:41.251223+0000","last_peered":"2024-04-21T08:44:41.251223+0000","last_clean":"2024-04-21T08:44:41.251223+0000","last_became_active":"2024-04-21T08:44:11.876108+0000","last_became_peered":"2024-04-21T08:44:11.876108+0000","last_unstale":"2024-04-21T08:44:41.251223+0000","last_undegraded":"2024-04-21T08:44:41.251223+0000","last_fullsized":"2024-04-21T08:44:41.251223+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:37.398743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250912+0000","last_change":"2024-04-21T08:42:16.640407+0000","last_active":"2024-04-21T08:44:41.250912+0000","last_peered":"2024-04-21T08:44:41.250912+0000","last_clean":"2024-04-21T08:44:41.250912+0000","last_became_active":"2024-04-21T08:42:16.640015+0000","last_became_peered":"2024-04-21T08:42:16.640015+0000","last_unstale":"2024-04-21T08:44:41.250912+0000","last_undegraded":"2024-04-21T08:44:41.250912+0000","last_fullsized":"2024-04-21T08:44:41.250912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:23:10.647184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374762+0000","last_change":"2024-04-21T08:42:17.545438+0000","last_active":"2024-04-21T08:44:41.374762+0000","last_peered":"2024-04-21T08:44:41.374762+0000","last_clean":"2024-04-21T08:44:41.374762+0000","last_became_active":"2024-04-21T08:42:17.544258+0000","last_became_peered":"2024-04-21T08:42:17.544258+0000","last_unstale":"2024-04-21T08:44:41.374762+0000","last_undegraded":"2024-04-21T08:44:41.374762+0000","last_fullsized":"2024-04-21T08:44:41.374762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:43.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376385+0000","last_change":"2024-04-21T08:42:21.671000+0000","last_active":"2024-04-21T08:44:41.376385+0000","last_peered":"2024-04-21T08:44:41.376385+0000","last_clean":"2024-04-21T08:44:41.376385+0000","last_became_active":"2024-04-21T08:42:21.670380+0000","last_became_peered":"2024-04-21T08:42:21.670380+0000","last_unstale":"2024-04-21T08:44:41.376385+0000","last_undegraded":"2024-04-21T08:44:41.376385+0000","last_fullsized":"2024-04-21T08:44:41.376385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:57.075686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250621+0000","last_change":"2024-04-21T08:42:16.649599+0000","last_active":"2024-04-21T08:44:41.250621+0000","last_peered":"2024-04-21T08:44:41.250621+0000","last_clean":"2024-04-21T08:44:41.250621+0000","last_became_active":"2024-04-21T08:42:16.649232+0000","last_became_peered":"2024-04-21T08:42:16.649232+0000","last_unstale":"2024-04-21T08:44:41.250621+0000","last_undegraded":"2024-04-21T08:44:41.250621+0000","last_fullsized":"2024-04-21T08:44:41.250621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:21.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356045+0000","last_change":"2024-04-21T08:44:11.776138+0000","last_active":"2024-04-21T08:44:41.356045+0000","last_peered":"2024-04-21T08:44:41.356045+0000","last_clean":"2024-04-21T08:44:41.356045+0000","last_became_active":"2024-04-21T08:44:11.775832+0000","last_became_peered":"2024-04-21T08:44:11.775832+0000","last_unstale":"2024-04-21T08:44:41.356045+0000","last_undegraded":"2024-04-21T08:44:41.356045+0000","last_fullsized":"2024-04-21T08:44:41.356045+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:03:34.210923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365172+0000","last_change":"2024-04-21T08:42:21.664811+0000","last_active":"2024-04-21T08:44:41.365172+0000","last_peered":"2024-04-21T08:44:41.365172+0000","last_clean":"2024-04-21T08:44:41.365172+0000","last_became_active":"2024-04-21T08:42:21.664332+0000","last_became_peered":"2024-04-21T08:42:21.664332+0000","last_unstale":"2024-04-21T08:44:41.365172+0000","last_undegraded":"2024-04-21T08:44:41.365172+0000","last_fullsized":"2024-04-21T08:44:41.365172+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:40.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251372+0000","last_change":"2024-04-21T08:42:16.641233+0000","last_active":"2024-04-21T08:44:41.251372+0000","last_peered":"2024-04-21T08:44:41.251372+0000","last_clean":"2024-04-21T08:44:41.251372+0000","last_became_active":"2024-04-21T08:42:16.640873+0000","last_became_peered":"2024-04-21T08:42:16.640873+0000","last_unstale":"2024-04-21T08:44:41.251372+0000","last_undegraded":"2024-04-21T08:44:41.251372+0000","last_fullsized":"2024-04-21T08:44:41.251372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:00.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375293+0000","last_change":"2024-04-21T08:42:17.550438+0000","last_active":"2024-04-21T08:44:41.375293+0000","last_peered":"2024-04-21T08:44:41.375293+0000","last_clean":"2024-04-21T08:44:41.375293+0000","last_became_active":"2024-04-21T08:42:17.550015+0000","last_became_peered":"2024-04-21T08:42:17.550015+0000","last_unstale":"2024-04-21T08:44:41.375293+0000","last_undegraded":"2024-04-21T08:44:41.375293+0000","last_fullsized":"2024-04-21T08:44:41.375293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:00:29.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250709+0000","last_change":"2024-04-21T08:42:21.970498+0000","last_active":"2024-04-21T08:44:41.250709+0000","last_peered":"2024-04-21T08:44:41.250709+0000","last_clean":"2024-04-21T08:44:41.250709+0000","last_became_active":"2024-04-21T08:42:21.969819+0000","last_became_peered":"2024-04-21T08:42:21.969819+0000","last_unstale":"2024-04-21T08:44:41.250709+0000","last_undegraded":"2024-04-21T08:44:41.250709+0000","last_fullsized":"2024-04-21T08:44:41.250709+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:13:16.552293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356527+0000","last_change":"2024-04-21T08:42:16.637146+0000","last_active":"2024-04-21T08:44:41.356527+0000","last_peered":"2024-04-21T08:44:41.356527+0000","last_clean":"2024-04-21T08:44:41.356527+0000","last_became_active":"2024-04-21T08:42:16.636781+0000","last_became_peered":"2024-04-21T08:42:16.636781+0000","last_unstale":"2024-04-21T08:44:41.356527+0000","last_undegraded":"2024-04-21T08:44:41.356527+0000","last_fullsized":"2024-04-21T08:44:41.356527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:39:27.641236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368622+0000","last_change":"2024-04-21T08:42:17.542325+0000","last_active":"2024-04-21T08:44:41.368622+0000","last_peered":"2024-04-21T08:44:41.368622+0000","last_clean":"2024-04-21T08:44:41.368622+0000","last_became_active":"2024-04-21T08:42:17.541103+0000","last_became_peered":"2024-04-21T08:42:17.541103+0000","last_unstale":"2024-04-21T08:44:41.368622+0000","last_undegraded":"2024-04-21T08:44:41.368622+0000","last_fullsized":"2024-04-21T08:44:41.368622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:46.903250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375702+0000","last_change":"2024-04-21T08:42:21.671146+0000","last_active":"2024-04-21T08:44:41.375702+0000","last_peered":"2024-04-21T08:44:41.375702+0000","last_clean":"2024-04-21T08:44:41.375702+0000","last_became_active":"2024-04-21T08:42:21.670561+0000","last_became_peered":"2024-04-21T08:42:21.670561+0000","last_unstale":"2024-04-21T08:44:41.375702+0000","last_undegraded":"2024-04-21T08:44:41.375702+0000","last_fullsized":"2024-04-21T08:44:41.375702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:38:28.028976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375066+0000","last_change":"2024-04-21T08:42:16.645132+0000","last_active":"2024-04-21T08:44:41.375066+0000","last_peered":"2024-04-21T08:44:41.375066+0000","last_clean":"2024-04-21T08:44:41.375066+0000","last_became_active":"2024-04-21T08:42:16.644786+0000","last_became_peered":"2024-04-21T08:42:16.644786+0000","last_unstale":"2024-04-21T08:44:41.375066+0000","last_undegraded":"2024-04-21T08:44:41.375066+0000","last_fullsized":"2024-04-21T08:44:41.375066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:08.298498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368567+0000","last_change":"2024-04-21T08:42:17.550424+0000","last_active":"2024-04-21T08:44:41.368567+0000","last_peered":"2024-04-21T08:44:41.368567+0000","last_clean":"2024-04-21T08:44:41.368567+0000","last_became_active":"2024-04-21T08:42:17.550077+0000","last_became_peered":"2024-04-21T08:42:17.550077+0000","last_unstale":"2024-04-21T08:44:41.368567+0000","last_undegraded":"2024-04-21T08:44:41.368567+0000","last_fullsized":"2024-04-21T08:44:41.368567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:17:07.016204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375856+0000","last_change":"2024-04-21T08:42:21.970652+0000","last_active":"2024-04-21T08:44:41.375856+0000","last_peered":"2024-04-21T08:44:41.375856+0000","last_clean":"2024-04-21T08:44:41.375856+0000","last_became_active":"2024-04-21T08:42:21.970251+0000","last_became_peered":"2024-04-21T08:42:21.970251+0000","last_unstale":"2024-04-21T08:44:41.375856+0000","last_undegraded":"2024-04-21T08:44:41.375856+0000","last_fullsized":"2024-04-21T08:44:41.375856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:45.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365285+0000","last_change":"2024-04-21T08:42:17.542820+0000","last_active":"2024-04-21T08:44:41.365285+0000","last_peered":"2024-04-21T08:44:41.365285+0000","last_clean":"2024-04-21T08:44:41.365285+0000","last_became_active":"2024-04-21T08:42:17.541320+0000","last_became_peered":"2024-04-21T08:42:17.541320+0000","last_unstale":"2024-04-21T08:44:41.365285+0000","last_undegraded":"2024-04-21T08:44:41.365285+0000","last_fullsized":"2024-04-21T08:44:41.365285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:23:06.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375209+0000","last_change":"2024-04-21T08:42:16.636950+0000","last_active":"2024-04-21T08:44:41.375209+0000","last_peered":"2024-04-21T08:44:41.375209+0000","last_clean":"2024-04-21T08:44:41.375209+0000","last_became_active":"2024-04-21T08:42:16.636590+0000","last_became_peered":"2024-04-21T08:42:16.636590+0000","last_unstale":"2024-04-21T08:44:41.375209+0000","last_undegraded":"2024-04-21T08:44:41.375209+0000","last_fullsized":"2024-04-21T08:44:41.375209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:55.693681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374884+0000","last_change":"2024-04-21T08:42:21.660404+0000","last_active":"2024-04-21T08:44:41.374884+0000","last_peered":"2024-04-21T08:44:41.374884+0000","last_clean":"2024-04-21T08:44:41.374884+0000","last_became_active":"2024-04-21T08:42:21.659999+0000","last_became_peered":"2024-04-21T08:42:21.659999+0000","last_unstale":"2024-04-21T08:44:41.374884+0000","last_undegraded":"2024-04-21T08:44:41.374884+0000","last_fullsized":"2024-04-21T08:44:41.374884+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:53:32.328873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356239+0000","last_change":"2024-04-21T08:43:10.984361+0000","last_active":"2024-04-21T08:44:41.356239+0000","last_peered":"2024-04-21T08:44:41.356239+0000","last_clean":"2024-04-21T08:44:41.356239+0000","last_became_active":"2024-04-21T08:43:10.984046+0000","last_became_peered":"2024-04-21T08:43:10.984046+0000","last_unstale":"2024-04-21T08:44:41.356239+0000","last_undegraded":"2024-04-21T08:44:41.356239+0000","last_fullsized":"2024-04-21T08:44:41.356239+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:33:23.154630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251236+0000","last_change":"2024-04-21T08:42:16.638993+0000","last_active":"2024-04-21T08:44:41.251236+0000","last_peered":"2024-04-21T08:44:41.251236+0000","last_clean":"2024-04-21T08:44:41.251236+0000","last_became_active":"2024-04-21T08:42:16.638617+0000","last_became_peered":"2024-04-21T08:42:16.638617+0000","last_unstale":"2024-04-21T08:44:41.251236+0000","last_undegraded":"2024-04-21T08:44:41.251236+0000","last_fullsized":"2024-04-21T08:44:41.251236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:14:28.271433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374510+0000","last_change":"2024-04-21T08:41:00.044984+0000","last_active":"2024-04-21T08:44:41.374510+0000","last_peered":"2024-04-21T08:44:41.374510+0000","last_clean":"2024-04-21T08:44:41.374510+0000","last_became_active":"2024-04-21T08:41:00.044485+0000","last_became_peered":"2024-04-21T08:41:00.044485+0000","last_unstale":"2024-04-21T08:44:41.374510+0000","last_undegraded":"2024-04-21T08:44:41.374510+0000","last_fullsized":"2024-04-21T08:44:41.374510+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.370031+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:01.031209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365508+0000","last_change":"2024-04-21T08:42:21.971045+0000","last_active":"2024-04-21T08:44:41.365508+0000","last_peered":"2024-04-21T08:44:41.365508+0000","last_clean":"2024-04-21T08:44:41.365508+0000","last_became_active":"2024-04-21T08:42:21.970577+0000","last_became_peered":"2024-04-21T08:42:21.970577+0000","last_unstale":"2024-04-21T08:44:41.365508+0000","last_undegraded":"2024-04-21T08:44:41.365508+0000","last_fullsized":"2024-04-21T08:44:41.365508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:55.681662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374964+0000","last_change":"2024-04-21T08:42:17.535570+0000","last_active":"2024-04-21T08:44:41.374964+0000","last_peered":"2024-04-21T08:44:41.374964+0000","last_clean":"2024-04-21T08:44:41.374964+0000","last_became_active":"2024-04-21T08:42:17.535217+0000","last_became_peered":"2024-04-21T08:42:17.535217+0000","last_unstale":"2024-04-21T08:44:41.374964+0000","last_undegraded":"2024-04-21T08:44:41.374964+0000","last_fullsized":"2024-04-21T08:44:41.374964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:49.510750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374650+0000","last_change":"2024-04-21T08:43:10.792548+0000","last_active":"2024-04-21T08:44:41.374650+0000","last_peered":"2024-04-21T08:44:41.374650+0000","last_clean":"2024-04-21T08:44:41.374650+0000","last_became_active":"2024-04-21T08:43:10.792165+0000","last_became_peered":"2024-04-21T08:43:10.792165+0000","last_unstale":"2024-04-21T08:44:41.374650+0000","last_undegraded":"2024-04-21T08:44:41.374650+0000","last_fullsized":"2024-04-21T08:44:41.374650+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:35.980536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374872+0000","last_change":"2024-04-21T08:42:21.661393+0000","last_active":"2024-04-21T08:44:41.374872+0000","last_peered":"2024-04-21T08:44:41.374872+0000","last_clean":"2024-04-21T08:44:41.374872+0000","last_became_active":"2024-04-21T08:42:21.660201+0000","last_became_peered":"2024-04-21T08:42:21.660201+0000","last_unstale":"2024-04-21T08:44:41.374872+0000","last_undegraded":"2024-04-21T08:44:41.374872+0000","last_fullsized":"2024-04-21T08:44:41.374872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:13.393389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250941+0000","last_change":"2024-04-21T08:42:16.650507+0000","last_active":"2024-04-21T08:44:41.250941+0000","last_peered":"2024-04-21T08:44:41.250941+0000","last_clean":"2024-04-21T08:44:41.250941+0000","last_became_active":"2024-04-21T08:42:16.650217+0000","last_became_peered":"2024-04-21T08:42:16.650217+0000","last_unstale":"2024-04-21T08:44:41.250941+0000","last_undegraded":"2024-04-21T08:44:41.250941+0000","last_fullsized":"2024-04-21T08:44:41.250941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:13:26.785092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375997+0000","last_change":"2024-04-21T08:42:17.530420+0000","last_active":"2024-04-21T08:44:41.375997+0000","last_peered":"2024-04-21T08:44:41.375997+0000","last_clean":"2024-04-21T08:44:41.375997+0000","last_became_active":"2024-04-21T08:42:17.529920+0000","last_became_peered":"2024-04-21T08:42:17.529920+0000","last_unstale":"2024-04-21T08:44:41.375997+0000","last_undegraded":"2024-04-21T08:44:41.375997+0000","last_fullsized":"2024-04-21T08:44:41.375997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:24.250197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249760+0000","last_change":"2024-04-21T08:42:21.970586+0000","last_active":"2024-04-21T08:44:41.249760+0000","last_peered":"2024-04-21T08:44:41.249760+0000","last_clean":"2024-04-21T08:44:41.249760+0000","last_became_active":"2024-04-21T08:42:21.970132+0000","last_became_peered":"2024-04-21T08:42:21.970132+0000","last_unstale":"2024-04-21T08:44:41.249760+0000","last_undegraded":"2024-04-21T08:44:41.249760+0000","last_fullsized":"2024-04-21T08:44:41.249760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:11.074370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'43","reported_seq":100,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.483501+0000","last_change":"2024-04-21T08:42:16.647916+0000","last_active":"2024-04-21T08:44:41.483501+0000","last_peered":"2024-04-21T08:44:41.483501+0000","last_clean":"2024-04-21T08:44:41.483501+0000","last_became_active":"2024-04-21T08:42:16.647476+0000","last_became_peered":"2024-04-21T08:42:16.647476+0000","last_unstale":"2024-04-21T08:44:41.483501+0000","last_undegraded":"2024-04-21T08:44:41.483501+0000","last_fullsized":"2024-04-21T08:44:41.483501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:27.242153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251070+0000","last_change":"2024-04-21T08:42:17.546144+0000","last_active":"2024-04-21T08:44:41.251070+0000","last_peered":"2024-04-21T08:44:41.251070+0000","last_clean":"2024-04-21T08:44:41.251070+0000","last_became_active":"2024-04-21T08:42:17.545730+0000","last_became_peered":"2024-04-21T08:42:17.545730+0000","last_unstale":"2024-04-21T08:44:41.251070+0000","last_undegraded":"2024-04-21T08:44:41.251070+0000","last_fullsized":"2024-04-21T08:44:41.251070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:42.802309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374618+0000","last_change":"2024-04-21T08:42:21.655802+0000","last_active":"2024-04-21T08:44:41.374618+0000","last_peered":"2024-04-21T08:44:41.374618+0000","last_clean":"2024-04-21T08:44:41.374618+0000","last_became_active":"2024-04-21T08:42:21.655379+0000","last_became_peered":"2024-04-21T08:42:21.655379+0000","last_unstale":"2024-04-21T08:44:41.374618+0000","last_undegraded":"2024-04-21T08:44:41.374618+0000","last_fullsized":"2024-04-21T08:44:41.374618+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:49.486057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251462+0000","last_change":"2024-04-21T08:42:16.642517+0000","last_active":"2024-04-21T08:44:41.251462+0000","last_peered":"2024-04-21T08:44:41.251462+0000","last_clean":"2024-04-21T08:44:41.251462+0000","last_became_active":"2024-04-21T08:42:16.641539+0000","last_became_peered":"2024-04-21T08:42:16.641539+0000","last_unstale":"2024-04-21T08:44:41.251462+0000","last_undegraded":"2024-04-21T08:44:41.251462+0000","last_fullsized":"2024-04-21T08:44:41.251462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:53.912199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418089+0000","last_change":"2024-04-21T08:42:17.550172+0000","last_active":"2024-04-21T08:44:41.418089+0000","last_peered":"2024-04-21T08:44:41.418089+0000","last_clean":"2024-04-21T08:44:41.418089+0000","last_became_active":"2024-04-21T08:42:17.549824+0000","last_became_peered":"2024-04-21T08:42:17.549824+0000","last_unstale":"2024-04-21T08:44:41.418089+0000","last_undegraded":"2024-04-21T08:44:41.418089+0000","last_fullsized":"2024-04-21T08:44:41.418089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:59.553645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251154+0000","last_change":"2024-04-21T08:44:11.774778+0000","last_active":"2024-04-21T08:44:41.251154+0000","last_peered":"2024-04-21T08:44:41.251154+0000","last_clean":"2024-04-21T08:44:41.251154+0000","last_became_active":"2024-04-21T08:44:11.774190+0000","last_became_peered":"2024-04-21T08:44:11.774190+0000","last_unstale":"2024-04-21T08:44:41.251154+0000","last_undegraded":"2024-04-21T08:44:41.251154+0000","last_fullsized":"2024-04-21T08:44:41.251154+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:06.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374515+0000","last_change":"2024-04-21T08:42:16.647985+0000","last_active":"2024-04-21T08:44:41.374515+0000","last_peered":"2024-04-21T08:44:41.374515+0000","last_clean":"2024-04-21T08:44:41.374515+0000","last_became_active":"2024-04-21T08:42:16.647577+0000","last_became_peered":"2024-04-21T08:42:16.647577+0000","last_unstale":"2024-04-21T08:44:41.374515+0000","last_undegraded":"2024-04-21T08:44:41.374515+0000","last_fullsized":"2024-04-21T08:44:41.374515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:00.731708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250576+0000","last_change":"2024-04-21T08:42:17.541757+0000","last_active":"2024-04-21T08:44:41.250576+0000","last_peered":"2024-04-21T08:44:41.250576+0000","last_clean":"2024-04-21T08:44:41.250576+0000","last_became_active":"2024-04-21T08:42:17.539280+0000","last_became_peered":"2024-04-21T08:42:17.539280+0000","last_unstale":"2024-04-21T08:44:41.250576+0000","last_undegraded":"2024-04-21T08:44:41.250576+0000","last_fullsized":"2024-04-21T08:44:41.250576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:45:54.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356961+0000","last_change":"2024-04-21T08:42:21.664566+0000","last_active":"2024-04-21T08:44:41.356961+0000","last_peered":"2024-04-21T08:44:41.356961+0000","last_clean":"2024-04-21T08:44:41.356961+0000","last_became_active":"2024-04-21T08:42:21.664136+0000","last_became_peered":"2024-04-21T08:42:21.664136+0000","last_unstale":"2024-04-21T08:44:41.356961+0000","last_undegraded":"2024-04-21T08:44:41.356961+0000","last_fullsized":"2024-04-21T08:44:41.356961+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:19.856577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249918+0000","last_change":"2024-04-21T08:42:16.650664+0000","last_active":"2024-04-21T08:44:41.249918+0000","last_peered":"2024-04-21T08:44:41.249918+0000","last_clean":"2024-04-21T08:44:41.249918+0000","last_became_active":"2024-04-21T08:42:16.650339+0000","last_became_peered":"2024-04-21T08:42:16.650339+0000","last_unstale":"2024-04-21T08:44:41.249918+0000","last_undegraded":"2024-04-21T08:44:41.249918+0000","last_fullsized":"2024-04-21T08:44:41.249918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:59.565455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374844+0000","last_change":"2024-04-21T08:42:17.538142+0000","last_active":"2024-04-21T08:44:41.374844+0000","last_peered":"2024-04-21T08:44:41.374844+0000","last_clean":"2024-04-21T08:44:41.374844+0000","last_became_active":"2024-04-21T08:42:17.537793+0000","last_became_peered":"2024-04-21T08:42:17.537793+0000","last_unstale":"2024-04-21T08:44:41.374844+0000","last_undegraded":"2024-04-21T08:44:41.374844+0000","last_fullsized":"2024-04-21T08:44:41.374844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:11.059294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251170+0000","last_change":"2024-04-21T08:42:21.670642+0000","last_active":"2024-04-21T08:44:41.251170+0000","last_peered":"2024-04-21T08:44:41.251170+0000","last_clean":"2024-04-21T08:44:41.251170+0000","last_became_active":"2024-04-21T08:42:21.670128+0000","last_became_peered":"2024-04-21T08:42:21.670128+0000","last_unstale":"2024-04-21T08:44:41.251170+0000","last_undegraded":"2024-04-21T08:44:41.251170+0000","last_fullsized":"2024-04-21T08:44:41.251170+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:51:28.604381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375278+0000","last_change":"2024-04-21T08:43:10.792114+0000","last_active":"2024-04-21T08:44:41.375278+0000","last_peered":"2024-04-21T08:44:41.375278+0000","last_clean":"2024-04-21T08:44:41.375278+0000","last_became_active":"2024-04-21T08:43:10.791664+0000","last_became_peered":"2024-04-21T08:43:10.791664+0000","last_unstale":"2024-04-21T08:44:41.375278+0000","last_undegraded":"2024-04-21T08:44:41.375278+0000","last_fullsized":"2024-04-21T08:44:41.375278+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:07.317804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250083+0000","last_change":"2024-04-21T08:42:17.547326+0000","last_active":"2024-04-21T08:44:41.250083+0000","last_peered":"2024-04-21T08:44:41.250083+0000","last_clean":"2024-04-21T08:44:41.250083+0000","last_became_active":"2024-04-21T08:42:17.546920+0000","last_became_peered":"2024-04-21T08:42:17.546920+0000","last_unstale":"2024-04-21T08:44:41.250083+0000","last_undegraded":"2024-04-21T08:44:41.250083+0000","last_fullsized":"2024-04-21T08:44:41.250083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:43.350877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251167+0000","last_change":"2024-04-21T08:42:21.971147+0000","last_active":"2024-04-21T08:44:41.251167+0000","last_peered":"2024-04-21T08:44:41.251167+0000","last_clean":"2024-04-21T08:44:41.251167+0000","last_became_active":"2024-04-21T08:42:21.970001+0000","last_became_peered":"2024-04-21T08:42:21.970001+0000","last_unstale":"2024-04-21T08:44:41.251167+0000","last_undegraded":"2024-04-21T08:44:41.251167+0000","last_fullsized":"2024-04-21T08:44:41.251167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:01:44.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250876+0000","last_change":"2024-04-21T08:42:16.643371+0000","last_active":"2024-04-21T08:44:41.250876+0000","last_peered":"2024-04-21T08:44:41.250876+0000","last_clean":"2024-04-21T08:44:41.250876+0000","last_became_active":"2024-04-21T08:42:16.642906+0000","last_became_peered":"2024-04-21T08:42:16.642906+0000","last_unstale":"2024-04-21T08:44:41.250876+0000","last_undegraded":"2024-04-21T08:44:41.250876+0000","last_fullsized":"2024-04-21T08:44:41.250876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:58.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375124+0000","last_change":"2024-04-21T08:42:17.545185+0000","last_active":"2024-04-21T08:44:41.375124+0000","last_peered":"2024-04-21T08:44:41.375124+0000","last_clean":"2024-04-21T08:44:41.375124+0000","last_became_active":"2024-04-21T08:42:17.544757+0000","last_became_peered":"2024-04-21T08:42:17.544757+0000","last_unstale":"2024-04-21T08:44:41.375124+0000","last_undegraded":"2024-04-21T08:44:41.375124+0000","last_fullsized":"2024-04-21T08:44:41.375124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:36:18.968041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374923+0000","last_change":"2024-04-21T08:42:21.662541+0000","last_active":"2024-04-21T08:44:41.374923+0000","last_peered":"2024-04-21T08:44:41.374923+0000","last_clean":"2024-04-21T08:44:41.374923+0000","last_became_active":"2024-04-21T08:42:21.662144+0000","last_became_peered":"2024-04-21T08:42:21.662144+0000","last_unstale":"2024-04-21T08:44:41.374923+0000","last_undegraded":"2024-04-21T08:44:41.374923+0000","last_fullsized":"2024-04-21T08:44:41.374923+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:04.347722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375374+0000","last_change":"2024-04-21T08:42:16.638601+0000","last_active":"2024-04-21T08:44:41.375374+0000","last_peered":"2024-04-21T08:44:41.375374+0000","last_clean":"2024-04-21T08:44:41.375374+0000","last_became_active":"2024-04-21T08:42:16.638211+0000","last_became_peered":"2024-04-21T08:42:16.638211+0000","last_unstale":"2024-04-21T08:44:41.375374+0000","last_undegraded":"2024-04-21T08:44:41.375374+0000","last_fullsized":"2024-04-21T08:44:41.375374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:02.633086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375257+0000","last_change":"2024-04-21T08:42:17.545495+0000","last_active":"2024-04-21T08:44:41.375257+0000","last_peered":"2024-04-21T08:44:41.375257+0000","last_clean":"2024-04-21T08:44:41.375257+0000","last_became_active":"2024-04-21T08:42:17.545159+0000","last_became_peered":"2024-04-21T08:42:17.545159+0000","last_unstale":"2024-04-21T08:44:41.375257+0000","last_undegraded":"2024-04-21T08:44:41.375257+0000","last_fullsized":"2024-04-21T08:44:41.375257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:27.150277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417564+0000","last_change":"2024-04-21T08:42:21.656890+0000","last_active":"2024-04-21T08:44:41.417564+0000","last_peered":"2024-04-21T08:44:41.417564+0000","last_clean":"2024-04-21T08:44:41.417564+0000","last_became_active":"2024-04-21T08:42:21.656475+0000","last_became_peered":"2024-04-21T08:42:21.656475+0000","last_unstale":"2024-04-21T08:44:41.417564+0000","last_undegraded":"2024-04-21T08:44:41.417564+0000","last_fullsized":"2024-04-21T08:44:41.417564+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:46.138591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'39","reported_seq":81,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.478489+0000","last_change":"2024-04-21T08:42:16.638244+0000","last_active":"2024-04-21T08:44:41.478489+0000","last_peered":"2024-04-21T08:44:41.478489+0000","last_clean":"2024-04-21T08:44:41.478489+0000","last_became_active":"2024-04-21T08:42:16.637883+0000","last_became_peered":"2024-04-21T08:42:16.637883+0000","last_unstale":"2024-04-21T08:44:41.478489+0000","last_undegraded":"2024-04-21T08:44:41.478489+0000","last_fullsized":"2024-04-21T08:44:41.478489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:20.888301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374936+0000","last_change":"2024-04-21T08:42:17.548032+0000","last_active":"2024-04-21T08:44:41.374936+0000","last_peered":"2024-04-21T08:44:41.374936+0000","last_clean":"2024-04-21T08:44:41.374936+0000","last_became_active":"2024-04-21T08:42:17.547660+0000","last_became_peered":"2024-04-21T08:42:17.547660+0000","last_unstale":"2024-04-21T08:44:41.374936+0000","last_undegraded":"2024-04-21T08:44:41.374936+0000","last_fullsized":"2024-04-21T08:44:41.374936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:14.710759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365005+0000","last_change":"2024-04-21T08:42:21.659286+0000","last_active":"2024-04-21T08:44:41.365005+0000","last_peered":"2024-04-21T08:44:41.365005+0000","last_clean":"2024-04-21T08:44:41.365005+0000","last_became_active":"2024-04-21T08:42:21.658844+0000","last_became_peered":"2024-04-21T08:42:21.658844+0000","last_unstale":"2024-04-21T08:44:41.365005+0000","last_undegraded":"2024-04-21T08:44:41.365005+0000","last_fullsized":"2024-04-21T08:44:41.365005+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:28.742646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376281+0000","last_change":"2024-04-21T08:42:16.643354+0000","last_active":"2024-04-21T08:44:41.376281+0000","last_peered":"2024-04-21T08:44:41.376281+0000","last_clean":"2024-04-21T08:44:41.376281+0000","last_became_active":"2024-04-21T08:42:16.642967+0000","last_became_peered":"2024-04-21T08:42:16.642967+0000","last_unstale":"2024-04-21T08:44:41.376281+0000","last_undegraded":"2024-04-21T08:44:41.376281+0000","last_fullsized":"2024-04-21T08:44:41.376281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:06:03.495181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375409+0000","last_change":"2024-04-21T08:42:17.539551+0000","last_active":"2024-04-21T08:44:41.375409+0000","last_peered":"2024-04-21T08:44:41.375409+0000","last_clean":"2024-04-21T08:44:41.375409+0000","last_became_active":"2024-04-21T08:42:17.539202+0000","last_became_peered":"2024-04-21T08:42:17.539202+0000","last_unstale":"2024-04-21T08:44:41.375409+0000","last_undegraded":"2024-04-21T08:44:41.375409+0000","last_fullsized":"2024-04-21T08:44:41.375409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:37.109421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375023+0000","last_change":"2024-04-21T08:42:17.544582+0000","last_active":"2024-04-21T08:44:41.375023+0000","last_peered":"2024-04-21T08:44:41.375023+0000","last_clean":"2024-04-21T08:44:41.375023+0000","last_became_active":"2024-04-21T08:42:17.543800+0000","last_became_peered":"2024-04-21T08:42:17.543800+0000","last_unstale":"2024-04-21T08:44:41.375023+0000","last_undegraded":"2024-04-21T08:44:41.375023+0000","last_fullsized":"2024-04-21T08:44:41.375023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:44:04.107207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374756+0000","last_change":"2024-04-21T08:42:17.543788+0000","last_active":"2024-04-21T08:44:41.374756+0000","last_peered":"2024-04-21T08:44:41.374756+0000","last_clean":"2024-04-21T08:44:41.374756+0000","last_became_active":"2024-04-21T08:42:17.540939+0000","last_became_peered":"2024-04-21T08:42:17.540939+0000","last_unstale":"2024-04-21T08:44:41.374756+0000","last_undegraded":"2024-04-21T08:44:41.374756+0000","last_fullsized":"2024-04-21T08:44:41.374756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:59.275098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374942+0000","last_change":"2024-04-21T08:42:17.540605+0000","last_active":"2024-04-21T08:44:41.374942+0000","last_peered":"2024-04-21T08:44:41.374942+0000","last_clean":"2024-04-21T08:44:41.374942+0000","last_became_active":"2024-04-21T08:42:17.540073+0000","last_became_peered":"2024-04-21T08:42:17.540073+0000","last_unstale":"2024-04-21T08:44:41.374942+0000","last_undegraded":"2024-04-21T08:44:41.374942+0000","last_fullsized":"2024-04-21T08:44:41.374942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:02:29.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417931+0000","last_change":"2024-04-21T08:42:17.551608+0000","last_active":"2024-04-21T08:44:41.417931+0000","last_peered":"2024-04-21T08:44:41.417931+0000","last_clean":"2024-04-21T08:44:41.417931+0000","last_became_active":"2024-04-21T08:42:17.551117+0000","last_became_peered":"2024-04-21T08:42:17.551117+0000","last_unstale":"2024-04-21T08:44:41.417931+0000","last_undegraded":"2024-04-21T08:44:41.417931+0000","last_fullsized":"2024-04-21T08:44:41.417931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:18.391928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375875+0000","last_change":"2024-04-21T08:42:17.536509+0000","last_active":"2024-04-21T08:44:41.375875+0000","last_peered":"2024-04-21T08:44:41.375875+0000","last_clean":"2024-04-21T08:44:41.375875+0000","last_became_active":"2024-04-21T08:42:17.536146+0000","last_became_peered":"2024-04-21T08:42:17.536146+0000","last_unstale":"2024-04-21T08:44:41.375875+0000","last_undegraded":"2024-04-21T08:44:41.375875+0000","last_fullsized":"2024-04-21T08:44:41.375875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:29.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375199+0000","last_change":"2024-04-21T08:42:17.543786+0000","last_active":"2024-04-21T08:44:41.375199+0000","last_peered":"2024-04-21T08:44:41.375199+0000","last_clean":"2024-04-21T08:44:41.375199+0000","last_became_active":"2024-04-21T08:42:17.543435+0000","last_became_peered":"2024-04-21T08:42:17.543435+0000","last_unstale":"2024-04-21T08:44:41.375199+0000","last_undegraded":"2024-04-21T08:44:41.375199+0000","last_fullsized":"2024-04-21T08:44:41.375199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375616+0000","last_change":"2024-04-21T08:42:17.541611+0000","last_active":"2024-04-21T08:44:41.375616+0000","last_peered":"2024-04-21T08:44:41.375616+0000","last_clean":"2024-04-21T08:44:41.375616+0000","last_became_active":"2024-04-21T08:42:17.541259+0000","last_became_peered":"2024-04-21T08:42:17.541259+0000","last_unstale":"2024-04-21T08:44:41.375616+0000","last_undegraded":"2024-04-21T08:44:41.375616+0000","last_fullsized":"2024-04-21T08:44:41.375616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:20.900629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356654+0000","last_change":"2024-04-21T08:44:11.774062+0000","last_active":"2024-04-21T08:44:41.356654+0000","last_peered":"2024-04-21T08:44:41.356654+0000","last_clean":"2024-04-21T08:44:41.356654+0000","last_became_active":"2024-04-21T08:44:11.773703+0000","last_became_peered":"2024-04-21T08:44:11.773703+0000","last_unstale":"2024-04-21T08:44:41.356654+0000","last_undegraded":"2024-04-21T08:44:41.356654+0000","last_fullsized":"2024-04-21T08:44:41.356654+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:10.739068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365197+0000","last_change":"2024-04-21T08:42:17.545526+0000","last_active":"2024-04-21T08:44:41.365197+0000","last_peered":"2024-04-21T08:44:41.365197+0000","last_clean":"2024-04-21T08:44:41.365197+0000","last_became_active":"2024-04-21T08:42:17.545135+0000","last_became_peered":"2024-04-21T08:42:17.545135+0000","last_unstale":"2024-04-21T08:44:41.365197+0000","last_undegraded":"2024-04-21T08:44:41.365197+0000","last_fullsized":"2024-04-21T08:44:41.365197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:37:58.038459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250029+0000","last_change":"2024-04-21T08:42:17.547063+0000","last_active":"2024-04-21T08:44:41.250029+0000","last_peered":"2024-04-21T08:44:41.250029+0000","last_clean":"2024-04-21T08:44:41.250029+0000","last_became_active":"2024-04-21T08:42:17.546689+0000","last_became_peered":"2024-04-21T08:42:17.546689+0000","last_unstale":"2024-04-21T08:44:41.250029+0000","last_undegraded":"2024-04-21T08:44:41.250029+0000","last_fullsized":"2024-04-21T08:44:41.250029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:26.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375261+0000","last_change":"2024-04-21T08:42:17.542545+0000","last_active":"2024-04-21T08:44:41.375261+0000","last_peered":"2024-04-21T08:44:41.375261+0000","last_clean":"2024-04-21T08:44:41.375261+0000","last_became_active":"2024-04-21T08:42:17.542147+0000","last_became_peered":"2024-04-21T08:42:17.542147+0000","last_unstale":"2024-04-21T08:44:41.375261+0000","last_undegraded":"2024-04-21T08:44:41.375261+0000","last_fullsized":"2024-04-21T08:44:41.375261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:15.064110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376582+0000","last_change":"2024-04-21T08:42:17.541262+0000","last_active":"2024-04-21T08:44:41.376582+0000","last_peered":"2024-04-21T08:44:41.376582+0000","last_clean":"2024-04-21T08:44:41.376582+0000","last_became_active":"2024-04-21T08:42:17.540817+0000","last_became_peered":"2024-04-21T08:42:17.540817+0000","last_unstale":"2024-04-21T08:44:41.376582+0000","last_undegraded":"2024-04-21T08:44:41.376582+0000","last_fullsized":"2024-04-21T08:44:41.376582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:22:59.597680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375116+0000","last_change":"2024-04-21T08:42:17.551109+0000","last_active":"2024-04-21T08:44:41.375116+0000","last_peered":"2024-04-21T08:44:41.375116+0000","last_clean":"2024-04-21T08:44:41.375116+0000","last_became_active":"2024-04-21T08:42:17.550104+0000","last_became_peered":"2024-04-21T08:42:17.550104+0000","last_unstale":"2024-04-21T08:44:41.375116+0000","last_undegraded":"2024-04-21T08:44:41.375116+0000","last_fullsized":"2024-04-21T08:44:41.375116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:22:19.361752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107298,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595682,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":350,"num_read_kb":280,"num_write":201,"num_write_kb":1411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":1787046,"data_compressed":80802,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":532412,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612655,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36496,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708656,"statfs":{"total":95995035648,"available":95957663744,"internally_reserved":0,"allocated":1183744,"data_stored":1314391,"data_compressed":73179,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776181,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1069056,"data_stored":930306,"data_compressed":48872,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":926232,"data_compressed":48872,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":1127458,"data_compressed":65492,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528316,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1105920,"data_stored":598384,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1036288,"data_stored":528406,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":598418,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1053236,"data_compressed":55028,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":532434,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1118208,"data_stored":1254562,"data_compressed":71648,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70102,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":201862,"data_compressed":7788,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:44:43.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:44:43.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.704+0000 7f8dfa57e640 1 -- 172.21.15.7:0/520354205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df4105330 msgr2=0x7f8df4107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.704+0000 7f8dfa57e640 1 --2- 172.21.15.7:0/520354205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df4105330 0x7f8df4107720 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009900 tx=0x7f8de802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:43.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.704+0000 7f8dfa57e640 1 -- 172.21.15.7:0/520354205 shutdown_connections 2024-04-21T08:44:43.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.704+0000 7f8dfa57e640 1 --2- 172.21.15.7:0/520354205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8df4107c60 0x7f8df410a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.704+0000 7f8dfa57e640 1 --2- 172.21.15.7:0/520354205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df4105330 0x7f8df4107720 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.704+0000 7f8dfa57e640 1 --2- 172.21.15.7:0/520354205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4102a00 0x7f8df4104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.704+0000 7f8dfa57e640 1 -- 172.21.15.7:0/520354205 >> 172.21.15.7:0/520354205 conn(0x7f8df40fc3f0 msgr2=0x7f8df40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:43.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.704+0000 7f8dfa57e640 1 -- 172.21.15.7:0/520354205 shutdown_connections 2024-04-21T08:44:43.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.704+0000 7f8dfa57e640 1 -- 172.21.15.7:0/520354205 wait complete. 2024-04-21T08:44:43.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.705+0000 7f8dfa57e640 1 Processor -- start 2024-04-21T08:44:43.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.705+0000 7f8dfa57e640 1 -- start start 2024-04-21T08:44:43.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8dfa57e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4102a00 0x7f8df4102370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:43.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8dfa57e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8df4105330 0x7f8df41009b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:43.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8dfa57e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df4107c60 0x7f8df4100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:43.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8dfa57e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8df410dab0 con 0x7f8df4107c60 2024-04-21T08:44:43.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8dfa57e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8df410d930 con 0x7f8df4105330 2024-04-21T08:44:43.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8dfa57e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8df410dc30 con 0x7f8df4102a00 2024-04-21T08:44:43.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4102a00 0x7f8df4102370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df8b1b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df4107c60 0x7f8df4100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4102a00 0x7f8df4102370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:42656/0 (socket says 172.21.15.7:42656) 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df3fff640 1 -- 172.21.15.7:0/458175787 learned_addr learned my addr 172.21.15.7:0/458175787 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df37fe640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8df4105330 0x7f8df41009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df3fff640 1 -- 172.21.15.7:0/458175787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8df4105330 msgr2=0x7f8df41009b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df3fff640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8df4105330 0x7f8df41009b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df3fff640 1 -- 172.21.15.7:0/458175787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df4107c60 msgr2=0x7f8df4100ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df3fff640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df4107c60 0x7f8df4100ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df3fff640 1 -- 172.21.15.7:0/458175787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df4101780 con 0x7f8df4102a00 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df3fff640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4102a00 0x7f8df4102370 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8de400e9f0 tx=0x7f8de400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.706+0000 7f8df17fa640 1 -- 172.21.15.7:0/458175787 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de400cd80 con 0x7f8df4102a00 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.707+0000 7f8df17fa640 1 -- 172.21.15.7:0/458175787 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8de40054d0 con 0x7f8df4102a00 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.707+0000 7f8dfa57e640 1 -- 172.21.15.7:0/458175787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df41a22e0 con 0x7f8df4102a00 2024-04-21T08:44:43.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.707+0000 7f8df17fa640 1 -- 172.21.15.7:0/458175787 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4010690 con 0x7f8df4102a00 2024-04-21T08:44:43.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.707+0000 7f8dfa57e640 1 -- 172.21.15.7:0/458175787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df41a27b0 con 0x7f8df4102a00 2024-04-21T08:44:43.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.708+0000 7f8dfa57e640 1 -- 172.21.15.7:0/458175787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8df4068e00 con 0x7f8df4102a00 2024-04-21T08:44:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.709+0000 7f8df17fa640 1 -- 172.21.15.7:0/458175787 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f8de4004e90 con 0x7f8df4102a00 2024-04-21T08:44:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.709+0000 7f8df17fa640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ddc0a39b0 0x7f8ddc0a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.709+0000 7f8df17fa640 1 -- 172.21.15.7:0/458175787 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f8de40d09a0 con 0x7f8df4102a00 2024-04-21T08:44:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.710+0000 7f8df37fe640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ddc0a39b0 0x7f8ddc0a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.710+0000 7f8df37fe640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ddc0a39b0 0x7f8ddc0a5e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8de8005ab0 tx=0x7f8de8005fd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:43.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.714+0000 7f8df17fa640 1 -- 172.21.15.7:0/458175787 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8de409d120 con 0x7f8df4102a00 2024-04-21T08:44:43.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.815+0000 7f8df17fa640 1 -- 172.21.15.7:0/458175787 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 23) ==== 138765+0+0 (secure 0 0 0) 0x7f8de4099090 con 0x7f8df4102a00 2024-04-21T08:44:43.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.910+0000 7f8dfa57e640 1 -- 172.21.15.7:0/458175787 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8df402d940 con 0x7f8ddc0a39b0 2024-04-21T08:44:43.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.930+0000 7f8df17fa640 1 -- 172.21.15.7:0/458175787 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339635 (secure 0 0 0) 0x7f8df402d940 con 0x7f8ddc0a39b0 2024-04-21T08:44:43.933 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:43.935 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-21T08:44:43.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 -- 172.21.15.7:0/458175787 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ddc0a39b0 msgr2=0x7f8ddc0a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ddc0a39b0 0x7f8ddc0a5e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8de8005ab0 tx=0x7f8de8005fd0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 -- 172.21.15.7:0/458175787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4102a00 msgr2=0x7f8df4102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4102a00 0x7f8df4102370 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8de400e9f0 tx=0x7f8de400edf0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 -- 172.21.15.7:0/458175787 shutdown_connections 2024-04-21T08:44:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8df4107c60 0x7f8df4100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8df4105330 0x7f8df41009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ddc0a39b0 0x7f8ddc0a5e70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 --2- 172.21.15.7:0/458175787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8df4102a00 0x7f8df4102370 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 -- 172.21.15.7:0/458175787 >> 172.21.15.7:0/458175787 conn(0x7f8df40fc3f0 msgr2=0x7f8df41032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 -- 172.21.15.7:0/458175787 shutdown_connections 2024-04-21T08:44:43.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:43.934+0000 7f8dfa57e640 1 -- 172.21.15.7:0/458175787 wait complete. 2024-04-21T08:44:43.945 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-21T08:44:43.262682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186284,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":486,"num_read_kb":399,"num_write":384,"num_write_kb":4351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":403472,"kb_used_data":12676,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124538352,"statfs":{"total":1151940427776,"available":1151527272448,"internally_reserved":0,"allocated":12980224,"data_stored":9924555,"data_compressed":544711,"data_compressed_allocated":737280,"data_compressed_original":4325376,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.989634"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375541+0000","last_change":"2024-04-21T08:42:17.545701+0000","last_active":"2024-04-21T08:44:41.375541+0000","last_peered":"2024-04-21T08:44:41.375541+0000","last_clean":"2024-04-21T08:44:41.375541+0000","last_became_active":"2024-04-21T08:42:17.545157+0000","last_became_peered":"2024-04-21T08:42:17.545157+0000","last_unstale":"2024-04-21T08:44:41.375541+0000","last_undegraded":"2024-04-21T08:44:41.375541+0000","last_fullsized":"2024-04-21T08:44:41.375541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:05:41.952879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418165+0000","last_change":"2024-04-21T08:42:17.541153+0000","last_active":"2024-04-21T08:44:41.418165+0000","last_peered":"2024-04-21T08:44:41.418165+0000","last_clean":"2024-04-21T08:44:41.418165+0000","last_became_active":"2024-04-21T08:42:17.540771+0000","last_became_peered":"2024-04-21T08:42:17.540771+0000","last_unstale":"2024-04-21T08:44:41.418165+0000","last_undegraded":"2024-04-21T08:44:41.418165+0000","last_fullsized":"2024-04-21T08:44:41.418165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:35:06.077367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417993+0000","last_change":"2024-04-21T08:42:17.545525+0000","last_active":"2024-04-21T08:44:41.417993+0000","last_peered":"2024-04-21T08:44:41.417993+0000","last_clean":"2024-04-21T08:44:41.417993+0000","last_became_active":"2024-04-21T08:42:17.545168+0000","last_became_peered":"2024-04-21T08:42:17.545168+0000","last_unstale":"2024-04-21T08:44:41.417993+0000","last_undegraded":"2024-04-21T08:44:41.417993+0000","last_fullsized":"2024-04-21T08:44:41.417993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:51.080385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251479+0000","last_change":"2024-04-21T08:42:17.543917+0000","last_active":"2024-04-21T08:44:41.251479+0000","last_peered":"2024-04-21T08:44:41.251479+0000","last_clean":"2024-04-21T08:44:41.251479+0000","last_became_active":"2024-04-21T08:42:17.543513+0000","last_became_peered":"2024-04-21T08:42:17.543513+0000","last_unstale":"2024-04-21T08:44:41.251479+0000","last_undegraded":"2024-04-21T08:44:41.251479+0000","last_fullsized":"2024-04-21T08:44:41.251479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:23.058600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250648+0000","last_change":"2024-04-21T08:42:17.549737+0000","last_active":"2024-04-21T08:44:41.250648+0000","last_peered":"2024-04-21T08:44:41.250648+0000","last_clean":"2024-04-21T08:44:41.250648+0000","last_became_active":"2024-04-21T08:42:17.549396+0000","last_became_peered":"2024-04-21T08:42:17.549396+0000","last_unstale":"2024-04-21T08:44:41.250648+0000","last_undegraded":"2024-04-21T08:44:41.250648+0000","last_fullsized":"2024-04-21T08:44:41.250648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:01:51.879960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368365+0000","last_change":"2024-04-21T08:43:10.792321+0000","last_active":"2024-04-21T08:44:41.368365+0000","last_peered":"2024-04-21T08:44:41.368365+0000","last_clean":"2024-04-21T08:44:41.368365+0000","last_became_active":"2024-04-21T08:43:10.791979+0000","last_became_peered":"2024-04-21T08:43:10.791979+0000","last_unstale":"2024-04-21T08:44:41.368365+0000","last_undegraded":"2024-04-21T08:44:41.368365+0000","last_fullsized":"2024-04-21T08:44:41.368365+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:11:16.056778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374889+0000","last_change":"2024-04-21T08:42:17.541172+0000","last_active":"2024-04-21T08:44:41.374889+0000","last_peered":"2024-04-21T08:44:41.374889+0000","last_clean":"2024-04-21T08:44:41.374889+0000","last_became_active":"2024-04-21T08:42:17.540829+0000","last_became_peered":"2024-04-21T08:42:17.540829+0000","last_unstale":"2024-04-21T08:44:41.374889+0000","last_undegraded":"2024-04-21T08:44:41.374889+0000","last_fullsized":"2024-04-21T08:44:41.374889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:16:12.633980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249753+0000","last_change":"2024-04-21T08:42:17.549744+0000","last_active":"2024-04-21T08:44:41.249753+0000","last_peered":"2024-04-21T08:44:41.249753+0000","last_clean":"2024-04-21T08:44:41.249753+0000","last_became_active":"2024-04-21T08:42:17.549378+0000","last_became_peered":"2024-04-21T08:42:17.549378+0000","last_unstale":"2024-04-21T08:44:41.249753+0000","last_undegraded":"2024-04-21T08:44:41.249753+0000","last_fullsized":"2024-04-21T08:44:41.249753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:03:07.702143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375768+0000","last_change":"2024-04-21T08:42:17.548675+0000","last_active":"2024-04-21T08:44:41.375768+0000","last_peered":"2024-04-21T08:44:41.375768+0000","last_clean":"2024-04-21T08:44:41.375768+0000","last_became_active":"2024-04-21T08:42:17.547809+0000","last_became_peered":"2024-04-21T08:42:17.547809+0000","last_unstale":"2024-04-21T08:44:41.375768+0000","last_undegraded":"2024-04-21T08:44:41.375768+0000","last_fullsized":"2024-04-21T08:44:41.375768+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:23.343473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368771+0000","last_change":"2024-04-21T08:42:17.551012+0000","last_active":"2024-04-21T08:44:41.368771+0000","last_peered":"2024-04-21T08:44:41.368771+0000","last_clean":"2024-04-21T08:44:41.368771+0000","last_became_active":"2024-04-21T08:42:17.550627+0000","last_became_peered":"2024-04-21T08:42:17.550627+0000","last_unstale":"2024-04-21T08:44:41.368771+0000","last_undegraded":"2024-04-21T08:44:41.368771+0000","last_fullsized":"2024-04-21T08:44:41.368771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:57.501588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375100+0000","last_change":"2024-04-21T08:42:17.546192+0000","last_active":"2024-04-21T08:44:41.375100+0000","last_peered":"2024-04-21T08:44:41.375100+0000","last_clean":"2024-04-21T08:44:41.375100+0000","last_became_active":"2024-04-21T08:42:17.545808+0000","last_became_peered":"2024-04-21T08:42:17.545808+0000","last_unstale":"2024-04-21T08:44:41.375100+0000","last_undegraded":"2024-04-21T08:44:41.375100+0000","last_fullsized":"2024-04-21T08:44:41.375100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:54.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251368+0000","last_change":"2024-04-21T08:42:17.553351+0000","last_active":"2024-04-21T08:44:41.251368+0000","last_peered":"2024-04-21T08:44:41.251368+0000","last_clean":"2024-04-21T08:44:41.251368+0000","last_became_active":"2024-04-21T08:42:17.550507+0000","last_became_peered":"2024-04-21T08:42:17.550507+0000","last_unstale":"2024-04-21T08:44:41.251368+0000","last_undegraded":"2024-04-21T08:44:41.251368+0000","last_fullsized":"2024-04-21T08:44:41.251368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:25.561375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376794+0000","last_change":"2024-04-21T08:42:17.540533+0000","last_active":"2024-04-21T08:44:41.376794+0000","last_peered":"2024-04-21T08:44:41.376794+0000","last_clean":"2024-04-21T08:44:41.376794+0000","last_became_active":"2024-04-21T08:42:17.539940+0000","last_became_peered":"2024-04-21T08:42:17.539940+0000","last_unstale":"2024-04-21T08:44:41.376794+0000","last_undegraded":"2024-04-21T08:44:41.376794+0000","last_fullsized":"2024-04-21T08:44:41.376794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:45:19.102899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417676+0000","last_change":"2024-04-21T08:42:17.537794+0000","last_active":"2024-04-21T08:44:41.417676+0000","last_peered":"2024-04-21T08:44:41.417676+0000","last_clean":"2024-04-21T08:44:41.417676+0000","last_became_active":"2024-04-21T08:42:17.537376+0000","last_became_peered":"2024-04-21T08:42:17.537376+0000","last_unstale":"2024-04-21T08:44:41.417676+0000","last_undegraded":"2024-04-21T08:44:41.417676+0000","last_fullsized":"2024-04-21T08:44:41.417676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:57.430036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417515+0000","last_change":"2024-04-21T08:42:17.550574+0000","last_active":"2024-04-21T08:44:41.417515+0000","last_peered":"2024-04-21T08:44:41.417515+0000","last_clean":"2024-04-21T08:44:41.417515+0000","last_became_active":"2024-04-21T08:42:17.550205+0000","last_became_peered":"2024-04-21T08:42:17.550205+0000","last_unstale":"2024-04-21T08:44:41.417515+0000","last_undegraded":"2024-04-21T08:44:41.417515+0000","last_fullsized":"2024-04-21T08:44:41.417515+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:41:08.226477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374544+0000","last_change":"2024-04-21T08:42:17.542239+0000","last_active":"2024-04-21T08:44:41.374544+0000","last_peered":"2024-04-21T08:44:41.374544+0000","last_clean":"2024-04-21T08:44:41.374544+0000","last_became_active":"2024-04-21T08:42:17.541848+0000","last_became_peered":"2024-04-21T08:42:17.541848+0000","last_unstale":"2024-04-21T08:44:41.374544+0000","last_undegraded":"2024-04-21T08:44:41.374544+0000","last_fullsized":"2024-04-21T08:44:41.374544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:47.717451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375417+0000","last_change":"2024-04-21T08:42:17.553200+0000","last_active":"2024-04-21T08:44:41.375417+0000","last_peered":"2024-04-21T08:44:41.375417+0000","last_clean":"2024-04-21T08:44:41.375417+0000","last_became_active":"2024-04-21T08:42:17.552913+0000","last_became_peered":"2024-04-21T08:42:17.552913+0000","last_unstale":"2024-04-21T08:44:41.375417+0000","last_undegraded":"2024-04-21T08:44:41.375417+0000","last_fullsized":"2024-04-21T08:44:41.375417+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:39:04.186928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356904+0000","last_change":"2024-04-21T08:42:17.534881+0000","last_active":"2024-04-21T08:44:41.356904+0000","last_peered":"2024-04-21T08:44:41.356904+0000","last_clean":"2024-04-21T08:44:41.356904+0000","last_became_active":"2024-04-21T08:42:17.534496+0000","last_became_peered":"2024-04-21T08:42:17.534496+0000","last_unstale":"2024-04-21T08:44:41.356904+0000","last_undegraded":"2024-04-21T08:44:41.356904+0000","last_fullsized":"2024-04-21T08:44:41.356904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:50:19.409391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375734+0000","last_change":"2024-04-21T08:42:17.540480+0000","last_active":"2024-04-21T08:44:41.375734+0000","last_peered":"2024-04-21T08:44:41.375734+0000","last_clean":"2024-04-21T08:44:41.375734+0000","last_became_active":"2024-04-21T08:42:17.539742+0000","last_became_peered":"2024-04-21T08:42:17.539742+0000","last_unstale":"2024-04-21T08:44:41.375734+0000","last_undegraded":"2024-04-21T08:44:41.375734+0000","last_fullsized":"2024-04-21T08:44:41.375734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:00.238976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374953+0000","last_change":"2024-04-21T08:44:11.876826+0000","last_active":"2024-04-21T08:44:41.374953+0000","last_peered":"2024-04-21T08:44:41.374953+0000","last_clean":"2024-04-21T08:44:41.374953+0000","last_became_active":"2024-04-21T08:44:11.876354+0000","last_became_peered":"2024-04-21T08:44:11.876354+0000","last_unstale":"2024-04-21T08:44:41.374953+0000","last_undegraded":"2024-04-21T08:44:41.374953+0000","last_fullsized":"2024-04-21T08:44:41.374953+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:00:05.574008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356782+0000","last_change":"2024-04-21T08:42:16.639888+0000","last_active":"2024-04-21T08:44:41.356782+0000","last_peered":"2024-04-21T08:44:41.356782+0000","last_clean":"2024-04-21T08:44:41.356782+0000","last_became_active":"2024-04-21T08:42:16.639403+0000","last_became_peered":"2024-04-21T08:42:16.639403+0000","last_unstale":"2024-04-21T08:44:41.356782+0000","last_undegraded":"2024-04-21T08:44:41.356782+0000","last_fullsized":"2024-04-21T08:44:41.356782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:08:17.164924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249935+0000","last_change":"2024-04-21T08:43:10.983368+0000","last_active":"2024-04-21T08:44:41.249935+0000","last_peered":"2024-04-21T08:44:41.249935+0000","last_clean":"2024-04-21T08:44:41.249935+0000","last_became_active":"2024-04-21T08:43:10.982956+0000","last_became_peered":"2024-04-21T08:43:10.982956+0000","last_unstale":"2024-04-21T08:44:41.249935+0000","last_undegraded":"2024-04-21T08:44:41.249935+0000","last_fullsized":"2024-04-21T08:44:41.249935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:12:44.919996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250725+0000","last_change":"2024-04-21T08:42:21.673197+0000","last_active":"2024-04-21T08:44:41.250725+0000","last_peered":"2024-04-21T08:44:41.250725+0000","last_clean":"2024-04-21T08:44:41.250725+0000","last_became_active":"2024-04-21T08:42:21.672793+0000","last_became_peered":"2024-04-21T08:42:21.672793+0000","last_unstale":"2024-04-21T08:44:41.250725+0000","last_undegraded":"2024-04-21T08:44:41.250725+0000","last_fullsized":"2024-04-21T08:44:41.250725+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:44:29.788459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250990+0000","last_change":"2024-04-21T08:42:16.648577+0000","last_active":"2024-04-21T08:44:41.250990+0000","last_peered":"2024-04-21T08:44:41.250990+0000","last_clean":"2024-04-21T08:44:41.250990+0000","last_became_active":"2024-04-21T08:42:16.646716+0000","last_became_peered":"2024-04-21T08:42:16.646716+0000","last_unstale":"2024-04-21T08:44:41.250990+0000","last_undegraded":"2024-04-21T08:44:41.250990+0000","last_fullsized":"2024-04-21T08:44:41.250990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:10.327078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417936+0000","last_change":"2024-04-21T08:42:17.545430+0000","last_active":"2024-04-21T08:44:41.417936+0000","last_peered":"2024-04-21T08:44:41.417936+0000","last_clean":"2024-04-21T08:44:41.417936+0000","last_became_active":"2024-04-21T08:42:17.545057+0000","last_became_peered":"2024-04-21T08:42:17.545057+0000","last_unstale":"2024-04-21T08:44:41.417936+0000","last_undegraded":"2024-04-21T08:44:41.417936+0000","last_fullsized":"2024-04-21T08:44:41.417936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:44:59.030830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250899+0000","last_change":"2024-04-21T08:42:21.673196+0000","last_active":"2024-04-21T08:44:41.250899+0000","last_peered":"2024-04-21T08:44:41.250899+0000","last_clean":"2024-04-21T08:44:41.250899+0000","last_became_active":"2024-04-21T08:42:21.672808+0000","last_became_peered":"2024-04-21T08:42:21.672808+0000","last_unstale":"2024-04-21T08:44:41.250899+0000","last_undegraded":"2024-04-21T08:44:41.250899+0000","last_fullsized":"2024-04-21T08:44:41.250899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:39.482353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250844+0000","last_change":"2024-04-21T08:42:16.641611+0000","last_active":"2024-04-21T08:44:41.250844+0000","last_peered":"2024-04-21T08:44:41.250844+0000","last_clean":"2024-04-21T08:44:41.250844+0000","last_became_active":"2024-04-21T08:42:16.641221+0000","last_became_peered":"2024-04-21T08:42:16.641221+0000","last_unstale":"2024-04-21T08:44:41.250844+0000","last_undegraded":"2024-04-21T08:44:41.250844+0000","last_fullsized":"2024-04-21T08:44:41.250844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:45.496532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375395+0000","last_change":"2024-04-21T08:42:17.532986+0000","last_active":"2024-04-21T08:44:41.375395+0000","last_peered":"2024-04-21T08:44:41.375395+0000","last_clean":"2024-04-21T08:44:41.375395+0000","last_became_active":"2024-04-21T08:42:17.532583+0000","last_became_peered":"2024-04-21T08:42:17.532583+0000","last_unstale":"2024-04-21T08:44:41.375395+0000","last_undegraded":"2024-04-21T08:44:41.375395+0000","last_fullsized":"2024-04-21T08:44:41.375395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:43.780935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250791+0000","last_change":"2024-04-21T08:42:21.667459+0000","last_active":"2024-04-21T08:44:41.250791+0000","last_peered":"2024-04-21T08:44:41.250791+0000","last_clean":"2024-04-21T08:44:41.250791+0000","last_became_active":"2024-04-21T08:42:21.667183+0000","last_became_peered":"2024-04-21T08:42:21.667183+0000","last_unstale":"2024-04-21T08:44:41.250791+0000","last_undegraded":"2024-04-21T08:44:41.250791+0000","last_fullsized":"2024-04-21T08:44:41.250791+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:32:23.626267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250969+0000","last_change":"2024-04-21T08:42:16.645210+0000","last_active":"2024-04-21T08:44:41.250969+0000","last_peered":"2024-04-21T08:44:41.250969+0000","last_clean":"2024-04-21T08:44:41.250969+0000","last_became_active":"2024-04-21T08:42:16.644747+0000","last_became_peered":"2024-04-21T08:42:16.644747+0000","last_unstale":"2024-04-21T08:44:41.250969+0000","last_undegraded":"2024-04-21T08:44:41.250969+0000","last_fullsized":"2024-04-21T08:44:41.250969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:34:45.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365569+0000","last_change":"2024-04-21T08:43:10.984152+0000","last_active":"2024-04-21T08:44:41.365569+0000","last_peered":"2024-04-21T08:44:41.365569+0000","last_clean":"2024-04-21T08:44:41.365569+0000","last_became_active":"2024-04-21T08:43:10.983736+0000","last_became_peered":"2024-04-21T08:43:10.983736+0000","last_unstale":"2024-04-21T08:44:41.365569+0000","last_undegraded":"2024-04-21T08:44:41.365569+0000","last_fullsized":"2024-04-21T08:44:41.365569+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:10.182667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375391+0000","last_change":"2024-04-21T08:42:21.660739+0000","last_active":"2024-04-21T08:44:41.375391+0000","last_peered":"2024-04-21T08:44:41.375391+0000","last_clean":"2024-04-21T08:44:41.375391+0000","last_became_active":"2024-04-21T08:42:21.660307+0000","last_became_peered":"2024-04-21T08:42:21.660307+0000","last_unstale":"2024-04-21T08:44:41.375391+0000","last_undegraded":"2024-04-21T08:44:41.375391+0000","last_fullsized":"2024-04-21T08:44:41.375391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:26.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365416+0000","last_change":"2024-04-21T08:43:10.986184+0000","last_active":"2024-04-21T08:44:41.365416+0000","last_peered":"2024-04-21T08:44:41.365416+0000","last_clean":"2024-04-21T08:44:41.365416+0000","last_became_active":"2024-04-21T08:43:10.985533+0000","last_became_peered":"2024-04-21T08:43:10.985533+0000","last_unstale":"2024-04-21T08:44:41.365416+0000","last_undegraded":"2024-04-21T08:44:41.365416+0000","last_fullsized":"2024-04-21T08:44:41.365416+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:26.023306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250682+0000","last_change":"2024-04-21T08:42:17.542315+0000","last_active":"2024-04-21T08:44:41.250682+0000","last_peered":"2024-04-21T08:44:41.250682+0000","last_clean":"2024-04-21T08:44:41.250682+0000","last_became_active":"2024-04-21T08:42:17.541895+0000","last_became_peered":"2024-04-21T08:42:17.541895+0000","last_unstale":"2024-04-21T08:44:41.250682+0000","last_undegraded":"2024-04-21T08:44:41.250682+0000","last_fullsized":"2024-04-21T08:44:41.250682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:23:33.909499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375270+0000","last_change":"2024-04-21T08:44:11.878135+0000","last_active":"2024-04-21T08:44:41.375270+0000","last_peered":"2024-04-21T08:44:41.375270+0000","last_clean":"2024-04-21T08:44:41.375270+0000","last_became_active":"2024-04-21T08:44:11.877271+0000","last_became_peered":"2024-04-21T08:44:11.877271+0000","last_unstale":"2024-04-21T08:44:41.375270+0000","last_undegraded":"2024-04-21T08:44:41.375270+0000","last_fullsized":"2024-04-21T08:44:41.375270+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:31.345519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368361+0000","last_change":"2024-04-21T08:43:10.984839+0000","last_active":"2024-04-21T08:44:41.368361+0000","last_peered":"2024-04-21T08:44:41.368361+0000","last_clean":"2024-04-21T08:44:41.368361+0000","last_became_active":"2024-04-21T08:43:10.984382+0000","last_became_peered":"2024-04-21T08:43:10.984382+0000","last_unstale":"2024-04-21T08:44:41.368361+0000","last_undegraded":"2024-04-21T08:44:41.368361+0000","last_fullsized":"2024-04-21T08:44:41.368361+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:47:20.774102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365597+0000","last_change":"2024-04-21T08:42:17.547770+0000","last_active":"2024-04-21T08:44:41.365597+0000","last_peered":"2024-04-21T08:44:41.365597+0000","last_clean":"2024-04-21T08:44:41.365597+0000","last_became_active":"2024-04-21T08:42:17.547440+0000","last_became_peered":"2024-04-21T08:42:17.547440+0000","last_unstale":"2024-04-21T08:44:41.365597+0000","last_undegraded":"2024-04-21T08:44:41.365597+0000","last_fullsized":"2024-04-21T08:44:41.365597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:37:35.555737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375243+0000","last_change":"2024-04-21T08:42:21.670684+0000","last_active":"2024-04-21T08:44:41.375243+0000","last_peered":"2024-04-21T08:44:41.375243+0000","last_clean":"2024-04-21T08:44:41.375243+0000","last_became_active":"2024-04-21T08:42:21.670260+0000","last_became_peered":"2024-04-21T08:42:21.670260+0000","last_unstale":"2024-04-21T08:44:41.375243+0000","last_undegraded":"2024-04-21T08:44:41.375243+0000","last_fullsized":"2024-04-21T08:44:41.375243+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:45:06.192040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376102+0000","last_change":"2024-04-21T08:42:16.639632+0000","last_active":"2024-04-21T08:44:41.376102+0000","last_peered":"2024-04-21T08:44:41.376102+0000","last_clean":"2024-04-21T08:44:41.376102+0000","last_became_active":"2024-04-21T08:42:16.639301+0000","last_became_peered":"2024-04-21T08:42:16.639301+0000","last_unstale":"2024-04-21T08:44:41.376102+0000","last_undegraded":"2024-04-21T08:44:41.376102+0000","last_fullsized":"2024-04-21T08:44:41.376102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:58.838667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249940+0000","last_change":"2024-04-21T08:42:17.546109+0000","last_active":"2024-04-21T08:44:41.249940+0000","last_peered":"2024-04-21T08:44:41.249940+0000","last_clean":"2024-04-21T08:44:41.249940+0000","last_became_active":"2024-04-21T08:42:17.545673+0000","last_became_peered":"2024-04-21T08:42:17.545673+0000","last_unstale":"2024-04-21T08:44:41.249940+0000","last_undegraded":"2024-04-21T08:44:41.249940+0000","last_fullsized":"2024-04-21T08:44:41.249940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:18.947848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417985+0000","last_change":"2024-04-21T08:42:21.666114+0000","last_active":"2024-04-21T08:44:41.417985+0000","last_peered":"2024-04-21T08:44:41.417985+0000","last_clean":"2024-04-21T08:44:41.417985+0000","last_became_active":"2024-04-21T08:42:21.665020+0000","last_became_peered":"2024-04-21T08:42:21.665020+0000","last_unstale":"2024-04-21T08:44:41.417985+0000","last_undegraded":"2024-04-21T08:44:41.417985+0000","last_fullsized":"2024-04-21T08:44:41.417985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:33.644178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365224+0000","last_change":"2024-04-21T08:42:16.639118+0000","last_active":"2024-04-21T08:44:41.365224+0000","last_peered":"2024-04-21T08:44:41.365224+0000","last_clean":"2024-04-21T08:44:41.365224+0000","last_became_active":"2024-04-21T08:42:16.638792+0000","last_became_peered":"2024-04-21T08:42:16.638792+0000","last_unstale":"2024-04-21T08:44:41.365224+0000","last_undegraded":"2024-04-21T08:44:41.365224+0000","last_fullsized":"2024-04-21T08:44:41.365224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:55.568712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356215+0000","last_change":"2024-04-21T08:42:17.539688+0000","last_active":"2024-04-21T08:44:41.356215+0000","last_peered":"2024-04-21T08:44:41.356215+0000","last_clean":"2024-04-21T08:44:41.356215+0000","last_became_active":"2024-04-21T08:42:17.539335+0000","last_became_peered":"2024-04-21T08:42:17.539335+0000","last_unstale":"2024-04-21T08:44:41.356215+0000","last_undegraded":"2024-04-21T08:44:41.356215+0000","last_fullsized":"2024-04-21T08:44:41.356215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:27:51.520232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249882+0000","last_change":"2024-04-21T08:42:21.660570+0000","last_active":"2024-04-21T08:44:41.249882+0000","last_peered":"2024-04-21T08:44:41.249882+0000","last_clean":"2024-04-21T08:44:41.249882+0000","last_became_active":"2024-04-21T08:42:21.659398+0000","last_became_peered":"2024-04-21T08:42:21.659398+0000","last_unstale":"2024-04-21T08:44:41.249882+0000","last_undegraded":"2024-04-21T08:44:41.249882+0000","last_fullsized":"2024-04-21T08:44:41.249882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:55.840334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368786+0000","last_change":"2024-04-21T08:42:16.648192+0000","last_active":"2024-04-21T08:44:41.368786+0000","last_peered":"2024-04-21T08:44:41.368786+0000","last_clean":"2024-04-21T08:44:41.368786+0000","last_became_active":"2024-04-21T08:42:16.647666+0000","last_became_peered":"2024-04-21T08:42:16.647666+0000","last_unstale":"2024-04-21T08:44:41.368786+0000","last_undegraded":"2024-04-21T08:44:41.368786+0000","last_fullsized":"2024-04-21T08:44:41.368786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:33.344680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375708+0000","last_change":"2024-04-21T08:42:17.545925+0000","last_active":"2024-04-21T08:44:41.375708+0000","last_peered":"2024-04-21T08:44:41.375708+0000","last_clean":"2024-04-21T08:44:41.375708+0000","last_became_active":"2024-04-21T08:42:17.545555+0000","last_became_peered":"2024-04-21T08:42:17.545555+0000","last_unstale":"2024-04-21T08:44:41.375708+0000","last_undegraded":"2024-04-21T08:44:41.375708+0000","last_fullsized":"2024-04-21T08:44:41.375708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:13:07.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356087+0000","last_change":"2024-04-21T08:42:21.662757+0000","last_active":"2024-04-21T08:44:41.356087+0000","last_peered":"2024-04-21T08:44:41.356087+0000","last_clean":"2024-04-21T08:44:41.356087+0000","last_became_active":"2024-04-21T08:42:21.662306+0000","last_became_peered":"2024-04-21T08:42:21.662306+0000","last_unstale":"2024-04-21T08:44:41.356087+0000","last_undegraded":"2024-04-21T08:44:41.356087+0000","last_fullsized":"2024-04-21T08:44:41.356087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:26.856147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375135+0000","last_change":"2024-04-21T08:43:10.984690+0000","last_active":"2024-04-21T08:44:41.375135+0000","last_peered":"2024-04-21T08:44:41.375135+0000","last_clean":"2024-04-21T08:44:41.375135+0000","last_became_active":"2024-04-21T08:43:10.984290+0000","last_became_peered":"2024-04-21T08:43:10.984290+0000","last_unstale":"2024-04-21T08:44:41.375135+0000","last_undegraded":"2024-04-21T08:44:41.375135+0000","last_fullsized":"2024-04-21T08:44:41.375135+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:37:36.400946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418211+0000","last_change":"2024-04-21T08:43:10.793316+0000","last_active":"2024-04-21T08:44:41.418211+0000","last_peered":"2024-04-21T08:44:41.418211+0000","last_clean":"2024-04-21T08:44:41.418211+0000","last_became_active":"2024-04-21T08:43:10.793035+0000","last_became_peered":"2024-04-21T08:43:10.793035+0000","last_unstale":"2024-04-21T08:44:41.418211+0000","last_undegraded":"2024-04-21T08:44:41.418211+0000","last_fullsized":"2024-04-21T08:44:41.418211+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:41:04.247385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251083+0000","last_change":"2024-04-21T08:42:21.670561+0000","last_active":"2024-04-21T08:44:41.251083+0000","last_peered":"2024-04-21T08:44:41.251083+0000","last_clean":"2024-04-21T08:44:41.251083+0000","last_became_active":"2024-04-21T08:42:21.670170+0000","last_became_peered":"2024-04-21T08:42:21.670170+0000","last_unstale":"2024-04-21T08:44:41.251083+0000","last_undegraded":"2024-04-21T08:44:41.251083+0000","last_fullsized":"2024-04-21T08:44:41.251083+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:35:54.538013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365460+0000","last_change":"2024-04-21T08:42:16.637319+0000","last_active":"2024-04-21T08:44:41.365460+0000","last_peered":"2024-04-21T08:44:41.365460+0000","last_clean":"2024-04-21T08:44:41.365460+0000","last_became_active":"2024-04-21T08:42:16.636940+0000","last_became_peered":"2024-04-21T08:42:16.636940+0000","last_unstale":"2024-04-21T08:44:41.365460+0000","last_undegraded":"2024-04-21T08:44:41.365460+0000","last_fullsized":"2024-04-21T08:44:41.365460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:59:09.826786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375697+0000","last_change":"2024-04-21T08:42:17.546202+0000","last_active":"2024-04-21T08:44:41.375697+0000","last_peered":"2024-04-21T08:44:41.375697+0000","last_clean":"2024-04-21T08:44:41.375697+0000","last_became_active":"2024-04-21T08:42:17.545781+0000","last_became_peered":"2024-04-21T08:42:17.545781+0000","last_unstale":"2024-04-21T08:44:41.375697+0000","last_undegraded":"2024-04-21T08:44:41.375697+0000","last_fullsized":"2024-04-21T08:44:41.375697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:23.990951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356160+0000","last_change":"2024-04-21T08:42:21.662822+0000","last_active":"2024-04-21T08:44:41.356160+0000","last_peered":"2024-04-21T08:44:41.356160+0000","last_clean":"2024-04-21T08:44:41.356160+0000","last_became_active":"2024-04-21T08:42:21.662398+0000","last_became_peered":"2024-04-21T08:42:21.662398+0000","last_unstale":"2024-04-21T08:44:41.356160+0000","last_undegraded":"2024-04-21T08:44:41.356160+0000","last_fullsized":"2024-04-21T08:44:41.356160+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:38:40.499294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":324,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:42.133017+0000","last_change":"2024-04-21T08:42:16.636330+0000","last_active":"2024-04-21T08:44:42.133017+0000","last_peered":"2024-04-21T08:44:42.133017+0000","last_clean":"2024-04-21T08:44:42.133017+0000","last_became_active":"2024-04-21T08:42:16.635762+0000","last_became_peered":"2024-04-21T08:42:16.635762+0000","last_unstale":"2024-04-21T08:44:42.133017+0000","last_undegraded":"2024-04-21T08:44:42.133017+0000","last_fullsized":"2024-04-21T08:44:42.133017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:31.137923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":195,"num_write":38,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251043+0000","last_change":"2024-04-21T08:42:17.543033+0000","last_active":"2024-04-21T08:44:41.251043+0000","last_peered":"2024-04-21T08:44:41.251043+0000","last_clean":"2024-04-21T08:44:41.251043+0000","last_became_active":"2024-04-21T08:42:17.542692+0000","last_became_peered":"2024-04-21T08:42:17.542692+0000","last_unstale":"2024-04-21T08:44:41.251043+0000","last_undegraded":"2024-04-21T08:44:41.251043+0000","last_fullsized":"2024-04-21T08:44:41.251043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:33:32.111999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365780+0000","last_change":"2024-04-21T08:42:21.668831+0000","last_active":"2024-04-21T08:44:41.365780+0000","last_peered":"2024-04-21T08:44:41.365780+0000","last_clean":"2024-04-21T08:44:41.365780+0000","last_became_active":"2024-04-21T08:42:21.668485+0000","last_became_peered":"2024-04-21T08:42:21.668485+0000","last_unstale":"2024-04-21T08:44:41.365780+0000","last_undegraded":"2024-04-21T08:44:41.365780+0000","last_fullsized":"2024-04-21T08:44:41.365780+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:49:29.049707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375241+0000","last_change":"2024-04-21T08:42:16.640288+0000","last_active":"2024-04-21T08:44:41.375241+0000","last_peered":"2024-04-21T08:44:41.375241+0000","last_clean":"2024-04-21T08:44:41.375241+0000","last_became_active":"2024-04-21T08:42:16.639909+0000","last_became_peered":"2024-04-21T08:42:16.639909+0000","last_unstale":"2024-04-21T08:44:41.375241+0000","last_undegraded":"2024-04-21T08:44:41.375241+0000","last_fullsized":"2024-04-21T08:44:41.375241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:33.839442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375479+0000","last_change":"2024-04-21T08:42:17.541154+0000","last_active":"2024-04-21T08:44:41.375479+0000","last_peered":"2024-04-21T08:44:41.375479+0000","last_clean":"2024-04-21T08:44:41.375479+0000","last_became_active":"2024-04-21T08:42:17.540747+0000","last_became_peered":"2024-04-21T08:42:17.540747+0000","last_unstale":"2024-04-21T08:44:41.375479+0000","last_undegraded":"2024-04-21T08:44:41.375479+0000","last_fullsized":"2024-04-21T08:44:41.375479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:14:50.641144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374983+0000","last_change":"2024-04-21T08:42:21.670767+0000","last_active":"2024-04-21T08:44:41.374983+0000","last_peered":"2024-04-21T08:44:41.374983+0000","last_clean":"2024-04-21T08:44:41.374983+0000","last_became_active":"2024-04-21T08:42:21.670338+0000","last_became_peered":"2024-04-21T08:42:21.670338+0000","last_unstale":"2024-04-21T08:44:41.374983+0000","last_undegraded":"2024-04-21T08:44:41.374983+0000","last_fullsized":"2024-04-21T08:44:41.374983+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:38:55.275603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365468+0000","last_change":"2024-04-21T08:42:16.640144+0000","last_active":"2024-04-21T08:44:41.365468+0000","last_peered":"2024-04-21T08:44:41.365468+0000","last_clean":"2024-04-21T08:44:41.365468+0000","last_became_active":"2024-04-21T08:42:16.639707+0000","last_became_peered":"2024-04-21T08:42:16.639707+0000","last_unstale":"2024-04-21T08:44:41.365468+0000","last_undegraded":"2024-04-21T08:44:41.365468+0000","last_fullsized":"2024-04-21T08:44:41.365468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:11:32.785519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375240+0000","last_change":"2024-04-21T08:42:17.555019+0000","last_active":"2024-04-21T08:44:41.375240+0000","last_peered":"2024-04-21T08:44:41.375240+0000","last_clean":"2024-04-21T08:44:41.375240+0000","last_became_active":"2024-04-21T08:42:17.554645+0000","last_became_peered":"2024-04-21T08:42:17.554645+0000","last_unstale":"2024-04-21T08:44:41.375240+0000","last_undegraded":"2024-04-21T08:44:41.375240+0000","last_fullsized":"2024-04-21T08:44:41.375240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:42:30.353945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251223+0000","last_change":"2024-04-21T08:44:11.876740+0000","last_active":"2024-04-21T08:44:41.251223+0000","last_peered":"2024-04-21T08:44:41.251223+0000","last_clean":"2024-04-21T08:44:41.251223+0000","last_became_active":"2024-04-21T08:44:11.876108+0000","last_became_peered":"2024-04-21T08:44:11.876108+0000","last_unstale":"2024-04-21T08:44:41.251223+0000","last_undegraded":"2024-04-21T08:44:41.251223+0000","last_fullsized":"2024-04-21T08:44:41.251223+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:37.398743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250912+0000","last_change":"2024-04-21T08:42:16.640407+0000","last_active":"2024-04-21T08:44:41.250912+0000","last_peered":"2024-04-21T08:44:41.250912+0000","last_clean":"2024-04-21T08:44:41.250912+0000","last_became_active":"2024-04-21T08:42:16.640015+0000","last_became_peered":"2024-04-21T08:42:16.640015+0000","last_unstale":"2024-04-21T08:44:41.250912+0000","last_undegraded":"2024-04-21T08:44:41.250912+0000","last_fullsized":"2024-04-21T08:44:41.250912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:23:10.647184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374762+0000","last_change":"2024-04-21T08:42:17.545438+0000","last_active":"2024-04-21T08:44:41.374762+0000","last_peered":"2024-04-21T08:44:41.374762+0000","last_clean":"2024-04-21T08:44:41.374762+0000","last_became_active":"2024-04-21T08:42:17.544258+0000","last_became_peered":"2024-04-21T08:42:17.544258+0000","last_unstale":"2024-04-21T08:44:41.374762+0000","last_undegraded":"2024-04-21T08:44:41.374762+0000","last_fullsized":"2024-04-21T08:44:41.374762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:43.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376385+0000","last_change":"2024-04-21T08:42:21.671000+0000","last_active":"2024-04-21T08:44:41.376385+0000","last_peered":"2024-04-21T08:44:41.376385+0000","last_clean":"2024-04-21T08:44:41.376385+0000","last_became_active":"2024-04-21T08:42:21.670380+0000","last_became_peered":"2024-04-21T08:42:21.670380+0000","last_unstale":"2024-04-21T08:44:41.376385+0000","last_undegraded":"2024-04-21T08:44:41.376385+0000","last_fullsized":"2024-04-21T08:44:41.376385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:57.075686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250621+0000","last_change":"2024-04-21T08:42:16.649599+0000","last_active":"2024-04-21T08:44:41.250621+0000","last_peered":"2024-04-21T08:44:41.250621+0000","last_clean":"2024-04-21T08:44:41.250621+0000","last_became_active":"2024-04-21T08:42:16.649232+0000","last_became_peered":"2024-04-21T08:42:16.649232+0000","last_unstale":"2024-04-21T08:44:41.250621+0000","last_undegraded":"2024-04-21T08:44:41.250621+0000","last_fullsized":"2024-04-21T08:44:41.250621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:21.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356045+0000","last_change":"2024-04-21T08:44:11.776138+0000","last_active":"2024-04-21T08:44:41.356045+0000","last_peered":"2024-04-21T08:44:41.356045+0000","last_clean":"2024-04-21T08:44:41.356045+0000","last_became_active":"2024-04-21T08:44:11.775832+0000","last_became_peered":"2024-04-21T08:44:11.775832+0000","last_unstale":"2024-04-21T08:44:41.356045+0000","last_undegraded":"2024-04-21T08:44:41.356045+0000","last_fullsized":"2024-04-21T08:44:41.356045+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:03:34.210923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365172+0000","last_change":"2024-04-21T08:42:21.664811+0000","last_active":"2024-04-21T08:44:41.365172+0000","last_peered":"2024-04-21T08:44:41.365172+0000","last_clean":"2024-04-21T08:44:41.365172+0000","last_became_active":"2024-04-21T08:42:21.664332+0000","last_became_peered":"2024-04-21T08:42:21.664332+0000","last_unstale":"2024-04-21T08:44:41.365172+0000","last_undegraded":"2024-04-21T08:44:41.365172+0000","last_fullsized":"2024-04-21T08:44:41.365172+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:40.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251372+0000","last_change":"2024-04-21T08:42:16.641233+0000","last_active":"2024-04-21T08:44:41.251372+0000","last_peered":"2024-04-21T08:44:41.251372+0000","last_clean":"2024-04-21T08:44:41.251372+0000","last_became_active":"2024-04-21T08:42:16.640873+0000","last_became_peered":"2024-04-21T08:42:16.640873+0000","last_unstale":"2024-04-21T08:44:41.251372+0000","last_undegraded":"2024-04-21T08:44:41.251372+0000","last_fullsized":"2024-04-21T08:44:41.251372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:00.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375293+0000","last_change":"2024-04-21T08:42:17.550438+0000","last_active":"2024-04-21T08:44:41.375293+0000","last_peered":"2024-04-21T08:44:41.375293+0000","last_clean":"2024-04-21T08:44:41.375293+0000","last_became_active":"2024-04-21T08:42:17.550015+0000","last_became_peered":"2024-04-21T08:42:17.550015+0000","last_unstale":"2024-04-21T08:44:41.375293+0000","last_undegraded":"2024-04-21T08:44:41.375293+0000","last_fullsized":"2024-04-21T08:44:41.375293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:00:29.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250709+0000","last_change":"2024-04-21T08:42:21.970498+0000","last_active":"2024-04-21T08:44:41.250709+0000","last_peered":"2024-04-21T08:44:41.250709+0000","last_clean":"2024-04-21T08:44:41.250709+0000","last_became_active":"2024-04-21T08:42:21.969819+0000","last_became_peered":"2024-04-21T08:42:21.969819+0000","last_unstale":"2024-04-21T08:44:41.250709+0000","last_undegraded":"2024-04-21T08:44:41.250709+0000","last_fullsized":"2024-04-21T08:44:41.250709+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:13:16.552293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356527+0000","last_change":"2024-04-21T08:42:16.637146+0000","last_active":"2024-04-21T08:44:41.356527+0000","last_peered":"2024-04-21T08:44:41.356527+0000","last_clean":"2024-04-21T08:44:41.356527+0000","last_became_active":"2024-04-21T08:42:16.636781+0000","last_became_peered":"2024-04-21T08:42:16.636781+0000","last_unstale":"2024-04-21T08:44:41.356527+0000","last_undegraded":"2024-04-21T08:44:41.356527+0000","last_fullsized":"2024-04-21T08:44:41.356527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:39:27.641236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368622+0000","last_change":"2024-04-21T08:42:17.542325+0000","last_active":"2024-04-21T08:44:41.368622+0000","last_peered":"2024-04-21T08:44:41.368622+0000","last_clean":"2024-04-21T08:44:41.368622+0000","last_became_active":"2024-04-21T08:42:17.541103+0000","last_became_peered":"2024-04-21T08:42:17.541103+0000","last_unstale":"2024-04-21T08:44:41.368622+0000","last_undegraded":"2024-04-21T08:44:41.368622+0000","last_fullsized":"2024-04-21T08:44:41.368622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:46.903250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375702+0000","last_change":"2024-04-21T08:42:21.671146+0000","last_active":"2024-04-21T08:44:41.375702+0000","last_peered":"2024-04-21T08:44:41.375702+0000","last_clean":"2024-04-21T08:44:41.375702+0000","last_became_active":"2024-04-21T08:42:21.670561+0000","last_became_peered":"2024-04-21T08:42:21.670561+0000","last_unstale":"2024-04-21T08:44:41.375702+0000","last_undegraded":"2024-04-21T08:44:41.375702+0000","last_fullsized":"2024-04-21T08:44:41.375702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:38:28.028976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375066+0000","last_change":"2024-04-21T08:42:16.645132+0000","last_active":"2024-04-21T08:44:41.375066+0000","last_peered":"2024-04-21T08:44:41.375066+0000","last_clean":"2024-04-21T08:44:41.375066+0000","last_became_active":"2024-04-21T08:42:16.644786+0000","last_became_peered":"2024-04-21T08:42:16.644786+0000","last_unstale":"2024-04-21T08:44:41.375066+0000","last_undegraded":"2024-04-21T08:44:41.375066+0000","last_fullsized":"2024-04-21T08:44:41.375066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:08.298498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368567+0000","last_change":"2024-04-21T08:42:17.550424+0000","last_active":"2024-04-21T08:44:41.368567+0000","last_peered":"2024-04-21T08:44:41.368567+0000","last_clean":"2024-04-21T08:44:41.368567+0000","last_became_active":"2024-04-21T08:42:17.550077+0000","last_became_peered":"2024-04-21T08:42:17.550077+0000","last_unstale":"2024-04-21T08:44:41.368567+0000","last_undegraded":"2024-04-21T08:44:41.368567+0000","last_fullsized":"2024-04-21T08:44:41.368567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:17:07.016204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375856+0000","last_change":"2024-04-21T08:42:21.970652+0000","last_active":"2024-04-21T08:44:41.375856+0000","last_peered":"2024-04-21T08:44:41.375856+0000","last_clean":"2024-04-21T08:44:41.375856+0000","last_became_active":"2024-04-21T08:42:21.970251+0000","last_became_peered":"2024-04-21T08:42:21.970251+0000","last_unstale":"2024-04-21T08:44:41.375856+0000","last_undegraded":"2024-04-21T08:44:41.375856+0000","last_fullsized":"2024-04-21T08:44:41.375856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:45.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365285+0000","last_change":"2024-04-21T08:42:17.542820+0000","last_active":"2024-04-21T08:44:41.365285+0000","last_peered":"2024-04-21T08:44:41.365285+0000","last_clean":"2024-04-21T08:44:41.365285+0000","last_became_active":"2024-04-21T08:42:17.541320+0000","last_became_peered":"2024-04-21T08:42:17.541320+0000","last_unstale":"2024-04-21T08:44:41.365285+0000","last_undegraded":"2024-04-21T08:44:41.365285+0000","last_fullsized":"2024-04-21T08:44:41.365285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:23:06.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375209+0000","last_change":"2024-04-21T08:42:16.636950+0000","last_active":"2024-04-21T08:44:41.375209+0000","last_peered":"2024-04-21T08:44:41.375209+0000","last_clean":"2024-04-21T08:44:41.375209+0000","last_became_active":"2024-04-21T08:42:16.636590+0000","last_became_peered":"2024-04-21T08:42:16.636590+0000","last_unstale":"2024-04-21T08:44:41.375209+0000","last_undegraded":"2024-04-21T08:44:41.375209+0000","last_fullsized":"2024-04-21T08:44:41.375209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:55.693681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374884+0000","last_change":"2024-04-21T08:42:21.660404+0000","last_active":"2024-04-21T08:44:41.374884+0000","last_peered":"2024-04-21T08:44:41.374884+0000","last_clean":"2024-04-21T08:44:41.374884+0000","last_became_active":"2024-04-21T08:42:21.659999+0000","last_became_peered":"2024-04-21T08:42:21.659999+0000","last_unstale":"2024-04-21T08:44:41.374884+0000","last_undegraded":"2024-04-21T08:44:41.374884+0000","last_fullsized":"2024-04-21T08:44:41.374884+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:53:32.328873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356239+0000","last_change":"2024-04-21T08:43:10.984361+0000","last_active":"2024-04-21T08:44:41.356239+0000","last_peered":"2024-04-21T08:44:41.356239+0000","last_clean":"2024-04-21T08:44:41.356239+0000","last_became_active":"2024-04-21T08:43:10.984046+0000","last_became_peered":"2024-04-21T08:43:10.984046+0000","last_unstale":"2024-04-21T08:44:41.356239+0000","last_undegraded":"2024-04-21T08:44:41.356239+0000","last_fullsized":"2024-04-21T08:44:41.356239+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:33:23.154630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251236+0000","last_change":"2024-04-21T08:42:16.638993+0000","last_active":"2024-04-21T08:44:41.251236+0000","last_peered":"2024-04-21T08:44:41.251236+0000","last_clean":"2024-04-21T08:44:41.251236+0000","last_became_active":"2024-04-21T08:42:16.638617+0000","last_became_peered":"2024-04-21T08:42:16.638617+0000","last_unstale":"2024-04-21T08:44:41.251236+0000","last_undegraded":"2024-04-21T08:44:41.251236+0000","last_fullsized":"2024-04-21T08:44:41.251236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:14:28.271433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374510+0000","last_change":"2024-04-21T08:41:00.044984+0000","last_active":"2024-04-21T08:44:41.374510+0000","last_peered":"2024-04-21T08:44:41.374510+0000","last_clean":"2024-04-21T08:44:41.374510+0000","last_became_active":"2024-04-21T08:41:00.044485+0000","last_became_peered":"2024-04-21T08:41:00.044485+0000","last_unstale":"2024-04-21T08:44:41.374510+0000","last_undegraded":"2024-04-21T08:44:41.374510+0000","last_fullsized":"2024-04-21T08:44:41.374510+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.370031+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:01.031209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365508+0000","last_change":"2024-04-21T08:42:21.971045+0000","last_active":"2024-04-21T08:44:41.365508+0000","last_peered":"2024-04-21T08:44:41.365508+0000","last_clean":"2024-04-21T08:44:41.365508+0000","last_became_active":"2024-04-21T08:42:21.970577+0000","last_became_peered":"2024-04-21T08:42:21.970577+0000","last_unstale":"2024-04-21T08:44:41.365508+0000","last_undegraded":"2024-04-21T08:44:41.365508+0000","last_fullsized":"2024-04-21T08:44:41.365508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:55.681662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374964+0000","last_change":"2024-04-21T08:42:17.535570+0000","last_active":"2024-04-21T08:44:41.374964+0000","last_peered":"2024-04-21T08:44:41.374964+0000","last_clean":"2024-04-21T08:44:41.374964+0000","last_became_active":"2024-04-21T08:42:17.535217+0000","last_became_peered":"2024-04-21T08:42:17.535217+0000","last_unstale":"2024-04-21T08:44:41.374964+0000","last_undegraded":"2024-04-21T08:44:41.374964+0000","last_fullsized":"2024-04-21T08:44:41.374964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:49.510750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374650+0000","last_change":"2024-04-21T08:43:10.792548+0000","last_active":"2024-04-21T08:44:41.374650+0000","last_peered":"2024-04-21T08:44:41.374650+0000","last_clean":"2024-04-21T08:44:41.374650+0000","last_became_active":"2024-04-21T08:43:10.792165+0000","last_became_peered":"2024-04-21T08:43:10.792165+0000","last_unstale":"2024-04-21T08:44:41.374650+0000","last_undegraded":"2024-04-21T08:44:41.374650+0000","last_fullsized":"2024-04-21T08:44:41.374650+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:35.980536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374872+0000","last_change":"2024-04-21T08:42:21.661393+0000","last_active":"2024-04-21T08:44:41.374872+0000","last_peered":"2024-04-21T08:44:41.374872+0000","last_clean":"2024-04-21T08:44:41.374872+0000","last_became_active":"2024-04-21T08:42:21.660201+0000","last_became_peered":"2024-04-21T08:42:21.660201+0000","last_unstale":"2024-04-21T08:44:41.374872+0000","last_undegraded":"2024-04-21T08:44:41.374872+0000","last_fullsized":"2024-04-21T08:44:41.374872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:13.393389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250941+0000","last_change":"2024-04-21T08:42:16.650507+0000","last_active":"2024-04-21T08:44:41.250941+0000","last_peered":"2024-04-21T08:44:41.250941+0000","last_clean":"2024-04-21T08:44:41.250941+0000","last_became_active":"2024-04-21T08:42:16.650217+0000","last_became_peered":"2024-04-21T08:42:16.650217+0000","last_unstale":"2024-04-21T08:44:41.250941+0000","last_undegraded":"2024-04-21T08:44:41.250941+0000","last_fullsized":"2024-04-21T08:44:41.250941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:13:26.785092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375997+0000","last_change":"2024-04-21T08:42:17.530420+0000","last_active":"2024-04-21T08:44:41.375997+0000","last_peered":"2024-04-21T08:44:41.375997+0000","last_clean":"2024-04-21T08:44:41.375997+0000","last_became_active":"2024-04-21T08:42:17.529920+0000","last_became_peered":"2024-04-21T08:42:17.529920+0000","last_unstale":"2024-04-21T08:44:41.375997+0000","last_undegraded":"2024-04-21T08:44:41.375997+0000","last_fullsized":"2024-04-21T08:44:41.375997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:24.250197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249760+0000","last_change":"2024-04-21T08:42:21.970586+0000","last_active":"2024-04-21T08:44:41.249760+0000","last_peered":"2024-04-21T08:44:41.249760+0000","last_clean":"2024-04-21T08:44:41.249760+0000","last_became_active":"2024-04-21T08:42:21.970132+0000","last_became_peered":"2024-04-21T08:42:21.970132+0000","last_unstale":"2024-04-21T08:44:41.249760+0000","last_undegraded":"2024-04-21T08:44:41.249760+0000","last_fullsized":"2024-04-21T08:44:41.249760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:11.074370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'43","reported_seq":100,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.483501+0000","last_change":"2024-04-21T08:42:16.647916+0000","last_active":"2024-04-21T08:44:41.483501+0000","last_peered":"2024-04-21T08:44:41.483501+0000","last_clean":"2024-04-21T08:44:41.483501+0000","last_became_active":"2024-04-21T08:42:16.647476+0000","last_became_peered":"2024-04-21T08:42:16.647476+0000","last_unstale":"2024-04-21T08:44:41.483501+0000","last_undegraded":"2024-04-21T08:44:41.483501+0000","last_fullsized":"2024-04-21T08:44:41.483501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:27.242153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251070+0000","last_change":"2024-04-21T08:42:17.546144+0000","last_active":"2024-04-21T08:44:41.251070+0000","last_peered":"2024-04-21T08:44:41.251070+0000","last_clean":"2024-04-21T08:44:41.251070+0000","last_became_active":"2024-04-21T08:42:17.545730+0000","last_became_peered":"2024-04-21T08:42:17.545730+0000","last_unstale":"2024-04-21T08:44:41.251070+0000","last_undegraded":"2024-04-21T08:44:41.251070+0000","last_fullsized":"2024-04-21T08:44:41.251070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:42.802309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374618+0000","last_change":"2024-04-21T08:42:21.655802+0000","last_active":"2024-04-21T08:44:41.374618+0000","last_peered":"2024-04-21T08:44:41.374618+0000","last_clean":"2024-04-21T08:44:41.374618+0000","last_became_active":"2024-04-21T08:42:21.655379+0000","last_became_peered":"2024-04-21T08:42:21.655379+0000","last_unstale":"2024-04-21T08:44:41.374618+0000","last_undegraded":"2024-04-21T08:44:41.374618+0000","last_fullsized":"2024-04-21T08:44:41.374618+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:49.486057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251462+0000","last_change":"2024-04-21T08:42:16.642517+0000","last_active":"2024-04-21T08:44:41.251462+0000","last_peered":"2024-04-21T08:44:41.251462+0000","last_clean":"2024-04-21T08:44:41.251462+0000","last_became_active":"2024-04-21T08:42:16.641539+0000","last_became_peered":"2024-04-21T08:42:16.641539+0000","last_unstale":"2024-04-21T08:44:41.251462+0000","last_undegraded":"2024-04-21T08:44:41.251462+0000","last_fullsized":"2024-04-21T08:44:41.251462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:53.912199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418089+0000","last_change":"2024-04-21T08:42:17.550172+0000","last_active":"2024-04-21T08:44:41.418089+0000","last_peered":"2024-04-21T08:44:41.418089+0000","last_clean":"2024-04-21T08:44:41.418089+0000","last_became_active":"2024-04-21T08:42:17.549824+0000","last_became_peered":"2024-04-21T08:42:17.549824+0000","last_unstale":"2024-04-21T08:44:41.418089+0000","last_undegraded":"2024-04-21T08:44:41.418089+0000","last_fullsized":"2024-04-21T08:44:41.418089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:59.553645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251154+0000","last_change":"2024-04-21T08:44:11.774778+0000","last_active":"2024-04-21T08:44:41.251154+0000","last_peered":"2024-04-21T08:44:41.251154+0000","last_clean":"2024-04-21T08:44:41.251154+0000","last_became_active":"2024-04-21T08:44:11.774190+0000","last_became_peered":"2024-04-21T08:44:11.774190+0000","last_unstale":"2024-04-21T08:44:41.251154+0000","last_undegraded":"2024-04-21T08:44:41.251154+0000","last_fullsized":"2024-04-21T08:44:41.251154+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:06.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374515+0000","last_change":"2024-04-21T08:42:16.647985+0000","last_active":"2024-04-21T08:44:41.374515+0000","last_peered":"2024-04-21T08:44:41.374515+0000","last_clean":"2024-04-21T08:44:41.374515+0000","last_became_active":"2024-04-21T08:42:16.647577+0000","last_became_peered":"2024-04-21T08:42:16.647577+0000","last_unstale":"2024-04-21T08:44:41.374515+0000","last_undegraded":"2024-04-21T08:44:41.374515+0000","last_fullsized":"2024-04-21T08:44:41.374515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:00.731708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250576+0000","last_change":"2024-04-21T08:42:17.541757+0000","last_active":"2024-04-21T08:44:41.250576+0000","last_peered":"2024-04-21T08:44:41.250576+0000","last_clean":"2024-04-21T08:44:41.250576+0000","last_became_active":"2024-04-21T08:42:17.539280+0000","last_became_peered":"2024-04-21T08:42:17.539280+0000","last_unstale":"2024-04-21T08:44:41.250576+0000","last_undegraded":"2024-04-21T08:44:41.250576+0000","last_fullsized":"2024-04-21T08:44:41.250576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:45:54.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356961+0000","last_change":"2024-04-21T08:42:21.664566+0000","last_active":"2024-04-21T08:44:41.356961+0000","last_peered":"2024-04-21T08:44:41.356961+0000","last_clean":"2024-04-21T08:44:41.356961+0000","last_became_active":"2024-04-21T08:42:21.664136+0000","last_became_peered":"2024-04-21T08:42:21.664136+0000","last_unstale":"2024-04-21T08:44:41.356961+0000","last_undegraded":"2024-04-21T08:44:41.356961+0000","last_fullsized":"2024-04-21T08:44:41.356961+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:19.856577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249918+0000","last_change":"2024-04-21T08:42:16.650664+0000","last_active":"2024-04-21T08:44:41.249918+0000","last_peered":"2024-04-21T08:44:41.249918+0000","last_clean":"2024-04-21T08:44:41.249918+0000","last_became_active":"2024-04-21T08:42:16.650339+0000","last_became_peered":"2024-04-21T08:42:16.650339+0000","last_unstale":"2024-04-21T08:44:41.249918+0000","last_undegraded":"2024-04-21T08:44:41.249918+0000","last_fullsized":"2024-04-21T08:44:41.249918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:59.565455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374844+0000","last_change":"2024-04-21T08:42:17.538142+0000","last_active":"2024-04-21T08:44:41.374844+0000","last_peered":"2024-04-21T08:44:41.374844+0000","last_clean":"2024-04-21T08:44:41.374844+0000","last_became_active":"2024-04-21T08:42:17.537793+0000","last_became_peered":"2024-04-21T08:42:17.537793+0000","last_unstale":"2024-04-21T08:44:41.374844+0000","last_undegraded":"2024-04-21T08:44:41.374844+0000","last_fullsized":"2024-04-21T08:44:41.374844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:11.059294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251170+0000","last_change":"2024-04-21T08:42:21.670642+0000","last_active":"2024-04-21T08:44:41.251170+0000","last_peered":"2024-04-21T08:44:41.251170+0000","last_clean":"2024-04-21T08:44:41.251170+0000","last_became_active":"2024-04-21T08:42:21.670128+0000","last_became_peered":"2024-04-21T08:42:21.670128+0000","last_unstale":"2024-04-21T08:44:41.251170+0000","last_undegraded":"2024-04-21T08:44:41.251170+0000","last_fullsized":"2024-04-21T08:44:41.251170+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:51:28.604381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375278+0000","last_change":"2024-04-21T08:43:10.792114+0000","last_active":"2024-04-21T08:44:41.375278+0000","last_peered":"2024-04-21T08:44:41.375278+0000","last_clean":"2024-04-21T08:44:41.375278+0000","last_became_active":"2024-04-21T08:43:10.791664+0000","last_became_peered":"2024-04-21T08:43:10.791664+0000","last_unstale":"2024-04-21T08:44:41.375278+0000","last_undegraded":"2024-04-21T08:44:41.375278+0000","last_fullsized":"2024-04-21T08:44:41.375278+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:07.317804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250083+0000","last_change":"2024-04-21T08:42:17.547326+0000","last_active":"2024-04-21T08:44:41.250083+0000","last_peered":"2024-04-21T08:44:41.250083+0000","last_clean":"2024-04-21T08:44:41.250083+0000","last_became_active":"2024-04-21T08:42:17.546920+0000","last_became_peered":"2024-04-21T08:42:17.546920+0000","last_unstale":"2024-04-21T08:44:41.250083+0000","last_undegraded":"2024-04-21T08:44:41.250083+0000","last_fullsized":"2024-04-21T08:44:41.250083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:43.350877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251167+0000","last_change":"2024-04-21T08:42:21.971147+0000","last_active":"2024-04-21T08:44:41.251167+0000","last_peered":"2024-04-21T08:44:41.251167+0000","last_clean":"2024-04-21T08:44:41.251167+0000","last_became_active":"2024-04-21T08:42:21.970001+0000","last_became_peered":"2024-04-21T08:42:21.970001+0000","last_unstale":"2024-04-21T08:44:41.251167+0000","last_undegraded":"2024-04-21T08:44:41.251167+0000","last_fullsized":"2024-04-21T08:44:41.251167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:01:44.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250876+0000","last_change":"2024-04-21T08:42:16.643371+0000","last_active":"2024-04-21T08:44:41.250876+0000","last_peered":"2024-04-21T08:44:41.250876+0000","last_clean":"2024-04-21T08:44:41.250876+0000","last_became_active":"2024-04-21T08:42:16.642906+0000","last_became_peered":"2024-04-21T08:42:16.642906+0000","last_unstale":"2024-04-21T08:44:41.250876+0000","last_undegraded":"2024-04-21T08:44:41.250876+0000","last_fullsized":"2024-04-21T08:44:41.250876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:58.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375124+0000","last_change":"2024-04-21T08:42:17.545185+0000","last_active":"2024-04-21T08:44:41.375124+0000","last_peered":"2024-04-21T08:44:41.375124+0000","last_clean":"2024-04-21T08:44:41.375124+0000","last_became_active":"2024-04-21T08:42:17.544757+0000","last_became_peered":"2024-04-21T08:42:17.544757+0000","last_unstale":"2024-04-21T08:44:41.375124+0000","last_undegraded":"2024-04-21T08:44:41.375124+0000","last_fullsized":"2024-04-21T08:44:41.375124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:36:18.968041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374923+0000","last_change":"2024-04-21T08:42:21.662541+0000","last_active":"2024-04-21T08:44:41.374923+0000","last_peered":"2024-04-21T08:44:41.374923+0000","last_clean":"2024-04-21T08:44:41.374923+0000","last_became_active":"2024-04-21T08:42:21.662144+0000","last_became_peered":"2024-04-21T08:42:21.662144+0000","last_unstale":"2024-04-21T08:44:41.374923+0000","last_undegraded":"2024-04-21T08:44:41.374923+0000","last_fullsized":"2024-04-21T08:44:41.374923+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:04.347722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375374+0000","last_change":"2024-04-21T08:42:16.638601+0000","last_active":"2024-04-21T08:44:41.375374+0000","last_peered":"2024-04-21T08:44:41.375374+0000","last_clean":"2024-04-21T08:44:41.375374+0000","last_became_active":"2024-04-21T08:42:16.638211+0000","last_became_peered":"2024-04-21T08:42:16.638211+0000","last_unstale":"2024-04-21T08:44:41.375374+0000","last_undegraded":"2024-04-21T08:44:41.375374+0000","last_fullsized":"2024-04-21T08:44:41.375374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:02.633086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375257+0000","last_change":"2024-04-21T08:42:17.545495+0000","last_active":"2024-04-21T08:44:41.375257+0000","last_peered":"2024-04-21T08:44:41.375257+0000","last_clean":"2024-04-21T08:44:41.375257+0000","last_became_active":"2024-04-21T08:42:17.545159+0000","last_became_peered":"2024-04-21T08:42:17.545159+0000","last_unstale":"2024-04-21T08:44:41.375257+0000","last_undegraded":"2024-04-21T08:44:41.375257+0000","last_fullsized":"2024-04-21T08:44:41.375257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:27.150277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417564+0000","last_change":"2024-04-21T08:42:21.656890+0000","last_active":"2024-04-21T08:44:41.417564+0000","last_peered":"2024-04-21T08:44:41.417564+0000","last_clean":"2024-04-21T08:44:41.417564+0000","last_became_active":"2024-04-21T08:42:21.656475+0000","last_became_peered":"2024-04-21T08:42:21.656475+0000","last_unstale":"2024-04-21T08:44:41.417564+0000","last_undegraded":"2024-04-21T08:44:41.417564+0000","last_fullsized":"2024-04-21T08:44:41.417564+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:46.138591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'39","reported_seq":81,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.478489+0000","last_change":"2024-04-21T08:42:16.638244+0000","last_active":"2024-04-21T08:44:41.478489+0000","last_peered":"2024-04-21T08:44:41.478489+0000","last_clean":"2024-04-21T08:44:41.478489+0000","last_became_active":"2024-04-21T08:42:16.637883+0000","last_became_peered":"2024-04-21T08:42:16.637883+0000","last_unstale":"2024-04-21T08:44:41.478489+0000","last_undegraded":"2024-04-21T08:44:41.478489+0000","last_fullsized":"2024-04-21T08:44:41.478489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:20.888301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374936+0000","last_change":"2024-04-21T08:42:17.548032+0000","last_active":"2024-04-21T08:44:41.374936+0000","last_peered":"2024-04-21T08:44:41.374936+0000","last_clean":"2024-04-21T08:44:41.374936+0000","last_became_active":"2024-04-21T08:42:17.547660+0000","last_became_peered":"2024-04-21T08:42:17.547660+0000","last_unstale":"2024-04-21T08:44:41.374936+0000","last_undegraded":"2024-04-21T08:44:41.374936+0000","last_fullsized":"2024-04-21T08:44:41.374936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:14.710759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365005+0000","last_change":"2024-04-21T08:42:21.659286+0000","last_active":"2024-04-21T08:44:41.365005+0000","last_peered":"2024-04-21T08:44:41.365005+0000","last_clean":"2024-04-21T08:44:41.365005+0000","last_became_active":"2024-04-21T08:42:21.658844+0000","last_became_peered":"2024-04-21T08:42:21.658844+0000","last_unstale":"2024-04-21T08:44:41.365005+0000","last_undegraded":"2024-04-21T08:44:41.365005+0000","last_fullsized":"2024-04-21T08:44:41.365005+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:28.742646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376281+0000","last_change":"2024-04-21T08:42:16.643354+0000","last_active":"2024-04-21T08:44:41.376281+0000","last_peered":"2024-04-21T08:44:41.376281+0000","last_clean":"2024-04-21T08:44:41.376281+0000","last_became_active":"2024-04-21T08:42:16.642967+0000","last_became_peered":"2024-04-21T08:42:16.642967+0000","last_unstale":"2024-04-21T08:44:41.376281+0000","last_undegraded":"2024-04-21T08:44:41.376281+0000","last_fullsized":"2024-04-21T08:44:41.376281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:06:03.495181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375409+0000","last_change":"2024-04-21T08:42:17.539551+0000","last_active":"2024-04-21T08:44:41.375409+0000","last_peered":"2024-04-21T08:44:41.375409+0000","last_clean":"2024-04-21T08:44:41.375409+0000","last_became_active":"2024-04-21T08:42:17.539202+0000","last_became_peered":"2024-04-21T08:42:17.539202+0000","last_unstale":"2024-04-21T08:44:41.375409+0000","last_undegraded":"2024-04-21T08:44:41.375409+0000","last_fullsized":"2024-04-21T08:44:41.375409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:37.109421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375023+0000","last_change":"2024-04-21T08:42:17.544582+0000","last_active":"2024-04-21T08:44:41.375023+0000","last_peered":"2024-04-21T08:44:41.375023+0000","last_clean":"2024-04-21T08:44:41.375023+0000","last_became_active":"2024-04-21T08:42:17.543800+0000","last_became_peered":"2024-04-21T08:42:17.543800+0000","last_unstale":"2024-04-21T08:44:41.375023+0000","last_undegraded":"2024-04-21T08:44:41.375023+0000","last_fullsized":"2024-04-21T08:44:41.375023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:44:04.107207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374756+0000","last_change":"2024-04-21T08:42:17.543788+0000","last_active":"2024-04-21T08:44:41.374756+0000","last_peered":"2024-04-21T08:44:41.374756+0000","last_clean":"2024-04-21T08:44:41.374756+0000","last_became_active":"2024-04-21T08:42:17.540939+0000","last_became_peered":"2024-04-21T08:42:17.540939+0000","last_unstale":"2024-04-21T08:44:41.374756+0000","last_undegraded":"2024-04-21T08:44:41.374756+0000","last_fullsized":"2024-04-21T08:44:41.374756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:59.275098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374942+0000","last_change":"2024-04-21T08:42:17.540605+0000","last_active":"2024-04-21T08:44:41.374942+0000","last_peered":"2024-04-21T08:44:41.374942+0000","last_clean":"2024-04-21T08:44:41.374942+0000","last_became_active":"2024-04-21T08:42:17.540073+0000","last_became_peered":"2024-04-21T08:42:17.540073+0000","last_unstale":"2024-04-21T08:44:41.374942+0000","last_undegraded":"2024-04-21T08:44:41.374942+0000","last_fullsized":"2024-04-21T08:44:41.374942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:02:29.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417931+0000","last_change":"2024-04-21T08:42:17.551608+0000","last_active":"2024-04-21T08:44:41.417931+0000","last_peered":"2024-04-21T08:44:41.417931+0000","last_clean":"2024-04-21T08:44:41.417931+0000","last_became_active":"2024-04-21T08:42:17.551117+0000","last_became_peered":"2024-04-21T08:42:17.551117+0000","last_unstale":"2024-04-21T08:44:41.417931+0000","last_undegraded":"2024-04-21T08:44:41.417931+0000","last_fullsized":"2024-04-21T08:44:41.417931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:18.391928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375875+0000","last_change":"2024-04-21T08:42:17.536509+0000","last_active":"2024-04-21T08:44:41.375875+0000","last_peered":"2024-04-21T08:44:41.375875+0000","last_clean":"2024-04-21T08:44:41.375875+0000","last_became_active":"2024-04-21T08:42:17.536146+0000","last_became_peered":"2024-04-21T08:42:17.536146+0000","last_unstale":"2024-04-21T08:44:41.375875+0000","last_undegraded":"2024-04-21T08:44:41.375875+0000","last_fullsized":"2024-04-21T08:44:41.375875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:29.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375199+0000","last_change":"2024-04-21T08:42:17.543786+0000","last_active":"2024-04-21T08:44:41.375199+0000","last_peered":"2024-04-21T08:44:41.375199+0000","last_clean":"2024-04-21T08:44:41.375199+0000","last_became_active":"2024-04-21T08:42:17.543435+0000","last_became_peered":"2024-04-21T08:42:17.543435+0000","last_unstale":"2024-04-21T08:44:41.375199+0000","last_undegraded":"2024-04-21T08:44:41.375199+0000","last_fullsized":"2024-04-21T08:44:41.375199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375616+0000","last_change":"2024-04-21T08:42:17.541611+0000","last_active":"2024-04-21T08:44:41.375616+0000","last_peered":"2024-04-21T08:44:41.375616+0000","last_clean":"2024-04-21T08:44:41.375616+0000","last_became_active":"2024-04-21T08:42:17.541259+0000","last_became_peered":"2024-04-21T08:42:17.541259+0000","last_unstale":"2024-04-21T08:44:41.375616+0000","last_undegraded":"2024-04-21T08:44:41.375616+0000","last_fullsized":"2024-04-21T08:44:41.375616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:20.900629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356654+0000","last_change":"2024-04-21T08:44:11.774062+0000","last_active":"2024-04-21T08:44:41.356654+0000","last_peered":"2024-04-21T08:44:41.356654+0000","last_clean":"2024-04-21T08:44:41.356654+0000","last_became_active":"2024-04-21T08:44:11.773703+0000","last_became_peered":"2024-04-21T08:44:11.773703+0000","last_unstale":"2024-04-21T08:44:41.356654+0000","last_undegraded":"2024-04-21T08:44:41.356654+0000","last_fullsized":"2024-04-21T08:44:41.356654+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:10.739068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365197+0000","last_change":"2024-04-21T08:42:17.545526+0000","last_active":"2024-04-21T08:44:41.365197+0000","last_peered":"2024-04-21T08:44:41.365197+0000","last_clean":"2024-04-21T08:44:41.365197+0000","last_became_active":"2024-04-21T08:42:17.545135+0000","last_became_peered":"2024-04-21T08:42:17.545135+0000","last_unstale":"2024-04-21T08:44:41.365197+0000","last_undegraded":"2024-04-21T08:44:41.365197+0000","last_fullsized":"2024-04-21T08:44:41.365197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:37:58.038459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250029+0000","last_change":"2024-04-21T08:42:17.547063+0000","last_active":"2024-04-21T08:44:41.250029+0000","last_peered":"2024-04-21T08:44:41.250029+0000","last_clean":"2024-04-21T08:44:41.250029+0000","last_became_active":"2024-04-21T08:42:17.546689+0000","last_became_peered":"2024-04-21T08:42:17.546689+0000","last_unstale":"2024-04-21T08:44:41.250029+0000","last_undegraded":"2024-04-21T08:44:41.250029+0000","last_fullsized":"2024-04-21T08:44:41.250029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:26.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375261+0000","last_change":"2024-04-21T08:42:17.542545+0000","last_active":"2024-04-21T08:44:41.375261+0000","last_peered":"2024-04-21T08:44:41.375261+0000","last_clean":"2024-04-21T08:44:41.375261+0000","last_became_active":"2024-04-21T08:42:17.542147+0000","last_became_peered":"2024-04-21T08:42:17.542147+0000","last_unstale":"2024-04-21T08:44:41.375261+0000","last_undegraded":"2024-04-21T08:44:41.375261+0000","last_fullsized":"2024-04-21T08:44:41.375261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:15.064110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376582+0000","last_change":"2024-04-21T08:42:17.541262+0000","last_active":"2024-04-21T08:44:41.376582+0000","last_peered":"2024-04-21T08:44:41.376582+0000","last_clean":"2024-04-21T08:44:41.376582+0000","last_became_active":"2024-04-21T08:42:17.540817+0000","last_became_peered":"2024-04-21T08:42:17.540817+0000","last_unstale":"2024-04-21T08:44:41.376582+0000","last_undegraded":"2024-04-21T08:44:41.376582+0000","last_fullsized":"2024-04-21T08:44:41.376582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:22:59.597680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375116+0000","last_change":"2024-04-21T08:42:17.551109+0000","last_active":"2024-04-21T08:44:41.375116+0000","last_peered":"2024-04-21T08:44:41.375116+0000","last_clean":"2024-04-21T08:44:41.375116+0000","last_became_active":"2024-04-21T08:42:17.550104+0000","last_became_peered":"2024-04-21T08:42:17.550104+0000","last_unstale":"2024-04-21T08:44:41.375116+0000","last_undegraded":"2024-04-21T08:44:41.375116+0000","last_fullsized":"2024-04-21T08:44:41.375116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:22:19.361752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107298,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595682,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":350,"num_read_kb":280,"num_write":201,"num_write_kb":1411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":1787046,"data_compressed":80802,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":532412,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612655,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36496,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708656,"statfs":{"total":95995035648,"available":95957663744,"internally_reserved":0,"allocated":1183744,"data_stored":1314391,"data_compressed":73179,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776181,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1069056,"data_stored":930306,"data_compressed":48872,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":926232,"data_compressed":48872,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":1127458,"data_compressed":65492,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528316,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1105920,"data_stored":598384,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1036288,"data_stored":528406,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":598418,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1053236,"data_compressed":55028,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":532434,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1118208,"data_stored":1254562,"data_compressed":71648,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70102,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":201862,"data_compressed":7788,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:44:43.950 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-21T08:44:43.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:44:44.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: [21/Apr/2024:08:44:42] ENGINE Bus STARTING 2024-04-21T08:44:44.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:44:44.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: [21/Apr/2024:08:44:42] ENGINE Serving on https://172.21.15.7:7150 2024-04-21T08:44:44.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: [21/Apr/2024:08:44:42] ENGINE Serving on http://172.21.15.7:8765 2024-04-21T08:44:44.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: [21/Apr/2024:08:44:42] ENGINE Bus STARTED 2024-04-21T08:44:44.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1193411548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:44.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:43 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.095+0000 7f0f25567640 1 -- 172.21.15.7:0/1327489699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f201011b0 msgr2=0x7f0f20101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.095+0000 7f0f25567640 1 --2- 172.21.15.7:0/1327489699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f201011b0 0x7f0f20101610 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009900 tx=0x7f0f1402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:44.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.096+0000 7f0f25567640 1 -- 172.21.15.7:0/1327489699 shutdown_connections 2024-04-21T08:44:44.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.096+0000 7f0f25567640 1 --2- 172.21.15.7:0/1327489699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f20101b50 0x7f0f20101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.096+0000 7f0f25567640 1 --2- 172.21.15.7:0/1327489699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f201011b0 0x7f0f20101610 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.096+0000 7f0f25567640 1 --2- 172.21.15.7:0/1327489699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f20100850 0x7f0f20100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.096+0000 7f0f25567640 1 -- 172.21.15.7:0/1327489699 >> 172.21.15.7:0/1327489699 conn(0x7f0f200fc3f0 msgr2=0x7f0f200fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.096+0000 7f0f25567640 1 -- 172.21.15.7:0/1327489699 shutdown_connections 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.096+0000 7f0f25567640 1 -- 172.21.15.7:0/1327489699 wait complete. 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.096+0000 7f0f25567640 1 Processor -- start 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.096+0000 7f0f25567640 1 -- start start 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f25567640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f20100850 0x7f0f2019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f25567640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f201011b0 0x7f0f2019a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f25567640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f20101b50 0x7f0f201a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f25567640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2010fcd0 con 0x7f0f20101b50 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f25567640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2010fb50 con 0x7f0f201011b0 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f25567640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2010fe50 con 0x7f0f20100850 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f1f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f20101b50 0x7f0f201a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f1effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f20100850 0x7f0f2019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f1f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f20101b50 0x7f0f201a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53720/0 (socket says 172.21.15.7:53720) 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f1f7fe640 1 -- 172.21.15.7:0/3270321428 learned_addr learned my addr 172.21.15.7:0/3270321428 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f1e7fc640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f201011b0 0x7f0f2019a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f1e7fc640 1 -- 172.21.15.7:0/3270321428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f20100850 msgr2=0x7f0f2019a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f1e7fc640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f20100850 0x7f0f2019a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f1e7fc640 1 -- 172.21.15.7:0/3270321428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f20101b50 msgr2=0x7f0f201a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f1e7fc640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f20101b50 0x7f0f201a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f1e7fc640 1 -- 172.21.15.7:0/3270321428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f201a1f90 con 0x7f0f201011b0 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.097+0000 7f0f1e7fc640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f201011b0 0x7f0f2019a8b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0f14000c00 tx=0x7f0f14005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.098+0000 7f0f07fff640 1 -- 172.21.15.7:0/3270321428 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14043020 con 0x7f0f201011b0 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.098+0000 7f0f25567640 1 -- 172.21.15.7:0/3270321428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f201a2220 con 0x7f0f201011b0 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.098+0000 7f0f07fff640 1 -- 172.21.15.7:0/3270321428 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f14004350 con 0x7f0f201011b0 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.098+0000 7f0f07fff640 1 -- 172.21.15.7:0/3270321428 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14003b50 con 0x7f0f201011b0 2024-04-21T08:44:44.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.098+0000 7f0f25567640 1 -- 172.21.15.7:0/3270321428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f201a2770 con 0x7f0f201011b0 2024-04-21T08:44:44.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.099+0000 7f0f25567640 1 -- 172.21.15.7:0/3270321428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f2010b7f0 con 0x7f0f201011b0 2024-04-21T08:44:44.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.101+0000 7f0f07fff640 1 -- 172.21.15.7:0/3270321428 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 138765+0+0 (secure 0 0 0) 0x7f0f14046020 con 0x7f0f201011b0 2024-04-21T08:44:44.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.101+0000 7f0f07fff640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0f080a3d20 0x7f0f080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.102+0000 7f0f1effd640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0f080a3d20 0x7f0f080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.102+0000 7f0f07fff640 1 -- 172.21.15.7:0/3270321428 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f0f140f31c0 con 0x7f0f201011b0 2024-04-21T08:44:44.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.102+0000 7f0f1effd640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0f080a3d20 0x7f0f080a61e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0f100095f0 tx=0x7f0f10005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:44.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.104+0000 7f0f07fff640 1 -- 172.21.15.7:0/3270321428 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f1404d020 con 0x7f0f201011b0 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: [21/Apr/2024:08:44:42] ENGINE Bus STARTING 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: [21/Apr/2024:08:44:42] ENGINE Serving on https://172.21.15.7:7150 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: [21/Apr/2024:08:44:42] ENGINE Serving on http://172.21.15.7:8765 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: [21/Apr/2024:08:44:42] ENGINE Bus STARTED 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1193411548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:43 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: [21/Apr/2024:08:44:42] ENGINE Bus STARTING 2024-04-21T08:44:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:44:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: [21/Apr/2024:08:44:42] ENGINE Serving on https://172.21.15.7:7150 2024-04-21T08:44:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: [21/Apr/2024:08:44:42] ENGINE Serving on http://172.21.15.7:8765 2024-04-21T08:44:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: [21/Apr/2024:08:44:42] ENGINE Bus STARTED 2024-04-21T08:44:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1193411548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:43 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:44.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.324+0000 7f0f25567640 1 -- 172.21.15.7:0/3270321428 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0f20062390 con 0x7f0f080a3d20 2024-04-21T08:44:44.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.342+0000 7f0f07fff640 1 -- 172.21.15.7:0/3270321428 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339635 (secure 0 0 0) 0x7f0f20062390 con 0x7f0f080a3d20 2024-04-21T08:44:44.345 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:44.347 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-21T08:44:44.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.345+0000 7f0f25567640 1 -- 172.21.15.7:0/3270321428 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0f080a3d20 msgr2=0x7f0f080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.345+0000 7f0f25567640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0f080a3d20 0x7f0f080a61e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0f100095f0 tx=0x7f0f10005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:44.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.345+0000 7f0f25567640 1 -- 172.21.15.7:0/3270321428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f201011b0 msgr2=0x7f0f2019a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.345+0000 7f0f25567640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f201011b0 0x7f0f2019a8b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0f14000c00 tx=0x7f0f14005dc0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.346+0000 7f0f25567640 1 -- 172.21.15.7:0/3270321428 shutdown_connections 2024-04-21T08:44:44.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.346+0000 7f0f25567640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f20101b50 0x7f0f201a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.346+0000 7f0f25567640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f201011b0 0x7f0f2019a8b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.346+0000 7f0f25567640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0f080a3d20 0x7f0f080a61e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.346+0000 7f0f25567640 1 --2- 172.21.15.7:0/3270321428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f20100850 0x7f0f2019a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.346+0000 7f0f25567640 1 -- 172.21.15.7:0/3270321428 >> 172.21.15.7:0/3270321428 conn(0x7f0f200fc3f0 msgr2=0x7f0f200fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:44.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.346+0000 7f0f25567640 1 -- 172.21.15.7:0/3270321428 shutdown_connections 2024-04-21T08:44:44.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.346+0000 7f0f25567640 1 -- 172.21.15.7:0/3270321428 wait complete. 2024-04-21T08:44:44.357 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-21T08:44:43.262682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186284,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":486,"num_read_kb":399,"num_write":384,"num_write_kb":4351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":403472,"kb_used_data":12676,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124538352,"statfs":{"total":1151940427776,"available":1151527272448,"internally_reserved":0,"allocated":12980224,"data_stored":9924555,"data_compressed":544711,"data_compressed_allocated":737280,"data_compressed_original":4325376,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.989634"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375541+0000","last_change":"2024-04-21T08:42:17.545701+0000","last_active":"2024-04-21T08:44:41.375541+0000","last_peered":"2024-04-21T08:44:41.375541+0000","last_clean":"2024-04-21T08:44:41.375541+0000","last_became_active":"2024-04-21T08:42:17.545157+0000","last_became_peered":"2024-04-21T08:42:17.545157+0000","last_unstale":"2024-04-21T08:44:41.375541+0000","last_undegraded":"2024-04-21T08:44:41.375541+0000","last_fullsized":"2024-04-21T08:44:41.375541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:05:41.952879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418165+0000","last_change":"2024-04-21T08:42:17.541153+0000","last_active":"2024-04-21T08:44:41.418165+0000","last_peered":"2024-04-21T08:44:41.418165+0000","last_clean":"2024-04-21T08:44:41.418165+0000","last_became_active":"2024-04-21T08:42:17.540771+0000","last_became_peered":"2024-04-21T08:42:17.540771+0000","last_unstale":"2024-04-21T08:44:41.418165+0000","last_undegraded":"2024-04-21T08:44:41.418165+0000","last_fullsized":"2024-04-21T08:44:41.418165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:35:06.077367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417993+0000","last_change":"2024-04-21T08:42:17.545525+0000","last_active":"2024-04-21T08:44:41.417993+0000","last_peered":"2024-04-21T08:44:41.417993+0000","last_clean":"2024-04-21T08:44:41.417993+0000","last_became_active":"2024-04-21T08:42:17.545168+0000","last_became_peered":"2024-04-21T08:42:17.545168+0000","last_unstale":"2024-04-21T08:44:41.417993+0000","last_undegraded":"2024-04-21T08:44:41.417993+0000","last_fullsized":"2024-04-21T08:44:41.417993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:51.080385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251479+0000","last_change":"2024-04-21T08:42:17.543917+0000","last_active":"2024-04-21T08:44:41.251479+0000","last_peered":"2024-04-21T08:44:41.251479+0000","last_clean":"2024-04-21T08:44:41.251479+0000","last_became_active":"2024-04-21T08:42:17.543513+0000","last_became_peered":"2024-04-21T08:42:17.543513+0000","last_unstale":"2024-04-21T08:44:41.251479+0000","last_undegraded":"2024-04-21T08:44:41.251479+0000","last_fullsized":"2024-04-21T08:44:41.251479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:23.058600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250648+0000","last_change":"2024-04-21T08:42:17.549737+0000","last_active":"2024-04-21T08:44:41.250648+0000","last_peered":"2024-04-21T08:44:41.250648+0000","last_clean":"2024-04-21T08:44:41.250648+0000","last_became_active":"2024-04-21T08:42:17.549396+0000","last_became_peered":"2024-04-21T08:42:17.549396+0000","last_unstale":"2024-04-21T08:44:41.250648+0000","last_undegraded":"2024-04-21T08:44:41.250648+0000","last_fullsized":"2024-04-21T08:44:41.250648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:01:51.879960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368365+0000","last_change":"2024-04-21T08:43:10.792321+0000","last_active":"2024-04-21T08:44:41.368365+0000","last_peered":"2024-04-21T08:44:41.368365+0000","last_clean":"2024-04-21T08:44:41.368365+0000","last_became_active":"2024-04-21T08:43:10.791979+0000","last_became_peered":"2024-04-21T08:43:10.791979+0000","last_unstale":"2024-04-21T08:44:41.368365+0000","last_undegraded":"2024-04-21T08:44:41.368365+0000","last_fullsized":"2024-04-21T08:44:41.368365+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:11:16.056778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374889+0000","last_change":"2024-04-21T08:42:17.541172+0000","last_active":"2024-04-21T08:44:41.374889+0000","last_peered":"2024-04-21T08:44:41.374889+0000","last_clean":"2024-04-21T08:44:41.374889+0000","last_became_active":"2024-04-21T08:42:17.540829+0000","last_became_peered":"2024-04-21T08:42:17.540829+0000","last_unstale":"2024-04-21T08:44:41.374889+0000","last_undegraded":"2024-04-21T08:44:41.374889+0000","last_fullsized":"2024-04-21T08:44:41.374889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:16:12.633980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249753+0000","last_change":"2024-04-21T08:42:17.549744+0000","last_active":"2024-04-21T08:44:41.249753+0000","last_peered":"2024-04-21T08:44:41.249753+0000","last_clean":"2024-04-21T08:44:41.249753+0000","last_became_active":"2024-04-21T08:42:17.549378+0000","last_became_peered":"2024-04-21T08:42:17.549378+0000","last_unstale":"2024-04-21T08:44:41.249753+0000","last_undegraded":"2024-04-21T08:44:41.249753+0000","last_fullsized":"2024-04-21T08:44:41.249753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:03:07.702143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375768+0000","last_change":"2024-04-21T08:42:17.548675+0000","last_active":"2024-04-21T08:44:41.375768+0000","last_peered":"2024-04-21T08:44:41.375768+0000","last_clean":"2024-04-21T08:44:41.375768+0000","last_became_active":"2024-04-21T08:42:17.547809+0000","last_became_peered":"2024-04-21T08:42:17.547809+0000","last_unstale":"2024-04-21T08:44:41.375768+0000","last_undegraded":"2024-04-21T08:44:41.375768+0000","last_fullsized":"2024-04-21T08:44:41.375768+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:23.343473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368771+0000","last_change":"2024-04-21T08:42:17.551012+0000","last_active":"2024-04-21T08:44:41.368771+0000","last_peered":"2024-04-21T08:44:41.368771+0000","last_clean":"2024-04-21T08:44:41.368771+0000","last_became_active":"2024-04-21T08:42:17.550627+0000","last_became_peered":"2024-04-21T08:42:17.550627+0000","last_unstale":"2024-04-21T08:44:41.368771+0000","last_undegraded":"2024-04-21T08:44:41.368771+0000","last_fullsized":"2024-04-21T08:44:41.368771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:57.501588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375100+0000","last_change":"2024-04-21T08:42:17.546192+0000","last_active":"2024-04-21T08:44:41.375100+0000","last_peered":"2024-04-21T08:44:41.375100+0000","last_clean":"2024-04-21T08:44:41.375100+0000","last_became_active":"2024-04-21T08:42:17.545808+0000","last_became_peered":"2024-04-21T08:42:17.545808+0000","last_unstale":"2024-04-21T08:44:41.375100+0000","last_undegraded":"2024-04-21T08:44:41.375100+0000","last_fullsized":"2024-04-21T08:44:41.375100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:54.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251368+0000","last_change":"2024-04-21T08:42:17.553351+0000","last_active":"2024-04-21T08:44:41.251368+0000","last_peered":"2024-04-21T08:44:41.251368+0000","last_clean":"2024-04-21T08:44:41.251368+0000","last_became_active":"2024-04-21T08:42:17.550507+0000","last_became_peered":"2024-04-21T08:42:17.550507+0000","last_unstale":"2024-04-21T08:44:41.251368+0000","last_undegraded":"2024-04-21T08:44:41.251368+0000","last_fullsized":"2024-04-21T08:44:41.251368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:25.561375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376794+0000","last_change":"2024-04-21T08:42:17.540533+0000","last_active":"2024-04-21T08:44:41.376794+0000","last_peered":"2024-04-21T08:44:41.376794+0000","last_clean":"2024-04-21T08:44:41.376794+0000","last_became_active":"2024-04-21T08:42:17.539940+0000","last_became_peered":"2024-04-21T08:42:17.539940+0000","last_unstale":"2024-04-21T08:44:41.376794+0000","last_undegraded":"2024-04-21T08:44:41.376794+0000","last_fullsized":"2024-04-21T08:44:41.376794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:45:19.102899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417676+0000","last_change":"2024-04-21T08:42:17.537794+0000","last_active":"2024-04-21T08:44:41.417676+0000","last_peered":"2024-04-21T08:44:41.417676+0000","last_clean":"2024-04-21T08:44:41.417676+0000","last_became_active":"2024-04-21T08:42:17.537376+0000","last_became_peered":"2024-04-21T08:42:17.537376+0000","last_unstale":"2024-04-21T08:44:41.417676+0000","last_undegraded":"2024-04-21T08:44:41.417676+0000","last_fullsized":"2024-04-21T08:44:41.417676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:57.430036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417515+0000","last_change":"2024-04-21T08:42:17.550574+0000","last_active":"2024-04-21T08:44:41.417515+0000","last_peered":"2024-04-21T08:44:41.417515+0000","last_clean":"2024-04-21T08:44:41.417515+0000","last_became_active":"2024-04-21T08:42:17.550205+0000","last_became_peered":"2024-04-21T08:42:17.550205+0000","last_unstale":"2024-04-21T08:44:41.417515+0000","last_undegraded":"2024-04-21T08:44:41.417515+0000","last_fullsized":"2024-04-21T08:44:41.417515+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:41:08.226477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374544+0000","last_change":"2024-04-21T08:42:17.542239+0000","last_active":"2024-04-21T08:44:41.374544+0000","last_peered":"2024-04-21T08:44:41.374544+0000","last_clean":"2024-04-21T08:44:41.374544+0000","last_became_active":"2024-04-21T08:42:17.541848+0000","last_became_peered":"2024-04-21T08:42:17.541848+0000","last_unstale":"2024-04-21T08:44:41.374544+0000","last_undegraded":"2024-04-21T08:44:41.374544+0000","last_fullsized":"2024-04-21T08:44:41.374544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:47.717451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375417+0000","last_change":"2024-04-21T08:42:17.553200+0000","last_active":"2024-04-21T08:44:41.375417+0000","last_peered":"2024-04-21T08:44:41.375417+0000","last_clean":"2024-04-21T08:44:41.375417+0000","last_became_active":"2024-04-21T08:42:17.552913+0000","last_became_peered":"2024-04-21T08:42:17.552913+0000","last_unstale":"2024-04-21T08:44:41.375417+0000","last_undegraded":"2024-04-21T08:44:41.375417+0000","last_fullsized":"2024-04-21T08:44:41.375417+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:39:04.186928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356904+0000","last_change":"2024-04-21T08:42:17.534881+0000","last_active":"2024-04-21T08:44:41.356904+0000","last_peered":"2024-04-21T08:44:41.356904+0000","last_clean":"2024-04-21T08:44:41.356904+0000","last_became_active":"2024-04-21T08:42:17.534496+0000","last_became_peered":"2024-04-21T08:42:17.534496+0000","last_unstale":"2024-04-21T08:44:41.356904+0000","last_undegraded":"2024-04-21T08:44:41.356904+0000","last_fullsized":"2024-04-21T08:44:41.356904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:50:19.409391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375734+0000","last_change":"2024-04-21T08:42:17.540480+0000","last_active":"2024-04-21T08:44:41.375734+0000","last_peered":"2024-04-21T08:44:41.375734+0000","last_clean":"2024-04-21T08:44:41.375734+0000","last_became_active":"2024-04-21T08:42:17.539742+0000","last_became_peered":"2024-04-21T08:42:17.539742+0000","last_unstale":"2024-04-21T08:44:41.375734+0000","last_undegraded":"2024-04-21T08:44:41.375734+0000","last_fullsized":"2024-04-21T08:44:41.375734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:00.238976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374953+0000","last_change":"2024-04-21T08:44:11.876826+0000","last_active":"2024-04-21T08:44:41.374953+0000","last_peered":"2024-04-21T08:44:41.374953+0000","last_clean":"2024-04-21T08:44:41.374953+0000","last_became_active":"2024-04-21T08:44:11.876354+0000","last_became_peered":"2024-04-21T08:44:11.876354+0000","last_unstale":"2024-04-21T08:44:41.374953+0000","last_undegraded":"2024-04-21T08:44:41.374953+0000","last_fullsized":"2024-04-21T08:44:41.374953+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:00:05.574008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356782+0000","last_change":"2024-04-21T08:42:16.639888+0000","last_active":"2024-04-21T08:44:41.356782+0000","last_peered":"2024-04-21T08:44:41.356782+0000","last_clean":"2024-04-21T08:44:41.356782+0000","last_became_active":"2024-04-21T08:42:16.639403+0000","last_became_peered":"2024-04-21T08:42:16.639403+0000","last_unstale":"2024-04-21T08:44:41.356782+0000","last_undegraded":"2024-04-21T08:44:41.356782+0000","last_fullsized":"2024-04-21T08:44:41.356782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:08:17.164924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249935+0000","last_change":"2024-04-21T08:43:10.983368+0000","last_active":"2024-04-21T08:44:41.249935+0000","last_peered":"2024-04-21T08:44:41.249935+0000","last_clean":"2024-04-21T08:44:41.249935+0000","last_became_active":"2024-04-21T08:43:10.982956+0000","last_became_peered":"2024-04-21T08:43:10.982956+0000","last_unstale":"2024-04-21T08:44:41.249935+0000","last_undegraded":"2024-04-21T08:44:41.249935+0000","last_fullsized":"2024-04-21T08:44:41.249935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:12:44.919996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250725+0000","last_change":"2024-04-21T08:42:21.673197+0000","last_active":"2024-04-21T08:44:41.250725+0000","last_peered":"2024-04-21T08:44:41.250725+0000","last_clean":"2024-04-21T08:44:41.250725+0000","last_became_active":"2024-04-21T08:42:21.672793+0000","last_became_peered":"2024-04-21T08:42:21.672793+0000","last_unstale":"2024-04-21T08:44:41.250725+0000","last_undegraded":"2024-04-21T08:44:41.250725+0000","last_fullsized":"2024-04-21T08:44:41.250725+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:44:29.788459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250990+0000","last_change":"2024-04-21T08:42:16.648577+0000","last_active":"2024-04-21T08:44:41.250990+0000","last_peered":"2024-04-21T08:44:41.250990+0000","last_clean":"2024-04-21T08:44:41.250990+0000","last_became_active":"2024-04-21T08:42:16.646716+0000","last_became_peered":"2024-04-21T08:42:16.646716+0000","last_unstale":"2024-04-21T08:44:41.250990+0000","last_undegraded":"2024-04-21T08:44:41.250990+0000","last_fullsized":"2024-04-21T08:44:41.250990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:10.327078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417936+0000","last_change":"2024-04-21T08:42:17.545430+0000","last_active":"2024-04-21T08:44:41.417936+0000","last_peered":"2024-04-21T08:44:41.417936+0000","last_clean":"2024-04-21T08:44:41.417936+0000","last_became_active":"2024-04-21T08:42:17.545057+0000","last_became_peered":"2024-04-21T08:42:17.545057+0000","last_unstale":"2024-04-21T08:44:41.417936+0000","last_undegraded":"2024-04-21T08:44:41.417936+0000","last_fullsized":"2024-04-21T08:44:41.417936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:44:59.030830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250899+0000","last_change":"2024-04-21T08:42:21.673196+0000","last_active":"2024-04-21T08:44:41.250899+0000","last_peered":"2024-04-21T08:44:41.250899+0000","last_clean":"2024-04-21T08:44:41.250899+0000","last_became_active":"2024-04-21T08:42:21.672808+0000","last_became_peered":"2024-04-21T08:42:21.672808+0000","last_unstale":"2024-04-21T08:44:41.250899+0000","last_undegraded":"2024-04-21T08:44:41.250899+0000","last_fullsized":"2024-04-21T08:44:41.250899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:39.482353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250844+0000","last_change":"2024-04-21T08:42:16.641611+0000","last_active":"2024-04-21T08:44:41.250844+0000","last_peered":"2024-04-21T08:44:41.250844+0000","last_clean":"2024-04-21T08:44:41.250844+0000","last_became_active":"2024-04-21T08:42:16.641221+0000","last_became_peered":"2024-04-21T08:42:16.641221+0000","last_unstale":"2024-04-21T08:44:41.250844+0000","last_undegraded":"2024-04-21T08:44:41.250844+0000","last_fullsized":"2024-04-21T08:44:41.250844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:45.496532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375395+0000","last_change":"2024-04-21T08:42:17.532986+0000","last_active":"2024-04-21T08:44:41.375395+0000","last_peered":"2024-04-21T08:44:41.375395+0000","last_clean":"2024-04-21T08:44:41.375395+0000","last_became_active":"2024-04-21T08:42:17.532583+0000","last_became_peered":"2024-04-21T08:42:17.532583+0000","last_unstale":"2024-04-21T08:44:41.375395+0000","last_undegraded":"2024-04-21T08:44:41.375395+0000","last_fullsized":"2024-04-21T08:44:41.375395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:43.780935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250791+0000","last_change":"2024-04-21T08:42:21.667459+0000","last_active":"2024-04-21T08:44:41.250791+0000","last_peered":"2024-04-21T08:44:41.250791+0000","last_clean":"2024-04-21T08:44:41.250791+0000","last_became_active":"2024-04-21T08:42:21.667183+0000","last_became_peered":"2024-04-21T08:42:21.667183+0000","last_unstale":"2024-04-21T08:44:41.250791+0000","last_undegraded":"2024-04-21T08:44:41.250791+0000","last_fullsized":"2024-04-21T08:44:41.250791+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:32:23.626267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250969+0000","last_change":"2024-04-21T08:42:16.645210+0000","last_active":"2024-04-21T08:44:41.250969+0000","last_peered":"2024-04-21T08:44:41.250969+0000","last_clean":"2024-04-21T08:44:41.250969+0000","last_became_active":"2024-04-21T08:42:16.644747+0000","last_became_peered":"2024-04-21T08:42:16.644747+0000","last_unstale":"2024-04-21T08:44:41.250969+0000","last_undegraded":"2024-04-21T08:44:41.250969+0000","last_fullsized":"2024-04-21T08:44:41.250969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:34:45.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365569+0000","last_change":"2024-04-21T08:43:10.984152+0000","last_active":"2024-04-21T08:44:41.365569+0000","last_peered":"2024-04-21T08:44:41.365569+0000","last_clean":"2024-04-21T08:44:41.365569+0000","last_became_active":"2024-04-21T08:43:10.983736+0000","last_became_peered":"2024-04-21T08:43:10.983736+0000","last_unstale":"2024-04-21T08:44:41.365569+0000","last_undegraded":"2024-04-21T08:44:41.365569+0000","last_fullsized":"2024-04-21T08:44:41.365569+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:10.182667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375391+0000","last_change":"2024-04-21T08:42:21.660739+0000","last_active":"2024-04-21T08:44:41.375391+0000","last_peered":"2024-04-21T08:44:41.375391+0000","last_clean":"2024-04-21T08:44:41.375391+0000","last_became_active":"2024-04-21T08:42:21.660307+0000","last_became_peered":"2024-04-21T08:42:21.660307+0000","last_unstale":"2024-04-21T08:44:41.375391+0000","last_undegraded":"2024-04-21T08:44:41.375391+0000","last_fullsized":"2024-04-21T08:44:41.375391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:26.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365416+0000","last_change":"2024-04-21T08:43:10.986184+0000","last_active":"2024-04-21T08:44:41.365416+0000","last_peered":"2024-04-21T08:44:41.365416+0000","last_clean":"2024-04-21T08:44:41.365416+0000","last_became_active":"2024-04-21T08:43:10.985533+0000","last_became_peered":"2024-04-21T08:43:10.985533+0000","last_unstale":"2024-04-21T08:44:41.365416+0000","last_undegraded":"2024-04-21T08:44:41.365416+0000","last_fullsized":"2024-04-21T08:44:41.365416+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:26.023306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250682+0000","last_change":"2024-04-21T08:42:17.542315+0000","last_active":"2024-04-21T08:44:41.250682+0000","last_peered":"2024-04-21T08:44:41.250682+0000","last_clean":"2024-04-21T08:44:41.250682+0000","last_became_active":"2024-04-21T08:42:17.541895+0000","last_became_peered":"2024-04-21T08:42:17.541895+0000","last_unstale":"2024-04-21T08:44:41.250682+0000","last_undegraded":"2024-04-21T08:44:41.250682+0000","last_fullsized":"2024-04-21T08:44:41.250682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:23:33.909499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375270+0000","last_change":"2024-04-21T08:44:11.878135+0000","last_active":"2024-04-21T08:44:41.375270+0000","last_peered":"2024-04-21T08:44:41.375270+0000","last_clean":"2024-04-21T08:44:41.375270+0000","last_became_active":"2024-04-21T08:44:11.877271+0000","last_became_peered":"2024-04-21T08:44:11.877271+0000","last_unstale":"2024-04-21T08:44:41.375270+0000","last_undegraded":"2024-04-21T08:44:41.375270+0000","last_fullsized":"2024-04-21T08:44:41.375270+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:31.345519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368361+0000","last_change":"2024-04-21T08:43:10.984839+0000","last_active":"2024-04-21T08:44:41.368361+0000","last_peered":"2024-04-21T08:44:41.368361+0000","last_clean":"2024-04-21T08:44:41.368361+0000","last_became_active":"2024-04-21T08:43:10.984382+0000","last_became_peered":"2024-04-21T08:43:10.984382+0000","last_unstale":"2024-04-21T08:44:41.368361+0000","last_undegraded":"2024-04-21T08:44:41.368361+0000","last_fullsized":"2024-04-21T08:44:41.368361+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:47:20.774102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365597+0000","last_change":"2024-04-21T08:42:17.547770+0000","last_active":"2024-04-21T08:44:41.365597+0000","last_peered":"2024-04-21T08:44:41.365597+0000","last_clean":"2024-04-21T08:44:41.365597+0000","last_became_active":"2024-04-21T08:42:17.547440+0000","last_became_peered":"2024-04-21T08:42:17.547440+0000","last_unstale":"2024-04-21T08:44:41.365597+0000","last_undegraded":"2024-04-21T08:44:41.365597+0000","last_fullsized":"2024-04-21T08:44:41.365597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:37:35.555737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375243+0000","last_change":"2024-04-21T08:42:21.670684+0000","last_active":"2024-04-21T08:44:41.375243+0000","last_peered":"2024-04-21T08:44:41.375243+0000","last_clean":"2024-04-21T08:44:41.375243+0000","last_became_active":"2024-04-21T08:42:21.670260+0000","last_became_peered":"2024-04-21T08:42:21.670260+0000","last_unstale":"2024-04-21T08:44:41.375243+0000","last_undegraded":"2024-04-21T08:44:41.375243+0000","last_fullsized":"2024-04-21T08:44:41.375243+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:45:06.192040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376102+0000","last_change":"2024-04-21T08:42:16.639632+0000","last_active":"2024-04-21T08:44:41.376102+0000","last_peered":"2024-04-21T08:44:41.376102+0000","last_clean":"2024-04-21T08:44:41.376102+0000","last_became_active":"2024-04-21T08:42:16.639301+0000","last_became_peered":"2024-04-21T08:42:16.639301+0000","last_unstale":"2024-04-21T08:44:41.376102+0000","last_undegraded":"2024-04-21T08:44:41.376102+0000","last_fullsized":"2024-04-21T08:44:41.376102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:58.838667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249940+0000","last_change":"2024-04-21T08:42:17.546109+0000","last_active":"2024-04-21T08:44:41.249940+0000","last_peered":"2024-04-21T08:44:41.249940+0000","last_clean":"2024-04-21T08:44:41.249940+0000","last_became_active":"2024-04-21T08:42:17.545673+0000","last_became_peered":"2024-04-21T08:42:17.545673+0000","last_unstale":"2024-04-21T08:44:41.249940+0000","last_undegraded":"2024-04-21T08:44:41.249940+0000","last_fullsized":"2024-04-21T08:44:41.249940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:18.947848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417985+0000","last_change":"2024-04-21T08:42:21.666114+0000","last_active":"2024-04-21T08:44:41.417985+0000","last_peered":"2024-04-21T08:44:41.417985+0000","last_clean":"2024-04-21T08:44:41.417985+0000","last_became_active":"2024-04-21T08:42:21.665020+0000","last_became_peered":"2024-04-21T08:42:21.665020+0000","last_unstale":"2024-04-21T08:44:41.417985+0000","last_undegraded":"2024-04-21T08:44:41.417985+0000","last_fullsized":"2024-04-21T08:44:41.417985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:33.644178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365224+0000","last_change":"2024-04-21T08:42:16.639118+0000","last_active":"2024-04-21T08:44:41.365224+0000","last_peered":"2024-04-21T08:44:41.365224+0000","last_clean":"2024-04-21T08:44:41.365224+0000","last_became_active":"2024-04-21T08:42:16.638792+0000","last_became_peered":"2024-04-21T08:42:16.638792+0000","last_unstale":"2024-04-21T08:44:41.365224+0000","last_undegraded":"2024-04-21T08:44:41.365224+0000","last_fullsized":"2024-04-21T08:44:41.365224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:55.568712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356215+0000","last_change":"2024-04-21T08:42:17.539688+0000","last_active":"2024-04-21T08:44:41.356215+0000","last_peered":"2024-04-21T08:44:41.356215+0000","last_clean":"2024-04-21T08:44:41.356215+0000","last_became_active":"2024-04-21T08:42:17.539335+0000","last_became_peered":"2024-04-21T08:42:17.539335+0000","last_unstale":"2024-04-21T08:44:41.356215+0000","last_undegraded":"2024-04-21T08:44:41.356215+0000","last_fullsized":"2024-04-21T08:44:41.356215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:27:51.520232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249882+0000","last_change":"2024-04-21T08:42:21.660570+0000","last_active":"2024-04-21T08:44:41.249882+0000","last_peered":"2024-04-21T08:44:41.249882+0000","last_clean":"2024-04-21T08:44:41.249882+0000","last_became_active":"2024-04-21T08:42:21.659398+0000","last_became_peered":"2024-04-21T08:42:21.659398+0000","last_unstale":"2024-04-21T08:44:41.249882+0000","last_undegraded":"2024-04-21T08:44:41.249882+0000","last_fullsized":"2024-04-21T08:44:41.249882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:55.840334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368786+0000","last_change":"2024-04-21T08:42:16.648192+0000","last_active":"2024-04-21T08:44:41.368786+0000","last_peered":"2024-04-21T08:44:41.368786+0000","last_clean":"2024-04-21T08:44:41.368786+0000","last_became_active":"2024-04-21T08:42:16.647666+0000","last_became_peered":"2024-04-21T08:42:16.647666+0000","last_unstale":"2024-04-21T08:44:41.368786+0000","last_undegraded":"2024-04-21T08:44:41.368786+0000","last_fullsized":"2024-04-21T08:44:41.368786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:33.344680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375708+0000","last_change":"2024-04-21T08:42:17.545925+0000","last_active":"2024-04-21T08:44:41.375708+0000","last_peered":"2024-04-21T08:44:41.375708+0000","last_clean":"2024-04-21T08:44:41.375708+0000","last_became_active":"2024-04-21T08:42:17.545555+0000","last_became_peered":"2024-04-21T08:42:17.545555+0000","last_unstale":"2024-04-21T08:44:41.375708+0000","last_undegraded":"2024-04-21T08:44:41.375708+0000","last_fullsized":"2024-04-21T08:44:41.375708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:13:07.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356087+0000","last_change":"2024-04-21T08:42:21.662757+0000","last_active":"2024-04-21T08:44:41.356087+0000","last_peered":"2024-04-21T08:44:41.356087+0000","last_clean":"2024-04-21T08:44:41.356087+0000","last_became_active":"2024-04-21T08:42:21.662306+0000","last_became_peered":"2024-04-21T08:42:21.662306+0000","last_unstale":"2024-04-21T08:44:41.356087+0000","last_undegraded":"2024-04-21T08:44:41.356087+0000","last_fullsized":"2024-04-21T08:44:41.356087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:26.856147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375135+0000","last_change":"2024-04-21T08:43:10.984690+0000","last_active":"2024-04-21T08:44:41.375135+0000","last_peered":"2024-04-21T08:44:41.375135+0000","last_clean":"2024-04-21T08:44:41.375135+0000","last_became_active":"2024-04-21T08:43:10.984290+0000","last_became_peered":"2024-04-21T08:43:10.984290+0000","last_unstale":"2024-04-21T08:44:41.375135+0000","last_undegraded":"2024-04-21T08:44:41.375135+0000","last_fullsized":"2024-04-21T08:44:41.375135+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:37:36.400946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418211+0000","last_change":"2024-04-21T08:43:10.793316+0000","last_active":"2024-04-21T08:44:41.418211+0000","last_peered":"2024-04-21T08:44:41.418211+0000","last_clean":"2024-04-21T08:44:41.418211+0000","last_became_active":"2024-04-21T08:43:10.793035+0000","last_became_peered":"2024-04-21T08:43:10.793035+0000","last_unstale":"2024-04-21T08:44:41.418211+0000","last_undegraded":"2024-04-21T08:44:41.418211+0000","last_fullsized":"2024-04-21T08:44:41.418211+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:41:04.247385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251083+0000","last_change":"2024-04-21T08:42:21.670561+0000","last_active":"2024-04-21T08:44:41.251083+0000","last_peered":"2024-04-21T08:44:41.251083+0000","last_clean":"2024-04-21T08:44:41.251083+0000","last_became_active":"2024-04-21T08:42:21.670170+0000","last_became_peered":"2024-04-21T08:42:21.670170+0000","last_unstale":"2024-04-21T08:44:41.251083+0000","last_undegraded":"2024-04-21T08:44:41.251083+0000","last_fullsized":"2024-04-21T08:44:41.251083+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:35:54.538013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365460+0000","last_change":"2024-04-21T08:42:16.637319+0000","last_active":"2024-04-21T08:44:41.365460+0000","last_peered":"2024-04-21T08:44:41.365460+0000","last_clean":"2024-04-21T08:44:41.365460+0000","last_became_active":"2024-04-21T08:42:16.636940+0000","last_became_peered":"2024-04-21T08:42:16.636940+0000","last_unstale":"2024-04-21T08:44:41.365460+0000","last_undegraded":"2024-04-21T08:44:41.365460+0000","last_fullsized":"2024-04-21T08:44:41.365460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:59:09.826786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375697+0000","last_change":"2024-04-21T08:42:17.546202+0000","last_active":"2024-04-21T08:44:41.375697+0000","last_peered":"2024-04-21T08:44:41.375697+0000","last_clean":"2024-04-21T08:44:41.375697+0000","last_became_active":"2024-04-21T08:42:17.545781+0000","last_became_peered":"2024-04-21T08:42:17.545781+0000","last_unstale":"2024-04-21T08:44:41.375697+0000","last_undegraded":"2024-04-21T08:44:41.375697+0000","last_fullsized":"2024-04-21T08:44:41.375697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:23.990951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356160+0000","last_change":"2024-04-21T08:42:21.662822+0000","last_active":"2024-04-21T08:44:41.356160+0000","last_peered":"2024-04-21T08:44:41.356160+0000","last_clean":"2024-04-21T08:44:41.356160+0000","last_became_active":"2024-04-21T08:42:21.662398+0000","last_became_peered":"2024-04-21T08:42:21.662398+0000","last_unstale":"2024-04-21T08:44:41.356160+0000","last_undegraded":"2024-04-21T08:44:41.356160+0000","last_fullsized":"2024-04-21T08:44:41.356160+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:38:40.499294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":324,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:42.133017+0000","last_change":"2024-04-21T08:42:16.636330+0000","last_active":"2024-04-21T08:44:42.133017+0000","last_peered":"2024-04-21T08:44:42.133017+0000","last_clean":"2024-04-21T08:44:42.133017+0000","last_became_active":"2024-04-21T08:42:16.635762+0000","last_became_peered":"2024-04-21T08:42:16.635762+0000","last_unstale":"2024-04-21T08:44:42.133017+0000","last_undegraded":"2024-04-21T08:44:42.133017+0000","last_fullsized":"2024-04-21T08:44:42.133017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:31.137923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":195,"num_write":38,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251043+0000","last_change":"2024-04-21T08:42:17.543033+0000","last_active":"2024-04-21T08:44:41.251043+0000","last_peered":"2024-04-21T08:44:41.251043+0000","last_clean":"2024-04-21T08:44:41.251043+0000","last_became_active":"2024-04-21T08:42:17.542692+0000","last_became_peered":"2024-04-21T08:42:17.542692+0000","last_unstale":"2024-04-21T08:44:41.251043+0000","last_undegraded":"2024-04-21T08:44:41.251043+0000","last_fullsized":"2024-04-21T08:44:41.251043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:33:32.111999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365780+0000","last_change":"2024-04-21T08:42:21.668831+0000","last_active":"2024-04-21T08:44:41.365780+0000","last_peered":"2024-04-21T08:44:41.365780+0000","last_clean":"2024-04-21T08:44:41.365780+0000","last_became_active":"2024-04-21T08:42:21.668485+0000","last_became_peered":"2024-04-21T08:42:21.668485+0000","last_unstale":"2024-04-21T08:44:41.365780+0000","last_undegraded":"2024-04-21T08:44:41.365780+0000","last_fullsized":"2024-04-21T08:44:41.365780+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:49:29.049707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375241+0000","last_change":"2024-04-21T08:42:16.640288+0000","last_active":"2024-04-21T08:44:41.375241+0000","last_peered":"2024-04-21T08:44:41.375241+0000","last_clean":"2024-04-21T08:44:41.375241+0000","last_became_active":"2024-04-21T08:42:16.639909+0000","last_became_peered":"2024-04-21T08:42:16.639909+0000","last_unstale":"2024-04-21T08:44:41.375241+0000","last_undegraded":"2024-04-21T08:44:41.375241+0000","last_fullsized":"2024-04-21T08:44:41.375241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:33.839442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375479+0000","last_change":"2024-04-21T08:42:17.541154+0000","last_active":"2024-04-21T08:44:41.375479+0000","last_peered":"2024-04-21T08:44:41.375479+0000","last_clean":"2024-04-21T08:44:41.375479+0000","last_became_active":"2024-04-21T08:42:17.540747+0000","last_became_peered":"2024-04-21T08:42:17.540747+0000","last_unstale":"2024-04-21T08:44:41.375479+0000","last_undegraded":"2024-04-21T08:44:41.375479+0000","last_fullsized":"2024-04-21T08:44:41.375479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:14:50.641144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374983+0000","last_change":"2024-04-21T08:42:21.670767+0000","last_active":"2024-04-21T08:44:41.374983+0000","last_peered":"2024-04-21T08:44:41.374983+0000","last_clean":"2024-04-21T08:44:41.374983+0000","last_became_active":"2024-04-21T08:42:21.670338+0000","last_became_peered":"2024-04-21T08:42:21.670338+0000","last_unstale":"2024-04-21T08:44:41.374983+0000","last_undegraded":"2024-04-21T08:44:41.374983+0000","last_fullsized":"2024-04-21T08:44:41.374983+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:38:55.275603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365468+0000","last_change":"2024-04-21T08:42:16.640144+0000","last_active":"2024-04-21T08:44:41.365468+0000","last_peered":"2024-04-21T08:44:41.365468+0000","last_clean":"2024-04-21T08:44:41.365468+0000","last_became_active":"2024-04-21T08:42:16.639707+0000","last_became_peered":"2024-04-21T08:42:16.639707+0000","last_unstale":"2024-04-21T08:44:41.365468+0000","last_undegraded":"2024-04-21T08:44:41.365468+0000","last_fullsized":"2024-04-21T08:44:41.365468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:11:32.785519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375240+0000","last_change":"2024-04-21T08:42:17.555019+0000","last_active":"2024-04-21T08:44:41.375240+0000","last_peered":"2024-04-21T08:44:41.375240+0000","last_clean":"2024-04-21T08:44:41.375240+0000","last_became_active":"2024-04-21T08:42:17.554645+0000","last_became_peered":"2024-04-21T08:42:17.554645+0000","last_unstale":"2024-04-21T08:44:41.375240+0000","last_undegraded":"2024-04-21T08:44:41.375240+0000","last_fullsized":"2024-04-21T08:44:41.375240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:42:30.353945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251223+0000","last_change":"2024-04-21T08:44:11.876740+0000","last_active":"2024-04-21T08:44:41.251223+0000","last_peered":"2024-04-21T08:44:41.251223+0000","last_clean":"2024-04-21T08:44:41.251223+0000","last_became_active":"2024-04-21T08:44:11.876108+0000","last_became_peered":"2024-04-21T08:44:11.876108+0000","last_unstale":"2024-04-21T08:44:41.251223+0000","last_undegraded":"2024-04-21T08:44:41.251223+0000","last_fullsized":"2024-04-21T08:44:41.251223+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:37.398743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250912+0000","last_change":"2024-04-21T08:42:16.640407+0000","last_active":"2024-04-21T08:44:41.250912+0000","last_peered":"2024-04-21T08:44:41.250912+0000","last_clean":"2024-04-21T08:44:41.250912+0000","last_became_active":"2024-04-21T08:42:16.640015+0000","last_became_peered":"2024-04-21T08:42:16.640015+0000","last_unstale":"2024-04-21T08:44:41.250912+0000","last_undegraded":"2024-04-21T08:44:41.250912+0000","last_fullsized":"2024-04-21T08:44:41.250912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:23:10.647184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374762+0000","last_change":"2024-04-21T08:42:17.545438+0000","last_active":"2024-04-21T08:44:41.374762+0000","last_peered":"2024-04-21T08:44:41.374762+0000","last_clean":"2024-04-21T08:44:41.374762+0000","last_became_active":"2024-04-21T08:42:17.544258+0000","last_became_peered":"2024-04-21T08:42:17.544258+0000","last_unstale":"2024-04-21T08:44:41.374762+0000","last_undegraded":"2024-04-21T08:44:41.374762+0000","last_fullsized":"2024-04-21T08:44:41.374762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:43.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376385+0000","last_change":"2024-04-21T08:42:21.671000+0000","last_active":"2024-04-21T08:44:41.376385+0000","last_peered":"2024-04-21T08:44:41.376385+0000","last_clean":"2024-04-21T08:44:41.376385+0000","last_became_active":"2024-04-21T08:42:21.670380+0000","last_became_peered":"2024-04-21T08:42:21.670380+0000","last_unstale":"2024-04-21T08:44:41.376385+0000","last_undegraded":"2024-04-21T08:44:41.376385+0000","last_fullsized":"2024-04-21T08:44:41.376385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:57.075686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250621+0000","last_change":"2024-04-21T08:42:16.649599+0000","last_active":"2024-04-21T08:44:41.250621+0000","last_peered":"2024-04-21T08:44:41.250621+0000","last_clean":"2024-04-21T08:44:41.250621+0000","last_became_active":"2024-04-21T08:42:16.649232+0000","last_became_peered":"2024-04-21T08:42:16.649232+0000","last_unstale":"2024-04-21T08:44:41.250621+0000","last_undegraded":"2024-04-21T08:44:41.250621+0000","last_fullsized":"2024-04-21T08:44:41.250621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:21.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356045+0000","last_change":"2024-04-21T08:44:11.776138+0000","last_active":"2024-04-21T08:44:41.356045+0000","last_peered":"2024-04-21T08:44:41.356045+0000","last_clean":"2024-04-21T08:44:41.356045+0000","last_became_active":"2024-04-21T08:44:11.775832+0000","last_became_peered":"2024-04-21T08:44:11.775832+0000","last_unstale":"2024-04-21T08:44:41.356045+0000","last_undegraded":"2024-04-21T08:44:41.356045+0000","last_fullsized":"2024-04-21T08:44:41.356045+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:03:34.210923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365172+0000","last_change":"2024-04-21T08:42:21.664811+0000","last_active":"2024-04-21T08:44:41.365172+0000","last_peered":"2024-04-21T08:44:41.365172+0000","last_clean":"2024-04-21T08:44:41.365172+0000","last_became_active":"2024-04-21T08:42:21.664332+0000","last_became_peered":"2024-04-21T08:42:21.664332+0000","last_unstale":"2024-04-21T08:44:41.365172+0000","last_undegraded":"2024-04-21T08:44:41.365172+0000","last_fullsized":"2024-04-21T08:44:41.365172+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:40.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251372+0000","last_change":"2024-04-21T08:42:16.641233+0000","last_active":"2024-04-21T08:44:41.251372+0000","last_peered":"2024-04-21T08:44:41.251372+0000","last_clean":"2024-04-21T08:44:41.251372+0000","last_became_active":"2024-04-21T08:42:16.640873+0000","last_became_peered":"2024-04-21T08:42:16.640873+0000","last_unstale":"2024-04-21T08:44:41.251372+0000","last_undegraded":"2024-04-21T08:44:41.251372+0000","last_fullsized":"2024-04-21T08:44:41.251372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:00.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375293+0000","last_change":"2024-04-21T08:42:17.550438+0000","last_active":"2024-04-21T08:44:41.375293+0000","last_peered":"2024-04-21T08:44:41.375293+0000","last_clean":"2024-04-21T08:44:41.375293+0000","last_became_active":"2024-04-21T08:42:17.550015+0000","last_became_peered":"2024-04-21T08:42:17.550015+0000","last_unstale":"2024-04-21T08:44:41.375293+0000","last_undegraded":"2024-04-21T08:44:41.375293+0000","last_fullsized":"2024-04-21T08:44:41.375293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:00:29.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250709+0000","last_change":"2024-04-21T08:42:21.970498+0000","last_active":"2024-04-21T08:44:41.250709+0000","last_peered":"2024-04-21T08:44:41.250709+0000","last_clean":"2024-04-21T08:44:41.250709+0000","last_became_active":"2024-04-21T08:42:21.969819+0000","last_became_peered":"2024-04-21T08:42:21.969819+0000","last_unstale":"2024-04-21T08:44:41.250709+0000","last_undegraded":"2024-04-21T08:44:41.250709+0000","last_fullsized":"2024-04-21T08:44:41.250709+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:13:16.552293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356527+0000","last_change":"2024-04-21T08:42:16.637146+0000","last_active":"2024-04-21T08:44:41.356527+0000","last_peered":"2024-04-21T08:44:41.356527+0000","last_clean":"2024-04-21T08:44:41.356527+0000","last_became_active":"2024-04-21T08:42:16.636781+0000","last_became_peered":"2024-04-21T08:42:16.636781+0000","last_unstale":"2024-04-21T08:44:41.356527+0000","last_undegraded":"2024-04-21T08:44:41.356527+0000","last_fullsized":"2024-04-21T08:44:41.356527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:39:27.641236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368622+0000","last_change":"2024-04-21T08:42:17.542325+0000","last_active":"2024-04-21T08:44:41.368622+0000","last_peered":"2024-04-21T08:44:41.368622+0000","last_clean":"2024-04-21T08:44:41.368622+0000","last_became_active":"2024-04-21T08:42:17.541103+0000","last_became_peered":"2024-04-21T08:42:17.541103+0000","last_unstale":"2024-04-21T08:44:41.368622+0000","last_undegraded":"2024-04-21T08:44:41.368622+0000","last_fullsized":"2024-04-21T08:44:41.368622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:46.903250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375702+0000","last_change":"2024-04-21T08:42:21.671146+0000","last_active":"2024-04-21T08:44:41.375702+0000","last_peered":"2024-04-21T08:44:41.375702+0000","last_clean":"2024-04-21T08:44:41.375702+0000","last_became_active":"2024-04-21T08:42:21.670561+0000","last_became_peered":"2024-04-21T08:42:21.670561+0000","last_unstale":"2024-04-21T08:44:41.375702+0000","last_undegraded":"2024-04-21T08:44:41.375702+0000","last_fullsized":"2024-04-21T08:44:41.375702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:38:28.028976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375066+0000","last_change":"2024-04-21T08:42:16.645132+0000","last_active":"2024-04-21T08:44:41.375066+0000","last_peered":"2024-04-21T08:44:41.375066+0000","last_clean":"2024-04-21T08:44:41.375066+0000","last_became_active":"2024-04-21T08:42:16.644786+0000","last_became_peered":"2024-04-21T08:42:16.644786+0000","last_unstale":"2024-04-21T08:44:41.375066+0000","last_undegraded":"2024-04-21T08:44:41.375066+0000","last_fullsized":"2024-04-21T08:44:41.375066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:08.298498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368567+0000","last_change":"2024-04-21T08:42:17.550424+0000","last_active":"2024-04-21T08:44:41.368567+0000","last_peered":"2024-04-21T08:44:41.368567+0000","last_clean":"2024-04-21T08:44:41.368567+0000","last_became_active":"2024-04-21T08:42:17.550077+0000","last_became_peered":"2024-04-21T08:42:17.550077+0000","last_unstale":"2024-04-21T08:44:41.368567+0000","last_undegraded":"2024-04-21T08:44:41.368567+0000","last_fullsized":"2024-04-21T08:44:41.368567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:17:07.016204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375856+0000","last_change":"2024-04-21T08:42:21.970652+0000","last_active":"2024-04-21T08:44:41.375856+0000","last_peered":"2024-04-21T08:44:41.375856+0000","last_clean":"2024-04-21T08:44:41.375856+0000","last_became_active":"2024-04-21T08:42:21.970251+0000","last_became_peered":"2024-04-21T08:42:21.970251+0000","last_unstale":"2024-04-21T08:44:41.375856+0000","last_undegraded":"2024-04-21T08:44:41.375856+0000","last_fullsized":"2024-04-21T08:44:41.375856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:45.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365285+0000","last_change":"2024-04-21T08:42:17.542820+0000","last_active":"2024-04-21T08:44:41.365285+0000","last_peered":"2024-04-21T08:44:41.365285+0000","last_clean":"2024-04-21T08:44:41.365285+0000","last_became_active":"2024-04-21T08:42:17.541320+0000","last_became_peered":"2024-04-21T08:42:17.541320+0000","last_unstale":"2024-04-21T08:44:41.365285+0000","last_undegraded":"2024-04-21T08:44:41.365285+0000","last_fullsized":"2024-04-21T08:44:41.365285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:23:06.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375209+0000","last_change":"2024-04-21T08:42:16.636950+0000","last_active":"2024-04-21T08:44:41.375209+0000","last_peered":"2024-04-21T08:44:41.375209+0000","last_clean":"2024-04-21T08:44:41.375209+0000","last_became_active":"2024-04-21T08:42:16.636590+0000","last_became_peered":"2024-04-21T08:42:16.636590+0000","last_unstale":"2024-04-21T08:44:41.375209+0000","last_undegraded":"2024-04-21T08:44:41.375209+0000","last_fullsized":"2024-04-21T08:44:41.375209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:55.693681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374884+0000","last_change":"2024-04-21T08:42:21.660404+0000","last_active":"2024-04-21T08:44:41.374884+0000","last_peered":"2024-04-21T08:44:41.374884+0000","last_clean":"2024-04-21T08:44:41.374884+0000","last_became_active":"2024-04-21T08:42:21.659999+0000","last_became_peered":"2024-04-21T08:42:21.659999+0000","last_unstale":"2024-04-21T08:44:41.374884+0000","last_undegraded":"2024-04-21T08:44:41.374884+0000","last_fullsized":"2024-04-21T08:44:41.374884+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:53:32.328873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356239+0000","last_change":"2024-04-21T08:43:10.984361+0000","last_active":"2024-04-21T08:44:41.356239+0000","last_peered":"2024-04-21T08:44:41.356239+0000","last_clean":"2024-04-21T08:44:41.356239+0000","last_became_active":"2024-04-21T08:43:10.984046+0000","last_became_peered":"2024-04-21T08:43:10.984046+0000","last_unstale":"2024-04-21T08:44:41.356239+0000","last_undegraded":"2024-04-21T08:44:41.356239+0000","last_fullsized":"2024-04-21T08:44:41.356239+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:33:23.154630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251236+0000","last_change":"2024-04-21T08:42:16.638993+0000","last_active":"2024-04-21T08:44:41.251236+0000","last_peered":"2024-04-21T08:44:41.251236+0000","last_clean":"2024-04-21T08:44:41.251236+0000","last_became_active":"2024-04-21T08:42:16.638617+0000","last_became_peered":"2024-04-21T08:42:16.638617+0000","last_unstale":"2024-04-21T08:44:41.251236+0000","last_undegraded":"2024-04-21T08:44:41.251236+0000","last_fullsized":"2024-04-21T08:44:41.251236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:14:28.271433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374510+0000","last_change":"2024-04-21T08:41:00.044984+0000","last_active":"2024-04-21T08:44:41.374510+0000","last_peered":"2024-04-21T08:44:41.374510+0000","last_clean":"2024-04-21T08:44:41.374510+0000","last_became_active":"2024-04-21T08:41:00.044485+0000","last_became_peered":"2024-04-21T08:41:00.044485+0000","last_unstale":"2024-04-21T08:44:41.374510+0000","last_undegraded":"2024-04-21T08:44:41.374510+0000","last_fullsized":"2024-04-21T08:44:41.374510+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.370031+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:01.031209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365508+0000","last_change":"2024-04-21T08:42:21.971045+0000","last_active":"2024-04-21T08:44:41.365508+0000","last_peered":"2024-04-21T08:44:41.365508+0000","last_clean":"2024-04-21T08:44:41.365508+0000","last_became_active":"2024-04-21T08:42:21.970577+0000","last_became_peered":"2024-04-21T08:42:21.970577+0000","last_unstale":"2024-04-21T08:44:41.365508+0000","last_undegraded":"2024-04-21T08:44:41.365508+0000","last_fullsized":"2024-04-21T08:44:41.365508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:55.681662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374964+0000","last_change":"2024-04-21T08:42:17.535570+0000","last_active":"2024-04-21T08:44:41.374964+0000","last_peered":"2024-04-21T08:44:41.374964+0000","last_clean":"2024-04-21T08:44:41.374964+0000","last_became_active":"2024-04-21T08:42:17.535217+0000","last_became_peered":"2024-04-21T08:42:17.535217+0000","last_unstale":"2024-04-21T08:44:41.374964+0000","last_undegraded":"2024-04-21T08:44:41.374964+0000","last_fullsized":"2024-04-21T08:44:41.374964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:49.510750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374650+0000","last_change":"2024-04-21T08:43:10.792548+0000","last_active":"2024-04-21T08:44:41.374650+0000","last_peered":"2024-04-21T08:44:41.374650+0000","last_clean":"2024-04-21T08:44:41.374650+0000","last_became_active":"2024-04-21T08:43:10.792165+0000","last_became_peered":"2024-04-21T08:43:10.792165+0000","last_unstale":"2024-04-21T08:44:41.374650+0000","last_undegraded":"2024-04-21T08:44:41.374650+0000","last_fullsized":"2024-04-21T08:44:41.374650+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:35.980536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374872+0000","last_change":"2024-04-21T08:42:21.661393+0000","last_active":"2024-04-21T08:44:41.374872+0000","last_peered":"2024-04-21T08:44:41.374872+0000","last_clean":"2024-04-21T08:44:41.374872+0000","last_became_active":"2024-04-21T08:42:21.660201+0000","last_became_peered":"2024-04-21T08:42:21.660201+0000","last_unstale":"2024-04-21T08:44:41.374872+0000","last_undegraded":"2024-04-21T08:44:41.374872+0000","last_fullsized":"2024-04-21T08:44:41.374872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:13.393389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250941+0000","last_change":"2024-04-21T08:42:16.650507+0000","last_active":"2024-04-21T08:44:41.250941+0000","last_peered":"2024-04-21T08:44:41.250941+0000","last_clean":"2024-04-21T08:44:41.250941+0000","last_became_active":"2024-04-21T08:42:16.650217+0000","last_became_peered":"2024-04-21T08:42:16.650217+0000","last_unstale":"2024-04-21T08:44:41.250941+0000","last_undegraded":"2024-04-21T08:44:41.250941+0000","last_fullsized":"2024-04-21T08:44:41.250941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:13:26.785092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375997+0000","last_change":"2024-04-21T08:42:17.530420+0000","last_active":"2024-04-21T08:44:41.375997+0000","last_peered":"2024-04-21T08:44:41.375997+0000","last_clean":"2024-04-21T08:44:41.375997+0000","last_became_active":"2024-04-21T08:42:17.529920+0000","last_became_peered":"2024-04-21T08:42:17.529920+0000","last_unstale":"2024-04-21T08:44:41.375997+0000","last_undegraded":"2024-04-21T08:44:41.375997+0000","last_fullsized":"2024-04-21T08:44:41.375997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:24.250197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249760+0000","last_change":"2024-04-21T08:42:21.970586+0000","last_active":"2024-04-21T08:44:41.249760+0000","last_peered":"2024-04-21T08:44:41.249760+0000","last_clean":"2024-04-21T08:44:41.249760+0000","last_became_active":"2024-04-21T08:42:21.970132+0000","last_became_peered":"2024-04-21T08:42:21.970132+0000","last_unstale":"2024-04-21T08:44:41.249760+0000","last_undegraded":"2024-04-21T08:44:41.249760+0000","last_fullsized":"2024-04-21T08:44:41.249760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:11.074370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'43","reported_seq":100,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.483501+0000","last_change":"2024-04-21T08:42:16.647916+0000","last_active":"2024-04-21T08:44:41.483501+0000","last_peered":"2024-04-21T08:44:41.483501+0000","last_clean":"2024-04-21T08:44:41.483501+0000","last_became_active":"2024-04-21T08:42:16.647476+0000","last_became_peered":"2024-04-21T08:42:16.647476+0000","last_unstale":"2024-04-21T08:44:41.483501+0000","last_undegraded":"2024-04-21T08:44:41.483501+0000","last_fullsized":"2024-04-21T08:44:41.483501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:27.242153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251070+0000","last_change":"2024-04-21T08:42:17.546144+0000","last_active":"2024-04-21T08:44:41.251070+0000","last_peered":"2024-04-21T08:44:41.251070+0000","last_clean":"2024-04-21T08:44:41.251070+0000","last_became_active":"2024-04-21T08:42:17.545730+0000","last_became_peered":"2024-04-21T08:42:17.545730+0000","last_unstale":"2024-04-21T08:44:41.251070+0000","last_undegraded":"2024-04-21T08:44:41.251070+0000","last_fullsized":"2024-04-21T08:44:41.251070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:42.802309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374618+0000","last_change":"2024-04-21T08:42:21.655802+0000","last_active":"2024-04-21T08:44:41.374618+0000","last_peered":"2024-04-21T08:44:41.374618+0000","last_clean":"2024-04-21T08:44:41.374618+0000","last_became_active":"2024-04-21T08:42:21.655379+0000","last_became_peered":"2024-04-21T08:42:21.655379+0000","last_unstale":"2024-04-21T08:44:41.374618+0000","last_undegraded":"2024-04-21T08:44:41.374618+0000","last_fullsized":"2024-04-21T08:44:41.374618+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:49.486057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251462+0000","last_change":"2024-04-21T08:42:16.642517+0000","last_active":"2024-04-21T08:44:41.251462+0000","last_peered":"2024-04-21T08:44:41.251462+0000","last_clean":"2024-04-21T08:44:41.251462+0000","last_became_active":"2024-04-21T08:42:16.641539+0000","last_became_peered":"2024-04-21T08:42:16.641539+0000","last_unstale":"2024-04-21T08:44:41.251462+0000","last_undegraded":"2024-04-21T08:44:41.251462+0000","last_fullsized":"2024-04-21T08:44:41.251462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:53.912199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418089+0000","last_change":"2024-04-21T08:42:17.550172+0000","last_active":"2024-04-21T08:44:41.418089+0000","last_peered":"2024-04-21T08:44:41.418089+0000","last_clean":"2024-04-21T08:44:41.418089+0000","last_became_active":"2024-04-21T08:42:17.549824+0000","last_became_peered":"2024-04-21T08:42:17.549824+0000","last_unstale":"2024-04-21T08:44:41.418089+0000","last_undegraded":"2024-04-21T08:44:41.418089+0000","last_fullsized":"2024-04-21T08:44:41.418089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:59.553645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251154+0000","last_change":"2024-04-21T08:44:11.774778+0000","last_active":"2024-04-21T08:44:41.251154+0000","last_peered":"2024-04-21T08:44:41.251154+0000","last_clean":"2024-04-21T08:44:41.251154+0000","last_became_active":"2024-04-21T08:44:11.774190+0000","last_became_peered":"2024-04-21T08:44:11.774190+0000","last_unstale":"2024-04-21T08:44:41.251154+0000","last_undegraded":"2024-04-21T08:44:41.251154+0000","last_fullsized":"2024-04-21T08:44:41.251154+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:06.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374515+0000","last_change":"2024-04-21T08:42:16.647985+0000","last_active":"2024-04-21T08:44:41.374515+0000","last_peered":"2024-04-21T08:44:41.374515+0000","last_clean":"2024-04-21T08:44:41.374515+0000","last_became_active":"2024-04-21T08:42:16.647577+0000","last_became_peered":"2024-04-21T08:42:16.647577+0000","last_unstale":"2024-04-21T08:44:41.374515+0000","last_undegraded":"2024-04-21T08:44:41.374515+0000","last_fullsized":"2024-04-21T08:44:41.374515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:00.731708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250576+0000","last_change":"2024-04-21T08:42:17.541757+0000","last_active":"2024-04-21T08:44:41.250576+0000","last_peered":"2024-04-21T08:44:41.250576+0000","last_clean":"2024-04-21T08:44:41.250576+0000","last_became_active":"2024-04-21T08:42:17.539280+0000","last_became_peered":"2024-04-21T08:42:17.539280+0000","last_unstale":"2024-04-21T08:44:41.250576+0000","last_undegraded":"2024-04-21T08:44:41.250576+0000","last_fullsized":"2024-04-21T08:44:41.250576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:45:54.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356961+0000","last_change":"2024-04-21T08:42:21.664566+0000","last_active":"2024-04-21T08:44:41.356961+0000","last_peered":"2024-04-21T08:44:41.356961+0000","last_clean":"2024-04-21T08:44:41.356961+0000","last_became_active":"2024-04-21T08:42:21.664136+0000","last_became_peered":"2024-04-21T08:42:21.664136+0000","last_unstale":"2024-04-21T08:44:41.356961+0000","last_undegraded":"2024-04-21T08:44:41.356961+0000","last_fullsized":"2024-04-21T08:44:41.356961+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:19.856577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249918+0000","last_change":"2024-04-21T08:42:16.650664+0000","last_active":"2024-04-21T08:44:41.249918+0000","last_peered":"2024-04-21T08:44:41.249918+0000","last_clean":"2024-04-21T08:44:41.249918+0000","last_became_active":"2024-04-21T08:42:16.650339+0000","last_became_peered":"2024-04-21T08:42:16.650339+0000","last_unstale":"2024-04-21T08:44:41.249918+0000","last_undegraded":"2024-04-21T08:44:41.249918+0000","last_fullsized":"2024-04-21T08:44:41.249918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:59.565455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374844+0000","last_change":"2024-04-21T08:42:17.538142+0000","last_active":"2024-04-21T08:44:41.374844+0000","last_peered":"2024-04-21T08:44:41.374844+0000","last_clean":"2024-04-21T08:44:41.374844+0000","last_became_active":"2024-04-21T08:42:17.537793+0000","last_became_peered":"2024-04-21T08:42:17.537793+0000","last_unstale":"2024-04-21T08:44:41.374844+0000","last_undegraded":"2024-04-21T08:44:41.374844+0000","last_fullsized":"2024-04-21T08:44:41.374844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:11.059294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251170+0000","last_change":"2024-04-21T08:42:21.670642+0000","last_active":"2024-04-21T08:44:41.251170+0000","last_peered":"2024-04-21T08:44:41.251170+0000","last_clean":"2024-04-21T08:44:41.251170+0000","last_became_active":"2024-04-21T08:42:21.670128+0000","last_became_peered":"2024-04-21T08:42:21.670128+0000","last_unstale":"2024-04-21T08:44:41.251170+0000","last_undegraded":"2024-04-21T08:44:41.251170+0000","last_fullsized":"2024-04-21T08:44:41.251170+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:51:28.604381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375278+0000","last_change":"2024-04-21T08:43:10.792114+0000","last_active":"2024-04-21T08:44:41.375278+0000","last_peered":"2024-04-21T08:44:41.375278+0000","last_clean":"2024-04-21T08:44:41.375278+0000","last_became_active":"2024-04-21T08:43:10.791664+0000","last_became_peered":"2024-04-21T08:43:10.791664+0000","last_unstale":"2024-04-21T08:44:41.375278+0000","last_undegraded":"2024-04-21T08:44:41.375278+0000","last_fullsized":"2024-04-21T08:44:41.375278+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:07.317804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250083+0000","last_change":"2024-04-21T08:42:17.547326+0000","last_active":"2024-04-21T08:44:41.250083+0000","last_peered":"2024-04-21T08:44:41.250083+0000","last_clean":"2024-04-21T08:44:41.250083+0000","last_became_active":"2024-04-21T08:42:17.546920+0000","last_became_peered":"2024-04-21T08:42:17.546920+0000","last_unstale":"2024-04-21T08:44:41.250083+0000","last_undegraded":"2024-04-21T08:44:41.250083+0000","last_fullsized":"2024-04-21T08:44:41.250083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:43.350877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251167+0000","last_change":"2024-04-21T08:42:21.971147+0000","last_active":"2024-04-21T08:44:41.251167+0000","last_peered":"2024-04-21T08:44:41.251167+0000","last_clean":"2024-04-21T08:44:41.251167+0000","last_became_active":"2024-04-21T08:42:21.970001+0000","last_became_peered":"2024-04-21T08:42:21.970001+0000","last_unstale":"2024-04-21T08:44:41.251167+0000","last_undegraded":"2024-04-21T08:44:41.251167+0000","last_fullsized":"2024-04-21T08:44:41.251167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:01:44.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250876+0000","last_change":"2024-04-21T08:42:16.643371+0000","last_active":"2024-04-21T08:44:41.250876+0000","last_peered":"2024-04-21T08:44:41.250876+0000","last_clean":"2024-04-21T08:44:41.250876+0000","last_became_active":"2024-04-21T08:42:16.642906+0000","last_became_peered":"2024-04-21T08:42:16.642906+0000","last_unstale":"2024-04-21T08:44:41.250876+0000","last_undegraded":"2024-04-21T08:44:41.250876+0000","last_fullsized":"2024-04-21T08:44:41.250876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:58.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375124+0000","last_change":"2024-04-21T08:42:17.545185+0000","last_active":"2024-04-21T08:44:41.375124+0000","last_peered":"2024-04-21T08:44:41.375124+0000","last_clean":"2024-04-21T08:44:41.375124+0000","last_became_active":"2024-04-21T08:42:17.544757+0000","last_became_peered":"2024-04-21T08:42:17.544757+0000","last_unstale":"2024-04-21T08:44:41.375124+0000","last_undegraded":"2024-04-21T08:44:41.375124+0000","last_fullsized":"2024-04-21T08:44:41.375124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:36:18.968041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374923+0000","last_change":"2024-04-21T08:42:21.662541+0000","last_active":"2024-04-21T08:44:41.374923+0000","last_peered":"2024-04-21T08:44:41.374923+0000","last_clean":"2024-04-21T08:44:41.374923+0000","last_became_active":"2024-04-21T08:42:21.662144+0000","last_became_peered":"2024-04-21T08:42:21.662144+0000","last_unstale":"2024-04-21T08:44:41.374923+0000","last_undegraded":"2024-04-21T08:44:41.374923+0000","last_fullsized":"2024-04-21T08:44:41.374923+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:04.347722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375374+0000","last_change":"2024-04-21T08:42:16.638601+0000","last_active":"2024-04-21T08:44:41.375374+0000","last_peered":"2024-04-21T08:44:41.375374+0000","last_clean":"2024-04-21T08:44:41.375374+0000","last_became_active":"2024-04-21T08:42:16.638211+0000","last_became_peered":"2024-04-21T08:42:16.638211+0000","last_unstale":"2024-04-21T08:44:41.375374+0000","last_undegraded":"2024-04-21T08:44:41.375374+0000","last_fullsized":"2024-04-21T08:44:41.375374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:02.633086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375257+0000","last_change":"2024-04-21T08:42:17.545495+0000","last_active":"2024-04-21T08:44:41.375257+0000","last_peered":"2024-04-21T08:44:41.375257+0000","last_clean":"2024-04-21T08:44:41.375257+0000","last_became_active":"2024-04-21T08:42:17.545159+0000","last_became_peered":"2024-04-21T08:42:17.545159+0000","last_unstale":"2024-04-21T08:44:41.375257+0000","last_undegraded":"2024-04-21T08:44:41.375257+0000","last_fullsized":"2024-04-21T08:44:41.375257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:27.150277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417564+0000","last_change":"2024-04-21T08:42:21.656890+0000","last_active":"2024-04-21T08:44:41.417564+0000","last_peered":"2024-04-21T08:44:41.417564+0000","last_clean":"2024-04-21T08:44:41.417564+0000","last_became_active":"2024-04-21T08:42:21.656475+0000","last_became_peered":"2024-04-21T08:42:21.656475+0000","last_unstale":"2024-04-21T08:44:41.417564+0000","last_undegraded":"2024-04-21T08:44:41.417564+0000","last_fullsized":"2024-04-21T08:44:41.417564+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:46.138591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'39","reported_seq":81,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.478489+0000","last_change":"2024-04-21T08:42:16.638244+0000","last_active":"2024-04-21T08:44:41.478489+0000","last_peered":"2024-04-21T08:44:41.478489+0000","last_clean":"2024-04-21T08:44:41.478489+0000","last_became_active":"2024-04-21T08:42:16.637883+0000","last_became_peered":"2024-04-21T08:42:16.637883+0000","last_unstale":"2024-04-21T08:44:41.478489+0000","last_undegraded":"2024-04-21T08:44:41.478489+0000","last_fullsized":"2024-04-21T08:44:41.478489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:20.888301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374936+0000","last_change":"2024-04-21T08:42:17.548032+0000","last_active":"2024-04-21T08:44:41.374936+0000","last_peered":"2024-04-21T08:44:41.374936+0000","last_clean":"2024-04-21T08:44:41.374936+0000","last_became_active":"2024-04-21T08:42:17.547660+0000","last_became_peered":"2024-04-21T08:42:17.547660+0000","last_unstale":"2024-04-21T08:44:41.374936+0000","last_undegraded":"2024-04-21T08:44:41.374936+0000","last_fullsized":"2024-04-21T08:44:41.374936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:14.710759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365005+0000","last_change":"2024-04-21T08:42:21.659286+0000","last_active":"2024-04-21T08:44:41.365005+0000","last_peered":"2024-04-21T08:44:41.365005+0000","last_clean":"2024-04-21T08:44:41.365005+0000","last_became_active":"2024-04-21T08:42:21.658844+0000","last_became_peered":"2024-04-21T08:42:21.658844+0000","last_unstale":"2024-04-21T08:44:41.365005+0000","last_undegraded":"2024-04-21T08:44:41.365005+0000","last_fullsized":"2024-04-21T08:44:41.365005+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:28.742646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376281+0000","last_change":"2024-04-21T08:42:16.643354+0000","last_active":"2024-04-21T08:44:41.376281+0000","last_peered":"2024-04-21T08:44:41.376281+0000","last_clean":"2024-04-21T08:44:41.376281+0000","last_became_active":"2024-04-21T08:42:16.642967+0000","last_became_peered":"2024-04-21T08:42:16.642967+0000","last_unstale":"2024-04-21T08:44:41.376281+0000","last_undegraded":"2024-04-21T08:44:41.376281+0000","last_fullsized":"2024-04-21T08:44:41.376281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:06:03.495181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375409+0000","last_change":"2024-04-21T08:42:17.539551+0000","last_active":"2024-04-21T08:44:41.375409+0000","last_peered":"2024-04-21T08:44:41.375409+0000","last_clean":"2024-04-21T08:44:41.375409+0000","last_became_active":"2024-04-21T08:42:17.539202+0000","last_became_peered":"2024-04-21T08:42:17.539202+0000","last_unstale":"2024-04-21T08:44:41.375409+0000","last_undegraded":"2024-04-21T08:44:41.375409+0000","last_fullsized":"2024-04-21T08:44:41.375409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:37.109421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375023+0000","last_change":"2024-04-21T08:42:17.544582+0000","last_active":"2024-04-21T08:44:41.375023+0000","last_peered":"2024-04-21T08:44:41.375023+0000","last_clean":"2024-04-21T08:44:41.375023+0000","last_became_active":"2024-04-21T08:42:17.543800+0000","last_became_peered":"2024-04-21T08:42:17.543800+0000","last_unstale":"2024-04-21T08:44:41.375023+0000","last_undegraded":"2024-04-21T08:44:41.375023+0000","last_fullsized":"2024-04-21T08:44:41.375023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:44:04.107207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374756+0000","last_change":"2024-04-21T08:42:17.543788+0000","last_active":"2024-04-21T08:44:41.374756+0000","last_peered":"2024-04-21T08:44:41.374756+0000","last_clean":"2024-04-21T08:44:41.374756+0000","last_became_active":"2024-04-21T08:42:17.540939+0000","last_became_peered":"2024-04-21T08:42:17.540939+0000","last_unstale":"2024-04-21T08:44:41.374756+0000","last_undegraded":"2024-04-21T08:44:41.374756+0000","last_fullsized":"2024-04-21T08:44:41.374756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:59.275098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374942+0000","last_change":"2024-04-21T08:42:17.540605+0000","last_active":"2024-04-21T08:44:41.374942+0000","last_peered":"2024-04-21T08:44:41.374942+0000","last_clean":"2024-04-21T08:44:41.374942+0000","last_became_active":"2024-04-21T08:42:17.540073+0000","last_became_peered":"2024-04-21T08:42:17.540073+0000","last_unstale":"2024-04-21T08:44:41.374942+0000","last_undegraded":"2024-04-21T08:44:41.374942+0000","last_fullsized":"2024-04-21T08:44:41.374942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:02:29.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417931+0000","last_change":"2024-04-21T08:42:17.551608+0000","last_active":"2024-04-21T08:44:41.417931+0000","last_peered":"2024-04-21T08:44:41.417931+0000","last_clean":"2024-04-21T08:44:41.417931+0000","last_became_active":"2024-04-21T08:42:17.551117+0000","last_became_peered":"2024-04-21T08:42:17.551117+0000","last_unstale":"2024-04-21T08:44:41.417931+0000","last_undegraded":"2024-04-21T08:44:41.417931+0000","last_fullsized":"2024-04-21T08:44:41.417931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:18.391928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375875+0000","last_change":"2024-04-21T08:42:17.536509+0000","last_active":"2024-04-21T08:44:41.375875+0000","last_peered":"2024-04-21T08:44:41.375875+0000","last_clean":"2024-04-21T08:44:41.375875+0000","last_became_active":"2024-04-21T08:42:17.536146+0000","last_became_peered":"2024-04-21T08:42:17.536146+0000","last_unstale":"2024-04-21T08:44:41.375875+0000","last_undegraded":"2024-04-21T08:44:41.375875+0000","last_fullsized":"2024-04-21T08:44:41.375875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:29.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375199+0000","last_change":"2024-04-21T08:42:17.543786+0000","last_active":"2024-04-21T08:44:41.375199+0000","last_peered":"2024-04-21T08:44:41.375199+0000","last_clean":"2024-04-21T08:44:41.375199+0000","last_became_active":"2024-04-21T08:42:17.543435+0000","last_became_peered":"2024-04-21T08:42:17.543435+0000","last_unstale":"2024-04-21T08:44:41.375199+0000","last_undegraded":"2024-04-21T08:44:41.375199+0000","last_fullsized":"2024-04-21T08:44:41.375199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375616+0000","last_change":"2024-04-21T08:42:17.541611+0000","last_active":"2024-04-21T08:44:41.375616+0000","last_peered":"2024-04-21T08:44:41.375616+0000","last_clean":"2024-04-21T08:44:41.375616+0000","last_became_active":"2024-04-21T08:42:17.541259+0000","last_became_peered":"2024-04-21T08:42:17.541259+0000","last_unstale":"2024-04-21T08:44:41.375616+0000","last_undegraded":"2024-04-21T08:44:41.375616+0000","last_fullsized":"2024-04-21T08:44:41.375616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:20.900629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356654+0000","last_change":"2024-04-21T08:44:11.774062+0000","last_active":"2024-04-21T08:44:41.356654+0000","last_peered":"2024-04-21T08:44:41.356654+0000","last_clean":"2024-04-21T08:44:41.356654+0000","last_became_active":"2024-04-21T08:44:11.773703+0000","last_became_peered":"2024-04-21T08:44:11.773703+0000","last_unstale":"2024-04-21T08:44:41.356654+0000","last_undegraded":"2024-04-21T08:44:41.356654+0000","last_fullsized":"2024-04-21T08:44:41.356654+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:10.739068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365197+0000","last_change":"2024-04-21T08:42:17.545526+0000","last_active":"2024-04-21T08:44:41.365197+0000","last_peered":"2024-04-21T08:44:41.365197+0000","last_clean":"2024-04-21T08:44:41.365197+0000","last_became_active":"2024-04-21T08:42:17.545135+0000","last_became_peered":"2024-04-21T08:42:17.545135+0000","last_unstale":"2024-04-21T08:44:41.365197+0000","last_undegraded":"2024-04-21T08:44:41.365197+0000","last_fullsized":"2024-04-21T08:44:41.365197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:37:58.038459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250029+0000","last_change":"2024-04-21T08:42:17.547063+0000","last_active":"2024-04-21T08:44:41.250029+0000","last_peered":"2024-04-21T08:44:41.250029+0000","last_clean":"2024-04-21T08:44:41.250029+0000","last_became_active":"2024-04-21T08:42:17.546689+0000","last_became_peered":"2024-04-21T08:42:17.546689+0000","last_unstale":"2024-04-21T08:44:41.250029+0000","last_undegraded":"2024-04-21T08:44:41.250029+0000","last_fullsized":"2024-04-21T08:44:41.250029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:26.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375261+0000","last_change":"2024-04-21T08:42:17.542545+0000","last_active":"2024-04-21T08:44:41.375261+0000","last_peered":"2024-04-21T08:44:41.375261+0000","last_clean":"2024-04-21T08:44:41.375261+0000","last_became_active":"2024-04-21T08:42:17.542147+0000","last_became_peered":"2024-04-21T08:42:17.542147+0000","last_unstale":"2024-04-21T08:44:41.375261+0000","last_undegraded":"2024-04-21T08:44:41.375261+0000","last_fullsized":"2024-04-21T08:44:41.375261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:15.064110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376582+0000","last_change":"2024-04-21T08:42:17.541262+0000","last_active":"2024-04-21T08:44:41.376582+0000","last_peered":"2024-04-21T08:44:41.376582+0000","last_clean":"2024-04-21T08:44:41.376582+0000","last_became_active":"2024-04-21T08:42:17.540817+0000","last_became_peered":"2024-04-21T08:42:17.540817+0000","last_unstale":"2024-04-21T08:44:41.376582+0000","last_undegraded":"2024-04-21T08:44:41.376582+0000","last_fullsized":"2024-04-21T08:44:41.376582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:22:59.597680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375116+0000","last_change":"2024-04-21T08:42:17.551109+0000","last_active":"2024-04-21T08:44:41.375116+0000","last_peered":"2024-04-21T08:44:41.375116+0000","last_clean":"2024-04-21T08:44:41.375116+0000","last_became_active":"2024-04-21T08:42:17.550104+0000","last_became_peered":"2024-04-21T08:42:17.550104+0000","last_unstale":"2024-04-21T08:44:41.375116+0000","last_undegraded":"2024-04-21T08:44:41.375116+0000","last_fullsized":"2024-04-21T08:44:41.375116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:22:19.361752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107298,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595682,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":350,"num_read_kb":280,"num_write":201,"num_write_kb":1411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":1787046,"data_compressed":80802,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":532412,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612655,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36496,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708656,"statfs":{"total":95995035648,"available":95957663744,"internally_reserved":0,"allocated":1183744,"data_stored":1314391,"data_compressed":73179,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776181,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1069056,"data_stored":930306,"data_compressed":48872,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":926232,"data_compressed":48872,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":1127458,"data_compressed":65492,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528316,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1105920,"data_stored":598384,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1036288,"data_stored":528406,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":598418,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1053236,"data_compressed":55028,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":532434,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1118208,"data_stored":1254562,"data_compressed":71648,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70102,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":201862,"data_compressed":7788,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:44:44.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.506+0000 7f90ec070640 1 -- 172.21.15.7:0/359104360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4105390 msgr2=0x7f90e4107780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.506+0000 7f90ec070640 1 --2- 172.21.15.7:0/359104360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4105390 0x7f90e4107780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f90d4009860 tx=0x7f90d402bb70 comp rx=0 tx=0).stop 2024-04-21T08:44:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.506+0000 7f90ec070640 1 -- 172.21.15.7:0/359104360 shutdown_connections 2024-04-21T08:44:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.506+0000 7f90ec070640 1 --2- 172.21.15.7:0/359104360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90e4107cc0 0x7f90e410a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.506+0000 7f90ec070640 1 --2- 172.21.15.7:0/359104360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4105390 0x7f90e4107780 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.506+0000 7f90ec070640 1 --2- 172.21.15.7:0/359104360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e4102a60 0x7f90e4104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.506+0000 7f90ec070640 1 -- 172.21.15.7:0/359104360 >> 172.21.15.7:0/359104360 conn(0x7f90e40fc3f0 msgr2=0x7f90e40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.506+0000 7f90ec070640 1 -- 172.21.15.7:0/359104360 shutdown_connections 2024-04-21T08:44:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.506+0000 7f90ec070640 1 -- 172.21.15.7:0/359104360 wait complete. 2024-04-21T08:44:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90ec070640 1 Processor -- start 2024-04-21T08:44:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90ec070640 1 -- start start 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90ec070640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90e4102a60 0x7f90e41023d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90ec070640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4105390 0x7f90e4100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90ec070640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e4107cc0 0x7f90e4100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90ec070640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f90e410da40 con 0x7f90e4102a60 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90ec070640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90e410d8c0 con 0x7f90e4107cc0 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90ec070640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f90e410dbc0 con 0x7f90e4105390 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90e9e0c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90e4102a60 0x7f90e41023d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90e9e0c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90e4102a60 0x7f90e41023d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53734/0 (socket says 172.21.15.7:53734) 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90e9e0c640 1 -- 172.21.15.7:0/2422856684 learned_addr learned my addr 172.21.15.7:0/2422856684 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90ea60d640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e4107cc0 0x7f90e4100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90e960b640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4105390 0x7f90e4100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90e9e0c640 1 -- 172.21.15.7:0/2422856684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4105390 msgr2=0x7f90e4100a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90e9e0c640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4105390 0x7f90e4100a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90e9e0c640 1 -- 172.21.15.7:0/2422856684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e4107cc0 msgr2=0x7f90e4100f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90e9e0c640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e4107cc0 0x7f90e4100f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90e9e0c640 1 -- 172.21.15.7:0/2422856684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e41017e0 con 0x7f90e4102a60 2024-04-21T08:44:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90ea60d640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e4107cc0 0x7f90e4100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.507+0000 7f90e960b640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4105390 0x7f90e4100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.508+0000 7f90e9e0c640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90e4102a60 0x7f90e41023d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f90e000e9f0 tx=0x7f90e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.508+0000 7f90daffd640 1 -- 172.21.15.7:0/2422856684 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e000cd80 con 0x7f90e4102a60 2024-04-21T08:44:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.508+0000 7f90ec070640 1 -- 172.21.15.7:0/2422856684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e41a22c0 con 0x7f90e4102a60 2024-04-21T08:44:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.508+0000 7f90daffd640 1 -- 172.21.15.7:0/2422856684 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90e00108e0 con 0x7f90e4102a60 2024-04-21T08:44:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.508+0000 7f90daffd640 1 -- 172.21.15.7:0/2422856684 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e0018980 con 0x7f90e4102a60 2024-04-21T08:44:44.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.508+0000 7f90ec070640 1 -- 172.21.15.7:0/2422856684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e41a2830 con 0x7f90e4102a60 2024-04-21T08:44:44.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.509+0000 7f90ec070640 1 -- 172.21.15.7:0/2422856684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e40fdee0 con 0x7f90e4102a60 2024-04-21T08:44:44.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.510+0000 7f90daffd640 1 -- 172.21.15.7:0/2422856684 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 23) ==== 138765+0+0 (secure 0 0 0) 0x7f90e0010430 con 0x7f90e4102a60 2024-04-21T08:44:44.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.511+0000 7f90daffd640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90d00a3d20 0x7f90d00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.511+0000 7f90daffd640 1 -- 172.21.15.7:0/2422856684 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f90e00d1380 con 0x7f90e4102a60 2024-04-21T08:44:44.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.511+0000 7f90e960b640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90d00a3d20 0x7f90d00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.512+0000 7f90e960b640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90d00a3d20 0x7f90d00a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f90d40348c0 tx=0x7f90d4034990 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:44.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.519+0000 7f90daffd640 1 -- 172.21.15.7:0/2422856684 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f90e009cfe0 con 0x7f90e4102a60 2024-04-21T08:44:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.709+0000 7f90ec070640 1 -- 172.21.15.7:0/2422856684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90e4062b50 con 0x7f90e4102a60 2024-04-21T08:44:44.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.710+0000 7f90daffd640 1 -- 172.21.15.7:0/2422856684 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25085 (secure 0 0 0) 0x7f90e00a12e0 con 0x7f90e4102a60 2024-04-21T08:44:44.710 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:44.710 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":95,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:44:41.244735+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3451243741":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/117789107":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/4167672076":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/3630897301":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:44.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.712+0000 7f90ec070640 1 -- 172.21.15.7:0/2422856684 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90d00a3d20 msgr2=0x7f90d00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.712+0000 7f90ec070640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90d00a3d20 0x7f90d00a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f90d40348c0 tx=0x7f90d4034990 comp rx=0 tx=0).stop 2024-04-21T08:44:44.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.712+0000 7f90ec070640 1 -- 172.21.15.7:0/2422856684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90e4102a60 msgr2=0x7f90e41023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.712+0000 7f90ec070640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90e4102a60 0x7f90e41023d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f90e000e9f0 tx=0x7f90e000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.712+0000 7f90ec070640 1 -- 172.21.15.7:0/2422856684 shutdown_connections 2024-04-21T08:44:44.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.712+0000 7f90ec070640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90e4107cc0 0x7f90e4100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.712+0000 7f90ec070640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90d00a3d20 0x7f90d00a61e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.712+0000 7f90ec070640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90e4105390 0x7f90e4100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.712+0000 7f90ec070640 1 --2- 172.21.15.7:0/2422856684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90e4102a60 0x7f90e41023d0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.712+0000 7f90ec070640 1 -- 172.21.15.7:0/2422856684 >> 172.21.15.7:0/2422856684 conn(0x7f90e40fc3f0 msgr2=0x7f90e41048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:44.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.712+0000 7f90ec070640 1 -- 172.21.15.7:0/2422856684 shutdown_connections 2024-04-21T08:44:44.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.713+0000 7f90ec070640 1 -- 172.21.15.7:0/2422856684 wait complete. 2024-04-21T08:44:44.723 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:44.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.839+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/1644582115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd78103b50 msgr2=0x7fdd7810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.839+0000 7fdd7e3b2640 1 --2- 172.21.15.7:0/1644582115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd78103b50 0x7fdd7810a3e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009ca0 tx=0x7fdd7002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.839+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/1644582115 shutdown_connections 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.839+0000 7fdd7e3b2640 1 --2- 172.21.15.7:0/1644582115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd78103b50 0x7fdd7810a3e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.839+0000 7fdd7e3b2640 1 --2- 172.21.15.7:0/1644582115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd78103190 0x7fdd78103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.839+0000 7fdd7e3b2640 1 --2- 172.21.15.7:0/1644582115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd78101fa0 0x7fdd781023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.839+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/1644582115 >> 172.21.15.7:0/1644582115 conn(0x7fdd780fd530 msgr2=0x7fdd780ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.839+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/1644582115 shutdown_connections 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.839+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/1644582115 wait complete. 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd7e3b2640 1 Processor -- start 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd7e3b2640 1 -- start start 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd7e3b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd78101fa0 0x7fdd78075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd7e3b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd78103190 0x7fdd78074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd7e3b2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd78103b50 0x7fdd78074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd7e3b2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7810dae0 con 0x7fdd78103b50 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd7e3b2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7810d960 con 0x7fdd78101fa0 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd7e3b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7810dc60 con 0x7fdd78103190 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd78101fa0 0x7fdd78075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd7c94f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd78103b50 0x7fdd78074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd77fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd78101fa0 0x7fdd78075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53360/0 (socket says 172.21.15.7:53360) 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd77fff640 1 -- 172.21.15.7:0/745460848 learned_addr learned my addr 172.21.15.7:0/745460848 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd777fe640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd78103190 0x7fdd78074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd77fff640 1 -- 172.21.15.7:0/745460848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd78103190 msgr2=0x7fdd78074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd77fff640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd78103190 0x7fdd78074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd77fff640 1 -- 172.21.15.7:0/745460848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd78103b50 msgr2=0x7fdd78074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd77fff640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd78103b50 0x7fdd78074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.840+0000 7fdd77fff640 1 -- 172.21.15.7:0/745460848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd78075110 con 0x7fdd78101fa0 2024-04-21T08:44:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.841+0000 7fdd77fff640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd78101fa0 0x7fdd78075d00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800e9c0 tx=0x7fdd6800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.841+0000 7fdd757fa640 1 -- 172.21.15.7:0/745460848 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6800cd80 con 0x7fdd78101fa0 2024-04-21T08:44:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.841+0000 7fdd757fa640 1 -- 172.21.15.7:0/745460848 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd680035f0 con 0x7fdd78101fa0 2024-04-21T08:44:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.841+0000 7fdd757fa640 1 -- 172.21.15.7:0/745460848 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd68010690 con 0x7fdd78101fa0 2024-04-21T08:44:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.841+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/745460848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd781a2280 con 0x7fdd78101fa0 2024-04-21T08:44:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.841+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/745460848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd781a2780 con 0x7fdd78101fa0 2024-04-21T08:44:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.842+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/745460848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd78109330 con 0x7fdd78101fa0 2024-04-21T08:44:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.843+0000 7fdd757fa640 1 -- 172.21.15.7:0/745460848 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 138765+0+0 (secure 0 0 0) 0x7fdd68004030 con 0x7fdd78101fa0 2024-04-21T08:44:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.843+0000 7fdd757fa640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdd600a3d20 0x7fdd600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.844+0000 7fdd757fa640 1 -- 172.21.15.7:0/745460848 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fdd680d1140 con 0x7fdd78101fa0 2024-04-21T08:44:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.844+0000 7fdd777fe640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdd600a3d20 0x7fdd600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.844+0000 7fdd777fe640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdd600a3d20 0x7fdd600a61e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd78075a70 tx=0x7fdd6c005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:44.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:44.848+0000 7fdd757fa640 1 -- 172.21.15.7:0/745460848 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd6809d3f0 con 0x7fdd78101fa0 2024-04-21T08:44:45.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.038+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/745460848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdd7802d850 con 0x7fdd78101fa0 2024-04-21T08:44:45.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.039+0000 7fdd757fa640 1 -- 172.21.15.7:0/745460848 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7fdd6809ccf0 con 0x7fdd78101fa0 2024-04-21T08:44:45.038 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:44:45.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.041+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/745460848 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdd600a3d20 msgr2=0x7fdd600a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.041+0000 7fdd7e3b2640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdd600a3d20 0x7fdd600a61e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd78075a70 tx=0x7fdd6c005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:45.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.041+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/745460848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd78101fa0 msgr2=0x7fdd78075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.041+0000 7fdd7e3b2640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd78101fa0 0x7fdd78075d00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800e9c0 tx=0x7fdd6800ee80 comp rx=0 tx=0).stop 2024-04-21T08:44:45.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.041+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/745460848 shutdown_connections 2024-04-21T08:44:45.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.041+0000 7fdd7e3b2640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd78103b50 0x7fdd78074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.041+0000 7fdd7e3b2640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdd600a3d20 0x7fdd600a61e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.041+0000 7fdd7e3b2640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd78103190 0x7fdd78074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.041+0000 7fdd7e3b2640 1 --2- 172.21.15.7:0/745460848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd78101fa0 0x7fdd78075d00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.041+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/745460848 >> 172.21.15.7:0/745460848 conn(0x7fdd780fd530 msgr2=0x7fdd78106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:45.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.042+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/745460848 shutdown_connections 2024-04-21T08:44:45.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.042+0000 7fdd7e3b2640 1 -- 172.21.15.7:0/745460848 wait complete. 2024-04-21T08:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T08:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2422856684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:44 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:44:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T08:44:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2422856684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:44 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.163+0000 7fba803ce640 1 -- 172.21.15.7:0/914034481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba781053b0 msgr2=0x7fba781077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.163+0000 7fba803ce640 1 --2- 172.21.15.7:0/914034481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba781053b0 0x7fba781077a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fba68009900 tx=0x7fba6802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.163+0000 7fba803ce640 1 -- 172.21.15.7:0/914034481 shutdown_connections 2024-04-21T08:44:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.163+0000 7fba803ce640 1 --2- 172.21.15.7:0/914034481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba78107ce0 0x7fba7810a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.163+0000 7fba803ce640 1 --2- 172.21.15.7:0/914034481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba781053b0 0x7fba781077a0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.163+0000 7fba803ce640 1 --2- 172.21.15.7:0/914034481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba78102a80 0x7fba78104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.163+0000 7fba803ce640 1 -- 172.21.15.7:0/914034481 >> 172.21.15.7:0/914034481 conn(0x7fba780fc430 msgr2=0x7fba780fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.163+0000 7fba803ce640 1 -- 172.21.15.7:0/914034481 shutdown_connections 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.163+0000 7fba803ce640 1 -- 172.21.15.7:0/914034481 wait complete. 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba803ce640 1 Processor -- start 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba803ce640 1 -- start start 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba803ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78102a80 0x7fba7819a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba803ce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba781053b0 0x7fba7819a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba803ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba78107ce0 0x7fba781a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba803ce640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810dae0 con 0x7fba78102a80 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba803ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810d960 con 0x7fba78107ce0 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba803ce640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810dc60 con 0x7fba781053b0 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7e16a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78102a80 0x7fba7819a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7e16a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78102a80 0x7fba7819a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47126/0 (socket says 172.21.15.7:47126) 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7e16a640 1 -- 172.21.15.7:0/2676965105 learned_addr learned my addr 172.21.15.7:0/2676965105 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7d969640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba781053b0 0x7fba7819a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7e96b640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba78107ce0 0x7fba781a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7d969640 1 -- 172.21.15.7:0/2676965105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba78107ce0 msgr2=0x7fba781a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7d969640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba78107ce0 0x7fba781a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7d969640 1 -- 172.21.15.7:0/2676965105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78102a80 msgr2=0x7fba7819a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7d969640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78102a80 0x7fba7819a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7d969640 1 -- 172.21.15.7:0/2676965105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba781a2050 con 0x7fba781053b0 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7e16a640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78102a80 0x7fba7819a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.164+0000 7fba7e96b640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba78107ce0 0x7fba781a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.165+0000 7fba7d969640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba781053b0 0x7fba7819a970 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fba68036040 tx=0x7fba68005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.165+0000 7fba6f7fe640 1 -- 172.21.15.7:0/2676965105 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68043020 con 0x7fba781053b0 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.165+0000 7fba6f7fe640 1 -- 172.21.15.7:0/2676965105 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba68004350 con 0x7fba781053b0 2024-04-21T08:44:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.165+0000 7fba6f7fe640 1 -- 172.21.15.7:0/2676965105 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68003ab0 con 0x7fba781053b0 2024-04-21T08:44:45.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.165+0000 7fba803ce640 1 -- 172.21.15.7:0/2676965105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba781a22e0 con 0x7fba781053b0 2024-04-21T08:44:45.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.165+0000 7fba803ce640 1 -- 172.21.15.7:0/2676965105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba781a2850 con 0x7fba781053b0 2024-04-21T08:44:45.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.166+0000 7fba803ce640 1 -- 172.21.15.7:0/2676965105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba78068570 con 0x7fba781053b0 2024-04-21T08:44:45.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.167+0000 7fba6f7fe640 1 -- 172.21.15.7:0/2676965105 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 23) ==== 138765+0+0 (secure 0 0 0) 0x7fba68046020 con 0x7fba781053b0 2024-04-21T08:44:45.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.168+0000 7fba6f7fe640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fba640a3da0 0x7fba640a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.168+0000 7fba7e16a640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fba640a3da0 0x7fba640a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.168+0000 7fba6f7fe640 1 -- 172.21.15.7:0/2676965105 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fba680f2b50 con 0x7fba781053b0 2024-04-21T08:44:45.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.169+0000 7fba7e16a640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fba640a3da0 0x7fba640a6260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba74011550 tx=0x7fba74005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:45.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.172+0000 7fba6f7fe640 1 -- 172.21.15.7:0/2676965105 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba6804d020 con 0x7fba781053b0 2024-04-21T08:44:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:44:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T08:44:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2422856684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:44 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:45.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.389+0000 7fba803ce640 1 -- 172.21.15.7:0/2676965105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fba7802d5c0 con 0x7fba781053b0 2024-04-21T08:44:45.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.390+0000 7fba6f7fe640 1 -- 172.21.15.7:0/2676965105 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7fba680bef10 con 0x7fba781053b0 2024-04-21T08:44:45.389 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:45.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 -- 172.21.15.7:0/2676965105 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fba640a3da0 msgr2=0x7fba640a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fba640a3da0 0x7fba640a6260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba74011550 tx=0x7fba74005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:45.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 -- 172.21.15.7:0/2676965105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba781053b0 msgr2=0x7fba7819a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba781053b0 0x7fba7819a970 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fba68036040 tx=0x7fba68005dc0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 -- 172.21.15.7:0/2676965105 shutdown_connections 2024-04-21T08:44:45.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba78107ce0 0x7fba781a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fba640a3da0 0x7fba640a6260 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba781053b0 0x7fba7819a970 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 --2- 172.21.15.7:0/2676965105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78102a80 0x7fba7819a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 -- 172.21.15.7:0/2676965105 >> 172.21.15.7:0/2676965105 conn(0x7fba780fc430 msgr2=0x7fba781021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:45.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 -- 172.21.15.7:0/2676965105 shutdown_connections 2024-04-21T08:44:45.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.392+0000 7fba803ce640 1 -- 172.21.15.7:0/2676965105 wait complete. 2024-04-21T08:44:45.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.526+0000 7f1020115640 1 -- 172.21.15.7:0/140370712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1018100e90 msgr2=0x7f10181012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.526+0000 7f1020115640 1 --2- 172.21.15.7:0/140370712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1018100e90 0x7f10181012b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1008009ca0 tx=0x7f100802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.526+0000 7f1020115640 1 -- 172.21.15.7:0/140370712 shutdown_connections 2024-04-21T08:44:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.526+0000 7f1020115640 1 --2- 172.21.15.7:0/140370712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f10181029a0 0x7f10181092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.526+0000 7f1020115640 1 --2- 172.21.15.7:0/140370712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1018101fe0 0x7f1018102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.526+0000 7f1020115640 1 --2- 172.21.15.7:0/140370712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1018100e90 0x7f10181012b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.526+0000 7f1020115640 1 -- 172.21.15.7:0/140370712 >> 172.21.15.7:0/140370712 conn(0x7f10180fc420 msgr2=0x7f10180fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.526+0000 7f1020115640 1 -- 172.21.15.7:0/140370712 shutdown_connections 2024-04-21T08:44:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.526+0000 7f1020115640 1 -- 172.21.15.7:0/140370712 wait complete. 2024-04-21T08:44:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f1020115640 1 Processor -- start 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f1020115640 1 -- start start 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f1020115640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1018100e90 0x7f1018104e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f1020115640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1018101fe0 0x7f1018103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f1020115640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10181029a0 0x7f1018103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f1020115640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f101810c9c0 con 0x7f10181029a0 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f1020115640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f101810c840 con 0x7f1018100e90 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f1020115640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f101810cb40 con 0x7f1018101fe0 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f101e6b2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10181029a0 0x7f1018103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f101deb1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1018100e90 0x7f1018104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f101deb1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1018100e90 0x7f1018104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53382/0 (socket says 172.21.15.7:53382) 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f101e6b2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10181029a0 0x7f1018103990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47148/0 (socket says 172.21.15.7:47148) 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f101deb1640 1 -- 172.21.15.7:0/2817285720 learned_addr learned my addr 172.21.15.7:0/2817285720 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f101d6b0640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1018101fe0 0x7f1018103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f101deb1640 1 -- 172.21.15.7:0/2817285720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1018101fe0 msgr2=0x7f1018103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f101deb1640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1018101fe0 0x7f1018103450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f101deb1640 1 -- 172.21.15.7:0/2817285720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10181029a0 msgr2=0x7f1018103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.527+0000 7f101deb1640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10181029a0 0x7f1018103990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.528+0000 7f101deb1640 1 -- 172.21.15.7:0/2817285720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1018104220 con 0x7f1018100e90 2024-04-21T08:44:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.528+0000 7f101deb1640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1018100e90 0x7f1018104e10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1008005e30 tx=0x7f10080346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.528+0000 7f1016ffd640 1 -- 172.21.15.7:0/2817285720 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008041020 con 0x7f1018100e90 2024-04-21T08:44:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.528+0000 7f1016ffd640 1 -- 172.21.15.7:0/2817285720 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1008034ca0 con 0x7f1018100e90 2024-04-21T08:44:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.528+0000 7f1020115640 1 -- 172.21.15.7:0/2817285720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10181a2300 con 0x7f1018100e90 2024-04-21T08:44:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.528+0000 7f1016ffd640 1 -- 172.21.15.7:0/2817285720 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008039040 con 0x7f1018100e90 2024-04-21T08:44:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.528+0000 7f1020115640 1 -- 172.21.15.7:0/2817285720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10181a2820 con 0x7f1018100e90 2024-04-21T08:44:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.529+0000 7f1020115640 1 -- 172.21.15.7:0/2817285720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10181a2a80 con 0x7f1018100e90 2024-04-21T08:44:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.531+0000 7f1016ffd640 1 -- 172.21.15.7:0/2817285720 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 23) ==== 138765+0+0 (secure 0 0 0) 0x7f1008044020 con 0x7f1018100e90 2024-04-21T08:44:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.531+0000 7f1016ffd640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f10000a3d20 0x7f10000a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.531+0000 7f101d6b0640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f10000a3d20 0x7f10000a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.531+0000 7f1016ffd640 1 -- 172.21.15.7:0/2817285720 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f10080f2820 con 0x7f1018100e90 2024-04-21T08:44:45.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.532+0000 7f101d6b0640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f10000a3d20 0x7f10000a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1018104b80 tx=0x7f100c005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:45.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.536+0000 7f1016ffd640 1 -- 172.21.15.7:0/2817285720 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f100804b020 con 0x7f1018100e90 2024-04-21T08:44:45.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.742+0000 7f1020115640 1 -- 172.21.15.7:0/2817285720 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f10181a2c50 con 0x7f1018100e90 2024-04-21T08:44:45.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.742+0000 7f1016ffd640 1 -- 172.21.15.7:0/2817285720 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f10080bec70 con 0x7f1018100e90 2024-04-21T08:44:45.742 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:44:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 -- 172.21.15.7:0/2817285720 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f10000a3d20 msgr2=0x7f10000a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f10000a3d20 0x7f10000a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1018104b80 tx=0x7f100c005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 -- 172.21.15.7:0/2817285720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1018100e90 msgr2=0x7f1018104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1018100e90 0x7f1018104e10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1008005e30 tx=0x7f10080346f0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 -- 172.21.15.7:0/2817285720 shutdown_connections 2024-04-21T08:44:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10181029a0 0x7f1018103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f10000a3d20 0x7f10000a61e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1018101fe0 0x7f1018103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 --2- 172.21.15.7:0/2817285720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1018100e90 0x7f1018104e10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 -- 172.21.15.7:0/2817285720 >> 172.21.15.7:0/2817285720 conn(0x7f10180fc420 msgr2=0x7f101806a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 -- 172.21.15.7:0/2817285720 shutdown_connections 2024-04-21T08:44:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.745+0000 7f1020115640 1 -- 172.21.15.7:0/2817285720 wait complete. 2024-04-21T08:44:45.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.876+0000 7f4758ed2640 1 -- 172.21.15.7:0/1307615889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f475410c500 msgr2=0x7f475410e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.876+0000 7f4758ed2640 1 --2- 172.21.15.7:0/1307615889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f475410c500 0x7f475410e9a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f474c009ca0 tx=0x7f474c02c180 comp rx=0 tx=0).stop 2024-04-21T08:44:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.876+0000 7f4758ed2640 1 -- 172.21.15.7:0/1307615889 shutdown_connections 2024-04-21T08:44:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.876+0000 7f4758ed2640 1 --2- 172.21.15.7:0/1307615889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f475410c500 0x7f475410e9a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.876+0000 7f4758ed2640 1 --2- 172.21.15.7:0/1307615889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4754103ae0 0x7f475410bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.876+0000 7f4758ed2640 1 --2- 172.21.15.7:0/1307615889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47541031c0 0x7f47541035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.876+0000 7f4758ed2640 1 -- 172.21.15.7:0/1307615889 >> 172.21.15.7:0/1307615889 conn(0x7f4754078000 msgr2=0x7f4754078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.876+0000 7f4758ed2640 1 -- 172.21.15.7:0/1307615889 shutdown_connections 2024-04-21T08:44:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.876+0000 7f4758ed2640 1 -- 172.21.15.7:0/1307615889 wait complete. 2024-04-21T08:44:45.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.877+0000 7f4758ed2640 1 Processor -- start 2024-04-21T08:44:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.877+0000 7f4758ed2640 1 -- start start 2024-04-21T08:44:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.877+0000 7f4758ed2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47541031c0 0x7f47541a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.877+0000 7f4758ed2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4754103ae0 0x7f47541a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.877+0000 7f4758ed2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f475410c500 0x7f475419ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.877+0000 7f4758ed2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f47541014a0 con 0x7f4754103ae0 2024-04-21T08:44:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.877+0000 7f4758ed2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4754101320 con 0x7f47541031c0 2024-04-21T08:44:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.877+0000 7f4758ed2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4754101620 con 0x7f475410c500 2024-04-21T08:44:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f475259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47541031c0 0x7f47541a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f4751d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4754103ae0 0x7f47541a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f475259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47541031c0 0x7f47541a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53400/0 (socket says 172.21.15.7:53400) 2024-04-21T08:44:45.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f4751d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4754103ae0 0x7f47541a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47158/0 (socket says 172.21.15.7:47158) 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f475259c640 1 -- 172.21.15.7:0/3873053900 learned_addr learned my addr 172.21.15.7:0/3873053900 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f4752d9d640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f475410c500 0x7f475419ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f475259c640 1 -- 172.21.15.7:0/3873053900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f475410c500 msgr2=0x7f475419ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f475259c640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f475410c500 0x7f475419ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f475259c640 1 -- 172.21.15.7:0/3873053900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4754103ae0 msgr2=0x7f47541a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f475259c640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4754103ae0 0x7f47541a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f475259c640 1 -- 172.21.15.7:0/3873053900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f475419b410 con 0x7f47541031c0 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f4752d9d640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f475410c500 0x7f475419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f475259c640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47541031c0 0x7f47541a0610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f474400e9f0 tx=0x7f474400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f47437fe640 1 -- 172.21.15.7:0/3873053900 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474400cd80 con 0x7f47541031c0 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f4758ed2640 1 -- 172.21.15.7:0/3873053900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f475419b700 con 0x7f47541031c0 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f47437fe640 1 -- 172.21.15.7:0/3873053900 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47440108e0 con 0x7f47541031c0 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f47437fe640 1 -- 172.21.15.7:0/3873053900 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744018980 con 0x7f47541031c0 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.878+0000 7f4758ed2640 1 -- 172.21.15.7:0/3873053900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47541a7590 con 0x7f47541031c0 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.879+0000 7f4758ed2640 1 -- 172.21.15.7:0/3873053900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4754069a60 con 0x7f47541031c0 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.880+0000 7f47437fe640 1 -- 172.21.15.7:0/3873053900 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f4744010430 con 0x7f47541031c0 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.881+0000 7f47437fe640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f473c0a3db0 0x7f473c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.881+0000 7f4751d9b640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f473c0a3db0 0x7f473c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.881+0000 7f47437fe640 1 -- 172.21.15.7:0/3873053900 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f47440d1100 con 0x7f47541031c0 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.882+0000 7f4751d9b640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f473c0a3db0 0x7f473c0a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f475419be90 tx=0x7f4748005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:45.886+0000 7f47437fe640 1 -- 172.21.15.7:0/3873053900 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f474409d240 con 0x7f47541031c0 2024-04-21T08:44:46.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.077+0000 7f4758ed2640 1 -- 172.21.15.7:0/3873053900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f47541a7760 con 0x7f47541031c0 2024-04-21T08:44:46.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.077+0000 7f47437fe640 1 -- 172.21.15.7:0/3873053900 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f474409cb40 con 0x7f47541031c0 2024-04-21T08:44:46.077 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 -- 172.21.15.7:0/3873053900 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f473c0a3db0 msgr2=0x7f473c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f473c0a3db0 0x7f473c0a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f475419be90 tx=0x7f4748005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 -- 172.21.15.7:0/3873053900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47541031c0 msgr2=0x7f47541a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47541031c0 0x7f47541a0610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f474400e9f0 tx=0x7f474400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 -- 172.21.15.7:0/3873053900 shutdown_connections 2024-04-21T08:44:46.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f473c0a3db0 0x7f473c0a6270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f475410c500 0x7f475419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4754103ae0 0x7f47541a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 --2- 172.21.15.7:0/3873053900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47541031c0 0x7f47541a0610 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 -- 172.21.15.7:0/3873053900 >> 172.21.15.7:0/3873053900 conn(0x7f4754078000 msgr2=0x7f475410d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 -- 172.21.15.7:0/3873053900 shutdown_connections 2024-04-21T08:44:46.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.079+0000 7f4758ed2640 1 -- 172.21.15.7:0/3873053900 wait complete. 2024-04-21T08:44:46.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:46.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:46.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: Unable to set osd_memory_target on smithi016 to 185893478: error parsing value: Value '185893478' is below minimum 939524096 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi007 to 177.2M 2024-04-21T08:44:46.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: Unable to set osd_memory_target on smithi007 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-21T08:44:46.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:46.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:46.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-21T08:44:46.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-21T08:44:46.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-21T08:44:46.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/745460848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:46.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2676965105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:46.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:45 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2817285720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:46.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:46.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: Unable to set osd_memory_target on smithi016 to 185893478: error parsing value: Value '185893478' is below minimum 939524096 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi007 to 177.2M 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: Unable to set osd_memory_target on smithi007 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/745460848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2676965105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:46.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:45 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2817285720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.197+0000 7f687c281640 1 -- 172.21.15.7:0/1586487314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68741072c0 msgr2=0x7f687410ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.197+0000 7f687c281640 1 --2- 172.21.15.7:0/1586487314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68741072c0 0x7f687410ee70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f686c009ca0 tx=0x7f686c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.197+0000 7f687c281640 1 -- 172.21.15.7:0/1586487314 shutdown_connections 2024-04-21T08:44:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.197+0000 7f687c281640 1 --2- 172.21.15.7:0/1586487314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68741072c0 0x7f687410ee70 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.197+0000 7f687c281640 1 --2- 172.21.15.7:0/1586487314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6874069f10 0x7f6874106be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.197+0000 7f687c281640 1 --2- 172.21.15.7:0/1586487314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6874069560 0x7f6874069940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.197+0000 7f687c281640 1 -- 172.21.15.7:0/1586487314 >> 172.21.15.7:0/1586487314 conn(0x7f68740771d0 msgr2=0x7f68740775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.197+0000 7f687c281640 1 -- 172.21.15.7:0/1586487314 shutdown_connections 2024-04-21T08:44:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.197+0000 7f687c281640 1 -- 172.21.15.7:0/1586487314 wait complete. 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687c281640 1 Processor -- start 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687c281640 1 -- start start 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687c281640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6874069560 0x7f6874198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687c281640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68741072c0 0x7f6874198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687c281640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f687419cf20 0x7f687419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687c281640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f68741121c0 con 0x7f68741072c0 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687c281640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6874112040 con 0x7f6874069560 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687c281640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6874112340 con 0x7f687419cf20 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687981c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68741072c0 0x7f6874198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687981c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68741072c0 0x7f6874198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47184/0 (socket says 172.21.15.7:47184) 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687981c640 1 -- 172.21.15.7:0/1164134150 learned_addr learned my addr 172.21.15.7:0/1164134150 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687a01d640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6874069560 0x7f6874198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687a81e640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f687419cf20 0x7f687419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687981c640 1 -- 172.21.15.7:0/1164134150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f687419cf20 msgr2=0x7f687419d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687981c640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f687419cf20 0x7f687419d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687981c640 1 -- 172.21.15.7:0/1164134150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6874069560 msgr2=0x7f6874198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.198+0000 7f687981c640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6874069560 0x7f6874198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.199+0000 7f687981c640 1 -- 172.21.15.7:0/1164134150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687419da10 con 0x7f68741072c0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.199+0000 7f687981c640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68741072c0 0x7f6874198ba0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f686400e9c0 tx=0x7f686400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.199+0000 7f686b7fe640 1 -- 172.21.15.7:0/1164134150 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686400ccc0 con 0x7f68741072c0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.199+0000 7f686b7fe640 1 -- 172.21.15.7:0/1164134150 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68640054d0 con 0x7f68741072c0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.199+0000 7f687c281640 1 -- 172.21.15.7:0/1164134150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f687419dd00 con 0x7f68741072c0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.199+0000 7f686b7fe640 1 -- 172.21.15.7:0/1164134150 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864010430 con 0x7f68741072c0 2024-04-21T08:44:46.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.199+0000 7f687c281640 1 -- 172.21.15.7:0/1164134150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68741a54a0 con 0x7f68741072c0 2024-04-21T08:44:46.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.200+0000 7f687c281640 1 -- 172.21.15.7:0/1164134150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6874068e00 con 0x7f68741072c0 2024-04-21T08:44:46.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.201+0000 7f686b7fe640 1 -- 172.21.15.7:0/1164134150 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f6864005020 con 0x7f68741072c0 2024-04-21T08:44:46.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.202+0000 7f686b7fe640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f68600a3db0 0x7f68600a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.202+0000 7f686b7fe640 1 -- 172.21.15.7:0/1164134150 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f68640d1150 con 0x7f68741072c0 2024-04-21T08:44:46.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.202+0000 7f687a01d640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f68600a3db0 0x7f68600a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.202+0000 7f687a01d640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f68600a3db0 0x7f68600a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6870009bd0 tx=0x7f6870005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.206+0000 7f686b7fe640 1 -- 172.21.15.7:0/1164134150 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f686409d290 con 0x7f68741072c0 2024-04-21T08:44:46.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:46.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:46.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: Unable to set osd_memory_target on smithi016 to 185893478: error parsing value: Value '185893478' is below minimum 939524096 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:46.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi007 to 177.2M 2024-04-21T08:44:46.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: Unable to set osd_memory_target on smithi007 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-21T08:44:46.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:46.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:46.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-21T08:44:46.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-21T08:44:46.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-21T08:44:46.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/745460848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:46.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2676965105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:46.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:45 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2817285720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:46.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.439+0000 7f687c281640 1 -- 172.21.15.7:0/1164134150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6874069f10 con 0x7f68741072c0 2024-04-21T08:44:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.440+0000 7f686b7fe640 1 -- 172.21.15.7:0/1164134150 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7f686409cb90 con 0x7f68741072c0 2024-04-21T08:44:46.439 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:46.439 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:44:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:44:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 -- 172.21.15.7:0/1164134150 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f68600a3db0 msgr2=0x7f68600a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f68600a3db0 0x7f68600a6270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6870009bd0 tx=0x7f6870005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 -- 172.21.15.7:0/1164134150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68741072c0 msgr2=0x7f6874198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68741072c0 0x7f6874198ba0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f686400e9c0 tx=0x7f686400ee80 comp rx=0 tx=0).stop 2024-04-21T08:44:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 -- 172.21.15.7:0/1164134150 shutdown_connections 2024-04-21T08:44:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f68600a3db0 0x7f68600a6270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f687419cf20 0x7f687419d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68741072c0 0x7f6874198ba0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 --2- 172.21.15.7:0/1164134150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6874069560 0x7f6874198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 -- 172.21.15.7:0/1164134150 >> 172.21.15.7:0/1164134150 conn(0x7f68740771d0 msgr2=0x7f68740fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 -- 172.21.15.7:0/1164134150 shutdown_connections 2024-04-21T08:44:46.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.442+0000 7f687c281640 1 -- 172.21.15.7:0/1164134150 wait complete. 2024-04-21T08:44:46.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:46.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.556+0000 7f7580663640 1 -- 172.21.15.7:0/995442832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75781072c0 msgr2=0x7f757810ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.556+0000 7f7580663640 1 --2- 172.21.15.7:0/995442832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75781072c0 0x7f757810ee70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7574009ca0 tx=0x7f757402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.556+0000 7f7580663640 1 -- 172.21.15.7:0/995442832 shutdown_connections 2024-04-21T08:44:46.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.556+0000 7f7580663640 1 --2- 172.21.15.7:0/995442832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75781072c0 0x7f757810ee70 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.556+0000 7f7580663640 1 --2- 172.21.15.7:0/995442832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7578069f10 0x7f7578106be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.556+0000 7f7580663640 1 --2- 172.21.15.7:0/995442832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7578069560 0x7f7578069940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.556+0000 7f7580663640 1 -- 172.21.15.7:0/995442832 >> 172.21.15.7:0/995442832 conn(0x7f75780771d0 msgr2=0x7f75780775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.556+0000 7f7580663640 1 -- 172.21.15.7:0/995442832 shutdown_connections 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.556+0000 7f7580663640 1 -- 172.21.15.7:0/995442832 wait complete. 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f7580663640 1 Processor -- start 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f7580663640 1 -- start start 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f7580663640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7578069560 0x7f7578198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f7580663640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75781072c0 0x7f7578198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f7580663640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f757819cf20 0x7f757819d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f7580663640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f75781121c0 con 0x7f757819cf20 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f7580663640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7578112040 con 0x7f75781072c0 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f7580663640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7578112340 con 0x7f7578069560 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f757ec00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f757819cf20 0x7f757819d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f757ec00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f757819cf20 0x7f757819d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47204/0 (socket says 172.21.15.7:47204) 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f757ec00640 1 -- 172.21.15.7:0/2767860952 learned_addr learned my addr 172.21.15.7:0/2767860952 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f757e3ff640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7578069560 0x7f7578198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.557+0000 7f757dbfe640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75781072c0 0x7f7578198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.558+0000 7f757ec00640 1 -- 172.21.15.7:0/2767860952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7578069560 msgr2=0x7f7578198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.558+0000 7f757ec00640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7578069560 0x7f7578198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.558+0000 7f757ec00640 1 -- 172.21.15.7:0/2767860952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75781072c0 msgr2=0x7f7578198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.558+0000 7f757ec00640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75781072c0 0x7f7578198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.558+0000 7f757ec00640 1 -- 172.21.15.7:0/2767860952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757819da10 con 0x7f757819cf20 2024-04-21T08:44:46.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.558+0000 7f757ec00640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f757819cf20 0x7f757819d3d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7574005e30 tx=0x7f7574034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.558+0000 7f756f7fe640 1 -- 172.21.15.7:0/2767860952 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7574041020 con 0x7f757819cf20 2024-04-21T08:44:46.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.558+0000 7f756f7fe640 1 -- 172.21.15.7:0/2767860952 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7574032970 con 0x7f757819cf20 2024-04-21T08:44:46.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.558+0000 7f7580663640 1 -- 172.21.15.7:0/2767860952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f757819dca0 con 0x7f757819cf20 2024-04-21T08:44:46.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.558+0000 7f756f7fe640 1 -- 172.21.15.7:0/2767860952 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7574039040 con 0x7f757819cf20 2024-04-21T08:44:46.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.558+0000 7f7580663640 1 -- 172.21.15.7:0/2767860952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75781a5460 con 0x7f757819cf20 2024-04-21T08:44:46.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.559+0000 7f7580663640 1 -- 172.21.15.7:0/2767860952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7578068e00 con 0x7f757819cf20 2024-04-21T08:44:46.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.560+0000 7f756f7fe640 1 -- 172.21.15.7:0/2767860952 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f7574044020 con 0x7f757819cf20 2024-04-21T08:44:46.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.561+0000 7f756f7fe640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75640a3db0 0x7f75640a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.561+0000 7f756f7fe640 1 -- 172.21.15.7:0/2767860952 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f75740f2f20 con 0x7f757819cf20 2024-04-21T08:44:46.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.561+0000 7f757e3ff640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75640a3db0 0x7f75640a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.561+0000 7f757e3ff640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75640a3db0 0x7f75640a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f75700095f0 tx=0x7f7570005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.565+0000 7f756f7fe640 1 -- 172.21.15.7:0/2767860952 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f757404b020 con 0x7f757819cf20 2024-04-21T08:44:46.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.753+0000 7f7580663640 1 -- 172.21.15.7:0/2767860952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7578069f10 con 0x7f757819cf20 2024-04-21T08:44:46.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.754+0000 7f756f7fe640 1 -- 172.21.15.7:0/2767860952 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25085 (secure 0 0 0) 0x7f75740bf110 con 0x7f757819cf20 2024-04-21T08:44:46.754 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:46.754 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":95,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:44:41.244735+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3451243741":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/117789107":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/4167672076":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/3630897301":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 -- 172.21.15.7:0/2767860952 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75640a3db0 msgr2=0x7f75640a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75640a3db0 0x7f75640a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f75700095f0 tx=0x7f7570005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 -- 172.21.15.7:0/2767860952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f757819cf20 msgr2=0x7f757819d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f757819cf20 0x7f757819d3d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7574005e30 tx=0x7f7574034670 comp rx=0 tx=0).stop 2024-04-21T08:44:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 -- 172.21.15.7:0/2767860952 shutdown_connections 2024-04-21T08:44:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f757819cf20 0x7f757819d3d0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75781072c0 0x7f7578198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75640a3db0 0x7f75640a6270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 --2- 172.21.15.7:0/2767860952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7578069560 0x7f7578198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 -- 172.21.15.7:0/2767860952 >> 172.21.15.7:0/2767860952 conn(0x7f75780771d0 msgr2=0x7f75780fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 -- 172.21.15.7:0/2767860952 shutdown_connections 2024-04-21T08:44:46.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.756+0000 7f7580663640 1 -- 172.21.15.7:0/2767860952 wait complete. 2024-04-21T08:44:46.767 DEBUG:tasks.fwd_scrub:Yielding 2024-04-21T08:44:46.767 INFO:teuthology.run_tasks:Running task exec... 2024-04-21T08:44:46.777 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:44:46.777 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-21T08:44:46.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-21T08:44:46.777 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-21T08:44:46.778 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:44:46.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:44:46.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:44:46.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.885+0000 7f36ac150640 1 -- 172.21.15.7:0/3195691943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a4100d90 msgr2=0x7f36a4101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.885+0000 7f36ac150640 1 --2- 172.21.15.7:0/3195691943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a4100d90 0x7f36a4101170 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009900 tx=0x7f36a002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:46.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.885+0000 7f36ac150640 1 -- 172.21.15.7:0/3195691943 shutdown_connections 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.885+0000 7f36ac150640 1 --2- 172.21.15.7:0/3195691943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36a410e940 0x7f36a4110d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.885+0000 7f36ac150640 1 --2- 172.21.15.7:0/3195691943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a4101740 0x7f36a410e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.885+0000 7f36ac150640 1 --2- 172.21.15.7:0/3195691943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a4100d90 0x7f36a4101170 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.885+0000 7f36ac150640 1 -- 172.21.15.7:0/3195691943 >> 172.21.15.7:0/3195691943 conn(0x7f36a40fc970 msgr2=0x7f36a40fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.885+0000 7f36ac150640 1 -- 172.21.15.7:0/3195691943 shutdown_connections 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.885+0000 7f36ac150640 1 -- 172.21.15.7:0/3195691943 wait complete. 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.885+0000 7f36ac150640 1 Processor -- start 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.885+0000 7f36ac150640 1 -- start start 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36ac150640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a4100d90 0x7f36a41a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36ac150640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a4101740 0x7f36a41a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36ac150640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36a410e940 0x7f36a419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36ac150640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f36a4114480 con 0x7f36a4100d90 2024-04-21T08:44:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36ac150640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36a4114300 con 0x7f36a410e940 2024-04-21T08:44:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36ac150640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f36a4114600 con 0x7f36a4101740 2024-04-21T08:44:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36a9eec640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a4100d90 0x7f36a41a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36a9eec640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a4100d90 0x7f36a41a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47222/0 (socket says 172.21.15.7:47222) 2024-04-21T08:44:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36a9eec640 1 -- 172.21.15.7:0/4160426380 learned_addr learned my addr 172.21.15.7:0/4160426380 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36a96eb640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a4101740 0x7f36a41a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f420057e640 1 -- 172.21.15.7:0/352585515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f81021b0 msgr2=0x7f41f8102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f420057e640 1 --2- 172.21.15.7:0/352585515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f81021b0 0x7f41f8102590 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009900 tx=0x7f41f402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:46.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36aa6ed640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36a410e940 0x7f36a419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f420057e640 1 -- 172.21.15.7:0/352585515 shutdown_connections 2024-04-21T08:44:46.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f420057e640 1 --2- 172.21.15.7:0/352585515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8107280 0x7f41f810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f420057e640 1 --2- 172.21.15.7:0/352585515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41f8102b60 0x7f41f8106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f420057e640 1 --2- 172.21.15.7:0/352585515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f81021b0 0x7f41f8102590 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f420057e640 1 -- 172.21.15.7:0/352585515 >> 172.21.15.7:0/352585515 conn(0x7f41f80fda30 msgr2=0x7f41f80ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f420057e640 1 -- 172.21.15.7:0/352585515 shutdown_connections 2024-04-21T08:44:46.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f420057e640 1 -- 172.21.15.7:0/352585515 wait complete. 2024-04-21T08:44:46.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36a9eec640 1 -- 172.21.15.7:0/4160426380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a4101740 msgr2=0x7f36a41a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36a9eec640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a4101740 0x7f36a41a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36a9eec640 1 -- 172.21.15.7:0/4160426380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36a410e940 msgr2=0x7f36a419a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36a9eec640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36a410e940 0x7f36a419a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36a9eec640 1 -- 172.21.15.7:0/4160426380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a419b040 con 0x7f36a4100d90 2024-04-21T08:44:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36a96eb640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a4101740 0x7f36a41a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36a9eec640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a4100d90 0x7f36a41a06c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f36a0005940 tx=0x7f36a00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f369affd640 1 -- 172.21.15.7:0/4160426380 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0043020 con 0x7f36a4100d90 2024-04-21T08:44:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f36ac150640 1 -- 172.21.15.7:0/4160426380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a419b2d0 con 0x7f36a4100d90 2024-04-21T08:44:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f369affd640 1 -- 172.21.15.7:0/4160426380 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36a002caf0 con 0x7f36a4100d90 2024-04-21T08:44:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.886+0000 7f369affd640 1 -- 172.21.15.7:0/4160426380 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0003af0 con 0x7f36a4100d90 2024-04-21T08:44:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f36ac150640 1 -- 172.21.15.7:0/4160426380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a41a7580 con 0x7f36a4100d90 2024-04-21T08:44:46.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f420057e640 1 Processor -- start 2024-04-21T08:44:46.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f420057e640 1 -- start start 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f420057e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f81021b0 0x7f41f8074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f420057e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41f8107280 0x7f41f8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f420057e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8074cd0 0x7f41f81a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f420057e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f41f8112210 con 0x7f41f81021b0 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f420057e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41f8112090 con 0x7f41f8074cd0 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f420057e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41f8112390 con 0x7f41f8107280 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f41fe31a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f81021b0 0x7f41f8074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f41fe31a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f81021b0 0x7f41f8074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47232/0 (socket says 172.21.15.7:47232) 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f41fe31a640 1 -- 172.21.15.7:0/1852051796 learned_addr learned my addr 172.21.15.7:0/1852051796 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f41feb1b640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8074cd0 0x7f41f81a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f41fdb19640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41f8107280 0x7f41f8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f41fe31a640 1 -- 172.21.15.7:0/1852051796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41f8107280 msgr2=0x7f41f8074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f41fe31a640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41f8107280 0x7f41f8074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f41fe31a640 1 -- 172.21.15.7:0/1852051796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8074cd0 msgr2=0x7f41f81a69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f41fe31a640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8074cd0 0x7f41f81a69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f41fe31a640 1 -- 172.21.15.7:0/1852051796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f81a70a0 con 0x7f41f81021b0 2024-04-21T08:44:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.887+0000 7f41feb1b640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8074cd0 0x7f41f81a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.888+0000 7f41fe31a640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f81021b0 0x7f41f8074250 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f41f4000c00 tx=0x7f41f402c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.888+0000 7f36ac150640 1 -- 172.21.15.7:0/4160426380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a406ade0 con 0x7f36a4100d90 2024-04-21T08:44:46.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.888+0000 7f41ef7fe640 1 -- 172.21.15.7:0/1852051796 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4043020 con 0x7f41f81021b0 2024-04-21T08:44:46.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.888+0000 7f41ef7fe640 1 -- 172.21.15.7:0/1852051796 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41f402cb30 con 0x7f41f81021b0 2024-04-21T08:44:46.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.888+0000 7f41ef7fe640 1 -- 172.21.15.7:0/1852051796 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4003a50 con 0x7f41f81021b0 2024-04-21T08:44:46.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.888+0000 7f420057e640 1 -- 172.21.15.7:0/1852051796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f81a7330 con 0x7f41f81021b0 2024-04-21T08:44:46.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.888+0000 7f420057e640 1 -- 172.21.15.7:0/1852051796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f81a78d0 con 0x7f41f81021b0 2024-04-21T08:44:46.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.889+0000 7f369affd640 1 -- 172.21.15.7:0/4160426380 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f36a0046020 con 0x7f36a4100d90 2024-04-21T08:44:46.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.890+0000 7f369affd640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36900a3da0 0x7f36900a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.890+0000 7f369affd640 1 -- 172.21.15.7:0/4160426380 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f36a00f2dc0 con 0x7f36a4100d90 2024-04-21T08:44:46.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.890+0000 7f36a96eb640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36900a3da0 0x7f36900a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.890+0000 7f41ef7fe640 1 -- 172.21.15.7:0/1852051796 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f41f4046020 con 0x7f41f81021b0 2024-04-21T08:44:46.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.890+0000 7f36a96eb640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36900a3da0 0x7f36900a6260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f36a4100090 tx=0x7f3694005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f420316c740 1 Processor -- start 2024-04-21T08:44:46.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f420316c740 1 -- start start 2024-04-21T08:44:46.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f420316c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2b98e0 0x55563a2b9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f420316c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2c2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f420316c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2c2e10 0x55563a2c5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f420316c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55563a21df00 con 0x55563a2ba290 2024-04-21T08:44:46.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f420316c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55563a1bb590 con 0x55563a2c2e10 2024-04-21T08:44:46.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f420316c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55563a219ec0 con 0x55563a2b98e0 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f41ef7fe640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41e40a3e60 0x7f41e40a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f41fe31a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f41feb1b640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41e40a3e60 0x7f41e40a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f41fe31a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2c2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47242/0 (socket says 172.21.15.7:47242) 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f41fe31a640 1 -- 172.21.15.7:0/3279019725 learned_addr learned my addr 172.21.15.7:0/3279019725 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.891+0000 7f41feb1b640 1 --2- 172.21.15.7:0/3279019725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2b98e0 0x55563a2b9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.892+0000 7f41fdb19640 1 --2- 172.21.15.7:0/3279019725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2c2e10 0x55563a2c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.892+0000 7f41ef7fe640 1 -- 172.21.15.7:0/1852051796 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f41f40f2d70 con 0x7f41f81021b0 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.892+0000 7f41fe31a640 1 -- 172.21.15.7:0/3279019725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2b98e0 msgr2=0x55563a2b9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.892+0000 7f41fe31a640 1 --2- 172.21.15.7:0/3279019725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2b98e0 0x55563a2b9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.892+0000 7f41fe31a640 1 -- 172.21.15.7:0/3279019725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2c2e10 msgr2=0x55563a2c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.892+0000 7f41fe31a640 1 --2- 172.21.15.7:0/3279019725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2c2e10 0x55563a2c5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.892+0000 7f41fe31a640 1 -- 172.21.15.7:0/3279019725 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555639ff87e0 con 0x55563a2ba290 2024-04-21T08:44:46.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.892+0000 7f41fdb19640 1 --2- 172.21.15.7:0/3279019725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2c2e10 0x55563a2c5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:46.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.892+0000 7f41feb1b640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41e40a3e60 0x7f41e40a6320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f41f0005fd0 tx=0x7f41f0009bb0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.892+0000 7f41fe31a640 1 --2- 172.21.15.7:0/3279019725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2c2840 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f41f40f06d0 tx=0x7f41f4091440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe38a69a79cb031 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.895+0000 7f41ecbfe640 1 -- 172.21.15.7:0/3279019725 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f40c3e90 con 0x55563a2ba290 2024-04-21T08:44:46.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.895+0000 7f41ecbfe640 1 -- 172.21.15.7:0/3279019725 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41f40bf050 con 0x55563a2ba290 2024-04-21T08:44:46.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.895+0000 7f41ecbfe640 1 -- 172.21.15.7:0/3279019725 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f40c7530 con 0x55563a2ba290 2024-04-21T08:44:46.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.895+0000 7f369affd640 1 -- 172.21.15.7:0/4160426380 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36a004d020 con 0x7f36a4100d90 2024-04-21T08:44:46.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.896+0000 7f420316c740 1 -- 172.21.15.7:0/3279019725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 msgr2=0x55563a2c2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.896+0000 7f420316c740 1 --2- 172.21.15.7:0/3279019725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2c2840 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f41f40f06d0 tx=0x7f41f4091440 comp rx=0 tx=0).stop 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.896+0000 7f420316c740 1 -- 172.21.15.7:0/3279019725 shutdown_connections 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.896+0000 7f420316c740 1 --2- 172.21.15.7:0/3279019725 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2c2e10 0x55563a2c5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.896+0000 7f420316c740 1 --2- 172.21.15.7:0/3279019725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2c2840 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.896+0000 7f420316c740 1 --2- 172.21.15.7:0/3279019725 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2b98e0 0x55563a2b9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.896+0000 7f420316c740 1 -- 172.21.15.7:0/3279019725 >> 172.21.15.7:0/3279019725 conn(0x55563a2b3fb0 msgr2=0x55563a2b4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.896+0000 7f420316c740 1 -- 172.21.15.7:0/3279019725 shutdown_connections 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.896+0000 7f420316c740 1 -- 172.21.15.7:0/3279019725 wait complete. 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.896+0000 7f420316c740 1 -- start start 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.896+0000 7f420316c740 1 Processor -- start 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f420316c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f420316c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2ba290 0x55563a2c0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f420316c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2c2e10 0x55563a2c0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f420316c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55563a2c5b40 con 0x55563a2b98e0 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f420316c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55563a1d7580 con 0x55563a2ba290 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f420316c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55563a2c5cc0 con 0x55563a2c2e10 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f420316c740 10 client.0 _ll_register_callbacks cb 0x55563a1dd650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41fe31a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41fe31a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2bfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47252/0 (socket says 172.21.15.7:47252) 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41fe31a640 1 -- 172.21.15.7:0/1653719094 learned_addr learned my addr 172.21.15.7:0/1653719094 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41fdb19640 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2ba290 0x55563a2c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41feb1b640 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2c2e10 0x55563a2c0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41fe31a640 1 -- 172.21.15.7:0/1653719094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2c2e10 msgr2=0x55563a2c0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41fe31a640 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2c2e10 0x55563a2c0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41fe31a640 1 -- 172.21.15.7:0/1653719094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2ba290 msgr2=0x55563a2c0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41fe31a640 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2ba290 0x55563a2c0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41fe31a640 1 -- 172.21.15.7:0/1653719094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55563a058f70 con 0x55563a2b98e0 2024-04-21T08:44:46.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41fe31a640 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2bfe00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f41f40bb320 tx=0x7f41f4005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed1facb1408a4387 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.897+0000 7f41e1ffb640 1 -- 172.21.15.7:0/1653719094 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f40be720 con 0x55563a2b98e0 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f41e1ffb640 1 -- 172.21.15.7:0/1653719094 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f41f40bebb0 con 0x55563a2b98e0 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f420316c740 1 -- 172.21.15.7:0/1653719094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41f40bf050 con 0x55563a2b98e0 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f41e1ffb640 1 -- 172.21.15.7:0/1653719094 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f41f40f04d0 con 0x55563a2b98e0 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f420316c740 10 client.15612 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f420316c740 1 -- 172.21.15.7:0/1653719094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555639ef0170 con 0x55563a2b98e0 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f41e1ffb640 1 -- 172.21.15.7:0/1653719094 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41f40bf050 con 0x55563a2b98e0 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f41e1ffb640 1 -- 172.21.15.7:0/1653719094 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f41f40a90b0 con 0x55563a2b98e0 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f420316c740 10 client.15612 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f420316c740 10 client.15612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f420316c740 20 client.15612 populate_metadata read hostname 'smithi007' 2024-04-21T08:44:46.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f420316c740 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55563a2d2f20 0x55563a2d5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f420316c740 4 client.15612 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f420316c740 1 -- 172.21.15.7:0/1653719094 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55563a22b920 con 0x55563a2d2f20 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.898+0000 7f41fdb19640 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55563a2d2f20 0x55563a2d5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.899+0000 7f41e1ffb640 10 client.15612 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.899+0000 7f41fdb19640 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55563a2d2f20 0x55563a2d5310 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb8a346e0ecad311 server_cookie=16abb7f36cc51fcb in_seq=0 out_seq=0 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.899+0000 7f41e1ffb640 1 -- 172.21.15.7:0/1653719094 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55563a22b920 con 0x55563a2d2f20 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.899+0000 7f41e1ffb640 10 client.15612 handle_command_reply: tid=0 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.899+0000 7f420316c740 1 client.15612 shutdown 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f420316c740 1 -- 172.21.15.7:0/1653719094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 msgr2=0x55563a2bfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f420316c740 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2bfe00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f41f40bb320 tx=0x7f41f4005640 comp rx=0 tx=0).stop 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f420316c740 1 -- 172.21.15.7:0/1653719094 shutdown_connections 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f420316c740 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55563a2d2f20 0x55563a2d5310 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f420316c740 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2c2e10 0x55563a2c0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f420316c740 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2ba290 0x55563a2c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f420316c740 1 --2- 172.21.15.7:0/1653719094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2bfe00 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f420316c740 1 -- 172.21.15.7:0/1653719094 >> 172.21.15.7:0/1653719094 conn(0x55563a2b3fb0 msgr2=0x55563a2b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f41e1ffb640 0 client.15612 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f420316c740 1 -- 172.21.15.7:0/1653719094 shutdown_connections 2024-04-21T08:44:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f420316c740 1 -- 172.21.15.7:0/1653719094 wait complete. 2024-04-21T08:44:46.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.900+0000 7f420316c740 20 client.15612 trim_cache size 0 max 16384 2024-04-21T08:44:46.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.917+0000 7f420316c740 1 Processor -- start 2024-04-21T08:44:46.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.917+0000 7f420316c740 1 -- start start 2024-04-21T08:44:46.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f420316c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2c2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f420316c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2ba290 0x55563a2ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f420316c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2c2e10 0x55563a2b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f420316c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f41f81af640 con 0x55563a2b98e0 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f420316c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41f81b1ec0 con 0x55563a2ba290 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f420316c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41f81afcb0 con 0x55563a2c2e10 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41fe31a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41fe31a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2c2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47258/0 (socket says 172.21.15.7:47258) 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41fe31a640 1 -- 172.21.15.7:0/137271091 learned_addr learned my addr 172.21.15.7:0/137271091 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41fdb19640 1 --2- 172.21.15.7:0/137271091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2ba290 0x55563a2ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41feb1b640 1 --2- 172.21.15.7:0/137271091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2c2e10 0x55563a2b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41fe31a640 1 -- 172.21.15.7:0/137271091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2c2e10 msgr2=0x55563a2b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41fe31a640 1 --2- 172.21.15.7:0/137271091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2c2e10 0x55563a2b5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41fe31a640 1 -- 172.21.15.7:0/137271091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2ba290 msgr2=0x55563a2ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41fe31a640 1 --2- 172.21.15.7:0/137271091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2ba290 0x55563a2ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41fe31a640 1 -- 172.21.15.7:0/137271091 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55563a2b4930 con 0x55563a2b98e0 2024-04-21T08:44:46.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41fdb19640 1 --2- 172.21.15.7:0/137271091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2ba290 0x55563a2ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41fe31a640 1 --2- 172.21.15.7:0/137271091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2c2530 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f41f40be650 tx=0x7f41f40bad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7004f8216c6a9955 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.918+0000 7f41e3fff640 1 -- 172.21.15.7:0/137271091 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f40c3e90 con 0x55563a2b98e0 2024-04-21T08:44:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.919+0000 7f41e3fff640 1 -- 172.21.15.7:0/137271091 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41f40f04d0 con 0x55563a2b98e0 2024-04-21T08:44:46.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.919+0000 7f41e3fff640 1 -- 172.21.15.7:0/137271091 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f40bebb0 con 0x55563a2b98e0 2024-04-21T08:44:46.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.919+0000 7f420316c740 1 -- 172.21.15.7:0/137271091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 msgr2=0x55563a2c2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.919+0000 7f420316c740 1 --2- 172.21.15.7:0/137271091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2c2530 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f41f40be650 tx=0x7f41f40bad00 comp rx=0 tx=0).stop 2024-04-21T08:44:46.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.919+0000 7f420316c740 1 -- 172.21.15.7:0/137271091 shutdown_connections 2024-04-21T08:44:46.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.919+0000 7f420316c740 1 --2- 172.21.15.7:0/137271091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2c2e10 0x55563a2b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.919+0000 7f420316c740 1 --2- 172.21.15.7:0/137271091 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2ba290 0x55563a2ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 --2- 172.21.15.7:0/137271091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2b98e0 0x55563a2c2530 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 -- 172.21.15.7:0/137271091 >> 172.21.15.7:0/137271091 conn(0x55563a2b3fb0 msgr2=0x55563a2b5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 -- 172.21.15.7:0/137271091 shutdown_connections 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 -- 172.21.15.7:0/137271091 wait complete. 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 -- start start 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 Processor -- start 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2b98e0 0x55563a2bcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2bd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2c2e10 0x55563a2bd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f41f81b0380 con 0x55563a2ba290 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41f81b17f0 con 0x55563a2c2e10 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41f81b0a50 con 0x55563a2b98e0 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.920+0000 7f420316c740 10 client.0 _ll_register_callbacks cb 0x7f41f402cce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41fdb19640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2bd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41fdb19640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2bd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47272/0 (socket says 172.21.15.7:47272) 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41fdb19640 1 -- 172.21.15.7:0/2453866927 learned_addr learned my addr 172.21.15.7:0/2453866927 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41fe31a640 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2b98e0 0x55563a2bcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41fdb19640 1 -- 172.21.15.7:0/2453866927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2b98e0 msgr2=0x55563a2bcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41fdb19640 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2b98e0 0x55563a2bcb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41fdb19640 1 -- 172.21.15.7:0/2453866927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2c2e10 msgr2=0x55563a2bd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41fdb19640 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2c2e10 0x55563a2bd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41fdb19640 1 -- 172.21.15.7:0/2453866927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55563a338370 con 0x55563a2ba290 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41fdb19640 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2bd040 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f41f8075ce0 tx=0x7f41e8012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe609aa0f74915a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41e1ffb640 1 -- 172.21.15.7:0/2453866927 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e8022020 con 0x55563a2ba290 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41e1ffb640 1 -- 172.21.15.7:0/2453866927 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f41e8004460 con 0x55563a2ba290 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f420316c740 1 -- 172.21.15.7:0/2453866927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41f40f04d0 con 0x55563a2ba290 2024-04-21T08:44:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f41e1ffb640 1 -- 172.21.15.7:0/2453866927 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f41e802d050 con 0x55563a2ba290 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f420316c740 10 client.15618 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.921+0000 7f420316c740 1 -- 172.21.15.7:0/2453866927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55563a339700 con 0x55563a2ba290 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.922+0000 7f41e1ffb640 1 -- 172.21.15.7:0/2453866927 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41f40f04d0 con 0x55563a2ba290 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.922+0000 7f41e1ffb640 1 -- 172.21.15.7:0/2453866927 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f41e8003a80 con 0x55563a2ba290 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.922+0000 7f420316c740 10 client.15618 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.922+0000 7f420316c740 10 client.15618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.922+0000 7f420316c740 20 client.15618 populate_metadata read hostname 'smithi007' 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.922+0000 7f420316c740 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55563a339ee0 0x55563a33c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.922+0000 7f420316c740 4 client.15618 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.922+0000 7f420316c740 1 -- 172.21.15.7:0/2453866927 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55563a225790 con 0x55563a339ee0 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.922+0000 7f41fe31a640 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55563a339ee0 0x55563a33c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.922+0000 7f41fe31a640 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55563a339ee0 0x55563a33c2d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=414efefccd523413 server_cookie=27b42fe82e9a3d16 in_seq=0 out_seq=0 2024-04-21T08:44:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.922+0000 7f41e1ffb640 10 client.15618 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.923+0000 7f41e1ffb640 1 -- 172.21.15.7:0/2453866927 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55563a225790 con 0x55563a339ee0 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.923+0000 7f41e1ffb640 10 client.15618 handle_command_reply: tid=0 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 1 client.15618 shutdown 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 1 -- 172.21.15.7:0/2453866927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 msgr2=0x55563a2bd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2bd040 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f41f8075ce0 tx=0x7f41e8012820 comp rx=0 tx=0).stop 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 1 -- 172.21.15.7:0/2453866927 shutdown_connections 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55563a339ee0 0x55563a33c2d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55563a2c2e10 0x55563a2bd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55563a2ba290 0x55563a2bd040 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 1 --2- 172.21.15.7:0/2453866927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55563a2b98e0 0x55563a2bcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f41e1ffb640 0 client.15618 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 1 -- 172.21.15.7:0/2453866927 >> 172.21.15.7:0/2453866927 conn(0x55563a2b3fb0 msgr2=0x55563a2b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 1 -- 172.21.15.7:0/2453866927 shutdown_connections 2024-04-21T08:44:46.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 1 -- 172.21.15.7:0/2453866927 wait complete. 2024-04-21T08:44:46.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420316c740 20 client.15618 trim_cache size 0 max 16384 2024-04-21T08:44:46.929 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:44:46.929 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:44:46.929 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "ed8efc86-501e-406f-92cc-770642ecc9f9", 2024-04-21T08:44:46.929 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:44:46.929 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:44:46.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420057e640 1 -- 172.21.15.7:0/1852051796 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41e40a3e60 msgr2=0x7f41e40a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420057e640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41e40a3e60 0x7f41e40a6320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f41f0005fd0 tx=0x7f41f0009bb0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420057e640 1 -- 172.21.15.7:0/1852051796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f81021b0 msgr2=0x7f41f8074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.924+0000 7f420057e640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f81021b0 0x7f41f8074250 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f41f4000c00 tx=0x7f41f402c630 comp rx=0 tx=0).stop 2024-04-21T08:44:46.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.925+0000 7f420057e640 1 -- 172.21.15.7:0/1852051796 shutdown_connections 2024-04-21T08:44:46.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.925+0000 7f420057e640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41f8074cd0 0x7f41f81a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.925+0000 7f420057e640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41e40a3e60 0x7f41e40a6320 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.925+0000 7f420057e640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41f8107280 0x7f41f8074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.925+0000 7f420057e640 1 --2- 172.21.15.7:0/1852051796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f81021b0 0x7f41f8074250 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.925+0000 7f420057e640 1 -- 172.21.15.7:0/1852051796 >> 172.21.15.7:0/1852051796 conn(0x7f41f80fda30 msgr2=0x7f41f80feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.925+0000 7f420057e640 1 -- 172.21.15.7:0/1852051796 shutdown_connections 2024-04-21T08:44:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:46.925+0000 7f420057e640 1 -- 172.21.15.7:0/1852051796 wait complete. 2024-04-21T08:44:46.935 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:44:46.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:44:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.040+0000 7f4439da4640 1 -- 172.21.15.7:0/2732456918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4434106120 msgr2=0x7f443410dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.040+0000 7f4439da4640 1 --2- 172.21.15.7:0/2732456918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4434106120 0x7f443410dcf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f442c009ca0 tx=0x7f442c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.040+0000 7f4439da4640 1 -- 172.21.15.7:0/2732456918 shutdown_connections 2024-04-21T08:44:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.040+0000 7f4439da4640 1 --2- 172.21.15.7:0/2732456918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4434106120 0x7f443410dcf0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.040+0000 7f4439da4640 1 --2- 172.21.15.7:0/2732456918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4434101a00 0x7f4434105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.040+0000 7f4439da4640 1 --2- 172.21.15.7:0/2732456918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4434101050 0x7f4434101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.041+0000 7f4439da4640 1 -- 172.21.15.7:0/2732456918 >> 172.21.15.7:0/2732456918 conn(0x7f4434077fa0 msgr2=0x7f44340783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.041+0000 7f4439da4640 1 -- 172.21.15.7:0/2732456918 shutdown_connections 2024-04-21T08:44:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.041+0000 7f4439da4640 1 -- 172.21.15.7:0/2732456918 wait complete. 2024-04-21T08:44:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.041+0000 7f4439da4640 1 Processor -- start 2024-04-21T08:44:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.041+0000 7f4439da4640 1 -- start start 2024-04-21T08:44:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f4439da4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4434101050 0x7f4434102ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f4439da4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4434106120 0x7f4434103020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f4439da4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4434103560 0x7f44341a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f4439da4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4434111070 con 0x7f4434101050 2024-04-21T08:44:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f4439da4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4434110ef0 con 0x7f4434106120 2024-04-21T08:44:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f4439da4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f44341111f0 con 0x7f4434103560 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f44337fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4434101050 0x7f4434102ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f44337fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4434101050 0x7f4434102ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47294/0 (socket says 172.21.15.7:47294) 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f44337fe640 1 -- 172.21.15.7:0/855185643 learned_addr learned my addr 172.21.15.7:0/855185643 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f4433fff640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4434103560 0x7f44341a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f4432ffd640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4434106120 0x7f4434103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f44337fe640 1 -- 172.21.15.7:0/855185643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4434103560 msgr2=0x7f44341a6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f44337fe640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4434103560 0x7f44341a6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f44337fe640 1 -- 172.21.15.7:0/855185643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4434106120 msgr2=0x7f4434103020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f44337fe640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4434106120 0x7f4434103020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f44337fe640 1 -- 172.21.15.7:0/855185643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44341a6f70 con 0x7f4434101050 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f44337fe640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4434101050 0x7f4434102ae0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f441c00e9f0 tx=0x7f441c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.042+0000 7f4430ff9640 1 -- 172.21.15.7:0/855185643 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c00cd80 con 0x7f4434101050 2024-04-21T08:44:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.043+0000 7f4439da4640 1 -- 172.21.15.7:0/855185643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44341a7260 con 0x7f4434101050 2024-04-21T08:44:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.043+0000 7f4430ff9640 1 -- 172.21.15.7:0/855185643 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f441c0108e0 con 0x7f4434101050 2024-04-21T08:44:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.043+0000 7f4430ff9640 1 -- 172.21.15.7:0/855185643 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c018980 con 0x7f4434101050 2024-04-21T08:44:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.043+0000 7f4439da4640 1 -- 172.21.15.7:0/855185643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44341a77b0 con 0x7f4434101050 2024-04-21T08:44:47.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.045+0000 7f4430ff9640 1 -- 172.21.15.7:0/855185643 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f441c010430 con 0x7f4434101050 2024-04-21T08:44:47.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.045+0000 7f443c994740 1 Processor -- start 2024-04-21T08:44:47.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.045+0000 7f443c994740 1 -- start start 2024-04-21T08:44:47.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.045+0000 7f443c994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf5518960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.045+0000 7f4430ff9640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f44200a3de0 0x7f44200a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.045+0000 7f443c994740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5518ea0 0x55fdf5521540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.045+0000 7f443c994740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5521a80 0x55fdf5523e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.045+0000 7f443c994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55fdf547d1e0 con 0x55fdf5518580 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f443c994740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fdf541a390 con 0x55fdf5518ea0 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f443c994740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fdf54791a0 con 0x55fdf5521a80 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4433fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf5518960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4433fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf5518960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47298/0 (socket says 172.21.15.7:47298) 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4433fff640 1 -- 172.21.15.7:0/883817342 learned_addr learned my addr 172.21.15.7:0/883817342 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4430ff9640 1 -- 172.21.15.7:0/855185643 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f441c0d1130 con 0x7f4434101050 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f44337fe640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f44200a3de0 0x7f44200a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4433fff640 1 --2- 172.21.15.7:0/883817342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5521a80 0x55fdf5523e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4432ffd640 1 --2- 172.21.15.7:0/883817342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5518ea0 0x55fdf5521540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4433fff640 1 -- 172.21.15.7:0/883817342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5521a80 msgr2=0x55fdf5523e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4433fff640 1 --2- 172.21.15.7:0/883817342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5521a80 0x55fdf5523e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4433fff640 1 -- 172.21.15.7:0/883817342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5518ea0 msgr2=0x55fdf5521540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4433fff640 1 --2- 172.21.15.7:0/883817342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5518ea0 0x55fdf5521540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4433fff640 1 -- 172.21.15.7:0/883817342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdf52597d0 con 0x55fdf5518580 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f44337fe640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f44200a3de0 0x7f44200a62a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f441c0ce430 tx=0x7f441c009310 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4433fff640 1 --2- 172.21.15.7:0/883817342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf5518960 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f442c009890 tx=0x7f442c009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78fc96793a056ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.046+0000 7f4429ffb640 1 -- 172.21.15.7:0/883817342 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c039040 con 0x55fdf5518580 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.047+0000 7f4429ffb640 1 -- 172.21.15.7:0/883817342 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f442c032970 con 0x55fdf5518580 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.047+0000 7f4429ffb640 1 -- 172.21.15.7:0/883817342 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c004290 con 0x55fdf5518580 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.047+0000 7f443c994740 1 -- 172.21.15.7:0/883817342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 msgr2=0x55fdf5518960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.047+0000 7f443c994740 1 --2- 172.21.15.7:0/883817342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf5518960 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f442c009890 tx=0x7f442c009820 comp rx=0 tx=0).stop 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.047+0000 7f443c994740 1 -- 172.21.15.7:0/883817342 shutdown_connections 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.047+0000 7f443c994740 1 --2- 172.21.15.7:0/883817342 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5521a80 0x55fdf5523e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.047+0000 7f443c994740 1 --2- 172.21.15.7:0/883817342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5518ea0 0x55fdf5521540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.047+0000 7f443c994740 1 --2- 172.21.15.7:0/883817342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf5518960 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.047+0000 7f443c994740 1 -- 172.21.15.7:0/883817342 >> 172.21.15.7:0/883817342 conn(0x55fdf5512cb0 msgr2=0x55fdf5513090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.047+0000 7f443c994740 1 -- 172.21.15.7:0/883817342 shutdown_connections 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.047+0000 7f443c994740 1 -- 172.21.15.7:0/883817342 wait complete. 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f443c994740 1 -- start start 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f443c994740 1 Processor -- start 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f443c994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf551eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f443c994740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5518ea0 0x55fdf551f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f443c994740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5521a80 0x55fdf551f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f443c994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55fdf5524810 con 0x55fdf5518580 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f443c994740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fdf54362f0 con 0x55fdf5521a80 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f443c994740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fdf5524990 con 0x55fdf5518ea0 2024-04-21T08:44:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f443c994740 10 client.0 _ll_register_callbacks cb 0x55fdf5443580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f4433fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf551eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f4433fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf551eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47302/0 (socket says 172.21.15.7:47302) 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f4433fff640 1 -- 172.21.15.7:0/277858713 learned_addr learned my addr 172.21.15.7:0/277858713 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f4432ffd640 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5518ea0 0x55fdf551f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f4433fff640 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5521a80 0x55fdf551f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f4433fff640 1 -- 172.21.15.7:0/277858713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5518ea0 msgr2=0x55fdf551f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f4433fff640 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5518ea0 0x55fdf551f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f4433fff640 1 -- 172.21.15.7:0/277858713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5521a80 msgr2=0x55fdf551f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f4433fff640 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5521a80 0x55fdf551f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.048+0000 7f4433fff640 1 -- 172.21.15.7:0/277858713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdf518f3f0 con 0x55fdf5518580 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f4433fff640 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf551eac0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f442c02c630 tx=0x7f442c003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f5dd78ab67f99d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f4412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:44:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f4412ffd640 1 -- 172.21.15.7:0/277858713 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c039070 con 0x55fdf5518580 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f4412ffd640 1 -- 172.21.15.7:0/277858713 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f442c003970 con 0x55fdf5518580 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f443c994740 1 -- 172.21.15.7:0/277858713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f442c032970 con 0x55fdf5518580 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f4412ffd640 1 -- 172.21.15.7:0/277858713 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f442c05c050 con 0x55fdf5518580 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f443c994740 10 client.15630 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f443c994740 1 -- 172.21.15.7:0/277858713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fdf514f0f0 con 0x55fdf5518580 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f4412ffd640 1 -- 172.21.15.7:0/277858713 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f442c032970 con 0x55fdf5518580 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f4412ffd640 1 -- 172.21.15.7:0/277858713 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f442c04d070 con 0x55fdf5518580 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f443c994740 10 client.15630 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f443c994740 10 client.15630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f443c994740 20 client.15630 populate_metadata read hostname 'smithi007' 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f443c994740 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fdf55318a0 0x55fdf5533c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f443c994740 4 client.15630 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.049+0000 7f443c994740 1 -- 172.21.15.7:0/277858713 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdf548ac00 con 0x55fdf55318a0 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.050+0000 7f4432ffd640 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fdf55318a0 0x55fdf5533c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.050+0000 7f4432ffd640 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fdf55318a0 0x55fdf5533c90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=382eb67e24e438ff server_cookie=757c45a68d921662 in_seq=0 out_seq=0 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.050+0000 7f4412ffd640 10 client.15630 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f4412ffd640 1 -- 172.21.15.7:0/277858713 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fdf548ac00 con 0x55fdf55318a0 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f4412ffd640 10 client.15630 handle_command_reply: tid=0 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 1 client.15630 shutdown 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 1 -- 172.21.15.7:0/277858713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 msgr2=0x55fdf551eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf551eac0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f442c02c630 tx=0x7f442c003ee0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 1 -- 172.21.15.7:0/277858713 shutdown_connections 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fdf55318a0 0x55fdf5533c90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5521a80 0x55fdf551f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5518ea0 0x55fdf551f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f4412ffd640 0 client.15630 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 1 --2- 172.21.15.7:0/277858713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf551eac0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 1 -- 172.21.15.7:0/277858713 >> 172.21.15.7:0/277858713 conn(0x55fdf5512cb0 msgr2=0x55fdf5514a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 1 -- 172.21.15.7:0/277858713 shutdown_connections 2024-04-21T08:44:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 1 -- 172.21.15.7:0/277858713 wait complete. 2024-04-21T08:44:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.051+0000 7f443c994740 20 client.15630 trim_cache size 0 max 16384 2024-04-21T08:44:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f443c994740 1 Processor -- start 2024-04-21T08:44:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f443c994740 1 -- start start 2024-04-21T08:44:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f443c994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf55211f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f443c994740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5518ea0 0x55fdf5536cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f443c994740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5521a80 0x55fdf55371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f443c994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f44341af5e0 con 0x55fdf5518580 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f443c994740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44341b1e60 con 0x55fdf5518ea0 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f443c994740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f44341afc50 con 0x55fdf5521a80 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f4433fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf55211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f4433fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf55211f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47308/0 (socket says 172.21.15.7:47308) 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f4433fff640 1 -- 172.21.15.7:0/3828078069 learned_addr learned my addr 172.21.15.7:0/3828078069 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f4433fff640 1 -- 172.21.15.7:0/3828078069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5521a80 msgr2=0x55fdf55371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f4433fff640 1 --2- 172.21.15.7:0/3828078069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5521a80 0x55fdf55371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f4433fff640 1 -- 172.21.15.7:0/3828078069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5518ea0 msgr2=0x55fdf5536cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f4433fff640 1 --2- 172.21.15.7:0/3828078069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5518ea0 0x55fdf5536cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.069+0000 7f4433fff640 1 -- 172.21.15.7:0/3828078069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdf5512320 con 0x55fdf5518580 2024-04-21T08:44:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f4433fff640 1 --2- 172.21.15.7:0/3828078069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf55211f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f442c032460 tx=0x7f442c0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5437b71ddad0a5a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f44297fa640 1 -- 172.21.15.7:0/3828078069 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c039070 con 0x55fdf5518580 2024-04-21T08:44:47.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f44297fa640 1 -- 172.21.15.7:0/3828078069 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f442c032c70 con 0x55fdf5518580 2024-04-21T08:44:47.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f44297fa640 1 -- 172.21.15.7:0/3828078069 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c034890 con 0x55fdf5518580 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f443c994740 1 -- 172.21.15.7:0/3828078069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 msgr2=0x55fdf55211f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f443c994740 1 --2- 172.21.15.7:0/3828078069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf55211f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f442c032460 tx=0x7f442c0346d0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f443c994740 1 -- 172.21.15.7:0/3828078069 shutdown_connections 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f443c994740 1 --2- 172.21.15.7:0/3828078069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5521a80 0x55fdf55371f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f443c994740 1 --2- 172.21.15.7:0/3828078069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5518ea0 0x55fdf5536cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f443c994740 1 --2- 172.21.15.7:0/3828078069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518580 0x55fdf55211f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f443c994740 1 -- 172.21.15.7:0/3828078069 >> 172.21.15.7:0/3828078069 conn(0x55fdf5512cb0 msgr2=0x55fdf551daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.070+0000 7f443c994740 1 -- 172.21.15.7:0/3828078069 shutdown_connections 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f443c994740 1 -- 172.21.15.7:0/3828078069 wait complete. 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f443c994740 1 -- start start 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f443c994740 1 Processor -- start 2024-04-21T08:44:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f443c994740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5518580 0x55fdf5534900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f443c994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518ea0 0x55fdf5534e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f443c994740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5521a80 0x55fdf5535380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f443c994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f44341b0320 con 0x55fdf5518ea0 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f443c994740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f44341b1790 con 0x55fdf5521a80 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f443c994740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f44341b09f0 con 0x55fdf5518580 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f443c994740 10 client.0 _ll_register_callbacks cb 0x7f442400b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f4432ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518ea0 0x55fdf5534e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f4432ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518ea0 0x55fdf5534e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47324/0 (socket says 172.21.15.7:47324) 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.071+0000 7f4432ffd640 1 -- 172.21.15.7:0/55908583 learned_addr learned my addr 172.21.15.7:0/55908583 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f4432ffd640 1 -- 172.21.15.7:0/55908583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5518580 msgr2=0x55fdf5534900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f4432ffd640 1 --2- 172.21.15.7:0/55908583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5518580 0x55fdf5534900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f4432ffd640 1 -- 172.21.15.7:0/55908583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5521a80 msgr2=0x55fdf5535380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f4432ffd640 1 --2- 172.21.15.7:0/55908583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5521a80 0x55fdf5535380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f4432ffd640 1 -- 172.21.15.7:0/55908583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdf5596cd0 con 0x55fdf5518ea0 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f4432ffd640 1 --2- 172.21.15.7:0/55908583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518ea0 0x55fdf5534e40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x55fdf5518050 tx=0x7f4424004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713160bc7cc82857 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f4412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f4412ffd640 1 -- 172.21.15.7:0/55908583 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442401c3b0 con 0x55fdf5518ea0 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f443c994740 1 -- 172.21.15.7:0/55908583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f442c032c70 con 0x55fdf5518ea0 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f4412ffd640 1 -- 172.21.15.7:0/55908583 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f442401c550 con 0x55fdf5518ea0 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f4412ffd640 1 -- 172.21.15.7:0/55908583 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f442402a050 con 0x55fdf5518ea0 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f443c994740 10 client.15636 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f443c994740 1 -- 172.21.15.7:0/55908583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fdf55980c0 con 0x55fdf5518ea0 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.072+0000 7f4412ffd640 1 -- 172.21.15.7:0/55908583 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f442c032c70 con 0x55fdf5518ea0 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f4412ffd640 1 -- 172.21.15.7:0/55908583 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f44240103d0 con 0x55fdf5518ea0 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f443c994740 10 client.15636 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f443c994740 10 client.15636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f443c994740 20 client.15636 populate_metadata read hostname 'smithi007' 2024-04-21T08:44:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f443c994740 1 --2- 172.21.15.7:0/55908583 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fdf5598870 0x55fdf559ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f443c994740 4 client.15636 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f443c994740 1 -- 172.21.15.7:0/55908583 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fdf5484a70 con 0x55fdf5598870 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f4433fff640 1 --2- 172.21.15.7:0/55908583 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fdf5598870 0x55fdf559ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f4412ffd640 10 client.15636 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f4433fff640 1 --2- 172.21.15.7:0/55908583 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fdf5598870 0x55fdf559ac90 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39c65cd834da5ea5 server_cookie=99b962cd4d282121 in_seq=0 out_seq=0 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f4412ffd640 1 -- 172.21.15.7:0/55908583 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fdf5484a70 con 0x55fdf5598870 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f4412ffd640 10 client.15636 handle_command_reply: tid=0 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.073+0000 7f443c994740 1 client.15636 shutdown 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f443c994740 1 -- 172.21.15.7:0/55908583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518ea0 msgr2=0x55fdf5534e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f443c994740 1 --2- 172.21.15.7:0/55908583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518ea0 0x55fdf5534e40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x55fdf5518050 tx=0x7f4424004410 comp rx=0 tx=0).stop 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f443c994740 1 -- 172.21.15.7:0/55908583 shutdown_connections 2024-04-21T08:44:47.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f443c994740 1 --2- 172.21.15.7:0/55908583 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fdf5598870 0x55fdf559ac90 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f443c994740 1 --2- 172.21.15.7:0/55908583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fdf5521a80 0x55fdf5535380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f443c994740 1 --2- 172.21.15.7:0/55908583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fdf5518ea0 0x55fdf5534e40 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4412ffd640 0 client.15636 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f443c994740 1 --2- 172.21.15.7:0/55908583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fdf5518580 0x55fdf5534900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f443c994740 1 -- 172.21.15.7:0/55908583 >> 172.21.15.7:0/55908583 conn(0x55fdf5512cb0 msgr2=0x55fdf552cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f443c994740 1 -- 172.21.15.7:0/55908583 shutdown_connections 2024-04-21T08:44:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f443c994740 1 -- 172.21.15.7:0/55908583 wait complete. 2024-04-21T08:44:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f443c994740 20 client.15636 trim_cache size 0 max 16384 2024-04-21T08:44:47.079 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:44:47.079 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:44:47.080 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:44:47.080 INFO:teuthology.orchestra.run.smithi007.stdout: "ed8efc86-501e-406f-92cc-770642ecc9f9": { 2024-04-21T08:44:47.080 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:44:47.080 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "ed8efc86-501e-406f-92cc-770642ecc9f9", 2024-04-21T08:44:47.080 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:44:47.080 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:44:47.080 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:44:47.080 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:44:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4439da4640 1 -- 172.21.15.7:0/855185643 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f44200a3de0 msgr2=0x7f44200a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4439da4640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f44200a3de0 0x7f44200a62a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f441c0ce430 tx=0x7f441c009310 comp rx=0 tx=0).stop 2024-04-21T08:44:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4439da4640 1 -- 172.21.15.7:0/855185643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4434101050 msgr2=0x7f4434102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4439da4640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4434101050 0x7f4434102ae0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f441c00e9f0 tx=0x7f441c00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4439da4640 1 -- 172.21.15.7:0/855185643 shutdown_connections 2024-04-21T08:44:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4439da4640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f44200a3de0 0x7f44200a62a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4439da4640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4434103560 0x7f44341a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4439da4640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4434106120 0x7f4434103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4439da4640 1 --2- 172.21.15.7:0/855185643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4434101050 0x7f4434102ae0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4439da4640 1 -- 172.21.15.7:0/855185643 >> 172.21.15.7:0/855185643 conn(0x7f4434077fa0 msgr2=0x7f4434069530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.074+0000 7f4439da4640 1 -- 172.21.15.7:0/855185643 shutdown_connections 2024-04-21T08:44:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.075+0000 7f4439da4640 1 -- 172.21.15.7:0/855185643 wait complete. 2024-04-21T08:44:47.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:44:47.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: Updating smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:44:47.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:44:47.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:44:47.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:44:47.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:44:47.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:44:47.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:44:47.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: Updating smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:44:47.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3873053900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1164134150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:46 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2767860952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.086 INFO:tasks.cephfs.filesystem:scrub status for tag:ed8efc86-501e-406f-92cc-770642ecc9f9 - {'path': '/', 'tag': 'ed8efc86-501e-406f-92cc-770642ecc9f9', 'options': 'recursive,force'} 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: Updating smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: Updating smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3873053900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:47.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:47.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1164134150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:46 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2767860952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.112+0000 7f36ac150640 1 -- 172.21.15.7:0/4160426380 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f36a4109b40 con 0x7f36900a3da0 2024-04-21T08:44:47.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.177+0000 7f369affd640 1 -- 172.21.15.7:0/4160426380 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3694004030 con 0x7f36900a3da0 2024-04-21T08:44:47.176 INFO:teuthology.orchestra.run.smithi007.stdout:Schedule set for path / 2024-04-21T08:44:47.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 -- 172.21.15.7:0/4160426380 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36900a3da0 msgr2=0x7f36900a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36900a3da0 0x7f36900a6260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f36a4100090 tx=0x7f3694005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:47.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 -- 172.21.15.7:0/4160426380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a4100d90 msgr2=0x7f36a41a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a4100d90 0x7f36a41a06c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f36a0005940 tx=0x7f36a00045e0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 -- 172.21.15.7:0/4160426380 shutdown_connections 2024-04-21T08:44:47.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36a410e940 0x7f36a419a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36900a3da0 0x7f36900a6260 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36a4101740 0x7f36a41a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 --2- 172.21.15.7:0/4160426380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36a4100d90 0x7f36a41a06c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 -- 172.21.15.7:0/4160426380 >> 172.21.15.7:0/4160426380 conn(0x7f36a40fc970 msgr2=0x7f36a40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 -- 172.21.15.7:0/4160426380 shutdown_connections 2024-04-21T08:44:47.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.179+0000 7f36ac150640 1 -- 172.21.15.7:0/4160426380 wait complete. 2024-04-21T08:44:47.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-21T08:44:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:44:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: Updating smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:44:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:44:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: Updating smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: Updating smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: Updating smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3873053900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:47.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:47.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:44:47.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1164134150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:46 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2767860952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.337+0000 7fee8de8c640 1 -- 172.21.15.7:0/182288472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee88102b60 msgr2=0x7fee88106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.337+0000 7fee8de8c640 1 --2- 172.21.15.7:0/182288472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee88102b60 0x7fee88106ba0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fee7c009900 tx=0x7fee7c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:47.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.337+0000 7fee8de8c640 1 -- 172.21.15.7:0/182288472 shutdown_connections 2024-04-21T08:44:47.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.337+0000 7fee8de8c640 1 --2- 172.21.15.7:0/182288472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee88107280 0x7fee8810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.337+0000 7fee8de8c640 1 --2- 172.21.15.7:0/182288472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee88102b60 0x7fee88106ba0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.337+0000 7fee8de8c640 1 --2- 172.21.15.7:0/182288472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881021b0 0x7fee88102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.337+0000 7fee8de8c640 1 -- 172.21.15.7:0/182288472 >> 172.21.15.7:0/182288472 conn(0x7fee880fda10 msgr2=0x7fee880ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.337+0000 7fee8de8c640 1 -- 172.21.15.7:0/182288472 shutdown_connections 2024-04-21T08:44:47.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.337+0000 7fee8de8c640 1 -- 172.21.15.7:0/182288472 wait complete. 2024-04-21T08:44:47.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee8de8c640 1 Processor -- start 2024-04-21T08:44:47.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee8de8c640 1 -- start start 2024-04-21T08:44:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee8de8c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881021b0 0x7fee8819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee8de8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee88107280 0x7fee8819e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee8de8c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee88198600 0x7fee88198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee8de8c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fee881121a0 con 0x7fee88198600 2024-04-21T08:44:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee8de8c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee88112020 con 0x7fee881021b0 2024-04-21T08:44:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee8de8c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fee88112320 con 0x7fee88107280 2024-04-21T08:44:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee87fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee88198600 0x7fee88198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee87fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee88198600 0x7fee88198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47336/0 (socket says 172.21.15.7:47336) 2024-04-21T08:44:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee87fff640 1 -- 172.21.15.7:0/2328663323 learned_addr learned my addr 172.21.15.7:0/2328663323 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee877fe640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881021b0 0x7fee8819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee86ffd640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee88107280 0x7fee8819e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee87fff640 1 -- 172.21.15.7:0/2328663323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee88107280 msgr2=0x7fee8819e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee87fff640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee88107280 0x7fee8819e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee87fff640 1 -- 172.21.15.7:0/2328663323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881021b0 msgr2=0x7fee8819e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee87fff640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881021b0 0x7fee8819e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee87fff640 1 -- 172.21.15.7:0/2328663323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee88199120 con 0x7fee88198600 2024-04-21T08:44:47.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.338+0000 7fee877fe640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881021b0 0x7fee8819e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:47.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.339+0000 7fee87fff640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee88198600 0x7fee88198ab0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fee8000cad0 tx=0x7fee8000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.339+0000 7fee84ff9640 1 -- 172.21.15.7:0/2328663323 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8000ae20 con 0x7fee88198600 2024-04-21T08:44:47.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.339+0000 7fee84ff9640 1 -- 172.21.15.7:0/2328663323 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee80005020 con 0x7fee88198600 2024-04-21T08:44:47.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.339+0000 7fee8de8c640 1 -- 172.21.15.7:0/2328663323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee88199410 con 0x7fee88198600 2024-04-21T08:44:47.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.339+0000 7fee84ff9640 1 -- 172.21.15.7:0/2328663323 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8000f6b0 con 0x7fee88198600 2024-04-21T08:44:47.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.339+0000 7fee8de8c640 1 -- 172.21.15.7:0/2328663323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee881a55c0 con 0x7fee88198600 2024-04-21T08:44:47.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.340+0000 7fee8de8c640 1 -- 172.21.15.7:0/2328663323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee88050530 con 0x7fee88198600 2024-04-21T08:44:47.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.341+0000 7fee84ff9640 1 -- 172.21.15.7:0/2328663323 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7fee800051d0 con 0x7fee88198600 2024-04-21T08:44:47.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.341+0000 7fee84ff9640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fee700a3da0 0x7fee700a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.342+0000 7fee877fe640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fee700a3da0 0x7fee700a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.342+0000 7fee84ff9640 1 -- 172.21.15.7:0/2328663323 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fee800d0ea0 con 0x7fee88198600 2024-04-21T08:44:47.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.342+0000 7fee877fe640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fee700a3da0 0x7fee700a6260 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee780058c0 tx=0x7fee78005ec0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.346+0000 7fee84ff9640 1 -- 172.21.15.7:0/2328663323 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee80014030 con 0x7fee88198600 2024-04-21T08:44:47.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.556+0000 7fee8de8c640 1 -- 172.21.15.7:0/2328663323 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fee88102b60 con 0x7fee700a3da0 2024-04-21T08:44:47.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.617+0000 7fee84ff9640 1 -- 172.21.15.7:0/2328663323 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fee88102b60 con 0x7fee700a3da0 2024-04-21T08:44:47.617 INFO:teuthology.orchestra.run.smithi007.stdout:Retention added to path / 2024-04-21T08:44:47.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 -- 172.21.15.7:0/2328663323 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fee700a3da0 msgr2=0x7fee700a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fee700a3da0 0x7fee700a6260 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee780058c0 tx=0x7fee78005ec0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 -- 172.21.15.7:0/2328663323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee88198600 msgr2=0x7fee88198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee88198600 0x7fee88198ab0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fee8000cad0 tx=0x7fee8000cf90 comp rx=0 tx=0).stop 2024-04-21T08:44:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 -- 172.21.15.7:0/2328663323 shutdown_connections 2024-04-21T08:44:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee88198600 0x7fee88198ab0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fee700a3da0 0x7fee700a6260 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee88107280 0x7fee8819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 --2- 172.21.15.7:0/2328663323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee881021b0 0x7fee8819e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 -- 172.21.15.7:0/2328663323 >> 172.21.15.7:0/2328663323 conn(0x7fee880fda10 msgr2=0x7fee880fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 -- 172.21.15.7:0/2328663323 shutdown_connections 2024-04-21T08:44:47.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.619+0000 7fee8de8c640 1 -- 172.21.15.7:0/2328663323 wait complete. 2024-04-21T08:44:47.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T08:44:47.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.733+0000 7f8ebb825640 1 -- 172.21.15.7:0/3147333913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8eb4101a50 msgr2=0x7f8eb4105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.733+0000 7f8ebb825640 1 --2- 172.21.15.7:0/3147333913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8eb4101a50 0x7f8eb4105a90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009900 tx=0x7f8eb002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.733+0000 7f8ebb825640 1 -- 172.21.15.7:0/3147333913 shutdown_connections 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.733+0000 7f8ebb825640 1 --2- 172.21.15.7:0/3147333913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb4106170 0x7f8eb410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.733+0000 7f8ebb825640 1 --2- 172.21.15.7:0/3147333913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8eb4101a50 0x7f8eb4105a90 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.733+0000 7f8ebb825640 1 --2- 172.21.15.7:0/3147333913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8eb41010a0 0x7f8eb4101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.733+0000 7f8ebb825640 1 -- 172.21.15.7:0/3147333913 >> 172.21.15.7:0/3147333913 conn(0x7f8eb4077f70 msgr2=0x7f8eb4078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.733+0000 7f8ebb825640 1 -- 172.21.15.7:0/3147333913 shutdown_connections 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.733+0000 7f8ebb825640 1 -- 172.21.15.7:0/3147333913 wait complete. 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.733+0000 7f8ebb825640 1 Processor -- start 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.733+0000 7f8ebb825640 1 -- start start 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8ebb825640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb41010a0 0x7f8eb4102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8ebb825640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8eb4106170 0x7f8eb4103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8ebb825640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8eb41035b0 0x7f8eb41a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8ebb825640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb4111090 con 0x7f8eb4106170 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8ebb825640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb4110f10 con 0x7f8eb41035b0 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8ebb825640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb4111210 con 0x7f8eb41010a0 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eb8dc0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8eb4106170 0x7f8eb4103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eb8dc0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8eb4106170 0x7f8eb4103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47356/0 (socket says 172.21.15.7:47356) 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eb95c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb41010a0 0x7f8eb4102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eb8dc0640 1 -- 172.21.15.7:0/259638740 learned_addr learned my addr 172.21.15.7:0/259638740 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eb9dc2640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8eb41035b0 0x7f8eb41a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eb8dc0640 1 -- 172.21.15.7:0/259638740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb41010a0 msgr2=0x7f8eb4102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eb8dc0640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb41010a0 0x7f8eb4102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eb8dc0640 1 -- 172.21.15.7:0/259638740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8eb41035b0 msgr2=0x7f8eb41a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eb8dc0640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8eb41035b0 0x7f8eb41a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eb8dc0640 1 -- 172.21.15.7:0/259638740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb41a6f90 con 0x7f8eb4106170 2024-04-21T08:44:47.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eb8dc0640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8eb4106170 0x7f8eb4103070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0036040 tx=0x7f8eb002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.734+0000 7f8eaa7fc640 1 -- 172.21.15.7:0/259638740 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0043020 con 0x7f8eb4106170 2024-04-21T08:44:47.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.735+0000 7f8ebb825640 1 -- 172.21.15.7:0/259638740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb41a7220 con 0x7f8eb4106170 2024-04-21T08:44:47.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.735+0000 7f8eaa7fc640 1 -- 172.21.15.7:0/259638740 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8eb002cbe0 con 0x7f8eb4106170 2024-04-21T08:44:47.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.735+0000 7f8eaa7fc640 1 -- 172.21.15.7:0/259638740 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0003bf0 con 0x7f8eb4106170 2024-04-21T08:44:47.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.735+0000 7f8ebb825640 1 -- 172.21.15.7:0/259638740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eb41a7770 con 0x7f8eb4106170 2024-04-21T08:44:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.736+0000 7f8ebb825640 1 -- 172.21.15.7:0/259638740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8eb4050530 con 0x7f8eb4106170 2024-04-21T08:44:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.737+0000 7f8eaa7fc640 1 -- 172.21.15.7:0/259638740 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f8eb0046020 con 0x7f8eb4106170 2024-04-21T08:44:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.737+0000 7f8eaa7fc640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ea00a3dd0 0x7f8ea00a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.737+0000 7f8eaa7fc640 1 -- 172.21.15.7:0/259638740 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f8eb00f3470 con 0x7f8eb4106170 2024-04-21T08:44:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.737+0000 7f8eb95c1640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ea00a3dd0 0x7f8ea00a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.738+0000 7f8eb95c1640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ea00a3dd0 0x7f8ea00a6290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400bbe0 tx=0x7f8ea400b390 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.741+0000 7f8eaa7fc640 1 -- 172.21.15.7:0/259638740 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8eb004d020 con 0x7f8eb4106170 2024-04-21T08:44:47.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:47 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:44:47.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:47 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:47.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:47 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:44:47.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:47 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.950+0000 7f8ebb825640 1 -- 172.21.15.7:0/259638740 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f8eb4101a50 con 0x7f8ea00a3dd0 2024-04-21T08:44:47.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.958+0000 7f8eaa7fc640 1 -- 172.21.15.7:0/259638740 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f8eb4101a50 con 0x7f8ea00a3dd0 2024-04-21T08:44:47.957 INFO:teuthology.orchestra.run.smithi007.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T08:44:47", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-21T08:44:47.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.959+0000 7f8ebb825640 1 -- 172.21.15.7:0/259638740 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ea00a3dd0 msgr2=0x7f8ea00a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.959+0000 7f8ebb825640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ea00a3dd0 0x7f8ea00a6290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400bbe0 tx=0x7f8ea400b390 comp rx=0 tx=0).stop 2024-04-21T08:44:47.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.959+0000 7f8ebb825640 1 -- 172.21.15.7:0/259638740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8eb4106170 msgr2=0x7f8eb4103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.959+0000 7f8ebb825640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8eb4106170 0x7f8eb4103070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0036040 tx=0x7f8eb002c7e0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.960+0000 7f8ebb825640 1 -- 172.21.15.7:0/259638740 shutdown_connections 2024-04-21T08:44:47.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.960+0000 7f8ebb825640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8eb41035b0 0x7f8eb41a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.960+0000 7f8ebb825640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8eb4106170 0x7f8eb4103070 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.960+0000 7f8ebb825640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ea00a3dd0 0x7f8ea00a6290 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.960+0000 7f8ebb825640 1 --2- 172.21.15.7:0/259638740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8eb41010a0 0x7f8eb4102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.960+0000 7f8ebb825640 1 -- 172.21.15.7:0/259638740 >> 172.21.15.7:0/259638740 conn(0x7f8eb4077f70 msgr2=0x7f8eb4069a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.960+0000 7f8ebb825640 1 -- 172.21.15.7:0/259638740 shutdown_connections 2024-04-21T08:44:47.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:47.960+0000 7f8ebb825640 1 -- 172.21.15.7:0/259638740 wait complete. 2024-04-21T08:44:47.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T08:44:48.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.074+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2737694142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe14101120 msgr2=0x7efe14101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.074+0000 7efe1b4c1640 1 --2- 172.21.15.7:0/2737694142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe14101120 0x7efe14101500 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7efe10009900 tx=0x7efe1002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.074+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2737694142 shutdown_connections 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.074+0000 7efe1b4c1640 1 --2- 172.21.15.7:0/2737694142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe141061a0 0x7efe1410dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.074+0000 7efe1b4c1640 1 --2- 172.21.15.7:0/2737694142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe14101ad0 0x7efe14105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.074+0000 7efe1b4c1640 1 --2- 172.21.15.7:0/2737694142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe14101120 0x7efe14101500 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.074+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2737694142 >> 172.21.15.7:0/2737694142 conn(0x7efe14077fa0 msgr2=0x7efe140783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.074+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2737694142 shutdown_connections 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.074+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2737694142 wait complete. 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe1b4c1640 1 Processor -- start 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe1b4c1640 1 -- start start 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe1b4c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14101120 0x7efe1419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe1b4c1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe141061a0 0x7efe1419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe1b4c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe141985a0 0x7efe14198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe1b4c1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efe14112220 con 0x7efe141061a0 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe1b4c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe141120a0 con 0x7efe14101120 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe1b4c1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efe141123a0 con 0x7efe141985a0 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe18a5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe141061a0 0x7efe1419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe18a5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe141061a0 0x7efe1419e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47384/0 (socket says 172.21.15.7:47384) 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe18a5c640 1 -- 172.21.15.7:0/2084238294 learned_addr learned my addr 172.21.15.7:0/2084238294 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe1925d640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14101120 0x7efe1419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe19a5e640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe141985a0 0x7efe14198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe18a5c640 1 -- 172.21.15.7:0/2084238294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe141985a0 msgr2=0x7efe14198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe18a5c640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe141985a0 0x7efe14198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe18a5c640 1 -- 172.21.15.7:0/2084238294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14101120 msgr2=0x7efe1419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe18a5c640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14101120 0x7efe1419e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe18a5c640 1 -- 172.21.15.7:0/2084238294 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe141990c0 con 0x7efe141061a0 2024-04-21T08:44:48.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.075+0000 7efe18a5c640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe141061a0 0x7efe1419e930 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7efe0400e9f0 tx=0x7efe0400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.076+0000 7efe0a7fc640 1 -- 172.21.15.7:0/2084238294 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0400ccc0 con 0x7efe141061a0 2024-04-21T08:44:48.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.076+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2084238294 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe141993b0 con 0x7efe141061a0 2024-04-21T08:44:48.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.076+0000 7efe0a7fc640 1 -- 172.21.15.7:0/2084238294 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe040054d0 con 0x7efe141061a0 2024-04-21T08:44:48.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.076+0000 7efe0a7fc640 1 -- 172.21.15.7:0/2084238294 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04010430 con 0x7efe141061a0 2024-04-21T08:44:48.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.076+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2084238294 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe141a55b0 con 0x7efe141061a0 2024-04-21T08:44:48.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.077+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2084238294 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe141025a0 con 0x7efe141061a0 2024-04-21T08:44:48.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.078+0000 7efe0a7fc640 1 -- 172.21.15.7:0/2084238294 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7efe040046c0 con 0x7efe141061a0 2024-04-21T08:44:48.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.078+0000 7efe0a7fc640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe000a3da0 0x7efe000a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.079+0000 7efe1925d640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe000a3da0 0x7efe000a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.079+0000 7efe0a7fc640 1 -- 172.21.15.7:0/2084238294 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7efe040d13b0 con 0x7efe141061a0 2024-04-21T08:44:48.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.079+0000 7efe1925d640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe000a3da0 0x7efe000a6260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efe10006010 tx=0x7efe10005e40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.083+0000 7efe0a7fc640 1 -- 172.21.15.7:0/2084238294 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe04053ff0 con 0x7efe141061a0 2024-04-21T08:44:48.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:47 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:44:48.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:47 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:48.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:47 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:44:48.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:47 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.292+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2084238294 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7efe14101ad0 con 0x7efe000a3da0 2024-04-21T08:44:48.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.298+0000 7efe0a7fc640 1 -- 172.21.15.7:0/2084238294 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7efe14101ad0 con 0x7efe000a3da0 2024-04-21T08:44:48.298 INFO:teuthology.orchestra.run.smithi007.stdout:/ 1m 6m3h 2024-04-21T08:44:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2084238294 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe000a3da0 msgr2=0x7efe000a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe000a3da0 0x7efe000a6260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efe10006010 tx=0x7efe10005e40 comp rx=0 tx=0).stop 2024-04-21T08:44:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2084238294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe141061a0 msgr2=0x7efe1419e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe141061a0 0x7efe1419e930 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7efe0400e9f0 tx=0x7efe0400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2084238294 shutdown_connections 2024-04-21T08:44:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7efe000a3da0 0x7efe000a6260 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe141985a0 0x7efe14198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe141061a0 0x7efe1419e930 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 --2- 172.21.15.7:0/2084238294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe14101120 0x7efe1419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2084238294 >> 172.21.15.7:0/2084238294 conn(0x7efe14077fa0 msgr2=0x7efe140ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2084238294 shutdown_connections 2024-04-21T08:44:48.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.300+0000 7efe1b4c1640 1 -- 172.21.15.7:0/2084238294 wait complete. 2024-04-21T08:44:48.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:47 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:44:48.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:47 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:48.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:47 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:44:48.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:47 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:44:48.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-21T08:44:48.336 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-21T08:44:48.346 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-21T08:44:48.387 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-21T08:44:48.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-21T08:44:48.390 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-21T08:44:48.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.525+0000 7f71629ce640 1 -- 172.21.15.7:0/3534631035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715c105dd0 msgr2=0x7f715c1061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.525+0000 7f71629ce640 1 --2- 172.21.15.7:0/3534631035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715c105dd0 0x7f715c1061b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f714c009900 tx=0x7f714c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.525+0000 7f71629ce640 1 -- 172.21.15.7:0/3534631035 shutdown_connections 2024-04-21T08:44:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.525+0000 7f71629ce640 1 --2- 172.21.15.7:0/3534631035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f715c101a40 0x7f715c10dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.525+0000 7f71629ce640 1 --2- 172.21.15.7:0/3534631035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f715c1010a0 0x7f715c101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.525+0000 7f71629ce640 1 --2- 172.21.15.7:0/3534631035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715c105dd0 0x7f715c1061b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.525+0000 7f71629ce640 1 -- 172.21.15.7:0/3534631035 >> 172.21.15.7:0/3534631035 conn(0x7f715c077c40 msgr2=0x7f715c078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.525+0000 7f71629ce640 1 -- 172.21.15.7:0/3534631035 shutdown_connections 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.525+0000 7f71629ce640 1 -- 172.21.15.7:0/3534631035 wait complete. 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f71629ce640 1 Processor -- start 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f71629ce640 1 -- start start 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f71629ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f715c1010a0 0x7f715c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f71629ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715c101a40 0x7f715c19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f71629ce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f715c105dd0 0x7f715c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f71629ce640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f715c1122d0 con 0x7f715c101a40 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f71629ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f715c112150 con 0x7f715c1010a0 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f71629ce640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f715c112450 con 0x7f715c105dd0 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f715b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715c101a40 0x7f715c19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f715b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715c101a40 0x7f715c19e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47402/0 (socket says 172.21.15.7:47402) 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f715b7fe640 1 -- 172.21.15.7:0/4240152574 learned_addr learned my addr 172.21.15.7:0/4240152574 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f715bfff640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f715c1010a0 0x7f715c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f7160f6b640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f715c105dd0 0x7f715c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f715b7fe640 1 -- 172.21.15.7:0/4240152574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f715c105dd0 msgr2=0x7f715c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.526+0000 7f715b7fe640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f715c105dd0 0x7f715c1985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.527+0000 7f715b7fe640 1 -- 172.21.15.7:0/4240152574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f715c1010a0 msgr2=0x7f715c19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.527+0000 7f715b7fe640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f715c1010a0 0x7f715c19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.527+0000 7f715b7fe640 1 -- 172.21.15.7:0/4240152574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715c198d50 con 0x7f715c101a40 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.527+0000 7f715bfff640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f715c1010a0 0x7f715c19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.527+0000 7f715b7fe640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715c101a40 0x7f715c19e960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f715000d770 tx=0x7f715000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.527+0000 7f71597fa640 1 -- 172.21.15.7:0/4240152574 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150017070 con 0x7f715c101a40 2024-04-21T08:44:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.527+0000 7f71597fa640 1 -- 172.21.15.7:0/4240152574 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71500054d0 con 0x7f715c101a40 2024-04-21T08:44:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.527+0000 7f71629ce640 1 -- 172.21.15.7:0/4240152574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f715c199040 con 0x7f715c101a40 2024-04-21T08:44:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.527+0000 7f71597fa640 1 -- 172.21.15.7:0/4240152574 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150004c30 con 0x7f715c101a40 2024-04-21T08:44:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.527+0000 7f71629ce640 1 -- 172.21.15.7:0/4240152574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f715c1a52e0 con 0x7f715c101a40 2024-04-21T08:44:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.528+0000 7f71629ce640 1 -- 172.21.15.7:0/4240152574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f715c102540 con 0x7f715c101a40 2024-04-21T08:44:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.529+0000 7f71597fa640 1 -- 172.21.15.7:0/4240152574 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f7150004dd0 con 0x7f715c101a40 2024-04-21T08:44:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.530+0000 7f71597fa640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f71440a3e30 0x7f71440a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.530+0000 7f71597fa640 1 -- 172.21.15.7:0/4240152574 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f71500d0d20 con 0x7f715c101a40 2024-04-21T08:44:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.530+0000 7f715bfff640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f71440a3e30 0x7f71440a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.530+0000 7f715bfff640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f71440a3e30 0x7f71440a62f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f714c005ab0 tx=0x7f714c005a20 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.533+0000 7f71597fa640 1 -- 172.21.15.7:0/4240152574 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f715009ce80 con 0x7f715c101a40 2024-04-21T08:44:48.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.724+0000 7f71629ce640 1 -- 172.21.15.7:0/4240152574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f715c109340 con 0x7f715c101a40 2024-04-21T08:44:48.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.725+0000 7f71597fa640 1 -- 172.21.15.7:0/4240152574 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25085 (secure 0 0 0) 0x7f715009c780 con 0x7f715c101a40 2024-04-21T08:44:48.725 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:48.725 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":95,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:44:41.244735+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3451243741":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/117789107":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/4167672076":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/3630897301":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:48.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 -- 172.21.15.7:0/4240152574 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f71440a3e30 msgr2=0x7f71440a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f71440a3e30 0x7f71440a62f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f714c005ab0 tx=0x7f714c005a20 comp rx=0 tx=0).stop 2024-04-21T08:44:48.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 -- 172.21.15.7:0/4240152574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715c101a40 msgr2=0x7f715c19e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715c101a40 0x7f715c19e960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f715000d770 tx=0x7f715000dc30 comp rx=0 tx=0).stop 2024-04-21T08:44:48.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 -- 172.21.15.7:0/4240152574 shutdown_connections 2024-04-21T08:44:48.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f71440a3e30 0x7f71440a62f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f715c105dd0 0x7f715c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715c101a40 0x7f715c19e960 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 --2- 172.21.15.7:0/4240152574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f715c1010a0 0x7f715c19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 -- 172.21.15.7:0/4240152574 >> 172.21.15.7:0/4240152574 conn(0x7f715c077c40 msgr2=0x7f715c0fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 -- 172.21.15.7:0/4240152574 shutdown_connections 2024-04-21T08:44:48.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.728+0000 7f71629ce640 1 -- 172.21.15.7:0/4240152574 wait complete. 2024-04-21T08:44:48.738 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:48.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:48.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.845+0000 7f4a2a967640 1 -- 172.21.15.7:0/2287439070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a241010a0 msgr2=0x7f4a24101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.845+0000 7f4a2a967640 1 --2- 172.21.15.7:0/2287439070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a241010a0 0x7f4a24101480 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009900 tx=0x7f4a1402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:48.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.845+0000 7f4a2a967640 1 -- 172.21.15.7:0/2287439070 shutdown_connections 2024-04-21T08:44:48.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.845+0000 7f4a2a967640 1 --2- 172.21.15.7:0/2287439070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a24106170 0x7f4a2410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.845+0000 7f4a2a967640 1 --2- 172.21.15.7:0/2287439070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a24101a50 0x7f4a24105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.845+0000 7f4a2a967640 1 --2- 172.21.15.7:0/2287439070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a241010a0 0x7f4a24101480 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.845+0000 7f4a2a967640 1 -- 172.21.15.7:0/2287439070 >> 172.21.15.7:0/2287439070 conn(0x7f4a24077f40 msgr2=0x7f4a24078350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.845+0000 7f4a2a967640 1 -- 172.21.15.7:0/2287439070 shutdown_connections 2024-04-21T08:44:48.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.845+0000 7f4a2a967640 1 -- 172.21.15.7:0/2287439070 wait complete. 2024-04-21T08:44:48.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a2a967640 1 Processor -- start 2024-04-21T08:44:48.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a2a967640 1 -- start start 2024-04-21T08:44:48.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a2a967640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a241010a0 0x7f4a24197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a2a967640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a24106170 0x7f4a24197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a2a967640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2419bf10 0x7f4a2419c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a2a967640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4a241110d0 con 0x7f4a2419bf10 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a2a967640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a24110f50 con 0x7f4a24106170 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a2a967640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4a24111250 con 0x7f4a241010a0 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a28f04640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2419bf10 0x7f4a2419c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a28f04640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2419bf10 0x7f4a2419c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47424/0 (socket says 172.21.15.7:47424) 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a28f04640 1 -- 172.21.15.7:0/1889578094 learned_addr learned my addr 172.21.15.7:0/1889578094 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a237fe640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a24106170 0x7f4a24197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a23fff640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a241010a0 0x7f4a24197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a28f04640 1 -- 172.21.15.7:0/1889578094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a241010a0 msgr2=0x7f4a24197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a28f04640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a241010a0 0x7f4a24197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a28f04640 1 -- 172.21.15.7:0/1889578094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a24106170 msgr2=0x7f4a24197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a28f04640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a24106170 0x7f4a24197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.846+0000 7f4a28f04640 1 -- 172.21.15.7:0/1889578094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2419ca00 con 0x7f4a2419bf10 2024-04-21T08:44:48.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.847+0000 7f4a28f04640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2419bf10 0x7f4a2419c3c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00eb10 tx=0x7f4a1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.847+0000 7f4a217fa640 1 -- 172.21.15.7:0/1889578094 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c004b20 con 0x7f4a2419bf10 2024-04-21T08:44:48.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.847+0000 7f4a2a967640 1 -- 172.21.15.7:0/1889578094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2419ccf0 con 0x7f4a2419bf10 2024-04-21T08:44:48.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.847+0000 7f4a217fa640 1 -- 172.21.15.7:0/1889578094 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a1c004cc0 con 0x7f4a2419bf10 2024-04-21T08:44:48.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.847+0000 7f4a217fa640 1 -- 172.21.15.7:0/1889578094 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c010670 con 0x7f4a2419bf10 2024-04-21T08:44:48.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.847+0000 7f4a2a967640 1 -- 172.21.15.7:0/1889578094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a241a4490 con 0x7f4a2419bf10 2024-04-21T08:44:48.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.848+0000 7f4a2a967640 1 -- 172.21.15.7:0/1889578094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a24068570 con 0x7f4a2419bf10 2024-04-21T08:44:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.849+0000 7f4a217fa640 1 -- 172.21.15.7:0/1889578094 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f4a1c005020 con 0x7f4a2419bf10 2024-04-21T08:44:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.850+0000 7f4a217fa640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4a0c0a3db0 0x7f4a0c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.850+0000 7f4a23fff640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4a0c0a3db0 0x7f4a0c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.850+0000 7f4a217fa640 1 -- 172.21.15.7:0/1889578094 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f4a1c0d1290 con 0x7f4a2419bf10 2024-04-21T08:44:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.850+0000 7f4a23fff640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4a0c0a3db0 0x7f4a0c0a6270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4a14006010 tx=0x7f4a14005e40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:48.854+0000 7f4a217fa640 1 -- 172.21.15.7:0/1889578094 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a1c09d3d0 con 0x7f4a2419bf10 2024-04-21T08:44:49.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.041+0000 7f4a2a967640 1 -- 172.21.15.7:0/1889578094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4a24101a50 con 0x7f4a2419bf10 2024-04-21T08:44:49.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.041+0000 7f4a217fa640 1 -- 172.21.15.7:0/1889578094 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f4a1c09ccd0 con 0x7f4a2419bf10 2024-04-21T08:44:49.041 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:44:49.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 -- 172.21.15.7:0/1889578094 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4a0c0a3db0 msgr2=0x7f4a0c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4a0c0a3db0 0x7f4a0c0a6270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4a14006010 tx=0x7f4a14005e40 comp rx=0 tx=0).stop 2024-04-21T08:44:49.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 -- 172.21.15.7:0/1889578094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2419bf10 msgr2=0x7f4a2419c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2419bf10 0x7f4a2419c3c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00eb10 tx=0x7f4a1c00c2d0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 -- 172.21.15.7:0/1889578094 shutdown_connections 2024-04-21T08:44:49.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2419bf10 0x7f4a2419c3c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a24106170 0x7f4a24197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4a0c0a3db0 0x7f4a0c0a6270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 --2- 172.21.15.7:0/1889578094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a241010a0 0x7f4a24197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 -- 172.21.15.7:0/1889578094 >> 172.21.15.7:0/1889578094 conn(0x7f4a24077f40 msgr2=0x7f4a24107340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 -- 172.21.15.7:0/1889578094 shutdown_connections 2024-04-21T08:44:49.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.043+0000 7f4a2a967640 1 -- 172.21.15.7:0/1889578094 wait complete. 2024-04-21T08:44:49.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:48 smithi007 ceph-mon[28477]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:49.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:48 smithi007 ceph-mon[28477]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 64 KiB/s wr, 29 op/s 2024-04-21T08:44:49.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:48 smithi007 ceph-mon[28477]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:49.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:48 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4240152574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:49.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:49.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:48 smithi077 ceph-mon[31615]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:49.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:48 smithi077 ceph-mon[31615]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 64 KiB/s wr, 29 op/s 2024-04-21T08:44:49.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:48 smithi077 ceph-mon[31615]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:49.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:48 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4240152574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.159+0000 7f3900632640 1 -- 172.21.15.7:0/3003644361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38f8101180 msgr2=0x7f38f8101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.159+0000 7f3900632640 1 --2- 172.21.15.7:0/3003644361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38f8101180 0x7f38f8101560 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f38f4009930 tx=0x7f38f402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.159+0000 7f3900632640 1 -- 172.21.15.7:0/3003644361 shutdown_connections 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.159+0000 7f3900632640 1 --2- 172.21.15.7:0/3003644361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8103400 0x7f38f810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.159+0000 7f3900632640 1 --2- 172.21.15.7:0/3003644361 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8102a60 0x7f38f8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.159+0000 7f3900632640 1 --2- 172.21.15.7:0/3003644361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38f8101180 0x7f38f8101560 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.159+0000 7f3900632640 1 -- 172.21.15.7:0/3003644361 >> 172.21.15.7:0/3003644361 conn(0x7f38f8077f70 msgr2=0x7f38f8078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.159+0000 7f3900632640 1 -- 172.21.15.7:0/3003644361 shutdown_connections 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.159+0000 7f3900632640 1 -- 172.21.15.7:0/3003644361 wait complete. 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f3900632640 1 Processor -- start 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f3900632640 1 -- start start 2024-04-21T08:44:49.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f3900632640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38f8101180 0x7f38f819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f3900632640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8102a60 0x7f38f819e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f3900632640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8103400 0x7f38f8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f3900632640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8112300 con 0x7f38f8101180 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f3900632640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8112180 con 0x7f38f8103400 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f3900632640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8112480 con 0x7f38f8102a60 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f38fe3ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38f8101180 0x7f38f819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f38fe3ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38f8101180 0x7f38f819e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47428/0 (socket says 172.21.15.7:47428) 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f38fe3ce640 1 -- 172.21.15.7:0/3905456429 learned_addr learned my addr 172.21.15.7:0/3905456429 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f38fdbcd640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8102a60 0x7f38f819e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f38fe3ce640 1 -- 172.21.15.7:0/3905456429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8102a60 msgr2=0x7f38f819e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f38febcf640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8103400 0x7f38f8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f38fe3ce640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8102a60 0x7f38f819e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f38fe3ce640 1 -- 172.21.15.7:0/3905456429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8103400 msgr2=0x7f38f8198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f38fe3ce640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8103400 0x7f38f8198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.160+0000 7f38fe3ce640 1 -- 172.21.15.7:0/3905456429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f8198da0 con 0x7f38f8101180 2024-04-21T08:44:49.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.161+0000 7f38fe3ce640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38f8101180 0x7f38f819e470 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f38f4009230 tx=0x7f38f402c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.161+0000 7f38ef7fe640 1 -- 172.21.15.7:0/3905456429 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f4043020 con 0x7f38f8101180 2024-04-21T08:44:49.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.161+0000 7f38ef7fe640 1 -- 172.21.15.7:0/3905456429 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38f402caf0 con 0x7f38f8101180 2024-04-21T08:44:49.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.161+0000 7f3900632640 1 -- 172.21.15.7:0/3905456429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f8199030 con 0x7f38f8101180 2024-04-21T08:44:49.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.161+0000 7f38ef7fe640 1 -- 172.21.15.7:0/3905456429 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f4003af0 con 0x7f38f8101180 2024-04-21T08:44:49.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.161+0000 7f3900632640 1 -- 172.21.15.7:0/3905456429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f81a5330 con 0x7f38f8101180 2024-04-21T08:44:49.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.162+0000 7f3900632640 1 -- 172.21.15.7:0/3905456429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38f8069250 con 0x7f38f8101180 2024-04-21T08:44:49.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.163+0000 7f38ef7fe640 1 -- 172.21.15.7:0/3905456429 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f38f4046020 con 0x7f38f8101180 2024-04-21T08:44:49.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.164+0000 7f38ef7fe640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38e40a3e30 0x7f38e40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.164+0000 7f38fdbcd640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38e40a3e30 0x7f38e40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.164+0000 7f38ef7fe640 1 -- 172.21.15.7:0/3905456429 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f38f40f31d0 con 0x7f38f8101180 2024-04-21T08:44:49.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.164+0000 7f38fdbcd640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38e40a3e30 0x7f38e40a62f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f38f80786d0 tx=0x7f38e8005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.167+0000 7f38ef7fe640 1 -- 172.21.15.7:0/3905456429 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38f404d020 con 0x7f38f8101180 2024-04-21T08:44:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:48 smithi016 ceph-mon[31259]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:48 smithi016 ceph-mon[31259]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 64 KiB/s wr, 29 op/s 2024-04-21T08:44:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:48 smithi016 ceph-mon[31259]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:48 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4240152574' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:49.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.353+0000 7f3900632640 1 -- 172.21.15.7:0/3905456429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f38f8109450 con 0x7f38f8101180 2024-04-21T08:44:49.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.354+0000 7f38ef7fe640 1 -- 172.21.15.7:0/3905456429 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f38f403e450 con 0x7f38f8101180 2024-04-21T08:44:49.353 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:49.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.355+0000 7f3900632640 1 -- 172.21.15.7:0/3905456429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38e40a3e30 msgr2=0x7f38e40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.355+0000 7f3900632640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38e40a3e30 0x7f38e40a62f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f38f80786d0 tx=0x7f38e8005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:49.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.355+0000 7f3900632640 1 -- 172.21.15.7:0/3905456429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38f8101180 msgr2=0x7f38f819e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.355+0000 7f3900632640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38f8101180 0x7f38f819e470 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f38f4009230 tx=0x7f38f402c710 comp rx=0 tx=0).stop 2024-04-21T08:44:49.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.355+0000 7f3900632640 1 -- 172.21.15.7:0/3905456429 shutdown_connections 2024-04-21T08:44:49.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.355+0000 7f3900632640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38f8103400 0x7f38f8198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.355+0000 7f3900632640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38e40a3e30 0x7f38e40a62f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.355+0000 7f3900632640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38f8102a60 0x7f38f819e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.356+0000 7f3900632640 1 --2- 172.21.15.7:0/3905456429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38f8101180 0x7f38f819e470 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.356+0000 7f3900632640 1 -- 172.21.15.7:0/3905456429 >> 172.21.15.7:0/3905456429 conn(0x7f38f8077f70 msgr2=0x7f38f8104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.356+0000 7f3900632640 1 -- 172.21.15.7:0/3905456429 shutdown_connections 2024-04-21T08:44:49.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.356+0000 7f3900632640 1 -- 172.21.15.7:0/3905456429 wait complete. 2024-04-21T08:44:49.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:49.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.472+0000 7f8159fb6640 1 -- 172.21.15.7:0/4285704266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8154103b70 msgr2=0x7f815410bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.472+0000 7f8159fb6640 1 --2- 172.21.15.7:0/4285704266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8154103b70 0x7f815410bfc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8148009900 tx=0x7f814802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:49.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.472+0000 7f8159fb6640 1 -- 172.21.15.7:0/4285704266 shutdown_connections 2024-04-21T08:44:49.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.472+0000 7f8159fb6640 1 --2- 172.21.15.7:0/4285704266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f815410c6d0 0x7f815410eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.472+0000 7f8159fb6640 1 --2- 172.21.15.7:0/4285704266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8154103b70 0x7f815410bfc0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.472+0000 7f8159fb6640 1 --2- 172.21.15.7:0/4285704266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81541031c0 0x7f81541035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.472+0000 7f8159fb6640 1 -- 172.21.15.7:0/4285704266 >> 172.21.15.7:0/4285704266 conn(0x7f8154077f70 msgr2=0x7f8154078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.472+0000 7f8159fb6640 1 -- 172.21.15.7:0/4285704266 shutdown_connections 2024-04-21T08:44:49.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.472+0000 7f8159fb6640 1 -- 172.21.15.7:0/4285704266 wait complete. 2024-04-21T08:44:49.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.472+0000 7f8159fb6640 1 Processor -- start 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.472+0000 7f8159fb6640 1 -- start start 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8159fb6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81541031c0 0x7f8154100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8159fb6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f815410c6d0 0x7f8154101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8159fb6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8154101890 0x7f81541a6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8159fb6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8154112330 con 0x7f8154101890 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8159fb6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f81541121b0 con 0x7f815410c6d0 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8159fb6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f81541124b0 con 0x7f81541031c0 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8153fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8154101890 0x7f81541a6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8153fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8154101890 0x7f81541a6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47442/0 (socket says 172.21.15.7:47442) 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8153fff640 1 -- 172.21.15.7:0/1548630137 learned_addr learned my addr 172.21.15.7:0/1548630137 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f81537fe640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81541031c0 0x7f8154100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8152ffd640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f815410c6d0 0x7f8154101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8153fff640 1 -- 172.21.15.7:0/1548630137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81541031c0 msgr2=0x7f8154100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8153fff640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81541031c0 0x7f8154100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8153fff640 1 -- 172.21.15.7:0/1548630137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f815410c6d0 msgr2=0x7f8154101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8153fff640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f815410c6d0 0x7f8154101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8153fff640 1 -- 172.21.15.7:0/1548630137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81541a71d0 con 0x7f8154101890 2024-04-21T08:44:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8152ffd640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f815410c6d0 0x7f8154101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8153fff640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8154101890 0x7f81541a6ad0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f814c00eb10 tx=0x7f814c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.473+0000 7f8150ff9640 1 -- 172.21.15.7:0/1548630137 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f814c004b20 con 0x7f8154101890 2024-04-21T08:44:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.474+0000 7f8150ff9640 1 -- 172.21.15.7:0/1548630137 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f814c004cc0 con 0x7f8154101890 2024-04-21T08:44:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.474+0000 7f8159fb6640 1 -- 172.21.15.7:0/1548630137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81541a74c0 con 0x7f8154101890 2024-04-21T08:44:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.474+0000 7f8150ff9640 1 -- 172.21.15.7:0/1548630137 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f814c010670 con 0x7f8154101890 2024-04-21T08:44:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.474+0000 7f8159fb6640 1 -- 172.21.15.7:0/1548630137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81541a7990 con 0x7f8154101890 2024-04-21T08:44:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.474+0000 7f8159fb6640 1 -- 172.21.15.7:0/1548630137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8154069250 con 0x7f8154101890 2024-04-21T08:44:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.476+0000 7f8150ff9640 1 -- 172.21.15.7:0/1548630137 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f814c005000 con 0x7f8154101890 2024-04-21T08:44:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.476+0000 7f8150ff9640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f813c0a3de0 0x7f813c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.476+0000 7f81537fe640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f813c0a3de0 0x7f813c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.476+0000 7f8150ff9640 1 -- 172.21.15.7:0/1548630137 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f814c0d13b0 con 0x7f8154101890 2024-04-21T08:44:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.477+0000 7f81537fe640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f813c0a3de0 0x7f813c0a62a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8144009b60 tx=0x7f8144005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.481+0000 7f8150ff9640 1 -- 172.21.15.7:0/1548630137 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f814c09d4f0 con 0x7f8154101890 2024-04-21T08:44:49.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.669+0000 7f8159fb6640 1 -- 172.21.15.7:0/1548630137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8154107680 con 0x7f8154101890 2024-04-21T08:44:49.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.669+0000 7f8150ff9640 1 -- 172.21.15.7:0/1548630137 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f814c09cdf0 con 0x7f8154101890 2024-04-21T08:44:49.669 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:44:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 -- 172.21.15.7:0/1548630137 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f813c0a3de0 msgr2=0x7f813c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f813c0a3de0 0x7f813c0a62a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8144009b60 tx=0x7f8144005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 -- 172.21.15.7:0/1548630137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8154101890 msgr2=0x7f81541a6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8154101890 0x7f81541a6ad0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f814c00eb10 tx=0x7f814c00c2d0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 -- 172.21.15.7:0/1548630137 shutdown_connections 2024-04-21T08:44:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8154101890 0x7f81541a6ad0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f815410c6d0 0x7f8154101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f813c0a3de0 0x7f813c0a62a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 --2- 172.21.15.7:0/1548630137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81541031c0 0x7f8154100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 -- 172.21.15.7:0/1548630137 >> 172.21.15.7:0/1548630137 conn(0x7f8154077f70 msgr2=0x7f81540fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 -- 172.21.15.7:0/1548630137 shutdown_connections 2024-04-21T08:44:49.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.671+0000 7f8159fb6640 1 -- 172.21.15.7:0/1548630137 wait complete. 2024-04-21T08:44:49.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.787+0000 7f40830d2640 1 -- 172.21.15.7:0/444713370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f407c1031c0 msgr2=0x7f407c1035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.787+0000 7f40830d2640 1 --2- 172.21.15.7:0/444713370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f407c1031c0 0x7f407c1035a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f406c009900 tx=0x7f406c02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.787+0000 7f40830d2640 1 -- 172.21.15.7:0/444713370 shutdown_connections 2024-04-21T08:44:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.787+0000 7f40830d2640 1 --2- 172.21.15.7:0/444713370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c10c500 0x7f407c10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.787+0000 7f40830d2640 1 --2- 172.21.15.7:0/444713370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f407c103ae0 0x7f407c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.787+0000 7f40830d2640 1 --2- 172.21.15.7:0/444713370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f407c1031c0 0x7f407c1035a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.787+0000 7f40830d2640 1 -- 172.21.15.7:0/444713370 >> 172.21.15.7:0/444713370 conn(0x7f407c078000 msgr2=0x7f407c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.787+0000 7f40830d2640 1 -- 172.21.15.7:0/444713370 shutdown_connections 2024-04-21T08:44:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f40830d2640 1 -- 172.21.15.7:0/444713370 wait complete. 2024-04-21T08:44:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f40830d2640 1 Processor -- start 2024-04-21T08:44:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f40830d2640 1 -- start start 2024-04-21T08:44:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f40830d2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f407c1031c0 0x7f407c100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f40830d2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f407c103ae0 0x7f407c101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f40830d2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c10c500 0x7f407c101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f40830d2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f407c112190 con 0x7f407c103ae0 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f40830d2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f407c112010 con 0x7f407c10c500 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f40830d2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f407c112310 con 0x7f407c1031c0 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f407bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f407c103ae0 0x7f407c101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f407bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f407c103ae0 0x7f407c101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47456/0 (socket says 172.21.15.7:47456) 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.788+0000 7f407bfff640 1 -- 172.21.15.7:0/4057045858 learned_addr learned my addr 172.21.15.7:0/4057045858 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f4080e6e640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f407c1031c0 0x7f407c100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f407bfff640 1 -- 172.21.15.7:0/4057045858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f407c1031c0 msgr2=0x7f407c100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f407bfff640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f407c1031c0 0x7f407c100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f408166f640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c10c500 0x7f407c101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f407bfff640 1 -- 172.21.15.7:0/4057045858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c10c500 msgr2=0x7f407c101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f407bfff640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c10c500 0x7f407c101890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f407bfff640 1 -- 172.21.15.7:0/4057045858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407c1a6c10 con 0x7f407c103ae0 2024-04-21T08:44:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f408166f640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c10c500 0x7f407c101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f407bfff640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f407c103ae0 0x7f407c101350 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9f0 tx=0x7f407000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f4079ffb640 1 -- 172.21.15.7:0/4057045858 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407000cd80 con 0x7f407c103ae0 2024-04-21T08:44:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f4079ffb640 1 -- 172.21.15.7:0/4057045858 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40700054d0 con 0x7f407c103ae0 2024-04-21T08:44:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f40830d2640 1 -- 172.21.15.7:0/4057045858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f407c1a6f00 con 0x7f407c103ae0 2024-04-21T08:44:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f4079ffb640 1 -- 172.21.15.7:0/4057045858 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070010690 con 0x7f407c103ae0 2024-04-21T08:44:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.789+0000 7f40830d2640 1 -- 172.21.15.7:0/4057045858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f407c1a7370 con 0x7f407c103ae0 2024-04-21T08:44:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.790+0000 7f40830d2640 1 -- 172.21.15.7:0/4057045858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f407c068e00 con 0x7f407c103ae0 2024-04-21T08:44:49.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.791+0000 7f4079ffb640 1 -- 172.21.15.7:0/4057045858 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f4070004e90 con 0x7f407c103ae0 2024-04-21T08:44:49.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.792+0000 7f4079ffb640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f40640a3db0 0x7f40640a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.792+0000 7f4079ffb640 1 -- 172.21.15.7:0/4057045858 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f40700d12d0 con 0x7f407c103ae0 2024-04-21T08:44:49.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.792+0000 7f4080e6e640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f40640a3db0 0x7f40640a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.793+0000 7f4080e6e640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f40640a3db0 0x7f40640a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f406c036040 tx=0x7f406c000bc0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.797+0000 7f4079ffb640 1 -- 172.21.15.7:0/4057045858 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f407009d410 con 0x7f407c103ae0 2024-04-21T08:44:49.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.983+0000 7f40830d2640 1 -- 172.21.15.7:0/4057045858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f407c1070f0 con 0x7f407c103ae0 2024-04-21T08:44:49.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.984+0000 7f4079ffb640 1 -- 172.21.15.7:0/4057045858 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f407009cd10 con 0x7f407c103ae0 2024-04-21T08:44:49.983 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 -- 172.21.15.7:0/4057045858 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f40640a3db0 msgr2=0x7f40640a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f40640a3db0 0x7f40640a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f406c036040 tx=0x7f406c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 -- 172.21.15.7:0/4057045858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f407c103ae0 msgr2=0x7f407c101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f407c103ae0 0x7f407c101350 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9f0 tx=0x7f407000edf0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 -- 172.21.15.7:0/4057045858 shutdown_connections 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f407c10c500 0x7f407c101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f407c103ae0 0x7f407c101350 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f40640a3db0 0x7f40640a6270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 --2- 172.21.15.7:0/4057045858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f407c1031c0 0x7f407c100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 -- 172.21.15.7:0/4057045858 >> 172.21.15.7:0/4057045858 conn(0x7f407c078000 msgr2=0x7f407c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 -- 172.21.15.7:0/4057045858 shutdown_connections 2024-04-21T08:44:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:49.986+0000 7f40830d2640 1 -- 172.21.15.7:0/4057045858 wait complete. 2024-04-21T08:44:49.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:49 smithi007 ceph-mon[28477]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:49.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:49 smithi007 ceph-mon[28477]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:49.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:49 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1889578094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:49 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3905456429' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.995 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:49 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1548630137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:50.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.099+0000 7f05fa3f4640 1 -- 172.21.15.7:0/1844490623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f4101a40 msgr2=0x7f05f410dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.099+0000 7f05fa3f4640 1 --2- 172.21.15.7:0/1844490623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f4101a40 0x7f05f410dca0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009ca0 tx=0x7f05ec02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.099+0000 7f05fa3f4640 1 -- 172.21.15.7:0/1844490623 shutdown_connections 2024-04-21T08:44:50.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.099+0000 7f05fa3f4640 1 --2- 172.21.15.7:0/1844490623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f4101a40 0x7f05f410dca0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.099+0000 7f05fa3f4640 1 --2- 172.21.15.7:0/1844490623 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f41010a0 0x7f05f4101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.099+0000 7f05fa3f4640 1 --2- 172.21.15.7:0/1844490623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05f4105dd0 0x7f05f41061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.099+0000 7f05fa3f4640 1 -- 172.21.15.7:0/1844490623 >> 172.21.15.7:0/1844490623 conn(0x7f05f4077c40 msgr2=0x7f05f4078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.099+0000 7f05fa3f4640 1 -- 172.21.15.7:0/1844490623 shutdown_connections 2024-04-21T08:44:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.099+0000 7f05fa3f4640 1 -- 172.21.15.7:0/1844490623 wait complete. 2024-04-21T08:44:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.100+0000 7f05fa3f4640 1 Processor -- start 2024-04-21T08:44:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.100+0000 7f05fa3f4640 1 -- start start 2024-04-21T08:44:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.100+0000 7f05fa3f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f41010a0 0x7f05f4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.100+0000 7f05fa3f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f4101a40 0x7f05f4198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.100+0000 7f05fa3f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05f4105dd0 0x7f05f419d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.100+0000 7f05fa3f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f05f4111180 con 0x7f05f41010a0 2024-04-21T08:44:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.100+0000 7f05fa3f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05f4111000 con 0x7f05f4101a40 2024-04-21T08:44:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.100+0000 7f05fa3f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05f4111300 con 0x7f05f4105dd0 2024-04-21T08:44:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.100+0000 7f05f3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f41010a0 0x7f05f4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.100+0000 7f05f3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f41010a0 0x7f05f4198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47460/0 (socket says 172.21.15.7:47460) 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.100+0000 7f05f3fff640 1 -- 172.21.15.7:0/2075375280 learned_addr learned my addr 172.21.15.7:0/2075375280 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05f8991640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05f4105dd0 0x7f05f419d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05f37fe640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f4101a40 0x7f05f4198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05f3fff640 1 -- 172.21.15.7:0/2075375280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05f4105dd0 msgr2=0x7f05f419d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05f3fff640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05f4105dd0 0x7f05f419d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05f3fff640 1 -- 172.21.15.7:0/2075375280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f4101a40 msgr2=0x7f05f4198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05f3fff640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f4101a40 0x7f05f4198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05f3fff640 1 -- 172.21.15.7:0/2075375280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f419d750 con 0x7f05f41010a0 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05f3fff640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f41010a0 0x7f05f4198760 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f05e400d770 tx=0x7f05e400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05f17fa640 1 -- 172.21.15.7:0/2075375280 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e4017070 con 0x7f05f41010a0 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05f17fa640 1 -- 172.21.15.7:0/2075375280 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05e40054d0 con 0x7f05f41010a0 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05fa3f4640 1 -- 172.21.15.7:0/2075375280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05f419da40 con 0x7f05f41010a0 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05f17fa640 1 -- 172.21.15.7:0/2075375280 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e4004c30 con 0x7f05f41010a0 2024-04-21T08:44:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.101+0000 7f05fa3f4640 1 -- 172.21.15.7:0/2075375280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05f41a52d0 con 0x7f05f41010a0 2024-04-21T08:44:50.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:49 smithi077 ceph-mon[31615]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:50.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:49 smithi077 ceph-mon[31615]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:50.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:49 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1889578094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:50.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:49 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3905456429' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:50.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:49 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1548630137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.102+0000 7f05fa3f4640 1 -- 172.21.15.7:0/2075375280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05f4102540 con 0x7f05f41010a0 2024-04-21T08:44:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.103+0000 7f05f17fa640 1 -- 172.21.15.7:0/2075375280 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f05e4004dd0 con 0x7f05f41010a0 2024-04-21T08:44:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.104+0000 7f05f17fa640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f05dc0a3db0 0x7f05dc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.104+0000 7f05f17fa640 1 -- 172.21.15.7:0/2075375280 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f05e40d0d60 con 0x7f05f41010a0 2024-04-21T08:44:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.104+0000 7f05f37fe640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f05dc0a3db0 0x7f05dc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.104+0000 7f05f37fe640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f05dc0a3db0 0x7f05dc0a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f05e8005fd0 tx=0x7f05e8005850 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.107+0000 7f05f17fa640 1 -- 172.21.15.7:0/2075375280 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f05e409ce40 con 0x7f05f41010a0 2024-04-21T08:44:50.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.294+0000 7f05fa3f4640 1 -- 172.21.15.7:0/2075375280 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05f41a54a0 con 0x7f05f41010a0 2024-04-21T08:44:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.295+0000 7f05f17fa640 1 -- 172.21.15.7:0/2075375280 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25085 (secure 0 0 0) 0x7f05e409c740 con 0x7f05f41010a0 2024-04-21T08:44:50.295 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:50.296 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":95,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:44:41.244735+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3451243741":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/117789107":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/4167672076":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/3630897301":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 -- 172.21.15.7:0/2075375280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f05dc0a3db0 msgr2=0x7f05dc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f05dc0a3db0 0x7f05dc0a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f05e8005fd0 tx=0x7f05e8005850 comp rx=0 tx=0).stop 2024-04-21T08:44:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 -- 172.21.15.7:0/2075375280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f41010a0 msgr2=0x7f05f4198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f41010a0 0x7f05f4198760 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f05e400d770 tx=0x7f05e400dc30 comp rx=0 tx=0).stop 2024-04-21T08:44:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 -- 172.21.15.7:0/2075375280 shutdown_connections 2024-04-21T08:44:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f05dc0a3db0 0x7f05dc0a6270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05f4105dd0 0x7f05f419d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f4101a40 0x7f05f4198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 --2- 172.21.15.7:0/2075375280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f41010a0 0x7f05f4198760 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 -- 172.21.15.7:0/2075375280 >> 172.21.15.7:0/2075375280 conn(0x7f05f4077c40 msgr2=0x7f05f4104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 -- 172.21.15.7:0/2075375280 shutdown_connections 2024-04-21T08:44:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.297+0000 7f05fa3f4640 1 -- 172.21.15.7:0/2075375280 wait complete. 2024-04-21T08:44:50.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:49 smithi016 ceph-mon[31259]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:50.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:49 smithi016 ceph-mon[31259]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:50.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:49 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1889578094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:50.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:49 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3905456429' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:50.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:49 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1548630137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:50.308 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:50.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.414+0000 7ff6e43f4640 1 -- 172.21.15.7:0/1822924904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc102a60 msgr2=0x7ff6dc102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.414+0000 7ff6e43f4640 1 --2- 172.21.15.7:0/1822924904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc102a60 0x7ff6dc102ec0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009930 tx=0x7ff6cc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.414+0000 7ff6e43f4640 1 -- 172.21.15.7:0/1822924904 shutdown_connections 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.414+0000 7ff6e43f4640 1 --2- 172.21.15.7:0/1822924904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6dc103400 0x7ff6dc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.414+0000 7ff6e43f4640 1 --2- 172.21.15.7:0/1822924904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc102a60 0x7ff6dc102ec0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.414+0000 7ff6e43f4640 1 --2- 172.21.15.7:0/1822924904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6dc101180 0x7ff6dc101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.414+0000 7ff6e43f4640 1 -- 172.21.15.7:0/1822924904 >> 172.21.15.7:0/1822924904 conn(0x7ff6dc077f70 msgr2=0x7ff6dc078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.414+0000 7ff6e43f4640 1 -- 172.21.15.7:0/1822924904 shutdown_connections 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.414+0000 7ff6e43f4640 1 -- 172.21.15.7:0/1822924904 wait complete. 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.414+0000 7ff6e43f4640 1 Processor -- start 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.414+0000 7ff6e43f4640 1 -- start start 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e43f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc101180 0x7ff6dc10e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e43f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6dc102a60 0x7ff6dc10edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e43f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6dc103400 0x7ff6dc10f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e43f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc112300 con 0x7ff6dc101180 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e43f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc112180 con 0x7ff6dc103400 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e43f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc112480 con 0x7ff6dc102a60 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e2190640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc101180 0x7ff6dc10e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e2190640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc101180 0x7ff6dc10e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47476/0 (socket says 172.21.15.7:47476) 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e2190640 1 -- 172.21.15.7:0/2485613652 learned_addr learned my addr 172.21.15.7:0/2485613652 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e198f640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6dc102a60 0x7ff6dc10edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e2991640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6dc103400 0x7ff6dc10f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e2190640 1 -- 172.21.15.7:0/2485613652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6dc102a60 msgr2=0x7ff6dc10edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e2190640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6dc102a60 0x7ff6dc10edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e2190640 1 -- 172.21.15.7:0/2485613652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6dc103400 msgr2=0x7ff6dc10f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e2190640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6dc103400 0x7ff6dc10f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e2190640 1 -- 172.21.15.7:0/2485613652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc1a6c60 con 0x7ff6dc101180 2024-04-21T08:44:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e2991640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6dc103400 0x7ff6dc10f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e2190640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc101180 0x7ff6dc10e890 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800d950 tx=0x7ff6d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6d37fe640 1 -- 172.21.15.7:0/2485613652 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8017070 con 0x7ff6dc101180 2024-04-21T08:44:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6d37fe640 1 -- 172.21.15.7:0/2485613652 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6d8009c30 con 0x7ff6dc101180 2024-04-21T08:44:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6e43f4640 1 -- 172.21.15.7:0/2485613652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc1a6f50 con 0x7ff6dc101180 2024-04-21T08:44:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.415+0000 7ff6d37fe640 1 -- 172.21.15.7:0/2485613652 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d800f620 con 0x7ff6dc101180 2024-04-21T08:44:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.416+0000 7ff6e43f4640 1 -- 172.21.15.7:0/2485613652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6dc1a7410 con 0x7ff6dc101180 2024-04-21T08:44:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.416+0000 7ff6e43f4640 1 -- 172.21.15.7:0/2485613652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6dc069250 con 0x7ff6dc101180 2024-04-21T08:44:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.418+0000 7ff6d37fe640 1 -- 172.21.15.7:0/2485613652 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7ff6d8010460 con 0x7ff6dc101180 2024-04-21T08:44:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.418+0000 7ff6d37fe640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff6c80a3e30 0x7ff6c80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.418+0000 7ff6e198f640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff6c80a3e30 0x7ff6c80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.418+0000 7ff6d37fe640 1 -- 172.21.15.7:0/2485613652 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7ff6d80d0930 con 0x7ff6dc101180 2024-04-21T08:44:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.419+0000 7ff6e198f640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff6c80a3e30 0x7ff6c80a62f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc005ab0 tx=0x7ff6cc005fd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.423+0000 7ff6d37fe640 1 -- 172.21.15.7:0/2485613652 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6d809ca70 con 0x7ff6dc101180 2024-04-21T08:44:50.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.611+0000 7ff6e43f4640 1 -- 172.21.15.7:0/2485613652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff6dc109480 con 0x7ff6dc101180 2024-04-21T08:44:50.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.611+0000 7ff6d37fe640 1 -- 172.21.15.7:0/2485613652 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7ff6d8009de0 con 0x7ff6dc101180 2024-04-21T08:44:50.611 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 -- 172.21.15.7:0/2485613652 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff6c80a3e30 msgr2=0x7ff6c80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff6c80a3e30 0x7ff6c80a62f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc005ab0 tx=0x7ff6cc005fd0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 -- 172.21.15.7:0/2485613652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc101180 msgr2=0x7ff6dc10e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc101180 0x7ff6dc10e890 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800d950 tx=0x7ff6d800de10 comp rx=0 tx=0).stop 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 -- 172.21.15.7:0/2485613652 shutdown_connections 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6dc103400 0x7ff6dc10f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff6c80a3e30 0x7ff6c80a62f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6dc102a60 0x7ff6dc10edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 --2- 172.21.15.7:0/2485613652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc101180 0x7ff6dc10e890 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 -- 172.21.15.7:0/2485613652 >> 172.21.15.7:0/2485613652 conn(0x7ff6dc077f70 msgr2=0x7ff6dc104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 -- 172.21.15.7:0/2485613652 shutdown_connections 2024-04-21T08:44:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.613+0000 7ff6e43f4640 1 -- 172.21.15.7:0/2485613652 wait complete. 2024-04-21T08:44:50.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.729+0000 7fee06fd5640 1 -- 172.21.15.7:0/222920298 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee001044f0 msgr2=0x7fee0010eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.729+0000 7fee06fd5640 1 --2- 172.21.15.7:0/222920298 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee001044f0 0x7fee0010eef0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009cd0 tx=0x7fedf802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.729+0000 7fee06fd5640 1 -- 172.21.15.7:0/222920298 shutdown_connections 2024-04-21T08:44:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.729+0000 7fee06fd5640 1 --2- 172.21.15.7:0/222920298 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee001044f0 0x7fee0010eef0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.729+0000 7fee06fd5640 1 --2- 172.21.15.7:0/222920298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee00103b50 0x7fee00103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.729+0000 7fee06fd5640 1 --2- 172.21.15.7:0/222920298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee00102270 0x7fee00102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.729+0000 7fee06fd5640 1 -- 172.21.15.7:0/222920298 >> 172.21.15.7:0/222920298 conn(0x7fee000fda10 msgr2=0x7fee000ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.729+0000 7fee06fd5640 1 -- 172.21.15.7:0/222920298 shutdown_connections 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.729+0000 7fee06fd5640 1 -- 172.21.15.7:0/222920298 wait complete. 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee06fd5640 1 Processor -- start 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee06fd5640 1 -- start start 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee06fd5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee00102270 0x7fee0019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee06fd5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee00103b50 0x7fee0019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee06fd5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee001044f0 0x7fee001985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee06fd5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fee00112290 con 0x7fee00102270 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee06fd5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee00112110 con 0x7fee00103b50 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee06fd5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fee00112410 con 0x7fee001044f0 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee04d71640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee00102270 0x7fee0019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee04d71640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee00102270 0x7fee0019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47496/0 (socket says 172.21.15.7:47496) 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee04d71640 1 -- 172.21.15.7:0/2325493252 learned_addr learned my addr 172.21.15.7:0/2325493252 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fedfffff640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee00103b50 0x7fee0019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee05572640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee001044f0 0x7fee001985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee04d71640 1 -- 172.21.15.7:0/2325493252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee001044f0 msgr2=0x7fee001985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee04d71640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee001044f0 0x7fee001985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee04d71640 1 -- 172.21.15.7:0/2325493252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee00103b50 msgr2=0x7fee0019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee04d71640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee00103b50 0x7fee0019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.730+0000 7fee04d71640 1 -- 172.21.15.7:0/2325493252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee00198d70 con 0x7fee00102270 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.731+0000 7fee04d71640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee00102270 0x7fee0019e3f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fedf400d950 tx=0x7fedf400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.731+0000 7fedfdffb640 1 -- 172.21.15.7:0/2325493252 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4017030 con 0x7fee00102270 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.731+0000 7fee06fd5640 1 -- 172.21.15.7:0/2325493252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee00199060 con 0x7fee00102270 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.731+0000 7fedfdffb640 1 -- 172.21.15.7:0/2325493252 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedf4004020 con 0x7fee00102270 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.731+0000 7fedfdffb640 1 -- 172.21.15.7:0/2325493252 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf40047a0 con 0x7fee00102270 2024-04-21T08:44:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.731+0000 7fee06fd5640 1 -- 172.21.15.7:0/2325493252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee001a52b0 con 0x7fee00102270 2024-04-21T08:44:50.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.732+0000 7fee06fd5640 1 -- 172.21.15.7:0/2325493252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee00069250 con 0x7fee00102270 2024-04-21T08:44:50.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.733+0000 7fedfdffb640 1 -- 172.21.15.7:0/2325493252 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7fedf4009a30 con 0x7fee00102270 2024-04-21T08:44:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.734+0000 7fedfdffb640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fedf00a3db0 0x7fedf00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.734+0000 7fedfffff640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fedf00a3db0 0x7fedf00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.734+0000 7fedfdffb640 1 -- 172.21.15.7:0/2325493252 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fedf40d0850 con 0x7fee00102270 2024-04-21T08:44:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.734+0000 7fedfffff640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fedf00a3db0 0x7fedf00a6270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fee001000f0 tx=0x7fedec00b390 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.738+0000 7fedfdffb640 1 -- 172.21.15.7:0/2325493252 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fedf409c990 con 0x7fee00102270 2024-04-21T08:44:50.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:50 smithi007 ceph-mon[28477]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 93 KiB/s wr, 43 op/s 2024-04-21T08:44:50.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:50 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4057045858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:50.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:50 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2075375280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:50.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:50 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2485613652' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.928+0000 7fee06fd5640 1 -- 172.21.15.7:0/2325493252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fee00199ad0 con 0x7fee00102270 2024-04-21T08:44:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.929+0000 7fedfdffb640 1 -- 172.21.15.7:0/2325493252 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7fedf409c290 con 0x7fee00102270 2024-04-21T08:44:50.928 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 -- 172.21.15.7:0/2325493252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fedf00a3db0 msgr2=0x7fedf00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fedf00a3db0 0x7fedf00a6270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fee001000f0 tx=0x7fedec00b390 comp rx=0 tx=0).stop 2024-04-21T08:44:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 -- 172.21.15.7:0/2325493252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee00102270 msgr2=0x7fee0019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee00102270 0x7fee0019e3f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fedf400d950 tx=0x7fedf400de10 comp rx=0 tx=0).stop 2024-04-21T08:44:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 -- 172.21.15.7:0/2325493252 shutdown_connections 2024-04-21T08:44:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fedf00a3db0 0x7fedf00a6270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee001044f0 0x7fee001985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee00103b50 0x7fee0019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 --2- 172.21.15.7:0/2325493252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee00102270 0x7fee0019e3f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 -- 172.21.15.7:0/2325493252 >> 172.21.15.7:0/2325493252 conn(0x7fee000fda10 msgr2=0x7fee000fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 -- 172.21.15.7:0/2325493252 shutdown_connections 2024-04-21T08:44:50.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:50.931+0000 7fee06fd5640 1 -- 172.21.15.7:0/2325493252 wait complete. 2024-04-21T08:44:50.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:51.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.047+0000 7f21fd57e640 1 -- 172.21.15.7:0/3240184047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f81044f0 msgr2=0x7f21f810eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.047+0000 7f21fd57e640 1 --2- 172.21.15.7:0/3240184047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f81044f0 0x7f21f810eef0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f21f0009cd0 tx=0x7f21f002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.047+0000 7f21fd57e640 1 -- 172.21.15.7:0/3240184047 shutdown_connections 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.047+0000 7f21fd57e640 1 --2- 172.21.15.7:0/3240184047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f81044f0 0x7f21f810eef0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.047+0000 7f21fd57e640 1 --2- 172.21.15.7:0/3240184047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f8103b50 0x7f21f8103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.047+0000 7f21fd57e640 1 --2- 172.21.15.7:0/3240184047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21f8102270 0x7f21f8102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.047+0000 7f21fd57e640 1 -- 172.21.15.7:0/3240184047 >> 172.21.15.7:0/3240184047 conn(0x7f21f80fda10 msgr2=0x7f21f80ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.047+0000 7f21fd57e640 1 -- 172.21.15.7:0/3240184047 shutdown_connections 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.047+0000 7f21fd57e640 1 -- 172.21.15.7:0/3240184047 wait complete. 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.047+0000 7f21fd57e640 1 Processor -- start 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21fd57e640 1 -- start start 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21fd57e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f8102270 0x7f21f81986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21fd57e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21f8103b50 0x7f21f8198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21fd57e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f81044f0 0x7f21f819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21fd57e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f21f8112270 con 0x7f21f81044f0 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21fd57e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21f81120f0 con 0x7f21f8102270 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21fd57e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f21f81123f0 con 0x7f21f8103b50 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21f77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f81044f0 0x7f21f819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21f77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f81044f0 0x7f21f819cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47510/0 (socket says 172.21.15.7:47510) 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21f77fe640 1 -- 172.21.15.7:0/3206156897 learned_addr learned my addr 172.21.15.7:0/3206156897 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21f6ffd640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f8102270 0x7f21f81986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21f67fc640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21f8103b50 0x7f21f8198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21f77fe640 1 -- 172.21.15.7:0/3206156897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21f8103b50 msgr2=0x7f21f8198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21f77fe640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21f8103b50 0x7f21f8198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21f77fe640 1 -- 172.21.15.7:0/3206156897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f8102270 msgr2=0x7f21f81986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21f77fe640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f8102270 0x7f21f81986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21f77fe640 1 -- 172.21.15.7:0/3206156897 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f819d6a0 con 0x7f21f81044f0 2024-04-21T08:44:51.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.048+0000 7f21f77fe640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f81044f0 0x7f21f819cf70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f21f0033000 tx=0x7f21f00346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:51.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.049+0000 7f21effff640 1 -- 172.21.15.7:0/3206156897 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f0041020 con 0x7f21f81044f0 2024-04-21T08:44:51.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.049+0000 7f21effff640 1 -- 172.21.15.7:0/3206156897 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21f0032970 con 0x7f21f81044f0 2024-04-21T08:44:51.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.049+0000 7f21fd57e640 1 -- 172.21.15.7:0/3206156897 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f819d930 con 0x7f21f81044f0 2024-04-21T08:44:51.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.049+0000 7f21effff640 1 -- 172.21.15.7:0/3206156897 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f0039040 con 0x7f21f81044f0 2024-04-21T08:44:51.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.049+0000 7f21fd57e640 1 -- 172.21.15.7:0/3206156897 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f81a5220 con 0x7f21f81044f0 2024-04-21T08:44:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.050+0000 7f21fd57e640 1 -- 172.21.15.7:0/3206156897 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f8069250 con 0x7f21f81044f0 2024-04-21T08:44:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.051+0000 7f21effff640 1 -- 172.21.15.7:0/3206156897 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f21f0044020 con 0x7f21f81044f0 2024-04-21T08:44:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.051+0000 7f21effff640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21e40a3db0 0x7f21e40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.052+0000 7f21f6ffd640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21e40a3db0 0x7f21e40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.052+0000 7f21effff640 1 -- 172.21.15.7:0/3206156897 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f21f00f2f40 con 0x7f21f81044f0 2024-04-21T08:44:51.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.052+0000 7f21f6ffd640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21e40a3db0 0x7f21e40a6270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f21e8005e10 tx=0x7f21e8005da0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:51.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.055+0000 7f21effff640 1 -- 172.21.15.7:0/3206156897 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21f004b020 con 0x7f21f81044f0 2024-04-21T08:44:51.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:50 smithi077 ceph-mon[31615]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 93 KiB/s wr, 43 op/s 2024-04-21T08:44:51.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:50 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4057045858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:51.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:50 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2075375280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:51.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:50 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2485613652' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.240+0000 7f21fd57e640 1 -- 172.21.15.7:0/3206156897 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f21f810a540 con 0x7f21f81044f0 2024-04-21T08:44:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.241+0000 7f21effff640 1 -- 172.21.15.7:0/3206156897 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f21f00bf250 con 0x7f21f81044f0 2024-04-21T08:44:51.240 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:44:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.242+0000 7f21fd57e640 1 -- 172.21.15.7:0/3206156897 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21e40a3db0 msgr2=0x7f21e40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.242+0000 7f21fd57e640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21e40a3db0 0x7f21e40a6270 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f21e8005e10 tx=0x7f21e8005da0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.242+0000 7f21fd57e640 1 -- 172.21.15.7:0/3206156897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f81044f0 msgr2=0x7f21f819cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.242+0000 7f21fd57e640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f81044f0 0x7f21f819cf70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f21f0033000 tx=0x7f21f00346e0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.243+0000 7f21fd57e640 1 -- 172.21.15.7:0/3206156897 shutdown_connections 2024-04-21T08:44:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.243+0000 7f21fd57e640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f81044f0 0x7f21f819cf70 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.243+0000 7f21fd57e640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21e40a3db0 0x7f21e40a6270 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.243+0000 7f21fd57e640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21f8103b50 0x7f21f8198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.243+0000 7f21fd57e640 1 --2- 172.21.15.7:0/3206156897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21f8102270 0x7f21f81986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.243+0000 7f21fd57e640 1 -- 172.21.15.7:0/3206156897 >> 172.21.15.7:0/3206156897 conn(0x7f21f80fda10 msgr2=0x7f21f80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.243+0000 7f21fd57e640 1 -- 172.21.15.7:0/3206156897 shutdown_connections 2024-04-21T08:44:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.243+0000 7f21fd57e640 1 -- 172.21.15.7:0/3206156897 wait complete. 2024-04-21T08:44:51.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:50 smithi016 ceph-mon[31259]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 93 KiB/s wr, 43 op/s 2024-04-21T08:44:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:50 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4057045858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:50 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2075375280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:50 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2485613652' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.359+0000 7f6da5470640 1 -- 172.21.15.7:0/1443834670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6da010c6d0 msgr2=0x7f6da010eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.359+0000 7f6da5470640 1 --2- 172.21.15.7:0/1443834670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6da010c6d0 0x7f6da010eb60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009ca0 tx=0x7f6d9802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.359+0000 7f6da5470640 1 -- 172.21.15.7:0/1443834670 shutdown_connections 2024-04-21T08:44:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.359+0000 7f6da5470640 1 --2- 172.21.15.7:0/1443834670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6da010c6d0 0x7f6da010eb60 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.359+0000 7f6da5470640 1 --2- 172.21.15.7:0/1443834670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da0103b70 0x7f6da010bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.359+0000 7f6da5470640 1 --2- 172.21.15.7:0/1443834670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da01031c0 0x7f6da01035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.359+0000 7f6da5470640 1 -- 172.21.15.7:0/1443834670 >> 172.21.15.7:0/1443834670 conn(0x7f6da0077f70 msgr2=0x7f6da0078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.359+0000 7f6da5470640 1 -- 172.21.15.7:0/1443834670 shutdown_connections 2024-04-21T08:44:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.359+0000 7f6da5470640 1 -- 172.21.15.7:0/1443834670 wait complete. 2024-04-21T08:44:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.359+0000 7f6da5470640 1 Processor -- start 2024-04-21T08:44:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6da5470640 1 -- start start 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6da5470640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da01031c0 0x7f6da0100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6da5470640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da010c6d0 0x7f6da0101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6da5470640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6da0101890 0x7f6da01a6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6da5470640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6da01123a0 con 0x7f6da0101890 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6da5470640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6da0112220 con 0x7f6da010c6d0 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6da5470640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6da0112520 con 0x7f6da01031c0 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6da0101890 0x7f6da01a6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6da0101890 0x7f6da01a6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47530/0 (socket says 172.21.15.7:47530) 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9f7fe640 1 -- 172.21.15.7:0/2622639273 learned_addr learned my addr 172.21.15.7:0/2622639273 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9effd640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da01031c0 0x7f6da0100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9e7fc640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da010c6d0 0x7f6da0101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9f7fe640 1 -- 172.21.15.7:0/2622639273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da01031c0 msgr2=0x7f6da0100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9f7fe640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da01031c0 0x7f6da0100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9f7fe640 1 -- 172.21.15.7:0/2622639273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da010c6d0 msgr2=0x7f6da0101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9f7fe640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da010c6d0 0x7f6da0101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9f7fe640 1 -- 172.21.15.7:0/2622639273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da01a71d0 con 0x7f6da0101890 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9e7fc640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da010c6d0 0x7f6da0101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9effd640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da01031c0 0x7f6da0100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d9f7fe640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6da0101890 0x7f6da01a6ad0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6d98005e30 tx=0x7f6d980346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.360+0000 7f6d87fff640 1 -- 172.21.15.7:0/2622639273 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d98041020 con 0x7f6da0101890 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.361+0000 7f6d87fff640 1 -- 172.21.15.7:0/2622639273 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d98032970 con 0x7f6da0101890 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.361+0000 7f6da5470640 1 -- 172.21.15.7:0/2622639273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da01a7460 con 0x7f6da0101890 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.361+0000 7f6d87fff640 1 -- 172.21.15.7:0/2622639273 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d98039040 con 0x7f6da0101890 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.361+0000 7f6da5470640 1 -- 172.21.15.7:0/2622639273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da01a7950 con 0x7f6da0101890 2024-04-21T08:44:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.362+0000 7f6da5470640 1 -- 172.21.15.7:0/2622639273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da0069250 con 0x7f6da0101890 2024-04-21T08:44:51.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.363+0000 7f6d87fff640 1 -- 172.21.15.7:0/2622639273 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f6d98044020 con 0x7f6da0101890 2024-04-21T08:44:51.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.363+0000 7f6d87fff640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6d880a3de0 0x7f6d880a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.363+0000 7f6d9effd640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6d880a3de0 0x7f6d880a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.363+0000 7f6d87fff640 1 -- 172.21.15.7:0/2622639273 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f6d980f2950 con 0x7f6da0101890 2024-04-21T08:44:51.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.364+0000 7f6d9effd640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6d880a3de0 0x7f6d880a62a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6d90009c20 tx=0x7f6d90005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:51.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.368+0000 7f6d87fff640 1 -- 172.21.15.7:0/2622639273 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d9804b020 con 0x7f6da0101890 2024-04-21T08:44:51.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.553+0000 7f6da5470640 1 -- 172.21.15.7:0/2622639273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6da0107680 con 0x7f6da0101890 2024-04-21T08:44:51.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.553+0000 7f6d87fff640 1 -- 172.21.15.7:0/2622639273 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f6d980beba0 con 0x7f6da0101890 2024-04-21T08:44:51.553 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:51.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 -- 172.21.15.7:0/2622639273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6d880a3de0 msgr2=0x7f6d880a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6d880a3de0 0x7f6d880a62a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6d90009c20 tx=0x7f6d90005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:51.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 -- 172.21.15.7:0/2622639273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6da0101890 msgr2=0x7f6da01a6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6da0101890 0x7f6da01a6ad0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6d98005e30 tx=0x7f6d980346d0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 -- 172.21.15.7:0/2622639273 shutdown_connections 2024-04-21T08:44:51.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6da0101890 0x7f6da01a6ad0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6da010c6d0 0x7f6da0101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6d880a3de0 0x7f6d880a62a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 --2- 172.21.15.7:0/2622639273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6da01031c0 0x7f6da0100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 -- 172.21.15.7:0/2622639273 >> 172.21.15.7:0/2622639273 conn(0x7f6da0077f70 msgr2=0x7f6da00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:51.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 -- 172.21.15.7:0/2622639273 shutdown_connections 2024-04-21T08:44:51.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.555+0000 7f6da5470640 1 -- 172.21.15.7:0/2622639273 wait complete. 2024-04-21T08:44:51.566 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-21T08:44:51.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:44:51.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.668+0000 7fcccfcb3640 1 -- 172.21.15.7:0/1837721804 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc8102b50 msgr2=0x7fccc8106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.668+0000 7fcccfcb3640 1 --2- 172.21.15.7:0/1837721804 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc8102b50 0x7fccc8106b90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009900 tx=0x7fccbc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:51.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 -- 172.21.15.7:0/1837721804 shutdown_connections 2024-04-21T08:44:51.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 --2- 172.21.15.7:0/1837721804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc8107270 0x7fccc810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 --2- 172.21.15.7:0/1837721804 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc8102b50 0x7fccc8106b90 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 --2- 172.21.15.7:0/1837721804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc81021a0 0x7fccc8102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 -- 172.21.15.7:0/1837721804 >> 172.21.15.7:0/1837721804 conn(0x7fccc80793e0 msgr2=0x7fccc80797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:51.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 -- 172.21.15.7:0/1837721804 shutdown_connections 2024-04-21T08:44:51.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 -- 172.21.15.7:0/1837721804 wait complete. 2024-04-21T08:44:51.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 Processor -- start 2024-04-21T08:44:51.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 -- start start 2024-04-21T08:44:51.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc81021a0 0x7fccc8074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc8107270 0x7fccc8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc8074cd0 0x7fccc81a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fccc8112200 con 0x7fccc81021a0 2024-04-21T08:44:51.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fccc8112080 con 0x7fccc8074cd0 2024-04-21T08:44:51.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccfcb3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fccc8112380 con 0x7fccc8107270 2024-04-21T08:44:51.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.669+0000 7fcccda4f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc81021a0 0x7fccc8074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccda4f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc81021a0 0x7fccc8074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47548/0 (socket says 172.21.15.7:47548) 2024-04-21T08:44:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccda4f640 1 -- 172.21.15.7:0/2706268525 learned_addr learned my addr 172.21.15.7:0/2706268525 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccd24e640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc8107270 0x7fccc8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fccce250640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc8074cd0 0x7fccc81a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccda4f640 1 -- 172.21.15.7:0/2706268525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc8107270 msgr2=0x7fccc8074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccda4f640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc8107270 0x7fccc8074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccda4f640 1 -- 172.21.15.7:0/2706268525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc8074cd0 msgr2=0x7fccc81a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccda4f640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc8074cd0 0x7fccc81a68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccda4f640 1 -- 172.21.15.7:0/2706268525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc81a6ff0 con 0x7fccc81021a0 2024-04-21T08:44:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fccce250640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc8074cd0 0x7fccc81a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccd24e640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc8107270 0x7fccc8074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:51.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccda4f640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc81021a0 0x7fccc8074250 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fccc000d770 tx=0x7fccc000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:51.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fccc6ffd640 1 -- 172.21.15.7:0/2706268525 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0017070 con 0x7fccc81021a0 2024-04-21T08:44:51.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fccc6ffd640 1 -- 172.21.15.7:0/2706268525 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccc00054d0 con 0x7fccc81021a0 2024-04-21T08:44:51.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccfcb3640 1 -- 172.21.15.7:0/2706268525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc81a72e0 con 0x7fccc81021a0 2024-04-21T08:44:51.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fccc6ffd640 1 -- 172.21.15.7:0/2706268525 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0004c30 con 0x7fccc81021a0 2024-04-21T08:44:51.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.670+0000 7fcccfcb3640 1 -- 172.21.15.7:0/2706268525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc81a77e0 con 0x7fccc81021a0 2024-04-21T08:44:51.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.671+0000 7fcccfcb3640 1 -- 172.21.15.7:0/2706268525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccc8069250 con 0x7fccc81021a0 2024-04-21T08:44:51.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.672+0000 7fccc6ffd640 1 -- 172.21.15.7:0/2706268525 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7fccc0004dd0 con 0x7fccc81021a0 2024-04-21T08:44:51.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.673+0000 7fccc6ffd640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fccb00a3de0 0x7fccb00a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:51.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.673+0000 7fccc6ffd640 1 -- 172.21.15.7:0/2706268525 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fccc00d0900 con 0x7fccc81021a0 2024-04-21T08:44:51.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.673+0000 7fcccd24e640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fccb00a3de0 0x7fccb00a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:51.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.673+0000 7fcccd24e640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fccb00a3de0 0x7fccb00a62a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fccbc036040 tx=0x7fccbc005fd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:51.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.678+0000 7fccc6ffd640 1 -- 172.21.15.7:0/2706268525 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fccc009ca40 con 0x7fccc81021a0 2024-04-21T08:44:51.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.862+0000 7fcccfcb3640 1 -- 172.21.15.7:0/2706268525 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fccc810a510 con 0x7fccb00a3de0 2024-04-21T08:44:51.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.872+0000 7fccc6ffd640 1 -- 172.21.15.7:0/2706268525 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339632 (secure 0 0 0) 0x7fccbc004270 con 0x7fccb00a3de0 2024-04-21T08:44:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:51.877 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-21T08:44:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 -- 172.21.15.7:0/2706268525 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fccb00a3de0 msgr2=0x7fccb00a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fccb00a3de0 0x7fccb00a62a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fccbc036040 tx=0x7fccbc005fd0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 -- 172.21.15.7:0/2706268525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc81021a0 msgr2=0x7fccc8074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc81021a0 0x7fccc8074250 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fccc000d770 tx=0x7fccc000dc30 comp rx=0 tx=0).stop 2024-04-21T08:44:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 -- 172.21.15.7:0/2706268525 shutdown_connections 2024-04-21T08:44:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc8074cd0 0x7fccc81a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fccb00a3de0 0x7fccb00a62a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc8107270 0x7fccc8074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 --2- 172.21.15.7:0/2706268525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc81021a0 0x7fccc8074250 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:51.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 -- 172.21.15.7:0/2706268525 >> 172.21.15.7:0/2706268525 conn(0x7fccc80793e0 msgr2=0x7fccc80ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:51.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 -- 172.21.15.7:0/2706268525 shutdown_connections 2024-04-21T08:44:51.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:51.876+0000 7fcccfcb3640 1 -- 172.21.15.7:0/2706268525 wait complete. 2024-04-21T08:44:51.886 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-21T08:44:51.264294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1382916,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":709,"num_read_kb":717,"num_write":468,"num_write_kb":5003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":844,"ondisk_log_size":844,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":403456,"kb_used_data":12680,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124538368,"statfs":{"total":1151940427776,"available":1151527288832,"internally_reserved":0,"allocated":12984320,"data_stored":9930710,"data_compressed":544941,"data_compressed_allocated":737280,"data_compressed_original":4325376,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":223,"num_read_kb":318,"num_write":84,"num_write_kb":652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.991246"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375541+0000","last_change":"2024-04-21T08:42:17.545701+0000","last_active":"2024-04-21T08:44:41.375541+0000","last_peered":"2024-04-21T08:44:41.375541+0000","last_clean":"2024-04-21T08:44:41.375541+0000","last_became_active":"2024-04-21T08:42:17.545157+0000","last_became_peered":"2024-04-21T08:42:17.545157+0000","last_unstale":"2024-04-21T08:44:41.375541+0000","last_undegraded":"2024-04-21T08:44:41.375541+0000","last_fullsized":"2024-04-21T08:44:41.375541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:05:41.952879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418165+0000","last_change":"2024-04-21T08:42:17.541153+0000","last_active":"2024-04-21T08:44:41.418165+0000","last_peered":"2024-04-21T08:44:41.418165+0000","last_clean":"2024-04-21T08:44:41.418165+0000","last_became_active":"2024-04-21T08:42:17.540771+0000","last_became_peered":"2024-04-21T08:42:17.540771+0000","last_unstale":"2024-04-21T08:44:41.418165+0000","last_undegraded":"2024-04-21T08:44:41.418165+0000","last_fullsized":"2024-04-21T08:44:41.418165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:35:06.077367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417993+0000","last_change":"2024-04-21T08:42:17.545525+0000","last_active":"2024-04-21T08:44:41.417993+0000","last_peered":"2024-04-21T08:44:41.417993+0000","last_clean":"2024-04-21T08:44:41.417993+0000","last_became_active":"2024-04-21T08:42:17.545168+0000","last_became_peered":"2024-04-21T08:42:17.545168+0000","last_unstale":"2024-04-21T08:44:41.417993+0000","last_undegraded":"2024-04-21T08:44:41.417993+0000","last_fullsized":"2024-04-21T08:44:41.417993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:51.080385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251479+0000","last_change":"2024-04-21T08:42:17.543917+0000","last_active":"2024-04-21T08:44:41.251479+0000","last_peered":"2024-04-21T08:44:41.251479+0000","last_clean":"2024-04-21T08:44:41.251479+0000","last_became_active":"2024-04-21T08:42:17.543513+0000","last_became_peered":"2024-04-21T08:42:17.543513+0000","last_unstale":"2024-04-21T08:44:41.251479+0000","last_undegraded":"2024-04-21T08:44:41.251479+0000","last_fullsized":"2024-04-21T08:44:41.251479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:23.058600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250648+0000","last_change":"2024-04-21T08:42:17.549737+0000","last_active":"2024-04-21T08:44:41.250648+0000","last_peered":"2024-04-21T08:44:41.250648+0000","last_clean":"2024-04-21T08:44:41.250648+0000","last_became_active":"2024-04-21T08:42:17.549396+0000","last_became_peered":"2024-04-21T08:42:17.549396+0000","last_unstale":"2024-04-21T08:44:41.250648+0000","last_undegraded":"2024-04-21T08:44:41.250648+0000","last_fullsized":"2024-04-21T08:44:41.250648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:01:51.879960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368365+0000","last_change":"2024-04-21T08:43:10.792321+0000","last_active":"2024-04-21T08:44:41.368365+0000","last_peered":"2024-04-21T08:44:41.368365+0000","last_clean":"2024-04-21T08:44:41.368365+0000","last_became_active":"2024-04-21T08:43:10.791979+0000","last_became_peered":"2024-04-21T08:43:10.791979+0000","last_unstale":"2024-04-21T08:44:41.368365+0000","last_undegraded":"2024-04-21T08:44:41.368365+0000","last_fullsized":"2024-04-21T08:44:41.368365+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:11:16.056778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374960+0000","last_change":"2024-04-21T08:42:17.541172+0000","last_active":"2024-04-21T08:44:41.374960+0000","last_peered":"2024-04-21T08:44:41.374960+0000","last_clean":"2024-04-21T08:44:41.374960+0000","last_became_active":"2024-04-21T08:42:17.540829+0000","last_became_peered":"2024-04-21T08:42:17.540829+0000","last_unstale":"2024-04-21T08:44:41.374960+0000","last_undegraded":"2024-04-21T08:44:41.374960+0000","last_fullsized":"2024-04-21T08:44:41.374960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:16:12.633980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249835+0000","last_change":"2024-04-21T08:42:17.549744+0000","last_active":"2024-04-21T08:44:41.249835+0000","last_peered":"2024-04-21T08:44:41.249835+0000","last_clean":"2024-04-21T08:44:41.249835+0000","last_became_active":"2024-04-21T08:42:17.549378+0000","last_became_peered":"2024-04-21T08:42:17.549378+0000","last_unstale":"2024-04-21T08:44:41.249835+0000","last_undegraded":"2024-04-21T08:44:41.249835+0000","last_fullsized":"2024-04-21T08:44:41.249835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:03:07.702143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375819+0000","last_change":"2024-04-21T08:42:17.548675+0000","last_active":"2024-04-21T08:44:41.375819+0000","last_peered":"2024-04-21T08:44:41.375819+0000","last_clean":"2024-04-21T08:44:41.375819+0000","last_became_active":"2024-04-21T08:42:17.547809+0000","last_became_peered":"2024-04-21T08:42:17.547809+0000","last_unstale":"2024-04-21T08:44:41.375819+0000","last_undegraded":"2024-04-21T08:44:41.375819+0000","last_fullsized":"2024-04-21T08:44:41.375819+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:23.343473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368808+0000","last_change":"2024-04-21T08:42:17.551012+0000","last_active":"2024-04-21T08:44:41.368808+0000","last_peered":"2024-04-21T08:44:41.368808+0000","last_clean":"2024-04-21T08:44:41.368808+0000","last_became_active":"2024-04-21T08:42:17.550627+0000","last_became_peered":"2024-04-21T08:42:17.550627+0000","last_unstale":"2024-04-21T08:44:41.368808+0000","last_undegraded":"2024-04-21T08:44:41.368808+0000","last_fullsized":"2024-04-21T08:44:41.368808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:57.501588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375100+0000","last_change":"2024-04-21T08:42:17.546192+0000","last_active":"2024-04-21T08:44:41.375100+0000","last_peered":"2024-04-21T08:44:41.375100+0000","last_clean":"2024-04-21T08:44:41.375100+0000","last_became_active":"2024-04-21T08:42:17.545808+0000","last_became_peered":"2024-04-21T08:42:17.545808+0000","last_unstale":"2024-04-21T08:44:41.375100+0000","last_undegraded":"2024-04-21T08:44:41.375100+0000","last_fullsized":"2024-04-21T08:44:41.375100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:54.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251368+0000","last_change":"2024-04-21T08:42:17.553351+0000","last_active":"2024-04-21T08:44:41.251368+0000","last_peered":"2024-04-21T08:44:41.251368+0000","last_clean":"2024-04-21T08:44:41.251368+0000","last_became_active":"2024-04-21T08:42:17.550507+0000","last_became_peered":"2024-04-21T08:42:17.550507+0000","last_unstale":"2024-04-21T08:44:41.251368+0000","last_undegraded":"2024-04-21T08:44:41.251368+0000","last_fullsized":"2024-04-21T08:44:41.251368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:25.561375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376794+0000","last_change":"2024-04-21T08:42:17.540533+0000","last_active":"2024-04-21T08:44:41.376794+0000","last_peered":"2024-04-21T08:44:41.376794+0000","last_clean":"2024-04-21T08:44:41.376794+0000","last_became_active":"2024-04-21T08:42:17.539940+0000","last_became_peered":"2024-04-21T08:42:17.539940+0000","last_unstale":"2024-04-21T08:44:41.376794+0000","last_undegraded":"2024-04-21T08:44:41.376794+0000","last_fullsized":"2024-04-21T08:44:41.376794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:45:19.102899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417676+0000","last_change":"2024-04-21T08:42:17.537794+0000","last_active":"2024-04-21T08:44:41.417676+0000","last_peered":"2024-04-21T08:44:41.417676+0000","last_clean":"2024-04-21T08:44:41.417676+0000","last_became_active":"2024-04-21T08:42:17.537376+0000","last_became_peered":"2024-04-21T08:42:17.537376+0000","last_unstale":"2024-04-21T08:44:41.417676+0000","last_undegraded":"2024-04-21T08:44:41.417676+0000","last_fullsized":"2024-04-21T08:44:41.417676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:57.430036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417515+0000","last_change":"2024-04-21T08:42:17.550574+0000","last_active":"2024-04-21T08:44:41.417515+0000","last_peered":"2024-04-21T08:44:41.417515+0000","last_clean":"2024-04-21T08:44:41.417515+0000","last_became_active":"2024-04-21T08:42:17.550205+0000","last_became_peered":"2024-04-21T08:42:17.550205+0000","last_unstale":"2024-04-21T08:44:41.417515+0000","last_undegraded":"2024-04-21T08:44:41.417515+0000","last_fullsized":"2024-04-21T08:44:41.417515+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:41:08.226477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374544+0000","last_change":"2024-04-21T08:42:17.542239+0000","last_active":"2024-04-21T08:44:41.374544+0000","last_peered":"2024-04-21T08:44:41.374544+0000","last_clean":"2024-04-21T08:44:41.374544+0000","last_became_active":"2024-04-21T08:42:17.541848+0000","last_became_peered":"2024-04-21T08:42:17.541848+0000","last_unstale":"2024-04-21T08:44:41.374544+0000","last_undegraded":"2024-04-21T08:44:41.374544+0000","last_fullsized":"2024-04-21T08:44:41.374544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:47.717451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375417+0000","last_change":"2024-04-21T08:42:17.553200+0000","last_active":"2024-04-21T08:44:41.375417+0000","last_peered":"2024-04-21T08:44:41.375417+0000","last_clean":"2024-04-21T08:44:41.375417+0000","last_became_active":"2024-04-21T08:42:17.552913+0000","last_became_peered":"2024-04-21T08:42:17.552913+0000","last_unstale":"2024-04-21T08:44:41.375417+0000","last_undegraded":"2024-04-21T08:44:41.375417+0000","last_fullsized":"2024-04-21T08:44:41.375417+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:39:04.186928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356904+0000","last_change":"2024-04-21T08:42:17.534881+0000","last_active":"2024-04-21T08:44:41.356904+0000","last_peered":"2024-04-21T08:44:41.356904+0000","last_clean":"2024-04-21T08:44:41.356904+0000","last_became_active":"2024-04-21T08:42:17.534496+0000","last_became_peered":"2024-04-21T08:42:17.534496+0000","last_unstale":"2024-04-21T08:44:41.356904+0000","last_undegraded":"2024-04-21T08:44:41.356904+0000","last_fullsized":"2024-04-21T08:44:41.356904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:50:19.409391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375734+0000","last_change":"2024-04-21T08:42:17.540480+0000","last_active":"2024-04-21T08:44:41.375734+0000","last_peered":"2024-04-21T08:44:41.375734+0000","last_clean":"2024-04-21T08:44:41.375734+0000","last_became_active":"2024-04-21T08:42:17.539742+0000","last_became_peered":"2024-04-21T08:42:17.539742+0000","last_unstale":"2024-04-21T08:44:41.375734+0000","last_undegraded":"2024-04-21T08:44:41.375734+0000","last_fullsized":"2024-04-21T08:44:41.375734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:00.238976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374953+0000","last_change":"2024-04-21T08:44:11.876826+0000","last_active":"2024-04-21T08:44:41.374953+0000","last_peered":"2024-04-21T08:44:41.374953+0000","last_clean":"2024-04-21T08:44:41.374953+0000","last_became_active":"2024-04-21T08:44:11.876354+0000","last_became_peered":"2024-04-21T08:44:11.876354+0000","last_unstale":"2024-04-21T08:44:41.374953+0000","last_undegraded":"2024-04-21T08:44:41.374953+0000","last_fullsized":"2024-04-21T08:44:41.374953+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:00:05.574008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356838+0000","last_change":"2024-04-21T08:42:16.639888+0000","last_active":"2024-04-21T08:44:41.356838+0000","last_peered":"2024-04-21T08:44:41.356838+0000","last_clean":"2024-04-21T08:44:41.356838+0000","last_became_active":"2024-04-21T08:42:16.639403+0000","last_became_peered":"2024-04-21T08:42:16.639403+0000","last_unstale":"2024-04-21T08:44:41.356838+0000","last_undegraded":"2024-04-21T08:44:41.356838+0000","last_fullsized":"2024-04-21T08:44:41.356838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:08:17.164924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249935+0000","last_change":"2024-04-21T08:43:10.983368+0000","last_active":"2024-04-21T08:44:41.249935+0000","last_peered":"2024-04-21T08:44:41.249935+0000","last_clean":"2024-04-21T08:44:41.249935+0000","last_became_active":"2024-04-21T08:43:10.982956+0000","last_became_peered":"2024-04-21T08:43:10.982956+0000","last_unstale":"2024-04-21T08:44:41.249935+0000","last_undegraded":"2024-04-21T08:44:41.249935+0000","last_fullsized":"2024-04-21T08:44:41.249935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:12:44.919996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250779+0000","last_change":"2024-04-21T08:42:21.673197+0000","last_active":"2024-04-21T08:44:41.250779+0000","last_peered":"2024-04-21T08:44:41.250779+0000","last_clean":"2024-04-21T08:44:41.250779+0000","last_became_active":"2024-04-21T08:42:21.672793+0000","last_became_peered":"2024-04-21T08:42:21.672793+0000","last_unstale":"2024-04-21T08:44:41.250779+0000","last_undegraded":"2024-04-21T08:44:41.250779+0000","last_fullsized":"2024-04-21T08:44:41.250779+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:44:29.788459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250990+0000","last_change":"2024-04-21T08:42:16.648577+0000","last_active":"2024-04-21T08:44:41.250990+0000","last_peered":"2024-04-21T08:44:41.250990+0000","last_clean":"2024-04-21T08:44:41.250990+0000","last_became_active":"2024-04-21T08:42:16.646716+0000","last_became_peered":"2024-04-21T08:42:16.646716+0000","last_unstale":"2024-04-21T08:44:41.250990+0000","last_undegraded":"2024-04-21T08:44:41.250990+0000","last_fullsized":"2024-04-21T08:44:41.250990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:10.327078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417936+0000","last_change":"2024-04-21T08:42:17.545430+0000","last_active":"2024-04-21T08:44:41.417936+0000","last_peered":"2024-04-21T08:44:41.417936+0000","last_clean":"2024-04-21T08:44:41.417936+0000","last_became_active":"2024-04-21T08:42:17.545057+0000","last_became_peered":"2024-04-21T08:42:17.545057+0000","last_unstale":"2024-04-21T08:44:41.417936+0000","last_undegraded":"2024-04-21T08:44:41.417936+0000","last_fullsized":"2024-04-21T08:44:41.417936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:44:59.030830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250899+0000","last_change":"2024-04-21T08:42:21.673196+0000","last_active":"2024-04-21T08:44:41.250899+0000","last_peered":"2024-04-21T08:44:41.250899+0000","last_clean":"2024-04-21T08:44:41.250899+0000","last_became_active":"2024-04-21T08:42:21.672808+0000","last_became_peered":"2024-04-21T08:42:21.672808+0000","last_unstale":"2024-04-21T08:44:41.250899+0000","last_undegraded":"2024-04-21T08:44:41.250899+0000","last_fullsized":"2024-04-21T08:44:41.250899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:39.482353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250844+0000","last_change":"2024-04-21T08:42:16.641611+0000","last_active":"2024-04-21T08:44:41.250844+0000","last_peered":"2024-04-21T08:44:41.250844+0000","last_clean":"2024-04-21T08:44:41.250844+0000","last_became_active":"2024-04-21T08:42:16.641221+0000","last_became_peered":"2024-04-21T08:42:16.641221+0000","last_unstale":"2024-04-21T08:44:41.250844+0000","last_undegraded":"2024-04-21T08:44:41.250844+0000","last_fullsized":"2024-04-21T08:44:41.250844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:45.496532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375448+0000","last_change":"2024-04-21T08:42:17.532986+0000","last_active":"2024-04-21T08:44:41.375448+0000","last_peered":"2024-04-21T08:44:41.375448+0000","last_clean":"2024-04-21T08:44:41.375448+0000","last_became_active":"2024-04-21T08:42:17.532583+0000","last_became_peered":"2024-04-21T08:42:17.532583+0000","last_unstale":"2024-04-21T08:44:41.375448+0000","last_undegraded":"2024-04-21T08:44:41.375448+0000","last_fullsized":"2024-04-21T08:44:41.375448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:43.780935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250791+0000","last_change":"2024-04-21T08:42:21.667459+0000","last_active":"2024-04-21T08:44:41.250791+0000","last_peered":"2024-04-21T08:44:41.250791+0000","last_clean":"2024-04-21T08:44:41.250791+0000","last_became_active":"2024-04-21T08:42:21.667183+0000","last_became_peered":"2024-04-21T08:42:21.667183+0000","last_unstale":"2024-04-21T08:44:41.250791+0000","last_undegraded":"2024-04-21T08:44:41.250791+0000","last_fullsized":"2024-04-21T08:44:41.250791+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:32:23.626267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250969+0000","last_change":"2024-04-21T08:42:16.645210+0000","last_active":"2024-04-21T08:44:41.250969+0000","last_peered":"2024-04-21T08:44:41.250969+0000","last_clean":"2024-04-21T08:44:41.250969+0000","last_became_active":"2024-04-21T08:42:16.644747+0000","last_became_peered":"2024-04-21T08:42:16.644747+0000","last_unstale":"2024-04-21T08:44:41.250969+0000","last_undegraded":"2024-04-21T08:44:41.250969+0000","last_fullsized":"2024-04-21T08:44:41.250969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:34:45.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365569+0000","last_change":"2024-04-21T08:43:10.984152+0000","last_active":"2024-04-21T08:44:41.365569+0000","last_peered":"2024-04-21T08:44:41.365569+0000","last_clean":"2024-04-21T08:44:41.365569+0000","last_became_active":"2024-04-21T08:43:10.983736+0000","last_became_peered":"2024-04-21T08:43:10.983736+0000","last_unstale":"2024-04-21T08:44:41.365569+0000","last_undegraded":"2024-04-21T08:44:41.365569+0000","last_fullsized":"2024-04-21T08:44:41.365569+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:10.182667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375391+0000","last_change":"2024-04-21T08:42:21.660739+0000","last_active":"2024-04-21T08:44:41.375391+0000","last_peered":"2024-04-21T08:44:41.375391+0000","last_clean":"2024-04-21T08:44:41.375391+0000","last_became_active":"2024-04-21T08:42:21.660307+0000","last_became_peered":"2024-04-21T08:42:21.660307+0000","last_unstale":"2024-04-21T08:44:41.375391+0000","last_undegraded":"2024-04-21T08:44:41.375391+0000","last_fullsized":"2024-04-21T08:44:41.375391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:26.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365469+0000","last_change":"2024-04-21T08:43:10.986184+0000","last_active":"2024-04-21T08:44:41.365469+0000","last_peered":"2024-04-21T08:44:41.365469+0000","last_clean":"2024-04-21T08:44:41.365469+0000","last_became_active":"2024-04-21T08:43:10.985533+0000","last_became_peered":"2024-04-21T08:43:10.985533+0000","last_unstale":"2024-04-21T08:44:41.365469+0000","last_undegraded":"2024-04-21T08:44:41.365469+0000","last_fullsized":"2024-04-21T08:44:41.365469+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:26.023306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250741+0000","last_change":"2024-04-21T08:42:17.542315+0000","last_active":"2024-04-21T08:44:41.250741+0000","last_peered":"2024-04-21T08:44:41.250741+0000","last_clean":"2024-04-21T08:44:41.250741+0000","last_became_active":"2024-04-21T08:42:17.541895+0000","last_became_peered":"2024-04-21T08:42:17.541895+0000","last_unstale":"2024-04-21T08:44:41.250741+0000","last_undegraded":"2024-04-21T08:44:41.250741+0000","last_fullsized":"2024-04-21T08:44:41.250741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:23:33.909499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375270+0000","last_change":"2024-04-21T08:44:11.878135+0000","last_active":"2024-04-21T08:44:41.375270+0000","last_peered":"2024-04-21T08:44:41.375270+0000","last_clean":"2024-04-21T08:44:41.375270+0000","last_became_active":"2024-04-21T08:44:11.877271+0000","last_became_peered":"2024-04-21T08:44:11.877271+0000","last_unstale":"2024-04-21T08:44:41.375270+0000","last_undegraded":"2024-04-21T08:44:41.375270+0000","last_fullsized":"2024-04-21T08:44:41.375270+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:31.345519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368426+0000","last_change":"2024-04-21T08:43:10.984839+0000","last_active":"2024-04-21T08:44:41.368426+0000","last_peered":"2024-04-21T08:44:41.368426+0000","last_clean":"2024-04-21T08:44:41.368426+0000","last_became_active":"2024-04-21T08:43:10.984382+0000","last_became_peered":"2024-04-21T08:43:10.984382+0000","last_unstale":"2024-04-21T08:44:41.368426+0000","last_undegraded":"2024-04-21T08:44:41.368426+0000","last_fullsized":"2024-04-21T08:44:41.368426+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:47:20.774102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365597+0000","last_change":"2024-04-21T08:42:17.547770+0000","last_active":"2024-04-21T08:44:41.365597+0000","last_peered":"2024-04-21T08:44:41.365597+0000","last_clean":"2024-04-21T08:44:41.365597+0000","last_became_active":"2024-04-21T08:42:17.547440+0000","last_became_peered":"2024-04-21T08:42:17.547440+0000","last_unstale":"2024-04-21T08:44:41.365597+0000","last_undegraded":"2024-04-21T08:44:41.365597+0000","last_fullsized":"2024-04-21T08:44:41.365597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:37:35.555737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375243+0000","last_change":"2024-04-21T08:42:21.670684+0000","last_active":"2024-04-21T08:44:41.375243+0000","last_peered":"2024-04-21T08:44:41.375243+0000","last_clean":"2024-04-21T08:44:41.375243+0000","last_became_active":"2024-04-21T08:42:21.670260+0000","last_became_peered":"2024-04-21T08:42:21.670260+0000","last_unstale":"2024-04-21T08:44:41.375243+0000","last_undegraded":"2024-04-21T08:44:41.375243+0000","last_fullsized":"2024-04-21T08:44:41.375243+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:45:06.192040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376149+0000","last_change":"2024-04-21T08:42:16.639632+0000","last_active":"2024-04-21T08:44:41.376149+0000","last_peered":"2024-04-21T08:44:41.376149+0000","last_clean":"2024-04-21T08:44:41.376149+0000","last_became_active":"2024-04-21T08:42:16.639301+0000","last_became_peered":"2024-04-21T08:42:16.639301+0000","last_unstale":"2024-04-21T08:44:41.376149+0000","last_undegraded":"2024-04-21T08:44:41.376149+0000","last_fullsized":"2024-04-21T08:44:41.376149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:58.838667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249940+0000","last_change":"2024-04-21T08:42:17.546109+0000","last_active":"2024-04-21T08:44:41.249940+0000","last_peered":"2024-04-21T08:44:41.249940+0000","last_clean":"2024-04-21T08:44:41.249940+0000","last_became_active":"2024-04-21T08:42:17.545673+0000","last_became_peered":"2024-04-21T08:42:17.545673+0000","last_unstale":"2024-04-21T08:44:41.249940+0000","last_undegraded":"2024-04-21T08:44:41.249940+0000","last_fullsized":"2024-04-21T08:44:41.249940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:18.947848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417985+0000","last_change":"2024-04-21T08:42:21.666114+0000","last_active":"2024-04-21T08:44:41.417985+0000","last_peered":"2024-04-21T08:44:41.417985+0000","last_clean":"2024-04-21T08:44:41.417985+0000","last_became_active":"2024-04-21T08:42:21.665020+0000","last_became_peered":"2024-04-21T08:42:21.665020+0000","last_unstale":"2024-04-21T08:44:41.417985+0000","last_undegraded":"2024-04-21T08:44:41.417985+0000","last_fullsized":"2024-04-21T08:44:41.417985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:33.644178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365224+0000","last_change":"2024-04-21T08:42:16.639118+0000","last_active":"2024-04-21T08:44:41.365224+0000","last_peered":"2024-04-21T08:44:41.365224+0000","last_clean":"2024-04-21T08:44:41.365224+0000","last_became_active":"2024-04-21T08:42:16.638792+0000","last_became_peered":"2024-04-21T08:42:16.638792+0000","last_unstale":"2024-04-21T08:44:41.365224+0000","last_undegraded":"2024-04-21T08:44:41.365224+0000","last_fullsized":"2024-04-21T08:44:41.365224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:55.568712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356215+0000","last_change":"2024-04-21T08:42:17.539688+0000","last_active":"2024-04-21T08:44:41.356215+0000","last_peered":"2024-04-21T08:44:41.356215+0000","last_clean":"2024-04-21T08:44:41.356215+0000","last_became_active":"2024-04-21T08:42:17.539335+0000","last_became_peered":"2024-04-21T08:42:17.539335+0000","last_unstale":"2024-04-21T08:44:41.356215+0000","last_undegraded":"2024-04-21T08:44:41.356215+0000","last_fullsized":"2024-04-21T08:44:41.356215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:27:51.520232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249882+0000","last_change":"2024-04-21T08:42:21.660570+0000","last_active":"2024-04-21T08:44:41.249882+0000","last_peered":"2024-04-21T08:44:41.249882+0000","last_clean":"2024-04-21T08:44:41.249882+0000","last_became_active":"2024-04-21T08:42:21.659398+0000","last_became_peered":"2024-04-21T08:42:21.659398+0000","last_unstale":"2024-04-21T08:44:41.249882+0000","last_undegraded":"2024-04-21T08:44:41.249882+0000","last_fullsized":"2024-04-21T08:44:41.249882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:55.840334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368786+0000","last_change":"2024-04-21T08:42:16.648192+0000","last_active":"2024-04-21T08:44:41.368786+0000","last_peered":"2024-04-21T08:44:41.368786+0000","last_clean":"2024-04-21T08:44:41.368786+0000","last_became_active":"2024-04-21T08:42:16.647666+0000","last_became_peered":"2024-04-21T08:42:16.647666+0000","last_unstale":"2024-04-21T08:44:41.368786+0000","last_undegraded":"2024-04-21T08:44:41.368786+0000","last_fullsized":"2024-04-21T08:44:41.368786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:33.344680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375708+0000","last_change":"2024-04-21T08:42:17.545925+0000","last_active":"2024-04-21T08:44:41.375708+0000","last_peered":"2024-04-21T08:44:41.375708+0000","last_clean":"2024-04-21T08:44:41.375708+0000","last_became_active":"2024-04-21T08:42:17.545555+0000","last_became_peered":"2024-04-21T08:42:17.545555+0000","last_unstale":"2024-04-21T08:44:41.375708+0000","last_undegraded":"2024-04-21T08:44:41.375708+0000","last_fullsized":"2024-04-21T08:44:41.375708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:13:07.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356087+0000","last_change":"2024-04-21T08:42:21.662757+0000","last_active":"2024-04-21T08:44:41.356087+0000","last_peered":"2024-04-21T08:44:41.356087+0000","last_clean":"2024-04-21T08:44:41.356087+0000","last_became_active":"2024-04-21T08:42:21.662306+0000","last_became_peered":"2024-04-21T08:42:21.662306+0000","last_unstale":"2024-04-21T08:44:41.356087+0000","last_undegraded":"2024-04-21T08:44:41.356087+0000","last_fullsized":"2024-04-21T08:44:41.356087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:26.856147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375135+0000","last_change":"2024-04-21T08:43:10.984690+0000","last_active":"2024-04-21T08:44:41.375135+0000","last_peered":"2024-04-21T08:44:41.375135+0000","last_clean":"2024-04-21T08:44:41.375135+0000","last_became_active":"2024-04-21T08:43:10.984290+0000","last_became_peered":"2024-04-21T08:43:10.984290+0000","last_unstale":"2024-04-21T08:44:41.375135+0000","last_undegraded":"2024-04-21T08:44:41.375135+0000","last_fullsized":"2024-04-21T08:44:41.375135+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:37:36.400946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418259+0000","last_change":"2024-04-21T08:43:10.793316+0000","last_active":"2024-04-21T08:44:41.418259+0000","last_peered":"2024-04-21T08:44:41.418259+0000","last_clean":"2024-04-21T08:44:41.418259+0000","last_became_active":"2024-04-21T08:43:10.793035+0000","last_became_peered":"2024-04-21T08:43:10.793035+0000","last_unstale":"2024-04-21T08:44:41.418259+0000","last_undegraded":"2024-04-21T08:44:41.418259+0000","last_fullsized":"2024-04-21T08:44:41.418259+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:41:04.247385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251083+0000","last_change":"2024-04-21T08:42:21.670561+0000","last_active":"2024-04-21T08:44:41.251083+0000","last_peered":"2024-04-21T08:44:41.251083+0000","last_clean":"2024-04-21T08:44:41.251083+0000","last_became_active":"2024-04-21T08:42:21.670170+0000","last_became_peered":"2024-04-21T08:42:21.670170+0000","last_unstale":"2024-04-21T08:44:41.251083+0000","last_undegraded":"2024-04-21T08:44:41.251083+0000","last_fullsized":"2024-04-21T08:44:41.251083+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:35:54.538013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365460+0000","last_change":"2024-04-21T08:42:16.637319+0000","last_active":"2024-04-21T08:44:41.365460+0000","last_peered":"2024-04-21T08:44:41.365460+0000","last_clean":"2024-04-21T08:44:41.365460+0000","last_became_active":"2024-04-21T08:42:16.636940+0000","last_became_peered":"2024-04-21T08:42:16.636940+0000","last_unstale":"2024-04-21T08:44:41.365460+0000","last_undegraded":"2024-04-21T08:44:41.365460+0000","last_fullsized":"2024-04-21T08:44:41.365460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:59:09.826786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375697+0000","last_change":"2024-04-21T08:42:17.546202+0000","last_active":"2024-04-21T08:44:41.375697+0000","last_peered":"2024-04-21T08:44:41.375697+0000","last_clean":"2024-04-21T08:44:41.375697+0000","last_became_active":"2024-04-21T08:42:17.545781+0000","last_became_peered":"2024-04-21T08:42:17.545781+0000","last_unstale":"2024-04-21T08:44:41.375697+0000","last_undegraded":"2024-04-21T08:44:41.375697+0000","last_fullsized":"2024-04-21T08:44:41.375697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:23.990951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356160+0000","last_change":"2024-04-21T08:42:21.662822+0000","last_active":"2024-04-21T08:44:41.356160+0000","last_peered":"2024-04-21T08:44:41.356160+0000","last_clean":"2024-04-21T08:44:41.356160+0000","last_became_active":"2024-04-21T08:42:21.662398+0000","last_became_peered":"2024-04-21T08:42:21.662398+0000","last_unstale":"2024-04-21T08:44:41.356160+0000","last_undegraded":"2024-04-21T08:44:41.356160+0000","last_fullsized":"2024-04-21T08:44:41.356160+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:38:40.499294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":335,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:47.138883+0000","last_change":"2024-04-21T08:42:16.636330+0000","last_active":"2024-04-21T08:44:47.138883+0000","last_peered":"2024-04-21T08:44:47.138883+0000","last_clean":"2024-04-21T08:44:47.138883+0000","last_became_active":"2024-04-21T08:42:16.635762+0000","last_became_peered":"2024-04-21T08:42:16.635762+0000","last_unstale":"2024-04-21T08:44:47.138883+0000","last_undegraded":"2024-04-21T08:44:47.138883+0000","last_fullsized":"2024-04-21T08:44:47.138883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:31.137923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":263,"num_read_kb":201,"num_write":38,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251043+0000","last_change":"2024-04-21T08:42:17.543033+0000","last_active":"2024-04-21T08:44:41.251043+0000","last_peered":"2024-04-21T08:44:41.251043+0000","last_clean":"2024-04-21T08:44:41.251043+0000","last_became_active":"2024-04-21T08:42:17.542692+0000","last_became_peered":"2024-04-21T08:42:17.542692+0000","last_unstale":"2024-04-21T08:44:41.251043+0000","last_undegraded":"2024-04-21T08:44:41.251043+0000","last_fullsized":"2024-04-21T08:44:41.251043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:33:32.111999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365806+0000","last_change":"2024-04-21T08:42:21.668831+0000","last_active":"2024-04-21T08:44:41.365806+0000","last_peered":"2024-04-21T08:44:41.365806+0000","last_clean":"2024-04-21T08:44:41.365806+0000","last_became_active":"2024-04-21T08:42:21.668485+0000","last_became_peered":"2024-04-21T08:42:21.668485+0000","last_unstale":"2024-04-21T08:44:41.365806+0000","last_undegraded":"2024-04-21T08:44:41.365806+0000","last_fullsized":"2024-04-21T08:44:41.365806+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:49:29.049707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375241+0000","last_change":"2024-04-21T08:42:16.640288+0000","last_active":"2024-04-21T08:44:41.375241+0000","last_peered":"2024-04-21T08:44:41.375241+0000","last_clean":"2024-04-21T08:44:41.375241+0000","last_became_active":"2024-04-21T08:42:16.639909+0000","last_became_peered":"2024-04-21T08:42:16.639909+0000","last_unstale":"2024-04-21T08:44:41.375241+0000","last_undegraded":"2024-04-21T08:44:41.375241+0000","last_fullsized":"2024-04-21T08:44:41.375241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:33.839442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375479+0000","last_change":"2024-04-21T08:42:17.541154+0000","last_active":"2024-04-21T08:44:41.375479+0000","last_peered":"2024-04-21T08:44:41.375479+0000","last_clean":"2024-04-21T08:44:41.375479+0000","last_became_active":"2024-04-21T08:42:17.540747+0000","last_became_peered":"2024-04-21T08:42:17.540747+0000","last_unstale":"2024-04-21T08:44:41.375479+0000","last_undegraded":"2024-04-21T08:44:41.375479+0000","last_fullsized":"2024-04-21T08:44:41.375479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:14:50.641144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375040+0000","last_change":"2024-04-21T08:42:21.670767+0000","last_active":"2024-04-21T08:44:41.375040+0000","last_peered":"2024-04-21T08:44:41.375040+0000","last_clean":"2024-04-21T08:44:41.375040+0000","last_became_active":"2024-04-21T08:42:21.670338+0000","last_became_peered":"2024-04-21T08:42:21.670338+0000","last_unstale":"2024-04-21T08:44:41.375040+0000","last_undegraded":"2024-04-21T08:44:41.375040+0000","last_fullsized":"2024-04-21T08:44:41.375040+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:38:55.275603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365468+0000","last_change":"2024-04-21T08:42:16.640144+0000","last_active":"2024-04-21T08:44:41.365468+0000","last_peered":"2024-04-21T08:44:41.365468+0000","last_clean":"2024-04-21T08:44:41.365468+0000","last_became_active":"2024-04-21T08:42:16.639707+0000","last_became_peered":"2024-04-21T08:42:16.639707+0000","last_unstale":"2024-04-21T08:44:41.365468+0000","last_undegraded":"2024-04-21T08:44:41.365468+0000","last_fullsized":"2024-04-21T08:44:41.365468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:11:32.785519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375240+0000","last_change":"2024-04-21T08:42:17.555019+0000","last_active":"2024-04-21T08:44:41.375240+0000","last_peered":"2024-04-21T08:44:41.375240+0000","last_clean":"2024-04-21T08:44:41.375240+0000","last_became_active":"2024-04-21T08:42:17.554645+0000","last_became_peered":"2024-04-21T08:42:17.554645+0000","last_unstale":"2024-04-21T08:44:41.375240+0000","last_undegraded":"2024-04-21T08:44:41.375240+0000","last_fullsized":"2024-04-21T08:44:41.375240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:42:30.353945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251223+0000","last_change":"2024-04-21T08:44:11.876740+0000","last_active":"2024-04-21T08:44:41.251223+0000","last_peered":"2024-04-21T08:44:41.251223+0000","last_clean":"2024-04-21T08:44:41.251223+0000","last_became_active":"2024-04-21T08:44:11.876108+0000","last_became_peered":"2024-04-21T08:44:11.876108+0000","last_unstale":"2024-04-21T08:44:41.251223+0000","last_undegraded":"2024-04-21T08:44:41.251223+0000","last_fullsized":"2024-04-21T08:44:41.251223+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:37.398743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250912+0000","last_change":"2024-04-21T08:42:16.640407+0000","last_active":"2024-04-21T08:44:41.250912+0000","last_peered":"2024-04-21T08:44:41.250912+0000","last_clean":"2024-04-21T08:44:41.250912+0000","last_became_active":"2024-04-21T08:42:16.640015+0000","last_became_peered":"2024-04-21T08:42:16.640015+0000","last_unstale":"2024-04-21T08:44:41.250912+0000","last_undegraded":"2024-04-21T08:44:41.250912+0000","last_fullsized":"2024-04-21T08:44:41.250912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:23:10.647184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374762+0000","last_change":"2024-04-21T08:42:17.545438+0000","last_active":"2024-04-21T08:44:41.374762+0000","last_peered":"2024-04-21T08:44:41.374762+0000","last_clean":"2024-04-21T08:44:41.374762+0000","last_became_active":"2024-04-21T08:42:17.544258+0000","last_became_peered":"2024-04-21T08:42:17.544258+0000","last_unstale":"2024-04-21T08:44:41.374762+0000","last_undegraded":"2024-04-21T08:44:41.374762+0000","last_fullsized":"2024-04-21T08:44:41.374762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:43.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376425+0000","last_change":"2024-04-21T08:42:21.671000+0000","last_active":"2024-04-21T08:44:41.376425+0000","last_peered":"2024-04-21T08:44:41.376425+0000","last_clean":"2024-04-21T08:44:41.376425+0000","last_became_active":"2024-04-21T08:42:21.670380+0000","last_became_peered":"2024-04-21T08:42:21.670380+0000","last_unstale":"2024-04-21T08:44:41.376425+0000","last_undegraded":"2024-04-21T08:44:41.376425+0000","last_fullsized":"2024-04-21T08:44:41.376425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:57.075686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250621+0000","last_change":"2024-04-21T08:42:16.649599+0000","last_active":"2024-04-21T08:44:41.250621+0000","last_peered":"2024-04-21T08:44:41.250621+0000","last_clean":"2024-04-21T08:44:41.250621+0000","last_became_active":"2024-04-21T08:42:16.649232+0000","last_became_peered":"2024-04-21T08:42:16.649232+0000","last_unstale":"2024-04-21T08:44:41.250621+0000","last_undegraded":"2024-04-21T08:44:41.250621+0000","last_fullsized":"2024-04-21T08:44:41.250621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:21.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356045+0000","last_change":"2024-04-21T08:44:11.776138+0000","last_active":"2024-04-21T08:44:41.356045+0000","last_peered":"2024-04-21T08:44:41.356045+0000","last_clean":"2024-04-21T08:44:41.356045+0000","last_became_active":"2024-04-21T08:44:11.775832+0000","last_became_peered":"2024-04-21T08:44:11.775832+0000","last_unstale":"2024-04-21T08:44:41.356045+0000","last_undegraded":"2024-04-21T08:44:41.356045+0000","last_fullsized":"2024-04-21T08:44:41.356045+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:03:34.210923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365172+0000","last_change":"2024-04-21T08:42:21.664811+0000","last_active":"2024-04-21T08:44:41.365172+0000","last_peered":"2024-04-21T08:44:41.365172+0000","last_clean":"2024-04-21T08:44:41.365172+0000","last_became_active":"2024-04-21T08:42:21.664332+0000","last_became_peered":"2024-04-21T08:42:21.664332+0000","last_unstale":"2024-04-21T08:44:41.365172+0000","last_undegraded":"2024-04-21T08:44:41.365172+0000","last_fullsized":"2024-04-21T08:44:41.365172+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:40.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251372+0000","last_change":"2024-04-21T08:42:16.641233+0000","last_active":"2024-04-21T08:44:41.251372+0000","last_peered":"2024-04-21T08:44:41.251372+0000","last_clean":"2024-04-21T08:44:41.251372+0000","last_became_active":"2024-04-21T08:42:16.640873+0000","last_became_peered":"2024-04-21T08:42:16.640873+0000","last_unstale":"2024-04-21T08:44:41.251372+0000","last_undegraded":"2024-04-21T08:44:41.251372+0000","last_fullsized":"2024-04-21T08:44:41.251372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:00.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375293+0000","last_change":"2024-04-21T08:42:17.550438+0000","last_active":"2024-04-21T08:44:41.375293+0000","last_peered":"2024-04-21T08:44:41.375293+0000","last_clean":"2024-04-21T08:44:41.375293+0000","last_became_active":"2024-04-21T08:42:17.550015+0000","last_became_peered":"2024-04-21T08:42:17.550015+0000","last_unstale":"2024-04-21T08:44:41.375293+0000","last_undegraded":"2024-04-21T08:44:41.375293+0000","last_fullsized":"2024-04-21T08:44:41.375293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:00:29.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250709+0000","last_change":"2024-04-21T08:42:21.970498+0000","last_active":"2024-04-21T08:44:41.250709+0000","last_peered":"2024-04-21T08:44:41.250709+0000","last_clean":"2024-04-21T08:44:41.250709+0000","last_became_active":"2024-04-21T08:42:21.969819+0000","last_became_peered":"2024-04-21T08:42:21.969819+0000","last_unstale":"2024-04-21T08:44:41.250709+0000","last_undegraded":"2024-04-21T08:44:41.250709+0000","last_fullsized":"2024-04-21T08:44:41.250709+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:13:16.552293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356527+0000","last_change":"2024-04-21T08:42:16.637146+0000","last_active":"2024-04-21T08:44:41.356527+0000","last_peered":"2024-04-21T08:44:41.356527+0000","last_clean":"2024-04-21T08:44:41.356527+0000","last_became_active":"2024-04-21T08:42:16.636781+0000","last_became_peered":"2024-04-21T08:42:16.636781+0000","last_unstale":"2024-04-21T08:44:41.356527+0000","last_undegraded":"2024-04-21T08:44:41.356527+0000","last_fullsized":"2024-04-21T08:44:41.356527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:39:27.641236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368622+0000","last_change":"2024-04-21T08:42:17.542325+0000","last_active":"2024-04-21T08:44:41.368622+0000","last_peered":"2024-04-21T08:44:41.368622+0000","last_clean":"2024-04-21T08:44:41.368622+0000","last_became_active":"2024-04-21T08:42:17.541103+0000","last_became_peered":"2024-04-21T08:42:17.541103+0000","last_unstale":"2024-04-21T08:44:41.368622+0000","last_undegraded":"2024-04-21T08:44:41.368622+0000","last_fullsized":"2024-04-21T08:44:41.368622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:46.903250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375702+0000","last_change":"2024-04-21T08:42:21.671146+0000","last_active":"2024-04-21T08:44:41.375702+0000","last_peered":"2024-04-21T08:44:41.375702+0000","last_clean":"2024-04-21T08:44:41.375702+0000","last_became_active":"2024-04-21T08:42:21.670561+0000","last_became_peered":"2024-04-21T08:42:21.670561+0000","last_unstale":"2024-04-21T08:44:41.375702+0000","last_undegraded":"2024-04-21T08:44:41.375702+0000","last_fullsized":"2024-04-21T08:44:41.375702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:38:28.028976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375066+0000","last_change":"2024-04-21T08:42:16.645132+0000","last_active":"2024-04-21T08:44:41.375066+0000","last_peered":"2024-04-21T08:44:41.375066+0000","last_clean":"2024-04-21T08:44:41.375066+0000","last_became_active":"2024-04-21T08:42:16.644786+0000","last_became_peered":"2024-04-21T08:42:16.644786+0000","last_unstale":"2024-04-21T08:44:41.375066+0000","last_undegraded":"2024-04-21T08:44:41.375066+0000","last_fullsized":"2024-04-21T08:44:41.375066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:08.298498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368567+0000","last_change":"2024-04-21T08:42:17.550424+0000","last_active":"2024-04-21T08:44:41.368567+0000","last_peered":"2024-04-21T08:44:41.368567+0000","last_clean":"2024-04-21T08:44:41.368567+0000","last_became_active":"2024-04-21T08:42:17.550077+0000","last_became_peered":"2024-04-21T08:42:17.550077+0000","last_unstale":"2024-04-21T08:44:41.368567+0000","last_undegraded":"2024-04-21T08:44:41.368567+0000","last_fullsized":"2024-04-21T08:44:41.368567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:17:07.016204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375856+0000","last_change":"2024-04-21T08:42:21.970652+0000","last_active":"2024-04-21T08:44:41.375856+0000","last_peered":"2024-04-21T08:44:41.375856+0000","last_clean":"2024-04-21T08:44:41.375856+0000","last_became_active":"2024-04-21T08:42:21.970251+0000","last_became_peered":"2024-04-21T08:42:21.970251+0000","last_unstale":"2024-04-21T08:44:41.375856+0000","last_undegraded":"2024-04-21T08:44:41.375856+0000","last_fullsized":"2024-04-21T08:44:41.375856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:45.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365285+0000","last_change":"2024-04-21T08:42:17.542820+0000","last_active":"2024-04-21T08:44:41.365285+0000","last_peered":"2024-04-21T08:44:41.365285+0000","last_clean":"2024-04-21T08:44:41.365285+0000","last_became_active":"2024-04-21T08:42:17.541320+0000","last_became_peered":"2024-04-21T08:42:17.541320+0000","last_unstale":"2024-04-21T08:44:41.365285+0000","last_undegraded":"2024-04-21T08:44:41.365285+0000","last_fullsized":"2024-04-21T08:44:41.365285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:23:06.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375209+0000","last_change":"2024-04-21T08:42:16.636950+0000","last_active":"2024-04-21T08:44:41.375209+0000","last_peered":"2024-04-21T08:44:41.375209+0000","last_clean":"2024-04-21T08:44:41.375209+0000","last_became_active":"2024-04-21T08:42:16.636590+0000","last_became_peered":"2024-04-21T08:42:16.636590+0000","last_unstale":"2024-04-21T08:44:41.375209+0000","last_undegraded":"2024-04-21T08:44:41.375209+0000","last_fullsized":"2024-04-21T08:44:41.375209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:55.693681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374884+0000","last_change":"2024-04-21T08:42:21.660404+0000","last_active":"2024-04-21T08:44:41.374884+0000","last_peered":"2024-04-21T08:44:41.374884+0000","last_clean":"2024-04-21T08:44:41.374884+0000","last_became_active":"2024-04-21T08:42:21.659999+0000","last_became_peered":"2024-04-21T08:42:21.659999+0000","last_unstale":"2024-04-21T08:44:41.374884+0000","last_undegraded":"2024-04-21T08:44:41.374884+0000","last_fullsized":"2024-04-21T08:44:41.374884+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:53:32.328873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356239+0000","last_change":"2024-04-21T08:43:10.984361+0000","last_active":"2024-04-21T08:44:41.356239+0000","last_peered":"2024-04-21T08:44:41.356239+0000","last_clean":"2024-04-21T08:44:41.356239+0000","last_became_active":"2024-04-21T08:43:10.984046+0000","last_became_peered":"2024-04-21T08:43:10.984046+0000","last_unstale":"2024-04-21T08:44:41.356239+0000","last_undegraded":"2024-04-21T08:44:41.356239+0000","last_fullsized":"2024-04-21T08:44:41.356239+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:33:23.154630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251236+0000","last_change":"2024-04-21T08:42:16.638993+0000","last_active":"2024-04-21T08:44:41.251236+0000","last_peered":"2024-04-21T08:44:41.251236+0000","last_clean":"2024-04-21T08:44:41.251236+0000","last_became_active":"2024-04-21T08:42:16.638617+0000","last_became_peered":"2024-04-21T08:42:16.638617+0000","last_unstale":"2024-04-21T08:44:41.251236+0000","last_undegraded":"2024-04-21T08:44:41.251236+0000","last_fullsized":"2024-04-21T08:44:41.251236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:14:28.271433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"95'148","reported_seq":410,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:43.370391+0000","last_change":"2024-04-21T08:41:00.044984+0000","last_active":"2024-04-21T08:44:43.370391+0000","last_peered":"2024-04-21T08:44:43.370391+0000","last_clean":"2024-04-21T08:44:43.370391+0000","last_became_active":"2024-04-21T08:41:00.044485+0000","last_became_peered":"2024-04-21T08:41:00.044485+0000","last_unstale":"2024-04-21T08:44:43.370391+0000","last_undegraded":"2024-04-21T08:44:43.370391+0000","last_fullsized":"2024-04-21T08:44:43.370391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.370031+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:01.031209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365508+0000","last_change":"2024-04-21T08:42:21.971045+0000","last_active":"2024-04-21T08:44:41.365508+0000","last_peered":"2024-04-21T08:44:41.365508+0000","last_clean":"2024-04-21T08:44:41.365508+0000","last_became_active":"2024-04-21T08:42:21.970577+0000","last_became_peered":"2024-04-21T08:42:21.970577+0000","last_unstale":"2024-04-21T08:44:41.365508+0000","last_undegraded":"2024-04-21T08:44:41.365508+0000","last_fullsized":"2024-04-21T08:44:41.365508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:55.681662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374964+0000","last_change":"2024-04-21T08:42:17.535570+0000","last_active":"2024-04-21T08:44:41.374964+0000","last_peered":"2024-04-21T08:44:41.374964+0000","last_clean":"2024-04-21T08:44:41.374964+0000","last_became_active":"2024-04-21T08:42:17.535217+0000","last_became_peered":"2024-04-21T08:42:17.535217+0000","last_unstale":"2024-04-21T08:44:41.374964+0000","last_undegraded":"2024-04-21T08:44:41.374964+0000","last_fullsized":"2024-04-21T08:44:41.374964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:49.510750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374650+0000","last_change":"2024-04-21T08:43:10.792548+0000","last_active":"2024-04-21T08:44:41.374650+0000","last_peered":"2024-04-21T08:44:41.374650+0000","last_clean":"2024-04-21T08:44:41.374650+0000","last_became_active":"2024-04-21T08:43:10.792165+0000","last_became_peered":"2024-04-21T08:43:10.792165+0000","last_unstale":"2024-04-21T08:44:41.374650+0000","last_undegraded":"2024-04-21T08:44:41.374650+0000","last_fullsized":"2024-04-21T08:44:41.374650+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:35.980536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374872+0000","last_change":"2024-04-21T08:42:21.661393+0000","last_active":"2024-04-21T08:44:41.374872+0000","last_peered":"2024-04-21T08:44:41.374872+0000","last_clean":"2024-04-21T08:44:41.374872+0000","last_became_active":"2024-04-21T08:42:21.660201+0000","last_became_peered":"2024-04-21T08:42:21.660201+0000","last_unstale":"2024-04-21T08:44:41.374872+0000","last_undegraded":"2024-04-21T08:44:41.374872+0000","last_fullsized":"2024-04-21T08:44:41.374872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:13.393389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250941+0000","last_change":"2024-04-21T08:42:16.650507+0000","last_active":"2024-04-21T08:44:41.250941+0000","last_peered":"2024-04-21T08:44:41.250941+0000","last_clean":"2024-04-21T08:44:41.250941+0000","last_became_active":"2024-04-21T08:42:16.650217+0000","last_became_peered":"2024-04-21T08:42:16.650217+0000","last_unstale":"2024-04-21T08:44:41.250941+0000","last_undegraded":"2024-04-21T08:44:41.250941+0000","last_fullsized":"2024-04-21T08:44:41.250941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:13:26.785092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375997+0000","last_change":"2024-04-21T08:42:17.530420+0000","last_active":"2024-04-21T08:44:41.375997+0000","last_peered":"2024-04-21T08:44:41.375997+0000","last_clean":"2024-04-21T08:44:41.375997+0000","last_became_active":"2024-04-21T08:42:17.529920+0000","last_became_peered":"2024-04-21T08:42:17.529920+0000","last_unstale":"2024-04-21T08:44:41.375997+0000","last_undegraded":"2024-04-21T08:44:41.375997+0000","last_fullsized":"2024-04-21T08:44:41.375997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:24.250197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249760+0000","last_change":"2024-04-21T08:42:21.970586+0000","last_active":"2024-04-21T08:44:41.249760+0000","last_peered":"2024-04-21T08:44:41.249760+0000","last_clean":"2024-04-21T08:44:41.249760+0000","last_became_active":"2024-04-21T08:42:21.970132+0000","last_became_peered":"2024-04-21T08:42:21.970132+0000","last_unstale":"2024-04-21T08:44:41.249760+0000","last_undegraded":"2024-04-21T08:44:41.249760+0000","last_fullsized":"2024-04-21T08:44:41.249760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:11.074370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'68","reported_seq":155,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:47.175970+0000","last_change":"2024-04-21T08:42:16.647916+0000","last_active":"2024-04-21T08:44:47.175970+0000","last_peered":"2024-04-21T08:44:47.175970+0000","last_clean":"2024-04-21T08:44:47.175970+0000","last_became_active":"2024-04-21T08:42:16.647476+0000","last_became_peered":"2024-04-21T08:42:16.647476+0000","last_unstale":"2024-04-21T08:44:47.175970+0000","last_undegraded":"2024-04-21T08:44:47.175970+0000","last_fullsized":"2024-04-21T08:44:47.175970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:27.242153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":160,"num_write":122,"num_write_kb":1157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251070+0000","last_change":"2024-04-21T08:42:17.546144+0000","last_active":"2024-04-21T08:44:41.251070+0000","last_peered":"2024-04-21T08:44:41.251070+0000","last_clean":"2024-04-21T08:44:41.251070+0000","last_became_active":"2024-04-21T08:42:17.545730+0000","last_became_peered":"2024-04-21T08:42:17.545730+0000","last_unstale":"2024-04-21T08:44:41.251070+0000","last_undegraded":"2024-04-21T08:44:41.251070+0000","last_fullsized":"2024-04-21T08:44:41.251070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:42.802309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374618+0000","last_change":"2024-04-21T08:42:21.655802+0000","last_active":"2024-04-21T08:44:41.374618+0000","last_peered":"2024-04-21T08:44:41.374618+0000","last_clean":"2024-04-21T08:44:41.374618+0000","last_became_active":"2024-04-21T08:42:21.655379+0000","last_became_peered":"2024-04-21T08:42:21.655379+0000","last_unstale":"2024-04-21T08:44:41.374618+0000","last_undegraded":"2024-04-21T08:44:41.374618+0000","last_fullsized":"2024-04-21T08:44:41.374618+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:49.486057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251462+0000","last_change":"2024-04-21T08:42:16.642517+0000","last_active":"2024-04-21T08:44:41.251462+0000","last_peered":"2024-04-21T08:44:41.251462+0000","last_clean":"2024-04-21T08:44:41.251462+0000","last_became_active":"2024-04-21T08:42:16.641539+0000","last_became_peered":"2024-04-21T08:42:16.641539+0000","last_unstale":"2024-04-21T08:44:41.251462+0000","last_undegraded":"2024-04-21T08:44:41.251462+0000","last_fullsized":"2024-04-21T08:44:41.251462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:53.912199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418089+0000","last_change":"2024-04-21T08:42:17.550172+0000","last_active":"2024-04-21T08:44:41.418089+0000","last_peered":"2024-04-21T08:44:41.418089+0000","last_clean":"2024-04-21T08:44:41.418089+0000","last_became_active":"2024-04-21T08:42:17.549824+0000","last_became_peered":"2024-04-21T08:42:17.549824+0000","last_unstale":"2024-04-21T08:44:41.418089+0000","last_undegraded":"2024-04-21T08:44:41.418089+0000","last_fullsized":"2024-04-21T08:44:41.418089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:59.553645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251154+0000","last_change":"2024-04-21T08:44:11.774778+0000","last_active":"2024-04-21T08:44:41.251154+0000","last_peered":"2024-04-21T08:44:41.251154+0000","last_clean":"2024-04-21T08:44:41.251154+0000","last_became_active":"2024-04-21T08:44:11.774190+0000","last_became_peered":"2024-04-21T08:44:11.774190+0000","last_unstale":"2024-04-21T08:44:41.251154+0000","last_undegraded":"2024-04-21T08:44:41.251154+0000","last_fullsized":"2024-04-21T08:44:41.251154+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:06.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374515+0000","last_change":"2024-04-21T08:42:16.647985+0000","last_active":"2024-04-21T08:44:41.374515+0000","last_peered":"2024-04-21T08:44:41.374515+0000","last_clean":"2024-04-21T08:44:41.374515+0000","last_became_active":"2024-04-21T08:42:16.647577+0000","last_became_peered":"2024-04-21T08:42:16.647577+0000","last_unstale":"2024-04-21T08:44:41.374515+0000","last_undegraded":"2024-04-21T08:44:41.374515+0000","last_fullsized":"2024-04-21T08:44:41.374515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:00.731708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250576+0000","last_change":"2024-04-21T08:42:17.541757+0000","last_active":"2024-04-21T08:44:41.250576+0000","last_peered":"2024-04-21T08:44:41.250576+0000","last_clean":"2024-04-21T08:44:41.250576+0000","last_became_active":"2024-04-21T08:42:17.539280+0000","last_became_peered":"2024-04-21T08:42:17.539280+0000","last_unstale":"2024-04-21T08:44:41.250576+0000","last_undegraded":"2024-04-21T08:44:41.250576+0000","last_fullsized":"2024-04-21T08:44:41.250576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:45:54.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356961+0000","last_change":"2024-04-21T08:42:21.664566+0000","last_active":"2024-04-21T08:44:41.356961+0000","last_peered":"2024-04-21T08:44:41.356961+0000","last_clean":"2024-04-21T08:44:41.356961+0000","last_became_active":"2024-04-21T08:42:21.664136+0000","last_became_peered":"2024-04-21T08:42:21.664136+0000","last_unstale":"2024-04-21T08:44:41.356961+0000","last_undegraded":"2024-04-21T08:44:41.356961+0000","last_fullsized":"2024-04-21T08:44:41.356961+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:19.856577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249918+0000","last_change":"2024-04-21T08:42:16.650664+0000","last_active":"2024-04-21T08:44:41.249918+0000","last_peered":"2024-04-21T08:44:41.249918+0000","last_clean":"2024-04-21T08:44:41.249918+0000","last_became_active":"2024-04-21T08:42:16.650339+0000","last_became_peered":"2024-04-21T08:42:16.650339+0000","last_unstale":"2024-04-21T08:44:41.249918+0000","last_undegraded":"2024-04-21T08:44:41.249918+0000","last_fullsized":"2024-04-21T08:44:41.249918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:59.565455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374844+0000","last_change":"2024-04-21T08:42:17.538142+0000","last_active":"2024-04-21T08:44:41.374844+0000","last_peered":"2024-04-21T08:44:41.374844+0000","last_clean":"2024-04-21T08:44:41.374844+0000","last_became_active":"2024-04-21T08:42:17.537793+0000","last_became_peered":"2024-04-21T08:42:17.537793+0000","last_unstale":"2024-04-21T08:44:41.374844+0000","last_undegraded":"2024-04-21T08:44:41.374844+0000","last_fullsized":"2024-04-21T08:44:41.374844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:11.059294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251170+0000","last_change":"2024-04-21T08:42:21.670642+0000","last_active":"2024-04-21T08:44:41.251170+0000","last_peered":"2024-04-21T08:44:41.251170+0000","last_clean":"2024-04-21T08:44:41.251170+0000","last_became_active":"2024-04-21T08:42:21.670128+0000","last_became_peered":"2024-04-21T08:42:21.670128+0000","last_unstale":"2024-04-21T08:44:41.251170+0000","last_undegraded":"2024-04-21T08:44:41.251170+0000","last_fullsized":"2024-04-21T08:44:41.251170+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:51:28.604381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375278+0000","last_change":"2024-04-21T08:43:10.792114+0000","last_active":"2024-04-21T08:44:41.375278+0000","last_peered":"2024-04-21T08:44:41.375278+0000","last_clean":"2024-04-21T08:44:41.375278+0000","last_became_active":"2024-04-21T08:43:10.791664+0000","last_became_peered":"2024-04-21T08:43:10.791664+0000","last_unstale":"2024-04-21T08:44:41.375278+0000","last_undegraded":"2024-04-21T08:44:41.375278+0000","last_fullsized":"2024-04-21T08:44:41.375278+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:07.317804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250083+0000","last_change":"2024-04-21T08:42:17.547326+0000","last_active":"2024-04-21T08:44:41.250083+0000","last_peered":"2024-04-21T08:44:41.250083+0000","last_clean":"2024-04-21T08:44:41.250083+0000","last_became_active":"2024-04-21T08:42:17.546920+0000","last_became_peered":"2024-04-21T08:42:17.546920+0000","last_unstale":"2024-04-21T08:44:41.250083+0000","last_undegraded":"2024-04-21T08:44:41.250083+0000","last_fullsized":"2024-04-21T08:44:41.250083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:43.350877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251167+0000","last_change":"2024-04-21T08:42:21.971147+0000","last_active":"2024-04-21T08:44:41.251167+0000","last_peered":"2024-04-21T08:44:41.251167+0000","last_clean":"2024-04-21T08:44:41.251167+0000","last_became_active":"2024-04-21T08:42:21.970001+0000","last_became_peered":"2024-04-21T08:42:21.970001+0000","last_unstale":"2024-04-21T08:44:41.251167+0000","last_undegraded":"2024-04-21T08:44:41.251167+0000","last_fullsized":"2024-04-21T08:44:41.251167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:01:44.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250876+0000","last_change":"2024-04-21T08:42:16.643371+0000","last_active":"2024-04-21T08:44:41.250876+0000","last_peered":"2024-04-21T08:44:41.250876+0000","last_clean":"2024-04-21T08:44:41.250876+0000","last_became_active":"2024-04-21T08:42:16.642906+0000","last_became_peered":"2024-04-21T08:42:16.642906+0000","last_unstale":"2024-04-21T08:44:41.250876+0000","last_undegraded":"2024-04-21T08:44:41.250876+0000","last_fullsized":"2024-04-21T08:44:41.250876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:58.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375124+0000","last_change":"2024-04-21T08:42:17.545185+0000","last_active":"2024-04-21T08:44:41.375124+0000","last_peered":"2024-04-21T08:44:41.375124+0000","last_clean":"2024-04-21T08:44:41.375124+0000","last_became_active":"2024-04-21T08:42:17.544757+0000","last_became_peered":"2024-04-21T08:42:17.544757+0000","last_unstale":"2024-04-21T08:44:41.375124+0000","last_undegraded":"2024-04-21T08:44:41.375124+0000","last_fullsized":"2024-04-21T08:44:41.375124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:36:18.968041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374923+0000","last_change":"2024-04-21T08:42:21.662541+0000","last_active":"2024-04-21T08:44:41.374923+0000","last_peered":"2024-04-21T08:44:41.374923+0000","last_clean":"2024-04-21T08:44:41.374923+0000","last_became_active":"2024-04-21T08:42:21.662144+0000","last_became_peered":"2024-04-21T08:42:21.662144+0000","last_unstale":"2024-04-21T08:44:41.374923+0000","last_undegraded":"2024-04-21T08:44:41.374923+0000","last_fullsized":"2024-04-21T08:44:41.374923+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:04.347722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375374+0000","last_change":"2024-04-21T08:42:16.638601+0000","last_active":"2024-04-21T08:44:41.375374+0000","last_peered":"2024-04-21T08:44:41.375374+0000","last_clean":"2024-04-21T08:44:41.375374+0000","last_became_active":"2024-04-21T08:42:16.638211+0000","last_became_peered":"2024-04-21T08:42:16.638211+0000","last_unstale":"2024-04-21T08:44:41.375374+0000","last_undegraded":"2024-04-21T08:44:41.375374+0000","last_fullsized":"2024-04-21T08:44:41.375374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:02.633086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375257+0000","last_change":"2024-04-21T08:42:17.545495+0000","last_active":"2024-04-21T08:44:41.375257+0000","last_peered":"2024-04-21T08:44:41.375257+0000","last_clean":"2024-04-21T08:44:41.375257+0000","last_became_active":"2024-04-21T08:42:17.545159+0000","last_became_peered":"2024-04-21T08:42:17.545159+0000","last_unstale":"2024-04-21T08:44:41.375257+0000","last_undegraded":"2024-04-21T08:44:41.375257+0000","last_fullsized":"2024-04-21T08:44:41.375257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:27.150277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417564+0000","last_change":"2024-04-21T08:42:21.656890+0000","last_active":"2024-04-21T08:44:41.417564+0000","last_peered":"2024-04-21T08:44:41.417564+0000","last_clean":"2024-04-21T08:44:41.417564+0000","last_became_active":"2024-04-21T08:42:21.656475+0000","last_became_peered":"2024-04-21T08:42:21.656475+0000","last_unstale":"2024-04-21T08:44:41.417564+0000","last_undegraded":"2024-04-21T08:44:41.417564+0000","last_fullsized":"2024-04-21T08:44:41.417564+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:46.138591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'57","reported_seq":109,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:47.171231+0000","last_change":"2024-04-21T08:42:16.638244+0000","last_active":"2024-04-21T08:44:47.171231+0000","last_peered":"2024-04-21T08:44:47.171231+0000","last_clean":"2024-04-21T08:44:47.171231+0000","last_became_active":"2024-04-21T08:42:16.637883+0000","last_became_peered":"2024-04-21T08:42:16.637883+0000","last_unstale":"2024-04-21T08:44:47.171231+0000","last_undegraded":"2024-04-21T08:44:47.171231+0000","last_fullsized":"2024-04-21T08:44:47.171231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:20.888301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":68,"num_write_kb":807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374936+0000","last_change":"2024-04-21T08:42:17.548032+0000","last_active":"2024-04-21T08:44:41.374936+0000","last_peered":"2024-04-21T08:44:41.374936+0000","last_clean":"2024-04-21T08:44:41.374936+0000","last_became_active":"2024-04-21T08:42:17.547660+0000","last_became_peered":"2024-04-21T08:42:17.547660+0000","last_unstale":"2024-04-21T08:44:41.374936+0000","last_undegraded":"2024-04-21T08:44:41.374936+0000","last_fullsized":"2024-04-21T08:44:41.374936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:14.710759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365005+0000","last_change":"2024-04-21T08:42:21.659286+0000","last_active":"2024-04-21T08:44:41.365005+0000","last_peered":"2024-04-21T08:44:41.365005+0000","last_clean":"2024-04-21T08:44:41.365005+0000","last_became_active":"2024-04-21T08:42:21.658844+0000","last_became_peered":"2024-04-21T08:42:21.658844+0000","last_unstale":"2024-04-21T08:44:41.365005+0000","last_undegraded":"2024-04-21T08:44:41.365005+0000","last_fullsized":"2024-04-21T08:44:41.365005+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:28.742646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376281+0000","last_change":"2024-04-21T08:42:16.643354+0000","last_active":"2024-04-21T08:44:41.376281+0000","last_peered":"2024-04-21T08:44:41.376281+0000","last_clean":"2024-04-21T08:44:41.376281+0000","last_became_active":"2024-04-21T08:42:16.642967+0000","last_became_peered":"2024-04-21T08:42:16.642967+0000","last_unstale":"2024-04-21T08:44:41.376281+0000","last_undegraded":"2024-04-21T08:44:41.376281+0000","last_fullsized":"2024-04-21T08:44:41.376281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:06:03.495181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375409+0000","last_change":"2024-04-21T08:42:17.539551+0000","last_active":"2024-04-21T08:44:41.375409+0000","last_peered":"2024-04-21T08:44:41.375409+0000","last_clean":"2024-04-21T08:44:41.375409+0000","last_became_active":"2024-04-21T08:42:17.539202+0000","last_became_peered":"2024-04-21T08:42:17.539202+0000","last_unstale":"2024-04-21T08:44:41.375409+0000","last_undegraded":"2024-04-21T08:44:41.375409+0000","last_fullsized":"2024-04-21T08:44:41.375409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:37.109421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375023+0000","last_change":"2024-04-21T08:42:17.544582+0000","last_active":"2024-04-21T08:44:41.375023+0000","last_peered":"2024-04-21T08:44:41.375023+0000","last_clean":"2024-04-21T08:44:41.375023+0000","last_became_active":"2024-04-21T08:42:17.543800+0000","last_became_peered":"2024-04-21T08:42:17.543800+0000","last_unstale":"2024-04-21T08:44:41.375023+0000","last_undegraded":"2024-04-21T08:44:41.375023+0000","last_fullsized":"2024-04-21T08:44:41.375023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:44:04.107207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374756+0000","last_change":"2024-04-21T08:42:17.543788+0000","last_active":"2024-04-21T08:44:41.374756+0000","last_peered":"2024-04-21T08:44:41.374756+0000","last_clean":"2024-04-21T08:44:41.374756+0000","last_became_active":"2024-04-21T08:42:17.540939+0000","last_became_peered":"2024-04-21T08:42:17.540939+0000","last_unstale":"2024-04-21T08:44:41.374756+0000","last_undegraded":"2024-04-21T08:44:41.374756+0000","last_fullsized":"2024-04-21T08:44:41.374756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:59.275098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374942+0000","last_change":"2024-04-21T08:42:17.540605+0000","last_active":"2024-04-21T08:44:41.374942+0000","last_peered":"2024-04-21T08:44:41.374942+0000","last_clean":"2024-04-21T08:44:41.374942+0000","last_became_active":"2024-04-21T08:42:17.540073+0000","last_became_peered":"2024-04-21T08:42:17.540073+0000","last_unstale":"2024-04-21T08:44:41.374942+0000","last_undegraded":"2024-04-21T08:44:41.374942+0000","last_fullsized":"2024-04-21T08:44:41.374942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:02:29.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417931+0000","last_change":"2024-04-21T08:42:17.551608+0000","last_active":"2024-04-21T08:44:41.417931+0000","last_peered":"2024-04-21T08:44:41.417931+0000","last_clean":"2024-04-21T08:44:41.417931+0000","last_became_active":"2024-04-21T08:42:17.551117+0000","last_became_peered":"2024-04-21T08:42:17.551117+0000","last_unstale":"2024-04-21T08:44:41.417931+0000","last_undegraded":"2024-04-21T08:44:41.417931+0000","last_fullsized":"2024-04-21T08:44:41.417931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:18.391928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375875+0000","last_change":"2024-04-21T08:42:17.536509+0000","last_active":"2024-04-21T08:44:41.375875+0000","last_peered":"2024-04-21T08:44:41.375875+0000","last_clean":"2024-04-21T08:44:41.375875+0000","last_became_active":"2024-04-21T08:42:17.536146+0000","last_became_peered":"2024-04-21T08:42:17.536146+0000","last_unstale":"2024-04-21T08:44:41.375875+0000","last_undegraded":"2024-04-21T08:44:41.375875+0000","last_fullsized":"2024-04-21T08:44:41.375875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:29.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375199+0000","last_change":"2024-04-21T08:42:17.543786+0000","last_active":"2024-04-21T08:44:41.375199+0000","last_peered":"2024-04-21T08:44:41.375199+0000","last_clean":"2024-04-21T08:44:41.375199+0000","last_became_active":"2024-04-21T08:42:17.543435+0000","last_became_peered":"2024-04-21T08:42:17.543435+0000","last_unstale":"2024-04-21T08:44:41.375199+0000","last_undegraded":"2024-04-21T08:44:41.375199+0000","last_fullsized":"2024-04-21T08:44:41.375199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375616+0000","last_change":"2024-04-21T08:42:17.541611+0000","last_active":"2024-04-21T08:44:41.375616+0000","last_peered":"2024-04-21T08:44:41.375616+0000","last_clean":"2024-04-21T08:44:41.375616+0000","last_became_active":"2024-04-21T08:42:17.541259+0000","last_became_peered":"2024-04-21T08:42:17.541259+0000","last_unstale":"2024-04-21T08:44:41.375616+0000","last_undegraded":"2024-04-21T08:44:41.375616+0000","last_fullsized":"2024-04-21T08:44:41.375616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:20.900629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356654+0000","last_change":"2024-04-21T08:44:11.774062+0000","last_active":"2024-04-21T08:44:41.356654+0000","last_peered":"2024-04-21T08:44:41.356654+0000","last_clean":"2024-04-21T08:44:41.356654+0000","last_became_active":"2024-04-21T08:44:11.773703+0000","last_became_peered":"2024-04-21T08:44:11.773703+0000","last_unstale":"2024-04-21T08:44:41.356654+0000","last_undegraded":"2024-04-21T08:44:41.356654+0000","last_fullsized":"2024-04-21T08:44:41.356654+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:10.739068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365197+0000","last_change":"2024-04-21T08:42:17.545526+0000","last_active":"2024-04-21T08:44:41.365197+0000","last_peered":"2024-04-21T08:44:41.365197+0000","last_clean":"2024-04-21T08:44:41.365197+0000","last_became_active":"2024-04-21T08:42:17.545135+0000","last_became_peered":"2024-04-21T08:42:17.545135+0000","last_unstale":"2024-04-21T08:44:41.365197+0000","last_undegraded":"2024-04-21T08:44:41.365197+0000","last_fullsized":"2024-04-21T08:44:41.365197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:37:58.038459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250029+0000","last_change":"2024-04-21T08:42:17.547063+0000","last_active":"2024-04-21T08:44:41.250029+0000","last_peered":"2024-04-21T08:44:41.250029+0000","last_clean":"2024-04-21T08:44:41.250029+0000","last_became_active":"2024-04-21T08:42:17.546689+0000","last_became_peered":"2024-04-21T08:42:17.546689+0000","last_unstale":"2024-04-21T08:44:41.250029+0000","last_undegraded":"2024-04-21T08:44:41.250029+0000","last_fullsized":"2024-04-21T08:44:41.250029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:26.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375261+0000","last_change":"2024-04-21T08:42:17.542545+0000","last_active":"2024-04-21T08:44:41.375261+0000","last_peered":"2024-04-21T08:44:41.375261+0000","last_clean":"2024-04-21T08:44:41.375261+0000","last_became_active":"2024-04-21T08:42:17.542147+0000","last_became_peered":"2024-04-21T08:42:17.542147+0000","last_unstale":"2024-04-21T08:44:41.375261+0000","last_undegraded":"2024-04-21T08:44:41.375261+0000","last_fullsized":"2024-04-21T08:44:41.375261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:15.064110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376582+0000","last_change":"2024-04-21T08:42:17.541262+0000","last_active":"2024-04-21T08:44:41.376582+0000","last_peered":"2024-04-21T08:44:41.376582+0000","last_clean":"2024-04-21T08:44:41.376582+0000","last_became_active":"2024-04-21T08:42:17.540817+0000","last_became_peered":"2024-04-21T08:42:17.540817+0000","last_unstale":"2024-04-21T08:44:41.376582+0000","last_undegraded":"2024-04-21T08:44:41.376582+0000","last_fullsized":"2024-04-21T08:44:41.376582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:22:59.597680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375116+0000","last_change":"2024-04-21T08:42:17.551109+0000","last_active":"2024-04-21T08:44:41.375116+0000","last_peered":"2024-04-21T08:44:41.375116+0000","last_clean":"2024-04-21T08:44:41.375116+0000","last_became_active":"2024-04-21T08:42:17.550104+0000","last_became_peered":"2024-04-21T08:42:17.550104+0000","last_unstale":"2024-04-21T08:44:41.375116+0000","last_undegraded":"2024-04-21T08:44:41.375116+0000","last_fullsized":"2024-04-21T08:44:41.375116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:22:19.361752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107298,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":792314,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":472,"num_read_kb":387,"num_write":265,"num_write_kb":2063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":2376942,"data_compressed":107241,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351019,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":532412,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36500,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708652,"statfs":{"total":95995035648,"available":95957659648,"internally_reserved":0,"allocated":1187840,"data_stored":1320546,"data_compressed":73280,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1069056,"data_stored":930306,"data_compressed":48872,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037819,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":926232,"data_compressed":49001,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299455,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":1127458,"data_compressed":65492,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462982,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528316,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1105920,"data_stored":598384,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1036288,"data_stored":528406,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":598418,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1053236,"data_compressed":55028,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888144,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":532434,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724618,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":1254562,"data_compressed":71648,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70102,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":721552,"data_compressed":34115,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":328414,"data_compressed":14840,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":398494,"data_compressed":16472,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:44:51.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:44:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.028+0000 7f73d6f7e640 1 -- 172.21.15.7:0/1026685244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73d0105dd0 msgr2=0x7f73d01061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.028+0000 7f73d6f7e640 1 --2- 172.21.15.7:0/1026685244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73d0105dd0 0x7f73d01061b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009900 tx=0x7f73c402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.028+0000 7f73d6f7e640 1 -- 172.21.15.7:0/1026685244 shutdown_connections 2024-04-21T08:44:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.028+0000 7f73d6f7e640 1 --2- 172.21.15.7:0/1026685244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73d0101a40 0x7f73d010dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.028+0000 7f73d6f7e640 1 --2- 172.21.15.7:0/1026685244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01010a0 0x7f73d0101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.028+0000 7f73d6f7e640 1 --2- 172.21.15.7:0/1026685244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73d0105dd0 0x7f73d01061b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.028+0000 7f73d6f7e640 1 -- 172.21.15.7:0/1026685244 >> 172.21.15.7:0/1026685244 conn(0x7f73d0077c40 msgr2=0x7f73d0078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.028+0000 7f73d6f7e640 1 -- 172.21.15.7:0/1026685244 shutdown_connections 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.028+0000 7f73d6f7e640 1 -- 172.21.15.7:0/1026685244 wait complete. 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73d6f7e640 1 Processor -- start 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73d6f7e640 1 -- start start 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73d6f7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01010a0 0x7f73d019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73d6f7e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73d0101a40 0x7f73d019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73d6f7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73d0105dd0 0x7f73d0198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73d6f7e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f73d01122f0 con 0x7f73d0101a40 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73d6f7e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f73d0112170 con 0x7f73d01010a0 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73d6f7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f73d0112470 con 0x7f73d0105dd0 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73cffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73d0101a40 0x7f73d019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73cffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73d0101a40 0x7f73d019ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47568/0 (socket says 172.21.15.7:47568) 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73cffff640 1 -- 172.21.15.7:0/111399378 learned_addr learned my addr 172.21.15.7:0/111399378 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73d4d1a640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01010a0 0x7f73d019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73d551b640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73d0105dd0 0x7f73d0198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73cffff640 1 -- 172.21.15.7:0/111399378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73d0105dd0 msgr2=0x7f73d0198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73cffff640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73d0105dd0 0x7f73d0198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73cffff640 1 -- 172.21.15.7:0/111399378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01010a0 msgr2=0x7f73d019e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73cffff640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01010a0 0x7f73d019e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73cffff640 1 -- 172.21.15.7:0/111399378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d0198f10 con 0x7f73d0101a40 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.029+0000 7f73d4d1a640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01010a0 0x7f73d019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.030+0000 7f73cffff640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73d0101a40 0x7f73d019ead0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00b4a0 tx=0x7f73bc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.030+0000 7f73cdffb640 1 -- 172.21.15.7:0/111399378 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc014070 con 0x7f73d0101a40 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.030+0000 7f73d6f7e640 1 -- 172.21.15.7:0/111399378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73d0199200 con 0x7f73d0101a40 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.030+0000 7f73cdffb640 1 -- 172.21.15.7:0/111399378 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73bc010ac0 con 0x7f73d0101a40 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.030+0000 7f73cdffb640 1 -- 172.21.15.7:0/111399378 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc017c30 con 0x7f73d0101a40 2024-04-21T08:44:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.030+0000 7f73d6f7e640 1 -- 172.21.15.7:0/111399378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73d01a5510 con 0x7f73d0101a40 2024-04-21T08:44:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.031+0000 7f73d6f7e640 1 -- 172.21.15.7:0/111399378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73d0069e50 con 0x7f73d0101a40 2024-04-21T08:44:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.032+0000 7f73cdffb640 1 -- 172.21.15.7:0/111399378 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f73bc017070 con 0x7f73d0101a40 2024-04-21T08:44:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.033+0000 7f73cdffb640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f73c00a3db0 0x7f73c00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.033+0000 7f73d4d1a640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f73c00a3db0 0x7f73c00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.033+0000 7f73cdffb640 1 -- 172.21.15.7:0/111399378 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f73bc0d0480 con 0x7f73d0101a40 2024-04-21T08:44:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.033+0000 7f73d4d1a640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f73c00a3db0 0x7f73c00a6270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f73c4005ab0 tx=0x7f73c4005a20 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.037+0000 7f73cdffb640 1 -- 172.21.15.7:0/111399378 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f73bc09c670 con 0x7f73d0101a40 2024-04-21T08:44:52.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:51 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2325493252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:52.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:51 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3206156897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:52.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:51 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:44:52.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:51 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2622639273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:52.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:51 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2325493252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:52.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:51 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3206156897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:52.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:51 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:44:52.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:51 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2622639273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:52.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:51 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2325493252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:52.206 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:51 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3206156897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:52.206 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:51 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:44:52.206 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:51 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2622639273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:52.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.221+0000 7f73d6f7e640 1 -- 172.21.15.7:0/111399378 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f73d0062390 con 0x7f73c00a3db0 2024-04-21T08:44:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.230+0000 7f73cdffb640 1 -- 172.21.15.7:0/111399378 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339632 (secure 0 0 0) 0x7f73d0062390 con 0x7f73c00a3db0 2024-04-21T08:44:52.233 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:52.235 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-21T08:44:52.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 -- 172.21.15.7:0/111399378 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f73c00a3db0 msgr2=0x7f73c00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f73c00a3db0 0x7f73c00a6270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f73c4005ab0 tx=0x7f73c4005a20 comp rx=0 tx=0).stop 2024-04-21T08:44:52.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 -- 172.21.15.7:0/111399378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73d0101a40 msgr2=0x7f73d019ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73d0101a40 0x7f73d019ead0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00b4a0 tx=0x7f73bc00b960 comp rx=0 tx=0).stop 2024-04-21T08:44:52.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 -- 172.21.15.7:0/111399378 shutdown_connections 2024-04-21T08:44:52.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f73c00a3db0 0x7f73c00a6270 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73d0105dd0 0x7f73d0198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73d0101a40 0x7f73d019ead0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 --2- 172.21.15.7:0/111399378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73d01010a0 0x7f73d019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 -- 172.21.15.7:0/111399378 >> 172.21.15.7:0/111399378 conn(0x7f73d0077c40 msgr2=0x7f73d00fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:52.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 -- 172.21.15.7:0/111399378 shutdown_connections 2024-04-21T08:44:52.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.234+0000 7f73d6f7e640 1 -- 172.21.15.7:0/111399378 wait complete. 2024-04-21T08:44:52.244 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-21T08:44:51.264294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1382916,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":709,"num_read_kb":717,"num_write":468,"num_write_kb":5003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":844,"ondisk_log_size":844,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":403456,"kb_used_data":12680,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124538368,"statfs":{"total":1151940427776,"available":1151527288832,"internally_reserved":0,"allocated":12984320,"data_stored":9930710,"data_compressed":544941,"data_compressed_allocated":737280,"data_compressed_original":4325376,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":223,"num_read_kb":318,"num_write":84,"num_write_kb":652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.991246"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375541+0000","last_change":"2024-04-21T08:42:17.545701+0000","last_active":"2024-04-21T08:44:41.375541+0000","last_peered":"2024-04-21T08:44:41.375541+0000","last_clean":"2024-04-21T08:44:41.375541+0000","last_became_active":"2024-04-21T08:42:17.545157+0000","last_became_peered":"2024-04-21T08:42:17.545157+0000","last_unstale":"2024-04-21T08:44:41.375541+0000","last_undegraded":"2024-04-21T08:44:41.375541+0000","last_fullsized":"2024-04-21T08:44:41.375541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:05:41.952879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418165+0000","last_change":"2024-04-21T08:42:17.541153+0000","last_active":"2024-04-21T08:44:41.418165+0000","last_peered":"2024-04-21T08:44:41.418165+0000","last_clean":"2024-04-21T08:44:41.418165+0000","last_became_active":"2024-04-21T08:42:17.540771+0000","last_became_peered":"2024-04-21T08:42:17.540771+0000","last_unstale":"2024-04-21T08:44:41.418165+0000","last_undegraded":"2024-04-21T08:44:41.418165+0000","last_fullsized":"2024-04-21T08:44:41.418165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:35:06.077367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417993+0000","last_change":"2024-04-21T08:42:17.545525+0000","last_active":"2024-04-21T08:44:41.417993+0000","last_peered":"2024-04-21T08:44:41.417993+0000","last_clean":"2024-04-21T08:44:41.417993+0000","last_became_active":"2024-04-21T08:42:17.545168+0000","last_became_peered":"2024-04-21T08:42:17.545168+0000","last_unstale":"2024-04-21T08:44:41.417993+0000","last_undegraded":"2024-04-21T08:44:41.417993+0000","last_fullsized":"2024-04-21T08:44:41.417993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:51.080385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251479+0000","last_change":"2024-04-21T08:42:17.543917+0000","last_active":"2024-04-21T08:44:41.251479+0000","last_peered":"2024-04-21T08:44:41.251479+0000","last_clean":"2024-04-21T08:44:41.251479+0000","last_became_active":"2024-04-21T08:42:17.543513+0000","last_became_peered":"2024-04-21T08:42:17.543513+0000","last_unstale":"2024-04-21T08:44:41.251479+0000","last_undegraded":"2024-04-21T08:44:41.251479+0000","last_fullsized":"2024-04-21T08:44:41.251479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:23.058600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250648+0000","last_change":"2024-04-21T08:42:17.549737+0000","last_active":"2024-04-21T08:44:41.250648+0000","last_peered":"2024-04-21T08:44:41.250648+0000","last_clean":"2024-04-21T08:44:41.250648+0000","last_became_active":"2024-04-21T08:42:17.549396+0000","last_became_peered":"2024-04-21T08:42:17.549396+0000","last_unstale":"2024-04-21T08:44:41.250648+0000","last_undegraded":"2024-04-21T08:44:41.250648+0000","last_fullsized":"2024-04-21T08:44:41.250648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:01:51.879960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368365+0000","last_change":"2024-04-21T08:43:10.792321+0000","last_active":"2024-04-21T08:44:41.368365+0000","last_peered":"2024-04-21T08:44:41.368365+0000","last_clean":"2024-04-21T08:44:41.368365+0000","last_became_active":"2024-04-21T08:43:10.791979+0000","last_became_peered":"2024-04-21T08:43:10.791979+0000","last_unstale":"2024-04-21T08:44:41.368365+0000","last_undegraded":"2024-04-21T08:44:41.368365+0000","last_fullsized":"2024-04-21T08:44:41.368365+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:11:16.056778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374960+0000","last_change":"2024-04-21T08:42:17.541172+0000","last_active":"2024-04-21T08:44:41.374960+0000","last_peered":"2024-04-21T08:44:41.374960+0000","last_clean":"2024-04-21T08:44:41.374960+0000","last_became_active":"2024-04-21T08:42:17.540829+0000","last_became_peered":"2024-04-21T08:42:17.540829+0000","last_unstale":"2024-04-21T08:44:41.374960+0000","last_undegraded":"2024-04-21T08:44:41.374960+0000","last_fullsized":"2024-04-21T08:44:41.374960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:16:12.633980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249835+0000","last_change":"2024-04-21T08:42:17.549744+0000","last_active":"2024-04-21T08:44:41.249835+0000","last_peered":"2024-04-21T08:44:41.249835+0000","last_clean":"2024-04-21T08:44:41.249835+0000","last_became_active":"2024-04-21T08:42:17.549378+0000","last_became_peered":"2024-04-21T08:42:17.549378+0000","last_unstale":"2024-04-21T08:44:41.249835+0000","last_undegraded":"2024-04-21T08:44:41.249835+0000","last_fullsized":"2024-04-21T08:44:41.249835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:03:07.702143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375819+0000","last_change":"2024-04-21T08:42:17.548675+0000","last_active":"2024-04-21T08:44:41.375819+0000","last_peered":"2024-04-21T08:44:41.375819+0000","last_clean":"2024-04-21T08:44:41.375819+0000","last_became_active":"2024-04-21T08:42:17.547809+0000","last_became_peered":"2024-04-21T08:42:17.547809+0000","last_unstale":"2024-04-21T08:44:41.375819+0000","last_undegraded":"2024-04-21T08:44:41.375819+0000","last_fullsized":"2024-04-21T08:44:41.375819+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:23.343473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368808+0000","last_change":"2024-04-21T08:42:17.551012+0000","last_active":"2024-04-21T08:44:41.368808+0000","last_peered":"2024-04-21T08:44:41.368808+0000","last_clean":"2024-04-21T08:44:41.368808+0000","last_became_active":"2024-04-21T08:42:17.550627+0000","last_became_peered":"2024-04-21T08:42:17.550627+0000","last_unstale":"2024-04-21T08:44:41.368808+0000","last_undegraded":"2024-04-21T08:44:41.368808+0000","last_fullsized":"2024-04-21T08:44:41.368808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:57.501588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375100+0000","last_change":"2024-04-21T08:42:17.546192+0000","last_active":"2024-04-21T08:44:41.375100+0000","last_peered":"2024-04-21T08:44:41.375100+0000","last_clean":"2024-04-21T08:44:41.375100+0000","last_became_active":"2024-04-21T08:42:17.545808+0000","last_became_peered":"2024-04-21T08:42:17.545808+0000","last_unstale":"2024-04-21T08:44:41.375100+0000","last_undegraded":"2024-04-21T08:44:41.375100+0000","last_fullsized":"2024-04-21T08:44:41.375100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:54.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251368+0000","last_change":"2024-04-21T08:42:17.553351+0000","last_active":"2024-04-21T08:44:41.251368+0000","last_peered":"2024-04-21T08:44:41.251368+0000","last_clean":"2024-04-21T08:44:41.251368+0000","last_became_active":"2024-04-21T08:42:17.550507+0000","last_became_peered":"2024-04-21T08:42:17.550507+0000","last_unstale":"2024-04-21T08:44:41.251368+0000","last_undegraded":"2024-04-21T08:44:41.251368+0000","last_fullsized":"2024-04-21T08:44:41.251368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:25.561375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376794+0000","last_change":"2024-04-21T08:42:17.540533+0000","last_active":"2024-04-21T08:44:41.376794+0000","last_peered":"2024-04-21T08:44:41.376794+0000","last_clean":"2024-04-21T08:44:41.376794+0000","last_became_active":"2024-04-21T08:42:17.539940+0000","last_became_peered":"2024-04-21T08:42:17.539940+0000","last_unstale":"2024-04-21T08:44:41.376794+0000","last_undegraded":"2024-04-21T08:44:41.376794+0000","last_fullsized":"2024-04-21T08:44:41.376794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:45:19.102899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417676+0000","last_change":"2024-04-21T08:42:17.537794+0000","last_active":"2024-04-21T08:44:41.417676+0000","last_peered":"2024-04-21T08:44:41.417676+0000","last_clean":"2024-04-21T08:44:41.417676+0000","last_became_active":"2024-04-21T08:42:17.537376+0000","last_became_peered":"2024-04-21T08:42:17.537376+0000","last_unstale":"2024-04-21T08:44:41.417676+0000","last_undegraded":"2024-04-21T08:44:41.417676+0000","last_fullsized":"2024-04-21T08:44:41.417676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:57.430036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417515+0000","last_change":"2024-04-21T08:42:17.550574+0000","last_active":"2024-04-21T08:44:41.417515+0000","last_peered":"2024-04-21T08:44:41.417515+0000","last_clean":"2024-04-21T08:44:41.417515+0000","last_became_active":"2024-04-21T08:42:17.550205+0000","last_became_peered":"2024-04-21T08:42:17.550205+0000","last_unstale":"2024-04-21T08:44:41.417515+0000","last_undegraded":"2024-04-21T08:44:41.417515+0000","last_fullsized":"2024-04-21T08:44:41.417515+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:41:08.226477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374544+0000","last_change":"2024-04-21T08:42:17.542239+0000","last_active":"2024-04-21T08:44:41.374544+0000","last_peered":"2024-04-21T08:44:41.374544+0000","last_clean":"2024-04-21T08:44:41.374544+0000","last_became_active":"2024-04-21T08:42:17.541848+0000","last_became_peered":"2024-04-21T08:42:17.541848+0000","last_unstale":"2024-04-21T08:44:41.374544+0000","last_undegraded":"2024-04-21T08:44:41.374544+0000","last_fullsized":"2024-04-21T08:44:41.374544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:47.717451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375417+0000","last_change":"2024-04-21T08:42:17.553200+0000","last_active":"2024-04-21T08:44:41.375417+0000","last_peered":"2024-04-21T08:44:41.375417+0000","last_clean":"2024-04-21T08:44:41.375417+0000","last_became_active":"2024-04-21T08:42:17.552913+0000","last_became_peered":"2024-04-21T08:42:17.552913+0000","last_unstale":"2024-04-21T08:44:41.375417+0000","last_undegraded":"2024-04-21T08:44:41.375417+0000","last_fullsized":"2024-04-21T08:44:41.375417+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:39:04.186928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356904+0000","last_change":"2024-04-21T08:42:17.534881+0000","last_active":"2024-04-21T08:44:41.356904+0000","last_peered":"2024-04-21T08:44:41.356904+0000","last_clean":"2024-04-21T08:44:41.356904+0000","last_became_active":"2024-04-21T08:42:17.534496+0000","last_became_peered":"2024-04-21T08:42:17.534496+0000","last_unstale":"2024-04-21T08:44:41.356904+0000","last_undegraded":"2024-04-21T08:44:41.356904+0000","last_fullsized":"2024-04-21T08:44:41.356904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:50:19.409391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375734+0000","last_change":"2024-04-21T08:42:17.540480+0000","last_active":"2024-04-21T08:44:41.375734+0000","last_peered":"2024-04-21T08:44:41.375734+0000","last_clean":"2024-04-21T08:44:41.375734+0000","last_became_active":"2024-04-21T08:42:17.539742+0000","last_became_peered":"2024-04-21T08:42:17.539742+0000","last_unstale":"2024-04-21T08:44:41.375734+0000","last_undegraded":"2024-04-21T08:44:41.375734+0000","last_fullsized":"2024-04-21T08:44:41.375734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:00.238976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374953+0000","last_change":"2024-04-21T08:44:11.876826+0000","last_active":"2024-04-21T08:44:41.374953+0000","last_peered":"2024-04-21T08:44:41.374953+0000","last_clean":"2024-04-21T08:44:41.374953+0000","last_became_active":"2024-04-21T08:44:11.876354+0000","last_became_peered":"2024-04-21T08:44:11.876354+0000","last_unstale":"2024-04-21T08:44:41.374953+0000","last_undegraded":"2024-04-21T08:44:41.374953+0000","last_fullsized":"2024-04-21T08:44:41.374953+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:00:05.574008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356838+0000","last_change":"2024-04-21T08:42:16.639888+0000","last_active":"2024-04-21T08:44:41.356838+0000","last_peered":"2024-04-21T08:44:41.356838+0000","last_clean":"2024-04-21T08:44:41.356838+0000","last_became_active":"2024-04-21T08:42:16.639403+0000","last_became_peered":"2024-04-21T08:42:16.639403+0000","last_unstale":"2024-04-21T08:44:41.356838+0000","last_undegraded":"2024-04-21T08:44:41.356838+0000","last_fullsized":"2024-04-21T08:44:41.356838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:08:17.164924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249935+0000","last_change":"2024-04-21T08:43:10.983368+0000","last_active":"2024-04-21T08:44:41.249935+0000","last_peered":"2024-04-21T08:44:41.249935+0000","last_clean":"2024-04-21T08:44:41.249935+0000","last_became_active":"2024-04-21T08:43:10.982956+0000","last_became_peered":"2024-04-21T08:43:10.982956+0000","last_unstale":"2024-04-21T08:44:41.249935+0000","last_undegraded":"2024-04-21T08:44:41.249935+0000","last_fullsized":"2024-04-21T08:44:41.249935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:12:44.919996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250779+0000","last_change":"2024-04-21T08:42:21.673197+0000","last_active":"2024-04-21T08:44:41.250779+0000","last_peered":"2024-04-21T08:44:41.250779+0000","last_clean":"2024-04-21T08:44:41.250779+0000","last_became_active":"2024-04-21T08:42:21.672793+0000","last_became_peered":"2024-04-21T08:42:21.672793+0000","last_unstale":"2024-04-21T08:44:41.250779+0000","last_undegraded":"2024-04-21T08:44:41.250779+0000","last_fullsized":"2024-04-21T08:44:41.250779+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:44:29.788459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250990+0000","last_change":"2024-04-21T08:42:16.648577+0000","last_active":"2024-04-21T08:44:41.250990+0000","last_peered":"2024-04-21T08:44:41.250990+0000","last_clean":"2024-04-21T08:44:41.250990+0000","last_became_active":"2024-04-21T08:42:16.646716+0000","last_became_peered":"2024-04-21T08:42:16.646716+0000","last_unstale":"2024-04-21T08:44:41.250990+0000","last_undegraded":"2024-04-21T08:44:41.250990+0000","last_fullsized":"2024-04-21T08:44:41.250990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:10.327078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417936+0000","last_change":"2024-04-21T08:42:17.545430+0000","last_active":"2024-04-21T08:44:41.417936+0000","last_peered":"2024-04-21T08:44:41.417936+0000","last_clean":"2024-04-21T08:44:41.417936+0000","last_became_active":"2024-04-21T08:42:17.545057+0000","last_became_peered":"2024-04-21T08:42:17.545057+0000","last_unstale":"2024-04-21T08:44:41.417936+0000","last_undegraded":"2024-04-21T08:44:41.417936+0000","last_fullsized":"2024-04-21T08:44:41.417936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:44:59.030830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250899+0000","last_change":"2024-04-21T08:42:21.673196+0000","last_active":"2024-04-21T08:44:41.250899+0000","last_peered":"2024-04-21T08:44:41.250899+0000","last_clean":"2024-04-21T08:44:41.250899+0000","last_became_active":"2024-04-21T08:42:21.672808+0000","last_became_peered":"2024-04-21T08:42:21.672808+0000","last_unstale":"2024-04-21T08:44:41.250899+0000","last_undegraded":"2024-04-21T08:44:41.250899+0000","last_fullsized":"2024-04-21T08:44:41.250899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:39.482353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250844+0000","last_change":"2024-04-21T08:42:16.641611+0000","last_active":"2024-04-21T08:44:41.250844+0000","last_peered":"2024-04-21T08:44:41.250844+0000","last_clean":"2024-04-21T08:44:41.250844+0000","last_became_active":"2024-04-21T08:42:16.641221+0000","last_became_peered":"2024-04-21T08:42:16.641221+0000","last_unstale":"2024-04-21T08:44:41.250844+0000","last_undegraded":"2024-04-21T08:44:41.250844+0000","last_fullsized":"2024-04-21T08:44:41.250844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:45.496532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375448+0000","last_change":"2024-04-21T08:42:17.532986+0000","last_active":"2024-04-21T08:44:41.375448+0000","last_peered":"2024-04-21T08:44:41.375448+0000","last_clean":"2024-04-21T08:44:41.375448+0000","last_became_active":"2024-04-21T08:42:17.532583+0000","last_became_peered":"2024-04-21T08:42:17.532583+0000","last_unstale":"2024-04-21T08:44:41.375448+0000","last_undegraded":"2024-04-21T08:44:41.375448+0000","last_fullsized":"2024-04-21T08:44:41.375448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:43.780935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250791+0000","last_change":"2024-04-21T08:42:21.667459+0000","last_active":"2024-04-21T08:44:41.250791+0000","last_peered":"2024-04-21T08:44:41.250791+0000","last_clean":"2024-04-21T08:44:41.250791+0000","last_became_active":"2024-04-21T08:42:21.667183+0000","last_became_peered":"2024-04-21T08:42:21.667183+0000","last_unstale":"2024-04-21T08:44:41.250791+0000","last_undegraded":"2024-04-21T08:44:41.250791+0000","last_fullsized":"2024-04-21T08:44:41.250791+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:32:23.626267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250969+0000","last_change":"2024-04-21T08:42:16.645210+0000","last_active":"2024-04-21T08:44:41.250969+0000","last_peered":"2024-04-21T08:44:41.250969+0000","last_clean":"2024-04-21T08:44:41.250969+0000","last_became_active":"2024-04-21T08:42:16.644747+0000","last_became_peered":"2024-04-21T08:42:16.644747+0000","last_unstale":"2024-04-21T08:44:41.250969+0000","last_undegraded":"2024-04-21T08:44:41.250969+0000","last_fullsized":"2024-04-21T08:44:41.250969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:34:45.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365569+0000","last_change":"2024-04-21T08:43:10.984152+0000","last_active":"2024-04-21T08:44:41.365569+0000","last_peered":"2024-04-21T08:44:41.365569+0000","last_clean":"2024-04-21T08:44:41.365569+0000","last_became_active":"2024-04-21T08:43:10.983736+0000","last_became_peered":"2024-04-21T08:43:10.983736+0000","last_unstale":"2024-04-21T08:44:41.365569+0000","last_undegraded":"2024-04-21T08:44:41.365569+0000","last_fullsized":"2024-04-21T08:44:41.365569+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:10.182667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375391+0000","last_change":"2024-04-21T08:42:21.660739+0000","last_active":"2024-04-21T08:44:41.375391+0000","last_peered":"2024-04-21T08:44:41.375391+0000","last_clean":"2024-04-21T08:44:41.375391+0000","last_became_active":"2024-04-21T08:42:21.660307+0000","last_became_peered":"2024-04-21T08:42:21.660307+0000","last_unstale":"2024-04-21T08:44:41.375391+0000","last_undegraded":"2024-04-21T08:44:41.375391+0000","last_fullsized":"2024-04-21T08:44:41.375391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:26.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365469+0000","last_change":"2024-04-21T08:43:10.986184+0000","last_active":"2024-04-21T08:44:41.365469+0000","last_peered":"2024-04-21T08:44:41.365469+0000","last_clean":"2024-04-21T08:44:41.365469+0000","last_became_active":"2024-04-21T08:43:10.985533+0000","last_became_peered":"2024-04-21T08:43:10.985533+0000","last_unstale":"2024-04-21T08:44:41.365469+0000","last_undegraded":"2024-04-21T08:44:41.365469+0000","last_fullsized":"2024-04-21T08:44:41.365469+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:26.023306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250741+0000","last_change":"2024-04-21T08:42:17.542315+0000","last_active":"2024-04-21T08:44:41.250741+0000","last_peered":"2024-04-21T08:44:41.250741+0000","last_clean":"2024-04-21T08:44:41.250741+0000","last_became_active":"2024-04-21T08:42:17.541895+0000","last_became_peered":"2024-04-21T08:42:17.541895+0000","last_unstale":"2024-04-21T08:44:41.250741+0000","last_undegraded":"2024-04-21T08:44:41.250741+0000","last_fullsized":"2024-04-21T08:44:41.250741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:23:33.909499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375270+0000","last_change":"2024-04-21T08:44:11.878135+0000","last_active":"2024-04-21T08:44:41.375270+0000","last_peered":"2024-04-21T08:44:41.375270+0000","last_clean":"2024-04-21T08:44:41.375270+0000","last_became_active":"2024-04-21T08:44:11.877271+0000","last_became_peered":"2024-04-21T08:44:11.877271+0000","last_unstale":"2024-04-21T08:44:41.375270+0000","last_undegraded":"2024-04-21T08:44:41.375270+0000","last_fullsized":"2024-04-21T08:44:41.375270+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:31.345519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368426+0000","last_change":"2024-04-21T08:43:10.984839+0000","last_active":"2024-04-21T08:44:41.368426+0000","last_peered":"2024-04-21T08:44:41.368426+0000","last_clean":"2024-04-21T08:44:41.368426+0000","last_became_active":"2024-04-21T08:43:10.984382+0000","last_became_peered":"2024-04-21T08:43:10.984382+0000","last_unstale":"2024-04-21T08:44:41.368426+0000","last_undegraded":"2024-04-21T08:44:41.368426+0000","last_fullsized":"2024-04-21T08:44:41.368426+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:47:20.774102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365597+0000","last_change":"2024-04-21T08:42:17.547770+0000","last_active":"2024-04-21T08:44:41.365597+0000","last_peered":"2024-04-21T08:44:41.365597+0000","last_clean":"2024-04-21T08:44:41.365597+0000","last_became_active":"2024-04-21T08:42:17.547440+0000","last_became_peered":"2024-04-21T08:42:17.547440+0000","last_unstale":"2024-04-21T08:44:41.365597+0000","last_undegraded":"2024-04-21T08:44:41.365597+0000","last_fullsized":"2024-04-21T08:44:41.365597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:37:35.555737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375243+0000","last_change":"2024-04-21T08:42:21.670684+0000","last_active":"2024-04-21T08:44:41.375243+0000","last_peered":"2024-04-21T08:44:41.375243+0000","last_clean":"2024-04-21T08:44:41.375243+0000","last_became_active":"2024-04-21T08:42:21.670260+0000","last_became_peered":"2024-04-21T08:42:21.670260+0000","last_unstale":"2024-04-21T08:44:41.375243+0000","last_undegraded":"2024-04-21T08:44:41.375243+0000","last_fullsized":"2024-04-21T08:44:41.375243+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:45:06.192040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376149+0000","last_change":"2024-04-21T08:42:16.639632+0000","last_active":"2024-04-21T08:44:41.376149+0000","last_peered":"2024-04-21T08:44:41.376149+0000","last_clean":"2024-04-21T08:44:41.376149+0000","last_became_active":"2024-04-21T08:42:16.639301+0000","last_became_peered":"2024-04-21T08:42:16.639301+0000","last_unstale":"2024-04-21T08:44:41.376149+0000","last_undegraded":"2024-04-21T08:44:41.376149+0000","last_fullsized":"2024-04-21T08:44:41.376149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:58.838667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249940+0000","last_change":"2024-04-21T08:42:17.546109+0000","last_active":"2024-04-21T08:44:41.249940+0000","last_peered":"2024-04-21T08:44:41.249940+0000","last_clean":"2024-04-21T08:44:41.249940+0000","last_became_active":"2024-04-21T08:42:17.545673+0000","last_became_peered":"2024-04-21T08:42:17.545673+0000","last_unstale":"2024-04-21T08:44:41.249940+0000","last_undegraded":"2024-04-21T08:44:41.249940+0000","last_fullsized":"2024-04-21T08:44:41.249940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:18.947848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417985+0000","last_change":"2024-04-21T08:42:21.666114+0000","last_active":"2024-04-21T08:44:41.417985+0000","last_peered":"2024-04-21T08:44:41.417985+0000","last_clean":"2024-04-21T08:44:41.417985+0000","last_became_active":"2024-04-21T08:42:21.665020+0000","last_became_peered":"2024-04-21T08:42:21.665020+0000","last_unstale":"2024-04-21T08:44:41.417985+0000","last_undegraded":"2024-04-21T08:44:41.417985+0000","last_fullsized":"2024-04-21T08:44:41.417985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:33.644178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365224+0000","last_change":"2024-04-21T08:42:16.639118+0000","last_active":"2024-04-21T08:44:41.365224+0000","last_peered":"2024-04-21T08:44:41.365224+0000","last_clean":"2024-04-21T08:44:41.365224+0000","last_became_active":"2024-04-21T08:42:16.638792+0000","last_became_peered":"2024-04-21T08:42:16.638792+0000","last_unstale":"2024-04-21T08:44:41.365224+0000","last_undegraded":"2024-04-21T08:44:41.365224+0000","last_fullsized":"2024-04-21T08:44:41.365224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:55.568712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356215+0000","last_change":"2024-04-21T08:42:17.539688+0000","last_active":"2024-04-21T08:44:41.356215+0000","last_peered":"2024-04-21T08:44:41.356215+0000","last_clean":"2024-04-21T08:44:41.356215+0000","last_became_active":"2024-04-21T08:42:17.539335+0000","last_became_peered":"2024-04-21T08:42:17.539335+0000","last_unstale":"2024-04-21T08:44:41.356215+0000","last_undegraded":"2024-04-21T08:44:41.356215+0000","last_fullsized":"2024-04-21T08:44:41.356215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:27:51.520232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249882+0000","last_change":"2024-04-21T08:42:21.660570+0000","last_active":"2024-04-21T08:44:41.249882+0000","last_peered":"2024-04-21T08:44:41.249882+0000","last_clean":"2024-04-21T08:44:41.249882+0000","last_became_active":"2024-04-21T08:42:21.659398+0000","last_became_peered":"2024-04-21T08:42:21.659398+0000","last_unstale":"2024-04-21T08:44:41.249882+0000","last_undegraded":"2024-04-21T08:44:41.249882+0000","last_fullsized":"2024-04-21T08:44:41.249882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:55.840334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368786+0000","last_change":"2024-04-21T08:42:16.648192+0000","last_active":"2024-04-21T08:44:41.368786+0000","last_peered":"2024-04-21T08:44:41.368786+0000","last_clean":"2024-04-21T08:44:41.368786+0000","last_became_active":"2024-04-21T08:42:16.647666+0000","last_became_peered":"2024-04-21T08:42:16.647666+0000","last_unstale":"2024-04-21T08:44:41.368786+0000","last_undegraded":"2024-04-21T08:44:41.368786+0000","last_fullsized":"2024-04-21T08:44:41.368786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:33.344680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375708+0000","last_change":"2024-04-21T08:42:17.545925+0000","last_active":"2024-04-21T08:44:41.375708+0000","last_peered":"2024-04-21T08:44:41.375708+0000","last_clean":"2024-04-21T08:44:41.375708+0000","last_became_active":"2024-04-21T08:42:17.545555+0000","last_became_peered":"2024-04-21T08:42:17.545555+0000","last_unstale":"2024-04-21T08:44:41.375708+0000","last_undegraded":"2024-04-21T08:44:41.375708+0000","last_fullsized":"2024-04-21T08:44:41.375708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:13:07.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356087+0000","last_change":"2024-04-21T08:42:21.662757+0000","last_active":"2024-04-21T08:44:41.356087+0000","last_peered":"2024-04-21T08:44:41.356087+0000","last_clean":"2024-04-21T08:44:41.356087+0000","last_became_active":"2024-04-21T08:42:21.662306+0000","last_became_peered":"2024-04-21T08:42:21.662306+0000","last_unstale":"2024-04-21T08:44:41.356087+0000","last_undegraded":"2024-04-21T08:44:41.356087+0000","last_fullsized":"2024-04-21T08:44:41.356087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:26.856147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375135+0000","last_change":"2024-04-21T08:43:10.984690+0000","last_active":"2024-04-21T08:44:41.375135+0000","last_peered":"2024-04-21T08:44:41.375135+0000","last_clean":"2024-04-21T08:44:41.375135+0000","last_became_active":"2024-04-21T08:43:10.984290+0000","last_became_peered":"2024-04-21T08:43:10.984290+0000","last_unstale":"2024-04-21T08:44:41.375135+0000","last_undegraded":"2024-04-21T08:44:41.375135+0000","last_fullsized":"2024-04-21T08:44:41.375135+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:37:36.400946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418259+0000","last_change":"2024-04-21T08:43:10.793316+0000","last_active":"2024-04-21T08:44:41.418259+0000","last_peered":"2024-04-21T08:44:41.418259+0000","last_clean":"2024-04-21T08:44:41.418259+0000","last_became_active":"2024-04-21T08:43:10.793035+0000","last_became_peered":"2024-04-21T08:43:10.793035+0000","last_unstale":"2024-04-21T08:44:41.418259+0000","last_undegraded":"2024-04-21T08:44:41.418259+0000","last_fullsized":"2024-04-21T08:44:41.418259+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:41:04.247385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251083+0000","last_change":"2024-04-21T08:42:21.670561+0000","last_active":"2024-04-21T08:44:41.251083+0000","last_peered":"2024-04-21T08:44:41.251083+0000","last_clean":"2024-04-21T08:44:41.251083+0000","last_became_active":"2024-04-21T08:42:21.670170+0000","last_became_peered":"2024-04-21T08:42:21.670170+0000","last_unstale":"2024-04-21T08:44:41.251083+0000","last_undegraded":"2024-04-21T08:44:41.251083+0000","last_fullsized":"2024-04-21T08:44:41.251083+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:35:54.538013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365460+0000","last_change":"2024-04-21T08:42:16.637319+0000","last_active":"2024-04-21T08:44:41.365460+0000","last_peered":"2024-04-21T08:44:41.365460+0000","last_clean":"2024-04-21T08:44:41.365460+0000","last_became_active":"2024-04-21T08:42:16.636940+0000","last_became_peered":"2024-04-21T08:42:16.636940+0000","last_unstale":"2024-04-21T08:44:41.365460+0000","last_undegraded":"2024-04-21T08:44:41.365460+0000","last_fullsized":"2024-04-21T08:44:41.365460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:59:09.826786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375697+0000","last_change":"2024-04-21T08:42:17.546202+0000","last_active":"2024-04-21T08:44:41.375697+0000","last_peered":"2024-04-21T08:44:41.375697+0000","last_clean":"2024-04-21T08:44:41.375697+0000","last_became_active":"2024-04-21T08:42:17.545781+0000","last_became_peered":"2024-04-21T08:42:17.545781+0000","last_unstale":"2024-04-21T08:44:41.375697+0000","last_undegraded":"2024-04-21T08:44:41.375697+0000","last_fullsized":"2024-04-21T08:44:41.375697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:23.990951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356160+0000","last_change":"2024-04-21T08:42:21.662822+0000","last_active":"2024-04-21T08:44:41.356160+0000","last_peered":"2024-04-21T08:44:41.356160+0000","last_clean":"2024-04-21T08:44:41.356160+0000","last_became_active":"2024-04-21T08:42:21.662398+0000","last_became_peered":"2024-04-21T08:42:21.662398+0000","last_unstale":"2024-04-21T08:44:41.356160+0000","last_undegraded":"2024-04-21T08:44:41.356160+0000","last_fullsized":"2024-04-21T08:44:41.356160+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:38:40.499294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":335,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:47.138883+0000","last_change":"2024-04-21T08:42:16.636330+0000","last_active":"2024-04-21T08:44:47.138883+0000","last_peered":"2024-04-21T08:44:47.138883+0000","last_clean":"2024-04-21T08:44:47.138883+0000","last_became_active":"2024-04-21T08:42:16.635762+0000","last_became_peered":"2024-04-21T08:42:16.635762+0000","last_unstale":"2024-04-21T08:44:47.138883+0000","last_undegraded":"2024-04-21T08:44:47.138883+0000","last_fullsized":"2024-04-21T08:44:47.138883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:31.137923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":263,"num_read_kb":201,"num_write":38,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251043+0000","last_change":"2024-04-21T08:42:17.543033+0000","last_active":"2024-04-21T08:44:41.251043+0000","last_peered":"2024-04-21T08:44:41.251043+0000","last_clean":"2024-04-21T08:44:41.251043+0000","last_became_active":"2024-04-21T08:42:17.542692+0000","last_became_peered":"2024-04-21T08:42:17.542692+0000","last_unstale":"2024-04-21T08:44:41.251043+0000","last_undegraded":"2024-04-21T08:44:41.251043+0000","last_fullsized":"2024-04-21T08:44:41.251043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:33:32.111999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365806+0000","last_change":"2024-04-21T08:42:21.668831+0000","last_active":"2024-04-21T08:44:41.365806+0000","last_peered":"2024-04-21T08:44:41.365806+0000","last_clean":"2024-04-21T08:44:41.365806+0000","last_became_active":"2024-04-21T08:42:21.668485+0000","last_became_peered":"2024-04-21T08:42:21.668485+0000","last_unstale":"2024-04-21T08:44:41.365806+0000","last_undegraded":"2024-04-21T08:44:41.365806+0000","last_fullsized":"2024-04-21T08:44:41.365806+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:49:29.049707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375241+0000","last_change":"2024-04-21T08:42:16.640288+0000","last_active":"2024-04-21T08:44:41.375241+0000","last_peered":"2024-04-21T08:44:41.375241+0000","last_clean":"2024-04-21T08:44:41.375241+0000","last_became_active":"2024-04-21T08:42:16.639909+0000","last_became_peered":"2024-04-21T08:42:16.639909+0000","last_unstale":"2024-04-21T08:44:41.375241+0000","last_undegraded":"2024-04-21T08:44:41.375241+0000","last_fullsized":"2024-04-21T08:44:41.375241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:33.839442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375479+0000","last_change":"2024-04-21T08:42:17.541154+0000","last_active":"2024-04-21T08:44:41.375479+0000","last_peered":"2024-04-21T08:44:41.375479+0000","last_clean":"2024-04-21T08:44:41.375479+0000","last_became_active":"2024-04-21T08:42:17.540747+0000","last_became_peered":"2024-04-21T08:42:17.540747+0000","last_unstale":"2024-04-21T08:44:41.375479+0000","last_undegraded":"2024-04-21T08:44:41.375479+0000","last_fullsized":"2024-04-21T08:44:41.375479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:14:50.641144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375040+0000","last_change":"2024-04-21T08:42:21.670767+0000","last_active":"2024-04-21T08:44:41.375040+0000","last_peered":"2024-04-21T08:44:41.375040+0000","last_clean":"2024-04-21T08:44:41.375040+0000","last_became_active":"2024-04-21T08:42:21.670338+0000","last_became_peered":"2024-04-21T08:42:21.670338+0000","last_unstale":"2024-04-21T08:44:41.375040+0000","last_undegraded":"2024-04-21T08:44:41.375040+0000","last_fullsized":"2024-04-21T08:44:41.375040+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:38:55.275603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365468+0000","last_change":"2024-04-21T08:42:16.640144+0000","last_active":"2024-04-21T08:44:41.365468+0000","last_peered":"2024-04-21T08:44:41.365468+0000","last_clean":"2024-04-21T08:44:41.365468+0000","last_became_active":"2024-04-21T08:42:16.639707+0000","last_became_peered":"2024-04-21T08:42:16.639707+0000","last_unstale":"2024-04-21T08:44:41.365468+0000","last_undegraded":"2024-04-21T08:44:41.365468+0000","last_fullsized":"2024-04-21T08:44:41.365468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:11:32.785519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375240+0000","last_change":"2024-04-21T08:42:17.555019+0000","last_active":"2024-04-21T08:44:41.375240+0000","last_peered":"2024-04-21T08:44:41.375240+0000","last_clean":"2024-04-21T08:44:41.375240+0000","last_became_active":"2024-04-21T08:42:17.554645+0000","last_became_peered":"2024-04-21T08:42:17.554645+0000","last_unstale":"2024-04-21T08:44:41.375240+0000","last_undegraded":"2024-04-21T08:44:41.375240+0000","last_fullsized":"2024-04-21T08:44:41.375240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:42:30.353945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251223+0000","last_change":"2024-04-21T08:44:11.876740+0000","last_active":"2024-04-21T08:44:41.251223+0000","last_peered":"2024-04-21T08:44:41.251223+0000","last_clean":"2024-04-21T08:44:41.251223+0000","last_became_active":"2024-04-21T08:44:11.876108+0000","last_became_peered":"2024-04-21T08:44:11.876108+0000","last_unstale":"2024-04-21T08:44:41.251223+0000","last_undegraded":"2024-04-21T08:44:41.251223+0000","last_fullsized":"2024-04-21T08:44:41.251223+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:37.398743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250912+0000","last_change":"2024-04-21T08:42:16.640407+0000","last_active":"2024-04-21T08:44:41.250912+0000","last_peered":"2024-04-21T08:44:41.250912+0000","last_clean":"2024-04-21T08:44:41.250912+0000","last_became_active":"2024-04-21T08:42:16.640015+0000","last_became_peered":"2024-04-21T08:42:16.640015+0000","last_unstale":"2024-04-21T08:44:41.250912+0000","last_undegraded":"2024-04-21T08:44:41.250912+0000","last_fullsized":"2024-04-21T08:44:41.250912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:23:10.647184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374762+0000","last_change":"2024-04-21T08:42:17.545438+0000","last_active":"2024-04-21T08:44:41.374762+0000","last_peered":"2024-04-21T08:44:41.374762+0000","last_clean":"2024-04-21T08:44:41.374762+0000","last_became_active":"2024-04-21T08:42:17.544258+0000","last_became_peered":"2024-04-21T08:42:17.544258+0000","last_unstale":"2024-04-21T08:44:41.374762+0000","last_undegraded":"2024-04-21T08:44:41.374762+0000","last_fullsized":"2024-04-21T08:44:41.374762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:43.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376425+0000","last_change":"2024-04-21T08:42:21.671000+0000","last_active":"2024-04-21T08:44:41.376425+0000","last_peered":"2024-04-21T08:44:41.376425+0000","last_clean":"2024-04-21T08:44:41.376425+0000","last_became_active":"2024-04-21T08:42:21.670380+0000","last_became_peered":"2024-04-21T08:42:21.670380+0000","last_unstale":"2024-04-21T08:44:41.376425+0000","last_undegraded":"2024-04-21T08:44:41.376425+0000","last_fullsized":"2024-04-21T08:44:41.376425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:57.075686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250621+0000","last_change":"2024-04-21T08:42:16.649599+0000","last_active":"2024-04-21T08:44:41.250621+0000","last_peered":"2024-04-21T08:44:41.250621+0000","last_clean":"2024-04-21T08:44:41.250621+0000","last_became_active":"2024-04-21T08:42:16.649232+0000","last_became_peered":"2024-04-21T08:42:16.649232+0000","last_unstale":"2024-04-21T08:44:41.250621+0000","last_undegraded":"2024-04-21T08:44:41.250621+0000","last_fullsized":"2024-04-21T08:44:41.250621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:21.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356045+0000","last_change":"2024-04-21T08:44:11.776138+0000","last_active":"2024-04-21T08:44:41.356045+0000","last_peered":"2024-04-21T08:44:41.356045+0000","last_clean":"2024-04-21T08:44:41.356045+0000","last_became_active":"2024-04-21T08:44:11.775832+0000","last_became_peered":"2024-04-21T08:44:11.775832+0000","last_unstale":"2024-04-21T08:44:41.356045+0000","last_undegraded":"2024-04-21T08:44:41.356045+0000","last_fullsized":"2024-04-21T08:44:41.356045+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:03:34.210923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365172+0000","last_change":"2024-04-21T08:42:21.664811+0000","last_active":"2024-04-21T08:44:41.365172+0000","last_peered":"2024-04-21T08:44:41.365172+0000","last_clean":"2024-04-21T08:44:41.365172+0000","last_became_active":"2024-04-21T08:42:21.664332+0000","last_became_peered":"2024-04-21T08:42:21.664332+0000","last_unstale":"2024-04-21T08:44:41.365172+0000","last_undegraded":"2024-04-21T08:44:41.365172+0000","last_fullsized":"2024-04-21T08:44:41.365172+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:40.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251372+0000","last_change":"2024-04-21T08:42:16.641233+0000","last_active":"2024-04-21T08:44:41.251372+0000","last_peered":"2024-04-21T08:44:41.251372+0000","last_clean":"2024-04-21T08:44:41.251372+0000","last_became_active":"2024-04-21T08:42:16.640873+0000","last_became_peered":"2024-04-21T08:42:16.640873+0000","last_unstale":"2024-04-21T08:44:41.251372+0000","last_undegraded":"2024-04-21T08:44:41.251372+0000","last_fullsized":"2024-04-21T08:44:41.251372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:00.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375293+0000","last_change":"2024-04-21T08:42:17.550438+0000","last_active":"2024-04-21T08:44:41.375293+0000","last_peered":"2024-04-21T08:44:41.375293+0000","last_clean":"2024-04-21T08:44:41.375293+0000","last_became_active":"2024-04-21T08:42:17.550015+0000","last_became_peered":"2024-04-21T08:42:17.550015+0000","last_unstale":"2024-04-21T08:44:41.375293+0000","last_undegraded":"2024-04-21T08:44:41.375293+0000","last_fullsized":"2024-04-21T08:44:41.375293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:00:29.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250709+0000","last_change":"2024-04-21T08:42:21.970498+0000","last_active":"2024-04-21T08:44:41.250709+0000","last_peered":"2024-04-21T08:44:41.250709+0000","last_clean":"2024-04-21T08:44:41.250709+0000","last_became_active":"2024-04-21T08:42:21.969819+0000","last_became_peered":"2024-04-21T08:42:21.969819+0000","last_unstale":"2024-04-21T08:44:41.250709+0000","last_undegraded":"2024-04-21T08:44:41.250709+0000","last_fullsized":"2024-04-21T08:44:41.250709+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:13:16.552293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356527+0000","last_change":"2024-04-21T08:42:16.637146+0000","last_active":"2024-04-21T08:44:41.356527+0000","last_peered":"2024-04-21T08:44:41.356527+0000","last_clean":"2024-04-21T08:44:41.356527+0000","last_became_active":"2024-04-21T08:42:16.636781+0000","last_became_peered":"2024-04-21T08:42:16.636781+0000","last_unstale":"2024-04-21T08:44:41.356527+0000","last_undegraded":"2024-04-21T08:44:41.356527+0000","last_fullsized":"2024-04-21T08:44:41.356527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:39:27.641236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368622+0000","last_change":"2024-04-21T08:42:17.542325+0000","last_active":"2024-04-21T08:44:41.368622+0000","last_peered":"2024-04-21T08:44:41.368622+0000","last_clean":"2024-04-21T08:44:41.368622+0000","last_became_active":"2024-04-21T08:42:17.541103+0000","last_became_peered":"2024-04-21T08:42:17.541103+0000","last_unstale":"2024-04-21T08:44:41.368622+0000","last_undegraded":"2024-04-21T08:44:41.368622+0000","last_fullsized":"2024-04-21T08:44:41.368622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:46.903250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375702+0000","last_change":"2024-04-21T08:42:21.671146+0000","last_active":"2024-04-21T08:44:41.375702+0000","last_peered":"2024-04-21T08:44:41.375702+0000","last_clean":"2024-04-21T08:44:41.375702+0000","last_became_active":"2024-04-21T08:42:21.670561+0000","last_became_peered":"2024-04-21T08:42:21.670561+0000","last_unstale":"2024-04-21T08:44:41.375702+0000","last_undegraded":"2024-04-21T08:44:41.375702+0000","last_fullsized":"2024-04-21T08:44:41.375702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:38:28.028976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375066+0000","last_change":"2024-04-21T08:42:16.645132+0000","last_active":"2024-04-21T08:44:41.375066+0000","last_peered":"2024-04-21T08:44:41.375066+0000","last_clean":"2024-04-21T08:44:41.375066+0000","last_became_active":"2024-04-21T08:42:16.644786+0000","last_became_peered":"2024-04-21T08:42:16.644786+0000","last_unstale":"2024-04-21T08:44:41.375066+0000","last_undegraded":"2024-04-21T08:44:41.375066+0000","last_fullsized":"2024-04-21T08:44:41.375066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:08.298498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368567+0000","last_change":"2024-04-21T08:42:17.550424+0000","last_active":"2024-04-21T08:44:41.368567+0000","last_peered":"2024-04-21T08:44:41.368567+0000","last_clean":"2024-04-21T08:44:41.368567+0000","last_became_active":"2024-04-21T08:42:17.550077+0000","last_became_peered":"2024-04-21T08:42:17.550077+0000","last_unstale":"2024-04-21T08:44:41.368567+0000","last_undegraded":"2024-04-21T08:44:41.368567+0000","last_fullsized":"2024-04-21T08:44:41.368567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:17:07.016204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375856+0000","last_change":"2024-04-21T08:42:21.970652+0000","last_active":"2024-04-21T08:44:41.375856+0000","last_peered":"2024-04-21T08:44:41.375856+0000","last_clean":"2024-04-21T08:44:41.375856+0000","last_became_active":"2024-04-21T08:42:21.970251+0000","last_became_peered":"2024-04-21T08:42:21.970251+0000","last_unstale":"2024-04-21T08:44:41.375856+0000","last_undegraded":"2024-04-21T08:44:41.375856+0000","last_fullsized":"2024-04-21T08:44:41.375856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:45.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365285+0000","last_change":"2024-04-21T08:42:17.542820+0000","last_active":"2024-04-21T08:44:41.365285+0000","last_peered":"2024-04-21T08:44:41.365285+0000","last_clean":"2024-04-21T08:44:41.365285+0000","last_became_active":"2024-04-21T08:42:17.541320+0000","last_became_peered":"2024-04-21T08:42:17.541320+0000","last_unstale":"2024-04-21T08:44:41.365285+0000","last_undegraded":"2024-04-21T08:44:41.365285+0000","last_fullsized":"2024-04-21T08:44:41.365285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:23:06.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375209+0000","last_change":"2024-04-21T08:42:16.636950+0000","last_active":"2024-04-21T08:44:41.375209+0000","last_peered":"2024-04-21T08:44:41.375209+0000","last_clean":"2024-04-21T08:44:41.375209+0000","last_became_active":"2024-04-21T08:42:16.636590+0000","last_became_peered":"2024-04-21T08:42:16.636590+0000","last_unstale":"2024-04-21T08:44:41.375209+0000","last_undegraded":"2024-04-21T08:44:41.375209+0000","last_fullsized":"2024-04-21T08:44:41.375209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:55.693681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374884+0000","last_change":"2024-04-21T08:42:21.660404+0000","last_active":"2024-04-21T08:44:41.374884+0000","last_peered":"2024-04-21T08:44:41.374884+0000","last_clean":"2024-04-21T08:44:41.374884+0000","last_became_active":"2024-04-21T08:42:21.659999+0000","last_became_peered":"2024-04-21T08:42:21.659999+0000","last_unstale":"2024-04-21T08:44:41.374884+0000","last_undegraded":"2024-04-21T08:44:41.374884+0000","last_fullsized":"2024-04-21T08:44:41.374884+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:53:32.328873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356239+0000","last_change":"2024-04-21T08:43:10.984361+0000","last_active":"2024-04-21T08:44:41.356239+0000","last_peered":"2024-04-21T08:44:41.356239+0000","last_clean":"2024-04-21T08:44:41.356239+0000","last_became_active":"2024-04-21T08:43:10.984046+0000","last_became_peered":"2024-04-21T08:43:10.984046+0000","last_unstale":"2024-04-21T08:44:41.356239+0000","last_undegraded":"2024-04-21T08:44:41.356239+0000","last_fullsized":"2024-04-21T08:44:41.356239+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:33:23.154630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251236+0000","last_change":"2024-04-21T08:42:16.638993+0000","last_active":"2024-04-21T08:44:41.251236+0000","last_peered":"2024-04-21T08:44:41.251236+0000","last_clean":"2024-04-21T08:44:41.251236+0000","last_became_active":"2024-04-21T08:42:16.638617+0000","last_became_peered":"2024-04-21T08:42:16.638617+0000","last_unstale":"2024-04-21T08:44:41.251236+0000","last_undegraded":"2024-04-21T08:44:41.251236+0000","last_fullsized":"2024-04-21T08:44:41.251236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:14:28.271433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"95'148","reported_seq":410,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:43.370391+0000","last_change":"2024-04-21T08:41:00.044984+0000","last_active":"2024-04-21T08:44:43.370391+0000","last_peered":"2024-04-21T08:44:43.370391+0000","last_clean":"2024-04-21T08:44:43.370391+0000","last_became_active":"2024-04-21T08:41:00.044485+0000","last_became_peered":"2024-04-21T08:41:00.044485+0000","last_unstale":"2024-04-21T08:44:43.370391+0000","last_undegraded":"2024-04-21T08:44:43.370391+0000","last_fullsized":"2024-04-21T08:44:43.370391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.370031+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:01.031209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365508+0000","last_change":"2024-04-21T08:42:21.971045+0000","last_active":"2024-04-21T08:44:41.365508+0000","last_peered":"2024-04-21T08:44:41.365508+0000","last_clean":"2024-04-21T08:44:41.365508+0000","last_became_active":"2024-04-21T08:42:21.970577+0000","last_became_peered":"2024-04-21T08:42:21.970577+0000","last_unstale":"2024-04-21T08:44:41.365508+0000","last_undegraded":"2024-04-21T08:44:41.365508+0000","last_fullsized":"2024-04-21T08:44:41.365508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:55.681662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374964+0000","last_change":"2024-04-21T08:42:17.535570+0000","last_active":"2024-04-21T08:44:41.374964+0000","last_peered":"2024-04-21T08:44:41.374964+0000","last_clean":"2024-04-21T08:44:41.374964+0000","last_became_active":"2024-04-21T08:42:17.535217+0000","last_became_peered":"2024-04-21T08:42:17.535217+0000","last_unstale":"2024-04-21T08:44:41.374964+0000","last_undegraded":"2024-04-21T08:44:41.374964+0000","last_fullsized":"2024-04-21T08:44:41.374964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:49.510750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374650+0000","last_change":"2024-04-21T08:43:10.792548+0000","last_active":"2024-04-21T08:44:41.374650+0000","last_peered":"2024-04-21T08:44:41.374650+0000","last_clean":"2024-04-21T08:44:41.374650+0000","last_became_active":"2024-04-21T08:43:10.792165+0000","last_became_peered":"2024-04-21T08:43:10.792165+0000","last_unstale":"2024-04-21T08:44:41.374650+0000","last_undegraded":"2024-04-21T08:44:41.374650+0000","last_fullsized":"2024-04-21T08:44:41.374650+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:35.980536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374872+0000","last_change":"2024-04-21T08:42:21.661393+0000","last_active":"2024-04-21T08:44:41.374872+0000","last_peered":"2024-04-21T08:44:41.374872+0000","last_clean":"2024-04-21T08:44:41.374872+0000","last_became_active":"2024-04-21T08:42:21.660201+0000","last_became_peered":"2024-04-21T08:42:21.660201+0000","last_unstale":"2024-04-21T08:44:41.374872+0000","last_undegraded":"2024-04-21T08:44:41.374872+0000","last_fullsized":"2024-04-21T08:44:41.374872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:13.393389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250941+0000","last_change":"2024-04-21T08:42:16.650507+0000","last_active":"2024-04-21T08:44:41.250941+0000","last_peered":"2024-04-21T08:44:41.250941+0000","last_clean":"2024-04-21T08:44:41.250941+0000","last_became_active":"2024-04-21T08:42:16.650217+0000","last_became_peered":"2024-04-21T08:42:16.650217+0000","last_unstale":"2024-04-21T08:44:41.250941+0000","last_undegraded":"2024-04-21T08:44:41.250941+0000","last_fullsized":"2024-04-21T08:44:41.250941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:13:26.785092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375997+0000","last_change":"2024-04-21T08:42:17.530420+0000","last_active":"2024-04-21T08:44:41.375997+0000","last_peered":"2024-04-21T08:44:41.375997+0000","last_clean":"2024-04-21T08:44:41.375997+0000","last_became_active":"2024-04-21T08:42:17.529920+0000","last_became_peered":"2024-04-21T08:42:17.529920+0000","last_unstale":"2024-04-21T08:44:41.375997+0000","last_undegraded":"2024-04-21T08:44:41.375997+0000","last_fullsized":"2024-04-21T08:44:41.375997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:24.250197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249760+0000","last_change":"2024-04-21T08:42:21.970586+0000","last_active":"2024-04-21T08:44:41.249760+0000","last_peered":"2024-04-21T08:44:41.249760+0000","last_clean":"2024-04-21T08:44:41.249760+0000","last_became_active":"2024-04-21T08:42:21.970132+0000","last_became_peered":"2024-04-21T08:42:21.970132+0000","last_unstale":"2024-04-21T08:44:41.249760+0000","last_undegraded":"2024-04-21T08:44:41.249760+0000","last_fullsized":"2024-04-21T08:44:41.249760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:11.074370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'68","reported_seq":155,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:47.175970+0000","last_change":"2024-04-21T08:42:16.647916+0000","last_active":"2024-04-21T08:44:47.175970+0000","last_peered":"2024-04-21T08:44:47.175970+0000","last_clean":"2024-04-21T08:44:47.175970+0000","last_became_active":"2024-04-21T08:42:16.647476+0000","last_became_peered":"2024-04-21T08:42:16.647476+0000","last_unstale":"2024-04-21T08:44:47.175970+0000","last_undegraded":"2024-04-21T08:44:47.175970+0000","last_fullsized":"2024-04-21T08:44:47.175970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:27.242153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":160,"num_write":122,"num_write_kb":1157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251070+0000","last_change":"2024-04-21T08:42:17.546144+0000","last_active":"2024-04-21T08:44:41.251070+0000","last_peered":"2024-04-21T08:44:41.251070+0000","last_clean":"2024-04-21T08:44:41.251070+0000","last_became_active":"2024-04-21T08:42:17.545730+0000","last_became_peered":"2024-04-21T08:42:17.545730+0000","last_unstale":"2024-04-21T08:44:41.251070+0000","last_undegraded":"2024-04-21T08:44:41.251070+0000","last_fullsized":"2024-04-21T08:44:41.251070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:42.802309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374618+0000","last_change":"2024-04-21T08:42:21.655802+0000","last_active":"2024-04-21T08:44:41.374618+0000","last_peered":"2024-04-21T08:44:41.374618+0000","last_clean":"2024-04-21T08:44:41.374618+0000","last_became_active":"2024-04-21T08:42:21.655379+0000","last_became_peered":"2024-04-21T08:42:21.655379+0000","last_unstale":"2024-04-21T08:44:41.374618+0000","last_undegraded":"2024-04-21T08:44:41.374618+0000","last_fullsized":"2024-04-21T08:44:41.374618+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:49.486057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251462+0000","last_change":"2024-04-21T08:42:16.642517+0000","last_active":"2024-04-21T08:44:41.251462+0000","last_peered":"2024-04-21T08:44:41.251462+0000","last_clean":"2024-04-21T08:44:41.251462+0000","last_became_active":"2024-04-21T08:42:16.641539+0000","last_became_peered":"2024-04-21T08:42:16.641539+0000","last_unstale":"2024-04-21T08:44:41.251462+0000","last_undegraded":"2024-04-21T08:44:41.251462+0000","last_fullsized":"2024-04-21T08:44:41.251462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:53.912199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418089+0000","last_change":"2024-04-21T08:42:17.550172+0000","last_active":"2024-04-21T08:44:41.418089+0000","last_peered":"2024-04-21T08:44:41.418089+0000","last_clean":"2024-04-21T08:44:41.418089+0000","last_became_active":"2024-04-21T08:42:17.549824+0000","last_became_peered":"2024-04-21T08:42:17.549824+0000","last_unstale":"2024-04-21T08:44:41.418089+0000","last_undegraded":"2024-04-21T08:44:41.418089+0000","last_fullsized":"2024-04-21T08:44:41.418089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:59.553645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251154+0000","last_change":"2024-04-21T08:44:11.774778+0000","last_active":"2024-04-21T08:44:41.251154+0000","last_peered":"2024-04-21T08:44:41.251154+0000","last_clean":"2024-04-21T08:44:41.251154+0000","last_became_active":"2024-04-21T08:44:11.774190+0000","last_became_peered":"2024-04-21T08:44:11.774190+0000","last_unstale":"2024-04-21T08:44:41.251154+0000","last_undegraded":"2024-04-21T08:44:41.251154+0000","last_fullsized":"2024-04-21T08:44:41.251154+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:06.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374515+0000","last_change":"2024-04-21T08:42:16.647985+0000","last_active":"2024-04-21T08:44:41.374515+0000","last_peered":"2024-04-21T08:44:41.374515+0000","last_clean":"2024-04-21T08:44:41.374515+0000","last_became_active":"2024-04-21T08:42:16.647577+0000","last_became_peered":"2024-04-21T08:42:16.647577+0000","last_unstale":"2024-04-21T08:44:41.374515+0000","last_undegraded":"2024-04-21T08:44:41.374515+0000","last_fullsized":"2024-04-21T08:44:41.374515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:00.731708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250576+0000","last_change":"2024-04-21T08:42:17.541757+0000","last_active":"2024-04-21T08:44:41.250576+0000","last_peered":"2024-04-21T08:44:41.250576+0000","last_clean":"2024-04-21T08:44:41.250576+0000","last_became_active":"2024-04-21T08:42:17.539280+0000","last_became_peered":"2024-04-21T08:42:17.539280+0000","last_unstale":"2024-04-21T08:44:41.250576+0000","last_undegraded":"2024-04-21T08:44:41.250576+0000","last_fullsized":"2024-04-21T08:44:41.250576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:45:54.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356961+0000","last_change":"2024-04-21T08:42:21.664566+0000","last_active":"2024-04-21T08:44:41.356961+0000","last_peered":"2024-04-21T08:44:41.356961+0000","last_clean":"2024-04-21T08:44:41.356961+0000","last_became_active":"2024-04-21T08:42:21.664136+0000","last_became_peered":"2024-04-21T08:42:21.664136+0000","last_unstale":"2024-04-21T08:44:41.356961+0000","last_undegraded":"2024-04-21T08:44:41.356961+0000","last_fullsized":"2024-04-21T08:44:41.356961+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:19.856577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249918+0000","last_change":"2024-04-21T08:42:16.650664+0000","last_active":"2024-04-21T08:44:41.249918+0000","last_peered":"2024-04-21T08:44:41.249918+0000","last_clean":"2024-04-21T08:44:41.249918+0000","last_became_active":"2024-04-21T08:42:16.650339+0000","last_became_peered":"2024-04-21T08:42:16.650339+0000","last_unstale":"2024-04-21T08:44:41.249918+0000","last_undegraded":"2024-04-21T08:44:41.249918+0000","last_fullsized":"2024-04-21T08:44:41.249918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:59.565455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374844+0000","last_change":"2024-04-21T08:42:17.538142+0000","last_active":"2024-04-21T08:44:41.374844+0000","last_peered":"2024-04-21T08:44:41.374844+0000","last_clean":"2024-04-21T08:44:41.374844+0000","last_became_active":"2024-04-21T08:42:17.537793+0000","last_became_peered":"2024-04-21T08:42:17.537793+0000","last_unstale":"2024-04-21T08:44:41.374844+0000","last_undegraded":"2024-04-21T08:44:41.374844+0000","last_fullsized":"2024-04-21T08:44:41.374844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:11.059294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251170+0000","last_change":"2024-04-21T08:42:21.670642+0000","last_active":"2024-04-21T08:44:41.251170+0000","last_peered":"2024-04-21T08:44:41.251170+0000","last_clean":"2024-04-21T08:44:41.251170+0000","last_became_active":"2024-04-21T08:42:21.670128+0000","last_became_peered":"2024-04-21T08:42:21.670128+0000","last_unstale":"2024-04-21T08:44:41.251170+0000","last_undegraded":"2024-04-21T08:44:41.251170+0000","last_fullsized":"2024-04-21T08:44:41.251170+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:51:28.604381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375278+0000","last_change":"2024-04-21T08:43:10.792114+0000","last_active":"2024-04-21T08:44:41.375278+0000","last_peered":"2024-04-21T08:44:41.375278+0000","last_clean":"2024-04-21T08:44:41.375278+0000","last_became_active":"2024-04-21T08:43:10.791664+0000","last_became_peered":"2024-04-21T08:43:10.791664+0000","last_unstale":"2024-04-21T08:44:41.375278+0000","last_undegraded":"2024-04-21T08:44:41.375278+0000","last_fullsized":"2024-04-21T08:44:41.375278+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:07.317804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250083+0000","last_change":"2024-04-21T08:42:17.547326+0000","last_active":"2024-04-21T08:44:41.250083+0000","last_peered":"2024-04-21T08:44:41.250083+0000","last_clean":"2024-04-21T08:44:41.250083+0000","last_became_active":"2024-04-21T08:42:17.546920+0000","last_became_peered":"2024-04-21T08:42:17.546920+0000","last_unstale":"2024-04-21T08:44:41.250083+0000","last_undegraded":"2024-04-21T08:44:41.250083+0000","last_fullsized":"2024-04-21T08:44:41.250083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:43.350877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251167+0000","last_change":"2024-04-21T08:42:21.971147+0000","last_active":"2024-04-21T08:44:41.251167+0000","last_peered":"2024-04-21T08:44:41.251167+0000","last_clean":"2024-04-21T08:44:41.251167+0000","last_became_active":"2024-04-21T08:42:21.970001+0000","last_became_peered":"2024-04-21T08:42:21.970001+0000","last_unstale":"2024-04-21T08:44:41.251167+0000","last_undegraded":"2024-04-21T08:44:41.251167+0000","last_fullsized":"2024-04-21T08:44:41.251167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:01:44.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250876+0000","last_change":"2024-04-21T08:42:16.643371+0000","last_active":"2024-04-21T08:44:41.250876+0000","last_peered":"2024-04-21T08:44:41.250876+0000","last_clean":"2024-04-21T08:44:41.250876+0000","last_became_active":"2024-04-21T08:42:16.642906+0000","last_became_peered":"2024-04-21T08:42:16.642906+0000","last_unstale":"2024-04-21T08:44:41.250876+0000","last_undegraded":"2024-04-21T08:44:41.250876+0000","last_fullsized":"2024-04-21T08:44:41.250876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:58.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375124+0000","last_change":"2024-04-21T08:42:17.545185+0000","last_active":"2024-04-21T08:44:41.375124+0000","last_peered":"2024-04-21T08:44:41.375124+0000","last_clean":"2024-04-21T08:44:41.375124+0000","last_became_active":"2024-04-21T08:42:17.544757+0000","last_became_peered":"2024-04-21T08:42:17.544757+0000","last_unstale":"2024-04-21T08:44:41.375124+0000","last_undegraded":"2024-04-21T08:44:41.375124+0000","last_fullsized":"2024-04-21T08:44:41.375124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:36:18.968041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374923+0000","last_change":"2024-04-21T08:42:21.662541+0000","last_active":"2024-04-21T08:44:41.374923+0000","last_peered":"2024-04-21T08:44:41.374923+0000","last_clean":"2024-04-21T08:44:41.374923+0000","last_became_active":"2024-04-21T08:42:21.662144+0000","last_became_peered":"2024-04-21T08:42:21.662144+0000","last_unstale":"2024-04-21T08:44:41.374923+0000","last_undegraded":"2024-04-21T08:44:41.374923+0000","last_fullsized":"2024-04-21T08:44:41.374923+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:04.347722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375374+0000","last_change":"2024-04-21T08:42:16.638601+0000","last_active":"2024-04-21T08:44:41.375374+0000","last_peered":"2024-04-21T08:44:41.375374+0000","last_clean":"2024-04-21T08:44:41.375374+0000","last_became_active":"2024-04-21T08:42:16.638211+0000","last_became_peered":"2024-04-21T08:42:16.638211+0000","last_unstale":"2024-04-21T08:44:41.375374+0000","last_undegraded":"2024-04-21T08:44:41.375374+0000","last_fullsized":"2024-04-21T08:44:41.375374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:02.633086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375257+0000","last_change":"2024-04-21T08:42:17.545495+0000","last_active":"2024-04-21T08:44:41.375257+0000","last_peered":"2024-04-21T08:44:41.375257+0000","last_clean":"2024-04-21T08:44:41.375257+0000","last_became_active":"2024-04-21T08:42:17.545159+0000","last_became_peered":"2024-04-21T08:42:17.545159+0000","last_unstale":"2024-04-21T08:44:41.375257+0000","last_undegraded":"2024-04-21T08:44:41.375257+0000","last_fullsized":"2024-04-21T08:44:41.375257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:27.150277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417564+0000","last_change":"2024-04-21T08:42:21.656890+0000","last_active":"2024-04-21T08:44:41.417564+0000","last_peered":"2024-04-21T08:44:41.417564+0000","last_clean":"2024-04-21T08:44:41.417564+0000","last_became_active":"2024-04-21T08:42:21.656475+0000","last_became_peered":"2024-04-21T08:42:21.656475+0000","last_unstale":"2024-04-21T08:44:41.417564+0000","last_undegraded":"2024-04-21T08:44:41.417564+0000","last_fullsized":"2024-04-21T08:44:41.417564+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:46.138591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'57","reported_seq":109,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:47.171231+0000","last_change":"2024-04-21T08:42:16.638244+0000","last_active":"2024-04-21T08:44:47.171231+0000","last_peered":"2024-04-21T08:44:47.171231+0000","last_clean":"2024-04-21T08:44:47.171231+0000","last_became_active":"2024-04-21T08:42:16.637883+0000","last_became_peered":"2024-04-21T08:42:16.637883+0000","last_unstale":"2024-04-21T08:44:47.171231+0000","last_undegraded":"2024-04-21T08:44:47.171231+0000","last_fullsized":"2024-04-21T08:44:47.171231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:20.888301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":68,"num_write_kb":807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374936+0000","last_change":"2024-04-21T08:42:17.548032+0000","last_active":"2024-04-21T08:44:41.374936+0000","last_peered":"2024-04-21T08:44:41.374936+0000","last_clean":"2024-04-21T08:44:41.374936+0000","last_became_active":"2024-04-21T08:42:17.547660+0000","last_became_peered":"2024-04-21T08:42:17.547660+0000","last_unstale":"2024-04-21T08:44:41.374936+0000","last_undegraded":"2024-04-21T08:44:41.374936+0000","last_fullsized":"2024-04-21T08:44:41.374936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:14.710759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365005+0000","last_change":"2024-04-21T08:42:21.659286+0000","last_active":"2024-04-21T08:44:41.365005+0000","last_peered":"2024-04-21T08:44:41.365005+0000","last_clean":"2024-04-21T08:44:41.365005+0000","last_became_active":"2024-04-21T08:42:21.658844+0000","last_became_peered":"2024-04-21T08:42:21.658844+0000","last_unstale":"2024-04-21T08:44:41.365005+0000","last_undegraded":"2024-04-21T08:44:41.365005+0000","last_fullsized":"2024-04-21T08:44:41.365005+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:28.742646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376281+0000","last_change":"2024-04-21T08:42:16.643354+0000","last_active":"2024-04-21T08:44:41.376281+0000","last_peered":"2024-04-21T08:44:41.376281+0000","last_clean":"2024-04-21T08:44:41.376281+0000","last_became_active":"2024-04-21T08:42:16.642967+0000","last_became_peered":"2024-04-21T08:42:16.642967+0000","last_unstale":"2024-04-21T08:44:41.376281+0000","last_undegraded":"2024-04-21T08:44:41.376281+0000","last_fullsized":"2024-04-21T08:44:41.376281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:06:03.495181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375409+0000","last_change":"2024-04-21T08:42:17.539551+0000","last_active":"2024-04-21T08:44:41.375409+0000","last_peered":"2024-04-21T08:44:41.375409+0000","last_clean":"2024-04-21T08:44:41.375409+0000","last_became_active":"2024-04-21T08:42:17.539202+0000","last_became_peered":"2024-04-21T08:42:17.539202+0000","last_unstale":"2024-04-21T08:44:41.375409+0000","last_undegraded":"2024-04-21T08:44:41.375409+0000","last_fullsized":"2024-04-21T08:44:41.375409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:37.109421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375023+0000","last_change":"2024-04-21T08:42:17.544582+0000","last_active":"2024-04-21T08:44:41.375023+0000","last_peered":"2024-04-21T08:44:41.375023+0000","last_clean":"2024-04-21T08:44:41.375023+0000","last_became_active":"2024-04-21T08:42:17.543800+0000","last_became_peered":"2024-04-21T08:42:17.543800+0000","last_unstale":"2024-04-21T08:44:41.375023+0000","last_undegraded":"2024-04-21T08:44:41.375023+0000","last_fullsized":"2024-04-21T08:44:41.375023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:44:04.107207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374756+0000","last_change":"2024-04-21T08:42:17.543788+0000","last_active":"2024-04-21T08:44:41.374756+0000","last_peered":"2024-04-21T08:44:41.374756+0000","last_clean":"2024-04-21T08:44:41.374756+0000","last_became_active":"2024-04-21T08:42:17.540939+0000","last_became_peered":"2024-04-21T08:42:17.540939+0000","last_unstale":"2024-04-21T08:44:41.374756+0000","last_undegraded":"2024-04-21T08:44:41.374756+0000","last_fullsized":"2024-04-21T08:44:41.374756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:59.275098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374942+0000","last_change":"2024-04-21T08:42:17.540605+0000","last_active":"2024-04-21T08:44:41.374942+0000","last_peered":"2024-04-21T08:44:41.374942+0000","last_clean":"2024-04-21T08:44:41.374942+0000","last_became_active":"2024-04-21T08:42:17.540073+0000","last_became_peered":"2024-04-21T08:42:17.540073+0000","last_unstale":"2024-04-21T08:44:41.374942+0000","last_undegraded":"2024-04-21T08:44:41.374942+0000","last_fullsized":"2024-04-21T08:44:41.374942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:02:29.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417931+0000","last_change":"2024-04-21T08:42:17.551608+0000","last_active":"2024-04-21T08:44:41.417931+0000","last_peered":"2024-04-21T08:44:41.417931+0000","last_clean":"2024-04-21T08:44:41.417931+0000","last_became_active":"2024-04-21T08:42:17.551117+0000","last_became_peered":"2024-04-21T08:42:17.551117+0000","last_unstale":"2024-04-21T08:44:41.417931+0000","last_undegraded":"2024-04-21T08:44:41.417931+0000","last_fullsized":"2024-04-21T08:44:41.417931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:18.391928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375875+0000","last_change":"2024-04-21T08:42:17.536509+0000","last_active":"2024-04-21T08:44:41.375875+0000","last_peered":"2024-04-21T08:44:41.375875+0000","last_clean":"2024-04-21T08:44:41.375875+0000","last_became_active":"2024-04-21T08:42:17.536146+0000","last_became_peered":"2024-04-21T08:42:17.536146+0000","last_unstale":"2024-04-21T08:44:41.375875+0000","last_undegraded":"2024-04-21T08:44:41.375875+0000","last_fullsized":"2024-04-21T08:44:41.375875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:29.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375199+0000","last_change":"2024-04-21T08:42:17.543786+0000","last_active":"2024-04-21T08:44:41.375199+0000","last_peered":"2024-04-21T08:44:41.375199+0000","last_clean":"2024-04-21T08:44:41.375199+0000","last_became_active":"2024-04-21T08:42:17.543435+0000","last_became_peered":"2024-04-21T08:42:17.543435+0000","last_unstale":"2024-04-21T08:44:41.375199+0000","last_undegraded":"2024-04-21T08:44:41.375199+0000","last_fullsized":"2024-04-21T08:44:41.375199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375616+0000","last_change":"2024-04-21T08:42:17.541611+0000","last_active":"2024-04-21T08:44:41.375616+0000","last_peered":"2024-04-21T08:44:41.375616+0000","last_clean":"2024-04-21T08:44:41.375616+0000","last_became_active":"2024-04-21T08:42:17.541259+0000","last_became_peered":"2024-04-21T08:42:17.541259+0000","last_unstale":"2024-04-21T08:44:41.375616+0000","last_undegraded":"2024-04-21T08:44:41.375616+0000","last_fullsized":"2024-04-21T08:44:41.375616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:20.900629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356654+0000","last_change":"2024-04-21T08:44:11.774062+0000","last_active":"2024-04-21T08:44:41.356654+0000","last_peered":"2024-04-21T08:44:41.356654+0000","last_clean":"2024-04-21T08:44:41.356654+0000","last_became_active":"2024-04-21T08:44:11.773703+0000","last_became_peered":"2024-04-21T08:44:11.773703+0000","last_unstale":"2024-04-21T08:44:41.356654+0000","last_undegraded":"2024-04-21T08:44:41.356654+0000","last_fullsized":"2024-04-21T08:44:41.356654+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:10.739068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365197+0000","last_change":"2024-04-21T08:42:17.545526+0000","last_active":"2024-04-21T08:44:41.365197+0000","last_peered":"2024-04-21T08:44:41.365197+0000","last_clean":"2024-04-21T08:44:41.365197+0000","last_became_active":"2024-04-21T08:42:17.545135+0000","last_became_peered":"2024-04-21T08:42:17.545135+0000","last_unstale":"2024-04-21T08:44:41.365197+0000","last_undegraded":"2024-04-21T08:44:41.365197+0000","last_fullsized":"2024-04-21T08:44:41.365197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:37:58.038459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250029+0000","last_change":"2024-04-21T08:42:17.547063+0000","last_active":"2024-04-21T08:44:41.250029+0000","last_peered":"2024-04-21T08:44:41.250029+0000","last_clean":"2024-04-21T08:44:41.250029+0000","last_became_active":"2024-04-21T08:42:17.546689+0000","last_became_peered":"2024-04-21T08:42:17.546689+0000","last_unstale":"2024-04-21T08:44:41.250029+0000","last_undegraded":"2024-04-21T08:44:41.250029+0000","last_fullsized":"2024-04-21T08:44:41.250029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:26.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375261+0000","last_change":"2024-04-21T08:42:17.542545+0000","last_active":"2024-04-21T08:44:41.375261+0000","last_peered":"2024-04-21T08:44:41.375261+0000","last_clean":"2024-04-21T08:44:41.375261+0000","last_became_active":"2024-04-21T08:42:17.542147+0000","last_became_peered":"2024-04-21T08:42:17.542147+0000","last_unstale":"2024-04-21T08:44:41.375261+0000","last_undegraded":"2024-04-21T08:44:41.375261+0000","last_fullsized":"2024-04-21T08:44:41.375261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:15.064110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376582+0000","last_change":"2024-04-21T08:42:17.541262+0000","last_active":"2024-04-21T08:44:41.376582+0000","last_peered":"2024-04-21T08:44:41.376582+0000","last_clean":"2024-04-21T08:44:41.376582+0000","last_became_active":"2024-04-21T08:42:17.540817+0000","last_became_peered":"2024-04-21T08:42:17.540817+0000","last_unstale":"2024-04-21T08:44:41.376582+0000","last_undegraded":"2024-04-21T08:44:41.376582+0000","last_fullsized":"2024-04-21T08:44:41.376582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:22:59.597680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375116+0000","last_change":"2024-04-21T08:42:17.551109+0000","last_active":"2024-04-21T08:44:41.375116+0000","last_peered":"2024-04-21T08:44:41.375116+0000","last_clean":"2024-04-21T08:44:41.375116+0000","last_became_active":"2024-04-21T08:42:17.550104+0000","last_became_peered":"2024-04-21T08:42:17.550104+0000","last_unstale":"2024-04-21T08:44:41.375116+0000","last_undegraded":"2024-04-21T08:44:41.375116+0000","last_fullsized":"2024-04-21T08:44:41.375116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:22:19.361752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107298,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":792314,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":472,"num_read_kb":387,"num_write":265,"num_write_kb":2063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":2376942,"data_compressed":107241,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351019,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":532412,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36500,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708652,"statfs":{"total":95995035648,"available":95957659648,"internally_reserved":0,"allocated":1187840,"data_stored":1320546,"data_compressed":73280,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1069056,"data_stored":930306,"data_compressed":48872,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037819,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":926232,"data_compressed":49001,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299455,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":1127458,"data_compressed":65492,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462982,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528316,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1105920,"data_stored":598384,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1036288,"data_stored":528406,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":598418,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1053236,"data_compressed":55028,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888144,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":532434,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724618,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":1254562,"data_compressed":71648,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70102,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":721552,"data_compressed":34115,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":328414,"data_compressed":14840,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":398494,"data_compressed":16472,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:44:52.249 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-21T08:44:52.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:44:52.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 -- 172.21.15.7:0/3845368480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8718102b10 msgr2=0x7f8718106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 --2- 172.21.15.7:0/3845368480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8718102b10 0x7f8718106b50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f870c009900 tx=0x7f870c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:52.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 -- 172.21.15.7:0/3845368480 shutdown_connections 2024-04-21T08:44:52.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 --2- 172.21.15.7:0/3845368480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8718107230 0x7f871810ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 --2- 172.21.15.7:0/3845368480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8718102b10 0x7f8718106b50 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 --2- 172.21.15.7:0/3845368480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8718102160 0x7f8718102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 -- 172.21.15.7:0/3845368480 >> 172.21.15.7:0/3845368480 conn(0x7f87180fd9e0 msgr2=0x7f87180ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 -- 172.21.15.7:0/3845368480 shutdown_connections 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 -- 172.21.15.7:0/3845368480 wait complete. 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 Processor -- start 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 -- start start 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8718102160 0x7f8718198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8718107230 0x7f8718198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871819cfd0 0x7f871819d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8718112150 con 0x7f8718107230 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8718111fd0 con 0x7f871819cfd0 2024-04-21T08:44:52.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.388+0000 7f871ed98640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f87181122d0 con 0x7f8718102160 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8717fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8718107230 0x7f8718198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8717fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8718107230 0x7f8718198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47594/0 (socket says 172.21.15.7:47594) 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8717fff640 1 -- 172.21.15.7:0/524378169 learned_addr learned my addr 172.21.15.7:0/524378169 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f871cb34640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8718102160 0x7f8718198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f871d335640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871819cfd0 0x7f871819d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8717fff640 1 -- 172.21.15.7:0/524378169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8718102160 msgr2=0x7f8718198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8717fff640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8718102160 0x7f8718198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8717fff640 1 -- 172.21.15.7:0/524378169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871819cfd0 msgr2=0x7f871819d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8717fff640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871819cfd0 0x7f871819d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8717fff640 1 -- 172.21.15.7:0/524378169 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871819dac0 con 0x7f8718107230 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f871d335640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871819cfd0 0x7f871819d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8717fff640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8718107230 0x7f8718198ca0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f870c036040 tx=0x7f870c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8715ffb640 1 -- 172.21.15.7:0/524378169 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c043020 con 0x7f8718107230 2024-04-21T08:44:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8715ffb640 1 -- 172.21.15.7:0/524378169 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f870c02cbe0 con 0x7f8718107230 2024-04-21T08:44:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f871ed98640 1 -- 172.21.15.7:0/524378169 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f871819dd50 con 0x7f8718107230 2024-04-21T08:44:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f8715ffb640 1 -- 172.21.15.7:0/524378169 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c003bf0 con 0x7f8718107230 2024-04-21T08:44:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.389+0000 7f871ed98640 1 -- 172.21.15.7:0/524378169 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87181a5570 con 0x7f8718107230 2024-04-21T08:44:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.390+0000 7f871ed98640 1 -- 172.21.15.7:0/524378169 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8718103630 con 0x7f8718107230 2024-04-21T08:44:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.391+0000 7f8715ffb640 1 -- 172.21.15.7:0/524378169 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f870c046020 con 0x7f8718107230 2024-04-21T08:44:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.392+0000 7f8715ffb640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87000a3db0 0x7f87000a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.392+0000 7f8715ffb640 1 -- 172.21.15.7:0/524378169 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f870c0f2ff0 con 0x7f8718107230 2024-04-21T08:44:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.392+0000 7f871cb34640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87000a3db0 0x7f87000a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.393+0000 7f871cb34640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87000a3db0 0x7f87000a6270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8708005fd0 tx=0x7f8708005850 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.397+0000 7f8715ffb640 1 -- 172.21.15.7:0/524378169 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f870c04d020 con 0x7f8718107230 2024-04-21T08:44:52.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.579+0000 7f871ed98640 1 -- 172.21.15.7:0/524378169 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f871810a4d0 con 0x7f87000a3db0 2024-04-21T08:44:52.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.588+0000 7f8715ffb640 1 -- 172.21.15.7:0/524378169 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339632 (secure 0 0 0) 0x7f870800b360 con 0x7f87000a3db0 2024-04-21T08:44:52.591 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:52.593 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.592+0000 7f871ed98640 1 -- 172.21.15.7:0/524378169 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87000a3db0 msgr2=0x7f87000a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.592+0000 7f871ed98640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87000a3db0 0x7f87000a6270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8708005fd0 tx=0x7f8708005850 comp rx=0 tx=0).stop 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.592+0000 7f871ed98640 1 -- 172.21.15.7:0/524378169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8718107230 msgr2=0x7f8718198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.592+0000 7f871ed98640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8718107230 0x7f8718198ca0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f870c036040 tx=0x7f870c02c7e0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.592+0000 7f871ed98640 1 -- 172.21.15.7:0/524378169 shutdown_connections 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.592+0000 7f871ed98640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f871819cfd0 0x7f871819d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.592+0000 7f871ed98640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8718107230 0x7f8718198ca0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.592+0000 7f871ed98640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87000a3db0 0x7f87000a6270 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.592+0000 7f871ed98640 1 --2- 172.21.15.7:0/524378169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8718102160 0x7f8718198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.592+0000 7f871ed98640 1 -- 172.21.15.7:0/524378169 >> 172.21.15.7:0/524378169 conn(0x7f87180fd9e0 msgr2=0x7f87180fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:52.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.592+0000 7f871ed98640 1 -- 172.21.15.7:0/524378169 shutdown_connections 2024-04-21T08:44:52.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.593+0000 7f871ed98640 1 -- 172.21.15.7:0/524378169 wait complete. 2024-04-21T08:44:52.603 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-21T08:44:51.264294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1382916,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":709,"num_read_kb":717,"num_write":468,"num_write_kb":5003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":844,"ondisk_log_size":844,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":403456,"kb_used_data":12680,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124538368,"statfs":{"total":1151940427776,"available":1151527288832,"internally_reserved":0,"allocated":12984320,"data_stored":9930710,"data_compressed":544941,"data_compressed_allocated":737280,"data_compressed_original":4325376,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":223,"num_read_kb":318,"num_write":84,"num_write_kb":652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.991246"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375541+0000","last_change":"2024-04-21T08:42:17.545701+0000","last_active":"2024-04-21T08:44:41.375541+0000","last_peered":"2024-04-21T08:44:41.375541+0000","last_clean":"2024-04-21T08:44:41.375541+0000","last_became_active":"2024-04-21T08:42:17.545157+0000","last_became_peered":"2024-04-21T08:42:17.545157+0000","last_unstale":"2024-04-21T08:44:41.375541+0000","last_undegraded":"2024-04-21T08:44:41.375541+0000","last_fullsized":"2024-04-21T08:44:41.375541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:05:41.952879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418165+0000","last_change":"2024-04-21T08:42:17.541153+0000","last_active":"2024-04-21T08:44:41.418165+0000","last_peered":"2024-04-21T08:44:41.418165+0000","last_clean":"2024-04-21T08:44:41.418165+0000","last_became_active":"2024-04-21T08:42:17.540771+0000","last_became_peered":"2024-04-21T08:42:17.540771+0000","last_unstale":"2024-04-21T08:44:41.418165+0000","last_undegraded":"2024-04-21T08:44:41.418165+0000","last_fullsized":"2024-04-21T08:44:41.418165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:35:06.077367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417993+0000","last_change":"2024-04-21T08:42:17.545525+0000","last_active":"2024-04-21T08:44:41.417993+0000","last_peered":"2024-04-21T08:44:41.417993+0000","last_clean":"2024-04-21T08:44:41.417993+0000","last_became_active":"2024-04-21T08:42:17.545168+0000","last_became_peered":"2024-04-21T08:42:17.545168+0000","last_unstale":"2024-04-21T08:44:41.417993+0000","last_undegraded":"2024-04-21T08:44:41.417993+0000","last_fullsized":"2024-04-21T08:44:41.417993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:51.080385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251479+0000","last_change":"2024-04-21T08:42:17.543917+0000","last_active":"2024-04-21T08:44:41.251479+0000","last_peered":"2024-04-21T08:44:41.251479+0000","last_clean":"2024-04-21T08:44:41.251479+0000","last_became_active":"2024-04-21T08:42:17.543513+0000","last_became_peered":"2024-04-21T08:42:17.543513+0000","last_unstale":"2024-04-21T08:44:41.251479+0000","last_undegraded":"2024-04-21T08:44:41.251479+0000","last_fullsized":"2024-04-21T08:44:41.251479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:23.058600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250648+0000","last_change":"2024-04-21T08:42:17.549737+0000","last_active":"2024-04-21T08:44:41.250648+0000","last_peered":"2024-04-21T08:44:41.250648+0000","last_clean":"2024-04-21T08:44:41.250648+0000","last_became_active":"2024-04-21T08:42:17.549396+0000","last_became_peered":"2024-04-21T08:42:17.549396+0000","last_unstale":"2024-04-21T08:44:41.250648+0000","last_undegraded":"2024-04-21T08:44:41.250648+0000","last_fullsized":"2024-04-21T08:44:41.250648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:01:51.879960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368365+0000","last_change":"2024-04-21T08:43:10.792321+0000","last_active":"2024-04-21T08:44:41.368365+0000","last_peered":"2024-04-21T08:44:41.368365+0000","last_clean":"2024-04-21T08:44:41.368365+0000","last_became_active":"2024-04-21T08:43:10.791979+0000","last_became_peered":"2024-04-21T08:43:10.791979+0000","last_unstale":"2024-04-21T08:44:41.368365+0000","last_undegraded":"2024-04-21T08:44:41.368365+0000","last_fullsized":"2024-04-21T08:44:41.368365+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:11:16.056778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374960+0000","last_change":"2024-04-21T08:42:17.541172+0000","last_active":"2024-04-21T08:44:41.374960+0000","last_peered":"2024-04-21T08:44:41.374960+0000","last_clean":"2024-04-21T08:44:41.374960+0000","last_became_active":"2024-04-21T08:42:17.540829+0000","last_became_peered":"2024-04-21T08:42:17.540829+0000","last_unstale":"2024-04-21T08:44:41.374960+0000","last_undegraded":"2024-04-21T08:44:41.374960+0000","last_fullsized":"2024-04-21T08:44:41.374960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:16:12.633980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249835+0000","last_change":"2024-04-21T08:42:17.549744+0000","last_active":"2024-04-21T08:44:41.249835+0000","last_peered":"2024-04-21T08:44:41.249835+0000","last_clean":"2024-04-21T08:44:41.249835+0000","last_became_active":"2024-04-21T08:42:17.549378+0000","last_became_peered":"2024-04-21T08:42:17.549378+0000","last_unstale":"2024-04-21T08:44:41.249835+0000","last_undegraded":"2024-04-21T08:44:41.249835+0000","last_fullsized":"2024-04-21T08:44:41.249835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:03:07.702143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375819+0000","last_change":"2024-04-21T08:42:17.548675+0000","last_active":"2024-04-21T08:44:41.375819+0000","last_peered":"2024-04-21T08:44:41.375819+0000","last_clean":"2024-04-21T08:44:41.375819+0000","last_became_active":"2024-04-21T08:42:17.547809+0000","last_became_peered":"2024-04-21T08:42:17.547809+0000","last_unstale":"2024-04-21T08:44:41.375819+0000","last_undegraded":"2024-04-21T08:44:41.375819+0000","last_fullsized":"2024-04-21T08:44:41.375819+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:23.343473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368808+0000","last_change":"2024-04-21T08:42:17.551012+0000","last_active":"2024-04-21T08:44:41.368808+0000","last_peered":"2024-04-21T08:44:41.368808+0000","last_clean":"2024-04-21T08:44:41.368808+0000","last_became_active":"2024-04-21T08:42:17.550627+0000","last_became_peered":"2024-04-21T08:42:17.550627+0000","last_unstale":"2024-04-21T08:44:41.368808+0000","last_undegraded":"2024-04-21T08:44:41.368808+0000","last_fullsized":"2024-04-21T08:44:41.368808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:57.501588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375100+0000","last_change":"2024-04-21T08:42:17.546192+0000","last_active":"2024-04-21T08:44:41.375100+0000","last_peered":"2024-04-21T08:44:41.375100+0000","last_clean":"2024-04-21T08:44:41.375100+0000","last_became_active":"2024-04-21T08:42:17.545808+0000","last_became_peered":"2024-04-21T08:42:17.545808+0000","last_unstale":"2024-04-21T08:44:41.375100+0000","last_undegraded":"2024-04-21T08:44:41.375100+0000","last_fullsized":"2024-04-21T08:44:41.375100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:54.538898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251368+0000","last_change":"2024-04-21T08:42:17.553351+0000","last_active":"2024-04-21T08:44:41.251368+0000","last_peered":"2024-04-21T08:44:41.251368+0000","last_clean":"2024-04-21T08:44:41.251368+0000","last_became_active":"2024-04-21T08:42:17.550507+0000","last_became_peered":"2024-04-21T08:42:17.550507+0000","last_unstale":"2024-04-21T08:44:41.251368+0000","last_undegraded":"2024-04-21T08:44:41.251368+0000","last_fullsized":"2024-04-21T08:44:41.251368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:25.561375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376794+0000","last_change":"2024-04-21T08:42:17.540533+0000","last_active":"2024-04-21T08:44:41.376794+0000","last_peered":"2024-04-21T08:44:41.376794+0000","last_clean":"2024-04-21T08:44:41.376794+0000","last_became_active":"2024-04-21T08:42:17.539940+0000","last_became_peered":"2024-04-21T08:42:17.539940+0000","last_unstale":"2024-04-21T08:44:41.376794+0000","last_undegraded":"2024-04-21T08:44:41.376794+0000","last_fullsized":"2024-04-21T08:44:41.376794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:45:19.102899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417676+0000","last_change":"2024-04-21T08:42:17.537794+0000","last_active":"2024-04-21T08:44:41.417676+0000","last_peered":"2024-04-21T08:44:41.417676+0000","last_clean":"2024-04-21T08:44:41.417676+0000","last_became_active":"2024-04-21T08:42:17.537376+0000","last_became_peered":"2024-04-21T08:42:17.537376+0000","last_unstale":"2024-04-21T08:44:41.417676+0000","last_undegraded":"2024-04-21T08:44:41.417676+0000","last_fullsized":"2024-04-21T08:44:41.417676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:55:57.430036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417515+0000","last_change":"2024-04-21T08:42:17.550574+0000","last_active":"2024-04-21T08:44:41.417515+0000","last_peered":"2024-04-21T08:44:41.417515+0000","last_clean":"2024-04-21T08:44:41.417515+0000","last_became_active":"2024-04-21T08:42:17.550205+0000","last_became_peered":"2024-04-21T08:42:17.550205+0000","last_unstale":"2024-04-21T08:44:41.417515+0000","last_undegraded":"2024-04-21T08:44:41.417515+0000","last_fullsized":"2024-04-21T08:44:41.417515+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:41:08.226477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374544+0000","last_change":"2024-04-21T08:42:17.542239+0000","last_active":"2024-04-21T08:44:41.374544+0000","last_peered":"2024-04-21T08:44:41.374544+0000","last_clean":"2024-04-21T08:44:41.374544+0000","last_became_active":"2024-04-21T08:42:17.541848+0000","last_became_peered":"2024-04-21T08:42:17.541848+0000","last_unstale":"2024-04-21T08:44:41.374544+0000","last_undegraded":"2024-04-21T08:44:41.374544+0000","last_fullsized":"2024-04-21T08:44:41.374544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:28:47.717451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375417+0000","last_change":"2024-04-21T08:42:17.553200+0000","last_active":"2024-04-21T08:44:41.375417+0000","last_peered":"2024-04-21T08:44:41.375417+0000","last_clean":"2024-04-21T08:44:41.375417+0000","last_became_active":"2024-04-21T08:42:17.552913+0000","last_became_peered":"2024-04-21T08:42:17.552913+0000","last_unstale":"2024-04-21T08:44:41.375417+0000","last_undegraded":"2024-04-21T08:44:41.375417+0000","last_fullsized":"2024-04-21T08:44:41.375417+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:39:04.186928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356904+0000","last_change":"2024-04-21T08:42:17.534881+0000","last_active":"2024-04-21T08:44:41.356904+0000","last_peered":"2024-04-21T08:44:41.356904+0000","last_clean":"2024-04-21T08:44:41.356904+0000","last_became_active":"2024-04-21T08:42:17.534496+0000","last_became_peered":"2024-04-21T08:42:17.534496+0000","last_unstale":"2024-04-21T08:44:41.356904+0000","last_undegraded":"2024-04-21T08:44:41.356904+0000","last_fullsized":"2024-04-21T08:44:41.356904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:50:19.409391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375734+0000","last_change":"2024-04-21T08:42:17.540480+0000","last_active":"2024-04-21T08:44:41.375734+0000","last_peered":"2024-04-21T08:44:41.375734+0000","last_clean":"2024-04-21T08:44:41.375734+0000","last_became_active":"2024-04-21T08:42:17.539742+0000","last_became_peered":"2024-04-21T08:42:17.539742+0000","last_unstale":"2024-04-21T08:44:41.375734+0000","last_undegraded":"2024-04-21T08:44:41.375734+0000","last_fullsized":"2024-04-21T08:44:41.375734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:00.238976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374953+0000","last_change":"2024-04-21T08:44:11.876826+0000","last_active":"2024-04-21T08:44:41.374953+0000","last_peered":"2024-04-21T08:44:41.374953+0000","last_clean":"2024-04-21T08:44:41.374953+0000","last_became_active":"2024-04-21T08:44:11.876354+0000","last_became_peered":"2024-04-21T08:44:11.876354+0000","last_unstale":"2024-04-21T08:44:41.374953+0000","last_undegraded":"2024-04-21T08:44:41.374953+0000","last_fullsized":"2024-04-21T08:44:41.374953+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:00:05.574008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356838+0000","last_change":"2024-04-21T08:42:16.639888+0000","last_active":"2024-04-21T08:44:41.356838+0000","last_peered":"2024-04-21T08:44:41.356838+0000","last_clean":"2024-04-21T08:44:41.356838+0000","last_became_active":"2024-04-21T08:42:16.639403+0000","last_became_peered":"2024-04-21T08:42:16.639403+0000","last_unstale":"2024-04-21T08:44:41.356838+0000","last_undegraded":"2024-04-21T08:44:41.356838+0000","last_fullsized":"2024-04-21T08:44:41.356838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:08:17.164924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249935+0000","last_change":"2024-04-21T08:43:10.983368+0000","last_active":"2024-04-21T08:44:41.249935+0000","last_peered":"2024-04-21T08:44:41.249935+0000","last_clean":"2024-04-21T08:44:41.249935+0000","last_became_active":"2024-04-21T08:43:10.982956+0000","last_became_peered":"2024-04-21T08:43:10.982956+0000","last_unstale":"2024-04-21T08:44:41.249935+0000","last_undegraded":"2024-04-21T08:44:41.249935+0000","last_fullsized":"2024-04-21T08:44:41.249935+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:12:44.919996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250779+0000","last_change":"2024-04-21T08:42:21.673197+0000","last_active":"2024-04-21T08:44:41.250779+0000","last_peered":"2024-04-21T08:44:41.250779+0000","last_clean":"2024-04-21T08:44:41.250779+0000","last_became_active":"2024-04-21T08:42:21.672793+0000","last_became_peered":"2024-04-21T08:42:21.672793+0000","last_unstale":"2024-04-21T08:44:41.250779+0000","last_undegraded":"2024-04-21T08:44:41.250779+0000","last_fullsized":"2024-04-21T08:44:41.250779+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:44:29.788459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250990+0000","last_change":"2024-04-21T08:42:16.648577+0000","last_active":"2024-04-21T08:44:41.250990+0000","last_peered":"2024-04-21T08:44:41.250990+0000","last_clean":"2024-04-21T08:44:41.250990+0000","last_became_active":"2024-04-21T08:42:16.646716+0000","last_became_peered":"2024-04-21T08:42:16.646716+0000","last_unstale":"2024-04-21T08:44:41.250990+0000","last_undegraded":"2024-04-21T08:44:41.250990+0000","last_fullsized":"2024-04-21T08:44:41.250990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:09:10.327078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417936+0000","last_change":"2024-04-21T08:42:17.545430+0000","last_active":"2024-04-21T08:44:41.417936+0000","last_peered":"2024-04-21T08:44:41.417936+0000","last_clean":"2024-04-21T08:44:41.417936+0000","last_became_active":"2024-04-21T08:42:17.545057+0000","last_became_peered":"2024-04-21T08:42:17.545057+0000","last_unstale":"2024-04-21T08:44:41.417936+0000","last_undegraded":"2024-04-21T08:44:41.417936+0000","last_fullsized":"2024-04-21T08:44:41.417936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:44:59.030830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250899+0000","last_change":"2024-04-21T08:42:21.673196+0000","last_active":"2024-04-21T08:44:41.250899+0000","last_peered":"2024-04-21T08:44:41.250899+0000","last_clean":"2024-04-21T08:44:41.250899+0000","last_became_active":"2024-04-21T08:42:21.672808+0000","last_became_peered":"2024-04-21T08:42:21.672808+0000","last_unstale":"2024-04-21T08:44:41.250899+0000","last_undegraded":"2024-04-21T08:44:41.250899+0000","last_fullsized":"2024-04-21T08:44:41.250899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:39.482353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250844+0000","last_change":"2024-04-21T08:42:16.641611+0000","last_active":"2024-04-21T08:44:41.250844+0000","last_peered":"2024-04-21T08:44:41.250844+0000","last_clean":"2024-04-21T08:44:41.250844+0000","last_became_active":"2024-04-21T08:42:16.641221+0000","last_became_peered":"2024-04-21T08:42:16.641221+0000","last_unstale":"2024-04-21T08:44:41.250844+0000","last_undegraded":"2024-04-21T08:44:41.250844+0000","last_fullsized":"2024-04-21T08:44:41.250844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:45.496532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375448+0000","last_change":"2024-04-21T08:42:17.532986+0000","last_active":"2024-04-21T08:44:41.375448+0000","last_peered":"2024-04-21T08:44:41.375448+0000","last_clean":"2024-04-21T08:44:41.375448+0000","last_became_active":"2024-04-21T08:42:17.532583+0000","last_became_peered":"2024-04-21T08:42:17.532583+0000","last_unstale":"2024-04-21T08:44:41.375448+0000","last_undegraded":"2024-04-21T08:44:41.375448+0000","last_fullsized":"2024-04-21T08:44:41.375448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:22:43.780935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250791+0000","last_change":"2024-04-21T08:42:21.667459+0000","last_active":"2024-04-21T08:44:41.250791+0000","last_peered":"2024-04-21T08:44:41.250791+0000","last_clean":"2024-04-21T08:44:41.250791+0000","last_became_active":"2024-04-21T08:42:21.667183+0000","last_became_peered":"2024-04-21T08:42:21.667183+0000","last_unstale":"2024-04-21T08:44:41.250791+0000","last_undegraded":"2024-04-21T08:44:41.250791+0000","last_fullsized":"2024-04-21T08:44:41.250791+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:32:23.626267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250969+0000","last_change":"2024-04-21T08:42:16.645210+0000","last_active":"2024-04-21T08:44:41.250969+0000","last_peered":"2024-04-21T08:44:41.250969+0000","last_clean":"2024-04-21T08:44:41.250969+0000","last_became_active":"2024-04-21T08:42:16.644747+0000","last_became_peered":"2024-04-21T08:42:16.644747+0000","last_unstale":"2024-04-21T08:44:41.250969+0000","last_undegraded":"2024-04-21T08:44:41.250969+0000","last_fullsized":"2024-04-21T08:44:41.250969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:34:45.131205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365569+0000","last_change":"2024-04-21T08:43:10.984152+0000","last_active":"2024-04-21T08:44:41.365569+0000","last_peered":"2024-04-21T08:44:41.365569+0000","last_clean":"2024-04-21T08:44:41.365569+0000","last_became_active":"2024-04-21T08:43:10.983736+0000","last_became_peered":"2024-04-21T08:43:10.983736+0000","last_unstale":"2024-04-21T08:44:41.365569+0000","last_undegraded":"2024-04-21T08:44:41.365569+0000","last_fullsized":"2024-04-21T08:44:41.365569+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:42:10.182667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375391+0000","last_change":"2024-04-21T08:42:21.660739+0000","last_active":"2024-04-21T08:44:41.375391+0000","last_peered":"2024-04-21T08:44:41.375391+0000","last_clean":"2024-04-21T08:44:41.375391+0000","last_became_active":"2024-04-21T08:42:21.660307+0000","last_became_peered":"2024-04-21T08:42:21.660307+0000","last_unstale":"2024-04-21T08:44:41.375391+0000","last_undegraded":"2024-04-21T08:44:41.375391+0000","last_fullsized":"2024-04-21T08:44:41.375391+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:26.447366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365469+0000","last_change":"2024-04-21T08:43:10.986184+0000","last_active":"2024-04-21T08:44:41.365469+0000","last_peered":"2024-04-21T08:44:41.365469+0000","last_clean":"2024-04-21T08:44:41.365469+0000","last_became_active":"2024-04-21T08:43:10.985533+0000","last_became_peered":"2024-04-21T08:43:10.985533+0000","last_unstale":"2024-04-21T08:44:41.365469+0000","last_undegraded":"2024-04-21T08:44:41.365469+0000","last_fullsized":"2024-04-21T08:44:41.365469+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:26.023306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250741+0000","last_change":"2024-04-21T08:42:17.542315+0000","last_active":"2024-04-21T08:44:41.250741+0000","last_peered":"2024-04-21T08:44:41.250741+0000","last_clean":"2024-04-21T08:44:41.250741+0000","last_became_active":"2024-04-21T08:42:17.541895+0000","last_became_peered":"2024-04-21T08:42:17.541895+0000","last_unstale":"2024-04-21T08:44:41.250741+0000","last_undegraded":"2024-04-21T08:44:41.250741+0000","last_fullsized":"2024-04-21T08:44:41.250741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:23:33.909499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375270+0000","last_change":"2024-04-21T08:44:11.878135+0000","last_active":"2024-04-21T08:44:41.375270+0000","last_peered":"2024-04-21T08:44:41.375270+0000","last_clean":"2024-04-21T08:44:41.375270+0000","last_became_active":"2024-04-21T08:44:11.877271+0000","last_became_peered":"2024-04-21T08:44:11.877271+0000","last_unstale":"2024-04-21T08:44:41.375270+0000","last_undegraded":"2024-04-21T08:44:41.375270+0000","last_fullsized":"2024-04-21T08:44:41.375270+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:04:31.345519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368426+0000","last_change":"2024-04-21T08:43:10.984839+0000","last_active":"2024-04-21T08:44:41.368426+0000","last_peered":"2024-04-21T08:44:41.368426+0000","last_clean":"2024-04-21T08:44:41.368426+0000","last_became_active":"2024-04-21T08:43:10.984382+0000","last_became_peered":"2024-04-21T08:43:10.984382+0000","last_unstale":"2024-04-21T08:44:41.368426+0000","last_undegraded":"2024-04-21T08:44:41.368426+0000","last_fullsized":"2024-04-21T08:44:41.368426+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:47:20.774102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365597+0000","last_change":"2024-04-21T08:42:17.547770+0000","last_active":"2024-04-21T08:44:41.365597+0000","last_peered":"2024-04-21T08:44:41.365597+0000","last_clean":"2024-04-21T08:44:41.365597+0000","last_became_active":"2024-04-21T08:42:17.547440+0000","last_became_peered":"2024-04-21T08:42:17.547440+0000","last_unstale":"2024-04-21T08:44:41.365597+0000","last_undegraded":"2024-04-21T08:44:41.365597+0000","last_fullsized":"2024-04-21T08:44:41.365597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:37:35.555737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375243+0000","last_change":"2024-04-21T08:42:21.670684+0000","last_active":"2024-04-21T08:44:41.375243+0000","last_peered":"2024-04-21T08:44:41.375243+0000","last_clean":"2024-04-21T08:44:41.375243+0000","last_became_active":"2024-04-21T08:42:21.670260+0000","last_became_peered":"2024-04-21T08:42:21.670260+0000","last_unstale":"2024-04-21T08:44:41.375243+0000","last_undegraded":"2024-04-21T08:44:41.375243+0000","last_fullsized":"2024-04-21T08:44:41.375243+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:45:06.192040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376149+0000","last_change":"2024-04-21T08:42:16.639632+0000","last_active":"2024-04-21T08:44:41.376149+0000","last_peered":"2024-04-21T08:44:41.376149+0000","last_clean":"2024-04-21T08:44:41.376149+0000","last_became_active":"2024-04-21T08:42:16.639301+0000","last_became_peered":"2024-04-21T08:42:16.639301+0000","last_unstale":"2024-04-21T08:44:41.376149+0000","last_undegraded":"2024-04-21T08:44:41.376149+0000","last_fullsized":"2024-04-21T08:44:41.376149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:58.838667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249940+0000","last_change":"2024-04-21T08:42:17.546109+0000","last_active":"2024-04-21T08:44:41.249940+0000","last_peered":"2024-04-21T08:44:41.249940+0000","last_clean":"2024-04-21T08:44:41.249940+0000","last_became_active":"2024-04-21T08:42:17.545673+0000","last_became_peered":"2024-04-21T08:42:17.545673+0000","last_unstale":"2024-04-21T08:44:41.249940+0000","last_undegraded":"2024-04-21T08:44:41.249940+0000","last_fullsized":"2024-04-21T08:44:41.249940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:35:18.947848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417985+0000","last_change":"2024-04-21T08:42:21.666114+0000","last_active":"2024-04-21T08:44:41.417985+0000","last_peered":"2024-04-21T08:44:41.417985+0000","last_clean":"2024-04-21T08:44:41.417985+0000","last_became_active":"2024-04-21T08:42:21.665020+0000","last_became_peered":"2024-04-21T08:42:21.665020+0000","last_unstale":"2024-04-21T08:44:41.417985+0000","last_undegraded":"2024-04-21T08:44:41.417985+0000","last_fullsized":"2024-04-21T08:44:41.417985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:33.644178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365224+0000","last_change":"2024-04-21T08:42:16.639118+0000","last_active":"2024-04-21T08:44:41.365224+0000","last_peered":"2024-04-21T08:44:41.365224+0000","last_clean":"2024-04-21T08:44:41.365224+0000","last_became_active":"2024-04-21T08:42:16.638792+0000","last_became_peered":"2024-04-21T08:42:16.638792+0000","last_unstale":"2024-04-21T08:44:41.365224+0000","last_undegraded":"2024-04-21T08:44:41.365224+0000","last_fullsized":"2024-04-21T08:44:41.365224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:55.568712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356215+0000","last_change":"2024-04-21T08:42:17.539688+0000","last_active":"2024-04-21T08:44:41.356215+0000","last_peered":"2024-04-21T08:44:41.356215+0000","last_clean":"2024-04-21T08:44:41.356215+0000","last_became_active":"2024-04-21T08:42:17.539335+0000","last_became_peered":"2024-04-21T08:42:17.539335+0000","last_unstale":"2024-04-21T08:44:41.356215+0000","last_undegraded":"2024-04-21T08:44:41.356215+0000","last_fullsized":"2024-04-21T08:44:41.356215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:27:51.520232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249882+0000","last_change":"2024-04-21T08:42:21.660570+0000","last_active":"2024-04-21T08:44:41.249882+0000","last_peered":"2024-04-21T08:44:41.249882+0000","last_clean":"2024-04-21T08:44:41.249882+0000","last_became_active":"2024-04-21T08:42:21.659398+0000","last_became_peered":"2024-04-21T08:42:21.659398+0000","last_unstale":"2024-04-21T08:44:41.249882+0000","last_undegraded":"2024-04-21T08:44:41.249882+0000","last_fullsized":"2024-04-21T08:44:41.249882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:55.840334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368786+0000","last_change":"2024-04-21T08:42:16.648192+0000","last_active":"2024-04-21T08:44:41.368786+0000","last_peered":"2024-04-21T08:44:41.368786+0000","last_clean":"2024-04-21T08:44:41.368786+0000","last_became_active":"2024-04-21T08:42:16.647666+0000","last_became_peered":"2024-04-21T08:42:16.647666+0000","last_unstale":"2024-04-21T08:44:41.368786+0000","last_undegraded":"2024-04-21T08:44:41.368786+0000","last_fullsized":"2024-04-21T08:44:41.368786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:33.344680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375708+0000","last_change":"2024-04-21T08:42:17.545925+0000","last_active":"2024-04-21T08:44:41.375708+0000","last_peered":"2024-04-21T08:44:41.375708+0000","last_clean":"2024-04-21T08:44:41.375708+0000","last_became_active":"2024-04-21T08:42:17.545555+0000","last_became_peered":"2024-04-21T08:42:17.545555+0000","last_unstale":"2024-04-21T08:44:41.375708+0000","last_undegraded":"2024-04-21T08:44:41.375708+0000","last_fullsized":"2024-04-21T08:44:41.375708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:13:07.788972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356087+0000","last_change":"2024-04-21T08:42:21.662757+0000","last_active":"2024-04-21T08:44:41.356087+0000","last_peered":"2024-04-21T08:44:41.356087+0000","last_clean":"2024-04-21T08:44:41.356087+0000","last_became_active":"2024-04-21T08:42:21.662306+0000","last_became_peered":"2024-04-21T08:42:21.662306+0000","last_unstale":"2024-04-21T08:44:41.356087+0000","last_undegraded":"2024-04-21T08:44:41.356087+0000","last_fullsized":"2024-04-21T08:44:41.356087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:26.856147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375135+0000","last_change":"2024-04-21T08:43:10.984690+0000","last_active":"2024-04-21T08:44:41.375135+0000","last_peered":"2024-04-21T08:44:41.375135+0000","last_clean":"2024-04-21T08:44:41.375135+0000","last_became_active":"2024-04-21T08:43:10.984290+0000","last_became_peered":"2024-04-21T08:43:10.984290+0000","last_unstale":"2024-04-21T08:44:41.375135+0000","last_undegraded":"2024-04-21T08:44:41.375135+0000","last_fullsized":"2024-04-21T08:44:41.375135+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:37:36.400946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418259+0000","last_change":"2024-04-21T08:43:10.793316+0000","last_active":"2024-04-21T08:44:41.418259+0000","last_peered":"2024-04-21T08:44:41.418259+0000","last_clean":"2024-04-21T08:44:41.418259+0000","last_became_active":"2024-04-21T08:43:10.793035+0000","last_became_peered":"2024-04-21T08:43:10.793035+0000","last_unstale":"2024-04-21T08:44:41.418259+0000","last_undegraded":"2024-04-21T08:44:41.418259+0000","last_fullsized":"2024-04-21T08:44:41.418259+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:41:04.247385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251083+0000","last_change":"2024-04-21T08:42:21.670561+0000","last_active":"2024-04-21T08:44:41.251083+0000","last_peered":"2024-04-21T08:44:41.251083+0000","last_clean":"2024-04-21T08:44:41.251083+0000","last_became_active":"2024-04-21T08:42:21.670170+0000","last_became_peered":"2024-04-21T08:42:21.670170+0000","last_unstale":"2024-04-21T08:44:41.251083+0000","last_undegraded":"2024-04-21T08:44:41.251083+0000","last_fullsized":"2024-04-21T08:44:41.251083+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:35:54.538013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365460+0000","last_change":"2024-04-21T08:42:16.637319+0000","last_active":"2024-04-21T08:44:41.365460+0000","last_peered":"2024-04-21T08:44:41.365460+0000","last_clean":"2024-04-21T08:44:41.365460+0000","last_became_active":"2024-04-21T08:42:16.636940+0000","last_became_peered":"2024-04-21T08:42:16.636940+0000","last_unstale":"2024-04-21T08:44:41.365460+0000","last_undegraded":"2024-04-21T08:44:41.365460+0000","last_fullsized":"2024-04-21T08:44:41.365460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:59:09.826786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375697+0000","last_change":"2024-04-21T08:42:17.546202+0000","last_active":"2024-04-21T08:44:41.375697+0000","last_peered":"2024-04-21T08:44:41.375697+0000","last_clean":"2024-04-21T08:44:41.375697+0000","last_became_active":"2024-04-21T08:42:17.545781+0000","last_became_peered":"2024-04-21T08:42:17.545781+0000","last_unstale":"2024-04-21T08:44:41.375697+0000","last_undegraded":"2024-04-21T08:44:41.375697+0000","last_fullsized":"2024-04-21T08:44:41.375697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:23.990951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356160+0000","last_change":"2024-04-21T08:42:21.662822+0000","last_active":"2024-04-21T08:44:41.356160+0000","last_peered":"2024-04-21T08:44:41.356160+0000","last_clean":"2024-04-21T08:44:41.356160+0000","last_became_active":"2024-04-21T08:42:21.662398+0000","last_became_peered":"2024-04-21T08:42:21.662398+0000","last_unstale":"2024-04-21T08:44:41.356160+0000","last_undegraded":"2024-04-21T08:44:41.356160+0000","last_fullsized":"2024-04-21T08:44:41.356160+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:38:40.499294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":335,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:47.138883+0000","last_change":"2024-04-21T08:42:16.636330+0000","last_active":"2024-04-21T08:44:47.138883+0000","last_peered":"2024-04-21T08:44:47.138883+0000","last_clean":"2024-04-21T08:44:47.138883+0000","last_became_active":"2024-04-21T08:42:16.635762+0000","last_became_peered":"2024-04-21T08:42:16.635762+0000","last_unstale":"2024-04-21T08:44:47.138883+0000","last_undegraded":"2024-04-21T08:44:47.138883+0000","last_fullsized":"2024-04-21T08:44:47.138883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:31.137923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":263,"num_read_kb":201,"num_write":38,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251043+0000","last_change":"2024-04-21T08:42:17.543033+0000","last_active":"2024-04-21T08:44:41.251043+0000","last_peered":"2024-04-21T08:44:41.251043+0000","last_clean":"2024-04-21T08:44:41.251043+0000","last_became_active":"2024-04-21T08:42:17.542692+0000","last_became_peered":"2024-04-21T08:42:17.542692+0000","last_unstale":"2024-04-21T08:44:41.251043+0000","last_undegraded":"2024-04-21T08:44:41.251043+0000","last_fullsized":"2024-04-21T08:44:41.251043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:33:32.111999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365806+0000","last_change":"2024-04-21T08:42:21.668831+0000","last_active":"2024-04-21T08:44:41.365806+0000","last_peered":"2024-04-21T08:44:41.365806+0000","last_clean":"2024-04-21T08:44:41.365806+0000","last_became_active":"2024-04-21T08:42:21.668485+0000","last_became_peered":"2024-04-21T08:42:21.668485+0000","last_unstale":"2024-04-21T08:44:41.365806+0000","last_undegraded":"2024-04-21T08:44:41.365806+0000","last_fullsized":"2024-04-21T08:44:41.365806+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:49:29.049707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375241+0000","last_change":"2024-04-21T08:42:16.640288+0000","last_active":"2024-04-21T08:44:41.375241+0000","last_peered":"2024-04-21T08:44:41.375241+0000","last_clean":"2024-04-21T08:44:41.375241+0000","last_became_active":"2024-04-21T08:42:16.639909+0000","last_became_peered":"2024-04-21T08:42:16.639909+0000","last_unstale":"2024-04-21T08:44:41.375241+0000","last_undegraded":"2024-04-21T08:44:41.375241+0000","last_fullsized":"2024-04-21T08:44:41.375241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:33.839442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375479+0000","last_change":"2024-04-21T08:42:17.541154+0000","last_active":"2024-04-21T08:44:41.375479+0000","last_peered":"2024-04-21T08:44:41.375479+0000","last_clean":"2024-04-21T08:44:41.375479+0000","last_became_active":"2024-04-21T08:42:17.540747+0000","last_became_peered":"2024-04-21T08:42:17.540747+0000","last_unstale":"2024-04-21T08:44:41.375479+0000","last_undegraded":"2024-04-21T08:44:41.375479+0000","last_fullsized":"2024-04-21T08:44:41.375479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:14:50.641144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375040+0000","last_change":"2024-04-21T08:42:21.670767+0000","last_active":"2024-04-21T08:44:41.375040+0000","last_peered":"2024-04-21T08:44:41.375040+0000","last_clean":"2024-04-21T08:44:41.375040+0000","last_became_active":"2024-04-21T08:42:21.670338+0000","last_became_peered":"2024-04-21T08:42:21.670338+0000","last_unstale":"2024-04-21T08:44:41.375040+0000","last_undegraded":"2024-04-21T08:44:41.375040+0000","last_fullsized":"2024-04-21T08:44:41.375040+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:38:55.275603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365468+0000","last_change":"2024-04-21T08:42:16.640144+0000","last_active":"2024-04-21T08:44:41.365468+0000","last_peered":"2024-04-21T08:44:41.365468+0000","last_clean":"2024-04-21T08:44:41.365468+0000","last_became_active":"2024-04-21T08:42:16.639707+0000","last_became_peered":"2024-04-21T08:42:16.639707+0000","last_unstale":"2024-04-21T08:44:41.365468+0000","last_undegraded":"2024-04-21T08:44:41.365468+0000","last_fullsized":"2024-04-21T08:44:41.365468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:11:32.785519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375240+0000","last_change":"2024-04-21T08:42:17.555019+0000","last_active":"2024-04-21T08:44:41.375240+0000","last_peered":"2024-04-21T08:44:41.375240+0000","last_clean":"2024-04-21T08:44:41.375240+0000","last_became_active":"2024-04-21T08:42:17.554645+0000","last_became_peered":"2024-04-21T08:42:17.554645+0000","last_unstale":"2024-04-21T08:44:41.375240+0000","last_undegraded":"2024-04-21T08:44:41.375240+0000","last_fullsized":"2024-04-21T08:44:41.375240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:42:30.353945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251223+0000","last_change":"2024-04-21T08:44:11.876740+0000","last_active":"2024-04-21T08:44:41.251223+0000","last_peered":"2024-04-21T08:44:41.251223+0000","last_clean":"2024-04-21T08:44:41.251223+0000","last_became_active":"2024-04-21T08:44:11.876108+0000","last_became_peered":"2024-04-21T08:44:11.876108+0000","last_unstale":"2024-04-21T08:44:41.251223+0000","last_undegraded":"2024-04-21T08:44:41.251223+0000","last_fullsized":"2024-04-21T08:44:41.251223+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:37.398743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250912+0000","last_change":"2024-04-21T08:42:16.640407+0000","last_active":"2024-04-21T08:44:41.250912+0000","last_peered":"2024-04-21T08:44:41.250912+0000","last_clean":"2024-04-21T08:44:41.250912+0000","last_became_active":"2024-04-21T08:42:16.640015+0000","last_became_peered":"2024-04-21T08:42:16.640015+0000","last_unstale":"2024-04-21T08:44:41.250912+0000","last_undegraded":"2024-04-21T08:44:41.250912+0000","last_fullsized":"2024-04-21T08:44:41.250912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:23:10.647184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374762+0000","last_change":"2024-04-21T08:42:17.545438+0000","last_active":"2024-04-21T08:44:41.374762+0000","last_peered":"2024-04-21T08:44:41.374762+0000","last_clean":"2024-04-21T08:44:41.374762+0000","last_became_active":"2024-04-21T08:42:17.544258+0000","last_became_peered":"2024-04-21T08:42:17.544258+0000","last_unstale":"2024-04-21T08:44:41.374762+0000","last_undegraded":"2024-04-21T08:44:41.374762+0000","last_fullsized":"2024-04-21T08:44:41.374762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:43.648248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376425+0000","last_change":"2024-04-21T08:42:21.671000+0000","last_active":"2024-04-21T08:44:41.376425+0000","last_peered":"2024-04-21T08:44:41.376425+0000","last_clean":"2024-04-21T08:44:41.376425+0000","last_became_active":"2024-04-21T08:42:21.670380+0000","last_became_peered":"2024-04-21T08:42:21.670380+0000","last_unstale":"2024-04-21T08:44:41.376425+0000","last_undegraded":"2024-04-21T08:44:41.376425+0000","last_fullsized":"2024-04-21T08:44:41.376425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:57.075686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250621+0000","last_change":"2024-04-21T08:42:16.649599+0000","last_active":"2024-04-21T08:44:41.250621+0000","last_peered":"2024-04-21T08:44:41.250621+0000","last_clean":"2024-04-21T08:44:41.250621+0000","last_became_active":"2024-04-21T08:42:16.649232+0000","last_became_peered":"2024-04-21T08:42:16.649232+0000","last_unstale":"2024-04-21T08:44:41.250621+0000","last_undegraded":"2024-04-21T08:44:41.250621+0000","last_fullsized":"2024-04-21T08:44:41.250621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:21.195422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356045+0000","last_change":"2024-04-21T08:44:11.776138+0000","last_active":"2024-04-21T08:44:41.356045+0000","last_peered":"2024-04-21T08:44:41.356045+0000","last_clean":"2024-04-21T08:44:41.356045+0000","last_became_active":"2024-04-21T08:44:11.775832+0000","last_became_peered":"2024-04-21T08:44:11.775832+0000","last_unstale":"2024-04-21T08:44:41.356045+0000","last_undegraded":"2024-04-21T08:44:41.356045+0000","last_fullsized":"2024-04-21T08:44:41.356045+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:03:34.210923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365172+0000","last_change":"2024-04-21T08:42:21.664811+0000","last_active":"2024-04-21T08:44:41.365172+0000","last_peered":"2024-04-21T08:44:41.365172+0000","last_clean":"2024-04-21T08:44:41.365172+0000","last_became_active":"2024-04-21T08:42:21.664332+0000","last_became_peered":"2024-04-21T08:42:21.664332+0000","last_unstale":"2024-04-21T08:44:41.365172+0000","last_undegraded":"2024-04-21T08:44:41.365172+0000","last_fullsized":"2024-04-21T08:44:41.365172+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:18:40.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251372+0000","last_change":"2024-04-21T08:42:16.641233+0000","last_active":"2024-04-21T08:44:41.251372+0000","last_peered":"2024-04-21T08:44:41.251372+0000","last_clean":"2024-04-21T08:44:41.251372+0000","last_became_active":"2024-04-21T08:42:16.640873+0000","last_became_peered":"2024-04-21T08:42:16.640873+0000","last_unstale":"2024-04-21T08:44:41.251372+0000","last_undegraded":"2024-04-21T08:44:41.251372+0000","last_fullsized":"2024-04-21T08:44:41.251372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:00.682097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375293+0000","last_change":"2024-04-21T08:42:17.550438+0000","last_active":"2024-04-21T08:44:41.375293+0000","last_peered":"2024-04-21T08:44:41.375293+0000","last_clean":"2024-04-21T08:44:41.375293+0000","last_became_active":"2024-04-21T08:42:17.550015+0000","last_became_peered":"2024-04-21T08:42:17.550015+0000","last_unstale":"2024-04-21T08:44:41.375293+0000","last_undegraded":"2024-04-21T08:44:41.375293+0000","last_fullsized":"2024-04-21T08:44:41.375293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:00:29.823151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250709+0000","last_change":"2024-04-21T08:42:21.970498+0000","last_active":"2024-04-21T08:44:41.250709+0000","last_peered":"2024-04-21T08:44:41.250709+0000","last_clean":"2024-04-21T08:44:41.250709+0000","last_became_active":"2024-04-21T08:42:21.969819+0000","last_became_peered":"2024-04-21T08:42:21.969819+0000","last_unstale":"2024-04-21T08:44:41.250709+0000","last_undegraded":"2024-04-21T08:44:41.250709+0000","last_fullsized":"2024-04-21T08:44:41.250709+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:13:16.552293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356527+0000","last_change":"2024-04-21T08:42:16.637146+0000","last_active":"2024-04-21T08:44:41.356527+0000","last_peered":"2024-04-21T08:44:41.356527+0000","last_clean":"2024-04-21T08:44:41.356527+0000","last_became_active":"2024-04-21T08:42:16.636781+0000","last_became_peered":"2024-04-21T08:42:16.636781+0000","last_unstale":"2024-04-21T08:44:41.356527+0000","last_undegraded":"2024-04-21T08:44:41.356527+0000","last_fullsized":"2024-04-21T08:44:41.356527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:39:27.641236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368622+0000","last_change":"2024-04-21T08:42:17.542325+0000","last_active":"2024-04-21T08:44:41.368622+0000","last_peered":"2024-04-21T08:44:41.368622+0000","last_clean":"2024-04-21T08:44:41.368622+0000","last_became_active":"2024-04-21T08:42:17.541103+0000","last_became_peered":"2024-04-21T08:42:17.541103+0000","last_unstale":"2024-04-21T08:44:41.368622+0000","last_undegraded":"2024-04-21T08:44:41.368622+0000","last_fullsized":"2024-04-21T08:44:41.368622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:46.903250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375702+0000","last_change":"2024-04-21T08:42:21.671146+0000","last_active":"2024-04-21T08:44:41.375702+0000","last_peered":"2024-04-21T08:44:41.375702+0000","last_clean":"2024-04-21T08:44:41.375702+0000","last_became_active":"2024-04-21T08:42:21.670561+0000","last_became_peered":"2024-04-21T08:42:21.670561+0000","last_unstale":"2024-04-21T08:44:41.375702+0000","last_undegraded":"2024-04-21T08:44:41.375702+0000","last_fullsized":"2024-04-21T08:44:41.375702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:38:28.028976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375066+0000","last_change":"2024-04-21T08:42:16.645132+0000","last_active":"2024-04-21T08:44:41.375066+0000","last_peered":"2024-04-21T08:44:41.375066+0000","last_clean":"2024-04-21T08:44:41.375066+0000","last_became_active":"2024-04-21T08:42:16.644786+0000","last_became_peered":"2024-04-21T08:42:16.644786+0000","last_unstale":"2024-04-21T08:44:41.375066+0000","last_undegraded":"2024-04-21T08:44:41.375066+0000","last_fullsized":"2024-04-21T08:44:41.375066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:56:08.298498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.368567+0000","last_change":"2024-04-21T08:42:17.550424+0000","last_active":"2024-04-21T08:44:41.368567+0000","last_peered":"2024-04-21T08:44:41.368567+0000","last_clean":"2024-04-21T08:44:41.368567+0000","last_became_active":"2024-04-21T08:42:17.550077+0000","last_became_peered":"2024-04-21T08:42:17.550077+0000","last_unstale":"2024-04-21T08:44:41.368567+0000","last_undegraded":"2024-04-21T08:44:41.368567+0000","last_fullsized":"2024-04-21T08:44:41.368567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:17:07.016204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375856+0000","last_change":"2024-04-21T08:42:21.970652+0000","last_active":"2024-04-21T08:44:41.375856+0000","last_peered":"2024-04-21T08:44:41.375856+0000","last_clean":"2024-04-21T08:44:41.375856+0000","last_became_active":"2024-04-21T08:42:21.970251+0000","last_became_peered":"2024-04-21T08:42:21.970251+0000","last_unstale":"2024-04-21T08:44:41.375856+0000","last_undegraded":"2024-04-21T08:44:41.375856+0000","last_fullsized":"2024-04-21T08:44:41.375856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:45.598654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365285+0000","last_change":"2024-04-21T08:42:17.542820+0000","last_active":"2024-04-21T08:44:41.365285+0000","last_peered":"2024-04-21T08:44:41.365285+0000","last_clean":"2024-04-21T08:44:41.365285+0000","last_became_active":"2024-04-21T08:42:17.541320+0000","last_became_peered":"2024-04-21T08:42:17.541320+0000","last_unstale":"2024-04-21T08:44:41.365285+0000","last_undegraded":"2024-04-21T08:44:41.365285+0000","last_fullsized":"2024-04-21T08:44:41.365285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:23:06.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375209+0000","last_change":"2024-04-21T08:42:16.636950+0000","last_active":"2024-04-21T08:44:41.375209+0000","last_peered":"2024-04-21T08:44:41.375209+0000","last_clean":"2024-04-21T08:44:41.375209+0000","last_became_active":"2024-04-21T08:42:16.636590+0000","last_became_peered":"2024-04-21T08:42:16.636590+0000","last_unstale":"2024-04-21T08:44:41.375209+0000","last_undegraded":"2024-04-21T08:44:41.375209+0000","last_fullsized":"2024-04-21T08:44:41.375209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:55.693681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374884+0000","last_change":"2024-04-21T08:42:21.660404+0000","last_active":"2024-04-21T08:44:41.374884+0000","last_peered":"2024-04-21T08:44:41.374884+0000","last_clean":"2024-04-21T08:44:41.374884+0000","last_became_active":"2024-04-21T08:42:21.659999+0000","last_became_peered":"2024-04-21T08:42:21.659999+0000","last_unstale":"2024-04-21T08:44:41.374884+0000","last_undegraded":"2024-04-21T08:44:41.374884+0000","last_fullsized":"2024-04-21T08:44:41.374884+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:53:32.328873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356239+0000","last_change":"2024-04-21T08:43:10.984361+0000","last_active":"2024-04-21T08:44:41.356239+0000","last_peered":"2024-04-21T08:44:41.356239+0000","last_clean":"2024-04-21T08:44:41.356239+0000","last_became_active":"2024-04-21T08:43:10.984046+0000","last_became_peered":"2024-04-21T08:43:10.984046+0000","last_unstale":"2024-04-21T08:44:41.356239+0000","last_undegraded":"2024-04-21T08:44:41.356239+0000","last_fullsized":"2024-04-21T08:44:41.356239+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:33:23.154630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251236+0000","last_change":"2024-04-21T08:42:16.638993+0000","last_active":"2024-04-21T08:44:41.251236+0000","last_peered":"2024-04-21T08:44:41.251236+0000","last_clean":"2024-04-21T08:44:41.251236+0000","last_became_active":"2024-04-21T08:42:16.638617+0000","last_became_peered":"2024-04-21T08:42:16.638617+0000","last_unstale":"2024-04-21T08:44:41.251236+0000","last_undegraded":"2024-04-21T08:44:41.251236+0000","last_fullsized":"2024-04-21T08:44:41.251236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:14:28.271433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"95'148","reported_seq":410,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:43.370391+0000","last_change":"2024-04-21T08:41:00.044984+0000","last_active":"2024-04-21T08:44:43.370391+0000","last_peered":"2024-04-21T08:44:43.370391+0000","last_clean":"2024-04-21T08:44:43.370391+0000","last_became_active":"2024-04-21T08:41:00.044485+0000","last_became_peered":"2024-04-21T08:41:00.044485+0000","last_unstale":"2024-04-21T08:44:43.370391+0000","last_undegraded":"2024-04-21T08:44:43.370391+0000","last_fullsized":"2024-04-21T08:44:43.370391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:37:34.370031+0000","last_clean_scrub_stamp":"2024-04-21T08:37:34.370031+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:01.031209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365508+0000","last_change":"2024-04-21T08:42:21.971045+0000","last_active":"2024-04-21T08:44:41.365508+0000","last_peered":"2024-04-21T08:44:41.365508+0000","last_clean":"2024-04-21T08:44:41.365508+0000","last_became_active":"2024-04-21T08:42:21.970577+0000","last_became_peered":"2024-04-21T08:42:21.970577+0000","last_unstale":"2024-04-21T08:44:41.365508+0000","last_undegraded":"2024-04-21T08:44:41.365508+0000","last_fullsized":"2024-04-21T08:44:41.365508+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:55.681662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374964+0000","last_change":"2024-04-21T08:42:17.535570+0000","last_active":"2024-04-21T08:44:41.374964+0000","last_peered":"2024-04-21T08:44:41.374964+0000","last_clean":"2024-04-21T08:44:41.374964+0000","last_became_active":"2024-04-21T08:42:17.535217+0000","last_became_peered":"2024-04-21T08:42:17.535217+0000","last_unstale":"2024-04-21T08:44:41.374964+0000","last_undegraded":"2024-04-21T08:44:41.374964+0000","last_fullsized":"2024-04-21T08:44:41.374964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:49.510750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374650+0000","last_change":"2024-04-21T08:43:10.792548+0000","last_active":"2024-04-21T08:44:41.374650+0000","last_peered":"2024-04-21T08:44:41.374650+0000","last_clean":"2024-04-21T08:44:41.374650+0000","last_became_active":"2024-04-21T08:43:10.792165+0000","last_became_peered":"2024-04-21T08:43:10.792165+0000","last_unstale":"2024-04-21T08:44:41.374650+0000","last_undegraded":"2024-04-21T08:44:41.374650+0000","last_fullsized":"2024-04-21T08:44:41.374650+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:21:35.980536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374872+0000","last_change":"2024-04-21T08:42:21.661393+0000","last_active":"2024-04-21T08:44:41.374872+0000","last_peered":"2024-04-21T08:44:41.374872+0000","last_clean":"2024-04-21T08:44:41.374872+0000","last_became_active":"2024-04-21T08:42:21.660201+0000","last_became_peered":"2024-04-21T08:42:21.660201+0000","last_unstale":"2024-04-21T08:44:41.374872+0000","last_undegraded":"2024-04-21T08:44:41.374872+0000","last_fullsized":"2024-04-21T08:44:41.374872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:30:13.393389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250941+0000","last_change":"2024-04-21T08:42:16.650507+0000","last_active":"2024-04-21T08:44:41.250941+0000","last_peered":"2024-04-21T08:44:41.250941+0000","last_clean":"2024-04-21T08:44:41.250941+0000","last_became_active":"2024-04-21T08:42:16.650217+0000","last_became_peered":"2024-04-21T08:42:16.650217+0000","last_unstale":"2024-04-21T08:44:41.250941+0000","last_undegraded":"2024-04-21T08:44:41.250941+0000","last_fullsized":"2024-04-21T08:44:41.250941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:13:26.785092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375997+0000","last_change":"2024-04-21T08:42:17.530420+0000","last_active":"2024-04-21T08:44:41.375997+0000","last_peered":"2024-04-21T08:44:41.375997+0000","last_clean":"2024-04-21T08:44:41.375997+0000","last_became_active":"2024-04-21T08:42:17.529920+0000","last_became_peered":"2024-04-21T08:42:17.529920+0000","last_unstale":"2024-04-21T08:44:41.375997+0000","last_undegraded":"2024-04-21T08:44:41.375997+0000","last_fullsized":"2024-04-21T08:44:41.375997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:24.250197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249760+0000","last_change":"2024-04-21T08:42:21.970586+0000","last_active":"2024-04-21T08:44:41.249760+0000","last_peered":"2024-04-21T08:44:41.249760+0000","last_clean":"2024-04-21T08:44:41.249760+0000","last_became_active":"2024-04-21T08:42:21.970132+0000","last_became_peered":"2024-04-21T08:42:21.970132+0000","last_unstale":"2024-04-21T08:44:41.249760+0000","last_undegraded":"2024-04-21T08:44:41.249760+0000","last_fullsized":"2024-04-21T08:44:41.249760+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:11.074370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'68","reported_seq":155,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:47.175970+0000","last_change":"2024-04-21T08:42:16.647916+0000","last_active":"2024-04-21T08:44:47.175970+0000","last_peered":"2024-04-21T08:44:47.175970+0000","last_clean":"2024-04-21T08:44:47.175970+0000","last_became_active":"2024-04-21T08:42:16.647476+0000","last_became_peered":"2024-04-21T08:42:16.647476+0000","last_unstale":"2024-04-21T08:44:47.175970+0000","last_undegraded":"2024-04-21T08:44:47.175970+0000","last_fullsized":"2024-04-21T08:44:47.175970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:32:27.242153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":160,"num_write":122,"num_write_kb":1157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251070+0000","last_change":"2024-04-21T08:42:17.546144+0000","last_active":"2024-04-21T08:44:41.251070+0000","last_peered":"2024-04-21T08:44:41.251070+0000","last_clean":"2024-04-21T08:44:41.251070+0000","last_became_active":"2024-04-21T08:42:17.545730+0000","last_became_peered":"2024-04-21T08:42:17.545730+0000","last_unstale":"2024-04-21T08:44:41.251070+0000","last_undegraded":"2024-04-21T08:44:41.251070+0000","last_fullsized":"2024-04-21T08:44:41.251070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:42.802309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374618+0000","last_change":"2024-04-21T08:42:21.655802+0000","last_active":"2024-04-21T08:44:41.374618+0000","last_peered":"2024-04-21T08:44:41.374618+0000","last_clean":"2024-04-21T08:44:41.374618+0000","last_became_active":"2024-04-21T08:42:21.655379+0000","last_became_peered":"2024-04-21T08:42:21.655379+0000","last_unstale":"2024-04-21T08:44:41.374618+0000","last_undegraded":"2024-04-21T08:44:41.374618+0000","last_fullsized":"2024-04-21T08:44:41.374618+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:49.486057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251462+0000","last_change":"2024-04-21T08:42:16.642517+0000","last_active":"2024-04-21T08:44:41.251462+0000","last_peered":"2024-04-21T08:44:41.251462+0000","last_clean":"2024-04-21T08:44:41.251462+0000","last_became_active":"2024-04-21T08:42:16.641539+0000","last_became_peered":"2024-04-21T08:42:16.641539+0000","last_unstale":"2024-04-21T08:44:41.251462+0000","last_undegraded":"2024-04-21T08:44:41.251462+0000","last_fullsized":"2024-04-21T08:44:41.251462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:53.912199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.418089+0000","last_change":"2024-04-21T08:42:17.550172+0000","last_active":"2024-04-21T08:44:41.418089+0000","last_peered":"2024-04-21T08:44:41.418089+0000","last_clean":"2024-04-21T08:44:41.418089+0000","last_became_active":"2024-04-21T08:42:17.549824+0000","last_became_peered":"2024-04-21T08:42:17.549824+0000","last_unstale":"2024-04-21T08:44:41.418089+0000","last_undegraded":"2024-04-21T08:44:41.418089+0000","last_fullsized":"2024-04-21T08:44:41.418089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:59.553645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251154+0000","last_change":"2024-04-21T08:44:11.774778+0000","last_active":"2024-04-21T08:44:41.251154+0000","last_peered":"2024-04-21T08:44:41.251154+0000","last_clean":"2024-04-21T08:44:41.251154+0000","last_became_active":"2024-04-21T08:44:11.774190+0000","last_became_peered":"2024-04-21T08:44:11.774190+0000","last_unstale":"2024-04-21T08:44:41.251154+0000","last_undegraded":"2024-04-21T08:44:41.251154+0000","last_fullsized":"2024-04-21T08:44:41.251154+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:06.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374515+0000","last_change":"2024-04-21T08:42:16.647985+0000","last_active":"2024-04-21T08:44:41.374515+0000","last_peered":"2024-04-21T08:44:41.374515+0000","last_clean":"2024-04-21T08:44:41.374515+0000","last_became_active":"2024-04-21T08:42:16.647577+0000","last_became_peered":"2024-04-21T08:42:16.647577+0000","last_unstale":"2024-04-21T08:44:41.374515+0000","last_undegraded":"2024-04-21T08:44:41.374515+0000","last_fullsized":"2024-04-21T08:44:41.374515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:00.731708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250576+0000","last_change":"2024-04-21T08:42:17.541757+0000","last_active":"2024-04-21T08:44:41.250576+0000","last_peered":"2024-04-21T08:44:41.250576+0000","last_clean":"2024-04-21T08:44:41.250576+0000","last_became_active":"2024-04-21T08:42:17.539280+0000","last_became_peered":"2024-04-21T08:42:17.539280+0000","last_unstale":"2024-04-21T08:44:41.250576+0000","last_undegraded":"2024-04-21T08:44:41.250576+0000","last_fullsized":"2024-04-21T08:44:41.250576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:45:54.628287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356961+0000","last_change":"2024-04-21T08:42:21.664566+0000","last_active":"2024-04-21T08:44:41.356961+0000","last_peered":"2024-04-21T08:44:41.356961+0000","last_clean":"2024-04-21T08:44:41.356961+0000","last_became_active":"2024-04-21T08:42:21.664136+0000","last_became_peered":"2024-04-21T08:42:21.664136+0000","last_unstale":"2024-04-21T08:44:41.356961+0000","last_undegraded":"2024-04-21T08:44:41.356961+0000","last_fullsized":"2024-04-21T08:44:41.356961+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:19.856577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.249918+0000","last_change":"2024-04-21T08:42:16.650664+0000","last_active":"2024-04-21T08:44:41.249918+0000","last_peered":"2024-04-21T08:44:41.249918+0000","last_clean":"2024-04-21T08:44:41.249918+0000","last_became_active":"2024-04-21T08:42:16.650339+0000","last_became_peered":"2024-04-21T08:42:16.650339+0000","last_unstale":"2024-04-21T08:44:41.249918+0000","last_undegraded":"2024-04-21T08:44:41.249918+0000","last_fullsized":"2024-04-21T08:44:41.249918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:34:59.565455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374844+0000","last_change":"2024-04-21T08:42:17.538142+0000","last_active":"2024-04-21T08:44:41.374844+0000","last_peered":"2024-04-21T08:44:41.374844+0000","last_clean":"2024-04-21T08:44:41.374844+0000","last_became_active":"2024-04-21T08:42:17.537793+0000","last_became_peered":"2024-04-21T08:42:17.537793+0000","last_unstale":"2024-04-21T08:44:41.374844+0000","last_undegraded":"2024-04-21T08:44:41.374844+0000","last_fullsized":"2024-04-21T08:44:41.374844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:11.059294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251170+0000","last_change":"2024-04-21T08:42:21.670642+0000","last_active":"2024-04-21T08:44:41.251170+0000","last_peered":"2024-04-21T08:44:41.251170+0000","last_clean":"2024-04-21T08:44:41.251170+0000","last_became_active":"2024-04-21T08:42:21.670128+0000","last_became_peered":"2024-04-21T08:42:21.670128+0000","last_unstale":"2024-04-21T08:44:41.251170+0000","last_undegraded":"2024-04-21T08:44:41.251170+0000","last_fullsized":"2024-04-21T08:44:41.251170+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:51:28.604381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375278+0000","last_change":"2024-04-21T08:43:10.792114+0000","last_active":"2024-04-21T08:44:41.375278+0000","last_peered":"2024-04-21T08:44:41.375278+0000","last_clean":"2024-04-21T08:44:41.375278+0000","last_became_active":"2024-04-21T08:43:10.791664+0000","last_became_peered":"2024-04-21T08:43:10.791664+0000","last_unstale":"2024-04-21T08:44:41.375278+0000","last_undegraded":"2024-04-21T08:44:41.375278+0000","last_fullsized":"2024-04-21T08:44:41.375278+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:07.317804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250083+0000","last_change":"2024-04-21T08:42:17.547326+0000","last_active":"2024-04-21T08:44:41.250083+0000","last_peered":"2024-04-21T08:44:41.250083+0000","last_clean":"2024-04-21T08:44:41.250083+0000","last_became_active":"2024-04-21T08:42:17.546920+0000","last_became_peered":"2024-04-21T08:42:17.546920+0000","last_unstale":"2024-04-21T08:44:41.250083+0000","last_undegraded":"2024-04-21T08:44:41.250083+0000","last_fullsized":"2024-04-21T08:44:41.250083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:29:43.350877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.251167+0000","last_change":"2024-04-21T08:42:21.971147+0000","last_active":"2024-04-21T08:44:41.251167+0000","last_peered":"2024-04-21T08:44:41.251167+0000","last_clean":"2024-04-21T08:44:41.251167+0000","last_became_active":"2024-04-21T08:42:21.970001+0000","last_became_peered":"2024-04-21T08:42:21.970001+0000","last_unstale":"2024-04-21T08:44:41.251167+0000","last_undegraded":"2024-04-21T08:44:41.251167+0000","last_fullsized":"2024-04-21T08:44:41.251167+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:01:44.565494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250876+0000","last_change":"2024-04-21T08:42:16.643371+0000","last_active":"2024-04-21T08:44:41.250876+0000","last_peered":"2024-04-21T08:44:41.250876+0000","last_clean":"2024-04-21T08:44:41.250876+0000","last_became_active":"2024-04-21T08:42:16.642906+0000","last_became_peered":"2024-04-21T08:42:16.642906+0000","last_unstale":"2024-04-21T08:44:41.250876+0000","last_undegraded":"2024-04-21T08:44:41.250876+0000","last_fullsized":"2024-04-21T08:44:41.250876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:00:58.015457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375124+0000","last_change":"2024-04-21T08:42:17.545185+0000","last_active":"2024-04-21T08:44:41.375124+0000","last_peered":"2024-04-21T08:44:41.375124+0000","last_clean":"2024-04-21T08:44:41.375124+0000","last_became_active":"2024-04-21T08:42:17.544757+0000","last_became_peered":"2024-04-21T08:42:17.544757+0000","last_unstale":"2024-04-21T08:44:41.375124+0000","last_undegraded":"2024-04-21T08:44:41.375124+0000","last_fullsized":"2024-04-21T08:44:41.375124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:36:18.968041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374923+0000","last_change":"2024-04-21T08:42:21.662541+0000","last_active":"2024-04-21T08:44:41.374923+0000","last_peered":"2024-04-21T08:44:41.374923+0000","last_clean":"2024-04-21T08:44:41.374923+0000","last_became_active":"2024-04-21T08:42:21.662144+0000","last_became_peered":"2024-04-21T08:42:21.662144+0000","last_unstale":"2024-04-21T08:44:41.374923+0000","last_undegraded":"2024-04-21T08:44:41.374923+0000","last_fullsized":"2024-04-21T08:44:41.374923+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:16:04.347722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375374+0000","last_change":"2024-04-21T08:42:16.638601+0000","last_active":"2024-04-21T08:44:41.375374+0000","last_peered":"2024-04-21T08:44:41.375374+0000","last_clean":"2024-04-21T08:44:41.375374+0000","last_became_active":"2024-04-21T08:42:16.638211+0000","last_became_peered":"2024-04-21T08:42:16.638211+0000","last_unstale":"2024-04-21T08:44:41.375374+0000","last_undegraded":"2024-04-21T08:44:41.375374+0000","last_fullsized":"2024-04-21T08:44:41.375374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:02.633086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375257+0000","last_change":"2024-04-21T08:42:17.545495+0000","last_active":"2024-04-21T08:44:41.375257+0000","last_peered":"2024-04-21T08:44:41.375257+0000","last_clean":"2024-04-21T08:44:41.375257+0000","last_became_active":"2024-04-21T08:42:17.545159+0000","last_became_peered":"2024-04-21T08:42:17.545159+0000","last_unstale":"2024-04-21T08:44:41.375257+0000","last_undegraded":"2024-04-21T08:44:41.375257+0000","last_fullsized":"2024-04-21T08:44:41.375257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:27.150277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417564+0000","last_change":"2024-04-21T08:42:21.656890+0000","last_active":"2024-04-21T08:44:41.417564+0000","last_peered":"2024-04-21T08:44:41.417564+0000","last_clean":"2024-04-21T08:44:41.417564+0000","last_became_active":"2024-04-21T08:42:21.656475+0000","last_became_peered":"2024-04-21T08:42:21.656475+0000","last_unstale":"2024-04-21T08:44:41.417564+0000","last_undegraded":"2024-04-21T08:44:41.417564+0000","last_fullsized":"2024-04-21T08:44:41.417564+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:22:46.138591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'57","reported_seq":109,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:47.171231+0000","last_change":"2024-04-21T08:42:16.638244+0000","last_active":"2024-04-21T08:44:47.171231+0000","last_peered":"2024-04-21T08:44:47.171231+0000","last_clean":"2024-04-21T08:44:47.171231+0000","last_became_active":"2024-04-21T08:42:16.637883+0000","last_became_peered":"2024-04-21T08:42:16.637883+0000","last_unstale":"2024-04-21T08:44:47.171231+0000","last_undegraded":"2024-04-21T08:44:47.171231+0000","last_fullsized":"2024-04-21T08:44:47.171231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:20.888301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":68,"num_write_kb":807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374936+0000","last_change":"2024-04-21T08:42:17.548032+0000","last_active":"2024-04-21T08:44:41.374936+0000","last_peered":"2024-04-21T08:44:41.374936+0000","last_clean":"2024-04-21T08:44:41.374936+0000","last_became_active":"2024-04-21T08:42:17.547660+0000","last_became_peered":"2024-04-21T08:42:17.547660+0000","last_unstale":"2024-04-21T08:44:41.374936+0000","last_undegraded":"2024-04-21T08:44:41.374936+0000","last_fullsized":"2024-04-21T08:44:41.374936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:02:14.710759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365005+0000","last_change":"2024-04-21T08:42:21.659286+0000","last_active":"2024-04-21T08:44:41.365005+0000","last_peered":"2024-04-21T08:44:41.365005+0000","last_clean":"2024-04-21T08:44:41.365005+0000","last_became_active":"2024-04-21T08:42:21.658844+0000","last_became_peered":"2024-04-21T08:42:21.658844+0000","last_unstale":"2024-04-21T08:44:41.365005+0000","last_undegraded":"2024-04-21T08:44:41.365005+0000","last_fullsized":"2024-04-21T08:44:41.365005+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:20.637386+0000","last_clean_scrub_stamp":"2024-04-21T08:42:20.637386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:28.742646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376281+0000","last_change":"2024-04-21T08:42:16.643354+0000","last_active":"2024-04-21T08:44:41.376281+0000","last_peered":"2024-04-21T08:44:41.376281+0000","last_clean":"2024-04-21T08:44:41.376281+0000","last_became_active":"2024-04-21T08:42:16.642967+0000","last_became_peered":"2024-04-21T08:42:16.642967+0000","last_unstale":"2024-04-21T08:44:41.376281+0000","last_undegraded":"2024-04-21T08:44:41.376281+0000","last_fullsized":"2024-04-21T08:44:41.376281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:15.612591+0000","last_clean_scrub_stamp":"2024-04-21T08:42:15.612591+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:06:03.495181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375409+0000","last_change":"2024-04-21T08:42:17.539551+0000","last_active":"2024-04-21T08:44:41.375409+0000","last_peered":"2024-04-21T08:44:41.375409+0000","last_clean":"2024-04-21T08:44:41.375409+0000","last_became_active":"2024-04-21T08:42:17.539202+0000","last_became_peered":"2024-04-21T08:42:17.539202+0000","last_unstale":"2024-04-21T08:44:41.375409+0000","last_undegraded":"2024-04-21T08:44:41.375409+0000","last_fullsized":"2024-04-21T08:44:41.375409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:37.109421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375023+0000","last_change":"2024-04-21T08:42:17.544582+0000","last_active":"2024-04-21T08:44:41.375023+0000","last_peered":"2024-04-21T08:44:41.375023+0000","last_clean":"2024-04-21T08:44:41.375023+0000","last_became_active":"2024-04-21T08:42:17.543800+0000","last_became_peered":"2024-04-21T08:42:17.543800+0000","last_unstale":"2024-04-21T08:44:41.375023+0000","last_undegraded":"2024-04-21T08:44:41.375023+0000","last_fullsized":"2024-04-21T08:44:41.375023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:44:04.107207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374756+0000","last_change":"2024-04-21T08:42:17.543788+0000","last_active":"2024-04-21T08:44:41.374756+0000","last_peered":"2024-04-21T08:44:41.374756+0000","last_clean":"2024-04-21T08:44:41.374756+0000","last_became_active":"2024-04-21T08:42:17.540939+0000","last_became_peered":"2024-04-21T08:42:17.540939+0000","last_unstale":"2024-04-21T08:44:41.374756+0000","last_undegraded":"2024-04-21T08:44:41.374756+0000","last_fullsized":"2024-04-21T08:44:41.374756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:59.275098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.374942+0000","last_change":"2024-04-21T08:42:17.540605+0000","last_active":"2024-04-21T08:44:41.374942+0000","last_peered":"2024-04-21T08:44:41.374942+0000","last_clean":"2024-04-21T08:44:41.374942+0000","last_became_active":"2024-04-21T08:42:17.540073+0000","last_became_peered":"2024-04-21T08:42:17.540073+0000","last_unstale":"2024-04-21T08:44:41.374942+0000","last_undegraded":"2024-04-21T08:44:41.374942+0000","last_fullsized":"2024-04-21T08:44:41.374942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:02:29.345205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.417931+0000","last_change":"2024-04-21T08:42:17.551608+0000","last_active":"2024-04-21T08:44:41.417931+0000","last_peered":"2024-04-21T08:44:41.417931+0000","last_clean":"2024-04-21T08:44:41.417931+0000","last_became_active":"2024-04-21T08:42:17.551117+0000","last_became_peered":"2024-04-21T08:42:17.551117+0000","last_unstale":"2024-04-21T08:44:41.417931+0000","last_undegraded":"2024-04-21T08:44:41.417931+0000","last_fullsized":"2024-04-21T08:44:41.417931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:18.391928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375875+0000","last_change":"2024-04-21T08:42:17.536509+0000","last_active":"2024-04-21T08:44:41.375875+0000","last_peered":"2024-04-21T08:44:41.375875+0000","last_clean":"2024-04-21T08:44:41.375875+0000","last_became_active":"2024-04-21T08:42:17.536146+0000","last_became_peered":"2024-04-21T08:42:17.536146+0000","last_unstale":"2024-04-21T08:44:41.375875+0000","last_undegraded":"2024-04-21T08:44:41.375875+0000","last_fullsized":"2024-04-21T08:44:41.375875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:29.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375199+0000","last_change":"2024-04-21T08:42:17.543786+0000","last_active":"2024-04-21T08:44:41.375199+0000","last_peered":"2024-04-21T08:44:41.375199+0000","last_clean":"2024-04-21T08:44:41.375199+0000","last_became_active":"2024-04-21T08:42:17.543435+0000","last_became_peered":"2024-04-21T08:42:17.543435+0000","last_unstale":"2024-04-21T08:44:41.375199+0000","last_undegraded":"2024-04-21T08:44:41.375199+0000","last_fullsized":"2024-04-21T08:44:41.375199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375616+0000","last_change":"2024-04-21T08:42:17.541611+0000","last_active":"2024-04-21T08:44:41.375616+0000","last_peered":"2024-04-21T08:44:41.375616+0000","last_clean":"2024-04-21T08:44:41.375616+0000","last_became_active":"2024-04-21T08:42:17.541259+0000","last_became_peered":"2024-04-21T08:42:17.541259+0000","last_unstale":"2024-04-21T08:44:41.375616+0000","last_undegraded":"2024-04-21T08:44:41.375616+0000","last_fullsized":"2024-04-21T08:44:41.375616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:49:20.900629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.356654+0000","last_change":"2024-04-21T08:44:11.774062+0000","last_active":"2024-04-21T08:44:41.356654+0000","last_peered":"2024-04-21T08:44:41.356654+0000","last_clean":"2024-04-21T08:44:41.356654+0000","last_became_active":"2024-04-21T08:44:11.773703+0000","last_became_peered":"2024-04-21T08:44:11.773703+0000","last_unstale":"2024-04-21T08:44:41.356654+0000","last_undegraded":"2024-04-21T08:44:41.356654+0000","last_fullsized":"2024-04-21T08:44:41.356654+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:45:10.739068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.365197+0000","last_change":"2024-04-21T08:42:17.545526+0000","last_active":"2024-04-21T08:44:41.365197+0000","last_peered":"2024-04-21T08:44:41.365197+0000","last_clean":"2024-04-21T08:44:41.365197+0000","last_became_active":"2024-04-21T08:42:17.545135+0000","last_became_peered":"2024-04-21T08:42:17.545135+0000","last_unstale":"2024-04-21T08:44:41.365197+0000","last_undegraded":"2024-04-21T08:44:41.365197+0000","last_fullsized":"2024-04-21T08:44:41.365197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:37:58.038459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.250029+0000","last_change":"2024-04-21T08:42:17.547063+0000","last_active":"2024-04-21T08:44:41.250029+0000","last_peered":"2024-04-21T08:44:41.250029+0000","last_clean":"2024-04-21T08:44:41.250029+0000","last_became_active":"2024-04-21T08:42:17.546689+0000","last_became_peered":"2024-04-21T08:42:17.546689+0000","last_unstale":"2024-04-21T08:44:41.250029+0000","last_undegraded":"2024-04-21T08:44:41.250029+0000","last_fullsized":"2024-04-21T08:44:41.250029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:17:26.236595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375261+0000","last_change":"2024-04-21T08:42:17.542545+0000","last_active":"2024-04-21T08:44:41.375261+0000","last_peered":"2024-04-21T08:44:41.375261+0000","last_clean":"2024-04-21T08:44:41.375261+0000","last_became_active":"2024-04-21T08:42:17.542147+0000","last_became_peered":"2024-04-21T08:42:17.542147+0000","last_unstale":"2024-04-21T08:44:41.375261+0000","last_undegraded":"2024-04-21T08:44:41.375261+0000","last_fullsized":"2024-04-21T08:44:41.375261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:18:15.064110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.376582+0000","last_change":"2024-04-21T08:42:17.541262+0000","last_active":"2024-04-21T08:44:41.376582+0000","last_peered":"2024-04-21T08:44:41.376582+0000","last_clean":"2024-04-21T08:44:41.376582+0000","last_became_active":"2024-04-21T08:42:17.540817+0000","last_became_peered":"2024-04-21T08:42:17.540817+0000","last_unstale":"2024-04-21T08:44:41.376582+0000","last_undegraded":"2024-04-21T08:44:41.376582+0000","last_fullsized":"2024-04-21T08:44:41.376582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:22:59.597680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T08:44:41.375116+0000","last_change":"2024-04-21T08:42:17.551109+0000","last_active":"2024-04-21T08:44:41.375116+0000","last_peered":"2024-04-21T08:44:41.375116+0000","last_clean":"2024-04-21T08:44:41.375116+0000","last_became_active":"2024-04-21T08:42:17.550104+0000","last_became_peered":"2024-04-21T08:42:17.550104+0000","last_unstale":"2024-04-21T08:44:41.375116+0000","last_undegraded":"2024-04-21T08:44:41.375116+0000","last_fullsized":"2024-04-21T08:44:41.375116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:16.615054+0000","last_clean_scrub_stamp":"2024-04-21T08:42:16.615054+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:22:19.361752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107298,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":792314,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":472,"num_read_kb":387,"num_write":265,"num_write_kb":2063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":2376942,"data_compressed":107241,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351019,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":532412,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36500,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708652,"statfs":{"total":95995035648,"available":95957659648,"internally_reserved":0,"allocated":1187840,"data_stored":1320546,"data_compressed":73280,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1069056,"data_stored":930306,"data_compressed":48872,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037819,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":926232,"data_compressed":49001,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299455,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":1127458,"data_compressed":65492,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462982,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528316,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1105920,"data_stored":598384,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1036288,"data_stored":528406,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":598418,"data_compressed":31358,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1053236,"data_compressed":55028,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888144,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":532434,"data_compressed":29726,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724618,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":1254562,"data_compressed":71648,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35766,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":70102,"data_compressed":1632,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":721552,"data_compressed":34115,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":328414,"data_compressed":14840,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":398494,"data_compressed":16472,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:44:52.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:52.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.746+0000 7f3ce9201640 1 -- 172.21.15.7:0/2553615485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce4101860 msgr2=0x7f3ce41058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.746+0000 7f3ce9201640 1 --2- 172.21.15.7:0/2553615485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce4101860 0x7f3ce41058a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009900 tx=0x7f3cd802bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:52.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.746+0000 7f3ce9201640 1 -- 172.21.15.7:0/2553615485 shutdown_connections 2024-04-21T08:44:52.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.746+0000 7f3ce9201640 1 --2- 172.21.15.7:0/2553615485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ce4105f80 0x7f3ce410db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.746+0000 7f3ce9201640 1 --2- 172.21.15.7:0/2553615485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce4101860 0x7f3ce41058a0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.746+0000 7f3ce9201640 1 --2- 172.21.15.7:0/2553615485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce4100eb0 0x7f3ce4101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.746+0000 7f3ce9201640 1 -- 172.21.15.7:0/2553615485 >> 172.21.15.7:0/2553615485 conn(0x7f3ce40fc730 msgr2=0x7f3ce40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.746+0000 7f3ce9201640 1 -- 172.21.15.7:0/2553615485 shutdown_connections 2024-04-21T08:44:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.746+0000 7f3ce9201640 1 -- 172.21.15.7:0/2553615485 wait complete. 2024-04-21T08:44:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.747+0000 7f3ce9201640 1 Processor -- start 2024-04-21T08:44:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.747+0000 7f3ce9201640 1 -- start start 2024-04-21T08:44:52.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.747+0000 7f3ce9201640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ce4100eb0 0x7f3ce4198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.747+0000 7f3ce9201640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce4105f80 0x7f3ce4198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.747+0000 7f3ce9201640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce419ccc0 0x7f3ce419d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.747+0000 7f3ce9201640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce4112000 con 0x7f3ce419ccc0 2024-04-21T08:44:52.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.747+0000 7f3ce9201640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce4111e80 con 0x7f3ce4105f80 2024-04-21T08:44:52.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.747+0000 7f3ce9201640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce4112180 con 0x7f3ce4100eb0 2024-04-21T08:44:52.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.747+0000 7f3ce359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce419ccc0 0x7f3ce419d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce419ccc0 0x7f3ce419d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47620/0 (socket says 172.21.15.7:47620) 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce359e640 1 -- 172.21.15.7:0/1609749056 learned_addr learned my addr 172.21.15.7:0/1609749056 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce2d9d640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ce4100eb0 0x7f3ce4198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce259c640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce4105f80 0x7f3ce4198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce359e640 1 -- 172.21.15.7:0/1609749056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ce4100eb0 msgr2=0x7f3ce4198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce359e640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ce4100eb0 0x7f3ce4198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce359e640 1 -- 172.21.15.7:0/1609749056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce4105f80 msgr2=0x7f3ce4198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce359e640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce4105f80 0x7f3ce4198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce359e640 1 -- 172.21.15.7:0/1609749056 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce419d7b0 con 0x7f3ce419ccc0 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce359e640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce419ccc0 0x7f3ce419d170 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00cce0 tx=0x7f3cdc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ccbfff640 1 -- 172.21.15.7:0/1609749056 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc00f450 con 0x7f3ce419ccc0 2024-04-21T08:44:52.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ccbfff640 1 -- 172.21.15.7:0/1609749056 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cdc00faa0 con 0x7f3ce419ccc0 2024-04-21T08:44:52.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce9201640 1 -- 172.21.15.7:0/1609749056 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce419daa0 con 0x7f3ce419ccc0 2024-04-21T08:44:52.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ccbfff640 1 -- 172.21.15.7:0/1609749056 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc018b70 con 0x7f3ce419ccc0 2024-04-21T08:44:52.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.748+0000 7f3ce9201640 1 -- 172.21.15.7:0/1609749056 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce41a5210 con 0x7f3ce419ccc0 2024-04-21T08:44:52.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.749+0000 7f3ce9201640 1 -- 172.21.15.7:0/1609749056 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce404fce0 con 0x7f3ce419ccc0 2024-04-21T08:44:52.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.751+0000 7f3ccbfff640 1 -- 172.21.15.7:0/1609749056 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f3cdc018d10 con 0x7f3ce419ccc0 2024-04-21T08:44:52.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.751+0000 7f3ccbfff640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ccc0a3e60 0x7f3ccc0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.751+0000 7f3ccbfff640 1 -- 172.21.15.7:0/1609749056 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f3cdc0d11a0 con 0x7f3ce419ccc0 2024-04-21T08:44:52.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.751+0000 7f3ce2d9d640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ccc0a3e60 0x7f3ccc0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.752+0000 7f3ce2d9d640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ccc0a3e60 0x7f3ccc0a6320 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400bbe0 tx=0x7f3cd400b390 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:52.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.756+0000 7f3ccbfff640 1 -- 172.21.15.7:0/1609749056 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3cdc014030 con 0x7f3ce419ccc0 2024-04-21T08:44:52.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.981+0000 7f3ce9201640 1 -- 172.21.15.7:0/1609749056 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ce4101860 con 0x7f3ce419ccc0 2024-04-21T08:44:52.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.982+0000 7f3ccbfff640 1 -- 172.21.15.7:0/1609749056 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7f3cdc09d2e0 con 0x7f3ce419ccc0 2024-04-21T08:44:52.982 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:52.982 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:44:52.982 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:44:52.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 -- 172.21.15.7:0/1609749056 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ccc0a3e60 msgr2=0x7f3ccc0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ccc0a3e60 0x7f3ccc0a6320 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400bbe0 tx=0x7f3cd400b390 comp rx=0 tx=0).stop 2024-04-21T08:44:52.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 -- 172.21.15.7:0/1609749056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce419ccc0 msgr2=0x7f3ce419d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce419ccc0 0x7f3ce419d170 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00cce0 tx=0x7f3cdc00a620 comp rx=0 tx=0).stop 2024-04-21T08:44:52.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 -- 172.21.15.7:0/1609749056 shutdown_connections 2024-04-21T08:44:52.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce419ccc0 0x7f3ce419d170 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce4105f80 0x7f3ce4198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ccc0a3e60 0x7f3ccc0a6320 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 --2- 172.21.15.7:0/1609749056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ce4100eb0 0x7f3ce4198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 -- 172.21.15.7:0/1609749056 >> 172.21.15.7:0/1609749056 conn(0x7f3ce40fc730 msgr2=0x7f3ce40fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:52.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 -- 172.21.15.7:0/1609749056 shutdown_connections 2024-04-21T08:44:52.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:52.984+0000 7f3ce9201640 1 -- 172.21.15.7:0/1609749056 wait complete. 2024-04-21T08:44:52.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:53.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.098+0000 7fc7009d5640 1 -- 172.21.15.7:0/862638516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6fc101a20 msgr2=0x7fc6fc105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.098+0000 7fc7009d5640 1 --2- 172.21.15.7:0/862638516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6fc101a20 0x7fc6fc105a60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0009900 tx=0x7fc6f002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.098+0000 7fc7009d5640 1 -- 172.21.15.7:0/862638516 shutdown_connections 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.098+0000 7fc7009d5640 1 --2- 172.21.15.7:0/862638516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6fc106140 0x7fc6fc10dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.098+0000 7fc7009d5640 1 --2- 172.21.15.7:0/862638516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6fc101a20 0x7fc6fc105a60 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.098+0000 7fc7009d5640 1 --2- 172.21.15.7:0/862638516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6fc101070 0x7fc6fc101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.098+0000 7fc7009d5640 1 -- 172.21.15.7:0/862638516 >> 172.21.15.7:0/862638516 conn(0x7fc6fc0771d0 msgr2=0x7fc6fc0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.098+0000 7fc7009d5640 1 -- 172.21.15.7:0/862638516 shutdown_connections 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.098+0000 7fc7009d5640 1 -- 172.21.15.7:0/862638516 wait complete. 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.098+0000 7fc7009d5640 1 Processor -- start 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc7009d5640 1 -- start start 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc7009d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6fc101070 0x7fc6fc198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc7009d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6fc106140 0x7fc6fc198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc7009d5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6fc19cf20 0x7fc6fc19d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc7009d5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc6fc112180 con 0x7fc6fc19cf20 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc7009d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc6fc112000 con 0x7fc6fc106140 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc7009d5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc6fc112300 con 0x7fc6fc101070 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6fad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6fc19cf20 0x7fc6fc19d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6fad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6fc19cf20 0x7fc6fc19d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47638/0 (socket says 172.21.15.7:47638) 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6fad9d640 1 -- 172.21.15.7:0/663174487 learned_addr learned my addr 172.21.15.7:0/663174487 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6fa59c640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6fc101070 0x7fc6fc198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6f9d9b640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6fc106140 0x7fc6fc198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6fad9d640 1 -- 172.21.15.7:0/663174487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6fc101070 msgr2=0x7fc6fc198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6fad9d640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6fc101070 0x7fc6fc198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6fad9d640 1 -- 172.21.15.7:0/663174487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6fc106140 msgr2=0x7fc6fc198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6fad9d640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6fc106140 0x7fc6fc198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6fad9d640 1 -- 172.21.15.7:0/663174487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6fc19da10 con 0x7fc6fc19cf20 2024-04-21T08:44:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6fad9d640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6fc19cf20 0x7fc6fc19d3d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00eaa0 tx=0x7fc6ec00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:53.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6eb7fe640 1 -- 172.21.15.7:0/663174487 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec004d20 con 0x7fc6fc19cf20 2024-04-21T08:44:53.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.099+0000 7fc6eb7fe640 1 -- 172.21.15.7:0/663174487 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6ec005020 con 0x7fc6fc19cf20 2024-04-21T08:44:53.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.100+0000 7fc7009d5640 1 -- 172.21.15.7:0/663174487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6fc19dd00 con 0x7fc6fc19cf20 2024-04-21T08:44:53.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.100+0000 7fc6eb7fe640 1 -- 172.21.15.7:0/663174487 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec0116b0 con 0x7fc6fc19cf20 2024-04-21T08:44:53.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.100+0000 7fc7009d5640 1 -- 172.21.15.7:0/663174487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6fc1a5470 con 0x7fc6fc19cf20 2024-04-21T08:44:53.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.101+0000 7fc7009d5640 1 -- 172.21.15.7:0/663174487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6fc068e00 con 0x7fc6fc19cf20 2024-04-21T08:44:53.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.102+0000 7fc6eb7fe640 1 -- 172.21.15.7:0/663174487 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7fc6ec0051d0 con 0x7fc6fc19cf20 2024-04-21T08:44:53.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.102+0000 7fc6eb7fe640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc6e40a3db0 0x7fc6e40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.102+0000 7fc6fa59c640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc6e40a3db0 0x7fc6e40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.103+0000 7fc6eb7fe640 1 -- 172.21.15.7:0/663174487 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fc6ec0d0fa0 con 0x7fc6fc19cf20 2024-04-21T08:44:53.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.103+0000 7fc6fa59c640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc6e40a3db0 0x7fc6e40a6270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400abe0 tx=0x7fc6f400a2d0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.107+0000 7fc6eb7fe640 1 -- 172.21.15.7:0/663174487 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc6ec016030 con 0x7fc6fc19cf20 2024-04-21T08:44:53.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:52 smithi007 ceph-mon[28477]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 73 KiB/s wr, 34 op/s 2024-04-21T08:44:53.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:52 smithi007 ceph-mon[28477]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:53.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.293+0000 7fc7009d5640 1 -- 172.21.15.7:0/663174487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6fc1093d0 con 0x7fc6fc19cf20 2024-04-21T08:44:53.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.294+0000 7fc6eb7fe640 1 -- 172.21.15.7:0/663174487 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25085 (secure 0 0 0) 0x7fc6ec09d0e0 con 0x7fc6fc19cf20 2024-04-21T08:44:53.294 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:53.294 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":95,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:44:41.244735+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3451243741":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/117789107":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/4167672076":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/3630897301":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 -- 172.21.15.7:0/663174487 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc6e40a3db0 msgr2=0x7fc6e40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc6e40a3db0 0x7fc6e40a6270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400abe0 tx=0x7fc6f400a2d0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 -- 172.21.15.7:0/663174487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6fc19cf20 msgr2=0x7fc6fc19d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6fc19cf20 0x7fc6fc19d3d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00eaa0 tx=0x7fc6ec00eea0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 -- 172.21.15.7:0/663174487 shutdown_connections 2024-04-21T08:44:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6fc19cf20 0x7fc6fc19d3d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6fc106140 0x7fc6fc198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc6e40a3db0 0x7fc6e40a6270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 --2- 172.21.15.7:0/663174487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6fc101070 0x7fc6fc198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 -- 172.21.15.7:0/663174487 >> 172.21.15.7:0/663174487 conn(0x7fc6fc0771d0 msgr2=0x7fc6fc107310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:53.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 -- 172.21.15.7:0/663174487 shutdown_connections 2024-04-21T08:44:53.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.296+0000 7fc7009d5640 1 -- 172.21.15.7:0/663174487 wait complete. 2024-04-21T08:44:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:52 smithi016 ceph-mon[31259]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 73 KiB/s wr, 34 op/s 2024-04-21T08:44:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:52 smithi016 ceph-mon[31259]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:53.307 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:53.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:53.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:52 smithi077 ceph-mon[31615]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 73 KiB/s wr, 34 op/s 2024-04-21T08:44:53.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:52 smithi077 ceph-mon[31615]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.413+0000 7f61a75f4640 1 -- 172.21.15.7:0/2626507845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a0106160 msgr2=0x7f61a010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.413+0000 7f61a75f4640 1 --2- 172.21.15.7:0/2626507845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a0106160 0x7f61a010dd30 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6198009ca0 tx=0x7f619802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.413+0000 7f61a75f4640 1 -- 172.21.15.7:0/2626507845 shutdown_connections 2024-04-21T08:44:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.413+0000 7f61a75f4640 1 --2- 172.21.15.7:0/2626507845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a0106160 0x7f61a010dd30 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.413+0000 7f61a75f4640 1 --2- 172.21.15.7:0/2626507845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a0101a40 0x7f61a0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.413+0000 7f61a75f4640 1 --2- 172.21.15.7:0/2626507845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a0101090 0x7f61a0101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.413+0000 7f61a75f4640 1 -- 172.21.15.7:0/2626507845 >> 172.21.15.7:0/2626507845 conn(0x7f61a0078000 msgr2=0x7f61a0078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.413+0000 7f61a75f4640 1 -- 172.21.15.7:0/2626507845 shutdown_connections 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.413+0000 7f61a75f4640 1 -- 172.21.15.7:0/2626507845 wait complete. 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.413+0000 7f61a75f4640 1 Processor -- start 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a75f4640 1 -- start start 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a75f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a0101090 0x7f61a01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a75f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a0106160 0x7f61a0198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a75f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a019cfb0 0x7f61a019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a75f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f61a01110b0 con 0x7f61a0101090 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a75f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61a0110f30 con 0x7f61a0106160 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a75f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f61a0111230 con 0x7f61a019cfb0 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a5390640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a0101090 0x7f61a01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a5390640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a0101090 0x7f61a01986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47642/0 (socket says 172.21.15.7:47642) 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a5390640 1 -- 172.21.15.7:0/2485387730 learned_addr learned my addr 172.21.15.7:0/2485387730 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a4b8f640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a0106160 0x7f61a0198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a5b91640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a019cfb0 0x7f61a019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a5390640 1 -- 172.21.15.7:0/2485387730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a019cfb0 msgr2=0x7f61a019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a5390640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a019cfb0 0x7f61a019d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a5390640 1 -- 172.21.15.7:0/2485387730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a0106160 msgr2=0x7f61a0198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a5390640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a0106160 0x7f61a0198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a5390640 1 -- 172.21.15.7:0/2485387730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a019daa0 con 0x7f61a0101090 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61a5390640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a0101090 0x7f61a01986f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f619c00d950 tx=0x7f619c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61967fc640 1 -- 172.21.15.7:0/2485387730 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c017070 con 0x7f61a0101090 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.414+0000 7f61967fc640 1 -- 172.21.15.7:0/2485387730 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f619c009c30 con 0x7f61a0101090 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.415+0000 7f61a75f4640 1 -- 172.21.15.7:0/2485387730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a019dd90 con 0x7f61a0101090 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.415+0000 7f61967fc640 1 -- 172.21.15.7:0/2485387730 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c00f620 con 0x7f61a0101090 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.415+0000 7f61a75f4640 1 -- 172.21.15.7:0/2485387730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a01a55d0 con 0x7f61a0101090 2024-04-21T08:44:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.415+0000 7f61a75f4640 1 -- 172.21.15.7:0/2485387730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61a0068e00 con 0x7f61a0101090 2024-04-21T08:44:53.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.417+0000 7f61967fc640 1 -- 172.21.15.7:0/2485387730 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f619c010460 con 0x7f61a0101090 2024-04-21T08:44:53.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.417+0000 7f61967fc640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f618c0a3e30 0x7f618c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.418+0000 7f61a4b8f640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f618c0a3e30 0x7f618c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.418+0000 7f61967fc640 1 -- 172.21.15.7:0/2485387730 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f619c0d0930 con 0x7f61a0101090 2024-04-21T08:44:53.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.418+0000 7f61a4b8f640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f618c0a3e30 0x7f618c0a62f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f61a0100ce0 tx=0x7f6190005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.421+0000 7f61967fc640 1 -- 172.21.15.7:0/2485387730 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f619c09caf0 con 0x7f61a0101090 2024-04-21T08:44:53.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.606+0000 7f61a75f4640 1 -- 172.21.15.7:0/2485387730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f61a0101a40 con 0x7f61a0101090 2024-04-21T08:44:53.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.607+0000 7f61967fc640 1 -- 172.21.15.7:0/2485387730 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f619c009de0 con 0x7f61a0101090 2024-04-21T08:44:53.606 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:44:53.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 -- 172.21.15.7:0/2485387730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f618c0a3e30 msgr2=0x7f618c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f618c0a3e30 0x7f618c0a62f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f61a0100ce0 tx=0x7f6190005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:53.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 -- 172.21.15.7:0/2485387730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a0101090 msgr2=0x7f61a01986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a0101090 0x7f61a01986f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f619c00d950 tx=0x7f619c00de10 comp rx=0 tx=0).stop 2024-04-21T08:44:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 -- 172.21.15.7:0/2485387730 shutdown_connections 2024-04-21T08:44:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f618c0a3e30 0x7f618c0a62f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f61a019cfb0 0x7f61a019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61a0106160 0x7f61a0198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 --2- 172.21.15.7:0/2485387730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61a0101090 0x7f61a01986f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 -- 172.21.15.7:0/2485387730 >> 172.21.15.7:0/2485387730 conn(0x7f61a0078000 msgr2=0x7f61a0069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 -- 172.21.15.7:0/2485387730 shutdown_connections 2024-04-21T08:44:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.609+0000 7f61a75f4640 1 -- 172.21.15.7:0/2485387730 wait complete. 2024-04-21T08:44:53.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:53.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.726+0000 7fcde26c0640 1 -- 172.21.15.7:0/1584941351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc103b50 msgr2=0x7fcddc103fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.726+0000 7fcde26c0640 1 --2- 172.21.15.7:0/1584941351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc103b50 0x7fcddc103fb0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009930 tx=0x7fcdd002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:53.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.726+0000 7fcde26c0640 1 -- 172.21.15.7:0/1584941351 shutdown_connections 2024-04-21T08:44:53.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.726+0000 7fcde26c0640 1 --2- 172.21.15.7:0/1584941351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcddc1044f0 0x7fcddc10eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.726+0000 7fcde26c0640 1 --2- 172.21.15.7:0/1584941351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc103b50 0x7fcddc103fb0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.726+0000 7fcde26c0640 1 --2- 172.21.15.7:0/1584941351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcddc102270 0x7fcddc102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.726+0000 7fcde26c0640 1 -- 172.21.15.7:0/1584941351 >> 172.21.15.7:0/1584941351 conn(0x7fcddc0fda10 msgr2=0x7fcddc0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:53.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.726+0000 7fcde26c0640 1 -- 172.21.15.7:0/1584941351 shutdown_connections 2024-04-21T08:44:53.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.726+0000 7fcde26c0640 1 -- 172.21.15.7:0/1584941351 wait complete. 2024-04-21T08:44:53.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcde26c0640 1 Processor -- start 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcde26c0640 1 -- start start 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcde26c0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcddc102270 0x7fcddc074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcde26c0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc103b50 0x7fcddc074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcde26c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcddc1044f0 0x7fcddc074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcde26c0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc112290 con 0x7fcddc103b50 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcde26c0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc112110 con 0x7fcddc1044f0 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcde26c0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc112410 con 0x7fcddc102270 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcddb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc103b50 0x7fcddc074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcddb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc103b50 0x7fcddc074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47658/0 (socket says 172.21.15.7:47658) 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcddb7fe640 1 -- 172.21.15.7:0/4017786665 learned_addr learned my addr 172.21.15.7:0/4017786665 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcddbfff640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcddc102270 0x7fcddc074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.727+0000 7fcde0c5d640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcddc1044f0 0x7fcddc074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.728+0000 7fcddb7fe640 1 -- 172.21.15.7:0/4017786665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcddc102270 msgr2=0x7fcddc074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.728+0000 7fcddb7fe640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcddc102270 0x7fcddc074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.728+0000 7fcddb7fe640 1 -- 172.21.15.7:0/4017786665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcddc1044f0 msgr2=0x7fcddc074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.728+0000 7fcddb7fe640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcddc1044f0 0x7fcddc074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.728+0000 7fcddb7fe640 1 -- 172.21.15.7:0/4017786665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcddc1a6d10 con 0x7fcddc103b50 2024-04-21T08:44:53.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.728+0000 7fcddb7fe640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc103b50 0x7fcddc074790 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009230 tx=0x7fcdd00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:53.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.728+0000 7fcdd97fa640 1 -- 172.21.15.7:0/4017786665 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0043020 con 0x7fcddc103b50 2024-04-21T08:44:53.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.728+0000 7fcdd97fa640 1 -- 172.21.15.7:0/4017786665 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd002cb70 con 0x7fcddc103b50 2024-04-21T08:44:53.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.728+0000 7fcde26c0640 1 -- 172.21.15.7:0/4017786665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcddc1a6fa0 con 0x7fcddc103b50 2024-04-21T08:44:53.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.728+0000 7fcdd97fa640 1 -- 172.21.15.7:0/4017786665 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0003b50 con 0x7fcddc103b50 2024-04-21T08:44:53.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.728+0000 7fcde26c0640 1 -- 172.21.15.7:0/4017786665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcddc1a7460 con 0x7fcddc103b50 2024-04-21T08:44:53.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.729+0000 7fcde26c0640 1 -- 172.21.15.7:0/4017786665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcddc069250 con 0x7fcddc103b50 2024-04-21T08:44:53.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.731+0000 7fcdd97fa640 1 -- 172.21.15.7:0/4017786665 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7fcdd0046020 con 0x7fcddc103b50 2024-04-21T08:44:53.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.731+0000 7fcdd97fa640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fcdc40a3db0 0x7fcdc40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:53.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.731+0000 7fcddbfff640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fcdc40a3db0 0x7fcdc40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:53.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.731+0000 7fcdd97fa640 1 -- 172.21.15.7:0/4017786665 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fcdd00f2db0 con 0x7fcddc103b50 2024-04-21T08:44:53.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.732+0000 7fcddbfff640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fcdc40a3db0 0x7fcdc40a6270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc005fd0 tx=0x7fcdcc00b390 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:53.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.734+0000 7fcdd97fa640 1 -- 172.21.15.7:0/4017786665 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcdd004d030 con 0x7fcddc103b50 2024-04-21T08:44:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:53 smithi007 ceph-mon[28477]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:53 smithi007 ceph-mon[28477]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:53 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1609749056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:53 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/663174487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:53 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2485387730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:53.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.922+0000 7fcde26c0640 1 -- 172.21.15.7:0/4017786665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcddc10a540 con 0x7fcddc103b50 2024-04-21T08:44:53.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.922+0000 7fcdd97fa640 1 -- 172.21.15.7:0/4017786665 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7fcdd00bf120 con 0x7fcddc103b50 2024-04-21T08:44:53.922 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 -- 172.21.15.7:0/4017786665 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fcdc40a3db0 msgr2=0x7fcdc40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fcdc40a3db0 0x7fcdc40a6270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc005fd0 tx=0x7fcdcc00b390 comp rx=0 tx=0).stop 2024-04-21T08:44:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 -- 172.21.15.7:0/4017786665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc103b50 msgr2=0x7fcddc074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc103b50 0x7fcddc074790 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009230 tx=0x7fcdd00045e0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 -- 172.21.15.7:0/4017786665 shutdown_connections 2024-04-21T08:44:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcddc1044f0 0x7fcddc074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc103b50 0x7fcddc074790 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fcdc40a3db0 0x7fcdc40a6270 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 --2- 172.21.15.7:0/4017786665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcddc102270 0x7fcddc074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 -- 172.21.15.7:0/4017786665 >> 172.21.15.7:0/4017786665 conn(0x7fcddc0fda10 msgr2=0x7fcddc0fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 -- 172.21.15.7:0/4017786665 shutdown_connections 2024-04-21T08:44:53.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:53.924+0000 7fcde26c0640 1 -- 172.21.15.7:0/4017786665 wait complete. 2024-04-21T08:44:53.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.042+0000 7fbdf6c81640 1 -- 172.21.15.7:0/2676747583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf01031c0 msgr2=0x7fbdf01035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.042+0000 7fbdf6c81640 1 --2- 172.21.15.7:0/2676747583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf01031c0 0x7fbdf01035a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009900 tx=0x7fbde402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.042+0000 7fbdf6c81640 1 -- 172.21.15.7:0/2676747583 shutdown_connections 2024-04-21T08:44:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.042+0000 7fbdf6c81640 1 --2- 172.21.15.7:0/2676747583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf010c6d0 0x7fbdf010eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.042+0000 7fbdf6c81640 1 --2- 172.21.15.7:0/2676747583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdf0103b70 0x7fbdf010bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.042+0000 7fbdf6c81640 1 --2- 172.21.15.7:0/2676747583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf01031c0 0x7fbdf01035a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.042+0000 7fbdf6c81640 1 -- 172.21.15.7:0/2676747583 >> 172.21.15.7:0/2676747583 conn(0x7fbdf0077f70 msgr2=0x7fbdf0078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.042+0000 7fbdf6c81640 1 -- 172.21.15.7:0/2676747583 shutdown_connections 2024-04-21T08:44:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.042+0000 7fbdf6c81640 1 -- 172.21.15.7:0/2676747583 wait complete. 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf6c81640 1 Processor -- start 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf6c81640 1 -- start start 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf6c81640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf01031c0 0x7fbdf019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf6c81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf010c6d0 0x7fbdf019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf6c81640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdf0198760 0x7fbdf0198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf6c81640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf01123a0 con 0x7fbdf01031c0 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf6c81640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf0112220 con 0x7fbdf0198760 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf6c81640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf0112520 con 0x7fbdf010c6d0 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf4a1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf01031c0 0x7fbdf019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf4a1d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf01031c0 0x7fbdf019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47676/0 (socket says 172.21.15.7:47676) 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf4a1d640 1 -- 172.21.15.7:0/3860759625 learned_addr learned my addr 172.21.15.7:0/3860759625 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdeffff640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf010c6d0 0x7fbdf019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf521e640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdf0198760 0x7fbdf0198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf4a1d640 1 -- 172.21.15.7:0/3860759625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf010c6d0 msgr2=0x7fbdf019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf4a1d640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf010c6d0 0x7fbdf019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf4a1d640 1 -- 172.21.15.7:0/3860759625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdf0198760 msgr2=0x7fbdf0198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf4a1d640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdf0198760 0x7fbdf0198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf4a1d640 1 -- 172.21.15.7:0/3860759625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf0199360 con 0x7fbdf01031c0 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.043+0000 7fbdf521e640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdf0198760 0x7fbdf0198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.044+0000 7fbdf4a1d640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf01031c0 0x7fbdf019e6a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fbde4005940 tx=0x7fbde402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.044+0000 7fbdedffb640 1 -- 172.21.15.7:0/3860759625 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde4043020 con 0x7fbdf01031c0 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.044+0000 7fbdedffb640 1 -- 172.21.15.7:0/3860759625 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbde402cbe0 con 0x7fbdf01031c0 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.044+0000 7fbdedffb640 1 -- 172.21.15.7:0/3860759625 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde4003bf0 con 0x7fbdf01031c0 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.044+0000 7fbdf6c81640 1 -- 172.21.15.7:0/3860759625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf01995f0 con 0x7fbdf01031c0 2024-04-21T08:44:54.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.044+0000 7fbdf6c81640 1 -- 172.21.15.7:0/3860759625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf01a5790 con 0x7fbdf01031c0 2024-04-21T08:44:54.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.045+0000 7fbdf6c81640 1 -- 172.21.15.7:0/3860759625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf006ae40 con 0x7fbdf01031c0 2024-04-21T08:44:54.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.046+0000 7fbdedffb640 1 -- 172.21.15.7:0/3860759625 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7fbde4046020 con 0x7fbdf01031c0 2024-04-21T08:44:54.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.047+0000 7fbdedffb640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbde00a3db0 0x7fbde00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.047+0000 7fbdeffff640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbde00a3db0 0x7fbde00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.047+0000 7fbdedffb640 1 -- 172.21.15.7:0/3860759625 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fbde40f3620 con 0x7fbdf01031c0 2024-04-21T08:44:54.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.047+0000 7fbdeffff640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbde00a3db0 0x7fbde00a6270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0199fb0 tx=0x7fbddc00a4c0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:54.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.050+0000 7fbdedffb640 1 -- 172.21.15.7:0/3860759625 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbde404d020 con 0x7fbdf01031c0 2024-04-21T08:44:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.238+0000 7fbdf6c81640 1 -- 172.21.15.7:0/3860759625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbdf0069250 con 0x7fbdf01031c0 2024-04-21T08:44:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.238+0000 7fbdedffb640 1 -- 172.21.15.7:0/3860759625 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fbde40bf810 con 0x7fbdf01031c0 2024-04-21T08:44:54.238 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:44:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 -- 172.21.15.7:0/3860759625 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbde00a3db0 msgr2=0x7fbde00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbde00a3db0 0x7fbde00a6270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0199fb0 tx=0x7fbddc00a4c0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 -- 172.21.15.7:0/3860759625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf01031c0 msgr2=0x7fbdf019e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf01031c0 0x7fbdf019e6a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fbde4005940 tx=0x7fbde402c7e0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 -- 172.21.15.7:0/3860759625 shutdown_connections 2024-04-21T08:44:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbdf0198760 0x7fbdf0198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbde00a3db0 0x7fbde00a6270 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf010c6d0 0x7fbdf019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 --2- 172.21.15.7:0/3860759625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf01031c0 0x7fbdf019e6a0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 -- 172.21.15.7:0/3860759625 >> 172.21.15.7:0/3860759625 conn(0x7fbdf0077f70 msgr2=0x7fbdf00fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 -- 172.21.15.7:0/3860759625 shutdown_connections 2024-04-21T08:44:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.240+0000 7fbdf6c81640 1 -- 172.21.15.7:0/3860759625 wait complete. 2024-04-21T08:44:54.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:54.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:53 smithi016 ceph-mon[31259]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:54.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:53 smithi016 ceph-mon[31259]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:54.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:53 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1609749056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:54.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:53 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/663174487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:54.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:53 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2485387730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:54.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:53 smithi077 ceph-mon[31615]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:54.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:53 smithi077 ceph-mon[31615]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:54.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:53 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1609749056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:54.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:53 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/663174487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:54.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:53 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2485387730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.356+0000 7f470dea8640 1 -- 172.21.15.7:0/2549868856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f470810c6d0 msgr2=0x7f470810eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.356+0000 7f470dea8640 1 --2- 172.21.15.7:0/2549868856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f470810c6d0 0x7f470810eb60 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f4700009ca0 tx=0x7f470002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.356+0000 7f470dea8640 1 -- 172.21.15.7:0/2549868856 shutdown_connections 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.356+0000 7f470dea8640 1 --2- 172.21.15.7:0/2549868856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f470810c6d0 0x7f470810eb60 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.356+0000 7f470dea8640 1 --2- 172.21.15.7:0/2549868856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4708103b70 0x7f470810bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.356+0000 7f470dea8640 1 --2- 172.21.15.7:0/2549868856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47081031c0 0x7f47081035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.356+0000 7f470dea8640 1 -- 172.21.15.7:0/2549868856 >> 172.21.15.7:0/2549868856 conn(0x7f4708077f70 msgr2=0x7f4708078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.356+0000 7f470dea8640 1 -- 172.21.15.7:0/2549868856 shutdown_connections 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.356+0000 7f470dea8640 1 -- 172.21.15.7:0/2549868856 wait complete. 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.356+0000 7f470dea8640 1 Processor -- start 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.356+0000 7f470dea8640 1 -- start start 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f470dea8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47081031c0 0x7f470819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f470dea8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470810c6d0 0x7f470819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f470dea8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708198700 0x7f4708198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f470dea8640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4708112380 con 0x7f4708198700 2024-04-21T08:44:54.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f470dea8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4708112200 con 0x7f47081031c0 2024-04-21T08:44:54.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f470dea8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4708112500 con 0x7f470810c6d0 2024-04-21T08:44:54.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4707fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708198700 0x7f4708198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4707fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708198700 0x7f4708198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47684/0 (socket says 172.21.15.7:47684) 2024-04-21T08:44:54.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4707fff640 1 -- 172.21.15.7:0/2011993683 learned_addr learned my addr 172.21.15.7:0/2011993683 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:54.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4706ffd640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470810c6d0 0x7f470819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f47077fe640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47081031c0 0x7f470819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4707fff640 1 -- 172.21.15.7:0/2011993683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470810c6d0 msgr2=0x7f470819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4707fff640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470810c6d0 0x7f470819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4707fff640 1 -- 172.21.15.7:0/2011993683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47081031c0 msgr2=0x7f470819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4707fff640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47081031c0 0x7f470819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4707fff640 1 -- 172.21.15.7:0/2011993683 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4708199300 con 0x7f4708198700 2024-04-21T08:44:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f47077fe640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47081031c0 0x7f470819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4707fff640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708198700 0x7f4708198b80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4700005e30 tx=0x7f4700034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4704ff9640 1 -- 172.21.15.7:0/2011993683 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700039040 con 0x7f4708198700 2024-04-21T08:44:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4704ff9640 1 -- 172.21.15.7:0/2011993683 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4700044070 con 0x7f4708198700 2024-04-21T08:44:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f470dea8640 1 -- 172.21.15.7:0/2011993683 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4708199590 con 0x7f4708198700 2024-04-21T08:44:54.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f4704ff9640 1 -- 172.21.15.7:0/2011993683 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700003e60 con 0x7f4708198700 2024-04-21T08:44:54.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.357+0000 7f470dea8640 1 -- 172.21.15.7:0/2011993683 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47081a56d0 con 0x7f4708198700 2024-04-21T08:44:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.359+0000 7f470dea8640 1 -- 172.21.15.7:0/2011993683 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470806ae40 con 0x7f4708198700 2024-04-21T08:44:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.360+0000 7f4704ff9640 1 -- 172.21.15.7:0/2011993683 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f47000324c0 con 0x7f4708198700 2024-04-21T08:44:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.361+0000 7f4704ff9640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f46f00a3db0 0x7f46f00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.361+0000 7f47077fe640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f46f00a3db0 0x7f46f00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.361+0000 7f4704ff9640 1 -- 172.21.15.7:0/2011993683 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f47000f3990 con 0x7f4708198700 2024-04-21T08:44:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.361+0000 7f47077fe640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f46f00a3db0 0x7f46f00a6270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f46f8009b40 tx=0x7f46f8005e50 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.364+0000 7f4704ff9640 1 -- 172.21.15.7:0/2011993683 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f47000bfbe0 con 0x7f4708198700 2024-04-21T08:44:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.549+0000 7f470dea8640 1 -- 172.21.15.7:0/2011993683 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4708107680 con 0x7f4708198700 2024-04-21T08:44:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.550+0000 7f4704ff9640 1 -- 172.21.15.7:0/2011993683 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f47000bf4e0 con 0x7f4708198700 2024-04-21T08:44:54.549 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:44:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.551+0000 7f470dea8640 1 -- 172.21.15.7:0/2011993683 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f46f00a3db0 msgr2=0x7f46f00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.551+0000 7f470dea8640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f46f00a3db0 0x7f46f00a6270 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f46f8009b40 tx=0x7f46f8005e50 comp rx=0 tx=0).stop 2024-04-21T08:44:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.551+0000 7f470dea8640 1 -- 172.21.15.7:0/2011993683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708198700 msgr2=0x7f4708198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.551+0000 7f470dea8640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708198700 0x7f4708198b80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4700005e30 tx=0x7f4700034760 comp rx=0 tx=0).stop 2024-04-21T08:44:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.551+0000 7f470dea8640 1 -- 172.21.15.7:0/2011993683 shutdown_connections 2024-04-21T08:44:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.551+0000 7f470dea8640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4708198700 0x7f4708198b80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.552+0000 7f470dea8640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f46f00a3db0 0x7f46f00a6270 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.552+0000 7f470dea8640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f470810c6d0 0x7f470819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.552+0000 7f470dea8640 1 --2- 172.21.15.7:0/2011993683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47081031c0 0x7f470819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.552+0000 7f470dea8640 1 -- 172.21.15.7:0/2011993683 >> 172.21.15.7:0/2011993683 conn(0x7f4708077f70 msgr2=0x7f47080fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.552+0000 7f470dea8640 1 -- 172.21.15.7:0/2011993683 shutdown_connections 2024-04-21T08:44:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.552+0000 7f470dea8640 1 -- 172.21.15.7:0/2011993683 wait complete. 2024-04-21T08:44:54.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:54.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.667+0000 7f455154b640 1 -- 172.21.15.7:0/604734759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f454c101070 msgr2=0x7f454c101450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.667+0000 7f455154b640 1 --2- 172.21.15.7:0/604734759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f454c101070 0x7f454c101450 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4534009900 tx=0x7f453402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:54.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 -- 172.21.15.7:0/604734759 shutdown_connections 2024-04-21T08:44:54.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 --2- 172.21.15.7:0/604734759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f454c106140 0x7f454c10dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 --2- 172.21.15.7:0/604734759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f454c101a20 0x7f454c105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 --2- 172.21.15.7:0/604734759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f454c101070 0x7f454c101450 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 -- 172.21.15.7:0/604734759 >> 172.21.15.7:0/604734759 conn(0x7f454c0771d0 msgr2=0x7f454c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 -- 172.21.15.7:0/604734759 shutdown_connections 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 -- 172.21.15.7:0/604734759 wait complete. 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 Processor -- start 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 -- start start 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f454c101070 0x7f454c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f454c106140 0x7f454c198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.668+0000 7f455154b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f454c19cf60 0x7f454c19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f455154b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f454c1121c0 con 0x7f454c101070 2024-04-21T08:44:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f455154b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f454c112040 con 0x7f454c106140 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f455154b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f454c112340 con 0x7f454c19cf60 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f454c101070 0x7f454c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f454c101070 0x7f454c1986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33102/0 (socket says 172.21.15.7:33102) 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454affd640 1 -- 172.21.15.7:0/1275308048 learned_addr learned my addr 172.21.15.7:0/1275308048 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454b7fe640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f454c19cf60 0x7f454c19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454a7fc640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f454c106140 0x7f454c198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454affd640 1 -- 172.21.15.7:0/1275308048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f454c19cf60 msgr2=0x7f454c19d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454affd640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f454c19cf60 0x7f454c19d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454affd640 1 -- 172.21.15.7:0/1275308048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f454c106140 msgr2=0x7f454c198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454affd640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f454c106140 0x7f454c198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454affd640 1 -- 172.21.15.7:0/1275308048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f454c19da50 con 0x7f454c101070 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454b7fe640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f454c19cf60 0x7f454c19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454a7fc640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f454c106140 0x7f454c198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f454affd640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f454c101070 0x7f454c1986a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f4534005940 tx=0x7f453402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f4543fff640 1 -- 172.21.15.7:0/1275308048 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534043020 con 0x7f454c101070 2024-04-21T08:44:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f4543fff640 1 -- 172.21.15.7:0/1275308048 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f453402cbe0 con 0x7f454c101070 2024-04-21T08:44:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f455154b640 1 -- 172.21.15.7:0/1275308048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f454c19dce0 con 0x7f454c101070 2024-04-21T08:44:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f4543fff640 1 -- 172.21.15.7:0/1275308048 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534003bf0 con 0x7f454c101070 2024-04-21T08:44:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.669+0000 7f455154b640 1 -- 172.21.15.7:0/1275308048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f454c1a5500 con 0x7f454c101070 2024-04-21T08:44:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.670+0000 7f455154b640 1 -- 172.21.15.7:0/1275308048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f454c068e00 con 0x7f454c101070 2024-04-21T08:44:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.672+0000 7f4543fff640 1 -- 172.21.15.7:0/1275308048 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f4534046020 con 0x7f454c101070 2024-04-21T08:44:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.672+0000 7f4543fff640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f45380a3db0 0x7f45380a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.672+0000 7f4543fff640 1 -- 172.21.15.7:0/1275308048 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f45340f3630 con 0x7f454c101070 2024-04-21T08:44:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.672+0000 7f454a7fc640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f45380a3db0 0x7f45380a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.673+0000 7f454a7fc640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f45380a3db0 0x7f45380a6270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f453c0058c0 tx=0x7f453c005850 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:54.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.676+0000 7f4543fff640 1 -- 172.21.15.7:0/1275308048 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f453404d020 con 0x7f454c101070 2024-04-21T08:44:54.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.906+0000 7f455154b640 1 -- 172.21.15.7:0/1275308048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f454c1999f0 con 0x7f454c101070 2024-04-21T08:44:54.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.907+0000 7f4543fff640 1 -- 172.21.15.7:0/1275308048 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7f454c1999f0 con 0x7f454c101070 2024-04-21T08:44:54.906 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:54.906 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:44:54.907 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:44:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 -- 172.21.15.7:0/1275308048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f45380a3db0 msgr2=0x7f45380a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f45380a3db0 0x7f45380a6270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f453c0058c0 tx=0x7f453c005850 comp rx=0 tx=0).stop 2024-04-21T08:44:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 -- 172.21.15.7:0/1275308048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f454c101070 msgr2=0x7f454c1986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f454c101070 0x7f454c1986a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f4534005940 tx=0x7f453402c7e0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 -- 172.21.15.7:0/1275308048 shutdown_connections 2024-04-21T08:44:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f45380a3db0 0x7f45380a6270 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f454c19cf60 0x7f454c19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f454c106140 0x7f454c198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 --2- 172.21.15.7:0/1275308048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f454c101070 0x7f454c1986a0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 -- 172.21.15.7:0/1275308048 >> 172.21.15.7:0/1275308048 conn(0x7f454c0771d0 msgr2=0x7f454c100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 -- 172.21.15.7:0/1275308048 shutdown_connections 2024-04-21T08:44:54.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:54.909+0000 7f455154b640 1 -- 172.21.15.7:0/1275308048 wait complete. 2024-04-21T08:44:54.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:55.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.023+0000 7f9df614b640 1 -- 172.21.15.7:0/634685336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df010c580 msgr2=0x7f9df010e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.023+0000 7f9df614b640 1 --2- 172.21.15.7:0/634685336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df010c580 0x7f9df010e9b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009ca0 tx=0x7f9de802c180 comp rx=0 tx=0).stop 2024-04-21T08:44:55.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.024+0000 7f9df614b640 1 -- 172.21.15.7:0/634685336 shutdown_connections 2024-04-21T08:44:55.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.024+0000 7f9df614b640 1 --2- 172.21.15.7:0/634685336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df010c580 0x7f9df010e9b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.024+0000 7f9df614b640 1 --2- 172.21.15.7:0/634685336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103ad0 0x7f9df010bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.024+0000 7f9df614b640 1 --2- 172.21.15.7:0/634685336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df01031b0 0x7f9df0103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.024+0000 7f9df614b640 1 -- 172.21.15.7:0/634685336 >> 172.21.15.7:0/634685336 conn(0x7f9df00771d0 msgr2=0x7f9df00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.024+0000 7f9df614b640 1 -- 172.21.15.7:0/634685336 shutdown_connections 2024-04-21T08:44:55.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.024+0000 7f9df614b640 1 -- 172.21.15.7:0/634685336 wait complete. 2024-04-21T08:44:55.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.024+0000 7f9df614b640 1 Processor -- start 2024-04-21T08:44:55.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.024+0000 7f9df614b640 1 -- start start 2024-04-21T08:44:55.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9df614b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df01031b0 0x7f9df0100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9df614b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103ad0 0x7f9df0101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9df614b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df010c580 0x7f9df0101880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9df614b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01121a0 con 0x7f9df01031b0 2024-04-21T08:44:55.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9df614b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9df0112020 con 0x7f9df0103ad0 2024-04-21T08:44:55.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9df614b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9df0112320 con 0x7f9df010c580 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9def7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df01031b0 0x7f9df0100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9def7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df01031b0 0x7f9df0100e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33128/0 (socket says 172.21.15.7:33128) 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9def7fe640 1 -- 172.21.15.7:0/2721100141 learned_addr learned my addr 172.21.15.7:0/2721100141 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9deeffd640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103ad0 0x7f9df0101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9deffff640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df010c580 0x7f9df0101880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9def7fe640 1 -- 172.21.15.7:0/2721100141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df010c580 msgr2=0x7f9df0101880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9def7fe640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df010c580 0x7f9df0101880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9def7fe640 1 -- 172.21.15.7:0/2721100141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103ad0 msgr2=0x7f9df0101340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9def7fe640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103ad0 0x7f9df0101340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9def7fe640 1 -- 172.21.15.7:0/2721100141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df01a6bc0 con 0x7f9df01031b0 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9deffff640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df010c580 0x7f9df0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9def7fe640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df01031b0 0x7f9df0100e00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9de000e9f0 tx=0x7f9de000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9decff9640 1 -- 172.21.15.7:0/2721100141 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de000cd80 con 0x7f9df01031b0 2024-04-21T08:44:55.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9decff9640 1 -- 172.21.15.7:0/2721100141 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9de00054d0 con 0x7f9df01031b0 2024-04-21T08:44:55.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9df614b640 1 -- 172.21.15.7:0/2721100141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df01a6eb0 con 0x7f9df01031b0 2024-04-21T08:44:55.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9decff9640 1 -- 172.21.15.7:0/2721100141 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0010690 con 0x7f9df01031b0 2024-04-21T08:44:55.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.025+0000 7f9df614b640 1 -- 172.21.15.7:0/2721100141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df01a7370 con 0x7f9df01031b0 2024-04-21T08:44:55.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.026+0000 7f9df614b640 1 -- 172.21.15.7:0/2721100141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df006ae80 con 0x7f9df01031b0 2024-04-21T08:44:55.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.028+0000 7f9decff9640 1 -- 172.21.15.7:0/2721100141 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f9de0004e90 con 0x7f9df01031b0 2024-04-21T08:44:55.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.028+0000 7f9decff9640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f9dd80a3e30 0x7f9dd80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.028+0000 7f9decff9640 1 -- 172.21.15.7:0/2721100141 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f9de00d11b0 con 0x7f9df01031b0 2024-04-21T08:44:55.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.028+0000 7f9deeffd640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f9dd80a3e30 0x7f9dd80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.029+0000 7f9deeffd640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f9dd80a3e30 0x7f9dd80a62f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9df0102890 tx=0x7f9de400b410 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.033+0000 7f9decff9640 1 -- 172.21.15.7:0/2721100141 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9de009d2f0 con 0x7f9df01031b0 2024-04-21T08:44:55.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:54 smithi007 ceph-mon[28477]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 83 KiB/s wr, 47 op/s 2024-04-21T08:44:55.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:54 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4017786665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:55.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:54 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3860759625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:55.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:54 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2011993683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.221+0000 7f9df614b640 1 -- 172.21.15.7:0/2721100141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9df002d5c0 con 0x7f9df01031b0 2024-04-21T08:44:55.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.222+0000 7f9decff9640 1 -- 172.21.15.7:0/2721100141 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25085 (secure 0 0 0) 0x7f9de009cbf0 con 0x7f9df01031b0 2024-04-21T08:44:55.223 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:55.223 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":95,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:44:41.244735+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3451243741":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/117789107":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/4167672076":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/3630897301":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.224+0000 7f9df614b640 1 -- 172.21.15.7:0/2721100141 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f9dd80a3e30 msgr2=0x7f9dd80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.224+0000 7f9df614b640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f9dd80a3e30 0x7f9dd80a62f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9df0102890 tx=0x7f9de400b410 comp rx=0 tx=0).stop 2024-04-21T08:44:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.224+0000 7f9df614b640 1 -- 172.21.15.7:0/2721100141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df01031b0 msgr2=0x7f9df0100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.224+0000 7f9df614b640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df01031b0 0x7f9df0100e00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9de000e9f0 tx=0x7f9de000edf0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.224+0000 7f9df614b640 1 -- 172.21.15.7:0/2721100141 shutdown_connections 2024-04-21T08:44:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.224+0000 7f9df614b640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f9dd80a3e30 0x7f9dd80a62f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.224+0000 7f9df614b640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9df010c580 0x7f9df0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.224+0000 7f9df614b640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9df0103ad0 0x7f9df0101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.224+0000 7f9df614b640 1 --2- 172.21.15.7:0/2721100141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9df01031b0 0x7f9df0100e00 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.224+0000 7f9df614b640 1 -- 172.21.15.7:0/2721100141 >> 172.21.15.7:0/2721100141 conn(0x7f9df00771d0 msgr2=0x7f9df00fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.224+0000 7f9df614b640 1 -- 172.21.15.7:0/2721100141 shutdown_connections 2024-04-21T08:44:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.225+0000 7f9df614b640 1 -- 172.21.15.7:0/2721100141 wait complete. 2024-04-21T08:44:55.236 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 485 2024-04-21T08:44:55.236 DEBUG:tasks.quiescer:Yielding 2024-04-21T08:44:55.236 INFO:teuthology.run_tasks:Running task workunit... 2024-04-21T08:44:55.251 INFO:tasks.workunit:Pulling workunits from ref 4f6bc5273643aaf5660909e85400647a8951c10b 2024-04-21T08:44:55.251 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-21T08:44:55.251 INFO:tasks.workunit:timeout=3h 2024-04-21T08:44:55.251 INFO:tasks.workunit:cleanup=True 2024-04-21T08:44:55.252 DEBUG:teuthology.orchestra.run.smithi007:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-21T08:44:55.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:55.268 INFO:teuthology.orchestra.run.smithi007.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-21T08:44:55.268 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-21T08:44:55.268 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-21T08:44:55.268 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:44:55.269 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-21T08:44:55.269 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-21 08:42:44.186053475 +0000 2024-04-21T08:44:55.269 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-21 08:42:44.186053475 +0000 2024-04-21T08:44:55.269 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-21 08:44:39.021866484 +0000 2024-04-21T08:44:55.269 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-21 08:42:44.186053475 +0000 2024-04-21T08:44:55.269 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-21T08:44:55.269 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-21T08:44:55.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:54 smithi016 ceph-mon[31259]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 83 KiB/s wr, 47 op/s 2024-04-21T08:44:55.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:54 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4017786665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:55.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:54 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3860759625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:55.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:54 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2011993683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:55.337 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/leonid-s-usov/ceph-ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4f6bc5273643aaf5660909e85400647a8951c10b 2024-04-21T08:44:55.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:54 smithi077 ceph-mon[31615]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 83 KiB/s wr, 47 op/s 2024-04-21T08:44:55.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:54 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4017786665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:55.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:54 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3860759625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:55.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:54 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2011993683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:55.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.356+0000 7fdea5b52640 1 -- 172.21.15.7:0/1883332542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdea0100d70 msgr2=0x7fdea010ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.356+0000 7fdea5b52640 1 --2- 172.21.15.7:0/1883332542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdea0100d70 0x7fdea010ee10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fde98009cd0 tx=0x7fde9802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.356+0000 7fdea5b52640 1 -- 172.21.15.7:0/1883332542 shutdown_connections 2024-04-21T08:44:55.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.356+0000 7fdea5b52640 1 --2- 172.21.15.7:0/1883332542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdea0100d70 0x7fdea010ee10 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.356+0000 7fdea5b52640 1 --2- 172.21.15.7:0/1883332542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea01003f0 0x7fdea0100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.356+0000 7fdea5b52640 1 --2- 172.21.15.7:0/1883332542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdea0107fe0 0x7fdea01083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.356+0000 7fdea5b52640 1 -- 172.21.15.7:0/1883332542 >> 172.21.15.7:0/1883332542 conn(0x7fdea0077370 msgr2=0x7fdea00fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.356+0000 7fdea5b52640 1 -- 172.21.15.7:0/1883332542 shutdown_connections 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.356+0000 7fdea5b52640 1 -- 172.21.15.7:0/1883332542 wait complete. 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.357+0000 7fdea5b52640 1 Processor -- start 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.357+0000 7fdea5b52640 1 -- start start 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.357+0000 7fdea5b52640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdea01003f0 0x7fdea01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.357+0000 7fdea5b52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea0100d70 0x7fdea0198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.357+0000 7fdea5b52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdea0107fe0 0x7fdea019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.357+0000 7fdea5b52640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdea01121a0 con 0x7fdea01003f0 2024-04-21T08:44:55.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.357+0000 7fdea5b52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdea0112020 con 0x7fdea0100d70 2024-04-21T08:44:55.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.357+0000 7fdea5b52640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdea0112320 con 0x7fdea0107fe0 2024-04-21T08:44:55.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.357+0000 7fde9f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdea01003f0 0x7fdea01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.357+0000 7fde9f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdea01003f0 0x7fdea01986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33148/0 (socket says 172.21.15.7:33148) 2024-04-21T08:44:55.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.357+0000 7fde9f7fe640 1 -- 172.21.15.7:0/4099683063 learned_addr learned my addr 172.21.15.7:0/4099683063 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:55.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fde9ffff640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdea0107fe0 0x7fdea019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fde9effd640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea0100d70 0x7fdea0198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fde9f7fe640 1 -- 172.21.15.7:0/4099683063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdea0107fe0 msgr2=0x7fdea019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fde9f7fe640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdea0107fe0 0x7fdea019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fde9f7fe640 1 -- 172.21.15.7:0/4099683063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea0100d70 msgr2=0x7fdea0198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fde9f7fe640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea0100d70 0x7fdea0198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fde9f7fe640 1 -- 172.21.15.7:0/4099683063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea019d6b0 con 0x7fdea01003f0 2024-04-21T08:44:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fde9f7fe640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdea01003f0 0x7fdea01986c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fde9000e9f0 tx=0x7fde9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fde9cff9640 1 -- 172.21.15.7:0/4099683063 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9000cd80 con 0x7fdea01003f0 2024-04-21T08:44:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fde9cff9640 1 -- 172.21.15.7:0/4099683063 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde900108e0 con 0x7fdea01003f0 2024-04-21T08:44:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fdea5b52640 1 -- 172.21.15.7:0/4099683063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdea019d9a0 con 0x7fdea01003f0 2024-04-21T08:44:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fde9cff9640 1 -- 172.21.15.7:0/4099683063 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde90018980 con 0x7fdea01003f0 2024-04-21T08:44:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.358+0000 7fdea5b52640 1 -- 172.21.15.7:0/4099683063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdea01a5230 con 0x7fdea01003f0 2024-04-21T08:44:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.359+0000 7fdea5b52640 1 -- 172.21.15.7:0/4099683063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdea0069250 con 0x7fdea01003f0 2024-04-21T08:44:55.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.360+0000 7fde9cff9640 1 -- 172.21.15.7:0/4099683063 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7fde90010430 con 0x7fdea01003f0 2024-04-21T08:44:55.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.361+0000 7fde9cff9640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde880a3db0 0x7fde880a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.361+0000 7fde9cff9640 1 -- 172.21.15.7:0/4099683063 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fde900d14b0 con 0x7fdea01003f0 2024-04-21T08:44:55.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.361+0000 7fde9effd640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde880a3db0 0x7fde880a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.362+0000 7fde9effd640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde880a3db0 0x7fde880a6270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdea00fe270 tx=0x7fde94005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.365+0000 7fde9cff9640 1 -- 172.21.15.7:0/4099683063 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde9009d690 con 0x7fdea01003f0 2024-04-21T08:44:55.372 INFO:tasks.workunit.client.0.smithi007.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-21T08:44:55.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.594+0000 7fdea5b52640 1 -- 172.21.15.7:0/4099683063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdea0199a10 con 0x7fdea01003f0 2024-04-21T08:44:55.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.595+0000 7fde9cff9640 1 -- 172.21.15.7:0/4099683063 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7fde9009cf90 con 0x7fdea01003f0 2024-04-21T08:44:55.594 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:44:55.595 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:44:55.595 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 -- 172.21.15.7:0/4099683063 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde880a3db0 msgr2=0x7fde880a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde880a3db0 0x7fde880a6270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdea00fe270 tx=0x7fde94005f90 comp rx=0 tx=0).stop 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 -- 172.21.15.7:0/4099683063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdea01003f0 msgr2=0x7fdea01986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdea01003f0 0x7fdea01986c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fde9000e9f0 tx=0x7fde9000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 -- 172.21.15.7:0/4099683063 shutdown_connections 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde880a3db0 0x7fde880a6270 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdea0107fe0 0x7fdea019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdea0100d70 0x7fdea0198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 --2- 172.21.15.7:0/4099683063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdea01003f0 0x7fdea01986c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 -- 172.21.15.7:0/4099683063 >> 172.21.15.7:0/4099683063 conn(0x7fdea0077370 msgr2=0x7fdea00feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 -- 172.21.15.7:0/4099683063 shutdown_connections 2024-04-21T08:44:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.597+0000 7fdea5b52640 1 -- 172.21.15.7:0/4099683063 wait complete. 2024-04-21T08:44:55.608 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:42:16.996517+0000', 'modified': '2024-04-21T08:42:42.080753+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24533}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24532': {'gid': 24532, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.77:6839/516970569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 516970569}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 516970569}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6839/1703654929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6838', 'nonce': 1703654929}, {'type': 'v1', 'addr': '172.21.15.16:6839', 'nonce': 1703654929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24533, 'qdb_cluster': [24533]} 2024-04-21T08:44:55.608 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T08:44:55.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:44:55.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.752+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/1312759734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c58106160 msgr2=0x7f4c5810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.752+0000 7f4c5f0d3640 1 --2- 172.21.15.7:0/1312759734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c58106160 0x7f4c5810dd30 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009ca0 tx=0x7f4c5002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.752+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/1312759734 shutdown_connections 2024-04-21T08:44:55.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.752+0000 7f4c5f0d3640 1 --2- 172.21.15.7:0/1312759734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c58106160 0x7f4c5810dd30 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.752+0000 7f4c5f0d3640 1 --2- 172.21.15.7:0/1312759734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c58101a40 0x7f4c58105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.752+0000 7f4c5f0d3640 1 --2- 172.21.15.7:0/1312759734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c58101090 0x7f4c58101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.752+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/1312759734 >> 172.21.15.7:0/1312759734 conn(0x7f4c58078000 msgr2=0x7f4c58078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.752+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/1312759734 shutdown_connections 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.752+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/1312759734 wait complete. 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5f0d3640 1 Processor -- start 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5f0d3640 1 -- start start 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5f0d3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c58101090 0x7f4c58198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5f0d3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c58106160 0x7f4c58198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5f0d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5819cf20 0x7f4c5819d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5f0d3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4c581121e0 con 0x7f4c58101090 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5f0d3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4c58112060 con 0x7f4c58106160 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5f0d3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4c58112360 con 0x7f4c5819cf20 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5ce6f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c58101090 0x7f4c58198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5ce6f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c58101090 0x7f4c58198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33172/0 (socket says 172.21.15.7:33172) 2024-04-21T08:44:55.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5ce6f640 1 -- 172.21.15.7:0/4285753536 learned_addr learned my addr 172.21.15.7:0/4285753536 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:55.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c57fff640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c58106160 0x7f4c58198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5d670640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5819cf20 0x7f4c5819d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5ce6f640 1 -- 172.21.15.7:0/4285753536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5819cf20 msgr2=0x7f4c5819d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5ce6f640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5819cf20 0x7f4c5819d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5ce6f640 1 -- 172.21.15.7:0/4285753536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c58106160 msgr2=0x7f4c58198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5ce6f640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c58106160 0x7f4c58198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5ce6f640 1 -- 172.21.15.7:0/4285753536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5819da10 con 0x7f4c58101090 2024-04-21T08:44:55.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5d670640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5819cf20 0x7f4c5819d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:55.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c57fff640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c58106160 0x7f4c58198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:55.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.753+0000 7f4c5ce6f640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c58101090 0x7f4c58198660 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800c920 tx=0x7f4c4800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.754+0000 7f4c55ffb640 1 -- 172.21.15.7:0/4285753536 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c48007c70 con 0x7f4c58101090 2024-04-21T08:44:55.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.754+0000 7f4c55ffb640 1 -- 172.21.15.7:0/4285753536 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c48007e10 con 0x7f4c58101090 2024-04-21T08:44:55.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.754+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/4285753536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5819dd00 con 0x7f4c58101090 2024-04-21T08:44:55.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.754+0000 7f4c55ffb640 1 -- 172.21.15.7:0/4285753536 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c48018820 con 0x7f4c58101090 2024-04-21T08:44:55.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.754+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/4285753536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c581a5540 con 0x7f4c58101090 2024-04-21T08:44:55.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.756+0000 7f4c55ffb640 1 -- 172.21.15.7:0/4285753536 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f4c4800f450 con 0x7f4c58101090 2024-04-21T08:44:55.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.756+0000 7f4c61d9d740 1 Processor -- start 2024-04-21T08:44:55.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.756+0000 7f4c61d9d740 1 -- start start 2024-04-21T08:44:55.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc6f7b0 0x55930fc6fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc78710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c55ffb640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4c400a3e30 0x7f4c400a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc78ce0 0x55930fc7b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55930fbd3b10 con 0x55930fc6f7b0 2024-04-21T08:44:55.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55930fb71240 con 0x55930fc78ce0 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55930fbcfad0 con 0x55930fc70160 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5d670640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc6f7b0 0x55930fc6fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5ce6f640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4c400a3e30 0x7f4c400a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5d670640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc6f7b0 0x55930fc6fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33186/0 (socket says 172.21.15.7:33186) 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5d670640 1 -- 172.21.15.7:0/1873007258 learned_addr learned my addr 172.21.15.7:0/1873007258 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c57fff640 1 --2- 172.21.15.7:0/1873007258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc78710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c55ffb640 1 -- 172.21.15.7:0/4285753536 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f4c480d0f90 con 0x7f4c58101090 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5d670640 1 --2- 172.21.15.7:0/1873007258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc78ce0 0x55930fc7b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5d670640 1 -- 172.21.15.7:0/1873007258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 msgr2=0x55930fc78710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5d670640 1 --2- 172.21.15.7:0/1873007258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc78710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5d670640 1 -- 172.21.15.7:0/1873007258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc78ce0 msgr2=0x55930fc7b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5d670640 1 --2- 172.21.15.7:0/1873007258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc78ce0 0x55930fc7b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5d670640 1 -- 172.21.15.7:0/1873007258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55930f8e62d0 con 0x55930fc6f7b0 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5d670640 1 --2- 172.21.15.7:0/1873007258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc6f7b0 0x55930fc6fb90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4c50037040 tx=0x7f4c50032340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39fe6597f186e4b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.757+0000 7f4c5ce6f640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4c400a3e30 0x7f4c400a62f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4c480ce230 tx=0x7f4c48087040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c3effd640 1 -- 172.21.15.7:0/1873007258 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50046070 con 0x55930fc6f7b0 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c3effd640 1 -- 172.21.15.7:0/1873007258 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c50032970 con 0x55930fc6f7b0 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c3effd640 1 -- 172.21.15.7:0/1873007258 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50039070 con 0x55930fc6f7b0 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c61d9d740 1 -- 172.21.15.7:0/1873007258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc6f7b0 msgr2=0x55930fc6fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/1873007258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc6f7b0 0x55930fc6fb90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4c50037040 tx=0x7f4c50032340 comp rx=0 tx=0).stop 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c61d9d740 1 -- 172.21.15.7:0/1873007258 shutdown_connections 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/1873007258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc78ce0 0x55930fc7b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/1873007258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc78710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/1873007258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc6f7b0 0x55930fc6fb90 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c61d9d740 1 -- 172.21.15.7:0/1873007258 >> 172.21.15.7:0/1873007258 conn(0x55930fc69a60 msgr2=0x55930fc69e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c61d9d740 1 -- 172.21.15.7:0/1873007258 shutdown_connections 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.758+0000 7f4c61d9d740 1 -- 172.21.15.7:0/1873007258 wait complete. 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c61d9d740 1 -- start start 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c61d9d740 1 Processor -- start 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc6f7b0 0x55930fc75d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc762d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc76810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55930fc7bbf0 con 0x55930fc78ce0 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55930fb5c190 con 0x55930fc6f7b0 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55930fc7bd70 con 0x55930fc70160 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c61d9d740 10 client.0 _ll_register_callbacks cb 0x55930fb578b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c5d670640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc76810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c5d670640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc76810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33196/0 (socket says 172.21.15.7:33196) 2024-04-21T08:44:55.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c5d670640 1 -- 172.21.15.7:0/2455679941 learned_addr learned my addr 172.21.15.7:0/2455679941 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c5d670640 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc6f7b0 0x55930fc75d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.759+0000 7f4c57fff640 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc762d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c5d670640 1 -- 172.21.15.7:0/2455679941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 msgr2=0x55930fc762d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c5d670640 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc762d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c5d670640 1 -- 172.21.15.7:0/2455679941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc6f7b0 msgr2=0x55930fc75d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c5d670640 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc6f7b0 0x55930fc75d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c5d670640 1 -- 172.21.15.7:0/2455679941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55930f9b06b0 con 0x55930fc78ce0 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c5d670640 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc76810 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4c50004000 tx=0x7f4c50003ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfdbc8a5ad386819 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c37fff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c37fff640 1 -- 172.21.15.7:0/2455679941 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50054060 con 0x55930fc78ce0 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c37fff640 1 -- 172.21.15.7:0/2455679941 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f4c50004300 con 0x55930fc78ce0 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2455679941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c50032970 con 0x55930fc78ce0 2024-04-21T08:44:55.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c37fff640 1 -- 172.21.15.7:0/2455679941 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4c5005b050 con 0x55930fc78ce0 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c61d9d740 10 client.15798 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2455679941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55930fc87fa0 con 0x55930fc78ce0 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c37fff640 1 -- 172.21.15.7:0/2455679941 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c50032970 con 0x55930fc78ce0 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.760+0000 7f4c37fff640 1 -- 172.21.15.7:0/2455679941 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f4c5004c070 con 0x55930fc78ce0 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.761+0000 7f4c61d9d740 10 client.15798 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.761+0000 7f4c61d9d740 10 client.15798 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.761+0000 7f4c61d9d740 20 client.15798 populate_metadata read hostname 'smithi007' 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.761+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55930fc892f0 0x55930fc8b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.761+0000 7f4c61d9d740 4 client.15798 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.761+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2455679941 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55930fbe1530 con 0x55930fc892f0 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.761+0000 7f4c57fff640 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55930fc892f0 0x55930fc8b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.761+0000 7f4c57fff640 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55930fc892f0 0x55930fc8b470 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9d9d66971b366f2 server_cookie=d5aec6f5b458d0c9 in_seq=0 out_seq=0 2024-04-21T08:44:55.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.761+0000 7f4c37fff640 10 client.15798 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c37fff640 1 -- 172.21.15.7:0/2455679941 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55930fbe1530 con 0x55930fc892f0 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c37fff640 10 client.15798 handle_command_reply: tid=0 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 1 client.15798 shutdown 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2455679941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 msgr2=0x55930fc76810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc76810 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4c50004000 tx=0x7f4c50003ec0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2455679941 shutdown_connections 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55930fc892f0 0x55930fc8b470 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc76810 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc762d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c37fff640 0 client.15798 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2455679941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc6f7b0 0x55930fc75d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2455679941 >> 172.21.15.7:0/2455679941 conn(0x55930fc69a60 msgr2=0x55930fc6bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2455679941 shutdown_connections 2024-04-21T08:44:55.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2455679941 wait complete. 2024-04-21T08:44:55.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.762+0000 7f4c61d9d740 20 client.15798 trim_cache size 0 max 16384 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.780+0000 7f4c61d9d740 1 Processor -- start 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.780+0000 7f4c61d9d740 1 -- start start 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc6f7b0 0x55930fc784c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc856c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc64800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55930fc897b0 con 0x55930fc78ce0 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55930fc89e80 con 0x55930fc6f7b0 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55930fc8a550 con 0x55930fc70160 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c5d670640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc64800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c5d670640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc64800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33206/0 (socket says 172.21.15.7:33206) 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c5d670640 1 -- 172.21.15.7:0/3780364127 learned_addr learned my addr 172.21.15.7:0/3780364127 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c57fff640 1 --2- 172.21.15.7:0/3780364127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc856c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c5d670640 1 -- 172.21.15.7:0/3780364127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 msgr2=0x55930fc856c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c5d670640 1 --2- 172.21.15.7:0/3780364127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc856c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c5d670640 1 -- 172.21.15.7:0/3780364127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc6f7b0 msgr2=0x55930fc784c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c5d670640 1 --2- 172.21.15.7:0/3780364127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc6f7b0 0x55930fc784c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c5d670640 1 -- 172.21.15.7:0/3780364127 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55930fc74300 con 0x55930fc78ce0 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c57fff640 1 --2- 172.21.15.7:0/3780364127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc856c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c5d670640 1 --2- 172.21.15.7:0/3780364127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc64800 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4c5002c6d0 tx=0x7f4c50009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659c7467eecc8e89 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c3e7fc640 1 -- 172.21.15.7:0/3780364127 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50054040 con 0x55930fc78ce0 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c3e7fc640 1 -- 172.21.15.7:0/3780364127 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c50032970 con 0x55930fc78ce0 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.781+0000 7f4c3e7fc640 1 -- 172.21.15.7:0/3780364127 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50046040 con 0x55930fc78ce0 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.782+0000 7f4c61d9d740 1 -- 172.21.15.7:0/3780364127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 msgr2=0x55930fc64800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.782+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/3780364127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc64800 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4c5002c6d0 tx=0x7f4c50009b20 comp rx=0 tx=0).stop 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.782+0000 7f4c61d9d740 1 -- 172.21.15.7:0/3780364127 shutdown_connections 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.782+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/3780364127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc64800 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.782+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/3780364127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc70160 0x55930fc856c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.782+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/3780364127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc6f7b0 0x55930fc784c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.782+0000 7f4c61d9d740 1 -- 172.21.15.7:0/3780364127 >> 172.21.15.7:0/3780364127 conn(0x55930fc69a60 msgr2=0x55930fc70590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.782+0000 7f4c61d9d740 1 -- 172.21.15.7:0/3780364127 shutdown_connections 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.782+0000 7f4c61d9d740 1 -- 172.21.15.7:0/3780364127 wait complete. 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.782+0000 7f4c61d9d740 1 -- start start 2024-04-21T08:44:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.782+0000 7f4c61d9d740 1 Processor -- start 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc6f7b0 0x55930fc93500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc70160 0x55930fc93a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c61d9d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc8e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55930fc8b2f0 con 0x55930fc78ce0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55930fc8ac20 con 0x55930fc70160 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c61d9d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55930fc8b9c0 con 0x55930fc6f7b0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c61d9d740 10 client.0 _ll_register_callbacks cb 0x7f4c50044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c5d670640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc8e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c5d670640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc8e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33222/0 (socket says 172.21.15.7:33222) 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c5d670640 1 -- 172.21.15.7:0/2376610191 learned_addr learned my addr 172.21.15.7:0/2376610191 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c5d670640 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc6f7b0 0x55930fc93500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c57fff640 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc70160 0x55930fc93a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c5d670640 1 -- 172.21.15.7:0/2376610191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc6f7b0 msgr2=0x55930fc93500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c5d670640 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc6f7b0 0x55930fc93500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c5d670640 1 -- 172.21.15.7:0/2376610191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc70160 msgr2=0x55930fc93a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c5d670640 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc70160 0x55930fc93a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c5d670640 1 -- 172.21.15.7:0/2376610191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55930fc8e850 con 0x55930fc78ce0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c5d670640 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc8e2c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4c50005e00 tx=0x7f4c50036040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66adc38e194de29d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.783+0000 7f4c37fff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c37fff640 1 -- 172.21.15.7:0/2376610191 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c50046070 con 0x55930fc78ce0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c37fff640 1 -- 172.21.15.7:0/2376610191 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f4c50041460 con 0x55930fc78ce0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2376610191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c50032970 con 0x55930fc78ce0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c37fff640 1 -- 172.21.15.7:0/2376610191 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4c50039040 con 0x55930fc78ce0 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c61d9d740 10 client.15804 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:55.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2376610191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55930fcf07a0 con 0x55930fc78ce0 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c37fff640 1 -- 172.21.15.7:0/2376610191 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c50032970 con 0x55930fc78ce0 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c37fff640 1 -- 172.21.15.7:0/2376610191 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f4c5004c040 con 0x55930fc78ce0 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c61d9d740 10 client.15804 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c61d9d740 10 client.15804 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c61d9d740 20 client.15804 populate_metadata read hostname 'smithi007' 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55930fcf4c40 0x55930fcf3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c61d9d740 4 client.15804 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2376610191 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "status"}) -- 0x55930fbdb3a0 con 0x55930fcf4c40 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.784+0000 7f4c57fff640 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55930fcf4c40 0x55930fcf3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c57fff640 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55930fcf4c40 0x55930fcf3950 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f43dcf563b139d7 server_cookie=8dba0469a8099fea in_seq=0 out_seq=0 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c37fff640 10 client.15804 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c37fff640 1 -- 172.21.15.7:0/2376610191 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55930fbdb3a0 con 0x55930fcf4c40 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c37fff640 10 client.15804 handle_command_reply: tid=0 2024-04-21T08:44:55.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 1 client.15804 shutdown 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2376610191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 msgr2=0x55930fc8e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc8e2c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4c50005e00 tx=0x7f4c50036040 comp rx=0 tx=0).stop 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2376610191 shutdown_connections 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55930fcf4c40 0x55930fcf3950 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55930fc78ce0 0x55930fc8e2c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55930fc70160 0x55930fc93a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c37fff640 0 client.15804 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 1 --2- 172.21.15.7:0/2376610191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55930fc6f7b0 0x55930fc93500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2376610191 >> 172.21.15.7:0/2376610191 conn(0x55930fc69a60 msgr2=0x55930fc64800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2376610191 shutdown_connections 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 1 -- 172.21.15.7:0/2376610191 wait complete. 2024-04-21T08:44:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.785+0000 7f4c61d9d740 20 client.15804 trim_cache size 0 max 16384 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "c9c60282-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24533, 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 158.161167256, 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 21, 2024-04-21T08:44:55.793 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 95, 2024-04-21T08:44:55.794 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T08:44:55.794 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 164.30265120300001 2024-04-21T08:44:55.794 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:44:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/4285753536 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4c400a3e30 msgr2=0x7f4c400a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4c400a3e30 0x7f4c400a62f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4c480ce230 tx=0x7f4c48087040 comp rx=0 tx=0).stop 2024-04-21T08:44:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/4285753536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c58101090 msgr2=0x7f4c58198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c58101090 0x7f4c58198660 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800c920 tx=0x7f4c4800cde0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/4285753536 shutdown_connections 2024-04-21T08:44:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4c400a3e30 0x7f4c400a62f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4c5819cf20 0x7f4c5819d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c58106160 0x7f4c58198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 --2- 172.21.15.7:0/4285753536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c58101090 0x7f4c58198660 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/4285753536 >> 172.21.15.7:0/4285753536 conn(0x7f4c58078000 msgr2=0x7f4c58100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/4285753536 shutdown_connections 2024-04-21T08:44:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:44:55.786+0000 7f4c5f0d3640 1 -- 172.21.15.7:0/4285753536 wait complete. 2024-04-21T08:44:55.797 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 120 sec 2024-04-21T08:44:55.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:55 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1275308048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:55.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:55 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2721100141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:55.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:55 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4099683063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:56.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:55 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1275308048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:56.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:55 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2721100141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:56.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:55 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4099683063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:56.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:55 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1275308048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:56.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:55 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2721100141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:56.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:55 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4099683063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:57.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:56 smithi007 ceph-mon[28477]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:44:57.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:56 smithi016 ceph-mon[31259]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:44:57.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:56 smithi077 ceph-mon[31615]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:44:59.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:44:58 smithi007 ceph-mon[28477]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:44:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:44:58 smithi016 ceph-mon[31259]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:44:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:44:58 smithi077 ceph-mon[31615]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:45:00.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:00 smithi007 ceph-mon[28477]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 50 KiB/s wr, 32 op/s 2024-04-21T08:45:00.902 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:45:00 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:45:00.626+0000 7f6dc92c5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:00.902 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:45:00 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:45:00.626+0000 7f6dc92c5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:00.902 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:45:00 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:45:00.626+0000 7f6dc92c5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:00.902 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:45:00 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:45:00.626+0000 7f6dc92c5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:00.902 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:45:00 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x[28711]: 2024-04-21T08:45:00.626+0000 7f6dc92c5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:01.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:00 smithi016 ceph-mon[31259]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 50 KiB/s wr, 32 op/s 2024-04-21T08:45:01.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:00 smithi077 ceph-mon[31615]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 50 KiB/s wr, 32 op/s 2024-04-21T08:45:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.528+0000 7fb4972a5640 1 -- 172.21.15.7:0/1261252658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb490101a50 msgr2=0x7fb490105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.528+0000 7fb4972a5640 1 --2- 172.21.15.7:0/1261252658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb490101a50 0x7fb490105a90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb480009900 tx=0x7fb48002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.529+0000 7fb4972a5640 1 -- 172.21.15.7:0/1261252658 shutdown_connections 2024-04-21T08:45:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.529+0000 7fb4972a5640 1 --2- 172.21.15.7:0/1261252658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb490106170 0x7fb49010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.529+0000 7fb4972a5640 1 --2- 172.21.15.7:0/1261252658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb490101a50 0x7fb490105a90 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.529+0000 7fb4972a5640 1 --2- 172.21.15.7:0/1261252658 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4901010a0 0x7fb490101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.529+0000 7fb4972a5640 1 -- 172.21.15.7:0/1261252658 >> 172.21.15.7:0/1261252658 conn(0x7fb490077f70 msgr2=0x7fb490078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.529+0000 7fb4972a5640 1 -- 172.21.15.7:0/1261252658 shutdown_connections 2024-04-21T08:45:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.529+0000 7fb4972a5640 1 -- 172.21.15.7:0/1261252658 wait complete. 2024-04-21T08:45:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.529+0000 7fb4972a5640 1 Processor -- start 2024-04-21T08:45:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb4972a5640 1 -- start start 2024-04-21T08:45:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb4972a5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901010a0 0x7fb490198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb4972a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb490106170 0x7fb490198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb4972a5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb49019d000 0x7fb49019d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb4972a5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb490111090 con 0x7fb49019d000 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb4972a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb490110f10 con 0x7fb490106170 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb4972a5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb490111210 con 0x7fb4901010a0 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb495842640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb49019d000 0x7fb49019d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb495842640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb49019d000 0x7fb49019d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33240/0 (socket says 172.21.15.7:33240) 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb495842640 1 -- 172.21.15.7:0/1766679288 learned_addr learned my addr 172.21.15.7:0/1766679288 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb495041640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901010a0 0x7fb490198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb494840640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb490106170 0x7fb490198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb495842640 1 -- 172.21.15.7:0/1766679288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901010a0 msgr2=0x7fb490198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb495842640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901010a0 0x7fb490198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb495842640 1 -- 172.21.15.7:0/1766679288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb490106170 msgr2=0x7fb490198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb495842640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb490106170 0x7fb490198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.530+0000 7fb495842640 1 -- 172.21.15.7:0/1766679288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49019daf0 con 0x7fb49019d000 2024-04-21T08:45:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.531+0000 7fb495842640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb49019d000 0x7fb49019d4b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb48800eaa0 tx=0x7fb48800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.531+0000 7fb4867fc640 1 -- 172.21.15.7:0/1766679288 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48800cdb0 con 0x7fb49019d000 2024-04-21T08:45:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.531+0000 7fb4867fc640 1 -- 172.21.15.7:0/1766679288 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb488005020 con 0x7fb49019d000 2024-04-21T08:45:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.531+0000 7fb4972a5640 1 -- 172.21.15.7:0/1766679288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb49019dde0 con 0x7fb49019d000 2024-04-21T08:45:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.531+0000 7fb4867fc640 1 -- 172.21.15.7:0/1766679288 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4880116d0 con 0x7fb49019d000 2024-04-21T08:45:01.533 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.531+0000 7fb4972a5640 1 -- 172.21.15.7:0/1766679288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4901a5550 con 0x7fb49019d000 2024-04-21T08:45:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.533+0000 7fb4867fc640 1 -- 172.21.15.7:0/1766679288 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7fb4880051d0 con 0x7fb49019d000 2024-04-21T08:45:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb499f72740 1 Processor -- start 2024-04-21T08:45:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb499f72740 1 -- start start 2024-04-21T08:45:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb499f72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbd3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb499f72740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbd3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb499f72740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb499f72740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55c0bfb388d0 con 0x55c0bfbe14f0 2024-04-21T08:45:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb499f72740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c0bfad5e80 con 0x55c0bfbd48d0 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb499f72740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c0bfb34890 con 0x55c0bfbd3b60 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb494840640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb4867fc640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb47c0a3da0 0x7fb47c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb494840640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33254/0 (socket says 172.21.15.7:33254) 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb494840640 1 -- 172.21.15.7:0/175120696 learned_addr learned my addr 172.21.15.7:0/175120696 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb494840640 1 --2- 172.21.15.7:0/175120696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbd3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb495842640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb47c0a3da0 0x7fb47c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb495041640 1 --2- 172.21.15.7:0/175120696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbd3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.534+0000 7fb4867fc640 1 -- 172.21.15.7:0/1766679288 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fb4880d0e80 con 0x7fb49019d000 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.535+0000 7fb494840640 1 -- 172.21.15.7:0/175120696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 msgr2=0x55c0bfbd3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.535+0000 7fb494840640 1 --2- 172.21.15.7:0/175120696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbd3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.535+0000 7fb494840640 1 -- 172.21.15.7:0/175120696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd48d0 msgr2=0x55c0bfbd3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:01.542 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.535+0000 7fb494840640 1 --2- 172.21.15.7:0/175120696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbd3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.542 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.535+0000 7fb494840640 1 -- 172.21.15.7:0/175120696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0bfaf7320 con 0x55c0bfbe14f0 2024-04-21T08:45:01.542 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.535+0000 7fb494840640 1 --2- 172.21.15.7:0/175120696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe38e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb4800096c0 tx=0x7fb480005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0fb89551fff986c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:01.542 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.535+0000 7fb495842640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb47c0a3da0 0x7fb47c0a6260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb4880ce180 tx=0x7fb488087040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:01.542 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.535+0000 7fb47b7fe640 1 -- 172.21.15.7:0/175120696 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb480047020 con 0x55c0bfbe14f0 2024-04-21T08:45:01.542 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.535+0000 7fb47b7fe640 1 -- 172.21.15.7:0/175120696 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb48002cc90 con 0x55c0bfbe14f0 2024-04-21T08:45:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.535+0000 7fb47b7fe640 1 -- 172.21.15.7:0/175120696 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb480004920 con 0x55c0bfbe14f0 2024-04-21T08:45:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.536+0000 7fb499f72740 1 -- 172.21.15.7:0/175120696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 msgr2=0x55c0bfbe38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.536+0000 7fb499f72740 1 --2- 172.21.15.7:0/175120696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe38e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb4800096c0 tx=0x7fb480005e40 comp rx=0 tx=0).stop 2024-04-21T08:45:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.536+0000 7fb499f72740 1 -- 172.21.15.7:0/175120696 shutdown_connections 2024-04-21T08:45:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.536+0000 7fb499f72740 1 --2- 172.21.15.7:0/175120696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe38e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.536+0000 7fb499f72740 1 --2- 172.21.15.7:0/175120696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbd3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.536+0000 7fb499f72740 1 --2- 172.21.15.7:0/175120696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbd3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.536+0000 7fb499f72740 1 -- 172.21.15.7:0/175120696 >> 172.21.15.7:0/175120696 conn(0x55c0bfb9ac90 msgr2=0x55c0bfbce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.536+0000 7fb499f72740 1 -- 172.21.15.7:0/175120696 shutdown_connections 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.536+0000 7fb499f72740 1 -- 172.21.15.7:0/175120696 wait complete. 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.536+0000 7fb499f72740 1 -- start start 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.536+0000 7fb499f72740 1 Processor -- start 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb499f72740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbe0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb499f72740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbde2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb499f72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbde8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb499f72740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55c0bfbe41c0 con 0x55c0bfbd48d0 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb499f72740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c0bfac0eb0 con 0x55c0bfbe14f0 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb499f72740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c0bfbe4340 con 0x55c0bfbd3b60 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb499f72740 10 client.0 _ll_register_callbacks cb 0x55c0bfaedb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb495041640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbde2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb495041640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbde2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33264/0 (socket says 172.21.15.7:33264) 2024-04-21T08:45:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb495041640 1 -- 172.21.15.7:0/764679931 learned_addr learned my addr 172.21.15.7:0/764679931 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb494840640 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbe0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb494840640 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbde8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb495041640 1 -- 172.21.15.7:0/764679931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 msgr2=0x55c0bfbe0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb495041640 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbe0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb495041640 1 -- 172.21.15.7:0/764679931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbe14f0 msgr2=0x55c0bfbde8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb495041640 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbde8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.537+0000 7fb495041640 1 -- 172.21.15.7:0/764679931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0bf84b210 con 0x55c0bfbd48d0 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb495041640 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbde2a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb48c010820 tx=0x7fb48c010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44eb33a2476a201e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb478ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb478ff9640 1 -- 172.21.15.7:0/764679931 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c00fb90 con 0x55c0bfbd48d0 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb478ff9640 1 -- 172.21.15.7:0/764679931 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fb48c00fd30 con 0x55c0bfbd48d0 2024-04-21T08:45:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb499f72740 1 -- 172.21.15.7:0/764679931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb48002cc90 con 0x55c0bfbd48d0 2024-04-21T08:45:01.546 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb478ff9640 1 -- 172.21.15.7:0/764679931 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb48c024050 con 0x55c0bfbd48d0 2024-04-21T08:45:01.546 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb499f72740 10 client.15822 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:01.546 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb499f72740 1 -- 172.21.15.7:0/764679931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c0bf917620 con 0x55c0bfbd48d0 2024-04-21T08:45:01.546 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb478ff9640 1 -- 172.21.15.7:0/764679931 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb48002cc90 con 0x55c0bfbd48d0 2024-04-21T08:45:01.546 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb478ff9640 1 -- 172.21.15.7:0/764679931 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb48c00e540 con 0x55c0bfbd48d0 2024-04-21T08:45:01.546 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb499f72740 10 client.15822 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:01.546 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb499f72740 10 client.15822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:01.546 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb499f72740 20 client.15822 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:01.546 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb499f72740 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55c0bfbf1670 0x55c0bfbf3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb499f72740 4 client.15822 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.538+0000 7fb499f72740 1 -- 172.21.15.7:0/764679931 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0bfb4c880 con 0x55c0bfbf1670 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.539+0000 7fb494840640 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55c0bfbf1670 0x55c0bfbf3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.539+0000 7fb494840640 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55c0bfbf1670 0x55c0bfbf3a80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7558295608a4b578 server_cookie=80d1838072df219 in_seq=0 out_seq=0 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.539+0000 7fb478ff9640 10 client.15822 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb478ff9640 1 -- 172.21.15.7:0/764679931 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0bfb4c880 con 0x55c0bfbf1670 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb478ff9640 10 client.15822 handle_command_reply: tid=0 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 1 client.15822 shutdown 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 1 -- 172.21.15.7:0/764679931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbd48d0 msgr2=0x55c0bfbde2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbde2a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb48c010820 tx=0x7fb48c010ce0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 1 -- 172.21.15.7:0/764679931 shutdown_connections 2024-04-21T08:45:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55c0bfbf1670 0x55c0bfbf3a80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbde8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbde2a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb478ff9640 0 client.15822 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 1 --2- 172.21.15.7:0/764679931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbe0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 1 -- 172.21.15.7:0/764679931 >> 172.21.15.7:0/764679931 conn(0x55c0bfb9ac90 msgr2=0x55c0bfbd05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 1 -- 172.21.15.7:0/764679931 shutdown_connections 2024-04-21T08:45:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 1 -- 172.21.15.7:0/764679931 wait complete. 2024-04-21T08:45:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.540+0000 7fb499f72740 20 client.15822 trim_cache size 0 max 16384 2024-04-21T08:45:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.555+0000 7fb499f72740 1 Processor -- start 2024-04-21T08:45:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.555+0000 7fb499f72740 1 -- start start 2024-04-21T08:45:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.555+0000 7fb499f72740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbed0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.555+0000 7fb499f72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbe09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.555+0000 7fb499f72740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.555+0000 7fb499f72740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb4901ab2b0 con 0x55c0bfbe14f0 2024-04-21T08:45:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.555+0000 7fb499f72740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4901adb30 con 0x55c0bfbd48d0 2024-04-21T08:45:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.555+0000 7fb499f72740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb4901ab920 con 0x55c0bfbd3b60 2024-04-21T08:45:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.555+0000 7fb494840640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb494840640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33272/0 (socket says 172.21.15.7:33272) 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb494840640 1 -- 172.21.15.7:0/751820118 learned_addr learned my addr 172.21.15.7:0/751820118 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb494840640 1 --2- 172.21.15.7:0/751820118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbed0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb495041640 1 --2- 172.21.15.7:0/751820118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbe09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb494840640 1 -- 172.21.15.7:0/751820118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 msgr2=0x55c0bfbed0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb494840640 1 --2- 172.21.15.7:0/751820118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbed0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb494840640 1 -- 172.21.15.7:0/751820118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd48d0 msgr2=0x55c0bfbe09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb494840640 1 --2- 172.21.15.7:0/751820118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbe09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb494840640 1 -- 172.21.15.7:0/751820118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0bfbd2e30 con 0x55c0bfbe14f0 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb494840640 1 --2- 172.21.15.7:0/751820118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe0f10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x55c0bfbd4600 tx=0x7fb480005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f61be20b68d5163 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb47affd640 1 -- 172.21.15.7:0/751820118 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb480047040 con 0x55c0bfbe14f0 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb47affd640 1 -- 172.21.15.7:0/751820118 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb48002ca60 con 0x55c0bfbe14f0 2024-04-21T08:45:01.557 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.556+0000 7fb47affd640 1 -- 172.21.15.7:0/751820118 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb480003950 con 0x55c0bfbe14f0 2024-04-21T08:45:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.557+0000 7fb499f72740 1 -- 172.21.15.7:0/751820118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 msgr2=0x55c0bfbe0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.557+0000 7fb499f72740 1 --2- 172.21.15.7:0/751820118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe0f10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x55c0bfbd4600 tx=0x7fb480005e40 comp rx=0 tx=0).stop 2024-04-21T08:45:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.557+0000 7fb499f72740 1 -- 172.21.15.7:0/751820118 shutdown_connections 2024-04-21T08:45:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.557+0000 7fb499f72740 1 --2- 172.21.15.7:0/751820118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe0f10 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.557+0000 7fb499f72740 1 --2- 172.21.15.7:0/751820118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbe09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.561 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.557+0000 7fb499f72740 1 --2- 172.21.15.7:0/751820118 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbed0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.557+0000 7fb499f72740 1 -- 172.21.15.7:0/751820118 >> 172.21.15.7:0/751820118 conn(0x55c0bfb9ac90 msgr2=0x55c0bfbce380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.557+0000 7fb499f72740 1 -- 172.21.15.7:0/751820118 shutdown_connections 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.557+0000 7fb499f72740 1 -- 172.21.15.7:0/751820118 wait complete. 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.557+0000 7fb499f72740 1 -- start start 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb499f72740 1 Processor -- start 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb499f72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbe98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb499f72740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbe1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb499f72740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb499f72740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb4901abff0 con 0x55c0bfbe14f0 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb499f72740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4901ad460 con 0x55c0bfbd3b60 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb499f72740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb4901ac6c0 con 0x55c0bfbd48d0 2024-04-21T08:45:01.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb499f72740 10 client.0 _ll_register_callbacks cb 0x7fb48c00a700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb494840640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb494840640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33286/0 (socket says 172.21.15.7:33286) 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb494840640 1 -- 172.21.15.7:0/4210068611 learned_addr learned my addr 172.21.15.7:0/4210068611 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb494840640 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbe98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb495041640 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbe1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb494840640 1 -- 172.21.15.7:0/4210068611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd48d0 msgr2=0x55c0bfbe1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb494840640 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbe1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb494840640 1 -- 172.21.15.7:0/4210068611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd3b60 msgr2=0x55c0bfbe98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb494840640 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbe98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.558+0000 7fb494840640 1 -- 172.21.15.7:0/4210068611 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0bfbe2b20 con 0x55c0bfbe14f0 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb494840640 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe2430 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb48002ce00 tx=0x7fb48005c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615f7895004ec632 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb478ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:01.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb478ff9640 1 -- 172.21.15.7:0/4210068611 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb480047070 con 0x55c0bfbe14f0 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb499f72740 1 -- 172.21.15.7:0/4210068611 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb48002ca60 con 0x55c0bfbe14f0 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb478ff9640 1 -- 172.21.15.7:0/4210068611 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fb48002c810 con 0x55c0bfbe14f0 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb478ff9640 1 -- 172.21.15.7:0/4210068611 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb480059150 con 0x55c0bfbe14f0 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb499f72740 10 client.15828 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb499f72740 1 -- 172.21.15.7:0/4210068611 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c0bfc57c00 con 0x55c0bfbe14f0 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb478ff9640 1 -- 172.21.15.7:0/4210068611 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb48002ca60 con 0x55c0bfbe14f0 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb478ff9640 1 -- 172.21.15.7:0/4210068611 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb4800039f0 con 0x55c0bfbe14f0 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb499f72740 10 client.15828 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb499f72740 10 client.15828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb499f72740 20 client.15828 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.559+0000 7fb499f72740 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55c0bfc57e90 0x55c0bfc5a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.560+0000 7fb499f72740 4 client.15828 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.560+0000 7fb499f72740 1 -- 172.21.15.7:0/4210068611 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c0bfad83d0 con 0x55c0bfc57e90 2024-04-21T08:45:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.560+0000 7fb495041640 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55c0bfc57e90 0x55c0bfc5a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.560+0000 7fb495041640 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55c0bfc57e90 0x55c0bfc5a280 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=390812001d8c3372 server_cookie=65d85da3a8776c79 in_seq=0 out_seq=0 2024-04-21T08:45:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.560+0000 7fb478ff9640 10 client.15828 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb478ff9640 1 -- 172.21.15.7:0/4210068611 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c0bfad83d0 con 0x55c0bfc57e90 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb478ff9640 10 client.15828 handle_command_reply: tid=0 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 1 client.15828 shutdown 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 1 -- 172.21.15.7:0/4210068611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 msgr2=0x55c0bfbe2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe2430 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb48002ce00 tx=0x7fb48005c040 comp rx=0 tx=0).stop 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 1 -- 172.21.15.7:0/4210068611 shutdown_connections 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55c0bfc57e90 0x55c0bfc5a280 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c0bfbe14f0 0x55c0bfbe2430 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb478ff9640 0 client.15828 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c0bfbd48d0 0x55c0bfbe1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 1 --2- 172.21.15.7:0/4210068611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c0bfbd3b60 0x55c0bfbe98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 1 -- 172.21.15.7:0/4210068611 >> 172.21.15.7:0/4210068611 conn(0x55c0bfb9ac90 msgr2=0x55c0bfbddb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 1 -- 172.21.15.7:0/4210068611 shutdown_connections 2024-04-21T08:45:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 1 -- 172.21.15.7:0/4210068611 wait complete. 2024-04-21T08:45:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.573+0000 7fb499f72740 20 client.15828 trim_cache size 0 max 16384 2024-04-21T08:45:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:45:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:45:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:45:01.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:45:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 -- 172.21.15.7:0/1766679288 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb47c0a3da0 msgr2=0x7fb47c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb47c0a3da0 0x7fb47c0a6260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb4880ce180 tx=0x7fb488087040 comp rx=0 tx=0).stop 2024-04-21T08:45:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 -- 172.21.15.7:0/1766679288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb49019d000 msgr2=0x7fb49019d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb49019d000 0x7fb49019d4b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb48800eaa0 tx=0x7fb48800ef60 comp rx=0 tx=0).stop 2024-04-21T08:45:01.575 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 -- 172.21.15.7:0/1766679288 shutdown_connections 2024-04-21T08:45:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb49019d000 0x7fb49019d4b0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb490106170 0x7fb490198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb47c0a3da0 0x7fb47c0a6260 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 --2- 172.21.15.7:0/1766679288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4901010a0 0x7fb490198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 -- 172.21.15.7:0/1766679288 >> 172.21.15.7:0/1766679288 conn(0x7fb490077f70 msgr2=0x7fb490104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 -- 172.21.15.7:0/1766679288 shutdown_connections 2024-04-21T08:45:01.576 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:01.574+0000 7fb4972a5640 1 -- 172.21.15.7:0/1766679288 wait complete. 2024-04-21T08:45:03.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:02 smithi007 ceph-mon[28477]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:45:03.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:02 smithi007 ceph-mon[28477]: mgrmap e25: x(active, since 20s), standbys: y, z 2024-04-21T08:45:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:02 smithi016 ceph-mon[31259]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:45:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:02 smithi016 ceph-mon[31259]: mgrmap e25: x(active, since 20s), standbys: y, z 2024-04-21T08:45:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:02 smithi077 ceph-mon[31615]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:45:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:02 smithi077 ceph-mon[31615]: mgrmap e25: x(active, since 20s), standbys: y, z 2024-04-21T08:45:05.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:04 smithi007 ceph-mon[28477]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 47 KiB/s wr, 39 op/s 2024-04-21T08:45:05.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:04 smithi016 ceph-mon[31259]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 47 KiB/s wr, 39 op/s 2024-04-21T08:45:05.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:04 smithi077 ceph-mon[31615]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 47 KiB/s wr, 39 op/s 2024-04-21T08:45:07.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:06 smithi007 ceph-mon[28477]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-04-21T08:45:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:06 smithi016 ceph-mon[31259]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-04-21T08:45:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:06 smithi077 ceph-mon[31615]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-04-21T08:45:09.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:08 smithi007 ceph-mon[28477]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-04-21T08:45:09.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:08 smithi016 ceph-mon[31259]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-04-21T08:45:09.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:08 smithi077 ceph-mon[31615]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-04-21T08:45:11.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:10 smithi007 ceph-mon[28477]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-21T08:45:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:10 smithi016 ceph-mon[31259]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-21T08:45:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:10 smithi077 ceph-mon[31615]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-21T08:45:13.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:12 smithi007 ceph-mon[28477]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-21T08:45:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:12 smithi016 ceph-mon[31259]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-21T08:45:13.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:12 smithi077 ceph-mon[31615]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-21T08:45:14.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.714+0000 7fdba732b640 1 -- 172.21.15.7:0/4221874360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdba0102b80 msgr2=0x7fdba0106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.714+0000 7fdba732b640 1 --2- 172.21.15.7:0/4221874360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdba0102b80 0x7fdba0106bc0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdb90009900 tx=0x7fdb9002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:14.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.715+0000 7fdba732b640 1 -- 172.21.15.7:0/4221874360 shutdown_connections 2024-04-21T08:45:14.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.715+0000 7fdba732b640 1 --2- 172.21.15.7:0/4221874360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdba01072a0 0x7fdba010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.715+0000 7fdba732b640 1 --2- 172.21.15.7:0/4221874360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdba0102b80 0x7fdba0106bc0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.715+0000 7fdba732b640 1 --2- 172.21.15.7:0/4221874360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdba01021d0 0x7fdba01025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.715+0000 7fdba732b640 1 -- 172.21.15.7:0/4221874360 >> 172.21.15.7:0/4221874360 conn(0x7fdba00fda30 msgr2=0x7fdba00ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.715+0000 7fdba732b640 1 -- 172.21.15.7:0/4221874360 shutdown_connections 2024-04-21T08:45:14.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.715+0000 7fdba732b640 1 -- 172.21.15.7:0/4221874360 wait complete. 2024-04-21T08:45:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba732b640 1 Processor -- start 2024-04-21T08:45:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba732b640 1 -- start start 2024-04-21T08:45:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba732b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdba01021d0 0x7fdba019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba732b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdba01072a0 0x7fdba019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba732b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdba01985b0 0x7fdba0198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba48c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdba01072a0 0x7fdba019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba48c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdba01072a0 0x7fdba019e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55450/0 (socket says 172.21.15.7:55450) 2024-04-21T08:45:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba48c6640 1 -- 172.21.15.7:0/2168759048 learned_addr learned my addr 172.21.15.7:0/2168759048 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba50c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdba01021d0 0x7fdba019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba732b640 1 -- 172.21.15.7:0/2168759048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdba0112250 con 0x7fdba01072a0 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba732b640 1 -- 172.21.15.7:0/2168759048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdba01120d0 con 0x7fdba01021d0 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.716+0000 7fdba732b640 1 -- 172.21.15.7:0/2168759048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdba01123d0 con 0x7fdba01985b0 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdba58c8640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdba01985b0 0x7fdba0198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdba48c6640 1 -- 172.21.15.7:0/2168759048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdba01985b0 msgr2=0x7fdba0198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdba48c6640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdba01985b0 0x7fdba0198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdba48c6640 1 -- 172.21.15.7:0/2168759048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdba01021d0 msgr2=0x7fdba019e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdba48c6640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdba01021d0 0x7fdba019e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdba48c6640 1 -- 172.21.15.7:0/2168759048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba01990d0 con 0x7fdba01072a0 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdba48c6640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdba01072a0 0x7fdba019e940 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdb90000c00 tx=0x7fdb9002c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdb967fc640 1 -- 172.21.15.7:0/2168759048 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb90043020 con 0x7fdba01072a0 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdba732b640 1 -- 172.21.15.7:0/2168759048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba0199360 con 0x7fdba01072a0 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdba732b640 1 -- 172.21.15.7:0/2168759048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba00744c0 con 0x7fdba01072a0 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdb967fc640 1 -- 172.21.15.7:0/2168759048 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb9002caf0 con 0x7fdba01072a0 2024-04-21T08:45:14.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.717+0000 7fdb967fc640 1 -- 172.21.15.7:0/2168759048 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb90003b20 con 0x7fdba01072a0 2024-04-21T08:45:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.721+0000 7fdb967fc640 1 -- 172.21.15.7:0/2168759048 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7fdb90046020 con 0x7fdba01072a0 2024-04-21T08:45:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.721+0000 7fdba9ffb740 1 Processor -- start 2024-04-21T08:45:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.721+0000 7fdba9ffb740 1 -- start start 2024-04-21T08:45:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.721+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 0x560e5689c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.721+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 0x560e5689cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.721+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e568a60e0 0x560e568a84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.721+0000 7fdba9ffb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x560e568018d0 con 0x560e568a60e0 2024-04-21T08:45:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba9ffb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560e5679ee80 con 0x560e5689d8d0 2024-04-21T08:45:14.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba9ffb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560e567fd890 con 0x560e5689cb60 2024-04-21T08:45:14.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba48c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e568a60e0 0x560e568a84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba48c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e568a60e0 0x560e568a84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55460/0 (socket says 172.21.15.7:55460) 2024-04-21T08:45:14.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba48c6640 1 -- 172.21.15.7:0/1144510171 learned_addr learned my addr 172.21.15.7:0/1144510171 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:14.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba58c8640 1 --2- 172.21.15.7:0/1144510171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 0x560e5689c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba50c7640 1 --2- 172.21.15.7:0/1144510171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 0x560e5689cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba48c6640 1 -- 172.21.15.7:0/1144510171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 msgr2=0x560e5689cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba48c6640 1 --2- 172.21.15.7:0/1144510171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 0x560e5689cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba48c6640 1 -- 172.21.15.7:0/1144510171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 msgr2=0x560e5689c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba48c6640 1 --2- 172.21.15.7:0/1144510171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 0x560e5689c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdba48c6640 1 -- 172.21.15.7:0/1144510171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e567c0320 con 0x560e568a60e0 2024-04-21T08:45:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.722+0000 7fdb967fc640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdb8c0a3e30 0x7fdb8c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.723+0000 7fdba58c8640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdb8c0a3e30 0x7fdb8c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.723+0000 7fdba48c6640 1 --2- 172.21.15.7:0/1144510171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e568a60e0 0x560e568a84d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fdb900f00f0 tx=0x7fdb900f6e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a903314f7d4ed28 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.723+0000 7fdb8b7fe640 1 -- 172.21.15.7:0/1144510171 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb900be530 con 0x560e568a60e0 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.723+0000 7fdb967fc640 1 -- 172.21.15.7:0/2168759048 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fdb900f2cf0 con 0x7fdba01072a0 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.723+0000 7fdb8b7fe640 1 -- 172.21.15.7:0/1144510171 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb90040e70 con 0x560e568a60e0 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.723+0000 7fdb8b7fe640 1 -- 172.21.15.7:0/1144510171 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb90078ce0 con 0x560e568a60e0 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.724+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1144510171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e568a60e0 msgr2=0x560e568a84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.724+0000 7fdba58c8640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdb8c0a3e30 0x7fdb8c0a62f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdb98005fd0 tx=0x7fdb98009500 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.724+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/1144510171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e568a60e0 0x560e568a84d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fdb900f00f0 tx=0x7fdb900f6e30 comp rx=0 tx=0).stop 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.724+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1144510171 shutdown_connections 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.724+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/1144510171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e568a60e0 0x560e568a84d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.724+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/1144510171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 0x560e5689cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.724+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/1144510171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 0x560e5689c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.724+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1144510171 >> 172.21.15.7:0/1144510171 conn(0x560e56863c90 msgr2=0x560e56897850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.724+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1144510171 shutdown_connections 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.724+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1144510171 wait complete. 2024-04-21T08:45:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.725+0000 7fdba9ffb740 1 -- start start 2024-04-21T08:45:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.725+0000 7fdba9ffb740 1 Processor -- start 2024-04-21T08:45:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.725+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.725+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 0x560e568a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.725+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e568a60e0 0x560e568a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba50c7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba50c7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568a59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55476/0 (socket says 172.21.15.7:55476) 2024-04-21T08:45:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba48c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 0x560e568a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba48c6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 0x560e568a2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33232/0 (socket says 172.21.15.7:33232) 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba50c7640 1 -- 172.21.15.7:0/2533746691 learned_addr learned my addr 172.21.15.7:0/2533746691 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba9ffb740 1 -- 172.21.15.7:0/2533746691 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x560e568a8bd0 con 0x560e5689cb60 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba9ffb740 1 -- 172.21.15.7:0/2533746691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560e56789eb0 con 0x560e5689d8d0 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba9ffb740 1 -- 172.21.15.7:0/2533746691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560e568a8d50 con 0x560e568a60e0 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba58c8640 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e568a60e0 0x560e568a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba9ffb740 10 client.0 _ll_register_callbacks cb 0x560e567b6b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba48c6640 1 -- 172.21.15.7:0/2533746691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e568a60e0 msgr2=0x560e568a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba48c6640 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e568a60e0 0x560e568a32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba48c6640 1 -- 172.21.15.7:0/2533746691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 msgr2=0x560e568a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba48c6640 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba48c6640 1 -- 172.21.15.7:0/2533746691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e56514210 con 0x560e5689d8d0 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba50c7640 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba9ffb740 1 -- 172.21.15.7:0/2533746691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560e56815880 con 0x560e5689d8d0 2024-04-21T08:45:14.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdba48c6640 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 0x560e568a2d60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb900036d0 tx=0x7fdb90003f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce959a3cc52eef61 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.726+0000 7fdb88ff9640 10 client.25439 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdb88ff9640 1 -- 172.21.15.7:0/2533746691 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb90040cd0 con 0x560e5689d8d0 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdb88ff9640 1 -- 172.21.15.7:0/2533746691 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fdb900a90d0 con 0x560e5689d8d0 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdb88ff9640 1 -- 172.21.15.7:0/2533746691 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdb9003f400 con 0x560e5689d8d0 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdba9ffb740 10 client.25439 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdba9ffb740 1 -- 172.21.15.7:0/2533746691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=21}) -- 0x560e565e0620 con 0x560e5689d8d0 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdb88ff9640 1 -- 172.21.15.7:0/2533746691 <== mon.2 v2:172.21.15.16:3300/0 4 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdb900f6350 con 0x560e5689d8d0 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdba9ffb740 10 client.25439 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdba9ffb740 10 client.25439 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdba9ffb740 20 client.25439 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560e568b59b0 0x560e568b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdba9ffb740 4 client.25439 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.727+0000 7fdba9ffb740 1 -- 172.21.15.7:0/2533746691 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e567a13d0 con 0x560e568b59b0 2024-04-21T08:45:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.728+0000 7fdba50c7640 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560e568b59b0 0x560e568b7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.728+0000 7fdba50c7640 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560e568b59b0 0x560e568b7da0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42f77b3e7f9be6db server_cookie=b1f154caee292765 in_seq=0 out_seq=0 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.728+0000 7fdb88ff9640 10 client.25439 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdb88ff9640 1 -- 172.21.15.7:0/2533746691 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e567a13d0 con 0x560e568b59b0 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdb88ff9640 10 client.25439 handle_command_reply: tid=0 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdba9ffb740 1 client.25439 shutdown 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdba9ffb740 1 -- 172.21.15.7:0/2533746691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 msgr2=0x560e568a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 0x560e568a2d60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb900036d0 tx=0x7fdb90003f70 comp rx=0 tx=0).stop 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdba9ffb740 1 -- 172.21.15.7:0/2533746691 shutdown_connections 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560e568b59b0 0x560e568b7da0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e568a60e0 0x560e568a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e5689d8d0 0x560e568a2d60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/2533746691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdba9ffb740 1 -- 172.21.15.7:0/2533746691 >> 172.21.15.7:0/2533746691 conn(0x560e56863c90 msgr2=0x560e568995a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdb88ff9640 0 client.25439 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdba9ffb740 1 -- 172.21.15.7:0/2533746691 shutdown_connections 2024-04-21T08:45:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.729+0000 7fdba9ffb740 1 -- 172.21.15.7:0/2533746691 wait complete. 2024-04-21T08:45:14.733 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.730+0000 7fdba9ffb740 20 client.25439 trim_cache size 0 max 16384 2024-04-21T08:45:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba9ffb740 1 Processor -- start 2024-04-21T08:45:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba9ffb740 1 -- start start 2024-04-21T08:45:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689d8d0 0x560e568a5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 0x560e568a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba9ffb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8400dfc0 con 0x560e5689cb60 2024-04-21T08:45:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba9ffb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84010840 con 0x560e568a60e0 2024-04-21T08:45:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba9ffb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8400e630 con 0x560e5689d8d0 2024-04-21T08:45:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba50c7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba50c7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55480/0 (socket says 172.21.15.7:55480) 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba50c7640 1 -- 172.21.15.7:0/3225511330 learned_addr learned my addr 172.21.15.7:0/3225511330 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.760+0000 7fdba48c6640 1 --2- 172.21.15.7:0/3225511330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689d8d0 0x560e568a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.761+0000 7fdba58c8640 1 --2- 172.21.15.7:0/3225511330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 0x560e568a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.761+0000 7fdba50c7640 1 -- 172.21.15.7:0/3225511330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689d8d0 msgr2=0x560e568a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.761+0000 7fdba50c7640 1 --2- 172.21.15.7:0/3225511330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689d8d0 0x560e568a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.761+0000 7fdba50c7640 1 -- 172.21.15.7:0/3225511330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 msgr2=0x560e568a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.761+0000 7fdba50c7640 1 --2- 172.21.15.7:0/3225511330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 0x560e568a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.761+0000 7fdba50c7640 1 -- 172.21.15.7:0/3225511330 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e56892f30 con 0x560e5689cb60 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.761+0000 7fdba58c8640 1 --2- 172.21.15.7:0/3225511330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 0x560e568a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.761+0000 7fdba50c7640 1 --2- 172.21.15.7:0/3225511330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568b1b20 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009710 tx=0x7fdb9c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c6f84fb496326e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.761+0000 7fdb8affd640 1 -- 172.21.15.7:0/3225511330 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c004030 con 0x560e5689cb60 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.761+0000 7fdb8affd640 1 -- 172.21.15.7:0/3225511330 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb9c00f8a0 con 0x560e5689cb60 2024-04-21T08:45:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.761+0000 7fdb8affd640 1 -- 172.21.15.7:0/3225511330 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c005440 con 0x560e5689cb60 2024-04-21T08:45:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.762+0000 7fdba9ffb740 1 -- 172.21.15.7:0/3225511330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 msgr2=0x560e568b1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.762+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/3225511330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568b1b20 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009710 tx=0x7fdb9c005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.762+0000 7fdba9ffb740 1 -- 172.21.15.7:0/3225511330 shutdown_connections 2024-04-21T08:45:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.762+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/3225511330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 0x560e568a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.762+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/3225511330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689d8d0 0x560e568a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.762+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/3225511330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689cb60 0x560e568b1b20 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.762+0000 7fdba9ffb740 1 -- 172.21.15.7:0/3225511330 >> 172.21.15.7:0/3225511330 conn(0x560e56863c90 msgr2=0x560e568a2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.762+0000 7fdba9ffb740 1 -- 172.21.15.7:0/3225511330 shutdown_connections 2024-04-21T08:45:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.763+0000 7fdba9ffb740 1 -- 172.21.15.7:0/3225511330 wait complete. 2024-04-21T08:45:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.763+0000 7fdba9ffb740 1 -- start start 2024-04-21T08:45:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.763+0000 7fdba9ffb740 1 Processor -- start 2024-04-21T08:45:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 0x560e568ae030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689d8d0 0x560e568ae570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba9ffb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 0x560e5689f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba9ffb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8400ed00 con 0x560e5689d8d0 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba9ffb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84010170 con 0x560e568a60e0 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba9ffb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8400f3d0 con 0x560e5689cb60 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba9ffb740 10 client.0 _ll_register_callbacks cb 0x7fdb980099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba50c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 0x560e568ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba48c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689d8d0 0x560e568ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba50c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 0x560e568ae030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:44818/0 (socket says 172.21.15.7:44818) 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba50c7640 1 -- 172.21.15.7:0/1147278408 learned_addr learned my addr 172.21.15.7:0/1147278408 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba58c8640 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 0x560e5689f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba50c7640 1 -- 172.21.15.7:0/1147278408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 msgr2=0x560e5689f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba50c7640 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 0x560e5689f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba50c7640 1 -- 172.21.15.7:0/1147278408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689d8d0 msgr2=0x560e568ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba50c7640 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689d8d0 0x560e568ae570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba50c7640 1 -- 172.21.15.7:0/1147278408 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e5689f760 con 0x560e5689cb60 2024-04-21T08:45:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba58c8640 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 0x560e5689f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.764+0000 7fdba48c6640 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689d8d0 0x560e568ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.765+0000 7fdba50c7640 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 0x560e568ae030 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c014010 tx=0x7fdb9c0121a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcbc8906c8a1a3ef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.765+0000 7fdb88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:45:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.765+0000 7fdb88ff9640 1 -- 172.21.15.7:0/1147278408 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c02e020 con 0x560e5689cb60 2024-04-21T08:45:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.765+0000 7fdb88ff9640 1 -- 172.21.15.7:0/1147278408 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fdb9c00fc50 con 0x560e5689cb60 2024-04-21T08:45:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.765+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1147278408 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb9c00f8a0 con 0x560e5689cb60 2024-04-21T08:45:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.765+0000 7fdb88ff9640 1 -- 172.21.15.7:0/1147278408 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdb9c039050 con 0x560e5689cb60 2024-04-21T08:45:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.765+0000 7fdba9ffb740 10 client.25471 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.765+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1147278408 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560e5691b570 con 0x560e5689cb60 2024-04-21T08:45:14.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.766+0000 7fdb88ff9640 1 -- 172.21.15.7:0/1147278408 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb9c00f8a0 con 0x560e5689cb60 2024-04-21T08:45:14.769 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.766+0000 7fdb88ff9640 1 -- 172.21.15.7:0/1147278408 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdb9c004bc0 con 0x560e5689cb60 2024-04-21T08:45:14.769 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.767+0000 7fdba9ffb740 10 client.25471 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:14.769 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.767+0000 7fdba9ffb740 10 client.25471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:14.769 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.767+0000 7fdba9ffb740 20 client.25471 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.767+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560e5691bbb0 0x560e5691dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.767+0000 7fdba9ffb740 4 client.25471 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:45:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.767+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1147278408 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560e5680f2f0 con 0x560e5691bbb0 2024-04-21T08:45:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.767+0000 7fdba48c6640 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560e5691bbb0 0x560e5691dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.767+0000 7fdba48c6640 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560e5691bbb0 0x560e5691dfd0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d31e912c2b22e3c3 server_cookie=80f94e605b2e1796 in_seq=0 out_seq=0 2024-04-21T08:45:14.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.767+0000 7fdb88ff9640 10 client.25471 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.771+0000 7fdb88ff9640 1 -- 172.21.15.7:0/1147278408 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560e5680f2f0 con 0x560e5691bbb0 2024-04-21T08:45:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.771+0000 7fdb88ff9640 10 client.25471 handle_command_reply: tid=0 2024-04-21T08:45:14.771 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.771+0000 7fdba9ffb740 1 client.25471 shutdown 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1147278408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 msgr2=0x560e568ae030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 0x560e568ae030 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c014010 tx=0x7fdb9c0121a0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1147278408 shutdown_connections 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560e5691bbb0 0x560e5691dfd0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560e568a60e0 0x560e5689f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560e5689d8d0 0x560e568ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdba9ffb740 1 --2- 172.21.15.7:0/1147278408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560e5689cb60 0x560e568ae030 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1147278408 >> 172.21.15.7:0/1147278408 conn(0x560e56863c90 msgr2=0x560e5689de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdb88ff9640 0 client.25471 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1147278408 shutdown_connections 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdba9ffb740 1 -- 172.21.15.7:0/1147278408 wait complete. 2024-04-21T08:45:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.772+0000 7fdba9ffb740 20 client.25471 trim_cache size 0 max 16384 2024-04-21T08:45:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:45:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:45:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:45:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:45:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.773+0000 7fdba732b640 1 -- 172.21.15.7:0/2168759048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdb8c0a3e30 msgr2=0x7fdb8c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.773+0000 7fdba732b640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdb8c0a3e30 0x7fdb8c0a62f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdb98005fd0 tx=0x7fdb98009500 comp rx=0 tx=0).stop 2024-04-21T08:45:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.773+0000 7fdba732b640 1 -- 172.21.15.7:0/2168759048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdba01072a0 msgr2=0x7fdba019e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.773+0000 7fdba732b640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdba01072a0 0x7fdba019e940 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdb90000c00 tx=0x7fdb9002c660 comp rx=0 tx=0).stop 2024-04-21T08:45:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.773+0000 7fdba732b640 1 -- 172.21.15.7:0/2168759048 shutdown_connections 2024-04-21T08:45:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.773+0000 7fdba732b640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdb8c0a3e30 0x7fdb8c0a62f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.773+0000 7fdba732b640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdba01985b0 0x7fdba0198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.773+0000 7fdba732b640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdba01072a0 0x7fdba019e940 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.773+0000 7fdba732b640 1 --2- 172.21.15.7:0/2168759048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdba01021d0 0x7fdba019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.773+0000 7fdba732b640 1 -- 172.21.15.7:0/2168759048 >> 172.21.15.7:0/2168759048 conn(0x7fdba00fda30 msgr2=0x7fdba00ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.773+0000 7fdba732b640 1 -- 172.21.15.7:0/2168759048 shutdown_connections 2024-04-21T08:45:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:14.774+0000 7fdba732b640 1 -- 172.21.15.7:0/2168759048 wait complete. 2024-04-21T08:45:15.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:14 smithi007 ceph-mon[28477]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T08:45:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:14 smithi016 ceph-mon[31259]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T08:45:15.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:14 smithi077 ceph-mon[31615]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T08:45:17.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:45:17.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:45:17.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:16 smithi007 ceph-mon[28477]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.197+0000 7f627740f640 1 -- 172.21.15.7:0/461641846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6270102b50 msgr2=0x7f6270106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.197+0000 7f627740f640 1 --2- 172.21.15.7:0/461641846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6270102b50 0x7f6270106b90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6260009900 tx=0x7f626002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.197+0000 7f627740f640 1 -- 172.21.15.7:0/461641846 shutdown_connections 2024-04-21T08:45:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.197+0000 7f627740f640 1 --2- 172.21.15.7:0/461641846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6270107270 0x7f627010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.197+0000 7f627740f640 1 --2- 172.21.15.7:0/461641846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6270102b50 0x7f6270106b90 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.197+0000 7f627740f640 1 --2- 172.21.15.7:0/461641846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62701021a0 0x7f6270102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.197+0000 7f627740f640 1 -- 172.21.15.7:0/461641846 >> 172.21.15.7:0/461641846 conn(0x7f62700fd9c0 msgr2=0x7f62700ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.197+0000 7f627740f640 1 -- 172.21.15.7:0/461641846 shutdown_connections 2024-04-21T08:45:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.197+0000 7f627740f640 1 -- 172.21.15.7:0/461641846 wait complete. 2024-04-21T08:45:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.198+0000 7f627740f640 1 Processor -- start 2024-04-21T08:45:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.198+0000 7f627740f640 1 -- start start 2024-04-21T08:45:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.198+0000 7f627740f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62701021a0 0x7f6270198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.198+0000 7f627740f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6270107270 0x7f6270198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.198+0000 7f627740f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627019cdf0 0x7f627019d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.198+0000 7f627740f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6270112190 con 0x7f627019cdf0 2024-04-21T08:45:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.198+0000 7f627740f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6270112010 con 0x7f6270107270 2024-04-21T08:45:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.198+0000 7f627740f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6270112310 con 0x7f62701021a0 2024-04-21T08:45:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62759ac640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627019cdf0 0x7f627019d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62751ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62701021a0 0x7f6270198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62751ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62701021a0 0x7f6270198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:44836/0 (socket says 172.21.15.7:44836) 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62751ab640 1 -- 172.21.15.7:0/3080579954 learned_addr learned my addr 172.21.15.7:0/3080579954 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62749aa640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6270107270 0x7f6270198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62749aa640 1 -- 172.21.15.7:0/3080579954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62701021a0 msgr2=0x7f6270198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62749aa640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62701021a0 0x7f6270198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62749aa640 1 -- 172.21.15.7:0/3080579954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627019cdf0 msgr2=0x7f627019d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62749aa640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627019cdf0 0x7f627019d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62749aa640 1 -- 172.21.15.7:0/3080579954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f627019d8e0 con 0x7f6270107270 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62759ac640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627019cdf0 0x7f627019d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.199+0000 7f62749aa640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6270107270 0x7f6270198a70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f626002c170 tx=0x7f626002ca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.200+0000 7f62667fc640 1 -- 172.21.15.7:0/3080579954 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260042020 con 0x7f6270107270 2024-04-21T08:45:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.200+0000 7f627740f640 1 -- 172.21.15.7:0/3080579954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f627019db70 con 0x7f6270107270 2024-04-21T08:45:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.200+0000 7f62667fc640 1 -- 172.21.15.7:0/3080579954 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f626002c5d0 con 0x7f6270107270 2024-04-21T08:45:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.201+0000 7f62667fc640 1 -- 172.21.15.7:0/3080579954 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260003c70 con 0x7f6270107270 2024-04-21T08:45:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.201+0000 7f627740f640 1 -- 172.21.15.7:0/3080579954 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6270074470 con 0x7f6270107270 2024-04-21T08:45:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.203+0000 7f62667fc640 1 -- 172.21.15.7:0/3080579954 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f6260045020 con 0x7f6270107270 2024-04-21T08:45:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.204+0000 7f62667fc640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f625c0a3d20 0x7f625c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.204+0000 7f6279ffd740 1 Processor -- start 2024-04-21T08:45:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.204+0000 7f6279ffd740 1 -- start start 2024-04-21T08:45:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.204+0000 7f62751ab640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f625c0a3d20 0x7f625c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.204+0000 7f6279ffd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb45b390 0x5590bb45b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.204+0000 7f6279ffd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb45bd40 0x5590bb46cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.204+0000 7f6279ffd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb46d290 0x5590bb46f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.204+0000 7f6279ffd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5590bb3c01e0 con 0x5590bb46d290 2024-04-21T08:45:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.204+0000 7f6279ffd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5590bb35d390 con 0x5590bb45b390 2024-04-21T08:45:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.204+0000 7f6279ffd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5590bb3bc1a0 con 0x5590bb45bd40 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.205+0000 7f62759ac640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb46d290 0x5590bb46f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.205+0000 7f62759ac640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb46d290 0x5590bb46f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55524/0 (socket says 172.21.15.7:55524) 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.205+0000 7f62759ac640 1 -- 172.21.15.7:0/377642159 learned_addr learned my addr 172.21.15.7:0/377642159 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.205+0000 7f62667fc640 1 -- 172.21.15.7:0/3080579954 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f62600f2cb0 con 0x7f6270107270 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.205+0000 7f62749aa640 1 --2- 172.21.15.7:0/377642159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb45bd40 0x5590bb46cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.205+0000 7f62759ac640 1 -- 172.21.15.7:0/377642159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb45bd40 msgr2=0x5590bb46cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.205+0000 7f62759ac640 1 --2- 172.21.15.7:0/377642159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb45bd40 0x5590bb46cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.205+0000 7f62759ac640 1 -- 172.21.15.7:0/377642159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb45b390 msgr2=0x5590bb45b770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.205+0000 7f62759ac640 1 --2- 172.21.15.7:0/377642159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb45b390 0x5590bb45b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.205+0000 7f62759ac640 1 -- 172.21.15.7:0/377642159 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590bb19c7d0 con 0x5590bb46d290 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.206+0000 7f62751ab640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f625c0a3d20 0x7f625c0a61e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f626c005fd0 tx=0x7f626c00b480 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.206+0000 7f62759ac640 1 --2- 172.21.15.7:0/377642159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb46d290 0x5590bb46f680 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f62680107a0 tx=0x7f6268010c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8ae87cc925a35 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.206+0000 7f625b7fe640 1 -- 172.21.15.7:0/377642159 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268003b70 con 0x5590bb46d290 2024-04-21T08:45:17.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.206+0000 7f625b7fe640 1 -- 172.21.15.7:0/377642159 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6268010e50 con 0x5590bb46d290 2024-04-21T08:45:17.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.206+0000 7f625b7fe640 1 -- 172.21.15.7:0/377642159 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62680046c0 con 0x5590bb46d290 2024-04-21T08:45:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.207+0000 7f6279ffd740 1 -- 172.21.15.7:0/377642159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb46d290 msgr2=0x5590bb46f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.207+0000 7f6279ffd740 1 --2- 172.21.15.7:0/377642159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb46d290 0x5590bb46f680 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f62680107a0 tx=0x7f6268010c60 comp rx=0 tx=0).stop 2024-04-21T08:45:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.207+0000 7f6279ffd740 1 -- 172.21.15.7:0/377642159 shutdown_connections 2024-04-21T08:45:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.207+0000 7f6279ffd740 1 --2- 172.21.15.7:0/377642159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb46d290 0x5590bb46f680 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.207+0000 7f6279ffd740 1 --2- 172.21.15.7:0/377642159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb45bd40 0x5590bb46cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.207+0000 7f6279ffd740 1 --2- 172.21.15.7:0/377642159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb45b390 0x5590bb45b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.207+0000 7f6279ffd740 1 -- 172.21.15.7:0/377642159 >> 172.21.15.7:0/377642159 conn(0x5590bb455cb0 msgr2=0x5590bb456090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.207+0000 7f6279ffd740 1 -- 172.21.15.7:0/377642159 shutdown_connections 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.207+0000 7f6279ffd740 1 -- 172.21.15.7:0/377642159 wait complete. 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.208+0000 7f6279ffd740 1 -- start start 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.208+0000 7f6279ffd740 1 Processor -- start 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.208+0000 7f6279ffd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45b390 0x5590bb46a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.208+0000 7f6279ffd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb45bd40 0x5590bb46a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.208+0000 7f6279ffd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb46d290 0x5590bb46ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.208+0000 7f6279ffd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5590bb470050 con 0x5590bb45b390 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.208+0000 7f6279ffd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5590bb3792f0 con 0x5590bb46d290 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.208+0000 7f6279ffd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5590bb4701d0 con 0x5590bb45bd40 2024-04-21T08:45:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.208+0000 7f6279ffd740 10 client.0 _ll_register_callbacks cb 0x5590bb386580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.209+0000 7f62751ab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45b390 0x5590bb46a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.209+0000 7f62751ab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45b390 0x5590bb46a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55526/0 (socket says 172.21.15.7:55526) 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.209+0000 7f62751ab640 1 -- 172.21.15.7:0/1009355212 learned_addr learned my addr 172.21.15.7:0/1009355212 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.209+0000 7f62749aa640 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb45bd40 0x5590bb46a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.209+0000 7f62759ac640 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb46d290 0x5590bb46ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.209+0000 7f62751ab640 1 -- 172.21.15.7:0/1009355212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb45bd40 msgr2=0x5590bb46a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.209+0000 7f62751ab640 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb45bd40 0x5590bb46a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.209+0000 7f62751ab640 1 -- 172.21.15.7:0/1009355212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb46d290 msgr2=0x5590bb46ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.209+0000 7f62751ab640 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb46d290 0x5590bb46ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.209+0000 7f62751ab640 1 -- 172.21.15.7:0/1009355212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590bb0d23f0 con 0x5590bb45b390 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.210+0000 7f62751ab640 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45b390 0x5590bb46a320 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f626c011010 tx=0x7f626c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb1bb48723a036f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.210+0000 7f6258ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.210+0000 7f6258ff9640 1 -- 172.21.15.7:0/1009355212 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c01ca40 con 0x5590bb45b390 2024-04-21T08:45:17.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.210+0000 7f6258ff9640 1 -- 172.21.15.7:0/1009355212 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f626c01cbe0 con 0x5590bb45b390 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.210+0000 7f6279ffd740 1 -- 172.21.15.7:0/1009355212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6268010e50 con 0x5590bb45b390 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.210+0000 7f6258ff9640 1 -- 172.21.15.7:0/1009355212 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f626c011e50 con 0x5590bb45b390 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.210+0000 7f6279ffd740 10 client.15858 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.210+0000 7f6279ffd740 1 -- 172.21.15.7:0/1009355212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5590bb0920f0 con 0x5590bb45b390 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.210+0000 7f6258ff9640 1 -- 172.21.15.7:0/1009355212 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6268010e50 con 0x5590bb45b390 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.211+0000 7f6258ff9640 1 -- 172.21.15.7:0/1009355212 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f626c01e530 con 0x5590bb45b390 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.211+0000 7f6279ffd740 10 client.15858 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.211+0000 7f6279ffd740 10 client.15858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.211+0000 7f6279ffd740 20 client.15858 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.211+0000 7f6279ffd740 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590bb47d5a0 0x5590bb47fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.211+0000 7f6279ffd740 4 client.15858 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.211+0000 7f6279ffd740 1 -- 172.21.15.7:0/1009355212 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590bb3cdc00 con 0x5590bb47d5a0 2024-04-21T08:45:17.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.211+0000 7f62749aa640 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590bb47d5a0 0x5590bb47fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.211+0000 7f6258ff9640 10 client.15858 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.211+0000 7f62749aa640 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590bb47d5a0 0x5590bb47fa20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b79ecc5b568d59b server_cookie=2964f902a29ceffb in_seq=0 out_seq=0 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6258ff9640 1 -- 172.21.15.7:0/1009355212 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590bb3cdc00 con 0x5590bb47d5a0 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6258ff9640 10 client.15858 handle_command_reply: tid=0 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6279ffd740 1 client.15858 shutdown 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6279ffd740 1 -- 172.21.15.7:0/1009355212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45b390 msgr2=0x5590bb46a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6279ffd740 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45b390 0x5590bb46a320 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f626c011010 tx=0x7f626c004730 comp rx=0 tx=0).stop 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6279ffd740 1 -- 172.21.15.7:0/1009355212 shutdown_connections 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6279ffd740 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590bb47d5a0 0x5590bb47fa20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6279ffd740 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb46d290 0x5590bb46ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6279ffd740 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb45bd40 0x5590bb46a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6279ffd740 1 --2- 172.21.15.7:0/1009355212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45b390 0x5590bb46a320 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6258ff9640 0 client.15858 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:17.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.212+0000 7f6279ffd740 1 -- 172.21.15.7:0/1009355212 >> 172.21.15.7:0/1009355212 conn(0x5590bb455cb0 msgr2=0x5590bb457a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.213+0000 7f6279ffd740 1 -- 172.21.15.7:0/1009355212 shutdown_connections 2024-04-21T08:45:17.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.213+0000 7f6279ffd740 1 -- 172.21.15.7:0/1009355212 wait complete. 2024-04-21T08:45:17.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.213+0000 7f6279ffd740 20 client.15858 trim_cache size 0 max 16384 2024-04-21T08:45:17.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.231+0000 7f6279ffd740 1 Processor -- start 2024-04-21T08:45:17.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.231+0000 7f6279ffd740 1 -- start start 2024-04-21T08:45:17.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.231+0000 7f6279ffd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb45bd40 0x5590bb46ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.231+0000 7f6279ffd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 0x5590bb478f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.231+0000 7f6279ffd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb469480 0x5590bb457810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.231+0000 7f6279ffd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f62701aad10 con 0x5590bb469480 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.231+0000 7f6279ffd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62701ad590 con 0x5590bb45bd40 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f6279ffd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f62701ab380 con 0x5590bb46d290 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62759ac640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb469480 0x5590bb457810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62749aa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 0x5590bb478f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62749aa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 0x5590bb478f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:44878/0 (socket says 172.21.15.7:44878) 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62749aa640 1 -- 172.21.15.7:0/3582455790 learned_addr learned my addr 172.21.15.7:0/3582455790 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62751ab640 1 --2- 172.21.15.7:0/3582455790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb45bd40 0x5590bb46ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62749aa640 1 -- 172.21.15.7:0/3582455790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb45bd40 msgr2=0x5590bb46ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62749aa640 1 --2- 172.21.15.7:0/3582455790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb45bd40 0x5590bb46ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62749aa640 1 -- 172.21.15.7:0/3582455790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb469480 msgr2=0x5590bb457810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62749aa640 1 --2- 172.21.15.7:0/3582455790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb469480 0x5590bb457810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62749aa640 1 -- 172.21.15.7:0/3582455790 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590bb47d350 con 0x5590bb46d290 2024-04-21T08:45:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62759ac640 1 --2- 172.21.15.7:0/3582455790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb469480 0x5590bb457810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.232+0000 7f62749aa640 1 --2- 172.21.15.7:0/3582455790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 0x5590bb478f50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62600be6b0 tx=0x7f62600a9040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2fe6a87ee11281b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.233+0000 7f625affd640 1 -- 172.21.15.7:0/3582455790 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62600be720 con 0x5590bb46d290 2024-04-21T08:45:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.233+0000 7f625affd640 1 -- 172.21.15.7:0/3582455790 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62600c2ad0 con 0x5590bb46d290 2024-04-21T08:45:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.233+0000 7f625affd640 1 -- 172.21.15.7:0/3582455790 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62600c6f00 con 0x5590bb46d290 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.234+0000 7f6279ffd740 1 -- 172.21.15.7:0/3582455790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 msgr2=0x5590bb478f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.234+0000 7f6279ffd740 1 --2- 172.21.15.7:0/3582455790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 0x5590bb478f50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62600be6b0 tx=0x7f62600a9040 comp rx=0 tx=0).stop 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.234+0000 7f6279ffd740 1 -- 172.21.15.7:0/3582455790 shutdown_connections 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.234+0000 7f6279ffd740 1 --2- 172.21.15.7:0/3582455790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb469480 0x5590bb457810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.234+0000 7f6279ffd740 1 --2- 172.21.15.7:0/3582455790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 0x5590bb478f50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.234+0000 7f6279ffd740 1 --2- 172.21.15.7:0/3582455790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb45bd40 0x5590bb46ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.234+0000 7f6279ffd740 1 -- 172.21.15.7:0/3582455790 >> 172.21.15.7:0/3582455790 conn(0x5590bb455cb0 msgr2=0x5590bb457100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.234+0000 7f6279ffd740 1 -- 172.21.15.7:0/3582455790 shutdown_connections 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.234+0000 7f6279ffd740 1 -- 172.21.15.7:0/3582455790 wait complete. 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.234+0000 7f6279ffd740 1 -- start start 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.234+0000 7f6279ffd740 1 Processor -- start 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.235+0000 7f6279ffd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45bd40 0x5590bb467150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.235+0000 7f62751ab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45bd40 0x5590bb467150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.235+0000 7f62751ab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45bd40 0x5590bb467150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55530/0 (socket says 172.21.15.7:55530) 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.235+0000 7f62751ab640 1 -- 172.21.15.7:0/2449750031 learned_addr learned my addr 172.21.15.7:0/2449750031 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.235+0000 7f6279ffd740 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb469480 0x5590bb467690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.235+0000 7f6279ffd740 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 0x5590bb467bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.235+0000 7f6279ffd740 1 -- 172.21.15.7:0/2449750031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f62701aba50 con 0x5590bb45bd40 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.235+0000 7f6279ffd740 1 -- 172.21.15.7:0/2449750031 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62701acec0 con 0x5590bb469480 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.235+0000 7f6279ffd740 1 -- 172.21.15.7:0/2449750031 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f62701ac120 con 0x5590bb46d290 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.235+0000 7f6279ffd740 10 client.0 _ll_register_callbacks cb 0x7f626c00ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f62749aa640 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb469480 0x5590bb467690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f62759ac640 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 0x5590bb467bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f62751ab640 1 -- 172.21.15.7:0/2449750031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 msgr2=0x5590bb467bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f62751ab640 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 0x5590bb467bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f62751ab640 1 -- 172.21.15.7:0/2449750031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb469480 msgr2=0x5590bb467690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f62751ab640 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb469480 0x5590bb467690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f62751ab640 1 -- 172.21.15.7:0/2449750031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590bb4e2d00 con 0x5590bb45bd40 2024-04-21T08:45:17.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f62749aa640 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb469480 0x5590bb467690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:17.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f62751ab640 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45bd40 0x5590bb467150 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f626c005940 tx=0x7f626c01ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2fbb318696ac7e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f6258ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f6258ff9640 1 -- 172.21.15.7:0/2449750031 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c011e50 con 0x5590bb45bd40 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f6279ffd740 1 -- 172.21.15.7:0/2449750031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62600c2ad0 con 0x5590bb45bd40 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f6258ff9640 1 -- 172.21.15.7:0/2449750031 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f626c01e420 con 0x5590bb45bd40 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.236+0000 7f6258ff9640 1 -- 172.21.15.7:0/2449750031 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f626c046050 con 0x5590bb45bd40 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.237+0000 7f6279ffd740 10 client.15864 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.237+0000 7f6279ffd740 1 -- 172.21.15.7:0/2449750031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5590bb4e3fd0 con 0x5590bb45bd40 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.237+0000 7f6258ff9640 1 -- 172.21.15.7:0/2449750031 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62600c2ad0 con 0x5590bb45bd40 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.237+0000 7f6258ff9640 1 -- 172.21.15.7:0/2449750031 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f626c03b020 con 0x5590bb45bd40 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.237+0000 7f6279ffd740 10 client.15864 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.237+0000 7f6279ffd740 10 client.15864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.237+0000 7f6279ffd740 20 client.15864 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.237+0000 7f6279ffd740 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590bb4e4780 0x5590bb4e6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.237+0000 7f6279ffd740 4 client.15864 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.237+0000 7f6279ffd740 1 -- 172.21.15.7:0/2449750031 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5590bb3c7a70 con 0x5590bb4e4780 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.237+0000 7f62749aa640 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590bb4e4780 0x5590bb4e6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.238+0000 7f6258ff9640 10 client.15864 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.238+0000 7f62749aa640 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590bb4e4780 0x5590bb4e6ba0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a920a04e958d7f2 server_cookie=635613c0c2fee548 in_seq=0 out_seq=0 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6258ff9640 1 -- 172.21.15.7:0/2449750031 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5590bb3c7a70 con 0x5590bb4e4780 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6258ff9640 10 client.15864 handle_command_reply: tid=0 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 1 client.15864 shutdown 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 1 -- 172.21.15.7:0/2449750031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45bd40 msgr2=0x5590bb467150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45bd40 0x5590bb467150 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f626c005940 tx=0x7f626c01ce90 comp rx=0 tx=0).stop 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 1 -- 172.21.15.7:0/2449750031 shutdown_connections 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590bb4e4780 0x5590bb4e6ba0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5590bb46d290 0x5590bb467bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6258ff9640 0 client.15864 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5590bb469480 0x5590bb467690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 1 --2- 172.21.15.7:0/2449750031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5590bb45bd40 0x5590bb467150 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 1 -- 172.21.15.7:0/2449750031 >> 172.21.15.7:0/2449750031 conn(0x5590bb455cb0 msgr2=0x5590bb457990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 1 -- 172.21.15.7:0/2449750031 shutdown_connections 2024-04-21T08:45:17.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 1 -- 172.21.15.7:0/2449750031 wait complete. 2024-04-21T08:45:17.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.239+0000 7f6279ffd740 20 client.15864 trim_cache size 0 max 16384 2024-04-21T08:45:17.242 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:45:17.242 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:45:17.242 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:45:17.242 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:45:17.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 -- 172.21.15.7:0/3080579954 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f625c0a3d20 msgr2=0x7f625c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f625c0a3d20 0x7f625c0a61e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f626c005fd0 tx=0x7f626c00b480 comp rx=0 tx=0).stop 2024-04-21T08:45:17.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 -- 172.21.15.7:0/3080579954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6270107270 msgr2=0x7f6270198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6270107270 0x7f6270198a70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f626002c170 tx=0x7f626002ca90 comp rx=0 tx=0).stop 2024-04-21T08:45:17.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 -- 172.21.15.7:0/3080579954 shutdown_connections 2024-04-21T08:45:17.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627019cdf0 0x7f627019d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6270107270 0x7f6270198a70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f625c0a3d20 0x7f625c0a61e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 --2- 172.21.15.7:0/3080579954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62701021a0 0x7f6270198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 -- 172.21.15.7:0/3080579954 >> 172.21.15.7:0/3080579954 conn(0x7f62700fd9c0 msgr2=0x7f62700fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 -- 172.21.15.7:0/3080579954 shutdown_connections 2024-04-21T08:45:17.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.240+0000 7f627740f640 1 -- 172.21.15.7:0/3080579954 wait complete. 2024-04-21T08:45:17.250 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:45:17.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:45:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:16 smithi016 ceph-mon[31259]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:17.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:16 smithi077 ceph-mon[31615]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:17.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.358+0000 7f86d90ba640 1 -- 172.21.15.7:0/1187896355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86d4101070 msgr2=0x7f86d4101450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.358+0000 7f86d90ba640 1 --2- 172.21.15.7:0/1187896355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86d4101070 0x7f86d4101450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f86c400b9f0 tx=0x7f86c402d8f0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.358+0000 7f86d90ba640 1 -- 172.21.15.7:0/1187896355 shutdown_connections 2024-04-21T08:45:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.358+0000 7f86d90ba640 1 --2- 172.21.15.7:0/1187896355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d4106140 0x7f86d410dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.358+0000 7f86d90ba640 1 --2- 172.21.15.7:0/1187896355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86d4101a20 0x7f86d4105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.358+0000 7f86d90ba640 1 --2- 172.21.15.7:0/1187896355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86d4101070 0x7f86d4101450 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f86c400b9f0 tx=0x7f86c402d8f0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.358+0000 7f86d90ba640 1 -- 172.21.15.7:0/1187896355 >> 172.21.15.7:0/1187896355 conn(0x7f86d40771d0 msgr2=0x7f86d40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.359+0000 7f86d90ba640 1 -- 172.21.15.7:0/1187896355 shutdown_connections 2024-04-21T08:45:17.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.359+0000 7f86d90ba640 1 -- 172.21.15.7:0/1187896355 wait complete. 2024-04-21T08:45:17.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d90ba640 1 Processor -- start 2024-04-21T08:45:17.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d90ba640 1 -- start start 2024-04-21T08:45:17.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d90ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86d4106140 0x7f86d41966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d90ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86d4196c30 0x7f86d419afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d90ba640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d419b4f0 0x7f86d419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d90ba640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f86d410ffa0 con 0x7f86d419b4f0 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d2d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86d4106140 0x7f86d41966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d2d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86d4106140 0x7f86d41966f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:44914/0 (socket says 172.21.15.7:44914) 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d2d9d640 1 -- 172.21.15.7:0/1165804297 learned_addr learned my addr 172.21.15.7:0/1165804297 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f86d410fe20 con 0x7f86d4196c30 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86d4110120 con 0x7f86d4106140 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.360+0000 7f86d359e640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d419b4f0 0x7f86d419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86d259c640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86d4196c30 0x7f86d419afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86d359e640 1 -- 172.21.15.7:0/1165804297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86d4106140 msgr2=0x7f86d41966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86d359e640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86d4106140 0x7f86d41966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86d359e640 1 -- 172.21.15.7:0/1165804297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86d4196c30 msgr2=0x7f86d419afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86d359e640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86d4196c30 0x7f86d419afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86d359e640 1 -- 172.21.15.7:0/1165804297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d41a3260 con 0x7f86d419b4f0 2024-04-21T08:45:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86d2d9d640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86d4106140 0x7f86d41966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86d259c640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86d4196c30 0x7f86d419afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86d359e640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d419b4f0 0x7f86d419b9a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00efc0 tx=0x7f86cc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86bbfff640 1 -- 172.21.15.7:0/1165804297 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc003970 con 0x7f86d419b4f0 2024-04-21T08:45:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d41a3490 con 0x7f86d419b4f0 2024-04-21T08:45:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86bbfff640 1 -- 172.21.15.7:0/1165804297 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86cc004020 con 0x7f86d419b4f0 2024-04-21T08:45:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d41a39b0 con 0x7f86d419b4f0 2024-04-21T08:45:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.361+0000 7f86bbfff640 1 -- 172.21.15.7:0/1165804297 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc00b940 con 0x7f86d419b4f0 2024-04-21T08:45:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.363+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86d4102540 con 0x7f86d419b4f0 2024-04-21T08:45:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.365+0000 7f86bbfff640 1 -- 172.21.15.7:0/1165804297 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f86cc01d020 con 0x7f86d419b4f0 2024-04-21T08:45:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.366+0000 7f86bbfff640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f86bc0a3f40 0x7f86bc0a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.366+0000 7f86bbfff640 1 -- 172.21.15.7:0/1165804297 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f86cc0d1320 con 0x7f86d419b4f0 2024-04-21T08:45:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.366+0000 7f86d2d9d640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f86bc0a3f40 0x7f86bc0a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.367+0000 7f86d2d9d640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f86bc0a3f40 0x7f86bc0a6400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f86c400b800 tx=0x7f86c4005d90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.368+0000 7f86bbfff640 1 -- 172.21.15.7:0/1165804297 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f86cc09d320 con 0x7f86d419b4f0 2024-04-21T08:45:17.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.621+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86d4068e00 con 0x7f86d419b4f0 2024-04-21T08:45:17.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.621+0000 7f86bbfff640 1 -- 172.21.15.7:0/1165804297 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7f86d4068e00 con 0x7f86d419b4f0 2024-04-21T08:45:17.621 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:45:17.621 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:45:17.622 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:45:17.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.624+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f86bc0a3f40 msgr2=0x7f86bc0a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.624+0000 7f86d90ba640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f86bc0a3f40 0x7f86bc0a6400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f86c400b800 tx=0x7f86c4005d90 comp rx=0 tx=0).stop 2024-04-21T08:45:17.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.624+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d419b4f0 msgr2=0x7f86d419b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.624+0000 7f86d90ba640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d419b4f0 0x7f86d419b9a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00efc0 tx=0x7f86cc00c800 comp rx=0 tx=0).stop 2024-04-21T08:45:17.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.624+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 shutdown_connections 2024-04-21T08:45:17.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.624+0000 7f86d90ba640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d419b4f0 0x7f86d419b9a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.624+0000 7f86d90ba640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86d4196c30 0x7f86d419afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.624+0000 7f86d90ba640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f86bc0a3f40 0x7f86bc0a6400 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.624+0000 7f86d90ba640 1 --2- 172.21.15.7:0/1165804297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86d4106140 0x7f86d41966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.624+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 >> 172.21.15.7:0/1165804297 conn(0x7f86d40771d0 msgr2=0x7f86d40fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.625+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 shutdown_connections 2024-04-21T08:45:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.625+0000 7f86d90ba640 1 -- 172.21.15.7:0/1165804297 wait complete. 2024-04-21T08:45:17.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:45:17.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.751+0000 7f1c33f75640 1 -- 172.21.15.7:0/3342535170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c2c101090 msgr2=0x7f1c2c101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.751+0000 7f1c33f75640 1 --2- 172.21.15.7:0/3342535170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c2c101090 0x7f1c2c101470 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009900 tx=0x7f1c2802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.751+0000 7f1c33f75640 1 -- 172.21.15.7:0/3342535170 shutdown_connections 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.751+0000 7f1c33f75640 1 --2- 172.21.15.7:0/3342535170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c2c106160 0x7f1c2c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.751+0000 7f1c33f75640 1 --2- 172.21.15.7:0/3342535170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2c101a40 0x7f1c2c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.751+0000 7f1c33f75640 1 --2- 172.21.15.7:0/3342535170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c2c101090 0x7f1c2c101470 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.751+0000 7f1c33f75640 1 -- 172.21.15.7:0/3342535170 >> 172.21.15.7:0/3342535170 conn(0x7f1c2c078000 msgr2=0x7f1c2c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.751+0000 7f1c33f75640 1 -- 172.21.15.7:0/3342535170 shutdown_connections 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.751+0000 7f1c33f75640 1 -- 172.21.15.7:0/3342535170 wait complete. 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.751+0000 7f1c33f75640 1 Processor -- start 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c33f75640 1 -- start start 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c33f75640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c2c101090 0x7f1c2c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c33f75640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2c106160 0x7f1c2c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c33f75640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c2c1985b0 0x7f1c2c198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c33f75640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c111120 con 0x7f1c2c1985b0 2024-04-21T08:45:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c33f75640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c110fa0 con 0x7f1c2c106160 2024-04-21T08:45:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c33f75640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1112a0 con 0x7f1c2c101090 2024-04-21T08:45:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c32512640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c2c1985b0 0x7f1c2c198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c32512640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c2c1985b0 0x7f1c2c198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55562/0 (socket says 172.21.15.7:55562) 2024-04-21T08:45:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c32512640 1 -- 172.21.15.7:0/3885365286 learned_addr learned my addr 172.21.15.7:0/3885365286 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c31d11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c2c101090 0x7f1c2c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c31510640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2c106160 0x7f1c2c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c32512640 1 -- 172.21.15.7:0/3885365286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c2c101090 msgr2=0x7f1c2c19e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c32512640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c2c101090 0x7f1c2c19e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c32512640 1 -- 172.21.15.7:0/3885365286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2c106160 msgr2=0x7f1c2c19e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c32512640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2c106160 0x7f1c2c19e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c32512640 1 -- 172.21.15.7:0/3885365286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2c1990d0 con 0x7f1c2c1985b0 2024-04-21T08:45:17.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.752+0000 7f1c31510640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2c106160 0x7f1c2c19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:17.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.753+0000 7f1c32512640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c2c1985b0 0x7f1c2c198a60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400ea60 tx=0x7f1c2400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.753+0000 7f1c22ffd640 1 -- 172.21.15.7:0/3885365286 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400ce40 con 0x7f1c2c1985b0 2024-04-21T08:45:17.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.753+0000 7f1c22ffd640 1 -- 172.21.15.7:0/3885365286 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c24005020 con 0x7f1c2c1985b0 2024-04-21T08:45:17.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.753+0000 7f1c22ffd640 1 -- 172.21.15.7:0/3885365286 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24010670 con 0x7f1c2c1985b0 2024-04-21T08:45:17.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.753+0000 7f1c33f75640 1 -- 172.21.15.7:0/3885365286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c2c1993c0 con 0x7f1c2c1985b0 2024-04-21T08:45:17.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.753+0000 7f1c33f75640 1 -- 172.21.15.7:0/3885365286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c2c1a55b0 con 0x7f1c2c1985b0 2024-04-21T08:45:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.755+0000 7f1c22ffd640 1 -- 172.21.15.7:0/3885365286 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f1c240054b0 con 0x7f1c2c1985b0 2024-04-21T08:45:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.756+0000 7f1c22ffd640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1c180a3f90 0x7f1c180a6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.756+0000 7f1c22ffd640 1 -- 172.21.15.7:0/3885365286 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f1c240d1290 con 0x7f1c2c1985b0 2024-04-21T08:45:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.756+0000 7f1c31d11640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1c180a3f90 0x7f1c180a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.757+0000 7f1c31d11640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1c180a3f90 0x7f1c180a6450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c28005ab0 tx=0x7f1c28005a20 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.757+0000 7f1c367fd740 1 Processor -- start 2024-04-21T08:45:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.757+0000 7f1c367fd740 1 -- start start 2024-04-21T08:45:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.757+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670b8e0 0x56202670bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.757+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670c290 0x562026714840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.757+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562026714e10 0x562026717260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.757+0000 7f1c367fd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56202666c1f0 con 0x56202670b8e0 2024-04-21T08:45:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.757+0000 7f1c367fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5620266094b0 con 0x562026714e10 2024-04-21T08:45:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.757+0000 7f1c367fd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5620266681b0 con 0x56202670c290 2024-04-21T08:45:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.758+0000 7f1c31d11640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670b8e0 0x56202670bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.758+0000 7f1c31d11640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670b8e0 0x56202670bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55576/0 (socket says 172.21.15.7:55576) 2024-04-21T08:45:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.758+0000 7f1c31d11640 1 -- 172.21.15.7:0/1541417421 learned_addr learned my addr 172.21.15.7:0/1541417421 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.758+0000 7f1c31510640 1 --2- 172.21.15.7:0/1541417421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670c290 0x562026714840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.758+0000 7f1c32512640 1 --2- 172.21.15.7:0/1541417421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562026714e10 0x562026717260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.758+0000 7f1c31d11640 1 -- 172.21.15.7:0/1541417421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670c290 msgr2=0x562026714840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.758+0000 7f1c31d11640 1 --2- 172.21.15.7:0/1541417421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670c290 0x562026714840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.758+0000 7f1c31d11640 1 -- 172.21.15.7:0/1541417421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562026714e10 msgr2=0x562026717260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.758+0000 7f1c31d11640 1 --2- 172.21.15.7:0/1541417421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562026714e10 0x562026717260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.758+0000 7f1c31d11640 1 -- 172.21.15.7:0/1541417421 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620265fb1b0 con 0x56202670b8e0 2024-04-21T08:45:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.758+0000 7f1c31d11640 1 --2- 172.21.15.7:0/1541417421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670b8e0 0x56202670bcc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1c2802c780 tx=0x7f1c2803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d2d2be7acbd369 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.759+0000 7f1c17fff640 1 -- 172.21.15.7:0/1541417421 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28004390 con 0x56202670b8e0 2024-04-21T08:45:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.759+0000 7f1c17fff640 1 -- 172.21.15.7:0/1541417421 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c28050070 con 0x56202670b8e0 2024-04-21T08:45:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.759+0000 7f1c17fff640 1 -- 172.21.15.7:0/1541417421 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28004dc0 con 0x56202670b8e0 2024-04-21T08:45:17.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.759+0000 7f1c367fd740 1 -- 172.21.15.7:0/1541417421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670b8e0 msgr2=0x56202670bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.759+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1541417421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670b8e0 0x56202670bcc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1c2802c780 tx=0x7f1c2803e000 comp rx=0 tx=0).stop 2024-04-21T08:45:17.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.759+0000 7f1c367fd740 1 -- 172.21.15.7:0/1541417421 shutdown_connections 2024-04-21T08:45:17.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.759+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1541417421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562026714e10 0x562026717260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.759+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1541417421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670c290 0x562026714840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.759+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1541417421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670b8e0 0x56202670bcc0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.759+0000 7f1c367fd740 1 -- 172.21.15.7:0/1541417421 >> 172.21.15.7:0/1541417421 conn(0x562026705fb0 msgr2=0x562026706390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.760+0000 7f1c367fd740 1 -- 172.21.15.7:0/1541417421 shutdown_connections 2024-04-21T08:45:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.760+0000 7f1c367fd740 1 -- 172.21.15.7:0/1541417421 wait complete. 2024-04-21T08:45:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.760+0000 7f1c367fd740 1 -- start start 2024-04-21T08:45:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.760+0000 7f1c367fd740 1 Processor -- start 2024-04-21T08:45:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.760+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x562026714710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.760+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670c290 0x562026711aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.760+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562026714e10 0x562026712020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.760+0000 7f1c367fd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562026717d20 con 0x56202670c290 2024-04-21T08:45:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.760+0000 7f1c367fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562026625500 con 0x562026714e10 2024-04-21T08:45:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.760+0000 7f1c367fd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562026717ea0 con 0x56202670b8e0 2024-04-21T08:45:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c367fd740 10 client.0 _ll_register_callbacks cb 0x5620265f3b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c31510640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670c290 0x562026711aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c31510640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670c290 0x562026711aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55590/0 (socket says 172.21.15.7:55590) 2024-04-21T08:45:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c31510640 1 -- 172.21.15.7:0/1498218488 learned_addr learned my addr 172.21.15.7:0/1498218488 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c31d11640 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x562026714710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c32512640 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562026714e10 0x562026712020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c31510640 1 -- 172.21.15.7:0/1498218488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 msgr2=0x562026714710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c31510640 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x562026714710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c31510640 1 -- 172.21.15.7:0/1498218488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562026714e10 msgr2=0x562026712020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c31510640 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562026714e10 0x562026712020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c31510640 1 -- 172.21.15.7:0/1498218488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56202637e3f0 con 0x56202670c290 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c157fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c31510640 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670c290 0x562026711aa0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c06b670 tx=0x7f1c1c013b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f679c5660f3a1071 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c157fa640 1 -- 172.21.15.7:0/1498218488 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c011dd0 con 0x56202670c290 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c157fa640 1 -- 172.21.15.7:0/1498218488 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f1c1c004020 con 0x56202670c290 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.761+0000 7f1c367fd740 1 -- 172.21.15.7:0/1498218488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c28050070 con 0x56202670c290 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c157fa640 1 -- 172.21.15.7:0/1498218488 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1c1c004220 con 0x56202670c290 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c367fd740 10 client.15882 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c367fd740 1 -- 172.21.15.7:0/1498218488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5620264487d0 con 0x56202670c290 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c157fa640 1 -- 172.21.15.7:0/1498218488 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c28050070 con 0x56202670c290 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c157fa640 1 -- 172.21.15.7:0/1498218488 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1c1c01a030 con 0x56202670c290 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c367fd740 10 client.15882 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c367fd740 10 client.15882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c367fd740 20 client.15882 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562026725100 0x562026727520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c367fd740 4 client.15882 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c367fd740 1 -- 172.21.15.7:0/1498218488 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56202660b930 con 0x562026725100 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.762+0000 7f1c31d11640 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562026725100 0x562026727520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.763+0000 7f1c157fa640 10 client.15882 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.763+0000 7f1c31d11640 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562026725100 0x562026727520 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dcdc2e6e836ebff server_cookie=14b1de51802e4ef0 in_seq=0 out_seq=0 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c157fa640 1 -- 172.21.15.7:0/1498218488 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56202660b930 con 0x562026725100 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c157fa640 10 client.15882 handle_command_reply: tid=0 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 1 client.15882 shutdown 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 1 -- 172.21.15.7:0/1498218488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670c290 msgr2=0x562026711aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670c290 0x562026711aa0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c06b670 tx=0x7f1c1c013b70 comp rx=0 tx=0).stop 2024-04-21T08:45:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 1 -- 172.21.15.7:0/1498218488 shutdown_connections 2024-04-21T08:45:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562026725100 0x562026727520 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562026714e10 0x562026712020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56202670c290 0x562026711aa0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c157fa640 0 client.15882 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1498218488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x562026714710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 1 -- 172.21.15.7:0/1498218488 >> 172.21.15.7:0/1498218488 conn(0x562026705fb0 msgr2=0x562026707d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 1 -- 172.21.15.7:0/1498218488 shutdown_connections 2024-04-21T08:45:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 1 -- 172.21.15.7:0/1498218488 wait complete. 2024-04-21T08:45:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.764+0000 7f1c367fd740 20 client.15882 trim_cache size 0 max 16384 2024-04-21T08:45:17.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c367fd740 1 Processor -- start 2024-04-21T08:45:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c367fd740 1 -- start start 2024-04-21T08:45:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x562026720c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 0x56202670cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x5620267141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c367fd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1ab220 con 0x562026714e10 2024-04-21T08:45:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c367fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1adaa0 con 0x56202670c290 2024-04-21T08:45:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c367fd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1ab890 con 0x56202670b8e0 2024-04-21T08:45:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c32512640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x5620267141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c31510640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 0x56202670cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c32512640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x5620267141d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55606/0 (socket says 172.21.15.7:55606) 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c32512640 1 -- 172.21.15.7:0/4080409296 learned_addr learned my addr 172.21.15.7:0/4080409296 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c31d11640 1 --2- 172.21.15.7:0/4080409296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x562026720c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c31510640 1 -- 172.21.15.7:0/4080409296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 msgr2=0x562026720c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c31510640 1 --2- 172.21.15.7:0/4080409296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x562026720c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c31510640 1 -- 172.21.15.7:0/4080409296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 msgr2=0x5620267141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.778+0000 7f1c31510640 1 --2- 172.21.15.7:0/4080409296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x5620267141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.779+0000 7f1c31510640 1 -- 172.21.15.7:0/4080409296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56202670c030 con 0x56202670c290 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.779+0000 7f1c32512640 1 --2- 172.21.15.7:0/4080409296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x5620267141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.779+0000 7f1c31510640 1 --2- 172.21.15.7:0/4080409296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 0x56202670cef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c004c10 tx=0x7f1c1c003670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f23f01dfc42ff55f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.779+0000 7f1c177fe640 1 -- 172.21.15.7:0/4080409296 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c011bd0 con 0x56202670c290 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.779+0000 7f1c177fe640 1 -- 172.21.15.7:0/4080409296 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c1c011d70 con 0x56202670c290 2024-04-21T08:45:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.779+0000 7f1c177fe640 1 -- 172.21.15.7:0/4080409296 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c0037e0 con 0x56202670c290 2024-04-21T08:45:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.780+0000 7f1c367fd740 1 -- 172.21.15.7:0/4080409296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 msgr2=0x56202670cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.780+0000 7f1c367fd740 1 --2- 172.21.15.7:0/4080409296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 0x56202670cef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c004c10 tx=0x7f1c1c003670 comp rx=0 tx=0).stop 2024-04-21T08:45:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.780+0000 7f1c367fd740 1 -- 172.21.15.7:0/4080409296 shutdown_connections 2024-04-21T08:45:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.780+0000 7f1c367fd740 1 --2- 172.21.15.7:0/4080409296 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x5620267141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.780+0000 7f1c367fd740 1 --2- 172.21.15.7:0/4080409296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 0x56202670cef0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.780+0000 7f1c367fd740 1 --2- 172.21.15.7:0/4080409296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x562026720c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.780+0000 7f1c367fd740 1 -- 172.21.15.7:0/4080409296 >> 172.21.15.7:0/4080409296 conn(0x562026705fb0 msgr2=0x56202670c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.780+0000 7f1c367fd740 1 -- 172.21.15.7:0/4080409296 shutdown_connections 2024-04-21T08:45:17.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.780+0000 7f1c367fd740 1 -- 172.21.15.7:0/4080409296 wait complete. 2024-04-21T08:45:17.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.780+0000 7f1c367fd740 1 -- start start 2024-04-21T08:45:17.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.780+0000 7f1c367fd740 1 Processor -- start 2024-04-21T08:45:17.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x56202671d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 0x56202670e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c367fd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x56202670e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c367fd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1abf60 con 0x562026714e10 2024-04-21T08:45:17.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c367fd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1ad3d0 con 0x56202670c290 2024-04-21T08:45:17.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c367fd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2c1ac630 con 0x56202670b8e0 2024-04-21T08:45:17.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c367fd740 10 client.0 _ll_register_callbacks cb 0x7f1c1c009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:17.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c31510640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 0x56202670e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c32512640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x56202670e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c31510640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 0x56202670e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33372/0 (socket says 172.21.15.7:33372) 2024-04-21T08:45:17.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c32512640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x56202670e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55618/0 (socket says 172.21.15.7:55618) 2024-04-21T08:45:17.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c31510640 1 -- 172.21.15.7:0/1030856935 learned_addr learned my addr 172.21.15.7:0/1030856935 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c31d11640 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x56202671d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c32512640 1 -- 172.21.15.7:0/1030856935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 msgr2=0x56202671d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c32512640 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x56202671d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c32512640 1 -- 172.21.15.7:0/1030856935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 msgr2=0x56202670e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c32512640 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 0x56202670e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c32512640 1 -- 172.21.15.7:0/1030856935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56202670ee80 con 0x562026714e10 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c31d11640 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x56202671d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c31510640 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 0x56202670e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c32512640 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x56202670e810 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1c2409cb80 tx=0x7f1c2409d2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a605500eb7da11 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.781+0000 7f1c157fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c157fa640 1 -- 172.21.15.7:0/1030856935 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c240b6070 con 0x562026714e10 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c367fd740 1 -- 172.21.15.7:0/1030856935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c1c011d70 con 0x562026714e10 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c157fa640 1 -- 172.21.15.7:0/1030856935 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f1c240a53d0 con 0x562026714e10 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c157fa640 1 -- 172.21.15.7:0/1030856935 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1c240a5860 con 0x562026714e10 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c367fd740 10 client.15888 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c367fd740 1 -- 172.21.15.7:0/1030856935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56202678b580 con 0x562026714e10 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c157fa640 1 -- 172.21.15.7:0/1030856935 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c1c011d70 con 0x562026714e10 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c157fa640 1 -- 172.21.15.7:0/1030856935 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1c24098940 con 0x562026714e10 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c367fd740 10 client.15888 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c367fd740 10 client.15888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c367fd740 20 client.15888 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56202678bcb0 0x56202678e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c367fd740 4 client.15888 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.782+0000 7f1c367fd740 1 -- 172.21.15.7:0/1030856935 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562026679c10 con 0x56202678bcb0 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c31d11640 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56202678bcb0 0x56202678e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c157fa640 10 client.15888 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c31d11640 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56202678bcb0 0x56202678e0d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c5f56d13cfe14e1 server_cookie=865a47cf08dcd66 in_seq=0 out_seq=0 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c157fa640 1 -- 172.21.15.7:0/1030856935 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562026679c10 con 0x56202678bcb0 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c157fa640 10 client.15888 handle_command_reply: tid=0 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c367fd740 1 client.15888 shutdown 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c367fd740 1 -- 172.21.15.7:0/1030856935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 msgr2=0x56202670e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x56202670e810 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1c2409cb80 tx=0x7f1c2409d2f0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c367fd740 1 -- 172.21.15.7:0/1030856935 shutdown_connections 2024-04-21T08:45:17.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56202678bcb0 0x56202678e0d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562026714e10 0x56202670e810 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56202670c290 0x56202670e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c367fd740 1 --2- 172.21.15.7:0/1030856935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56202670b8e0 0x56202671d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c367fd740 1 -- 172.21.15.7:0/1030856935 >> 172.21.15.7:0/1030856935 conn(0x562026705fb0 msgr2=0x562026700c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c157fa640 0 client.15888 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:17.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c367fd740 1 -- 172.21.15.7:0/1030856935 shutdown_connections 2024-04-21T08:45:17.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.783+0000 7f1c367fd740 1 -- 172.21.15.7:0/1030856935 wait complete. 2024-04-21T08:45:17.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c367fd740 20 client.15888 trim_cache size 0 max 16384 2024-04-21T08:45:17.794 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:45:17.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 -- 172.21.15.7:0/3885365286 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1c180a3f90 msgr2=0x7f1c180a6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1c180a3f90 0x7f1c180a6450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c28005ab0 tx=0x7f1c28005a20 comp rx=0 tx=0).stop 2024-04-21T08:45:17.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 -- 172.21.15.7:0/3885365286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c2c1985b0 msgr2=0x7f1c2c198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c2c1985b0 0x7f1c2c198a60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400ea60 tx=0x7f1c2400ef20 comp rx=0 tx=0).stop 2024-04-21T08:45:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 -- 172.21.15.7:0/3885365286 shutdown_connections 2024-04-21T08:45:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c2c1985b0 0x7f1c2c198a60 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2c106160 0x7f1c2c19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1c180a3f90 0x7f1c180a6450 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 --2- 172.21.15.7:0/3885365286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c2c101090 0x7f1c2c19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 -- 172.21.15.7:0/3885365286 >> 172.21.15.7:0/3885365286 conn(0x7f1c2c078000 msgr2=0x7f1c2c069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 -- 172.21.15.7:0/3885365286 shutdown_connections 2024-04-21T08:45:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:17.784+0000 7f1c33f75640 1 -- 172.21.15.7:0/3885365286 wait complete. 2024-04-21T08:45:18.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:17 smithi007 ceph-mon[28477]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:45:18.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:17 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1165804297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:18.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:17 smithi016 ceph-mon[31259]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:45:18.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:17 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1165804297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:18.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:17 smithi077 ceph-mon[31615]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:45:18.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:17 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1165804297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:18.798 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:45:18.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:18.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:45:18.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.929+0000 7f1ae93b6640 1 -- 172.21.15.7:0/1297985067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ae4102190 msgr2=0x7f1ae4102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.929+0000 7f1ae93b6640 1 --2- 172.21.15.7:0/1297985067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ae4102190 0x7f1ae4102570 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009900 tx=0x7f1ad402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.929+0000 7f1ae93b6640 1 -- 172.21.15.7:0/1297985067 shutdown_connections 2024-04-21T08:45:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.929+0000 7f1ae93b6640 1 --2- 172.21.15.7:0/1297985067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ae4107260 0x7f1ae410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.929+0000 7f1ae93b6640 1 --2- 172.21.15.7:0/1297985067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ae4102b40 0x7f1ae4106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.929+0000 7f1ae93b6640 1 --2- 172.21.15.7:0/1297985067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ae4102190 0x7f1ae4102570 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.929+0000 7f1ae93b6640 1 -- 172.21.15.7:0/1297985067 >> 172.21.15.7:0/1297985067 conn(0x7f1ae40fda10 msgr2=0x7f1ae40ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.929+0000 7f1ae93b6640 1 -- 172.21.15.7:0/1297985067 shutdown_connections 2024-04-21T08:45:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.929+0000 7f1ae93b6640 1 -- 172.21.15.7:0/1297985067 wait complete. 2024-04-21T08:45:18.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.929+0000 7f1ae93b6640 1 Processor -- start 2024-04-21T08:45:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae93b6640 1 -- start start 2024-04-21T08:45:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae93b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ae4102190 0x7f1ae419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae93b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ae4107260 0x7f1ae419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae93b6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ae4198690 0x7f1ae4198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae93b6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae41121d0 con 0x7f1ae4198690 2024-04-21T08:45:18.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae93b6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae4112050 con 0x7f1ae4107260 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae93b6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae4112350 con 0x7f1ae4102190 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ae4107260 0x7f1ae419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ae4107260 0x7f1ae419eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33384/0 (socket says 172.21.15.7:33384) 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ae4198690 0x7f1ae4198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae27fc640 1 -- 172.21.15.7:0/2145919151 learned_addr learned my addr 172.21.15.7:0/2145919151 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ae4102190 0x7f1ae419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae27fc640 1 -- 172.21.15.7:0/2145919151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ae4102190 msgr2=0x7f1ae419e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ae4102190 0x7f1ae419e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae27fc640 1 -- 172.21.15.7:0/2145919151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ae4198690 msgr2=0x7f1ae4198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ae4198690 0x7f1ae4198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.930+0000 7f1ae27fc640 1 -- 172.21.15.7:0/2145919151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae4199290 con 0x7f1ae4107260 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.931+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ae4107260 0x7f1ae419eac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800e9c0 tx=0x7f1ad800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:18.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.931+0000 7f1acbfff640 1 -- 172.21.15.7:0/2145919151 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad800cd80 con 0x7f1ae4107260 2024-04-21T08:45:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.931+0000 7f1acbfff640 1 -- 172.21.15.7:0/2145919151 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ad80054d0 con 0x7f1ae4107260 2024-04-21T08:45:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.931+0000 7f1ae93b6640 1 -- 172.21.15.7:0/2145919151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae41a5320 con 0x7f1ae4107260 2024-04-21T08:45:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.931+0000 7f1acbfff640 1 -- 172.21.15.7:0/2145919151 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8010430 con 0x7f1ae4107260 2024-04-21T08:45:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.931+0000 7f1ae93b6640 1 -- 172.21.15.7:0/2145919151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae41a56d0 con 0x7f1ae4107260 2024-04-21T08:45:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.933+0000 7f1acbfff640 1 -- 172.21.15.7:0/2145919151 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f1ad8004fd0 con 0x7f1ae4107260 2024-04-21T08:45:18.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.934+0000 7f1acbfff640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1acc0a3e40 0x7f1acc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.934+0000 7f1aebfaa740 1 Processor -- start 2024-04-21T08:45:18.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.934+0000 7f1aebfaa740 1 -- start start 2024-04-21T08:45:18.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1acbfff640 1 -- 172.21.15.7:0/2145919151 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f1ad80d12a0 con 0x7f1ae4107260 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1acc0a3e40 0x7f1acc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1aebfaa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606590 0x55871c606970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1aebfaa740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606eb0 0x55871c618420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1aebfaa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c618960 0x55871c61ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1aebfaa740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55871c56af00 con 0x55871c606eb0 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1aebfaa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55871c508590 con 0x55871c618960 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1aebfaa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55871c566ec0 con 0x55871c606590 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae27fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606eb0 0x55871c618420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae27fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606eb0 0x55871c618420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55650/0 (socket says 172.21.15.7:55650) 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae27fc640 1 -- 172.21.15.7:0/3228806558 learned_addr learned my addr 172.21.15.7:0/3228806558 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/3228806558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606590 0x55871c606970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1acc0a3e40 0x7f1acc0a6300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4006010 tx=0x7f1ad4005e40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae37fe640 1 --2- 172.21.15.7:0/3228806558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c618960 0x55871c61ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae27fc640 1 -- 172.21.15.7:0/3228806558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606590 msgr2=0x55871c606970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/3228806558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606590 0x55871c606970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae27fc640 1 -- 172.21.15.7:0/3228806558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c618960 msgr2=0x55871c61ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/3228806558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c618960 0x55871c61ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.935+0000 7f1ae27fc640 1 -- 172.21.15.7:0/3228806558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55871c3457e0 con 0x55871c606eb0 2024-04-21T08:45:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.936+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/3228806558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606590 0x55871c606970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.936+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/3228806558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606eb0 0x55871c618420 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80995d0 tx=0x7f1ad809c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c4aa765e1400158 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.936+0000 7f1ac97fa640 1 -- 172.21.15.7:0/3228806558 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad80b6070 con 0x55871c606eb0 2024-04-21T08:45:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.936+0000 7f1ac97fa640 1 -- 172.21.15.7:0/3228806558 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ad809cc00 con 0x55871c606eb0 2024-04-21T08:45:18.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.936+0000 7f1ac97fa640 1 -- 172.21.15.7:0/3228806558 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad80bd0d0 con 0x55871c606eb0 2024-04-21T08:45:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.937+0000 7f1aebfaa740 1 -- 172.21.15.7:0/3228806558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606eb0 msgr2=0x55871c618420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.937+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/3228806558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606eb0 0x55871c618420 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80995d0 tx=0x7f1ad809c560 comp rx=0 tx=0).stop 2024-04-21T08:45:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.937+0000 7f1aebfaa740 1 -- 172.21.15.7:0/3228806558 shutdown_connections 2024-04-21T08:45:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.937+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/3228806558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c618960 0x55871c61ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.937+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/3228806558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606eb0 0x55871c618420 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.937+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/3228806558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606590 0x55871c606970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.937+0000 7f1aebfaa740 1 -- 172.21.15.7:0/3228806558 >> 172.21.15.7:0/3228806558 conn(0x55871c600fb0 msgr2=0x55871c601390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.937+0000 7f1aebfaa740 1 -- 172.21.15.7:0/3228806558 shutdown_connections 2024-04-21T08:45:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.937+0000 7f1aebfaa740 1 -- 172.21.15.7:0/3228806558 wait complete. 2024-04-21T08:45:18.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.937+0000 7f1aebfaa740 1 -- start start 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.937+0000 7f1aebfaa740 1 Processor -- start 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1aebfaa740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c615980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1aebfaa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606eb0 0x55871c615ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1aebfaa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c618960 0x55871c616400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1ae2ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c615980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1ae2ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c615980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55652/0 (socket says 172.21.15.7:55652) 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1ae2ffd640 1 -- 172.21.15.7:0/2260111374 learned_addr learned my addr 172.21.15.7:0/2260111374 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1aebfaa740 1 -- 172.21.15.7:0/2260111374 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55871c61b590 con 0x55871c606590 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1aebfaa740 1 -- 172.21.15.7:0/2260111374 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55871c524580 con 0x55871c606eb0 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1aebfaa740 1 -- 172.21.15.7:0/2260111374 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55871c61b710 con 0x55871c618960 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606eb0 0x55871c615ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1aebfaa740 10 client.0 _ll_register_callbacks cb 0x55871c52a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1ae37fe640 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c618960 0x55871c616400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1ae2ffd640 1 -- 172.21.15.7:0/2260111374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c618960 msgr2=0x55871c616400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c618960 0x55871c616400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1ae2ffd640 1 -- 172.21.15.7:0/2260111374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606eb0 msgr2=0x55871c615ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606eb0 0x55871c615ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1ae2ffd640 1 -- 172.21.15.7:0/2260111374 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55871c3a5f70 con 0x55871c606590 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.938+0000 7f1ae37fe640 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c618960 0x55871c616400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.939+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c615980 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009900 tx=0x7f1ad4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f2f8c0ab869325b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.939+0000 7f1ac6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.939+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/2260111374 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4050030 con 0x55871c606590 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.939+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/2260111374 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f1ad403a040 con 0x55871c606590 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.939+0000 7f1aebfaa740 1 -- 172.21.15.7:0/2260111374 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ad809cc00 con 0x55871c606590 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.939+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/2260111374 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1ad405d050 con 0x55871c606590 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.939+0000 7f1aebfaa740 10 client.15900 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:18.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.939+0000 7f1aebfaa740 1 -- 172.21.15.7:0/2260111374 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55871c23d170 con 0x55871c606590 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.939+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/2260111374 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ad809cc00 con 0x55871c606590 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.940+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/2260111374 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1ad4003950 con 0x55871c606590 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.940+0000 7f1aebfaa740 10 client.15900 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.940+0000 7f1aebfaa740 10 client.15900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.940+0000 7f1aebfaa740 20 client.15900 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.940+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55871c628ba0 0x55871c62b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.940+0000 7f1aebfaa740 4 client.15900 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.940+0000 7f1aebfaa740 1 -- 172.21.15.7:0/2260111374 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55871c578920 con 0x55871c628ba0 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.940+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55871c628ba0 0x55871c62b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.940+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55871c628ba0 0x55871c62b010 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=859454aa0435cbb4 server_cookie=14bb8866f50279b1 in_seq=0 out_seq=0 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.940+0000 7f1ac6ff5640 10 client.15900 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.941+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/2260111374 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55871c578920 con 0x55871c628ba0 2024-04-21T08:45:18.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.941+0000 7f1ac6ff5640 10 client.15900 handle_command_reply: tid=0 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.941+0000 7f1aebfaa740 1 client.15900 shutdown 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.941+0000 7f1aebfaa740 1 -- 172.21.15.7:0/2260111374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 msgr2=0x55871c615980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.941+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c615980 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009900 tx=0x7f1ad4009260 comp rx=0 tx=0).stop 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.942+0000 7f1aebfaa740 1 -- 172.21.15.7:0/2260111374 shutdown_connections 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.942+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55871c628ba0 0x55871c62b010 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.942+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c618960 0x55871c616400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.942+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606eb0 0x55871c615ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.942+0000 7f1ac6ff5640 0 client.15900 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.942+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/2260111374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c615980 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.942+0000 7f1aebfaa740 1 -- 172.21.15.7:0/2260111374 >> 172.21.15.7:0/2260111374 conn(0x55871c600fb0 msgr2=0x55871c602d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.942+0000 7f1aebfaa740 1 -- 172.21.15.7:0/2260111374 shutdown_connections 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.942+0000 7f1aebfaa740 1 -- 172.21.15.7:0/2260111374 wait complete. 2024-04-21T08:45:18.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.942+0000 7f1aebfaa740 20 client.15900 trim_cache size 0 max 16384 2024-04-21T08:45:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1aebfaa740 1 Processor -- start 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1aebfaa740 1 -- start start 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1aebfaa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606590 0x55871c6180b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1aebfaa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606eb0 0x55871c6244c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1aebfaa740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c618960 0x55871c6308e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1aebfaa740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae41ab340 con 0x55871c618960 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1aebfaa740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae41adbc0 con 0x55871c606590 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1aebfaa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae41ab9b0 con 0x55871c606eb0 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1ae37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c618960 0x55871c6308e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1ae37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c618960 0x55871c6308e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55656/0 (socket says 172.21.15.7:55656) 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1ae37fe640 1 -- 172.21.15.7:0/3513864575 learned_addr learned my addr 172.21.15.7:0/3513864575 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/3513864575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606590 0x55871c6180b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.976+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/3513864575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606eb0 0x55871c6244c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.977+0000 7f1ae2ffd640 1 -- 172.21.15.7:0/3513864575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606eb0 msgr2=0x55871c6244c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.977+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/3513864575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606eb0 0x55871c6244c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.977+0000 7f1ae2ffd640 1 -- 172.21.15.7:0/3513864575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c618960 msgr2=0x55871c6308e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.977+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/3513864575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c618960 0x55871c6308e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.977+0000 7f1ae2ffd640 1 -- 172.21.15.7:0/3513864575 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55871c6075a0 con 0x55871c606590 2024-04-21T08:45:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.977+0000 7f1ae37fe640 1 --2- 172.21.15.7:0/3513864575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c618960 0x55871c6308e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.977+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/3513864575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606eb0 0x55871c6244c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.977+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/3513864575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606590 0x55871c6180b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009850 tx=0x7f1ad402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6102fd88129b8d11 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.977+0000 7f1ac8ff9640 1 -- 172.21.15.7:0/3513864575 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4050050 con 0x55871c606590 2024-04-21T08:45:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.977+0000 7f1ac8ff9640 1 -- 172.21.15.7:0/3513864575 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ad4045070 con 0x55871c606590 2024-04-21T08:45:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.977+0000 7f1ac8ff9640 1 -- 172.21.15.7:0/3513864575 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4004cb0 con 0x55871c606590 2024-04-21T08:45:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.978+0000 7f1aebfaa740 1 -- 172.21.15.7:0/3513864575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606590 msgr2=0x55871c6180b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.978+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/3513864575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606590 0x55871c6180b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009850 tx=0x7f1ad402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.978+0000 7f1aebfaa740 1 -- 172.21.15.7:0/3513864575 shutdown_connections 2024-04-21T08:45:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.978+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/3513864575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c618960 0x55871c6308e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.978+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/3513864575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c606eb0 0x55871c6244c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.978+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/3513864575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606590 0x55871c6180b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.978+0000 7f1aebfaa740 1 -- 172.21.15.7:0/3513864575 >> 172.21.15.7:0/3513864575 conn(0x55871c600fb0 msgr2=0x55871c602c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.978+0000 7f1aebfaa740 1 -- 172.21.15.7:0/3513864575 shutdown_connections 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.978+0000 7f1aebfaa740 1 -- 172.21.15.7:0/3513864575 wait complete. 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.979+0000 7f1aebfaa740 1 -- start start 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.979+0000 7f1aebfaa740 1 Processor -- start 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.979+0000 7f1aebfaa740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c620980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.979+0000 7f1ae2ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c620980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.979+0000 7f1ae2ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c620980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55658/0 (socket says 172.21.15.7:55658) 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.979+0000 7f1aebfaa740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606eb0 0x55871c620ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1ae2ffd640 1 -- 172.21.15.7:0/114570932 learned_addr learned my addr 172.21.15.7:0/114570932 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c618960 0x55871c61d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1aebfaa740 1 -- 172.21.15.7:0/114570932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae41ac080 con 0x55871c606590 2024-04-21T08:45:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1aebfaa740 1 -- 172.21.15.7:0/114570932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae41ad4f0 con 0x55871c606eb0 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1aebfaa740 1 -- 172.21.15.7:0/114570932 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae41ac750 con 0x55871c618960 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1ae37fe640 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c618960 0x55871c61d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1aebfaa740 10 client.0 _ll_register_callbacks cb 0x7f1ad4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606eb0 0x55871c620ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1ae2ffd640 1 -- 172.21.15.7:0/114570932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c618960 msgr2=0x55871c61d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c618960 0x55871c61d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1ae2ffd640 1 -- 172.21.15.7:0/114570932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606eb0 msgr2=0x55871c620ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606eb0 0x55871c620ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1ae2ffd640 1 -- 172.21.15.7:0/114570932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55871c61dfd0 con 0x55871c606590 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1ae2ffd640 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c620980 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4003820 tx=0x7f1ad4003850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2bade98e9efb9d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.980+0000 7f1ac6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/114570932 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4050040 con 0x55871c606590 2024-04-21T08:45:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/114570932 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f1ad40048b0 con 0x55871c606590 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1aebfaa740 1 -- 172.21.15.7:0/114570932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ad4045070 con 0x55871c606590 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/114570932 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1ad4072050 con 0x55871c606590 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1aebfaa740 10 client.15906 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1aebfaa740 1 -- 172.21.15.7:0/114570932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55871c68fc30 con 0x55871c606590 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/114570932 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ad4045070 con 0x55871c606590 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/114570932 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1ad4041840 con 0x55871c606590 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1aebfaa740 10 client.15906 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1aebfaa740 10 client.15906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1aebfaa740 20 client.15906 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.981+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55871c690240 0x55871c692660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.982+0000 7f1aebfaa740 4 client.15906 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.982+0000 7f1aebfaa740 1 -- 172.21.15.7:0/114570932 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55871c572790 con 0x55871c690240 2024-04-21T08:45:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.982+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55871c690240 0x55871c692660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.982+0000 7f1ac6ff5640 10 client.15906 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.982+0000 7f1ae27fc640 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55871c690240 0x55871c692660 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6095c70410243e79 server_cookie=34431023eca5cf50 in_seq=0 out_seq=0 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.983+0000 7f1ac6ff5640 1 -- 172.21.15.7:0/114570932 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55871c572790 con 0x55871c690240 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.983+0000 7f1ac6ff5640 10 client.15906 handle_command_reply: tid=0 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.983+0000 7f1aebfaa740 1 client.15906 shutdown 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1aebfaa740 1 -- 172.21.15.7:0/114570932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 msgr2=0x55871c620980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c620980 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4003820 tx=0x7f1ad4003850 comp rx=0 tx=0).stop 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1aebfaa740 1 -- 172.21.15.7:0/114570932 shutdown_connections 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55871c690240 0x55871c692660 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55871c618960 0x55871c61d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1ac6ff5640 0 client.15906 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55871c606eb0 0x55871c620ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1aebfaa740 1 --2- 172.21.15.7:0/114570932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55871c606590 0x55871c620980 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1aebfaa740 1 -- 172.21.15.7:0/114570932 >> 172.21.15.7:0/114570932 conn(0x55871c600fb0 msgr2=0x55871c6023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1aebfaa740 1 -- 172.21.15.7:0/114570932 shutdown_connections 2024-04-21T08:45:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1aebfaa740 1 -- 172.21.15.7:0/114570932 wait complete. 2024-04-21T08:45:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.984+0000 7f1aebfaa740 20 client.15906 trim_cache size 0 max 16384 2024-04-21T08:45:18.989 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:45:18.989 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:45:18.989 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "eb254f36-82c9-4750-b31b-79d1ce10b774", 2024-04-21T08:45:18.989 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:45:18.989 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:45:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 -- 172.21.15.7:0/2145919151 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1acc0a3e40 msgr2=0x7f1acc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1acc0a3e40 0x7f1acc0a6300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4006010 tx=0x7f1ad4005e40 comp rx=0 tx=0).stop 2024-04-21T08:45:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 -- 172.21.15.7:0/2145919151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ae4107260 msgr2=0x7f1ae419eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ae4107260 0x7f1ae419eac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800e9c0 tx=0x7f1ad800ee80 comp rx=0 tx=0).stop 2024-04-21T08:45:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 -- 172.21.15.7:0/2145919151 shutdown_connections 2024-04-21T08:45:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ae4198690 0x7f1ae4198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ae4107260 0x7f1ae419eac0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1acc0a3e40 0x7f1acc0a6300 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 --2- 172.21.15.7:0/2145919151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ae4102190 0x7f1ae419e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 -- 172.21.15.7:0/2145919151 >> 172.21.15.7:0/2145919151 conn(0x7f1ae40fda10 msgr2=0x7f1ae40fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 -- 172.21.15.7:0/2145919151 shutdown_connections 2024-04-21T08:45:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:18.985+0000 7f1ae93b6640 1 -- 172.21.15.7:0/2145919151 wait complete. 2024-04-21T08:45:18.997 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:45:18.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:45:19.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.105+0000 7f21c1cf2640 1 -- 172.21.15.7:0/1512929392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21bc10e8f0 msgr2=0x7f21bc110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:19.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.105+0000 7f21c1cf2640 1 --2- 172.21.15.7:0/1512929392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21bc10e8f0 0x7f21bc110ce0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f21b4009ca0 tx=0x7f21b402c180 comp rx=0 tx=0).stop 2024-04-21T08:45:19.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.105+0000 7f21c1cf2640 1 -- 172.21.15.7:0/1512929392 shutdown_connections 2024-04-21T08:45:19.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.105+0000 7f21c1cf2640 1 --2- 172.21.15.7:0/1512929392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21bc10e8f0 0x7f21bc110ce0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.105+0000 7f21c1cf2640 1 --2- 172.21.15.7:0/1512929392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21bc069c10 0x7f21bc10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.105+0000 7f21c1cf2640 1 --2- 172.21.15.7:0/1512929392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc0692f0 0x7f21bc0696d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.105+0000 7f21c1cf2640 1 -- 172.21.15.7:0/1512929392 >> 172.21.15.7:0/1512929392 conn(0x7f21bc077fa0 msgr2=0x7f21bc0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:19.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.105+0000 7f21c1cf2640 1 -- 172.21.15.7:0/1512929392 shutdown_connections 2024-04-21T08:45:19.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.105+0000 7f21c1cf2640 1 -- 172.21.15.7:0/1512929392 wait complete. 2024-04-21T08:45:19.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21c1cf2640 1 Processor -- start 2024-04-21T08:45:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21c1cf2640 1 -- start start 2024-04-21T08:45:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21c1cf2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21bc0692f0 0x7f21bc1a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21c1cf2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21bc069c10 0x7f21bc1a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21c1cf2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc10e8f0 0x7f21bc19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21c1cf2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc114330 con 0x7f21bc0692f0 2024-04-21T08:45:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21c1cf2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1141b0 con 0x7f21bc10e8f0 2024-04-21T08:45:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21c1cf2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1144b0 con 0x7f21bc069c10 2024-04-21T08:45:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21bc0692f0 0x7f21bc1a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21bc0692f0 0x7f21bc1a0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55672/0 (socket says 172.21.15.7:55672) 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21bb7fe640 1 -- 172.21.15.7:0/2029136701 learned_addr learned my addr 172.21.15.7:0/2029136701 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.106+0000 7f21baffd640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21bc069c10 0x7f21bc1a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21bbfff640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc10e8f0 0x7f21bc19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21bbfff640 1 -- 172.21.15.7:0/2029136701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21bc069c10 msgr2=0x7f21bc1a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21bbfff640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21bc069c10 0x7f21bc1a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21bbfff640 1 -- 172.21.15.7:0/2029136701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21bc0692f0 msgr2=0x7f21bc1a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21bbfff640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21bc0692f0 0x7f21bc1a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21bbfff640 1 -- 172.21.15.7:0/2029136701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21bc19ad70 con 0x7f21bc10e8f0 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21baffd640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21bc069c10 0x7f21bc1a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21bb7fe640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21bc0692f0 0x7f21bc1a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21bbfff640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc10e8f0 0x7f21bc19a5f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f21b402c790 tx=0x7f21b4003c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21b8ff9640 1 -- 172.21.15.7:0/2029136701 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b4039040 con 0x7f21bc10e8f0 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21b8ff9640 1 -- 172.21.15.7:0/2029136701 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b4044070 con 0x7f21bc10e8f0 2024-04-21T08:45:19.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21c1cf2640 1 -- 172.21.15.7:0/2029136701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21bc19b000 con 0x7f21bc10e8f0 2024-04-21T08:45:19.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.107+0000 7f21b8ff9640 1 -- 172.21.15.7:0/2029136701 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b4032950 con 0x7f21bc10e8f0 2024-04-21T08:45:19.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.108+0000 7f21c1cf2640 1 -- 172.21.15.7:0/2029136701 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21bc1a7300 con 0x7f21bc10e8f0 2024-04-21T08:45:19.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.110+0000 7f21c49be740 1 Processor -- start 2024-04-21T08:45:19.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21b8ff9640 1 -- 172.21.15.7:0/2029136701 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f21b4032af0 con 0x7f21bc10e8f0 2024-04-21T08:45:19.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21c49be740 1 -- start start 2024-04-21T08:45:19.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21c49be740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee40d90 0x56081ee41170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21c49be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee416b0 0x56081ee49d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21c49be740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee4a290 0x56081ee4c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21c49be740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56081ed9d1e0 con 0x56081ee40d90 2024-04-21T08:45:19.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21c49be740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56081ed3a390 con 0x56081ee416b0 2024-04-21T08:45:19.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21c49be740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56081ed991a0 con 0x56081ee4a290 2024-04-21T08:45:19.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21baffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee40d90 0x56081ee41170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21b8ff9640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21a40a3e40 0x7f21a40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21baffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee40d90 0x56081ee41170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55688/0 (socket says 172.21.15.7:55688) 2024-04-21T08:45:19.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.111+0000 7f21baffd640 1 -- 172.21.15.7:0/532293464 learned_addr learned my addr 172.21.15.7:0/532293464 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21bb7fe640 1 --2- 172.21.15.7:0/532293464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee416b0 0x56081ee49d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21b8ff9640 1 -- 172.21.15.7:0/2029136701 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f21b40f3270 con 0x7f21bc10e8f0 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21bbfff640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21a40a3e40 0x7f21a40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21baffd640 1 -- 172.21.15.7:0/532293464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee4a290 msgr2=0x56081ee4c680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21baffd640 1 --2- 172.21.15.7:0/532293464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee4a290 0x56081ee4c680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21baffd640 1 -- 172.21.15.7:0/532293464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee416b0 msgr2=0x56081ee49d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21baffd640 1 --2- 172.21.15.7:0/532293464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee416b0 0x56081ee49d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21baffd640 1 -- 172.21.15.7:0/532293464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56081eb797d0 con 0x56081ee40d90 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21baffd640 1 --2- 172.21.15.7:0/532293464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee40d90 0x56081ee41170 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f21b0010400 tx=0x7f21b00108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11f8732354f6382 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21bbfff640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21a40a3e40 0x7f21a40a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f21b40f05c0 tx=0x7f21b40095c0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21a1ffb640 1 -- 172.21.15.7:0/532293464 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b0004220 con 0x56081ee40d90 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.112+0000 7f21a1ffb640 1 -- 172.21.15.7:0/532293464 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b00043c0 con 0x56081ee40d90 2024-04-21T08:45:19.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.113+0000 7f21a1ffb640 1 -- 172.21.15.7:0/532293464 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b00119b0 con 0x56081ee40d90 2024-04-21T08:45:19.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.113+0000 7f21c49be740 1 -- 172.21.15.7:0/532293464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee40d90 msgr2=0x56081ee41170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:19.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.113+0000 7f21c49be740 1 --2- 172.21.15.7:0/532293464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee40d90 0x56081ee41170 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f21b0010400 tx=0x7f21b00108c0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.113+0000 7f21c49be740 1 -- 172.21.15.7:0/532293464 shutdown_connections 2024-04-21T08:45:19.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.113+0000 7f21c49be740 1 --2- 172.21.15.7:0/532293464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee4a290 0x56081ee4c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.113+0000 7f21c49be740 1 --2- 172.21.15.7:0/532293464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee416b0 0x56081ee49d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.113+0000 7f21c49be740 1 --2- 172.21.15.7:0/532293464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee40d90 0x56081ee41170 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.113+0000 7f21c49be740 1 -- 172.21.15.7:0/532293464 >> 172.21.15.7:0/532293464 conn(0x56081ee3b4c0 msgr2=0x56081ee3b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:19.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.113+0000 7f21c49be740 1 -- 172.21.15.7:0/532293464 shutdown_connections 2024-04-21T08:45:19.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.113+0000 7f21c49be740 1 -- 172.21.15.7:0/532293464 wait complete. 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.114+0000 7f21c49be740 1 -- start start 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.114+0000 7f21c49be740 1 Processor -- start 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.114+0000 7f21c49be740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee38510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.114+0000 7f21c49be740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 0x56081ee38a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.114+0000 7f21c49be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee4a290 0x56081ee48d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.114+0000 7f21c49be740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56081ee327c0 con 0x56081ee416b0 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21c49be740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56081ed562f0 con 0x56081ee4a290 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21c49be740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56081ee32940 con 0x56081ee40d90 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 0x56081ee38a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee4a290 0x56081ee48d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21c49be740 10 client.0 _ll_register_callbacks cb 0x56081ed63580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee4a290 0x56081ee48d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33446/0 (socket says 172.21.15.7:33446) 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21baffd640 1 -- 172.21.15.7:0/1743707550 learned_addr learned my addr 172.21.15.7:0/1743707550 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21baffd640 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee38510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 0x56081ee38a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55698/0 (socket says 172.21.15.7:55698) 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21baffd640 1 -- 172.21.15.7:0/1743707550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 msgr2=0x56081ee38510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21baffd640 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee38510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21baffd640 1 -- 172.21.15.7:0/1743707550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 msgr2=0x56081ee38a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21baffd640 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 0x56081ee38a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21baffd640 1 -- 172.21.15.7:0/1743707550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56081eaaf3f0 con 0x56081ee4a290 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21baffd640 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee4a290 0x56081ee48d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f21b0011b50 tx=0x7f21b0013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f79c0c7c6d83219c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f219affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f219affd640 1 -- 172.21.15.7:0/1743707550 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b00035f0 con 0x56081ee4a290 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.115+0000 7f21c49be740 1 -- 172.21.15.7:0/1743707550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21b00043c0 con 0x56081ee4a290 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f219affd640 1 -- 172.21.15.7:0/1743707550 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f21b0003790 con 0x56081ee4a290 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f219affd640 1 -- 172.21.15.7:0/1743707550 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f21b0003990 con 0x56081ee4a290 2024-04-21T08:45:19.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f21c49be740 10 client.25511 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f21c49be740 1 -- 172.21.15.7:0/1743707550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56081ea6f0f0 con 0x56081ee4a290 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f219affd640 1 -- 172.21.15.7:0/1743707550 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b00043c0 con 0x56081ee4a290 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f219affd640 1 -- 172.21.15.7:0/1743707550 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f21b0029040 con 0x56081ee4a290 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f21c49be740 10 client.25511 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f21c49be740 10 client.25511 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f21c49be740 20 client.25511 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f21c49be740 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56081ee519b0 0x56081ee53da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f21c49be740 4 client.25511 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f21c49be740 1 -- 172.21.15.7:0/1743707550 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56081edaac00 con 0x56081ee519b0 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.116+0000 7f21bb7fe640 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56081ee519b0 0x56081ee53da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.117+0000 7f219affd640 10 client.25511 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.117+0000 7f21bb7fe640 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56081ee519b0 0x56081ee53da0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8b8eecd07abbf85 server_cookie=818592f2cad0e1ff in_seq=0 out_seq=0 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f219affd640 1 -- 172.21.15.7:0/1743707550 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56081edaac00 con 0x56081ee519b0 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f219affd640 10 client.25511 handle_command_reply: tid=0 2024-04-21T08:45:19.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 1 client.25511 shutdown 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 1 -- 172.21.15.7:0/1743707550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee4a290 msgr2=0x56081ee48d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee4a290 0x56081ee48d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f21b0011b50 tx=0x7f21b0013040 comp rx=0 tx=0).stop 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 1 -- 172.21.15.7:0/1743707550 shutdown_connections 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56081ee519b0 0x56081ee53da0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee4a290 0x56081ee48d40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 0x56081ee38a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 1 --2- 172.21.15.7:0/1743707550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee38510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 1 -- 172.21.15.7:0/1743707550 >> 172.21.15.7:0/1743707550 conn(0x56081ee3b4c0 msgr2=0x56081ee3d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f219affd640 0 client.25511 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 1 -- 172.21.15.7:0/1743707550 shutdown_connections 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 1 -- 172.21.15.7:0/1743707550 wait complete. 2024-04-21T08:45:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.118+0000 7f21c49be740 20 client.25511 trim_cache size 0 max 16384 2024-04-21T08:45:19.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.153+0000 7f21c49be740 1 Processor -- start 2024-04-21T08:45:19.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21c49be740 1 -- start start 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21c49be740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee38950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21c49be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee416b0 0x56081ee48ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21c49be740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee4a290 0x56081ee594e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21c49be740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1ab140 con 0x56081ee4a290 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21c49be740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1ad9c0 con 0x56081ee416b0 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21c49be740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1ab7b0 con 0x56081ee40d90 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21baffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee4a290 0x56081ee594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21baffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee4a290 0x56081ee594e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55710/0 (socket says 172.21.15.7:55710) 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21baffd640 1 -- 172.21.15.7:0/1236110462 learned_addr learned my addr 172.21.15.7:0/1236110462 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21baffd640 1 --2- 172.21.15.7:0/1236110462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee38950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.154+0000 7f21bb7fe640 1 --2- 172.21.15.7:0/1236110462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee416b0 0x56081ee48ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.155+0000 7f21baffd640 1 -- 172.21.15.7:0/1236110462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 msgr2=0x56081ee38950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:19.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.155+0000 7f21baffd640 1 --2- 172.21.15.7:0/1236110462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee38950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.155+0000 7f21baffd640 1 -- 172.21.15.7:0/1236110462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee416b0 msgr2=0x56081ee48ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:19.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.155+0000 7f21baffd640 1 --2- 172.21.15.7:0/1236110462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee416b0 0x56081ee48ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.155+0000 7f21baffd640 1 -- 172.21.15.7:0/1236110462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56081ee414e0 con 0x56081ee4a290 2024-04-21T08:45:19.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.155+0000 7f21bb7fe640 1 --2- 172.21.15.7:0/1236110462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee416b0 0x56081ee48ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:19.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.155+0000 7f21baffd640 1 --2- 172.21.15.7:0/1236110462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee4a290 0x56081ee594e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f21b00116e0 tx=0x7f21b00108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da6e3e418dd44a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:19.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.155+0000 7f21a17fa640 1 -- 172.21.15.7:0/1236110462 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b000bb20 con 0x56081ee4a290 2024-04-21T08:45:19.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.155+0000 7f21a17fa640 1 -- 172.21.15.7:0/1236110462 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b002a3b0 con 0x56081ee4a290 2024-04-21T08:45:19.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.155+0000 7f21a17fa640 1 -- 172.21.15.7:0/1236110462 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b0005680 con 0x56081ee4a290 2024-04-21T08:45:19.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.156+0000 7f21c49be740 1 -- 172.21.15.7:0/1236110462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee4a290 msgr2=0x56081ee594e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:19.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.156+0000 7f21c49be740 1 --2- 172.21.15.7:0/1236110462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee4a290 0x56081ee594e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f21b00116e0 tx=0x7f21b00108c0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.156+0000 7f21c49be740 1 -- 172.21.15.7:0/1236110462 shutdown_connections 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.156+0000 7f21c49be740 1 --2- 172.21.15.7:0/1236110462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee4a290 0x56081ee594e0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.156+0000 7f21c49be740 1 --2- 172.21.15.7:0/1236110462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee416b0 0x56081ee48ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.156+0000 7f21c49be740 1 --2- 172.21.15.7:0/1236110462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee38950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.156+0000 7f21c49be740 1 -- 172.21.15.7:0/1236110462 >> 172.21.15.7:0/1236110462 conn(0x56081ee3b4c0 msgr2=0x56081ee31d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.156+0000 7f21c49be740 1 -- 172.21.15.7:0/1236110462 shutdown_connections 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.156+0000 7f21c49be740 1 -- 172.21.15.7:0/1236110462 wait complete. 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.157+0000 7f21c49be740 1 -- start start 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.157+0000 7f21c49be740 1 Processor -- start 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.157+0000 7f21c49be740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee44210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.157+0000 7f21c49be740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 0x56081ee44750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.157+0000 7f21c49be740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee44c90 0x56081eeb6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21c49be740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1abe80 con 0x56081ee416b0 2024-04-21T08:45:19.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21c49be740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1ad2f0 con 0x56081ee44c90 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21c49be740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f21bc1ac550 con 0x56081ee40d90 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21c49be740 10 client.0 _ll_register_callbacks cb 0x7f21ac00bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 0x56081ee44750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 0x56081ee44750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55726/0 (socket says 172.21.15.7:55726) 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21bb7fe640 1 -- 172.21.15.7:0/327611633 learned_addr learned my addr 172.21.15.7:0/327611633 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21baffd640 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee44210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21baffd640 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee44c90 0x56081eeb6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21bb7fe640 1 -- 172.21.15.7:0/327611633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 msgr2=0x56081ee44210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21bb7fe640 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee44210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21bb7fe640 1 -- 172.21.15.7:0/327611633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee44c90 msgr2=0x56081eeb6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21bb7fe640 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee44c90 0x56081eeb6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21bb7fe640 1 -- 172.21.15.7:0/327611633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56081eeb7290 con 0x56081ee416b0 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21baffd640 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee44210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:19.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f21bb7fe640 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 0x56081ee44750 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f21ac005fd0 tx=0x7f21ac0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42fc2a20669eed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.158+0000 7f219affd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.159+0000 7f219affd640 1 -- 172.21.15.7:0/327611633 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac020620 con 0x56081ee416b0 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.159+0000 7f219affd640 1 -- 172.21.15.7:0/327611633 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f21ac0207c0 con 0x56081ee416b0 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.159+0000 7f21c49be740 1 -- 172.21.15.7:0/327611633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21b002a3b0 con 0x56081ee416b0 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.159+0000 7f219affd640 1 -- 172.21.15.7:0/327611633 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f21ac02f050 con 0x56081ee416b0 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.159+0000 7f21c49be740 10 client.15924 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.159+0000 7f21c49be740 1 -- 172.21.15.7:0/327611633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56081eeb8540 con 0x56081ee416b0 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.159+0000 7f219affd640 1 -- 172.21.15.7:0/327611633 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b002a3b0 con 0x56081ee416b0 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.159+0000 7f219affd640 1 -- 172.21.15.7:0/327611633 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f21ac011440 con 0x56081ee416b0 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.160+0000 7f21c49be740 10 client.15924 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.160+0000 7f21c49be740 10 client.15924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.160+0000 7f21c49be740 20 client.15924 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.160+0000 7f21c49be740 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56081eeb8d40 0x56081eebb160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:19.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.160+0000 7f21c49be740 4 client.15924 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.160+0000 7f21c49be740 1 -- 172.21.15.7:0/327611633 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56081eda4a70 con 0x56081eeb8d40 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.160+0000 7f21baffd640 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56081eeb8d40 0x56081eebb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.160+0000 7f219affd640 10 client.15924 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.160+0000 7f21baffd640 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56081eeb8d40 0x56081eebb160 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23cceb43359ff263 server_cookie=bff9bfcc4b59e8b9 in_seq=0 out_seq=0 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f219affd640 1 -- 172.21.15.7:0/327611633 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56081eda4a70 con 0x56081eeb8d40 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f219affd640 10 client.15924 handle_command_reply: tid=0 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f21c49be740 1 client.15924 shutdown 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f21c49be740 1 -- 172.21.15.7:0/327611633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 msgr2=0x56081ee44750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f21c49be740 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 0x56081ee44750 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f21ac005fd0 tx=0x7f21ac0039c0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f21c49be740 1 -- 172.21.15.7:0/327611633 shutdown_connections 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f21c49be740 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56081eeb8d40 0x56081eebb160 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f21c49be740 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56081ee44c90 0x56081eeb6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f21c49be740 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56081ee416b0 0x56081ee44750 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f21c49be740 1 --2- 172.21.15.7:0/327611633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56081ee40d90 0x56081ee44210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f21c49be740 1 -- 172.21.15.7:0/327611633 >> 172.21.15.7:0/327611633 conn(0x56081ee3b4c0 msgr2=0x56081ee36ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:19.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f219affd640 0 client.15924 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:19.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f21c49be740 1 -- 172.21.15.7:0/327611633 shutdown_connections 2024-04-21T08:45:19.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.161+0000 7f21c49be740 1 -- 172.21.15.7:0/327611633 wait complete. 2024-04-21T08:45:19.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.162+0000 7f21c49be740 20 client.15924 trim_cache size 0 max 16384 2024-04-21T08:45:19.166 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:45:19.166 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:45:19.166 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:45:19.166 INFO:teuthology.orchestra.run.smithi007.stdout: "eb254f36-82c9-4750-b31b-79d1ce10b774": { 2024-04-21T08:45:19.166 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:45:19.167 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "eb254f36-82c9-4750-b31b-79d1ce10b774", 2024-04-21T08:45:19.167 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:45:19.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:45:19.167 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:45:19.167 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:45:19.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.162+0000 7f21c1cf2640 1 -- 172.21.15.7:0/2029136701 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21a40a3e40 msgr2=0x7f21a40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:19.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.162+0000 7f21c1cf2640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21a40a3e40 0x7f21a40a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f21b40f05c0 tx=0x7f21b40095c0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.162+0000 7f21c1cf2640 1 -- 172.21.15.7:0/2029136701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc10e8f0 msgr2=0x7f21bc19a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:19.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.162+0000 7f21c1cf2640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc10e8f0 0x7f21bc19a5f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f21b402c790 tx=0x7f21b4003c20 comp rx=0 tx=0).stop 2024-04-21T08:45:19.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.162+0000 7f21c1cf2640 1 -- 172.21.15.7:0/2029136701 shutdown_connections 2024-04-21T08:45:19.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.162+0000 7f21c1cf2640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21bc10e8f0 0x7f21bc19a5f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.163+0000 7f21c1cf2640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f21a40a3e40 0x7f21a40a6300 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.163+0000 7f21c1cf2640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21bc069c10 0x7f21bc1a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.163+0000 7f21c1cf2640 1 --2- 172.21.15.7:0/2029136701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21bc0692f0 0x7f21bc1a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:19.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.163+0000 7f21c1cf2640 1 -- 172.21.15.7:0/2029136701 >> 172.21.15.7:0/2029136701 conn(0x7f21bc077fa0 msgr2=0x7f21bc10fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:19.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.163+0000 7f21c1cf2640 1 -- 172.21.15.7:0/2029136701 shutdown_connections 2024-04-21T08:45:19.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:19.163+0000 7f21c1cf2640 1 -- 172.21.15.7:0/2029136701 wait complete. 2024-04-21T08:45:19.177 INFO:tasks.cephfs.filesystem:scrub status for tag:eb254f36-82c9-4750-b31b-79d1ce10b774 - {'path': '/', 'tag': 'eb254f36-82c9-4750-b31b-79d1ce10b774', 'options': 'recursive,force'} 2024-04-21T08:45:20.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:20 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:45:20.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:20 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:20.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:20 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:45:20.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:20 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:20.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:20 smithi077 ceph-mon[31615]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:45:20.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:20 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:45:20.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:20 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:20.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:20 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:45:20.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:20 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:20.654 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:20 smithi007 ceph-mon[28477]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:45:20.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:20 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:45:20.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:20 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:20.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:20 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:45:20.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:20 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:20.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:20 smithi016 ceph-mon[31259]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:45:22.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:22 smithi077 ceph-mon[31615]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:45:22.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:22 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:45:22.606 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:22 smithi016 ceph-mon[31259]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:45:22.607 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:22 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:45:22.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:22 smithi007 ceph-mon[28477]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:45:22.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:22 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:45:24.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:24 smithi077 ceph-mon[31615]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T08:45:24.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:24 smithi007 ceph-mon[28477]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T08:45:24.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:24 smithi016 ceph-mon[31259]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T08:45:26.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:26 smithi077 ceph-mon[31615]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T08:45:26.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:26 smithi007 ceph-mon[28477]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T08:45:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:26 smithi016 ceph-mon[31259]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T08:45:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.873+0000 7f7007225640 1 -- 172.21.15.7:0/3816076950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7000101660 msgr2=0x7f700010e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.873+0000 7f7007225640 1 --2- 172.21.15.7:0/3816076950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7000101660 0x7f700010e3b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009900 tx=0x7f6fec02bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.873+0000 7f7007225640 1 -- 172.21.15.7:0/3816076950 shutdown_connections 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.873+0000 7f7007225640 1 --2- 172.21.15.7:0/3816076950 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f700010e8f0 0x7f7000110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.873+0000 7f7007225640 1 --2- 172.21.15.7:0/3816076950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7000101660 0x7f700010e3b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.873+0000 7f7007225640 1 --2- 172.21.15.7:0/3816076950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7000100d40 0x7f7000101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.873+0000 7f7007225640 1 -- 172.21.15.7:0/3816076950 >> 172.21.15.7:0/3816076950 conn(0x7f7000077f70 msgr2=0x7f7000078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.873+0000 7f7007225640 1 -- 172.21.15.7:0/3816076950 shutdown_connections 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.873+0000 7f7007225640 1 -- 172.21.15.7:0/3816076950 wait complete. 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f7007225640 1 Processor -- start 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f7007225640 1 -- start start 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f7007225640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7000100d40 0x7f70001a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f7007225640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7000101660 0x7f70001a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f7007225640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f700010e8f0 0x7f700019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f7007225640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7000114440 con 0x7f700010e8f0 2024-04-21T08:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f70057c2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f700010e8f0 0x7f700019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f70057c2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f700010e8f0 0x7f700019a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58422/0 (socket says 172.21.15.7:58422) 2024-04-21T08:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f7007225640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70001142c0 con 0x7f7000100d40 2024-04-21T08:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f70057c2640 1 -- 172.21.15.7:0/2177830090 learned_addr learned my addr 172.21.15.7:0/2177830090 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f7007225640 1 -- 172.21.15.7:0/2177830090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f70001145c0 con 0x7f7000101660 2024-04-21T08:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.874+0000 7f6ffffff640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7000101660 0x7f70001a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.875+0000 7f7004fc1640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7000100d40 0x7f70001a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.875+0000 7f70057c2640 1 -- 172.21.15.7:0/2177830090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7000101660 msgr2=0x7f70001a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.875+0000 7f70057c2640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7000101660 0x7f70001a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.875+0000 7f70057c2640 1 -- 172.21.15.7:0/2177830090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7000100d40 msgr2=0x7f70001a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.875+0000 7f70057c2640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7000100d40 0x7f70001a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.875+0000 7f70057c2640 1 -- 172.21.15.7:0/2177830090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f700019af50 con 0x7f700010e8f0 2024-04-21T08:45:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.875+0000 7f70057c2640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f700010e8f0 0x7f700019a780 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800eb10 tx=0x7f6ff800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.875+0000 7f6ffdffb640 1 -- 172.21.15.7:0/2177830090 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8004b20 con 0x7f700010e8f0 2024-04-21T08:45:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.875+0000 7f6ffdffb640 1 -- 172.21.15.7:0/2177830090 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ff8004cc0 con 0x7f700010e8f0 2024-04-21T08:45:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.875+0000 7f6ffdffb640 1 -- 172.21.15.7:0/2177830090 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff80107b0 con 0x7f700010e8f0 2024-04-21T08:45:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.875+0000 7f7007225640 1 -- 172.21.15.7:0/2177830090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f700019b240 con 0x7f700010e8f0 2024-04-21T08:45:27.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.876+0000 7f7007225640 1 -- 172.21.15.7:0/2177830090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70001a7460 con 0x7f700010e8f0 2024-04-21T08:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f6ffdffb640 1 -- 172.21.15.7:0/2177830090 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f6ff8005020 con 0x7f700010e8f0 2024-04-21T08:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f7009e11740 1 Processor -- start 2024-04-21T08:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f7009e11740 1 -- start start 2024-04-21T08:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f7009e11740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6e1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f7009e11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6e1b60 0x5624bb6e1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f7009e11740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6eb0e0 0x5624bb6ed4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f7009e11740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5624bb6468d0 con 0x5624bb6e28d0 2024-04-21T08:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f7009e11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5624bb5e3e80 con 0x5624bb6e1b60 2024-04-21T08:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f7009e11740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624bb642890 con 0x5624bb6eb0e0 2024-04-21T08:45:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f6ffffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6e1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f6ffffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6e1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58434/0 (socket says 172.21.15.7:58434) 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f6ffffff640 1 -- 172.21.15.7:0/3161633209 learned_addr learned my addr 172.21.15.7:0/3161633209 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f7004fc1640 1 --2- 172.21.15.7:0/3161633209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6e1b60 0x5624bb6e1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f6ffdffb640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ff00a3eb0 0x7f6ff00a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.879+0000 7f6ffffff640 1 --2- 172.21.15.7:0/3161633209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6eb0e0 0x5624bb6ed4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.880+0000 7f70057c2640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ff00a3eb0 0x7f6ff00a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.880+0000 7f6ffdffb640 1 -- 172.21.15.7:0/2177830090 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f6ff80d14e0 con 0x7f700010e8f0 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.880+0000 7f70057c2640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ff00a3eb0 0x7f6ff00a6370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80ce720 tx=0x7f6ff80c7f50 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.880+0000 7f7004fc1640 1 -- 172.21.15.7:0/3161633209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6eb0e0 msgr2=0x5624bb6ed4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.880+0000 7f7004fc1640 1 --2- 172.21.15.7:0/3161633209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6eb0e0 0x5624bb6ed4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.880+0000 7f7004fc1640 1 -- 172.21.15.7:0/3161633209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 msgr2=0x5624bb6e1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.880+0000 7f7004fc1640 1 --2- 172.21.15.7:0/3161633209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6e1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.880+0000 7f7004fc1640 1 -- 172.21.15.7:0/3161633209 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624bb605320 con 0x5624bb6e1b60 2024-04-21T08:45:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.880+0000 7f6ffffff640 1 --2- 172.21.15.7:0/3161633209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6e1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.881+0000 7f7004fc1640 1 --2- 172.21.15.7:0/3161633209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6e1b60 0x5624bb6e1fe0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400bbf0 tx=0x7f6ff40107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f45f1944cd6f69bf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.881+0000 7f6feaffd640 1 -- 172.21.15.7:0/3161633209 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4004030 con 0x5624bb6e1b60 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.881+0000 7f6feaffd640 1 -- 172.21.15.7:0/3161633209 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ff40035f0 con 0x5624bb6e1b60 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.881+0000 7f6feaffd640 1 -- 172.21.15.7:0/3161633209 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4011680 con 0x5624bb6e1b60 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.882+0000 7f7009e11740 1 -- 172.21.15.7:0/3161633209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6e1b60 msgr2=0x5624bb6e1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.882+0000 7f7009e11740 1 --2- 172.21.15.7:0/3161633209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6e1b60 0x5624bb6e1fe0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400bbf0 tx=0x7f6ff40107a0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.882+0000 7f7009e11740 1 -- 172.21.15.7:0/3161633209 shutdown_connections 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.882+0000 7f7009e11740 1 --2- 172.21.15.7:0/3161633209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6eb0e0 0x5624bb6ed4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.882+0000 7f7009e11740 1 --2- 172.21.15.7:0/3161633209 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6e1b60 0x5624bb6e1fe0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.882+0000 7f7009e11740 1 --2- 172.21.15.7:0/3161633209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6e1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.882+0000 7f7009e11740 1 -- 172.21.15.7:0/3161633209 >> 172.21.15.7:0/3161633209 conn(0x5624bb6a8c90 msgr2=0x5624bb6dc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.882+0000 7f7009e11740 1 -- 172.21.15.7:0/3161633209 shutdown_connections 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.882+0000 7f7009e11740 1 -- 172.21.15.7:0/3161633209 wait complete. 2024-04-21T08:45:27.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.882+0000 7f7009e11740 1 -- start start 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.882+0000 7f7009e11740 1 Processor -- start 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f7009e11740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f7009e11740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e28d0 0x5624bb6e7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f7009e11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 0x5624bb6e8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f7009e11740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5624bb6ede10 con 0x5624bb6e1b60 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f7009e11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5624bb5ceeb0 con 0x5624bb6eb0e0 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f7009e11740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624bb6edf90 con 0x5624bb6e28d0 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f7009e11740 10 client.0 _ll_register_callbacks cb 0x5624bb5fbb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f6ffffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f6ffffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6eaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58448/0 (socket says 172.21.15.7:58448) 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f6ffffff640 1 -- 172.21.15.7:0/2311325016 learned_addr learned my addr 172.21.15.7:0/2311325016 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f6ffffff640 1 -- 172.21.15.7:0/2311325016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e28d0 msgr2=0x5624bb6e7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f6ffffff640 1 --2- 172.21.15.7:0/2311325016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e28d0 0x5624bb6e7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f6ffffff640 1 -- 172.21.15.7:0/2311325016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 msgr2=0x5624bb6e8470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f6ffffff640 1 --2- 172.21.15.7:0/2311325016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 0x5624bb6e8470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.883+0000 7f6ffffff640 1 -- 172.21.15.7:0/2311325016 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624bb359210 con 0x5624bb6e1b60 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.884+0000 7f6ffffff640 1 --2- 172.21.15.7:0/2311325016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6eaad0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6fec036040 tx=0x7f6fec009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15888ef31aa251e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.884+0000 7f6fdffff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.884+0000 7f6fdffff640 1 -- 172.21.15.7:0/2311325016 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec005250 con 0x5624bb6e1b60 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.884+0000 7f6fdffff640 1 -- 172.21.15.7:0/2311325016 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f6fec03a040 con 0x5624bb6e1b60 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.884+0000 7f7009e11740 1 -- 172.21.15.7:0/2311325016 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ff40035f0 con 0x5624bb6e1b60 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.884+0000 7f6fdffff640 1 -- 172.21.15.7:0/2311325016 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6fec0053f0 con 0x5624bb6e1b60 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.884+0000 7f7009e11740 10 client.15936 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.884+0000 7f7009e11740 1 -- 172.21.15.7:0/2311325016 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5624bb425620 con 0x5624bb6e1b60 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.885+0000 7f6fdffff640 1 -- 172.21.15.7:0/2311325016 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ff40035f0 con 0x5624bb6e1b60 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.885+0000 7f6fdffff640 1 -- 172.21.15.7:0/2311325016 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6fec003660 con 0x5624bb6e1b60 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.885+0000 7f7009e11740 10 client.15936 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:27.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.885+0000 7f7009e11740 10 client.15936 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.885+0000 7f7009e11740 20 client.15936 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.885+0000 7f7009e11740 1 --2- 172.21.15.7:0/2311325016 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5624bb6fb0e0 0x5624bb6fd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.885+0000 7f7009e11740 4 client.15936 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.885+0000 7f7009e11740 1 -- 172.21.15.7:0/2311325016 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624bb65a880 con 0x5624bb6fb0e0 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.885+0000 7f7004fc1640 1 --2- 172.21.15.7:0/2311325016 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5624bb6fb0e0 0x5624bb6fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.886+0000 7f6fdffff640 10 client.15936 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.886+0000 7f7004fc1640 1 --2- 172.21.15.7:0/2311325016 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5624bb6fb0e0 0x5624bb6fd510 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=681fb13e50dd676 server_cookie=26adf7e634250575 in_seq=0 out_seq=0 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f6fdffff640 1 -- 172.21.15.7:0/2311325016 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5624bb65a880 con 0x5624bb6fb0e0 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f6fdffff640 10 client.15936 handle_command_reply: tid=0 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 1 client.15936 shutdown 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 1 -- 172.21.15.7:0/2311325016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 msgr2=0x5624bb6eaad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 1 --2- 172.21.15.7:0/2311325016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6eaad0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6fec036040 tx=0x7f6fec009660 comp rx=0 tx=0).stop 2024-04-21T08:45:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 1 -- 172.21.15.7:0/2311325016 shutdown_connections 2024-04-21T08:45:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 1 --2- 172.21.15.7:0/2311325016 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5624bb6fb0e0 0x5624bb6fd510 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 1 --2- 172.21.15.7:0/2311325016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 0x5624bb6e8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 1 --2- 172.21.15.7:0/2311325016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e28d0 0x5624bb6e7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 1 --2- 172.21.15.7:0/2311325016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6eaad0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 1 -- 172.21.15.7:0/2311325016 >> 172.21.15.7:0/2311325016 conn(0x5624bb6a8c90 msgr2=0x5624bb6de5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f6fdffff640 0 client.15936 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 1 -- 172.21.15.7:0/2311325016 shutdown_connections 2024-04-21T08:45:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 1 -- 172.21.15.7:0/2311325016 wait complete. 2024-04-21T08:45:27.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.887+0000 7f7009e11740 20 client.15936 trim_cache size 0 max 16384 2024-04-21T08:45:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.902+0000 7f7009e11740 1 Processor -- start 2024-04-21T08:45:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f7009e11740 1 -- start start 2024-04-21T08:45:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f7009e11740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e1b60 0x5624bb6e1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f7009e11740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f7009e11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 0x5624bb6ea590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f7009e11740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f70001af7e0 con 0x5624bb6e28d0 2024-04-21T08:45:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f7009e11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70001b2060 con 0x5624bb6eb0e0 2024-04-21T08:45:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f7009e11740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f70001afe50 con 0x5624bb6e1b60 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f7004fc1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f7004fc1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6f6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58452/0 (socket says 172.21.15.7:58452) 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f7004fc1640 1 -- 172.21.15.7:0/3879616388 learned_addr learned my addr 172.21.15.7:0/3879616388 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f6ffffff640 1 --2- 172.21.15.7:0/3879616388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e1b60 0x5624bb6e1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f6ffffff640 1 --2- 172.21.15.7:0/3879616388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 0x5624bb6ea590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f6ffffff640 1 -- 172.21.15.7:0/3879616388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 msgr2=0x5624bb6ea590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f6ffffff640 1 --2- 172.21.15.7:0/3879616388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 0x5624bb6ea590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f6ffffff640 1 -- 172.21.15.7:0/3879616388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 msgr2=0x5624bb6f6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f6ffffff640 1 --2- 172.21.15.7:0/3879616388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6f6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f6ffffff640 1 -- 172.21.15.7:0/3879616388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624bb6e0e30 con 0x5624bb6e1b60 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.903+0000 7f7004fc1640 1 --2- 172.21.15.7:0/3879616388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6f6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.904+0000 7f6ffffff640 1 --2- 172.21.15.7:0/3879616388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e1b60 0x5624bb6e1080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6fec001eb0 tx=0x7f6fec005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc9059bbffa886aa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.904+0000 7f6fea7fc640 1 -- 172.21.15.7:0/3879616388 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec005590 con 0x5624bb6e1b60 2024-04-21T08:45:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.904+0000 7f6fea7fc640 1 -- 172.21.15.7:0/3879616388 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fec048020 con 0x5624bb6e1b60 2024-04-21T08:45:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.904+0000 7f6fea7fc640 1 -- 172.21.15.7:0/3879616388 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec004dd0 con 0x5624bb6e1b60 2024-04-21T08:45:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.905+0000 7f7009e11740 1 -- 172.21.15.7:0/3879616388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e1b60 msgr2=0x5624bb6e1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.905+0000 7f7009e11740 1 --2- 172.21.15.7:0/3879616388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e1b60 0x5624bb6e1080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6fec001eb0 tx=0x7f6fec005e00 comp rx=0 tx=0).stop 2024-04-21T08:45:27.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.905+0000 7f7009e11740 1 -- 172.21.15.7:0/3879616388 shutdown_connections 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.905+0000 7f7009e11740 1 --2- 172.21.15.7:0/3879616388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 0x5624bb6ea590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.905+0000 7f7009e11740 1 --2- 172.21.15.7:0/3879616388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e28d0 0x5624bb6f6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.905+0000 7f7009e11740 1 --2- 172.21.15.7:0/3879616388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e1b60 0x5624bb6e1080 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.905+0000 7f7009e11740 1 -- 172.21.15.7:0/3879616388 >> 172.21.15.7:0/3879616388 conn(0x5624bb6a8c90 msgr2=0x5624bb6dc380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.905+0000 7f7009e11740 1 -- 172.21.15.7:0/3879616388 shutdown_connections 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.905+0000 7f7009e11740 1 -- 172.21.15.7:0/3879616388 wait complete. 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.905+0000 7f7009e11740 1 -- start start 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.905+0000 7f7009e11740 1 Processor -- start 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f7009e11740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6ebac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f7009e11740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e28d0 0x5624bb6ec000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f7009e11740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 0x5624bb6f2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f7009e11740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f70001b0520 con 0x5624bb6e1b60 2024-04-21T08:45:27.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f6ffffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6ebac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f7009e11740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70001b1990 con 0x5624bb6eb0e0 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f7009e11740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f70001b0bf0 con 0x5624bb6e28d0 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f7009e11740 10 client.0 _ll_register_callbacks cb 0x7f6fec02cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f6ffffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6ebac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58458/0 (socket says 172.21.15.7:58458) 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f6ffffff640 1 -- 172.21.15.7:0/3920123498 learned_addr learned my addr 172.21.15.7:0/3920123498 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f7004fc1640 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e28d0 0x5624bb6ec000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f6ffffff640 1 -- 172.21.15.7:0/3920123498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e28d0 msgr2=0x5624bb6ec000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f6ffffff640 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e28d0 0x5624bb6ec000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f6ffffff640 1 -- 172.21.15.7:0/3920123498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 msgr2=0x5624bb6f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f6ffffff640 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 0x5624bb6f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f6ffffff640 1 -- 172.21.15.7:0/3920123498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624bb6f27e0 con 0x5624bb6e1b60 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.906+0000 7f7004fc1640 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e28d0 0x5624bb6ec000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:27.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.907+0000 7f6ffffff640 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6ebac0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009900 tx=0x7f6fec004010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beaa564cc7692a06 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.907+0000 7f6fdffff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.907+0000 7f6fdffff640 1 -- 172.21.15.7:0/3920123498 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec067060 con 0x5624bb6e1b60 2024-04-21T08:45:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.907+0000 7f6fdffff640 1 -- 172.21.15.7:0/3920123498 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f6fec0035f0 con 0x5624bb6e1b60 2024-04-21T08:45:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.907+0000 7f7009e11740 1 -- 172.21.15.7:0/3920123498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fec048020 con 0x5624bb6e1b60 2024-04-21T08:45:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.907+0000 7f6fdffff640 1 -- 172.21.15.7:0/3920123498 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6fec0037f0 con 0x5624bb6e1b60 2024-04-21T08:45:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.907+0000 7f7009e11740 10 client.15942 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.907+0000 7f7009e11740 1 -- 172.21.15.7:0/3920123498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5624bb6f3a90 con 0x5624bb6e1b60 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.907+0000 7f6fdffff640 1 -- 172.21.15.7:0/3920123498 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fec004b70 con 0x5624bb6e1b60 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.908+0000 7f6fdffff640 1 -- 172.21.15.7:0/3920123498 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6fec0627a0 con 0x5624bb6e1b60 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.908+0000 7f7009e11740 10 client.15942 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.908+0000 7f7009e11740 10 client.15942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.908+0000 7f7009e11740 20 client.15942 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.908+0000 7f7009e11740 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5624bb7618e0 0x5624bb763cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.908+0000 7f7009e11740 4 client.15942 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.908+0000 7f7009e11740 1 -- 172.21.15.7:0/3920123498 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f6fec048020 con 0x5624bb7618e0 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.908+0000 7f7004fc1640 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5624bb7618e0 0x5624bb763cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.908+0000 7f6fdffff640 10 client.15942 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.908+0000 7f7004fc1640 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5624bb7618e0 0x5624bb763cd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f669163700c25fca server_cookie=1f6151af5eeecd12 in_seq=0 out_seq=0 2024-04-21T08:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f6fdffff640 1 -- 172.21.15.7:0/3920123498 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f6fec048020 con 0x5624bb7618e0 2024-04-21T08:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f6fdffff640 10 client.15942 handle_command_reply: tid=0 2024-04-21T08:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f7009e11740 1 client.15942 shutdown 2024-04-21T08:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f7009e11740 1 -- 172.21.15.7:0/3920123498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 msgr2=0x5624bb6ebac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f7009e11740 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6ebac0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009900 tx=0x7f6fec004010 comp rx=0 tx=0).stop 2024-04-21T08:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f7009e11740 1 -- 172.21.15.7:0/3920123498 shutdown_connections 2024-04-21T08:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f7009e11740 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5624bb7618e0 0x5624bb763cd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f6fdffff640 0 client.15942 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f7009e11740 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5624bb6eb0e0 0x5624bb6f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f7009e11740 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624bb6e28d0 0x5624bb6ec000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f7009e11740 1 --2- 172.21.15.7:0/3920123498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5624bb6e1b60 0x5624bb6ebac0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.912+0000 7f7009e11740 1 -- 172.21.15.7:0/3920123498 >> 172.21.15.7:0/3920123498 conn(0x5624bb6a8c90 msgr2=0x5624bb6e6680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7009e11740 1 -- 172.21.15.7:0/3920123498 shutdown_connections 2024-04-21T08:45:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7009e11740 1 -- 172.21.15.7:0/3920123498 wait complete. 2024-04-21T08:45:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7009e11740 20 client.15942 trim_cache size 0 max 16384 2024-04-21T08:45:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:45:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:45:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:45:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:45:27.916 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7007225640 1 -- 172.21.15.7:0/2177830090 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ff00a3eb0 msgr2=0x7f6ff00a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.916 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7007225640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ff00a3eb0 0x7f6ff00a6370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80ce720 tx=0x7f6ff80c7f50 comp rx=0 tx=0).stop 2024-04-21T08:45:27.916 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7007225640 1 -- 172.21.15.7:0/2177830090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f700010e8f0 msgr2=0x7f700019a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.916 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7007225640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f700010e8f0 0x7f700019a780 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800eb10 tx=0x7f6ff800c2d0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.916 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7007225640 1 -- 172.21.15.7:0/2177830090 shutdown_connections 2024-04-21T08:45:27.916 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7007225640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f700010e8f0 0x7f700019a780 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.916 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7007225640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ff00a3eb0 0x7f6ff00a6370 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.917 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7007225640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7000101660 0x7f70001a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.917 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7007225640 1 --2- 172.21.15.7:0/2177830090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7000100d40 0x7f70001a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.917 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.913+0000 7f7007225640 1 -- 172.21.15.7:0/2177830090 >> 172.21.15.7:0/2177830090 conn(0x7f7000077f70 msgr2=0x7f700010fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.917 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.914+0000 7f7007225640 1 -- 172.21.15.7:0/2177830090 shutdown_connections 2024-04-21T08:45:27.917 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:27.914+0000 7f7007225640 1 -- 172.21.15.7:0/2177830090 wait complete. 2024-04-21T08:45:28.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:28 smithi077 ceph-mon[31615]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T08:45:28.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:28 smithi007 ceph-mon[28477]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T08:45:28.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:28 smithi016 ceph-mon[31259]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T08:45:30.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:30 smithi077 ceph-mon[31615]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-21T08:45:30.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:30 smithi007 ceph-mon[28477]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-21T08:45:30.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:30 smithi016 ceph-mon[31259]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-21T08:45:32.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:32 smithi077 ceph-mon[31615]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-21T08:45:32.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:32 smithi007 ceph-mon[28477]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-21T08:45:32.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:32 smithi016 ceph-mon[31259]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-21T08:45:34.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:34 smithi077 ceph-mon[31615]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-21T08:45:34.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:34 smithi007 ceph-mon[28477]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-21T08:45:34.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:34 smithi016 ceph-mon[31259]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-21T08:45:36.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:36 smithi077 ceph-mon[31615]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-21T08:45:36.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:36 smithi007 ceph-mon[28477]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-21T08:45:36.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:36 smithi016 ceph-mon[31259]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-21T08:45:37.029 INFO:tasks.workunit.client.0.smithi007.stderr:Note: switching to '4f6bc5273643aaf5660909e85400647a8951c10b'. 2024-04-21T08:45:37.029 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr:state without impacting any branches by switching back to a branch. 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr: git switch -c 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr:Or undo this operation with: 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr: git switch - 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-04-21T08:45:37.030 INFO:tasks.workunit.client.0.smithi007.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-21T08:45:37.031 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-04-21T08:45:37.031 INFO:tasks.workunit.client.0.smithi007.stderr:HEAD is now at 4f6bc527364 use ubuntu for fs/workload tests 2024-04-21T08:45:37.033 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-21T08:45:37.062 INFO:tasks.workunit.client.0.smithi007.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-21T08:45:37.064 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T08:45:37.064 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-21T08:45:37.640 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-21T08:45:37.694 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-21T08:45:37.734 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T08:45:37.735 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T08:45:37.735 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-21T08:45:37.775 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T08:45:37.779 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:45:37.779 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-21T08:45:37.836 INFO:tasks.workunit:Running workunits matching kernel_untar_build.sh on client.0... 2024-04-21T08:45:37.837 INFO:tasks.workunit:Running workunit kernel_untar_build.sh... 2024-04-21T08:45:37.837 DEBUG:teuthology.orchestra.run.smithi007:workunit test kernel_untar_build.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh 2024-04-21T08:45:37.897 INFO:tasks.workunit.client.0.smithi007.stderr:+ wget -O linux.tar.xz http://download.ceph.com/qa/linux-6.5.11.tar.xz 2024-04-21T08:45:37.918 INFO:tasks.workunit.client.0.smithi007.stderr:--2024-04-21 08:45:37-- http://download.ceph.com/qa/linux-6.5.11.tar.xz 2024-04-21T08:45:37.918 INFO:tasks.workunit.client.0.smithi007.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-21T08:45:37.939 INFO:tasks.workunit.client.0.smithi007.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-04-21T08:45:37.960 INFO:tasks.workunit.client.0.smithi007.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-21T08:45:37.960 INFO:tasks.workunit.client.0.smithi007.stderr:Length: 139010916 (133M) [application/octet-stream] 2024-04-21T08:45:37.960 INFO:tasks.workunit.client.0.smithi007.stderr:Saving to: ‘linux.tar.xz’ 2024-04-21T08:45:37.963 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-04-21T08:45:38.001 INFO:tasks.workunit.client.0.smithi007.stderr: 0K .......... .......... .......... .......... .......... 0% 1.20M 1m51s 2024-04-21T08:45:38.024 INFO:tasks.workunit.client.0.smithi007.stderr: 50K .......... .......... .......... .......... .......... 0% 2.42M 83s 2024-04-21T08:45:38.025 INFO:tasks.workunit.client.0.smithi007.stderr: 100K .......... .......... .......... .......... .......... 0% 195M 55s 2024-04-21T08:45:38.025 INFO:tasks.workunit.client.0.smithi007.stderr: 150K .......... .......... .......... .......... .......... 0% 203M 42s 2024-04-21T08:45:38.044 INFO:tasks.workunit.client.0.smithi007.stderr: 200K .......... .......... .......... .......... .......... 0% 2.43M 44s 2024-04-21T08:45:38.044 INFO:tasks.workunit.client.0.smithi007.stderr: 250K .......... .......... .......... .......... .......... 0% 357M 37s 2024-04-21T08:45:38.044 INFO:tasks.workunit.client.0.smithi007.stderr: 300K .......... .......... .......... .......... .......... 0% 519M 32s 2024-04-21T08:45:38.044 INFO:tasks.workunit.client.0.smithi007.stderr: 350K .......... .......... .......... .......... .......... 0% 608M 28s 2024-04-21T08:45:38.066 INFO:tasks.workunit.client.0.smithi007.stderr: 400K .......... .......... .......... .......... .......... 0% 2.48M 31s 2024-04-21T08:45:38.067 INFO:tasks.workunit.client.0.smithi007.stderr: 450K .......... .......... .......... .......... .......... 0% 443M 27s 2024-04-21T08:45:38.067 INFO:tasks.workunit.client.0.smithi007.stderr: 500K .......... .......... .......... .......... .......... 0% 321M 25s 2024-04-21T08:45:38.067 INFO:tasks.workunit.client.0.smithi007.stderr: 550K .......... .......... .......... .......... .......... 0% 528M 23s 2024-04-21T08:45:38.067 INFO:tasks.workunit.client.0.smithi007.stderr: 600K .......... .......... .......... .......... .......... 0% 425M 21s 2024-04-21T08:45:38.067 INFO:tasks.workunit.client.0.smithi007.stderr: 650K .......... .......... .......... .......... .......... 0% 611M 20s 2024-04-21T08:45:38.067 INFO:tasks.workunit.client.0.smithi007.stderr: 700K .......... .......... .......... .......... .......... 0% 29.4M 19s 2024-04-21T08:45:38.067 INFO:tasks.workunit.client.0.smithi007.stderr: 750K .......... .......... .......... .......... .......... 0% 24.0M 18s 2024-04-21T08:45:38.088 INFO:tasks.workunit.client.0.smithi007.stderr: 800K .......... .......... .......... .......... .......... 0% 3.08M 19s 2024-04-21T08:45:38.088 INFO:tasks.workunit.client.0.smithi007.stderr: 850K .......... .......... .......... .......... .......... 0% 338M 18s 2024-04-21T08:45:38.088 INFO:tasks.workunit.client.0.smithi007.stderr: 900K .......... .......... .......... .......... .......... 0% 578M 17s 2024-04-21T08:45:38.088 INFO:tasks.workunit.client.0.smithi007.stderr: 950K .......... .......... .......... .......... .......... 0% 579M 16s 2024-04-21T08:45:38.088 INFO:tasks.workunit.client.0.smithi007.stderr: 1000K .......... .......... .......... .......... .......... 0% 459M 16s 2024-04-21T08:45:38.088 INFO:tasks.workunit.client.0.smithi007.stderr: 1050K .......... .......... .......... .......... .......... 0% 578M 15s 2024-04-21T08:45:38.088 INFO:tasks.workunit.client.0.smithi007.stderr: 1100K .......... .......... .......... .......... .......... 0% 62.4M 14s 2024-04-21T08:45:38.088 INFO:tasks.workunit.client.0.smithi007.stderr: 1150K .......... .......... .......... .......... .......... 0% 21.9M 14s 2024-04-21T08:45:38.088 INFO:tasks.workunit.client.0.smithi007.stderr: 1200K .......... .......... .......... .......... .......... 0% 22.3M 14s 2024-04-21T08:45:38.089 INFO:tasks.workunit.client.0.smithi007.stderr: 1250K .......... .......... .......... .......... .......... 0% 23.6M 13s 2024-04-21T08:45:38.089 INFO:tasks.workunit.client.0.smithi007.stderr: 1300K .......... .......... .......... .......... .......... 0% 235M 13s 2024-04-21T08:45:38.091 INFO:tasks.workunit.client.0.smithi007.stderr: 1350K .......... .......... .......... .......... .......... 1% 25.7M 13s 2024-04-21T08:45:38.093 INFO:tasks.workunit.client.0.smithi007.stderr: 1400K .......... .......... .......... .......... .......... 1% 33.7M 12s 2024-04-21T08:45:38.096 INFO:tasks.workunit.client.0.smithi007.stderr: 1450K .......... .......... .......... .......... .......... 1% 20.4M 12s 2024-04-21T08:45:38.098 INFO:tasks.workunit.client.0.smithi007.stderr: 1500K .......... .......... .......... .......... .......... 1% 23.3M 12s 2024-04-21T08:45:38.098 INFO:tasks.workunit.client.0.smithi007.stderr: 1550K .......... .......... .......... .......... .......... 1% 88.4M 12s 2024-04-21T08:45:38.100 INFO:tasks.workunit.client.0.smithi007.stderr: 1600K .......... .......... .......... .......... .......... 1% 28.6M 11s 2024-04-21T08:45:38.102 INFO:tasks.workunit.client.0.smithi007.stderr: 1650K .......... .......... .......... .......... .......... 1% 20.0M 11s 2024-04-21T08:45:38.104 INFO:tasks.workunit.client.0.smithi007.stderr: 1700K .......... .......... .......... .......... .......... 1% 24.5M 11s 2024-04-21T08:45:38.106 INFO:tasks.workunit.client.0.smithi007.stderr: 1750K .......... .......... .......... .......... .......... 1% 23.6M 11s 2024-04-21T08:45:38.107 INFO:tasks.workunit.client.0.smithi007.stderr: 1800K .......... .......... .......... .......... .......... 1% 206M 11s 2024-04-21T08:45:38.109 INFO:tasks.workunit.client.0.smithi007.stderr: 1850K .......... .......... .......... .......... .......... 1% 22.5M 10s 2024-04-21T08:45:38.111 INFO:tasks.workunit.client.0.smithi007.stderr: 1900K .......... .......... .......... .......... .......... 1% 23.7M 10s 2024-04-21T08:45:38.113 INFO:tasks.workunit.client.0.smithi007.stderr: 1950K .......... .......... .......... .......... .......... 1% 22.8M 10s 2024-04-21T08:45:38.115 INFO:tasks.workunit.client.0.smithi007.stderr: 2000K .......... .......... .......... .......... .......... 1% 23.0M 10s 2024-04-21T08:45:38.115 INFO:tasks.workunit.client.0.smithi007.stderr: 2050K .......... .......... .......... .......... .......... 1% 197M 10s 2024-04-21T08:45:38.117 INFO:tasks.workunit.client.0.smithi007.stderr: 2100K .......... .......... .......... .......... .......... 1% 23.9M 10s 2024-04-21T08:45:38.119 INFO:tasks.workunit.client.0.smithi007.stderr: 2150K .......... .......... .......... .......... .......... 1% 26.1M 10s 2024-04-21T08:45:38.121 INFO:tasks.workunit.client.0.smithi007.stderr: 2200K .......... .......... .......... .......... .......... 1% 23.0M 10s 2024-04-21T08:45:38.123 INFO:tasks.workunit.client.0.smithi007.stderr: 2250K .......... .......... .......... .......... .......... 1% 24.3M 9s 2024-04-21T08:45:38.123 INFO:tasks.workunit.client.0.smithi007.stderr: 2300K .......... .......... .......... .......... .......... 1% 447M 9s 2024-04-21T08:45:38.125 INFO:tasks.workunit.client.0.smithi007.stderr: 2350K .......... .......... .......... .......... .......... 1% 22.9M 9s 2024-04-21T08:45:38.127 INFO:tasks.workunit.client.0.smithi007.stderr: 2400K .......... .......... .......... .......... .......... 1% 24.2M 9s 2024-04-21T08:45:38.129 INFO:tasks.workunit.client.0.smithi007.stderr: 2450K .......... .......... .......... .......... .......... 1% 23.9M 9s 2024-04-21T08:45:38.132 INFO:tasks.workunit.client.0.smithi007.stderr: 2500K .......... .......... .......... .......... .......... 1% 24.2M 9s 2024-04-21T08:45:38.132 INFO:tasks.workunit.client.0.smithi007.stderr: 2550K .......... .......... .......... .......... .......... 1% 336M 9s 2024-04-21T08:45:38.134 INFO:tasks.workunit.client.0.smithi007.stderr: 2600K .......... .......... .......... .......... .......... 1% 22.2M 9s 2024-04-21T08:45:38.136 INFO:tasks.workunit.client.0.smithi007.stderr: 2650K .......... .......... .......... .......... .......... 1% 23.2M 9s 2024-04-21T08:45:38.138 INFO:tasks.workunit.client.0.smithi007.stderr: 2700K .......... .......... .......... .......... .......... 2% 23.0M 9s 2024-04-21T08:45:38.140 INFO:tasks.workunit.client.0.smithi007.stderr: 2750K .......... .......... .......... .......... .......... 2% 24.2M 9s 2024-04-21T08:45:38.140 INFO:tasks.workunit.client.0.smithi007.stderr: 2800K .......... .......... .......... .......... .......... 2% 368M 8s 2024-04-21T08:45:38.142 INFO:tasks.workunit.client.0.smithi007.stderr: 2850K .......... .......... .......... .......... .......... 2% 22.3M 8s 2024-04-21T08:45:38.143 INFO:tasks.workunit.client.0.smithi007.stderr: 2900K .......... .......... .......... .......... .......... 2% 44.7M 8s 2024-04-21T08:45:38.145 INFO:tasks.workunit.client.0.smithi007.stderr: 2950K .......... .......... .......... .......... .......... 2% 22.2M 8s 2024-04-21T08:45:38.148 INFO:tasks.workunit.client.0.smithi007.stderr: 3000K .......... .......... .......... .......... .......... 2% 19.7M 8s 2024-04-21T08:45:38.148 INFO:tasks.workunit.client.0.smithi007.stderr: 3050K .......... .......... .......... .......... .......... 2% 176M 8s 2024-04-21T08:45:38.150 INFO:tasks.workunit.client.0.smithi007.stderr: 3100K .......... .......... .......... .......... .......... 2% 29.5M 8s 2024-04-21T08:45:38.152 INFO:tasks.workunit.client.0.smithi007.stderr: 3150K .......... .......... .......... .......... .......... 2% 23.8M 8s 2024-04-21T08:45:38.154 INFO:tasks.workunit.client.0.smithi007.stderr: 3200K .......... .......... .......... .......... .......... 2% 22.9M 8s 2024-04-21T08:45:38.157 INFO:tasks.workunit.client.0.smithi007.stderr: 3250K .......... .......... .......... .......... .......... 2% 19.2M 8s 2024-04-21T08:45:38.157 INFO:tasks.workunit.client.0.smithi007.stderr: 3300K .......... .......... .......... .......... .......... 2% 172M 8s 2024-04-21T08:45:38.159 INFO:tasks.workunit.client.0.smithi007.stderr: 3350K .......... .......... .......... .......... .......... 2% 24.3M 8s 2024-04-21T08:45:38.161 INFO:tasks.workunit.client.0.smithi007.stderr: 3400K .......... .......... .......... .......... .......... 2% 22.8M 8s 2024-04-21T08:45:38.165 INFO:tasks.workunit.client.0.smithi007.stderr: 3450K .......... .......... .......... .......... .......... 2% 13.1M 8s 2024-04-21T08:45:38.167 INFO:tasks.workunit.client.0.smithi007.stderr: 3500K .......... .......... .......... .......... .......... 2% 26.6M 8s 2024-04-21T08:45:38.167 INFO:tasks.workunit.client.0.smithi007.stderr: 3550K .......... .......... .......... .......... .......... 2% 369M 8s 2024-04-21T08:45:38.169 INFO:tasks.workunit.client.0.smithi007.stderr: 3600K .......... .......... .......... .......... .......... 2% 22.0M 8s 2024-04-21T08:45:38.171 INFO:tasks.workunit.client.0.smithi007.stderr: 3650K .......... .......... .......... .......... .......... 2% 26.6M 8s 2024-04-21T08:45:38.173 INFO:tasks.workunit.client.0.smithi007.stderr: 3700K .......... .......... .......... .......... .......... 2% 26.8M 7s 2024-04-21T08:45:38.176 INFO:tasks.workunit.client.0.smithi007.stderr: 3750K .......... .......... .......... .......... .......... 2% 23.9M 7s 2024-04-21T08:45:38.176 INFO:tasks.workunit.client.0.smithi007.stderr: 3800K .......... .......... .......... .......... .......... 2% 101M 7s 2024-04-21T08:45:38.177 INFO:tasks.workunit.client.0.smithi007.stderr: 3850K .......... .......... .......... .......... .......... 2% 23.8M 7s 2024-04-21T08:45:38.179 INFO:tasks.workunit.client.0.smithi007.stderr: 3900K .......... .......... .......... .......... .......... 2% 28.5M 7s 2024-04-21T08:45:38.181 INFO:tasks.workunit.client.0.smithi007.stderr: 3950K .......... .......... .......... .......... .......... 2% 23.0M 7s 2024-04-21T08:45:38.186 INFO:tasks.workunit.client.0.smithi007.stderr: 4000K .......... .......... .......... .......... .......... 2% 13.9M 7s 2024-04-21T08:45:38.187 INFO:tasks.workunit.client.0.smithi007.stderr: 4050K .......... .......... .......... .......... .......... 3% 309M 7s 2024-04-21T08:45:38.187 INFO:tasks.workunit.client.0.smithi007.stderr: 4100K .......... .......... .......... .......... .......... 3% 49.3M 7s 2024-04-21T08:45:38.188 INFO:tasks.workunit.client.0.smithi007.stderr: 4150K .......... .......... .......... .......... .......... 3% 22.7M 7s 2024-04-21T08:45:38.189 INFO:tasks.workunit.client.0.smithi007.stderr: 4200K .......... .......... .......... .......... .......... 3% 36.2M 7s 2024-04-21T08:45:38.190 INFO:tasks.workunit.client.0.smithi007.stderr: 4250K .......... .......... .......... .......... .......... 3% 40.5M 7s 2024-04-21T08:45:38.192 INFO:tasks.workunit.client.0.smithi007.stderr: 4300K .......... .......... .......... .......... .......... 3% 31.9M 7s 2024-04-21T08:45:38.194 INFO:tasks.workunit.client.0.smithi007.stderr: 4350K .......... .......... .......... .......... .......... 3% 22.7M 7s 2024-04-21T08:45:38.195 INFO:tasks.workunit.client.0.smithi007.stderr: 4400K .......... .......... .......... .......... .......... 3% 38.9M 7s 2024-04-21T08:45:38.197 INFO:tasks.workunit.client.0.smithi007.stderr: 4450K .......... .......... .......... .......... .......... 3% 21.7M 7s 2024-04-21T08:45:38.199 INFO:tasks.workunit.client.0.smithi007.stderr: 4500K .......... .......... .......... .......... .......... 3% 25.1M 7s 2024-04-21T08:45:38.200 INFO:tasks.workunit.client.0.smithi007.stderr: 4550K .......... .......... .......... .......... .......... 3% 50.7M 7s 2024-04-21T08:45:38.202 INFO:tasks.workunit.client.0.smithi007.stderr: 4600K .......... .......... .......... .......... .......... 3% 25.2M 7s 2024-04-21T08:45:38.204 INFO:tasks.workunit.client.0.smithi007.stderr: 4650K .......... .......... .......... .......... .......... 3% 34.5M 7s 2024-04-21T08:45:38.206 INFO:tasks.workunit.client.0.smithi007.stderr: 4700K .......... .......... .......... .......... .......... 3% 22.7M 7s 2024-04-21T08:45:38.208 INFO:tasks.workunit.client.0.smithi007.stderr: 4750K .......... .......... .......... .......... .......... 3% 23.3M 7s 2024-04-21T08:45:38.209 INFO:tasks.workunit.client.0.smithi007.stderr: 4800K .......... .......... .......... .......... .......... 3% 58.1M 7s 2024-04-21T08:45:38.212 INFO:tasks.workunit.client.0.smithi007.stderr: 4850K .......... .......... .......... .......... .......... 3% 14.7M 7s 2024-04-21T08:45:38.214 INFO:tasks.workunit.client.0.smithi007.stderr: 4900K .......... .......... .......... .......... .......... 3% 32.2M 7s 2024-04-21T08:45:38.216 INFO:tasks.workunit.client.0.smithi007.stderr: 4950K .......... .......... .......... .......... .......... 3% 24.2M 7s 2024-04-21T08:45:38.218 INFO:tasks.workunit.client.0.smithi007.stderr: 5000K .......... .......... .......... .......... .......... 3% 23.6M 7s 2024-04-21T08:45:38.219 INFO:tasks.workunit.client.0.smithi007.stderr: 5050K .......... .......... .......... .......... .......... 3% 41.2M 7s 2024-04-21T08:45:38.221 INFO:tasks.workunit.client.0.smithi007.stderr: 5100K .......... .......... .......... .......... .......... 3% 24.3M 7s 2024-04-21T08:45:38.223 INFO:tasks.workunit.client.0.smithi007.stderr: 5150K .......... .......... .......... .......... .......... 3% 33.6M 7s 2024-04-21T08:45:38.224 INFO:tasks.workunit.client.0.smithi007.stderr: 5200K .......... .......... .......... .......... .......... 3% 22.7M 7s 2024-04-21T08:45:38.227 INFO:tasks.workunit.client.0.smithi007.stderr: 5250K .......... .......... .......... .......... .......... 3% 23.6M 7s 2024-04-21T08:45:38.227 INFO:tasks.workunit.client.0.smithi007.stderr: 5300K .......... .......... .......... .......... .......... 3% 65.1M 7s 2024-04-21T08:45:38.229 INFO:tasks.workunit.client.0.smithi007.stderr: 5350K .......... .......... .......... .......... .......... 3% 23.3M 7s 2024-04-21T08:45:38.231 INFO:tasks.workunit.client.0.smithi007.stderr: 5400K .......... .......... .......... .......... .......... 4% 24.8M 6s 2024-04-21T08:45:38.233 INFO:tasks.workunit.client.0.smithi007.stderr: 5450K .......... .......... .......... .......... .......... 4% 34.2M 6s 2024-04-21T08:45:38.235 INFO:tasks.workunit.client.0.smithi007.stderr: 5500K .......... .......... .......... .......... .......... 4% 26.0M 6s 2024-04-21T08:45:38.237 INFO:tasks.workunit.client.0.smithi007.stderr: 5550K .......... .......... .......... .......... .......... 4% 39.7M 6s 2024-04-21T08:45:38.238 INFO:tasks.workunit.client.0.smithi007.stderr: 5600K .......... .......... .......... .......... .......... 4% 24.3M 6s 2024-04-21T08:45:38.240 INFO:tasks.workunit.client.0.smithi007.stderr: 5650K .......... .......... .......... .......... .......... 4% 24.9M 6s 2024-04-21T08:45:38.241 INFO:tasks.workunit.client.0.smithi007.stderr: 5700K .......... .......... .......... .......... .......... 4% 40.2M 6s 2024-04-21T08:45:38.243 INFO:tasks.workunit.client.0.smithi007.stderr: 5750K .......... .......... .......... .......... .......... 4% 22.4M 6s 2024-04-21T08:45:38.246 INFO:tasks.workunit.client.0.smithi007.stderr: 5800K .......... .......... .......... .......... .......... 4% 23.8M 6s 2024-04-21T08:45:38.247 INFO:tasks.workunit.client.0.smithi007.stderr: 5850K .......... .......... .......... .......... .......... 4% 30.3M 6s 2024-04-21T08:45:38.249 INFO:tasks.workunit.client.0.smithi007.stderr: 5900K .......... .......... .......... .......... .......... 4% 24.2M 6s 2024-04-21T08:45:38.250 INFO:tasks.workunit.client.0.smithi007.stderr: 5950K .......... .......... .......... .......... .......... 4% 41.4M 6s 2024-04-21T08:45:38.252 INFO:tasks.workunit.client.0.smithi007.stderr: 6000K .......... .......... .......... .......... .......... 4% 22.2M 6s 2024-04-21T08:45:38.254 INFO:tasks.workunit.client.0.smithi007.stderr: 6050K .......... .......... .......... .......... .......... 4% 24.5M 6s 2024-04-21T08:45:38.255 INFO:tasks.workunit.client.0.smithi007.stderr: 6100K .......... .......... .......... .......... .......... 4% 48.8M 6s 2024-04-21T08:45:38.257 INFO:tasks.workunit.client.0.smithi007.stderr: 6150K .......... .......... .......... .......... .......... 4% 35.6M 6s 2024-04-21T08:45:38.259 INFO:tasks.workunit.client.0.smithi007.stderr: 6200K .......... .......... .......... .......... .......... 4% 24.5M 6s 2024-04-21T08:45:38.261 INFO:tasks.workunit.client.0.smithi007.stderr: 6250K .......... .......... .......... .......... .......... 4% 23.8M 6s 2024-04-21T08:45:38.263 INFO:tasks.workunit.client.0.smithi007.stderr: 6300K .......... .......... .......... .......... .......... 4% 24.9M 6s 2024-04-21T08:45:38.264 INFO:tasks.workunit.client.0.smithi007.stderr: 6350K .......... .......... .......... .......... .......... 4% 47.0M 6s 2024-04-21T08:45:38.266 INFO:tasks.workunit.client.0.smithi007.stderr: 6400K .......... .......... .......... .......... .......... 4% 21.3M 6s 2024-04-21T08:45:38.268 INFO:tasks.workunit.client.0.smithi007.stderr: 6450K .......... .......... .......... .......... .......... 4% 30.1M 6s 2024-04-21T08:45:38.270 INFO:tasks.workunit.client.0.smithi007.stderr: 6500K .......... .......... .......... .......... .......... 4% 19.3M 6s 2024-04-21T08:45:38.273 INFO:tasks.workunit.client.0.smithi007.stderr: 6550K .......... .......... .......... .......... .......... 4% 20.0M 6s 2024-04-21T08:45:38.273 INFO:tasks.workunit.client.0.smithi007.stderr: 6600K .......... .......... .......... .......... .......... 4% 56.6M 6s 2024-04-21T08:45:38.275 INFO:tasks.workunit.client.0.smithi007.stderr: 6650K .......... .......... .......... .......... .......... 4% 23.6M 6s 2024-04-21T08:45:38.277 INFO:tasks.workunit.client.0.smithi007.stderr: 6700K .......... .......... .......... .......... .......... 4% 34.2M 6s 2024-04-21T08:45:38.279 INFO:tasks.workunit.client.0.smithi007.stderr: 6750K .......... .......... .......... .......... .......... 5% 29.4M 6s 2024-04-21T08:45:38.281 INFO:tasks.workunit.client.0.smithi007.stderr: 6800K .......... .......... .......... .......... .......... 5% 20.7M 6s 2024-04-21T08:45:38.282 INFO:tasks.workunit.client.0.smithi007.stderr: 6850K .......... .......... .......... .......... .......... 5% 53.3M 6s 2024-04-21T08:45:38.284 INFO:tasks.workunit.client.0.smithi007.stderr: 6900K .......... .......... .......... .......... .......... 5% 23.9M 6s 2024-04-21T08:45:38.285 INFO:tasks.workunit.client.0.smithi007.stderr: 6950K .......... .......... .......... .......... .......... 5% 36.8M 6s 2024-04-21T08:45:38.288 INFO:tasks.workunit.client.0.smithi007.stderr: 7000K .......... .......... .......... .......... .......... 5% 22.8M 6s 2024-04-21T08:45:38.290 INFO:tasks.workunit.client.0.smithi007.stderr: 7050K .......... .......... .......... .......... .......... 5% 23.8M 6s 2024-04-21T08:45:38.290 INFO:tasks.workunit.client.0.smithi007.stderr: 7100K .......... .......... .......... .......... .......... 5% 57.5M 6s 2024-04-21T08:45:38.292 INFO:tasks.workunit.client.0.smithi007.stderr: 7150K .......... .......... .......... .......... .......... 5% 23.8M 6s 2024-04-21T08:45:38.294 INFO:tasks.workunit.client.0.smithi007.stderr: 7200K .......... .......... .......... .......... .......... 5% 35.7M 6s 2024-04-21T08:45:38.296 INFO:tasks.workunit.client.0.smithi007.stderr: 7250K .......... .......... .......... .......... .......... 5% 24.1M 6s 2024-04-21T08:45:38.298 INFO:tasks.workunit.client.0.smithi007.stderr: 7300K .......... .......... .......... .......... .......... 5% 23.7M 6s 2024-04-21T08:45:38.299 INFO:tasks.workunit.client.0.smithi007.stderr: 7350K .......... .......... .......... .......... .......... 5% 51.5M 6s 2024-04-21T08:45:38.301 INFO:tasks.workunit.client.0.smithi007.stderr: 7400K .......... .......... .......... .......... .......... 5% 25.2M 6s 2024-04-21T08:45:38.302 INFO:tasks.workunit.client.0.smithi007.stderr: 7450K .......... .......... .......... .......... .......... 5% 33.9M 6s 2024-04-21T08:45:38.304 INFO:tasks.workunit.client.0.smithi007.stderr: 7500K .......... .......... .......... .......... .......... 5% 25.7M 6s 2024-04-21T08:45:38.306 INFO:tasks.workunit.client.0.smithi007.stderr: 7550K .......... .......... .......... .......... .......... 5% 22.9M 6s 2024-04-21T08:45:38.307 INFO:tasks.workunit.client.0.smithi007.stderr: 7600K .......... .......... .......... .......... .......... 5% 56.6M 6s 2024-04-21T08:45:38.309 INFO:tasks.workunit.client.0.smithi007.stderr: 7650K .......... .......... .......... .......... .......... 5% 25.6M 6s 2024-04-21T08:45:38.310 INFO:tasks.workunit.client.0.smithi007.stderr: 7700K .......... .......... .......... .......... .......... 5% 35.8M 6s 2024-04-21T08:45:38.313 INFO:tasks.workunit.client.0.smithi007.stderr: 7750K .......... .......... .......... .......... .......... 5% 19.4M 6s 2024-04-21T08:45:38.315 INFO:tasks.workunit.client.0.smithi007.stderr: 7800K .......... .......... .......... .......... .......... 5% 22.8M 6s 2024-04-21T08:45:38.316 INFO:tasks.workunit.client.0.smithi007.stderr: 7850K .......... .......... .......... .......... .......... 5% 48.5M 6s 2024-04-21T08:45:38.318 INFO:tasks.workunit.client.0.smithi007.stderr: 7900K .......... .......... .......... .......... .......... 5% 24.9M 6s 2024-04-21T08:45:38.319 INFO:tasks.workunit.client.0.smithi007.stderr: 7950K .......... .......... .......... .......... .......... 5% 41.5M 6s 2024-04-21T08:45:38.321 INFO:tasks.workunit.client.0.smithi007.stderr: 8000K .......... .......... .......... .......... .......... 5% 22.4M 6s 2024-04-21T08:45:38.323 INFO:tasks.workunit.client.0.smithi007.stderr: 8050K .......... .......... .......... .......... .......... 5% 25.6M 6s 2024-04-21T08:45:38.324 INFO:tasks.workunit.client.0.smithi007.stderr: 8100K .......... .......... .......... .......... .......... 6% 43.1M 6s 2024-04-21T08:45:38.326 INFO:tasks.workunit.client.0.smithi007.stderr: 8150K .......... .......... .......... .......... .......... 6% 24.9M 6s 2024-04-21T08:45:38.328 INFO:tasks.workunit.client.0.smithi007.stderr: 8200K .......... .......... .......... .......... .......... 6% 36.5M 6s 2024-04-21T08:45:38.330 INFO:tasks.workunit.client.0.smithi007.stderr: 8250K .......... .......... .......... .......... .......... 6% 23.2M 6s 2024-04-21T08:45:38.332 INFO:tasks.workunit.client.0.smithi007.stderr: 8300K .......... .......... .......... .......... .......... 6% 24.9M 6s 2024-04-21T08:45:38.332 INFO:tasks.workunit.client.0.smithi007.stderr: 8350K .......... .......... .......... .......... .......... 6% 55.2M 6s 2024-04-21T08:45:38.335 INFO:tasks.workunit.client.0.smithi007.stderr: 8400K .......... .......... .......... .......... .......... 6% 22.7M 6s 2024-04-21T08:45:38.336 INFO:tasks.workunit.client.0.smithi007.stderr: 8450K .......... .......... .......... .......... .......... 6% 38.4M 6s 2024-04-21T08:45:38.338 INFO:tasks.workunit.client.0.smithi007.stderr: 8500K .......... .......... .......... .......... .......... 6% 24.4M 6s 2024-04-21T08:45:38.340 INFO:tasks.workunit.client.0.smithi007.stderr: 8550K .......... .......... .......... .......... .......... 6% 26.1M 6s 2024-04-21T08:45:38.341 INFO:tasks.workunit.client.0.smithi007.stderr: 8600K .......... .......... .......... .......... .......... 6% 40.7M 6s 2024-04-21T08:45:38.343 INFO:tasks.workunit.client.0.smithi007.stderr: 8650K .......... .......... .......... .......... .......... 6% 24.5M 6s 2024-04-21T08:45:38.345 INFO:tasks.workunit.client.0.smithi007.stderr: 8700K .......... .......... .......... .......... .......... 6% 36.1M 6s 2024-04-21T08:45:38.347 INFO:tasks.workunit.client.0.smithi007.stderr: 8750K .......... .......... .......... .......... .......... 6% 23.9M 6s 2024-04-21T08:45:38.349 INFO:tasks.workunit.client.0.smithi007.stderr: 8800K .......... .......... .......... .......... .......... 6% 23.4M 6s 2024-04-21T08:45:38.349 INFO:tasks.workunit.client.0.smithi007.stderr: 8850K .......... .......... .......... .......... .......... 6% 51.5M 6s 2024-04-21T08:45:38.351 INFO:tasks.workunit.client.0.smithi007.stderr: 8900K .......... .......... .......... .......... .......... 6% 24.0M 6s 2024-04-21T08:45:38.353 INFO:tasks.workunit.client.0.smithi007.stderr: 8950K .......... .......... .......... .......... .......... 6% 38.9M 6s 2024-04-21T08:45:38.355 INFO:tasks.workunit.client.0.smithi007.stderr: 9000K .......... .......... .......... .......... .......... 6% 22.5M 6s 2024-04-21T08:45:38.357 INFO:tasks.workunit.client.0.smithi007.stderr: 9050K .......... .......... .......... .......... .......... 6% 23.7M 6s 2024-04-21T08:45:38.358 INFO:tasks.workunit.client.0.smithi007.stderr: 9100K .......... .......... .......... .......... .......... 6% 46.5M 6s 2024-04-21T08:45:38.360 INFO:tasks.workunit.client.0.smithi007.stderr: 9150K .......... .......... .......... .......... .......... 6% 21.1M 6s 2024-04-21T08:45:38.363 INFO:tasks.workunit.client.0.smithi007.stderr: 9200K .......... .......... .......... .......... .......... 6% 38.3M 5s 2024-04-21T08:45:38.364 INFO:tasks.workunit.client.0.smithi007.stderr: 9250K .......... .......... .......... .......... .......... 6% 22.6M 5s 2024-04-21T08:45:38.366 INFO:tasks.workunit.client.0.smithi007.stderr: 9300K .......... .......... .......... .......... .......... 6% 21.7M 5s 2024-04-21T08:45:38.367 INFO:tasks.workunit.client.0.smithi007.stderr: 9350K .......... .......... .......... .......... .......... 6% 53.9M 5s 2024-04-21T08:45:38.369 INFO:tasks.workunit.client.0.smithi007.stderr: 9400K .......... .......... .......... .......... .......... 6% 30.6M 5s 2024-04-21T08:45:38.371 INFO:tasks.workunit.client.0.smithi007.stderr: 9450K .......... .......... .......... .......... .......... 6% 20.4M 5s 2024-04-21T08:45:38.373 INFO:tasks.workunit.client.0.smithi007.stderr: 9500K .......... .......... .......... .......... .......... 7% 23.9M 5s 2024-04-21T08:45:38.375 INFO:tasks.workunit.client.0.smithi007.stderr: 9550K .......... .......... .......... .......... .......... 7% 31.7M 5s 2024-04-21T08:45:38.385 INFO:tasks.workunit.client.0.smithi007.stderr: 9600K .......... .......... .......... .......... .......... 7% 53.3M 5s 2024-04-21T08:45:38.385 INFO:tasks.workunit.client.0.smithi007.stderr: 9650K .......... .......... .......... .......... .......... 7% 21.5M 5s 2024-04-21T08:45:38.386 INFO:tasks.workunit.client.0.smithi007.stderr: 9700K .......... .......... .......... .......... .......... 7% 36.3M 5s 2024-04-21T08:45:38.386 INFO:tasks.workunit.client.0.smithi007.stderr: 9750K .......... .......... .......... .......... .......... 7% 24.0M 5s 2024-04-21T08:45:38.386 INFO:tasks.workunit.client.0.smithi007.stderr: 9800K .......... .......... .......... .......... .......... 7% 23.8M 5s 2024-04-21T08:45:38.386 INFO:tasks.workunit.client.0.smithi007.stderr: 9850K .......... .......... .......... .......... .......... 7% 52.0M 5s 2024-04-21T08:45:38.386 INFO:tasks.workunit.client.0.smithi007.stderr: 9900K .......... .......... .......... .......... .......... 7% 22.5M 5s 2024-04-21T08:45:38.388 INFO:tasks.workunit.client.0.smithi007.stderr: 9950K .......... .......... .......... .......... .......... 7% 38.2M 5s 2024-04-21T08:45:38.390 INFO:tasks.workunit.client.0.smithi007.stderr: 10000K .......... .......... .......... .......... .......... 7% 22.6M 5s 2024-04-21T08:45:38.392 INFO:tasks.workunit.client.0.smithi007.stderr: 10050K .......... .......... .......... .......... .......... 7% 23.6M 5s 2024-04-21T08:45:38.393 INFO:tasks.workunit.client.0.smithi007.stderr: 10100K .......... .......... .......... .......... .......... 7% 52.0M 5s 2024-04-21T08:45:38.395 INFO:tasks.workunit.client.0.smithi007.stderr: 10150K .......... .......... .......... .......... .......... 7% 24.4M 5s 2024-04-21T08:45:38.397 INFO:tasks.workunit.client.0.smithi007.stderr: 10200K .......... .......... .......... .......... .......... 7% 23.9M 5s 2024-04-21T08:45:38.398 INFO:tasks.workunit.client.0.smithi007.stderr: 10250K .......... .......... .......... .......... .......... 7% 38.1M 5s 2024-04-21T08:45:38.400 INFO:tasks.workunit.client.0.smithi007.stderr: 10300K .......... .......... .......... .......... .......... 7% 22.5M 5s 2024-04-21T08:45:38.401 INFO:tasks.workunit.client.0.smithi007.stderr: 10350K .......... .......... .......... .......... .......... 7% 67.3M 5s 2024-04-21T08:45:38.403 INFO:tasks.workunit.client.0.smithi007.stderr: 10400K .......... .......... .......... .......... .......... 7% 22.8M 5s 2024-04-21T08:45:38.405 INFO:tasks.workunit.client.0.smithi007.stderr: 10450K .......... .......... .......... .......... .......... 7% 38.1M 5s 2024-04-21T08:45:38.407 INFO:tasks.workunit.client.0.smithi007.stderr: 10500K .......... .......... .......... .......... .......... 7% 22.9M 5s 2024-04-21T08:45:38.409 INFO:tasks.workunit.client.0.smithi007.stderr: 10550K .......... .......... .......... .......... .......... 7% 23.5M 5s 2024-04-21T08:45:38.410 INFO:tasks.workunit.client.0.smithi007.stderr: 10600K .......... .......... .......... .......... .......... 7% 52.3M 5s 2024-04-21T08:45:38.412 INFO:tasks.workunit.client.0.smithi007.stderr: 10650K .......... .......... .......... .......... .......... 7% 24.7M 5s 2024-04-21T08:45:38.414 INFO:tasks.workunit.client.0.smithi007.stderr: 10700K .......... .......... .......... .......... .......... 7% 30.0M 5s 2024-04-21T08:45:38.415 INFO:tasks.workunit.client.0.smithi007.stderr: 10750K .......... .......... .......... .......... .......... 7% 24.6M 5s 2024-04-21T08:45:38.417 INFO:tasks.workunit.client.0.smithi007.stderr: 10800K .......... .......... .......... .......... .......... 7% 26.9M 5s 2024-04-21T08:45:38.419 INFO:tasks.workunit.client.0.smithi007.stderr: 10850K .......... .......... .......... .......... .......... 8% 54.4M 5s 2024-04-21T08:45:38.420 INFO:tasks.workunit.client.0.smithi007.stderr: 10900K .......... .......... .......... .......... .......... 8% 24.8M 5s 2024-04-21T08:45:38.422 INFO:tasks.workunit.client.0.smithi007.stderr: 10950K .......... .......... .......... .......... .......... 8% 26.4M 5s 2024-04-21T08:45:38.424 INFO:tasks.workunit.client.0.smithi007.stderr: 11000K .......... .......... .......... .......... .......... 8% 23.0M 5s 2024-04-21T08:45:38.425 INFO:tasks.workunit.client.0.smithi007.stderr: 11050K .......... .......... .......... .......... .......... 8% 31.7M 5s 2024-04-21T08:45:38.427 INFO:tasks.workunit.client.0.smithi007.stderr: 11100K .......... .......... .......... .......... .......... 8% 25.2M 5s 2024-04-21T08:45:38.428 INFO:tasks.workunit.client.0.smithi007.stderr: 11150K .......... .......... .......... .......... .......... 8% 52.1M 5s 2024-04-21T08:45:38.430 INFO:tasks.workunit.client.0.smithi007.stderr: 11200K .......... .......... .......... .......... .......... 8% 24.9M 5s 2024-04-21T08:45:38.432 INFO:tasks.workunit.client.0.smithi007.stderr: 11250K .......... .......... .......... .......... .......... 8% 34.4M 5s 2024-04-21T08:45:38.433 INFO:tasks.workunit.client.0.smithi007.stderr: 11300K .......... .......... .......... .......... .......... 8% 25.7M 5s 2024-04-21T08:45:38.436 INFO:tasks.workunit.client.0.smithi007.stderr: 11350K .......... .......... .......... .......... .......... 8% 22.7M 5s 2024-04-21T08:45:38.436 INFO:tasks.workunit.client.0.smithi007.stderr: 11400K .......... .......... .......... .......... .......... 8% 54.6M 5s 2024-04-21T08:45:38.438 INFO:tasks.workunit.client.0.smithi007.stderr: 11450K .......... .......... .......... .......... .......... 8% 24.3M 5s 2024-04-21T08:45:38.440 INFO:tasks.workunit.client.0.smithi007.stderr: 11500K .......... .......... .......... .......... .......... 8% 33.4M 5s 2024-04-21T08:45:38.442 INFO:tasks.workunit.client.0.smithi007.stderr: 11550K .......... .......... .......... .......... .......... 8% 23.9M 5s 2024-04-21T08:45:38.444 INFO:tasks.workunit.client.0.smithi007.stderr: 11600K .......... .......... .......... .......... .......... 8% 25.7M 5s 2024-04-21T08:45:38.445 INFO:tasks.workunit.client.0.smithi007.stderr: 11650K .......... .......... .......... .......... .......... 8% 37.7M 5s 2024-04-21T08:45:38.447 INFO:tasks.workunit.client.0.smithi007.stderr: 11700K .......... .......... .......... .......... .......... 8% 22.7M 5s 2024-04-21T08:45:38.450 INFO:tasks.workunit.client.0.smithi007.stderr: 11750K .......... .......... .......... .......... .......... 8% 21.5M 5s 2024-04-21T08:45:38.453 INFO:tasks.workunit.client.0.smithi007.stderr: 11800K .......... .......... .......... .......... .......... 8% 17.2M 5s 2024-04-21T08:45:38.455 INFO:tasks.workunit.client.0.smithi007.stderr: 11850K .......... .......... .......... .......... .......... 8% 23.7M 5s 2024-04-21T08:45:38.455 INFO:tasks.workunit.client.0.smithi007.stderr: 11900K .......... .......... .......... .......... .......... 8% 46.7M 5s 2024-04-21T08:45:38.458 INFO:tasks.workunit.client.0.smithi007.stderr: 11950K .......... .......... .......... .......... .......... 8% 24.9M 5s 2024-04-21T08:45:38.460 INFO:tasks.workunit.client.0.smithi007.stderr: 12000K .......... .......... .......... .......... .......... 8% 24.9M 5s 2024-04-21T08:45:38.461 INFO:tasks.workunit.client.0.smithi007.stderr: 12050K .......... .......... .......... .......... .......... 8% 36.1M 5s 2024-04-21T08:45:38.462 INFO:tasks.workunit.client.0.smithi007.stderr: 12100K .......... .......... .......... .......... .......... 8% 30.0M 5s 2024-04-21T08:45:38.464 INFO:tasks.workunit.client.0.smithi007.stderr: 12150K .......... .......... .......... .......... .......... 8% 37.5M 5s 2024-04-21T08:45:38.466 INFO:tasks.workunit.client.0.smithi007.stderr: 12200K .......... .......... .......... .......... .......... 9% 22.1M 5s 2024-04-21T08:45:38.467 INFO:tasks.workunit.client.0.smithi007.stderr: 12250K .......... .......... .......... .......... .......... 9% 40.2M 5s 2024-04-21T08:45:38.470 INFO:tasks.workunit.client.0.smithi007.stderr: 12300K .......... .......... .......... .......... .......... 9% 23.2M 5s 2024-04-21T08:45:38.472 INFO:tasks.workunit.client.0.smithi007.stderr: 12350K .......... .......... .......... .......... .......... 9% 24.5M 5s 2024-04-21T08:45:38.472 INFO:tasks.workunit.client.0.smithi007.stderr: 12400K .......... .......... .......... .......... .......... 9% 46.6M 5s 2024-04-21T08:45:38.474 INFO:tasks.workunit.client.0.smithi007.stderr: 12450K .......... .......... .......... .......... .......... 9% 25.4M 5s 2024-04-21T08:45:38.476 INFO:tasks.workunit.client.0.smithi007.stderr: 12500K .......... .......... .......... .......... .......... 9% 36.5M 5s 2024-04-21T08:45:38.478 INFO:tasks.workunit.client.0.smithi007.stderr: 12550K .......... .......... .......... .......... .......... 9% 23.4M 5s 2024-04-21T08:45:38.480 INFO:tasks.workunit.client.0.smithi007.stderr: 12600K .......... .......... .......... .......... .......... 9% 25.4M 5s 2024-04-21T08:45:38.481 INFO:tasks.workunit.client.0.smithi007.stderr: 12650K .......... .......... .......... .......... .......... 9% 45.2M 5s 2024-04-21T08:45:38.487 INFO:tasks.workunit.client.0.smithi007.stderr: 12700K .......... .......... .......... .......... .......... 9% 24.9M 5s 2024-04-21T08:45:38.488 INFO:tasks.workunit.client.0.smithi007.stderr: 12750K .......... .......... .......... .......... .......... 9% 38.1M 5s 2024-04-21T08:45:38.488 INFO:tasks.workunit.client.0.smithi007.stderr: 12800K .......... .......... .......... .......... .......... 9% 23.7M 5s 2024-04-21T08:45:38.488 INFO:tasks.workunit.client.0.smithi007.stderr: 12850K .......... .......... .......... .......... .......... 9% 24.5M 5s 2024-04-21T08:45:38.489 INFO:tasks.workunit.client.0.smithi007.stderr: 12900K .......... .......... .......... .......... .......... 9% 51.2M 5s 2024-04-21T08:45:38.491 INFO:tasks.workunit.client.0.smithi007.stderr: 12950K .......... .......... .......... .......... .......... 9% 22.9M 5s 2024-04-21T08:45:38.493 INFO:tasks.workunit.client.0.smithi007.stderr: 13000K .......... .......... .......... .......... .......... 9% 36.9M 5s 2024-04-21T08:45:38.495 INFO:tasks.workunit.client.0.smithi007.stderr: 13050K .......... .......... .......... .......... .......... 9% 23.8M 5s 2024-04-21T08:45:38.498 INFO:tasks.workunit.client.0.smithi007.stderr: 13100K .......... .......... .......... .......... .......... 9% 23.1M 5s 2024-04-21T08:45:38.498 INFO:tasks.workunit.client.0.smithi007.stderr: 13150K .......... .......... .......... .......... .......... 9% 50.5M 5s 2024-04-21T08:45:38.499 INFO:tasks.workunit.client.0.smithi007.stderr: 13200K .......... .......... .......... .......... .......... 9% 25.5M 5s 2024-04-21T08:45:38.501 INFO:tasks.workunit.client.0.smithi007.stderr: 13250K .......... .......... .......... .......... .......... 9% 37.2M 5s 2024-04-21T08:45:38.504 INFO:tasks.workunit.client.0.smithi007.stderr: 13300K .......... .......... .......... .......... .......... 9% 14.8M 5s 2024-04-21T08:45:38.506 INFO:tasks.workunit.client.0.smithi007.stderr: 13350K .......... .......... .......... .......... .......... 9% 21.5M 5s 2024-04-21T08:45:38.507 INFO:tasks.workunit.client.0.smithi007.stderr: 13400K .......... .......... .......... .......... .......... 9% 57.0M 5s 2024-04-21T08:45:38.509 INFO:tasks.workunit.client.0.smithi007.stderr: 13450K .......... .......... .......... .......... .......... 9% 23.6M 5s 2024-04-21T08:45:38.511 INFO:tasks.workunit.client.0.smithi007.stderr: 13500K .......... .......... .......... .......... .......... 9% 35.7M 5s 2024-04-21T08:45:38.513 INFO:tasks.workunit.client.0.smithi007.stderr: 13550K .......... .......... .......... .......... .......... 10% 19.7M 5s 2024-04-21T08:45:38.515 INFO:tasks.workunit.client.0.smithi007.stderr: 13600K .......... .......... .......... .......... .......... 10% 23.3M 5s 2024-04-21T08:45:38.516 INFO:tasks.workunit.client.0.smithi007.stderr: 13650K .......... .......... .......... .......... .......... 10% 51.1M 5s 2024-04-21T08:45:38.518 INFO:tasks.workunit.client.0.smithi007.stderr: 13700K .......... .......... .......... .......... .......... 10% 24.6M 5s 2024-04-21T08:45:38.519 INFO:tasks.workunit.client.0.smithi007.stderr: 13750K .......... .......... .......... .......... .......... 10% 37.2M 5s 2024-04-21T08:45:38.521 INFO:tasks.workunit.client.0.smithi007.stderr: 13800K .......... .......... .......... .......... .......... 10% 23.5M 5s 2024-04-21T08:45:38.524 INFO:tasks.workunit.client.0.smithi007.stderr: 13850K .......... .......... .......... .......... .......... 10% 24.2M 5s 2024-04-21T08:45:38.525 INFO:tasks.workunit.client.0.smithi007.stderr: 13900K .......... .......... .......... .......... .......... 10% 48.1M 5s 2024-04-21T08:45:38.527 INFO:tasks.workunit.client.0.smithi007.stderr: 13950K .......... .......... .......... .......... .......... 10% 23.4M 5s 2024-04-21T08:45:38.528 INFO:tasks.workunit.client.0.smithi007.stderr: 14000K .......... .......... .......... .......... .......... 10% 40.3M 5s 2024-04-21T08:45:38.530 INFO:tasks.workunit.client.0.smithi007.stderr: 14050K .......... .......... .......... .......... .......... 10% 22.8M 5s 2024-04-21T08:45:38.531 INFO:tasks.workunit.client.0.smithi007.stderr: 14100K .......... .......... .......... .......... .......... 10% 38.3M 5s 2024-04-21T08:45:38.533 INFO:tasks.workunit.client.0.smithi007.stderr: 14150K .......... .......... .......... .......... .......... 10% 33.3M 5s 2024-04-21T08:45:38.535 INFO:tasks.workunit.client.0.smithi007.stderr: 14200K .......... .......... .......... .......... .......... 10% 22.0M 5s 2024-04-21T08:45:38.536 INFO:tasks.workunit.client.0.smithi007.stderr: 14250K .......... .......... .......... .......... .......... 10% 37.0M 5s 2024-04-21T08:45:38.538 INFO:tasks.workunit.client.0.smithi007.stderr: 14300K .......... .......... .......... .......... .......... 10% 21.5M 5s 2024-04-21T08:45:38.541 INFO:tasks.workunit.client.0.smithi007.stderr: 14350K .......... .......... .......... .......... .......... 10% 19.1M 5s 2024-04-21T08:45:38.542 INFO:tasks.workunit.client.0.smithi007.stderr: 14400K .......... .......... .......... .......... .......... 10% 48.7M 5s 2024-04-21T08:45:38.544 INFO:tasks.workunit.client.0.smithi007.stderr: 14450K .......... .......... .......... .......... .......... 10% 22.5M 5s 2024-04-21T08:45:38.546 INFO:tasks.workunit.client.0.smithi007.stderr: 14500K .......... .......... .......... .......... .......... 10% 37.7M 5s 2024-04-21T08:45:38.548 INFO:tasks.workunit.client.0.smithi007.stderr: 14550K .......... .......... .......... .......... .......... 10% 25.0M 5s 2024-04-21T08:45:38.550 INFO:tasks.workunit.client.0.smithi007.stderr: 14600K .......... .......... .......... .......... .......... 10% 22.1M 5s 2024-04-21T08:45:38.551 INFO:tasks.workunit.client.0.smithi007.stderr: 14650K .......... .......... .......... .......... .......... 10% 61.4M 5s 2024-04-21T08:45:38.553 INFO:tasks.workunit.client.0.smithi007.stderr: 14700K .......... .......... .......... .......... .......... 10% 24.6M 5s 2024-04-21T08:45:38.554 INFO:tasks.workunit.client.0.smithi007.stderr: 14750K .......... .......... .......... .......... .......... 10% 35.2M 5s 2024-04-21T08:45:38.556 INFO:tasks.workunit.client.0.smithi007.stderr: 14800K .......... .......... .......... .......... .......... 10% 24.1M 5s 2024-04-21T08:45:38.557 INFO:tasks.workunit.client.0.smithi007.stderr: 14850K .......... .......... .......... .......... .......... 10% 31.8M 5s 2024-04-21T08:45:38.560 INFO:tasks.workunit.client.0.smithi007.stderr: 14900K .......... .......... .......... .......... .......... 11% 33.5M 5s 2024-04-21T08:45:38.562 INFO:tasks.workunit.client.0.smithi007.stderr: 14950K .......... .......... .......... .......... .......... 11% 18.5M 5s 2024-04-21T08:45:38.564 INFO:tasks.workunit.client.0.smithi007.stderr: 15000K .......... .......... .......... .......... .......... 11% 18.9M 5s 2024-04-21T08:45:38.566 INFO:tasks.workunit.client.0.smithi007.stderr: 15050K .......... .......... .......... .......... .......... 11% 41.0M 5s 2024-04-21T08:45:38.568 INFO:tasks.workunit.client.0.smithi007.stderr: 15100K .......... .......... .......... .......... .......... 11% 23.6M 5s 2024-04-21T08:45:38.569 INFO:tasks.workunit.client.0.smithi007.stderr: 15150K .......... .......... .......... .......... .......... 11% 43.0M 5s 2024-04-21T08:45:38.571 INFO:tasks.workunit.client.0.smithi007.stderr: 15200K .......... .......... .......... .......... .......... 11% 23.4M 5s 2024-04-21T08:45:38.573 INFO:tasks.workunit.client.0.smithi007.stderr: 15250K .......... .......... .......... .......... .......... 11% 42.8M 5s 2024-04-21T08:45:38.575 INFO:tasks.workunit.client.0.smithi007.stderr: 15300K .......... .......... .......... .......... .......... 11% 23.8M 5s 2024-04-21T08:45:38.576 INFO:tasks.workunit.client.0.smithi007.stderr: 15350K .......... .......... .......... .......... .......... 11% 24.1M 5s 2024-04-21T08:45:38.577 INFO:tasks.workunit.client.0.smithi007.stderr: 15400K .......... .......... .......... .......... .......... 11% 44.5M 5s 2024-04-21T08:45:38.581 INFO:tasks.workunit.client.0.smithi007.stderr: 15450K .......... .......... .......... .......... .......... 11% 25.3M 5s 2024-04-21T08:45:38.581 INFO:tasks.workunit.client.0.smithi007.stderr: 15500K .......... .......... .......... .......... .......... 11% 27.3M 5s 2024-04-21T08:45:38.588 INFO:tasks.workunit.client.0.smithi007.stderr: 15550K .......... .......... .......... .......... .......... 11% 35.5M 5s 2024-04-21T08:45:38.588 INFO:tasks.workunit.client.0.smithi007.stderr: 15600K .......... .......... .......... .......... .......... 11% 23.4M 5s 2024-04-21T08:45:38.588 INFO:tasks.workunit.client.0.smithi007.stderr: 15650K .......... .......... .......... .......... .......... 11% 49.4M 5s 2024-04-21T08:45:38.588 INFO:tasks.workunit.client.0.smithi007.stderr: 15700K .......... .......... .......... .......... .......... 11% 23.6M 5s 2024-04-21T08:45:38.590 INFO:tasks.workunit.client.0.smithi007.stderr: 15750K .......... .......... .......... .......... .......... 11% 22.5M 5s 2024-04-21T08:45:38.592 INFO:tasks.workunit.client.0.smithi007.stderr: 15800K .......... .......... .......... .......... .......... 11% 39.1M 5s 2024-04-21T08:45:38.594 INFO:tasks.workunit.client.0.smithi007.stderr: 15850K .......... .......... .......... .......... .......... 11% 24.8M 5s 2024-04-21T08:45:38.596 INFO:tasks.workunit.client.0.smithi007.stderr: 15900K .......... .......... .......... .......... .......... 11% 45.0M 5s 2024-04-21T08:45:38.597 INFO:tasks.workunit.client.0.smithi007.stderr: 15950K .......... .......... .......... .......... .......... 11% 25.5M 5s 2024-04-21T08:45:38.601 INFO:tasks.workunit.client.0.smithi007.stderr: 16000K .......... .......... .......... .......... .......... 11% 23.1M 5s 2024-04-21T08:45:38.601 INFO:tasks.workunit.client.0.smithi007.stderr: 16050K .......... .......... .......... .......... .......... 11% 31.5M 5s 2024-04-21T08:45:38.605 INFO:tasks.workunit.client.0.smithi007.stderr: 16100K .......... .......... .......... .......... .......... 11% 26.8M 5s 2024-04-21T08:45:38.606 INFO:tasks.workunit.client.0.smithi007.stderr: 16150K .......... .......... .......... .......... .......... 11% 61.1M 5s 2024-04-21T08:45:38.606 INFO:tasks.workunit.client.0.smithi007.stderr: 16200K .......... .......... .......... .......... .......... 11% 23.6M 5s 2024-04-21T08:45:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:38 smithi077 ceph-mon[31615]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-21T08:45:38.607 INFO:tasks.workunit.client.0.smithi007.stderr: 16250K .......... .......... .......... .......... .......... 12% 23.9M 5s 2024-04-21T08:45:38.608 INFO:tasks.workunit.client.0.smithi007.stderr: 16300K .......... .......... .......... .......... .......... 12% 28.7M 5s 2024-04-21T08:45:38.611 INFO:tasks.workunit.client.0.smithi007.stderr: 16350K .......... .......... .......... .......... .......... 12% 22.0M 5s 2024-04-21T08:45:38.612 INFO:tasks.workunit.client.0.smithi007.stderr: 16400K .......... .......... .......... .......... .......... 12% 24.6M 5s 2024-04-21T08:45:38.613 INFO:tasks.workunit.client.0.smithi007.stderr: 16450K .......... .......... .......... .......... .......... 12% 49.7M 5s 2024-04-21T08:45:38.615 INFO:tasks.workunit.client.0.smithi007.stderr: 16500K .......... .......... .......... .......... .......... 12% 24.5M 5s 2024-04-21T08:45:38.617 INFO:tasks.workunit.client.0.smithi007.stderr: 16550K .......... .......... .......... .......... .......... 12% 37.9M 5s 2024-04-21T08:45:38.619 INFO:tasks.workunit.client.0.smithi007.stderr: 16600K .......... .......... .......... .......... .......... 12% 25.1M 5s 2024-04-21T08:45:38.622 INFO:tasks.workunit.client.0.smithi007.stderr: 16650K .......... .......... .......... .......... .......... 12% 22.0M 5s 2024-04-21T08:45:38.622 INFO:tasks.workunit.client.0.smithi007.stderr: 16700K .......... .......... .......... .......... .......... 12% 50.1M 5s 2024-04-21T08:45:38.623 INFO:tasks.workunit.client.0.smithi007.stderr: 16750K .......... .......... .......... .......... .......... 12% 25.0M 5s 2024-04-21T08:45:38.625 INFO:tasks.workunit.client.0.smithi007.stderr: 16800K .......... .......... .......... .......... .......... 12% 37.7M 5s 2024-04-21T08:45:38.627 INFO:tasks.workunit.client.0.smithi007.stderr: 16850K .......... .......... .......... .......... .......... 12% 22.9M 5s 2024-04-21T08:45:38.629 INFO:tasks.workunit.client.0.smithi007.stderr: 16900K .......... .......... .......... .......... .......... 12% 25.0M 5s 2024-04-21T08:45:38.630 INFO:tasks.workunit.client.0.smithi007.stderr: 16950K .......... .......... .......... .......... .......... 12% 45.2M 5s 2024-04-21T08:45:38.632 INFO:tasks.workunit.client.0.smithi007.stderr: 17000K .......... .......... .......... .......... .......... 12% 23.7M 5s 2024-04-21T08:45:38.633 INFO:tasks.workunit.client.0.smithi007.stderr: 17050K .......... .......... .......... .......... .......... 12% 39.8M 5s 2024-04-21T08:45:38.635 INFO:tasks.workunit.client.0.smithi007.stderr: 17100K .......... .......... .......... .......... .......... 12% 23.7M 5s 2024-04-21T08:45:38.637 INFO:tasks.workunit.client.0.smithi007.stderr: 17150K .......... .......... .......... .......... .......... 12% 23.9M 5s 2024-04-21T08:45:38.638 INFO:tasks.workunit.client.0.smithi007.stderr: 17200K .......... .......... .......... .......... .......... 12% 62.2M 5s 2024-04-21T08:45:38.640 INFO:tasks.workunit.client.0.smithi007.stderr: 17250K .......... .......... .......... .......... .......... 12% 22.8M 5s 2024-04-21T08:45:38.642 INFO:tasks.workunit.client.0.smithi007.stderr: 17300K .......... .......... .......... .......... .......... 12% 28.1M 5s 2024-04-21T08:45:38.643 INFO:tasks.workunit.client.0.smithi007.stderr: 17350K .......... .......... .......... .......... .......... 12% 33.5M 5s 2024-04-21T08:45:38.645 INFO:tasks.workunit.client.0.smithi007.stderr: 17400K .......... .......... .......... .......... .......... 12% 22.4M 5s 2024-04-21T08:45:38.646 INFO:tasks.workunit.client.0.smithi007.stderr: 17450K .......... .......... .......... .......... .......... 12% 52.4M 5s 2024-04-21T08:45:38.648 INFO:tasks.workunit.client.0.smithi007.stderr: 17500K .......... .......... .......... .......... .......... 12% 22.1M 5s 2024-04-21T08:45:38.650 INFO:tasks.workunit.client.0.smithi007.stderr: 17550K .......... .......... .......... .......... .......... 12% 40.6M 5s 2024-04-21T08:45:38.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:38 smithi007 ceph-mon[28477]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-21T08:45:38.652 INFO:tasks.workunit.client.0.smithi007.stderr: 17600K .......... .......... .......... .......... .......... 13% 29.5M 5s 2024-04-21T08:45:38.654 INFO:tasks.workunit.client.0.smithi007.stderr: 17650K .......... .......... .......... .......... .......... 13% 19.3M 5s 2024-04-21T08:45:38.655 INFO:tasks.workunit.client.0.smithi007.stderr: 17700K .......... .......... .......... .......... .......... 13% 54.3M 5s 2024-04-21T08:45:38.657 INFO:tasks.workunit.client.0.smithi007.stderr: 17750K .......... .......... .......... .......... .......... 13% 21.0M 5s 2024-04-21T08:45:38.659 INFO:tasks.workunit.client.0.smithi007.stderr: 17800K .......... .......... .......... .......... .......... 13% 39.7M 5s 2024-04-21T08:45:38.661 INFO:tasks.workunit.client.0.smithi007.stderr: 17850K .......... .......... .......... .......... .......... 13% 22.1M 5s 2024-04-21T08:45:38.663 INFO:tasks.workunit.client.0.smithi007.stderr: 17900K .......... .......... .......... .......... .......... 13% 21.4M 5s 2024-04-21T08:45:38.664 INFO:tasks.workunit.client.0.smithi007.stderr: 17950K .......... .......... .......... .......... .......... 13% 54.7M 5s 2024-04-21T08:45:38.665 INFO:tasks.workunit.client.0.smithi007.stderr: 18000K .......... .......... .......... .......... .......... 13% 27.6M 5s 2024-04-21T08:45:38.667 INFO:tasks.workunit.client.0.smithi007.stderr: 18050K .......... .......... .......... .......... .......... 13% 27.1M 5s 2024-04-21T08:45:38.670 INFO:tasks.workunit.client.0.smithi007.stderr: 18100K .......... .......... .......... .......... .......... 13% 22.4M 5s 2024-04-21T08:45:38.671 INFO:tasks.workunit.client.0.smithi007.stderr: 18150K .......... .......... .......... .......... .......... 13% 25.1M 5s 2024-04-21T08:45:38.672 INFO:tasks.workunit.client.0.smithi007.stderr: 18200K .......... .......... .......... .......... .......... 13% 47.0M 5s 2024-04-21T08:45:38.674 INFO:tasks.workunit.client.0.smithi007.stderr: 18250K .......... .......... .......... .......... .......... 13% 25.7M 5s 2024-04-21T08:45:38.676 INFO:tasks.workunit.client.0.smithi007.stderr: 18300K .......... .......... .......... .......... .......... 13% 32.8M 5s 2024-04-21T08:45:38.678 INFO:tasks.workunit.client.0.smithi007.stderr: 18350K .......... .......... .......... .......... .......... 13% 22.8M 5s 2024-04-21T08:45:38.680 INFO:tasks.workunit.client.0.smithi007.stderr: 18400K .......... .......... .......... .......... .......... 13% 22.4M 5s 2024-04-21T08:45:38.683 INFO:tasks.workunit.client.0.smithi007.stderr: 18450K .......... .......... .......... .......... .......... 13% 26.8M 5s 2024-04-21T08:45:38.683 INFO:tasks.workunit.client.0.smithi007.stderr: 18500K .......... .......... .......... .......... .......... 13% 41.3M 5s 2024-04-21T08:45:38.685 INFO:tasks.workunit.client.0.smithi007.stderr: 18550K .......... .......... .......... .......... .......... 13% 38.9M 5s 2024-04-21T08:45:38.692 INFO:tasks.workunit.client.0.smithi007.stderr: 18600K .......... .......... .......... .......... .......... 13% 26.5M 5s 2024-04-21T08:45:38.692 INFO:tasks.workunit.client.0.smithi007.stderr: 18650K .......... .......... .......... .......... .......... 13% 20.2M 5s 2024-04-21T08:45:38.692 INFO:tasks.workunit.client.0.smithi007.stderr: 18700K .......... .......... .......... .......... .......... 13% 55.2M 5s 2024-04-21T08:45:38.692 INFO:tasks.workunit.client.0.smithi007.stderr: 18750K .......... .......... .......... .......... .......... 13% 22.4M 5s 2024-04-21T08:45:38.693 INFO:tasks.workunit.client.0.smithi007.stderr: 18800K .......... .......... .......... .......... .......... 13% 40.3M 5s 2024-04-21T08:45:38.695 INFO:tasks.workunit.client.0.smithi007.stderr: 18850K .......... .......... .......... .......... .......... 13% 23.6M 5s 2024-04-21T08:45:38.697 INFO:tasks.workunit.client.0.smithi007.stderr: 18900K .......... .......... .......... .......... .......... 13% 24.1M 5s 2024-04-21T08:45:38.698 INFO:tasks.workunit.client.0.smithi007.stderr: 18950K .......... .......... .......... .......... .......... 13% 54.5M 5s 2024-04-21T08:45:38.700 INFO:tasks.workunit.client.0.smithi007.stderr: 19000K .......... .......... .......... .......... .......... 14% 24.4M 5s 2024-04-21T08:45:38.701 INFO:tasks.workunit.client.0.smithi007.stderr: 19050K .......... .......... .......... .......... .......... 14% 37.8M 5s 2024-04-21T08:45:38.704 INFO:tasks.workunit.client.0.smithi007.stderr: 19100K .......... .......... .......... .......... .......... 14% 22.1M 5s 2024-04-21T08:45:38.706 INFO:tasks.workunit.client.0.smithi007.stderr: 19150K .......... .......... .......... .......... .......... 14% 25.6M 5s 2024-04-21T08:45:38.707 INFO:tasks.workunit.client.0.smithi007.stderr: 19200K .......... .......... .......... .......... .......... 14% 47.4M 5s 2024-04-21T08:45:38.708 INFO:tasks.workunit.client.0.smithi007.stderr: 19250K .......... .......... .......... .......... .......... 14% 25.7M 5s 2024-04-21T08:45:38.710 INFO:tasks.workunit.client.0.smithi007.stderr: 19300K .......... .......... .......... .......... .......... 14% 39.0M 5s 2024-04-21T08:45:38.712 INFO:tasks.workunit.client.0.smithi007.stderr: 19350K .......... .......... .......... .......... .......... 14% 22.0M 5s 2024-04-21T08:45:38.714 INFO:tasks.workunit.client.0.smithi007.stderr: 19400K .......... .......... .......... .......... .......... 14% 23.7M 5s 2024-04-21T08:45:38.715 INFO:tasks.workunit.client.0.smithi007.stderr: 19450K .......... .......... .......... .......... .......... 14% 56.3M 5s 2024-04-21T08:45:38.717 INFO:tasks.workunit.client.0.smithi007.stderr: 19500K .......... .......... .......... .......... .......... 14% 23.3M 5s 2024-04-21T08:45:38.718 INFO:tasks.workunit.client.0.smithi007.stderr: 19550K .......... .......... .......... .......... .......... 14% 41.5M 4s 2024-04-21T08:45:38.720 INFO:tasks.workunit.client.0.smithi007.stderr: 19600K .......... .......... .......... .......... .......... 14% 23.4M 4s 2024-04-21T08:45:38.722 INFO:tasks.workunit.client.0.smithi007.stderr: 19650K .......... .......... .......... .......... .......... 14% 22.5M 4s 2024-04-21T08:45:38.723 INFO:tasks.workunit.client.0.smithi007.stderr: 19700K .......... .......... .......... .......... .......... 14% 54.8M 4s 2024-04-21T08:45:38.725 INFO:tasks.workunit.client.0.smithi007.stderr: 19750K .......... .......... .......... .......... .......... 14% 24.3M 4s 2024-04-21T08:45:38.727 INFO:tasks.workunit.client.0.smithi007.stderr: 19800K .......... .......... .......... .......... .......... 14% 34.3M 4s 2024-04-21T08:45:38.728 INFO:tasks.workunit.client.0.smithi007.stderr: 19850K .......... .......... .......... .......... .......... 14% 26.2M 4s 2024-04-21T08:45:38.731 INFO:tasks.workunit.client.0.smithi007.stderr: 19900K .......... .......... .......... .......... .......... 14% 22.4M 4s 2024-04-21T08:45:38.731 INFO:tasks.workunit.client.0.smithi007.stderr: 19950K .......... .......... .......... .......... .......... 14% 56.1M 4s 2024-04-21T08:45:38.734 INFO:tasks.workunit.client.0.smithi007.stderr: 20000K .......... .......... .......... .......... .......... 14% 23.4M 4s 2024-04-21T08:45:38.735 INFO:tasks.workunit.client.0.smithi007.stderr: 20050K .......... .......... .......... .......... .......... 14% 31.1M 4s 2024-04-21T08:45:38.736 INFO:tasks.workunit.client.0.smithi007.stderr: 20100K .......... .......... .......... .......... .......... 14% 39.7M 4s 2024-04-21T08:45:38.739 INFO:tasks.workunit.client.0.smithi007.stderr: 20150K .......... .......... .......... .......... .......... 14% 19.4M 4s 2024-04-21T08:45:38.740 INFO:tasks.workunit.client.0.smithi007.stderr: 20200K .......... .......... .......... .......... .......... 14% 53.9M 4s 2024-04-21T08:45:38.742 INFO:tasks.workunit.client.0.smithi007.stderr: 20250K .......... .......... .......... .......... .......... 14% 23.1M 4s 2024-04-21T08:45:38.744 INFO:tasks.workunit.client.0.smithi007.stderr: 20300K .......... .......... .......... .......... .......... 14% 39.9M 4s 2024-04-21T08:45:38.745 INFO:tasks.workunit.client.0.smithi007.stderr: 20350K .......... .......... .......... .......... .......... 15% 21.8M 4s 2024-04-21T08:45:38.748 INFO:tasks.workunit.client.0.smithi007.stderr: 20400K .......... .......... .......... .......... .......... 15% 23.4M 4s 2024-04-21T08:45:38.748 INFO:tasks.workunit.client.0.smithi007.stderr: 20450K .......... .......... .......... .......... .......... 15% 57.6M 4s 2024-04-21T08:45:38.751 INFO:tasks.workunit.client.0.smithi007.stderr: 20500K .......... .......... .......... .......... .......... 15% 16.4M 4s 2024-04-21T08:45:38.754 INFO:tasks.workunit.client.0.smithi007.stderr: 20550K .......... .......... .......... .......... .......... 15% 28.2M 4s 2024-04-21T08:45:38.756 INFO:tasks.workunit.client.0.smithi007.stderr: 20600K .......... .......... .......... .......... .......... 15% 19.2M 4s 2024-04-21T08:45:38.757 INFO:tasks.workunit.client.0.smithi007.stderr: 20650K .......... .......... .......... .......... .......... 15% 33.6M 4s 2024-04-21T08:45:38.759 INFO:tasks.workunit.client.0.smithi007.stderr: 20700K .......... .......... .......... .......... .......... 15% 33.4M 4s 2024-04-21T08:45:38.761 INFO:tasks.workunit.client.0.smithi007.stderr: 20750K .......... .......... .......... .......... .......... 15% 23.0M 4s 2024-04-21T08:45:38.763 INFO:tasks.workunit.client.0.smithi007.stderr: 20800K .......... .......... .......... .......... .......... 15% 24.9M 4s 2024-04-21T08:45:38.764 INFO:tasks.workunit.client.0.smithi007.stderr: 20850K .......... .......... .......... .......... .......... 15% 34.4M 4s 2024-04-21T08:45:38.766 INFO:tasks.workunit.client.0.smithi007.stderr: 20900K .......... .......... .......... .......... .......... 15% 24.0M 4s 2024-04-21T08:45:38.767 INFO:tasks.workunit.client.0.smithi007.stderr: 20950K .......... .......... .......... .......... .......... 15% 64.7M 4s 2024-04-21T08:45:38.769 INFO:tasks.workunit.client.0.smithi007.stderr: 21000K .......... .......... .......... .......... .......... 15% 22.4M 4s 2024-04-21T08:45:38.771 INFO:tasks.workunit.client.0.smithi007.stderr: 21050K .......... .......... .......... .......... .......... 15% 25.0M 4s 2024-04-21T08:45:38.772 INFO:tasks.workunit.client.0.smithi007.stderr: 21100K .......... .......... .......... .......... .......... 15% 34.8M 4s 2024-04-21T08:45:38.774 INFO:tasks.workunit.client.0.smithi007.stderr: 21150K .......... .......... .......... .......... .......... 15% 31.0M 4s 2024-04-21T08:45:38.776 INFO:tasks.workunit.client.0.smithi007.stderr: 21200K .......... .......... .......... .......... .......... 15% 39.8M 4s 2024-04-21T08:45:38.777 INFO:tasks.workunit.client.0.smithi007.stderr: 21250K .......... .......... .......... .......... .......... 15% 23.5M 4s 2024-04-21T08:45:38.779 INFO:tasks.workunit.client.0.smithi007.stderr: 21300K .......... .......... .......... .......... .......... 15% 29.1M 4s 2024-04-21T08:45:38.781 INFO:tasks.workunit.client.0.smithi007.stderr: 21350K .......... .......... .......... .......... .......... 15% 26.1M 4s 2024-04-21T08:45:38.783 INFO:tasks.workunit.client.0.smithi007.stderr: 21400K .......... .......... .......... .......... .......... 15% 25.6M 4s 2024-04-21T08:45:38.785 INFO:tasks.workunit.client.0.smithi007.stderr: 21450K .......... .......... .......... .......... .......... 15% 56.2M 4s 2024-04-21T08:45:38.786 INFO:tasks.workunit.client.0.smithi007.stderr: 21500K .......... .......... .......... .......... .......... 15% 22.9M 4s 2024-04-21T08:45:38.788 INFO:tasks.workunit.client.0.smithi007.stderr: 21550K .......... .......... .......... .......... .......... 15% 22.6M 4s 2024-04-21T08:45:38.789 INFO:tasks.workunit.client.0.smithi007.stderr: 21600K .......... .......... .......... .......... .......... 15% 45.7M 4s 2024-04-21T08:45:38.791 INFO:tasks.workunit.client.0.smithi007.stderr: 21650K .......... .......... .......... .......... .......... 15% 22.2M 4s 2024-04-21T08:45:38.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:38 smithi016 ceph-mon[31259]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-21T08:45:38.883 INFO:tasks.workunit.client.0.smithi007.stderr: 21700K .......... .......... .......... .......... .......... 16% 595K 5s 2024-04-21T08:45:38.883 INFO:tasks.workunit.client.0.smithi007.stderr: 21750K .......... .......... .......... .......... .......... 16% 542M 5s 2024-04-21T08:45:38.883 INFO:tasks.workunit.client.0.smithi007.stderr: 21800K .......... .......... .......... .......... .......... 16% 491M 5s 2024-04-21T08:45:38.883 INFO:tasks.workunit.client.0.smithi007.stderr: 21850K .......... .......... .......... .......... .......... 16% 441M 5s 2024-04-21T08:45:38.883 INFO:tasks.workunit.client.0.smithi007.stderr: 21900K .......... .......... .......... .......... .......... 16% 527M 5s 2024-04-21T08:45:38.883 INFO:tasks.workunit.client.0.smithi007.stderr: 21950K .......... .......... .......... .......... .......... 16% 517M 5s 2024-04-21T08:45:38.884 INFO:tasks.workunit.client.0.smithi007.stderr: 22000K .......... .......... .......... .......... .......... 16% 556M 5s 2024-04-21T08:45:38.884 INFO:tasks.workunit.client.0.smithi007.stderr: 22050K .......... .......... .......... .......... .......... 16% 491M 5s 2024-04-21T08:45:38.884 INFO:tasks.workunit.client.0.smithi007.stderr: 22100K .......... .......... .......... .......... .......... 16% 489M 5s 2024-04-21T08:45:38.884 INFO:tasks.workunit.client.0.smithi007.stderr: 22150K .......... .......... .......... .......... .......... 16% 502M 5s 2024-04-21T08:45:38.887 INFO:tasks.workunit.client.0.smithi007.stderr: 22200K .......... .......... .......... .......... .......... 16% 4.36M 5s 2024-04-21T08:45:38.890 INFO:tasks.workunit.client.0.smithi007.stderr: 22250K .......... .......... .......... .......... .......... 16% 85.5M 5s 2024-04-21T08:45:38.894 INFO:tasks.workunit.client.0.smithi007.stderr: 22300K .......... .......... .......... .......... .......... 16% 162M 5s 2024-04-21T08:45:38.894 INFO:tasks.workunit.client.0.smithi007.stderr: 22350K .......... .......... .......... .......... .......... 16% 487M 5s 2024-04-21T08:45:38.895 INFO:tasks.workunit.client.0.smithi007.stderr: 22400K .......... .......... .......... .......... .......... 16% 26.2M 5s 2024-04-21T08:45:38.956 INFO:tasks.workunit.client.0.smithi007.stderr: 22450K .......... .......... .......... .......... .......... 16% 69.7M 5s 2024-04-21T08:45:38.956 INFO:tasks.workunit.client.0.smithi007.stderr: 22500K .......... .......... .......... .......... .......... 16% 91.8M 5s 2024-04-21T08:45:38.956 INFO:tasks.workunit.client.0.smithi007.stderr: 22550K .......... .......... .......... .......... .......... 16% 155M 5s 2024-04-21T08:45:38.956 INFO:tasks.workunit.client.0.smithi007.stderr: 22600K .......... .......... .......... .......... .......... 16% 517M 5s 2024-04-21T08:45:38.956 INFO:tasks.workunit.client.0.smithi007.stderr: 22650K .......... .......... .......... .......... .......... 16% 490M 5s 2024-04-21T08:45:38.956 INFO:tasks.workunit.client.0.smithi007.stderr: 22700K .......... .......... .......... .......... .......... 16% 415M 5s 2024-04-21T08:45:38.956 INFO:tasks.workunit.client.0.smithi007.stderr: 22750K .......... .......... .......... .......... .......... 16% 572M 5s 2024-04-21T08:45:38.956 INFO:tasks.workunit.client.0.smithi007.stderr: 22800K .......... .......... .......... .......... .......... 16% 555M 5s 2024-04-21T08:45:38.956 INFO:tasks.workunit.client.0.smithi007.stderr: 22850K .......... .......... .......... .......... .......... 16% 520M 5s 2024-04-21T08:45:38.956 INFO:tasks.workunit.client.0.smithi007.stderr: 22900K .......... .......... .......... .......... .......... 16% 506M 5s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 22950K .......... .......... .......... .......... .......... 16% 575M 5s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23000K .......... .......... .......... .......... .......... 16% 570M 5s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23050K .......... .......... .......... .......... .......... 17% 558M 5s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23100K .......... .......... .......... .......... .......... 17% 499M 5s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23150K .......... .......... .......... .......... .......... 17% 563M 5s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23200K .......... .......... .......... .......... .......... 17% 580M 5s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23250K .......... .......... .......... .......... .......... 17% 536M 5s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23300K .......... .......... .......... .......... .......... 17% 479M 4s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23350K .......... .......... .......... .......... .......... 17% 581M 4s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23400K .......... .......... .......... .......... .......... 17% 577M 4s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23450K .......... .......... .......... .......... .......... 17% 580M 4s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23500K .......... .......... .......... .......... .......... 17% 504M 4s 2024-04-21T08:45:38.957 INFO:tasks.workunit.client.0.smithi007.stderr: 23550K .......... .......... .......... .......... .......... 17% 554M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 23600K .......... .......... .......... .......... .......... 17% 570M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 23650K .......... .......... .......... .......... .......... 17% 390M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 23700K .......... .......... .......... .......... .......... 17% 484M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 23750K .......... .......... .......... .......... .......... 17% 565M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 23800K .......... .......... .......... .......... .......... 17% 488M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 23850K .......... .......... .......... .......... .......... 17% 570M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 23900K .......... .......... .......... .......... .......... 17% 495M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 23950K .......... .......... .......... .......... .......... 17% 3.56M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 24000K .......... .......... .......... .......... .......... 17% 48.3M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 24050K .......... .......... .......... .......... .......... 17% 235M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 24100K .......... .......... .......... .......... .......... 17% 434M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 24150K .......... .......... .......... .......... .......... 17% 521M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 24200K .......... .......... .......... .......... .......... 17% 527M 4s 2024-04-21T08:45:38.958 INFO:tasks.workunit.client.0.smithi007.stderr: 24250K .......... .......... .......... .......... .......... 17% 533M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24300K .......... .......... .......... .......... .......... 17% 473M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24350K .......... .......... .......... .......... .......... 17% 529M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24400K .......... .......... .......... .......... .......... 18% 524M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24450K .......... .......... .......... .......... .......... 18% 529M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24500K .......... .......... .......... .......... .......... 18% 441M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24550K .......... .......... .......... .......... .......... 18% 503M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24600K .......... .......... .......... .......... .......... 18% 529M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24650K .......... .......... .......... .......... .......... 18% 548M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24700K .......... .......... .......... .......... .......... 18% 484M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24750K .......... .......... .......... .......... .......... 18% 548M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24800K .......... .......... .......... .......... .......... 18% 579M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24850K .......... .......... .......... .......... .......... 18% 546M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24900K .......... .......... .......... .......... .......... 18% 481M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 24950K .......... .......... .......... .......... .......... 18% 594M 4s 2024-04-21T08:45:38.959 INFO:tasks.workunit.client.0.smithi007.stderr: 25000K .......... .......... .......... .......... .......... 18% 601M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25050K .......... .......... .......... .......... .......... 18% 484M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25100K .......... .......... .......... .......... .......... 18% 303M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25150K .......... .......... .......... .......... .......... 18% 19.5M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25200K .......... .......... .......... .......... .......... 18% 32.3M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25250K .......... .......... .......... .......... .......... 18% 74.0M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25300K .......... .......... .......... .......... .......... 18% 24.1M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25350K .......... .......... .......... .......... .......... 18% 34.0M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25400K .......... .......... .......... .......... .......... 18% 21.7M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25450K .......... .......... .......... .......... .......... 18% 24.6M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25500K .......... .......... .......... .......... .......... 18% 81.7M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25550K .......... .......... .......... .......... .......... 18% 23.6M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25600K .......... .......... .......... .......... .......... 18% 28.6M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25650K .......... .......... .......... .......... .......... 18% 32.9M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25700K .......... .......... .......... .......... .......... 18% 27.8M 4s 2024-04-21T08:45:38.960 INFO:tasks.workunit.client.0.smithi007.stderr: 25750K .......... .......... .......... .......... .......... 19% 23.8M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 25800K .......... .......... .......... .......... .......... 19% 29.7M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 25850K .......... .......... .......... .......... .......... 19% 24.9M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 25900K .......... .......... .......... .......... .......... 19% 28.9M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 25950K .......... .......... .......... .......... .......... 19% 25.6M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 26000K .......... .......... .......... .......... .......... 19% 39.5M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 26050K .......... .......... .......... .......... .......... 19% 28.1M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 26100K .......... .......... .......... .......... .......... 19% 26.5M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 26150K .......... .......... .......... .......... .......... 19% 28.5M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 26200K .......... .......... .......... .......... .......... 19% 24.6M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 26250K .......... .......... .......... .......... .......... 19% 21.3M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 26300K .......... .......... .......... .......... .......... 19% 71.2M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 26350K .......... .......... .......... .......... .......... 19% 33.7M 4s 2024-04-21T08:45:38.961 INFO:tasks.workunit.client.0.smithi007.stderr: 26400K .......... .......... .......... .......... .......... 19% 22.5M 4s 2024-04-21T08:45:38.964 INFO:tasks.workunit.client.0.smithi007.stderr: 26450K .......... .......... .......... .......... .......... 19% 25.0M 4s 2024-04-21T08:45:38.964 INFO:tasks.workunit.client.0.smithi007.stderr: 26500K .......... .......... .......... .......... .......... 19% 24.5M 4s 2024-04-21T08:45:38.965 INFO:tasks.workunit.client.0.smithi007.stderr: 26550K .......... .......... .......... .......... .......... 19% 67.2M 4s 2024-04-21T08:45:38.965 INFO:tasks.workunit.client.0.smithi007.stderr: 26600K .......... .......... .......... .......... .......... 19% 32.9M 4s 2024-04-21T08:45:38.965 INFO:tasks.workunit.client.0.smithi007.stderr: 26650K .......... .......... .......... .......... .......... 19% 22.4M 4s 2024-04-21T08:45:38.966 INFO:tasks.workunit.client.0.smithi007.stderr: 26700K .......... .......... .......... .......... .......... 19% 25.1M 4s 2024-04-21T08:45:38.967 INFO:tasks.workunit.client.0.smithi007.stderr: 26750K .......... .......... .......... .......... .......... 19% 48.3M 4s 2024-04-21T08:45:38.968 INFO:tasks.workunit.client.0.smithi007.stderr: 26800K .......... .......... .......... .......... .......... 19% 33.6M 4s 2024-04-21T08:45:38.971 INFO:tasks.workunit.client.0.smithi007.stderr: 26850K .......... .......... .......... .......... .......... 19% 20.0M 4s 2024-04-21T08:45:38.973 INFO:tasks.workunit.client.0.smithi007.stderr: 26900K .......... .......... .......... .......... .......... 19% 27.2M 4s 2024-04-21T08:45:38.975 INFO:tasks.workunit.client.0.smithi007.stderr: 26950K .......... .......... .......... .......... .......... 19% 21.7M 4s 2024-04-21T08:45:38.976 INFO:tasks.workunit.client.0.smithi007.stderr: 27000K .......... .......... .......... .......... .......... 19% 30.8M 4s 2024-04-21T08:45:38.977 INFO:tasks.workunit.client.0.smithi007.stderr: 27050K .......... .......... .......... .......... .......... 19% 69.4M 4s 2024-04-21T08:45:38.979 INFO:tasks.workunit.client.0.smithi007.stderr: 27100K .......... .......... .......... .......... .......... 19% 20.9M 4s 2024-04-21T08:45:38.982 INFO:tasks.workunit.client.0.smithi007.stderr: 27150K .......... .......... .......... .......... .......... 20% 26.1M 4s 2024-04-21T08:45:38.983 INFO:tasks.workunit.client.0.smithi007.stderr: 27200K .......... .......... .......... .......... .......... 20% 31.4M 4s 2024-04-21T08:45:38.985 INFO:tasks.workunit.client.0.smithi007.stderr: 27250K .......... .......... .......... .......... .......... 20% 25.6M 4s 2024-04-21T08:45:38.985 INFO:tasks.workunit.client.0.smithi007.stderr: 27300K .......... .......... .......... .......... .......... 20% 60.2M 4s 2024-04-21T08:45:38.988 INFO:tasks.workunit.client.0.smithi007.stderr: 27350K .......... .......... .......... .......... .......... 20% 29.4M 4s 2024-04-21T08:45:38.989 INFO:tasks.workunit.client.0.smithi007.stderr: 27400K .......... .......... .......... .......... .......... 20% 28.7M 4s 2024-04-21T08:45:38.992 INFO:tasks.workunit.client.0.smithi007.stderr: 27450K .......... .......... .......... .......... .......... 20% 21.4M 4s 2024-04-21T08:45:38.994 INFO:tasks.workunit.client.0.smithi007.stderr: 27500K .......... .......... .......... .......... .......... 20% 20.5M 4s 2024-04-21T08:45:38.994 INFO:tasks.workunit.client.0.smithi007.stderr: 27550K .......... .......... .......... .......... .......... 20% 180M 4s 2024-04-21T08:45:38.996 INFO:tasks.workunit.client.0.smithi007.stderr: 27600K .......... .......... .......... .......... .......... 20% 26.1M 4s 2024-04-21T08:45:38.998 INFO:tasks.workunit.client.0.smithi007.stderr: 27650K .......... .......... .......... .......... .......... 20% 22.8M 4s 2024-04-21T08:45:39.000 INFO:tasks.workunit.client.0.smithi007.stderr: 27700K .......... .......... .......... .......... .......... 20% 26.5M 4s 2024-04-21T08:45:39.002 INFO:tasks.workunit.client.0.smithi007.stderr: 27750K .......... .......... .......... .......... .......... 20% 24.9M 4s 2024-04-21T08:45:39.002 INFO:tasks.workunit.client.0.smithi007.stderr: 27800K .......... .......... .......... .......... .......... 20% 99.9M 4s 2024-04-21T08:45:39.004 INFO:tasks.workunit.client.0.smithi007.stderr: 27850K .......... .......... .......... .......... .......... 20% 28.8M 4s 2024-04-21T08:45:39.007 INFO:tasks.workunit.client.0.smithi007.stderr: 27900K .......... .......... .......... .......... .......... 20% 21.7M 4s 2024-04-21T08:45:39.008 INFO:tasks.workunit.client.0.smithi007.stderr: 27950K .......... .......... .......... .......... .......... 20% 25.5M 4s 2024-04-21T08:45:39.011 INFO:tasks.workunit.client.0.smithi007.stderr: 28000K .......... .......... .......... .......... .......... 20% 24.1M 4s 2024-04-21T08:45:39.011 INFO:tasks.workunit.client.0.smithi007.stderr: 28050K .......... .......... .......... .......... .......... 20% 112M 4s 2024-04-21T08:45:39.013 INFO:tasks.workunit.client.0.smithi007.stderr: 28100K .......... .......... .......... .......... .......... 20% 26.5M 4s 2024-04-21T08:45:39.015 INFO:tasks.workunit.client.0.smithi007.stderr: 28150K .......... .......... .......... .......... .......... 20% 25.0M 4s 2024-04-21T08:45:39.017 INFO:tasks.workunit.client.0.smithi007.stderr: 28200K .......... .......... .......... .......... .......... 20% 24.5M 4s 2024-04-21T08:45:39.018 INFO:tasks.workunit.client.0.smithi007.stderr: 28250K .......... .......... .......... .......... .......... 20% 25.0M 4s 2024-04-21T08:45:39.019 INFO:tasks.workunit.client.0.smithi007.stderr: 28300K .......... .......... .......... .......... .......... 20% 64.1M 4s 2024-04-21T08:45:39.021 INFO:tasks.workunit.client.0.smithi007.stderr: 28350K .......... .......... .......... .......... .......... 20% 24.3M 4s 2024-04-21T08:45:39.022 INFO:tasks.workunit.client.0.smithi007.stderr: 28400K .......... .......... .......... .......... .......... 20% 33.8M 4s 2024-04-21T08:45:39.024 INFO:tasks.workunit.client.0.smithi007.stderr: 28450K .......... .......... .......... .......... .......... 20% 22.0M 4s 2024-04-21T08:45:39.026 INFO:tasks.workunit.client.0.smithi007.stderr: 28500K .......... .......... .......... .......... .......... 21% 25.0M 4s 2024-04-21T08:45:39.028 INFO:tasks.workunit.client.0.smithi007.stderr: 28550K .......... .......... .......... .......... .......... 21% 46.4M 4s 2024-04-21T08:45:39.029 INFO:tasks.workunit.client.0.smithi007.stderr: 28600K .......... .......... .......... .......... .......... 21% 24.8M 4s 2024-04-21T08:45:39.031 INFO:tasks.workunit.client.0.smithi007.stderr: 28650K .......... .......... .......... .......... .......... 21% 38.4M 4s 2024-04-21T08:45:39.034 INFO:tasks.workunit.client.0.smithi007.stderr: 28700K .......... .......... .......... .......... .......... 21% 16.9M 4s 2024-04-21T08:45:39.036 INFO:tasks.workunit.client.0.smithi007.stderr: 28750K .......... .......... .......... .......... .......... 21% 20.1M 4s 2024-04-21T08:45:39.037 INFO:tasks.workunit.client.0.smithi007.stderr: 28800K .......... .......... .......... .......... .......... 21% 37.6M 4s 2024-04-21T08:45:39.039 INFO:tasks.workunit.client.0.smithi007.stderr: 28850K .......... .......... .......... .......... .......... 21% 27.6M 4s 2024-04-21T08:45:39.041 INFO:tasks.workunit.client.0.smithi007.stderr: 28900K .......... .......... .......... .......... .......... 21% 29.9M 4s 2024-04-21T08:45:39.043 INFO:tasks.workunit.client.0.smithi007.stderr: 28950K .......... .......... .......... .......... .......... 21% 22.3M 4s 2024-04-21T08:45:39.045 INFO:tasks.workunit.client.0.smithi007.stderr: 29000K .......... .......... .......... .......... .......... 21% 25.2M 4s 2024-04-21T08:45:39.046 INFO:tasks.workunit.client.0.smithi007.stderr: 29050K .......... .......... .......... .......... .......... 21% 46.1M 4s 2024-04-21T08:45:39.048 INFO:tasks.workunit.client.0.smithi007.stderr: 29100K .......... .......... .......... .......... .......... 21% 24.4M 4s 2024-04-21T08:45:39.049 INFO:tasks.workunit.client.0.smithi007.stderr: 29150K .......... .......... .......... .......... .......... 21% 39.4M 4s 2024-04-21T08:45:39.051 INFO:tasks.workunit.client.0.smithi007.stderr: 29200K .......... .......... .......... .......... .......... 21% 22.7M 4s 2024-04-21T08:45:39.054 INFO:tasks.workunit.client.0.smithi007.stderr: 29250K .......... .......... .......... .......... .......... 21% 23.4M 4s 2024-04-21T08:45:39.054 INFO:tasks.workunit.client.0.smithi007.stderr: 29300K .......... .......... .......... .......... .......... 21% 57.7M 4s 2024-04-21T08:45:39.056 INFO:tasks.workunit.client.0.smithi007.stderr: 29350K .......... .......... .......... .......... .......... 21% 22.3M 4s 2024-04-21T08:45:39.059 INFO:tasks.workunit.client.0.smithi007.stderr: 29400K .......... .......... .......... .......... .......... 21% 35.9M 4s 2024-04-21T08:45:39.060 INFO:tasks.workunit.client.0.smithi007.stderr: 29450K .......... .......... .......... .......... .......... 21% 27.7M 4s 2024-04-21T08:45:39.062 INFO:tasks.workunit.client.0.smithi007.stderr: 29500K .......... .......... .......... .......... .......... 21% 20.6M 4s 2024-04-21T08:45:39.063 INFO:tasks.workunit.client.0.smithi007.stderr: 29550K .......... .......... .......... .......... .......... 21% 54.1M 4s 2024-04-21T08:45:39.065 INFO:tasks.workunit.client.0.smithi007.stderr: 29600K .......... .......... .......... .......... .......... 21% 23.7M 4s 2024-04-21T08:45:39.067 INFO:tasks.workunit.client.0.smithi007.stderr: 29650K .......... .......... .......... .......... .......... 21% 36.6M 4s 2024-04-21T08:45:39.068 INFO:tasks.workunit.client.0.smithi007.stderr: 29700K .......... .......... .......... .......... .......... 21% 25.4M 4s 2024-04-21T08:45:39.070 INFO:tasks.workunit.client.0.smithi007.stderr: 29750K .......... .......... .......... .......... .......... 21% 24.7M 4s 2024-04-21T08:45:39.071 INFO:tasks.workunit.client.0.smithi007.stderr: 29800K .......... .......... .......... .......... .......... 21% 52.0M 4s 2024-04-21T08:45:39.073 INFO:tasks.workunit.client.0.smithi007.stderr: 29850K .......... .......... .......... .......... .......... 22% 22.6M 4s 2024-04-21T08:45:39.075 INFO:tasks.workunit.client.0.smithi007.stderr: 29900K .......... .......... .......... .......... .......... 22% 36.2M 4s 2024-04-21T08:45:39.077 INFO:tasks.workunit.client.0.smithi007.stderr: 29950K .......... .......... .......... .......... .......... 22% 24.3M 4s 2024-04-21T08:45:39.079 INFO:tasks.workunit.client.0.smithi007.stderr: 30000K .......... .......... .......... .......... .......... 22% 24.2M 4s 2024-04-21T08:45:39.080 INFO:tasks.workunit.client.0.smithi007.stderr: 30050K .......... .......... .......... .......... .......... 22% 45.5M 4s 2024-04-21T08:45:39.082 INFO:tasks.workunit.client.0.smithi007.stderr: 30100K .......... .......... .......... .......... .......... 22% 24.2M 4s 2024-04-21T08:45:39.084 INFO:tasks.workunit.client.0.smithi007.stderr: 30150K .......... .......... .......... .......... .......... 22% 24.7M 4s 2024-04-21T08:45:39.085 INFO:tasks.workunit.client.0.smithi007.stderr: 30200K .......... .......... .......... .......... .......... 22% 35.8M 4s 2024-04-21T08:45:39.087 INFO:tasks.workunit.client.0.smithi007.stderr: 30250K .......... .......... .......... .......... .......... 22% 24.8M 4s 2024-04-21T08:45:39.088 INFO:tasks.workunit.client.0.smithi007.stderr: 30300K .......... .......... .......... .......... .......... 22% 47.1M 4s 2024-04-21T08:45:39.091 INFO:tasks.workunit.client.0.smithi007.stderr: 30350K .......... .......... .......... .......... .......... 22% 21.0M 4s 2024-04-21T08:45:39.093 INFO:tasks.workunit.client.0.smithi007.stderr: 30400K .......... .......... .......... .......... .......... 22% 21.9M 4s 2024-04-21T08:45:39.094 INFO:tasks.workunit.client.0.smithi007.stderr: 30450K .......... .......... .......... .......... .......... 22% 36.7M 4s 2024-04-21T08:45:39.096 INFO:tasks.workunit.client.0.smithi007.stderr: 30500K .......... .......... .......... .......... .......... 22% 24.2M 4s 2024-04-21T08:45:39.098 INFO:tasks.workunit.client.0.smithi007.stderr: 30550K .......... .......... .......... .......... .......... 22% 48.0M 4s 2024-04-21T08:45:39.100 INFO:tasks.workunit.client.0.smithi007.stderr: 30600K .......... .......... .......... .......... .......... 22% 19.1M 4s 2024-04-21T08:45:39.102 INFO:tasks.workunit.client.0.smithi007.stderr: 30650K .......... .......... .......... .......... .......... 22% 23.6M 4s 2024-04-21T08:45:39.103 INFO:tasks.workunit.client.0.smithi007.stderr: 30700K .......... .......... .......... .......... .......... 22% 41.3M 4s 2024-04-21T08:45:39.105 INFO:tasks.workunit.client.0.smithi007.stderr: 30750K .......... .......... .......... .......... .......... 22% 23.3M 4s 2024-04-21T08:45:39.107 INFO:tasks.workunit.client.0.smithi007.stderr: 30800K .......... .......... .......... .......... .......... 22% 58.6M 4s 2024-04-21T08:45:39.108 INFO:tasks.workunit.client.0.smithi007.stderr: 30850K .......... .......... .......... .......... .......... 22% 22.2M 4s 2024-04-21T08:45:39.110 INFO:tasks.workunit.client.0.smithi007.stderr: 30900K .......... .......... .......... .......... .......... 22% 23.7M 4s 2024-04-21T08:45:39.111 INFO:tasks.workunit.client.0.smithi007.stderr: 30950K .......... .......... .......... .......... .......... 22% 40.7M 4s 2024-04-21T08:45:39.113 INFO:tasks.workunit.client.0.smithi007.stderr: 31000K .......... .......... .......... .......... .......... 22% 24.4M 4s 2024-04-21T08:45:39.116 INFO:tasks.workunit.client.0.smithi007.stderr: 31050K .......... .......... .......... .......... .......... 22% 22.5M 4s 2024-04-21T08:45:39.116 INFO:tasks.workunit.client.0.smithi007.stderr: 31100K .......... .......... .......... .......... .......... 22% 47.3M 4s 2024-04-21T08:45:39.118 INFO:tasks.workunit.client.0.smithi007.stderr: 31150K .......... .......... .......... .......... .......... 22% 24.4M 4s 2024-04-21T08:45:39.120 INFO:tasks.workunit.client.0.smithi007.stderr: 31200K .......... .......... .......... .......... .......... 23% 40.9M 4s 2024-04-21T08:45:39.122 INFO:tasks.workunit.client.0.smithi007.stderr: 31250K .......... .......... .......... .......... .......... 23% 23.3M 4s 2024-04-21T08:45:39.124 INFO:tasks.workunit.client.0.smithi007.stderr: 31300K .......... .......... .......... .......... .......... 23% 21.4M 4s 2024-04-21T08:45:39.125 INFO:tasks.workunit.client.0.smithi007.stderr: 31350K .......... .......... .......... .......... .......... 23% 49.0M 4s 2024-04-21T08:45:39.128 INFO:tasks.workunit.client.0.smithi007.stderr: 31400K .......... .......... .......... .......... .......... 23% 19.0M 4s 2024-04-21T08:45:39.129 INFO:tasks.workunit.client.0.smithi007.stderr: 31450K .......... .......... .......... .......... .......... 23% 42.7M 4s 2024-04-21T08:45:39.131 INFO:tasks.workunit.client.0.smithi007.stderr: 31500K .......... .......... .......... .......... .......... 23% 22.5M 4s 2024-04-21T08:45:39.133 INFO:tasks.workunit.client.0.smithi007.stderr: 31550K .......... .......... .......... .......... .......... 23% 23.5M 4s 2024-04-21T08:45:39.134 INFO:tasks.workunit.client.0.smithi007.stderr: 31600K .......... .......... .......... .......... .......... 23% 56.1M 4s 2024-04-21T08:45:39.136 INFO:tasks.workunit.client.0.smithi007.stderr: 31650K .......... .......... .......... .......... .......... 23% 23.9M 4s 2024-04-21T08:45:39.138 INFO:tasks.workunit.client.0.smithi007.stderr: 31700K .......... .......... .......... .......... .......... 23% 41.1M 4s 2024-04-21T08:45:39.139 INFO:tasks.workunit.client.0.smithi007.stderr: 31750K .......... .......... .......... .......... .......... 23% 29.6M 4s 2024-04-21T08:45:39.141 INFO:tasks.workunit.client.0.smithi007.stderr: 31800K .......... .......... .......... .......... .......... 23% 19.4M 4s 2024-04-21T08:45:39.142 INFO:tasks.workunit.client.0.smithi007.stderr: 31850K .......... .......... .......... .......... .......... 23% 48.1M 4s 2024-04-21T08:45:39.144 INFO:tasks.workunit.client.0.smithi007.stderr: 31900K .......... .......... .......... .......... .......... 23% 24.2M 4s 2024-04-21T08:45:39.146 INFO:tasks.workunit.client.0.smithi007.stderr: 31950K .......... .......... .......... .......... .......... 23% 34.4M 4s 2024-04-21T08:45:39.147 INFO:tasks.workunit.client.0.smithi007.stderr: 32000K .......... .......... .......... .......... .......... 23% 35.1M 4s 2024-04-21T08:45:39.150 INFO:tasks.workunit.client.0.smithi007.stderr: 32050K .......... .......... .......... .......... .......... 23% 16.9M 4s 2024-04-21T08:45:39.151 INFO:tasks.workunit.client.0.smithi007.stderr: 32100K .......... .......... .......... .......... .......... 23% 43.9M 4s 2024-04-21T08:45:39.154 INFO:tasks.workunit.client.0.smithi007.stderr: 32150K .......... .......... .......... .......... .......... 23% 20.1M 4s 2024-04-21T08:45:39.155 INFO:tasks.workunit.client.0.smithi007.stderr: 32200K .......... .......... .......... .......... .......... 23% 39.9M 4s 2024-04-21T08:45:39.156 INFO:tasks.workunit.client.0.smithi007.stderr: 32250K .......... .......... .......... .......... .......... 23% 28.3M 4s 2024-04-21T08:45:39.159 INFO:tasks.workunit.client.0.smithi007.stderr: 32300K .......... .......... .......... .......... .......... 23% 19.7M 4s 2024-04-21T08:45:39.160 INFO:tasks.workunit.client.0.smithi007.stderr: 32350K .......... .......... .......... .......... .......... 23% 50.1M 4s 2024-04-21T08:45:39.162 INFO:tasks.workunit.client.0.smithi007.stderr: 32400K .......... .......... .......... .......... .......... 23% 24.6M 4s 2024-04-21T08:45:39.164 INFO:tasks.workunit.client.0.smithi007.stderr: 32450K .......... .......... .......... .......... .......... 23% 24.5M 4s 2024-04-21T08:45:39.165 INFO:tasks.workunit.client.0.smithi007.stderr: 32500K .......... .......... .......... .......... .......... 23% 35.1M 4s 2024-04-21T08:45:39.167 INFO:tasks.workunit.client.0.smithi007.stderr: 32550K .......... .......... .......... .......... .......... 24% 24.9M 4s 2024-04-21T08:45:39.169 INFO:tasks.workunit.client.0.smithi007.stderr: 32600K .......... .......... .......... .......... .......... 24% 44.1M 4s 2024-04-21T08:45:39.170 INFO:tasks.workunit.client.0.smithi007.stderr: 32650K .......... .......... .......... .......... .......... 24% 31.2M 4s 2024-04-21T08:45:39.172 INFO:tasks.workunit.client.0.smithi007.stderr: 32700K .......... .......... .......... .......... .......... 24% 29.2M 4s 2024-04-21T08:45:39.175 INFO:tasks.workunit.client.0.smithi007.stderr: 32750K .......... .......... .......... .......... .......... 24% 23.0M 4s 2024-04-21T08:45:39.177 INFO:tasks.workunit.client.0.smithi007.stderr: 32800K .......... .......... .......... .......... .......... 24% 24.6M 4s 2024-04-21T08:45:39.177 INFO:tasks.workunit.client.0.smithi007.stderr: 32850K .......... .......... .......... .......... .......... 24% 48.3M 4s 2024-04-21T08:45:39.179 INFO:tasks.workunit.client.0.smithi007.stderr: 32900K .......... .......... .......... .......... .......... 24% 24.9M 4s 2024-04-21T08:45:39.181 INFO:tasks.workunit.client.0.smithi007.stderr: 32950K .......... .......... .......... .......... .......... 24% 37.4M 4s 2024-04-21T08:45:39.187 INFO:tasks.workunit.client.0.smithi007.stderr: 33000K .......... .......... .......... .......... .......... 24% 12.0M 4s 2024-04-21T08:45:39.187 INFO:tasks.workunit.client.0.smithi007.stderr: 33050K .......... .......... .......... .......... .......... 24% 55.2M 4s 2024-04-21T08:45:39.187 INFO:tasks.workunit.client.0.smithi007.stderr: 33100K .......... .......... .......... .......... .......... 24% 50.5M 4s 2024-04-21T08:45:39.189 INFO:tasks.workunit.client.0.smithi007.stderr: 33150K .......... .......... .......... .......... .......... 24% 23.0M 4s 2024-04-21T08:45:39.191 INFO:tasks.workunit.client.0.smithi007.stderr: 33200K .......... .......... .......... .......... .......... 24% 29.2M 4s 2024-04-21T08:45:39.191 INFO:tasks.workunit.client.0.smithi007.stderr: 33250K .......... .......... .......... .......... .......... 24% 42.3M 4s 2024-04-21T08:45:39.194 INFO:tasks.workunit.client.0.smithi007.stderr: 33300K .......... .......... .......... .......... .......... 24% 20.2M 4s 2024-04-21T08:45:39.196 INFO:tasks.workunit.client.0.smithi007.stderr: 33350K .......... .......... .......... .......... .......... 24% 34.4M 4s 2024-04-21T08:45:39.197 INFO:tasks.workunit.client.0.smithi007.stderr: 33400K .......... .......... .......... .......... .......... 24% 25.7M 4s 2024-04-21T08:45:39.199 INFO:tasks.workunit.client.0.smithi007.stderr: 33450K .......... .......... .......... .......... .......... 24% 23.9M 4s 2024-04-21T08:45:39.200 INFO:tasks.workunit.client.0.smithi007.stderr: 33500K .......... .......... .......... .......... .......... 24% 34.4M 4s 2024-04-21T08:45:39.202 INFO:tasks.workunit.client.0.smithi007.stderr: 33550K .......... .......... .......... .......... .......... 24% 23.4M 4s 2024-04-21T08:45:39.203 INFO:tasks.workunit.client.0.smithi007.stderr: 33600K .......... .......... .......... .......... .......... 24% 45.8M 4s 2024-04-21T08:45:39.205 INFO:tasks.workunit.client.0.smithi007.stderr: 33650K .......... .......... .......... .......... .......... 24% 23.7M 4s 2024-04-21T08:45:39.206 INFO:tasks.workunit.client.0.smithi007.stderr: 33700K .......... .......... .......... .......... .......... 24% 38.9M 4s 2024-04-21T08:45:39.209 INFO:tasks.workunit.client.0.smithi007.stderr: 33750K .......... .......... .......... .......... .......... 24% 24.0M 4s 2024-04-21T08:45:39.210 INFO:tasks.workunit.client.0.smithi007.stderr: 33800K .......... .......... .......... .......... .......... 24% 24.1M 4s 2024-04-21T08:45:39.211 INFO:tasks.workunit.client.0.smithi007.stderr: 33850K .......... .......... .......... .......... .......... 24% 52.9M 4s 2024-04-21T08:45:39.213 INFO:tasks.workunit.client.0.smithi007.stderr: 33900K .......... .......... .......... .......... .......... 25% 24.8M 4s 2024-04-21T08:45:39.216 INFO:tasks.workunit.client.0.smithi007.stderr: 33950K .......... .......... .......... .......... .......... 25% 22.4M 4s 2024-04-21T08:45:39.217 INFO:tasks.workunit.client.0.smithi007.stderr: 34000K .......... .......... .......... .......... .......... 25% 37.7M 4s 2024-04-21T08:45:39.219 INFO:tasks.workunit.client.0.smithi007.stderr: 34050K .......... .......... .......... .......... .......... 25% 24.1M 4s 2024-04-21T08:45:39.220 INFO:tasks.workunit.client.0.smithi007.stderr: 34100K .......... .......... .......... .......... .......... 25% 49.3M 4s 2024-04-21T08:45:39.222 INFO:tasks.workunit.client.0.smithi007.stderr: 34150K .......... .......... .......... .......... .......... 25% 21.7M 4s 2024-04-21T08:45:39.223 INFO:tasks.workunit.client.0.smithi007.stderr: 34200K .......... .......... .......... .......... .......... 25% 41.9M 4s 2024-04-21T08:45:39.226 INFO:tasks.workunit.client.0.smithi007.stderr: 34250K .......... .......... .......... .......... .......... 25% 21.4M 4s 2024-04-21T08:45:39.228 INFO:tasks.workunit.client.0.smithi007.stderr: 34300K .......... .......... .......... .......... .......... 25% 23.8M 4s 2024-04-21T08:45:39.229 INFO:tasks.workunit.client.0.smithi007.stderr: 34350K .......... .......... .......... .......... .......... 25% 48.6M 4s 2024-04-21T08:45:39.231 INFO:tasks.workunit.client.0.smithi007.stderr: 34400K .......... .......... .......... .......... .......... 25% 25.0M 4s 2024-04-21T08:45:39.233 INFO:tasks.workunit.client.0.smithi007.stderr: 34450K .......... .......... .......... .......... .......... 25% 42.4M 4s 2024-04-21T08:45:39.234 INFO:tasks.workunit.client.0.smithi007.stderr: 34500K .......... .......... .......... .......... .......... 25% 24.1M 4s 2024-04-21T08:45:39.236 INFO:tasks.workunit.client.0.smithi007.stderr: 34550K .......... .......... .......... .......... .......... 25% 22.4M 4s 2024-04-21T08:45:39.237 INFO:tasks.workunit.client.0.smithi007.stderr: 34600K .......... .......... .......... .......... .......... 25% 47.4M 4s 2024-04-21T08:45:39.239 INFO:tasks.workunit.client.0.smithi007.stderr: 34650K .......... .......... .......... .......... .......... 25% 24.3M 4s 2024-04-21T08:45:39.241 INFO:tasks.workunit.client.0.smithi007.stderr: 34700K .......... .......... .......... .......... .......... 25% 24.2M 4s 2024-04-21T08:45:39.242 INFO:tasks.workunit.client.0.smithi007.stderr: 34750K .......... .......... .......... .......... .......... 25% 34.1M 4s 2024-04-21T08:45:39.245 INFO:tasks.workunit.client.0.smithi007.stderr: 34800K .......... .......... .......... .......... .......... 25% 24.6M 4s 2024-04-21T08:45:39.245 INFO:tasks.workunit.client.0.smithi007.stderr: 34850K .......... .......... .......... .......... .......... 25% 50.0M 4s 2024-04-21T08:45:39.247 INFO:tasks.workunit.client.0.smithi007.stderr: 34900K .......... .......... .......... .......... .......... 25% 26.0M 4s 2024-04-21T08:45:39.250 INFO:tasks.workunit.client.0.smithi007.stderr: 34950K .......... .......... .......... .......... .......... 25% 22.6M 4s 2024-04-21T08:45:39.251 INFO:tasks.workunit.client.0.smithi007.stderr: 35000K .......... .......... .......... .......... .......... 25% 42.4M 4s 2024-04-21T08:45:39.253 INFO:tasks.workunit.client.0.smithi007.stderr: 35050K .......... .......... .......... .......... .......... 25% 21.9M 4s 2024-04-21T08:45:39.254 INFO:tasks.workunit.client.0.smithi007.stderr: 35100K .......... .......... .......... .......... .......... 25% 49.4M 4s 2024-04-21T08:45:39.256 INFO:tasks.workunit.client.0.smithi007.stderr: 35150K .......... .......... .......... .......... .......... 25% 24.1M 4s 2024-04-21T08:45:39.258 INFO:tasks.workunit.client.0.smithi007.stderr: 35200K .......... .......... .......... .......... .......... 25% 24.9M 4s 2024-04-21T08:45:39.259 INFO:tasks.workunit.client.0.smithi007.stderr: 35250K .......... .......... .......... .......... .......... 26% 51.6M 4s 2024-04-21T08:45:39.261 INFO:tasks.workunit.client.0.smithi007.stderr: 35300K .......... .......... .......... .......... .......... 26% 21.8M 4s 2024-04-21T08:45:39.262 INFO:tasks.workunit.client.0.smithi007.stderr: 35350K .......... .......... .......... .......... .......... 26% 48.7M 4s 2024-04-21T08:45:39.264 INFO:tasks.workunit.client.0.smithi007.stderr: 35400K .......... .......... .......... .......... .......... 26% 22.8M 4s 2024-04-21T08:45:39.266 INFO:tasks.workunit.client.0.smithi007.stderr: 35450K .......... .......... .......... .......... .......... 26% 24.4M 4s 2024-04-21T08:45:39.267 INFO:tasks.workunit.client.0.smithi007.stderr: 35500K .......... .......... .......... .......... .......... 26% 38.4M 4s 2024-04-21T08:45:39.270 INFO:tasks.workunit.client.0.smithi007.stderr: 35550K .......... .......... .......... .......... .......... 26% 23.0M 4s 2024-04-21T08:45:39.270 INFO:tasks.workunit.client.0.smithi007.stderr: 35600K .......... .......... .......... .......... .......... 26% 58.0M 4s 2024-04-21T08:45:39.273 INFO:tasks.workunit.client.0.smithi007.stderr: 35650K .......... .......... .......... .......... .......... 26% 22.8M 4s 2024-04-21T08:45:39.275 INFO:tasks.workunit.client.0.smithi007.stderr: 35700K .......... .......... .......... .......... .......... 26% 24.8M 4s 2024-04-21T08:45:39.276 INFO:tasks.workunit.client.0.smithi007.stderr: 35750K .......... .......... .......... .......... .......... 26% 41.6M 4s 2024-04-21T08:45:39.278 INFO:tasks.workunit.client.0.smithi007.stderr: 35800K .......... .......... .......... .......... .......... 26% 23.1M 4s 2024-04-21T08:45:39.280 INFO:tasks.workunit.client.0.smithi007.stderr: 35850K .......... .......... .......... .......... .......... 26% 44.8M 4s 2024-04-21T08:45:39.281 INFO:tasks.workunit.client.0.smithi007.stderr: 35900K .......... .......... .......... .......... .......... 26% 23.5M 4s 2024-04-21T08:45:39.283 INFO:tasks.workunit.client.0.smithi007.stderr: 35950K .......... .......... .......... .......... .......... 26% 25.2M 4s 2024-04-21T08:45:39.284 INFO:tasks.workunit.client.0.smithi007.stderr: 36000K .......... .......... .......... .......... .......... 26% 41.3M 4s 2024-04-21T08:45:39.287 INFO:tasks.workunit.client.0.smithi007.stderr: 36050K .......... .......... .......... .......... .......... 26% 22.9M 4s 2024-04-21T08:45:39.290 INFO:tasks.workunit.client.0.smithi007.stderr: 36100K .......... .......... .......... .......... .......... 26% 52.9M 4s 2024-04-21T08:45:39.290 INFO:tasks.workunit.client.0.smithi007.stderr: 36150K .......... .......... .......... .......... .......... 26% 21.8M 4s 2024-04-21T08:45:39.292 INFO:tasks.workunit.client.0.smithi007.stderr: 36200K .......... .......... .......... .......... .......... 26% 23.8M 4s 2024-04-21T08:45:39.294 INFO:tasks.workunit.client.0.smithi007.stderr: 36250K .......... .......... .......... .......... .......... 26% 40.4M 4s 2024-04-21T08:45:39.295 INFO:tasks.workunit.client.0.smithi007.stderr: 36300K .......... .......... .......... .......... .......... 26% 22.9M 4s 2024-04-21T08:45:39.297 INFO:tasks.workunit.client.0.smithi007.stderr: 36350K .......... .......... .......... .......... .......... 26% 26.3M 4s 2024-04-21T08:45:39.298 INFO:tasks.workunit.client.0.smithi007.stderr: 36400K .......... .......... .......... .......... .......... 26% 40.5M 4s 2024-04-21T08:45:39.300 INFO:tasks.workunit.client.0.smithi007.stderr: 36450K .......... .......... .......... .......... .......... 26% 25.7M 4s 2024-04-21T08:45:39.301 INFO:tasks.workunit.client.0.smithi007.stderr: 36500K .......... .......... .......... .......... .......... 26% 35.9M 4s 2024-04-21T08:45:39.303 INFO:tasks.workunit.client.0.smithi007.stderr: 36550K .......... .......... .......... .......... .......... 26% 24.8M 4s 2024-04-21T08:45:39.305 INFO:tasks.workunit.client.0.smithi007.stderr: 36600K .......... .......... .......... .......... .......... 26% 23.7M 4s 2024-04-21T08:45:39.306 INFO:tasks.workunit.client.0.smithi007.stderr: 36650K .......... .......... .......... .......... .......... 27% 46.6M 4s 2024-04-21T08:45:39.308 INFO:tasks.workunit.client.0.smithi007.stderr: 36700K .......... .......... .......... .......... .......... 27% 25.1M 4s 2024-04-21T08:45:39.309 INFO:tasks.workunit.client.0.smithi007.stderr: 36750K .......... .......... .......... .......... .......... 27% 37.6M 4s 2024-04-21T08:45:39.312 INFO:tasks.workunit.client.0.smithi007.stderr: 36800K .......... .......... .......... .......... .......... 27% 21.6M 4s 2024-04-21T08:45:39.314 INFO:tasks.workunit.client.0.smithi007.stderr: 36850K .......... .......... .......... .......... .......... 27% 24.5M 4s 2024-04-21T08:45:39.315 INFO:tasks.workunit.client.0.smithi007.stderr: 36900K .......... .......... .......... .......... .......... 27% 46.7M 4s 2024-04-21T08:45:39.317 INFO:tasks.workunit.client.0.smithi007.stderr: 36950K .......... .......... .......... .......... .......... 27% 21.6M 4s 2024-04-21T08:45:39.318 INFO:tasks.workunit.client.0.smithi007.stderr: 37000K .......... .......... .......... .......... .......... 27% 38.8M 4s 2024-04-21T08:45:39.321 INFO:tasks.workunit.client.0.smithi007.stderr: 37050K .......... .......... .......... .......... .......... 27% 19.1M 4s 2024-04-21T08:45:39.323 INFO:tasks.workunit.client.0.smithi007.stderr: 37100K .......... .......... .......... .......... .......... 27% 21.6M 4s 2024-04-21T08:45:39.324 INFO:tasks.workunit.client.0.smithi007.stderr: 37150K .......... .......... .......... .......... .......... 27% 44.1M 4s 2024-04-21T08:45:39.326 INFO:tasks.workunit.client.0.smithi007.stderr: 37200K .......... .......... .......... .......... .......... 27% 22.5M 4s 2024-04-21T08:45:39.328 INFO:tasks.workunit.client.0.smithi007.stderr: 37250K .......... .......... .......... .......... .......... 27% 38.7M 4s 2024-04-21T08:45:39.330 INFO:tasks.workunit.client.0.smithi007.stderr: 37300K .......... .......... .......... .......... .......... 27% 23.6M 4s 2024-04-21T08:45:39.332 INFO:tasks.workunit.client.0.smithi007.stderr: 37350K .......... .......... .......... .......... .......... 27% 23.7M 4s 2024-04-21T08:45:39.332 INFO:tasks.workunit.client.0.smithi007.stderr: 37400K .......... .......... .......... .......... .......... 27% 57.8M 4s 2024-04-21T08:45:39.335 INFO:tasks.workunit.client.0.smithi007.stderr: 37450K .......... .......... .......... .......... .......... 27% 23.1M 4s 2024-04-21T08:45:39.336 INFO:tasks.workunit.client.0.smithi007.stderr: 37500K .......... .......... .......... .......... .......... 27% 39.6M 4s 2024-04-21T08:45:39.338 INFO:tasks.workunit.client.0.smithi007.stderr: 37550K .......... .......... .......... .......... .......... 27% 23.1M 4s 2024-04-21T08:45:39.340 INFO:tasks.workunit.client.0.smithi007.stderr: 37600K .......... .......... .......... .......... .......... 27% 25.0M 4s 2024-04-21T08:45:39.341 INFO:tasks.workunit.client.0.smithi007.stderr: 37650K .......... .......... .......... .......... .......... 27% 57.1M 4s 2024-04-21T08:45:39.343 INFO:tasks.workunit.client.0.smithi007.stderr: 37700K .......... .......... .......... .......... .......... 27% 28.6M 4s 2024-04-21T08:45:39.345 INFO:tasks.workunit.client.0.smithi007.stderr: 37750K .......... .......... .......... .......... .......... 27% 24.8M 4s 2024-04-21T08:45:39.347 INFO:tasks.workunit.client.0.smithi007.stderr: 37800K .......... .......... .......... .......... .......... 27% 25.6M 4s 2024-04-21T08:45:39.349 INFO:tasks.workunit.client.0.smithi007.stderr: 37850K .......... .......... .......... .......... .......... 27% 24.6M 4s 2024-04-21T08:45:39.349 INFO:tasks.workunit.client.0.smithi007.stderr: 37900K .......... .......... .......... .......... .......... 27% 52.3M 4s 2024-04-21T08:45:39.351 INFO:tasks.workunit.client.0.smithi007.stderr: 37950K .......... .......... .......... .......... .......... 27% 23.4M 4s 2024-04-21T08:45:39.353 INFO:tasks.workunit.client.0.smithi007.stderr: 38000K .......... .......... .......... .......... .......... 28% 36.4M 4s 2024-04-21T08:45:39.356 INFO:tasks.workunit.client.0.smithi007.stderr: 38050K .......... .......... .......... .......... .......... 28% 22.8M 4s 2024-04-21T08:45:39.357 INFO:tasks.workunit.client.0.smithi007.stderr: 38100K .......... .......... .......... .......... .......... 28% 25.1M 4s 2024-04-21T08:45:39.358 INFO:tasks.workunit.client.0.smithi007.stderr: 38150K .......... .......... .......... .......... .......... 28% 56.0M 4s 2024-04-21T08:45:39.359 INFO:tasks.workunit.client.0.smithi007.stderr: 38200K .......... .......... .......... .......... .......... 28% 31.2M 4s 2024-04-21T08:45:39.361 INFO:tasks.workunit.client.0.smithi007.stderr: 38250K .......... .......... .......... .......... .......... 28% 28.2M 4s 2024-04-21T08:45:39.364 INFO:tasks.workunit.client.0.smithi007.stderr: 38300K .......... .......... .......... .......... .......... 28% 23.4M 4s 2024-04-21T08:45:39.365 INFO:tasks.workunit.client.0.smithi007.stderr: 38350K .......... .......... .......... .......... .......... 28% 23.3M 4s 2024-04-21T08:45:39.366 INFO:tasks.workunit.client.0.smithi007.stderr: 38400K .......... .......... .......... .......... .......... 28% 55.7M 4s 2024-04-21T08:45:39.368 INFO:tasks.workunit.client.0.smithi007.stderr: 38450K .......... .......... .......... .......... .......... 28% 23.2M 4s 2024-04-21T08:45:39.370 INFO:tasks.workunit.client.0.smithi007.stderr: 38500K .......... .......... .......... .......... .......... 28% 36.6M 4s 2024-04-21T08:45:39.372 INFO:tasks.workunit.client.0.smithi007.stderr: 38550K .......... .......... .......... .......... .......... 28% 23.5M 4s 2024-04-21T08:45:39.374 INFO:tasks.workunit.client.0.smithi007.stderr: 38600K .......... .......... .......... .......... .......... 28% 25.6M 4s 2024-04-21T08:45:39.375 INFO:tasks.workunit.client.0.smithi007.stderr: 38650K .......... .......... .......... .......... .......... 28% 44.7M 4s 2024-04-21T08:45:39.377 INFO:tasks.workunit.client.0.smithi007.stderr: 38700K .......... .......... .......... .......... .......... 28% 17.9M 4s 2024-04-21T08:45:39.378 INFO:tasks.workunit.client.0.smithi007.stderr: 38750K .......... .......... .......... .......... .......... 28% 64.9M 4s 2024-04-21T08:45:39.381 INFO:tasks.workunit.client.0.smithi007.stderr: 38800K .......... .......... .......... .......... .......... 28% 20.4M 4s 2024-04-21T08:45:39.383 INFO:tasks.workunit.client.0.smithi007.stderr: 38850K .......... .......... .......... .......... .......... 28% 24.2M 4s 2024-04-21T08:45:39.384 INFO:tasks.workunit.client.0.smithi007.stderr: 38900K .......... .......... .......... .......... .......... 28% 35.3M 4s 2024-04-21T08:45:39.386 INFO:tasks.workunit.client.0.smithi007.stderr: 38950K .......... .......... .......... .......... .......... 28% 22.2M 4s 2024-04-21T08:45:39.387 INFO:tasks.workunit.client.0.smithi007.stderr: 39000K .......... .......... .......... .......... .......... 28% 37.1M 4s 2024-04-21T08:45:39.390 INFO:tasks.workunit.client.0.smithi007.stderr: 39050K .......... .......... .......... .......... .......... 28% 23.7M 4s 2024-04-21T08:45:39.391 INFO:tasks.workunit.client.0.smithi007.stderr: 39100K .......... .......... .......... .......... .......... 28% 25.8M 4s 2024-04-21T08:45:39.392 INFO:tasks.workunit.client.0.smithi007.stderr: 39150K .......... .......... .......... .......... .......... 28% 50.3M 4s 2024-04-21T08:45:39.395 INFO:tasks.workunit.client.0.smithi007.stderr: 39200K .......... .......... .......... .......... .......... 28% 21.1M 4s 2024-04-21T08:45:39.397 INFO:tasks.workunit.client.0.smithi007.stderr: 39250K .......... .......... .......... .......... .......... 28% 34.6M 4s 2024-04-21T08:45:39.398 INFO:tasks.workunit.client.0.smithi007.stderr: 39300K .......... .......... .......... .......... .......... 28% 24.3M 4s 2024-04-21T08:45:39.400 INFO:tasks.workunit.client.0.smithi007.stderr: 39350K .......... .......... .......... .......... .......... 29% 25.5M 4s 2024-04-21T08:45:39.401 INFO:tasks.workunit.client.0.smithi007.stderr: 39400K .......... .......... .......... .......... .......... 29% 45.4M 4s 2024-04-21T08:45:39.403 INFO:tasks.workunit.client.0.smithi007.stderr: 39450K .......... .......... .......... .......... .......... 29% 24.5M 4s 2024-04-21T08:45:39.405 INFO:tasks.workunit.client.0.smithi007.stderr: 39500K .......... .......... .......... .......... .......... 29% 43.4M 4s 2024-04-21T08:45:39.406 INFO:tasks.workunit.client.0.smithi007.stderr: 39550K .......... .......... .......... .......... .......... 29% 21.6M 4s 2024-04-21T08:45:39.408 INFO:tasks.workunit.client.0.smithi007.stderr: 39600K .......... .......... .......... .......... .......... 29% 24.8M 4s 2024-04-21T08:45:39.409 INFO:tasks.workunit.client.0.smithi007.stderr: 39650K .......... .......... .......... .......... .......... 29% 55.2M 4s 2024-04-21T08:45:39.412 INFO:tasks.workunit.client.0.smithi007.stderr: 39700K .......... .......... .......... .......... .......... 29% 20.1M 4s 2024-04-21T08:45:39.414 INFO:tasks.workunit.client.0.smithi007.stderr: 39750K .......... .......... .......... .......... .......... 29% 30.3M 4s 2024-04-21T08:45:39.416 INFO:tasks.workunit.client.0.smithi007.stderr: 39800K .......... .......... .......... .......... .......... 29% 22.6M 4s 2024-04-21T08:45:39.418 INFO:tasks.workunit.client.0.smithi007.stderr: 39850K .......... .......... .......... .......... .......... 29% 23.6M 4s 2024-04-21T08:45:39.419 INFO:tasks.workunit.client.0.smithi007.stderr: 39900K .......... .......... .......... .......... .......... 29% 45.1M 3s 2024-04-21T08:45:39.421 INFO:tasks.workunit.client.0.smithi007.stderr: 39950K .......... .......... .......... .......... .......... 29% 24.8M 3s 2024-04-21T08:45:39.427 INFO:tasks.workunit.client.0.smithi007.stderr: 40000K .......... .......... .......... .......... .......... 29% 23.8M 3s 2024-04-21T08:45:39.427 INFO:tasks.workunit.client.0.smithi007.stderr: 40050K .......... .......... .......... .......... .......... 29% 43.8M 3s 2024-04-21T08:45:39.427 INFO:tasks.workunit.client.0.smithi007.stderr: 40100K .......... .......... .......... .......... .......... 29% 22.0M 3s 2024-04-21T08:45:39.428 INFO:tasks.workunit.client.0.smithi007.stderr: 40150K .......... .......... .......... .......... .......... 29% 57.6M 3s 2024-04-21T08:45:39.429 INFO:tasks.workunit.client.0.smithi007.stderr: 40200K .......... .......... .......... .......... .......... 29% 23.6M 3s 2024-04-21T08:45:39.431 INFO:tasks.workunit.client.0.smithi007.stderr: 40250K .......... .......... .......... .......... .......... 29% 23.9M 3s 2024-04-21T08:45:39.432 INFO:tasks.workunit.client.0.smithi007.stderr: 40300K .......... .......... .......... .......... .......... 29% 38.1M 3s 2024-04-21T08:45:39.435 INFO:tasks.workunit.client.0.smithi007.stderr: 40350K .......... .......... .......... .......... .......... 29% 21.0M 3s 2024-04-21T08:45:39.435 INFO:tasks.workunit.client.0.smithi007.stderr: 40400K .......... .......... .......... .......... .......... 29% 66.5M 3s 2024-04-21T08:45:39.438 INFO:tasks.workunit.client.0.smithi007.stderr: 40450K .......... .......... .......... .......... .......... 29% 18.0M 3s 2024-04-21T08:45:39.440 INFO:tasks.workunit.client.0.smithi007.stderr: 40500K .......... .......... .......... .......... .......... 29% 21.2M 3s 2024-04-21T08:45:39.442 INFO:tasks.workunit.client.0.smithi007.stderr: 40550K .......... .......... .......... .......... .......... 29% 40.2M 3s 2024-04-21T08:45:39.444 INFO:tasks.workunit.client.0.smithi007.stderr: 40600K .......... .......... .......... .......... .......... 29% 23.2M 3s 2024-04-21T08:45:39.444 INFO:tasks.workunit.client.0.smithi007.stderr: 40650K .......... .......... .......... .......... .......... 29% 53.4M 3s 2024-04-21T08:45:39.447 INFO:tasks.workunit.client.0.smithi007.stderr: 40700K .......... .......... .......... .......... .......... 30% 23.4M 3s 2024-04-21T08:45:39.449 INFO:tasks.workunit.client.0.smithi007.stderr: 40750K .......... .......... .......... .......... .......... 30% 24.3M 3s 2024-04-21T08:45:39.450 INFO:tasks.workunit.client.0.smithi007.stderr: 40800K .......... .......... .......... .......... .......... 30% 36.7M 3s 2024-04-21T08:45:39.452 INFO:tasks.workunit.client.0.smithi007.stderr: 40850K .......... .......... .......... .......... .......... 30% 25.1M 3s 2024-04-21T08:45:39.453 INFO:tasks.workunit.client.0.smithi007.stderr: 40900K .......... .......... .......... .......... .......... 30% 48.9M 3s 2024-04-21T08:45:39.455 INFO:tasks.workunit.client.0.smithi007.stderr: 40950K .......... .......... .......... .......... .......... 30% 24.4M 3s 2024-04-21T08:45:39.457 INFO:tasks.workunit.client.0.smithi007.stderr: 41000K .......... .......... .......... .......... .......... 30% 23.7M 3s 2024-04-21T08:45:39.458 INFO:tasks.workunit.client.0.smithi007.stderr: 41050K .......... .......... .......... .......... .......... 30% 36.2M 3s 2024-04-21T08:45:39.460 INFO:tasks.workunit.client.0.smithi007.stderr: 41100K .......... .......... .......... .......... .......... 30% 23.1M 3s 2024-04-21T08:45:39.463 INFO:tasks.workunit.client.0.smithi007.stderr: 41150K .......... .......... .......... .......... .......... 30% 49.9M 3s 2024-04-21T08:45:39.464 INFO:tasks.workunit.client.0.smithi007.stderr: 41200K .......... .......... .......... .......... .......... 30% 23.0M 3s 2024-04-21T08:45:39.466 INFO:tasks.workunit.client.0.smithi007.stderr: 41250K .......... .......... .......... .......... .......... 30% 22.6M 3s 2024-04-21T08:45:39.467 INFO:tasks.workunit.client.0.smithi007.stderr: 41300K .......... .......... .......... .......... .......... 30% 38.3M 3s 2024-04-21T08:45:39.469 INFO:tasks.workunit.client.0.smithi007.stderr: 41350K .......... .......... .......... .......... .......... 30% 23.4M 3s 2024-04-21T08:45:39.471 INFO:tasks.workunit.client.0.smithi007.stderr: 41400K .......... .......... .......... .......... .......... 30% 23.5M 3s 2024-04-21T08:45:39.472 INFO:tasks.workunit.client.0.smithi007.stderr: 41450K .......... .......... .......... .......... .......... 30% 45.3M 3s 2024-04-21T08:45:39.474 INFO:tasks.workunit.client.0.smithi007.stderr: 41500K .......... .......... .......... .......... .......... 30% 23.3M 3s 2024-04-21T08:45:39.475 INFO:tasks.workunit.client.0.smithi007.stderr: 41550K .......... .......... .......... .......... .......... 30% 40.1M 3s 2024-04-21T08:45:39.478 INFO:tasks.workunit.client.0.smithi007.stderr: 41600K .......... .......... .......... .......... .......... 30% 23.8M 3s 2024-04-21T08:45:39.480 INFO:tasks.workunit.client.0.smithi007.stderr: 41650K .......... .......... .......... .......... .......... 30% 25.6M 3s 2024-04-21T08:45:39.481 INFO:tasks.workunit.client.0.smithi007.stderr: 41700K .......... .......... .......... .......... .......... 30% 32.8M 3s 2024-04-21T08:45:39.484 INFO:tasks.workunit.client.0.smithi007.stderr: 41750K .......... .......... .......... .......... .......... 30% 21.2M 3s 2024-04-21T08:45:39.484 INFO:tasks.workunit.client.0.smithi007.stderr: 41800K .......... .......... .......... .......... .......... 30% 47.2M 3s 2024-04-21T08:45:39.487 INFO:tasks.workunit.client.0.smithi007.stderr: 41850K .......... .......... .......... .......... .......... 30% 22.2M 3s 2024-04-21T08:45:39.488 INFO:tasks.workunit.client.0.smithi007.stderr: 41900K .......... .......... .......... .......... .......... 30% 24.3M 3s 2024-04-21T08:45:39.490 INFO:tasks.workunit.client.0.smithi007.stderr: 41950K .......... .......... .......... .......... .......... 30% 47.5M 3s 2024-04-21T08:45:39.492 INFO:tasks.workunit.client.0.smithi007.stderr: 42000K .......... .......... .......... .......... .......... 30% 23.6M 3s 2024-04-21T08:45:39.493 INFO:tasks.workunit.client.0.smithi007.stderr: 42050K .......... .......... .......... .......... .......... 31% 35.2M 3s 2024-04-21T08:45:39.495 INFO:tasks.workunit.client.0.smithi007.stderr: 42100K .......... .......... .......... .......... .......... 31% 23.1M 3s 2024-04-21T08:45:39.497 INFO:tasks.workunit.client.0.smithi007.stderr: 42150K .......... .......... .......... .......... .......... 31% 26.9M 3s 2024-04-21T08:45:39.498 INFO:tasks.workunit.client.0.smithi007.stderr: 42200K .......... .......... .......... .......... .......... 31% 45.1M 3s 2024-04-21T08:45:39.499 INFO:tasks.workunit.client.0.smithi007.stderr: 42250K .......... .......... .......... .......... .......... 31% 36.6M 3s 2024-04-21T08:45:39.501 INFO:tasks.workunit.client.0.smithi007.stderr: 42300K .......... .......... .......... .......... .......... 31% 28.3M 3s 2024-04-21T08:45:39.503 INFO:tasks.workunit.client.0.smithi007.stderr: 42350K .......... .......... .......... .......... .......... 31% 21.4M 3s 2024-04-21T08:45:39.506 INFO:tasks.workunit.client.0.smithi007.stderr: 42400K .......... .......... .......... .......... .......... 31% 23.3M 3s 2024-04-21T08:45:39.506 INFO:tasks.workunit.client.0.smithi007.stderr: 42450K .......... .......... .......... .......... .......... 31% 53.8M 3s 2024-04-21T08:45:39.509 INFO:tasks.workunit.client.0.smithi007.stderr: 42500K .......... .......... .......... .......... .......... 31% 21.7M 3s 2024-04-21T08:45:39.511 INFO:tasks.workunit.client.0.smithi007.stderr: 42550K .......... .......... .......... .......... .......... 31% 24.4M 3s 2024-04-21T08:45:39.512 INFO:tasks.workunit.client.0.smithi007.stderr: 42600K .......... .......... .......... .......... .......... 31% 43.1M 3s 2024-04-21T08:45:39.514 INFO:tasks.workunit.client.0.smithi007.stderr: 42650K .......... .......... .......... .......... .......... 31% 23.0M 3s 2024-04-21T08:45:39.515 INFO:tasks.workunit.client.0.smithi007.stderr: 42700K .......... .......... .......... .......... .......... 31% 48.2M 3s 2024-04-21T08:45:39.517 INFO:tasks.workunit.client.0.smithi007.stderr: 42750K .......... .......... .......... .......... .......... 31% 23.4M 3s 2024-04-21T08:45:39.518 INFO:tasks.workunit.client.0.smithi007.stderr: 42800K .......... .......... .......... .......... .......... 31% 43.6M 3s 2024-04-21T08:45:39.520 INFO:tasks.workunit.client.0.smithi007.stderr: 42850K .......... .......... .......... .......... .......... 31% 22.8M 3s 2024-04-21T08:45:39.522 INFO:tasks.workunit.client.0.smithi007.stderr: 42900K .......... .......... .......... .......... .......... 31% 24.0M 3s 2024-04-21T08:45:39.523 INFO:tasks.workunit.client.0.smithi007.stderr: 42950K .......... .......... .......... .......... .......... 31% 44.2M 3s 2024-04-21T08:45:39.525 INFO:tasks.workunit.client.0.smithi007.stderr: 43000K .......... .......... .......... .......... .......... 31% 25.0M 3s 2024-04-21T08:45:39.526 INFO:tasks.workunit.client.0.smithi007.stderr: 43050K .......... .......... .......... .......... .......... 31% 40.9M 3s 2024-04-21T08:45:39.528 INFO:tasks.workunit.client.0.smithi007.stderr: 43100K .......... .......... .......... .......... .......... 31% 30.4M 3s 2024-04-21T08:45:39.531 INFO:tasks.workunit.client.0.smithi007.stderr: 43150K .......... .......... .......... .......... .......... 31% 19.0M 3s 2024-04-21T08:45:39.532 INFO:tasks.workunit.client.0.smithi007.stderr: 43200K .......... .......... .......... .......... .......... 31% 53.3M 3s 2024-04-21T08:45:39.534 INFO:tasks.workunit.client.0.smithi007.stderr: 43250K .......... .......... .......... .......... .......... 31% 23.6M 3s 2024-04-21T08:45:39.536 INFO:tasks.workunit.client.0.smithi007.stderr: 43300K .......... .......... .......... .......... .......... 31% 26.4M 3s 2024-04-21T08:45:39.537 INFO:tasks.workunit.client.0.smithi007.stderr: 43350K .......... .......... .......... .......... .......... 31% 34.9M 3s 2024-04-21T08:45:39.539 INFO:tasks.workunit.client.0.smithi007.stderr: 43400K .......... .......... .......... .......... .......... 32% 23.9M 3s 2024-04-21T08:45:39.540 INFO:tasks.workunit.client.0.smithi007.stderr: 43450K .......... .......... .......... .......... .......... 32% 44.5M 3s 2024-04-21T08:45:39.542 INFO:tasks.workunit.client.0.smithi007.stderr: 43500K .......... .......... .......... .......... .......... 32% 24.1M 3s 2024-04-21T08:45:39.543 INFO:tasks.workunit.client.0.smithi007.stderr: 43550K .......... .......... .......... .......... .......... 32% 42.5M 3s 2024-04-21T08:45:39.545 INFO:tasks.workunit.client.0.smithi007.stderr: 43600K .......... .......... .......... .......... .......... 32% 23.6M 3s 2024-04-21T08:45:39.548 INFO:tasks.workunit.client.0.smithi007.stderr: 43650K .......... .......... .......... .......... .......... 32% 21.4M 3s 2024-04-21T08:45:39.548 INFO:tasks.workunit.client.0.smithi007.stderr: 43700K .......... .......... .......... .......... .......... 32% 69.3M 3s 2024-04-21T08:45:39.551 INFO:tasks.workunit.client.0.smithi007.stderr: 43750K .......... .......... .......... .......... .......... 32% 22.0M 3s 2024-04-21T08:45:39.552 INFO:tasks.workunit.client.0.smithi007.stderr: 43800K .......... .......... .......... .......... .......... 32% 42.5M 3s 2024-04-21T08:45:39.554 INFO:tasks.workunit.client.0.smithi007.stderr: 43850K .......... .......... .......... .......... .......... 32% 23.1M 3s 2024-04-21T08:45:39.556 INFO:tasks.workunit.client.0.smithi007.stderr: 43900K .......... .......... .......... .......... .......... 32% 24.5M 3s 2024-04-21T08:45:39.557 INFO:tasks.workunit.client.0.smithi007.stderr: 43950K .......... .......... .......... .......... .......... 32% 46.4M 3s 2024-04-21T08:45:39.559 INFO:tasks.workunit.client.0.smithi007.stderr: 44000K .......... .......... .......... .......... .......... 32% 23.6M 3s 2024-04-21T08:45:39.560 INFO:tasks.workunit.client.0.smithi007.stderr: 44050K .......... .......... .......... .......... .......... 32% 42.2M 3s 2024-04-21T08:45:39.562 INFO:tasks.workunit.client.0.smithi007.stderr: 44100K .......... .......... .......... .......... .......... 32% 23.5M 3s 2024-04-21T08:45:39.564 INFO:tasks.workunit.client.0.smithi007.stderr: 44150K .......... .......... .......... .......... .......... 32% 25.0M 3s 2024-04-21T08:45:39.565 INFO:tasks.workunit.client.0.smithi007.stderr: 44200K .......... .......... .......... .......... .......... 32% 47.0M 3s 2024-04-21T08:45:39.567 INFO:tasks.workunit.client.0.smithi007.stderr: 44250K .......... .......... .......... .......... .......... 32% 21.9M 3s 2024-04-21T08:45:39.569 INFO:tasks.workunit.client.0.smithi007.stderr: 44300K .......... .......... .......... .......... .......... 32% 37.9M 3s 2024-04-21T08:45:39.571 INFO:tasks.workunit.client.0.smithi007.stderr: 44350K .......... .......... .......... .......... .......... 32% 26.1M 3s 2024-04-21T08:45:39.573 INFO:tasks.workunit.client.0.smithi007.stderr: 44400K .......... .......... .......... .......... .......... 32% 23.1M 3s 2024-04-21T08:45:39.574 INFO:tasks.workunit.client.0.smithi007.stderr: 44450K .......... .......... .......... .......... .......... 32% 46.6M 3s 2024-04-21T08:45:39.576 INFO:tasks.workunit.client.0.smithi007.stderr: 44500K .......... .......... .......... .......... .......... 32% 24.9M 3s 2024-04-21T08:45:39.577 INFO:tasks.workunit.client.0.smithi007.stderr: 44550K .......... .......... .......... .......... .......... 32% 33.3M 3s 2024-04-21T08:45:39.579 INFO:tasks.workunit.client.0.smithi007.stderr: 44600K .......... .......... .......... .......... .......... 32% 25.2M 3s 2024-04-21T08:45:39.580 INFO:tasks.workunit.client.0.smithi007.stderr: 44650K .......... .......... .......... .......... .......... 32% 32.9M 3s 2024-04-21T08:45:39.582 INFO:tasks.workunit.client.0.smithi007.stderr: 44700K .......... .......... .......... .......... .......... 32% 33.8M 3s 2024-04-21T08:45:39.584 INFO:tasks.workunit.client.0.smithi007.stderr: 44750K .......... .......... .......... .......... .......... 33% 22.4M 3s 2024-04-21T08:45:39.586 INFO:tasks.workunit.client.0.smithi007.stderr: 44800K .......... .......... .......... .......... .......... 33% 24.4M 3s 2024-04-21T08:45:39.587 INFO:tasks.workunit.client.0.smithi007.stderr: 44850K .......... .......... .......... .......... .......... 33% 42.7M 3s 2024-04-21T08:45:39.590 INFO:tasks.workunit.client.0.smithi007.stderr: 44900K .......... .......... .......... .......... .......... 33% 22.3M 3s 2024-04-21T08:45:39.590 INFO:tasks.workunit.client.0.smithi007.stderr: 44950K .......... .......... .......... .......... .......... 33% 49.9M 3s 2024-04-21T08:45:39.593 INFO:tasks.workunit.client.0.smithi007.stderr: 45000K .......... .......... .......... .......... .......... 33% 23.1M 3s 2024-04-21T08:45:39.595 INFO:tasks.workunit.client.0.smithi007.stderr: 45050K .......... .......... .......... .......... .......... 33% 42.4M 3s 2024-04-21T08:45:39.596 INFO:tasks.workunit.client.0.smithi007.stderr: 45100K .......... .......... .......... .......... .......... 33% 21.7M 3s 2024-04-21T08:45:39.598 INFO:tasks.workunit.client.0.smithi007.stderr: 45150K .......... .......... .......... .......... .......... 33% 24.6M 3s 2024-04-21T08:45:39.599 INFO:tasks.workunit.client.0.smithi007.stderr: 45200K .......... .......... .......... .......... .......... 33% 44.2M 3s 2024-04-21T08:45:39.602 INFO:tasks.workunit.client.0.smithi007.stderr: 45250K .......... .......... .......... .......... .......... 33% 20.0M 3s 2024-04-21T08:45:39.603 INFO:tasks.workunit.client.0.smithi007.stderr: 45300K .......... .......... .......... .......... .......... 33% 26.8M 3s 2024-04-21T08:45:39.605 INFO:tasks.workunit.client.0.smithi007.stderr: 45350K .......... .......... .......... .......... .......... 33% 37.4M 3s 2024-04-21T08:45:39.607 INFO:tasks.workunit.client.0.smithi007.stderr: 45400K .......... .......... .......... .......... .......... 33% 20.8M 3s 2024-04-21T08:45:39.608 INFO:tasks.workunit.client.0.smithi007.stderr: 45450K .......... .......... .......... .......... .......... 33% 39.3M 3s 2024-04-21T08:45:39.611 INFO:tasks.workunit.client.0.smithi007.stderr: 45500K .......... .......... .......... .......... .......... 33% 18.2M 3s 2024-04-21T08:45:39.613 INFO:tasks.workunit.client.0.smithi007.stderr: 45550K .......... .......... .......... .......... .......... 33% 25.7M 3s 2024-04-21T08:45:39.614 INFO:tasks.workunit.client.0.smithi007.stderr: 45600K .......... .......... .......... .......... .......... 33% 41.5M 3s 2024-04-21T08:45:39.616 INFO:tasks.workunit.client.0.smithi007.stderr: 45650K .......... .......... .......... .......... .......... 33% 23.2M 3s 2024-04-21T08:45:39.617 INFO:tasks.workunit.client.0.smithi007.stderr: 45700K .......... .......... .......... .......... .......... 33% 44.4M 3s 2024-04-21T08:45:39.619 INFO:tasks.workunit.client.0.smithi007.stderr: 45750K .......... .......... .......... .......... .......... 33% 24.7M 3s 2024-04-21T08:45:39.621 INFO:tasks.workunit.client.0.smithi007.stderr: 45800K .......... .......... .......... .......... .......... 33% 24.6M 3s 2024-04-21T08:45:39.623 INFO:tasks.workunit.client.0.smithi007.stderr: 45850K .......... .......... .......... .......... .......... 33% 34.5M 3s 2024-04-21T08:45:39.625 INFO:tasks.workunit.client.0.smithi007.stderr: 45900K .......... .......... .......... .......... .......... 33% 23.8M 3s 2024-04-21T08:45:39.625 INFO:tasks.workunit.client.0.smithi007.stderr: 45950K .......... .......... .......... .......... .......... 33% 59.6M 3s 2024-04-21T08:45:39.627 INFO:tasks.workunit.client.0.smithi007.stderr: 46000K .......... .......... .......... .......... .......... 33% 24.0M 3s 2024-04-21T08:45:39.630 INFO:tasks.workunit.client.0.smithi007.stderr: 46050K .......... .......... .......... .......... .......... 33% 24.1M 3s 2024-04-21T08:45:39.631 INFO:tasks.workunit.client.0.smithi007.stderr: 46100K .......... .......... .......... .......... .......... 33% 31.4M 3s 2024-04-21T08:45:39.633 INFO:tasks.workunit.client.0.smithi007.stderr: 46150K .......... .......... .......... .......... .......... 34% 29.8M 3s 2024-04-21T08:45:39.634 INFO:tasks.workunit.client.0.smithi007.stderr: 46200K .......... .......... .......... .......... .......... 34% 43.4M 3s 2024-04-21T08:45:39.636 INFO:tasks.workunit.client.0.smithi007.stderr: 46250K .......... .......... .......... .......... .......... 34% 22.5M 3s 2024-04-21T08:45:39.638 INFO:tasks.workunit.client.0.smithi007.stderr: 46300K .......... .......... .......... .......... .......... 34% 25.4M 3s 2024-04-21T08:45:39.639 INFO:tasks.workunit.client.0.smithi007.stderr: 46350K .......... .......... .......... .......... .......... 34% 38.0M 3s 2024-04-21T08:45:39.641 INFO:tasks.workunit.client.0.smithi007.stderr: 46400K .......... .......... .......... .......... .......... 34% 22.0M 3s 2024-04-21T08:45:39.643 INFO:tasks.workunit.client.0.smithi007.stderr: 46450K .......... .......... .......... .......... .......... 34% 58.9M 3s 2024-04-21T08:45:39.644 INFO:tasks.workunit.client.0.smithi007.stderr: 46500K .......... .......... .......... .......... .......... 34% 24.2M 3s 2024-04-21T08:45:39.646 INFO:tasks.workunit.client.0.smithi007.stderr: 46550K .......... .......... .......... .......... .......... 34% 23.5M 3s 2024-04-21T08:45:39.648 INFO:tasks.workunit.client.0.smithi007.stderr: 46600K .......... .......... .......... .......... .......... 34% 39.2M 3s 2024-04-21T08:45:39.650 INFO:tasks.workunit.client.0.smithi007.stderr: 46650K .......... .......... .......... .......... .......... 34% 23.5M 3s 2024-04-21T08:45:39.652 INFO:tasks.workunit.client.0.smithi007.stderr: 46700K .......... .......... .......... .......... .......... 34% 22.9M 3s 2024-04-21T08:45:39.653 INFO:tasks.workunit.client.0.smithi007.stderr: 46750K .......... .......... .......... .......... .......... 34% 46.8M 3s 2024-04-21T08:45:39.655 INFO:tasks.workunit.client.0.smithi007.stderr: 46800K .......... .......... .......... .......... .......... 34% 25.4M 3s 2024-04-21T08:45:39.656 INFO:tasks.workunit.client.0.smithi007.stderr: 46850K .......... .......... .......... .......... .......... 34% 39.0M 3s 2024-04-21T08:45:39.658 INFO:tasks.workunit.client.0.smithi007.stderr: 46900K .......... .......... .......... .......... .......... 34% 23.8M 3s 2024-04-21T08:45:39.660 INFO:tasks.workunit.client.0.smithi007.stderr: 46950K .......... .......... .......... .......... .......... 34% 23.4M 3s 2024-04-21T08:45:39.661 INFO:tasks.workunit.client.0.smithi007.stderr: 47000K .......... .......... .......... .......... .......... 34% 48.8M 3s 2024-04-21T08:45:39.663 INFO:tasks.workunit.client.0.smithi007.stderr: 47050K .......... .......... .......... .......... .......... 34% 20.3M 3s 2024-04-21T08:45:39.665 INFO:tasks.workunit.client.0.smithi007.stderr: 47100K .......... .......... .......... .......... .......... 34% 40.1M 3s 2024-04-21T08:45:39.667 INFO:tasks.workunit.client.0.smithi007.stderr: 47150K .......... .......... .......... .......... .......... 34% 21.1M 3s 2024-04-21T08:45:39.669 INFO:tasks.workunit.client.0.smithi007.stderr: 47200K .......... .......... .......... .......... .......... 34% 22.0M 3s 2024-04-21T08:45:39.670 INFO:tasks.workunit.client.0.smithi007.stderr: 47250K .......... .......... .......... .......... .......... 34% 46.6M 3s 2024-04-21T08:45:39.673 INFO:tasks.workunit.client.0.smithi007.stderr: 47300K .......... .......... .......... .......... .......... 34% 20.0M 3s 2024-04-21T08:45:39.674 INFO:tasks.workunit.client.0.smithi007.stderr: 47350K .......... .......... .......... .......... .......... 34% 33.9M 3s 2024-04-21T08:45:39.676 INFO:tasks.workunit.client.0.smithi007.stderr: 47400K .......... .......... .......... .......... .......... 34% 25.3M 3s 2024-04-21T08:45:39.678 INFO:tasks.workunit.client.0.smithi007.stderr: 47450K .......... .......... .......... .......... .......... 34% 23.9M 3s 2024-04-21T08:45:39.679 INFO:tasks.workunit.client.0.smithi007.stderr: 47500K .......... .......... .......... .......... .......... 35% 49.4M 3s 2024-04-21T08:45:39.681 INFO:tasks.workunit.client.0.smithi007.stderr: 47550K .......... .......... .......... .......... .......... 35% 23.1M 3s 2024-04-21T08:45:39.683 INFO:tasks.workunit.client.0.smithi007.stderr: 47600K .......... .......... .......... .......... .......... 35% 38.5M 3s 2024-04-21T08:45:39.685 INFO:tasks.workunit.client.0.smithi007.stderr: 47650K .......... .......... .......... .......... .......... 35% 24.1M 3s 2024-04-21T08:45:39.687 INFO:tasks.workunit.client.0.smithi007.stderr: 47700K .......... .......... .......... .......... .......... 35% 24.4M 3s 2024-04-21T08:45:39.688 INFO:tasks.workunit.client.0.smithi007.stderr: 47750K .......... .......... .......... .......... .......... 35% 54.2M 3s 2024-04-21T08:45:39.690 INFO:tasks.workunit.client.0.smithi007.stderr: 47800K .......... .......... .......... .......... .......... 35% 23.1M 3s 2024-04-21T08:45:39.691 INFO:tasks.workunit.client.0.smithi007.stderr: 47850K .......... .......... .......... .......... .......... 35% 37.0M 3s 2024-04-21T08:45:39.693 INFO:tasks.workunit.client.0.smithi007.stderr: 47900K .......... .......... .......... .......... .......... 35% 24.5M 3s 2024-04-21T08:45:39.695 INFO:tasks.workunit.client.0.smithi007.stderr: 47950K .......... .......... .......... .......... .......... 35% 24.0M 3s 2024-04-21T08:45:39.696 INFO:tasks.workunit.client.0.smithi007.stderr: 48000K .......... .......... .......... .......... .......... 35% 45.6M 3s 2024-04-21T08:45:39.697 INFO:tasks.workunit.client.0.smithi007.stderr: 48050K .......... .......... .......... .......... .......... 35% 35.9M 3s 2024-04-21T08:45:39.700 INFO:tasks.workunit.client.0.smithi007.stderr: 48100K .......... .......... .......... .......... .......... 35% 19.1M 3s 2024-04-21T08:45:39.702 INFO:tasks.workunit.client.0.smithi007.stderr: 48150K .......... .......... .......... .......... .......... 35% 22.2M 3s 2024-04-21T08:45:39.704 INFO:tasks.workunit.client.0.smithi007.stderr: 48200K .......... .......... .......... .......... .......... 35% 23.2M 3s 2024-04-21T08:45:39.705 INFO:tasks.workunit.client.0.smithi007.stderr: 48250K .......... .......... .......... .......... .......... 35% 52.9M 3s 2024-04-21T08:45:39.711 INFO:tasks.workunit.client.0.smithi007.stderr: 48300K .......... .......... .......... .......... .......... 35% 23.2M 3s 2024-04-21T08:45:39.711 INFO:tasks.workunit.client.0.smithi007.stderr: 48350K .......... .......... .......... .......... .......... 35% 37.3M 3s 2024-04-21T08:45:39.711 INFO:tasks.workunit.client.0.smithi007.stderr: 48400K .......... .......... .......... .......... .......... 35% 24.2M 3s 2024-04-21T08:45:39.713 INFO:tasks.workunit.client.0.smithi007.stderr: 48450K .......... .......... .......... .......... .......... 35% 25.2M 3s 2024-04-21T08:45:39.714 INFO:tasks.workunit.client.0.smithi007.stderr: 48500K .......... .......... .......... .......... .......... 35% 45.5M 3s 2024-04-21T08:45:39.716 INFO:tasks.workunit.client.0.smithi007.stderr: 48550K .......... .......... .......... .......... .......... 35% 25.1M 3s 2024-04-21T08:45:39.717 INFO:tasks.workunit.client.0.smithi007.stderr: 48600K .......... .......... .......... .......... .......... 35% 36.6M 3s 2024-04-21T08:45:39.719 INFO:tasks.workunit.client.0.smithi007.stderr: 48650K .......... .......... .......... .......... .......... 35% 21.6M 3s 2024-04-21T08:45:39.721 INFO:tasks.workunit.client.0.smithi007.stderr: 48700K .......... .......... .......... .......... .......... 35% 23.8M 3s 2024-04-21T08:45:39.722 INFO:tasks.workunit.client.0.smithi007.stderr: 48750K .......... .......... .......... .......... .......... 35% 51.6M 3s 2024-04-21T08:45:39.724 INFO:tasks.workunit.client.0.smithi007.stderr: 48800K .......... .......... .......... .......... .......... 35% 21.3M 3s 2024-04-21T08:45:39.726 INFO:tasks.workunit.client.0.smithi007.stderr: 48850K .......... .......... .......... .......... .......... 36% 39.9M 3s 2024-04-21T08:45:39.728 INFO:tasks.workunit.client.0.smithi007.stderr: 48900K .......... .......... .......... .......... .......... 36% 17.8M 3s 2024-04-21T08:45:39.730 INFO:tasks.workunit.client.0.smithi007.stderr: 48950K .......... .......... .......... .......... .......... 36% 25.8M 3s 2024-04-21T08:45:39.731 INFO:tasks.workunit.client.0.smithi007.stderr: 49000K .......... .......... .......... .......... .......... 36% 50.0M 3s 2024-04-21T08:45:39.734 INFO:tasks.workunit.client.0.smithi007.stderr: 49050K .......... .......... .......... .......... .......... 36% 22.3M 3s 2024-04-21T08:45:39.735 INFO:tasks.workunit.client.0.smithi007.stderr: 49100K .......... .......... .......... .......... .......... 36% 41.8M 3s 2024-04-21T08:45:39.737 INFO:tasks.workunit.client.0.smithi007.stderr: 49150K .......... .......... .......... .......... .......... 36% 23.2M 3s 2024-04-21T08:45:39.739 INFO:tasks.workunit.client.0.smithi007.stderr: 49200K .......... .......... .......... .......... .......... 36% 24.7M 3s 2024-04-21T08:45:39.739 INFO:tasks.workunit.client.0.smithi007.stderr: 49250K .......... .......... .......... .......... .......... 36% 59.5M 3s 2024-04-21T08:45:39.742 INFO:tasks.workunit.client.0.smithi007.stderr: 49300K .......... .......... .......... .......... .......... 36% 22.0M 3s 2024-04-21T08:45:39.743 INFO:tasks.workunit.client.0.smithi007.stderr: 49350K .......... .......... .......... .......... .......... 36% 38.0M 3s 2024-04-21T08:45:39.745 INFO:tasks.workunit.client.0.smithi007.stderr: 49400K .......... .......... .......... .......... .......... 36% 23.5M 3s 2024-04-21T08:45:39.747 INFO:tasks.workunit.client.0.smithi007.stderr: 49450K .......... .......... .......... .......... .......... 36% 22.8M 3s 2024-04-21T08:45:39.748 INFO:tasks.workunit.client.0.smithi007.stderr: 49500K .......... .......... .......... .......... .......... 36% 45.5M 3s 2024-04-21T08:45:39.750 INFO:tasks.workunit.client.0.smithi007.stderr: 49550K .......... .......... .......... .......... .......... 36% 24.7M 3s 2024-04-21T08:45:39.752 INFO:tasks.workunit.client.0.smithi007.stderr: 49600K .......... .......... .......... .......... .......... 36% 37.9M 3s 2024-04-21T08:45:39.754 INFO:tasks.workunit.client.0.smithi007.stderr: 49650K .......... .......... .......... .......... .......... 36% 24.9M 3s 2024-04-21T08:45:39.756 INFO:tasks.workunit.client.0.smithi007.stderr: 49700K .......... .......... .......... .......... .......... 36% 22.9M 3s 2024-04-21T08:45:39.757 INFO:tasks.workunit.client.0.smithi007.stderr: 49750K .......... .......... .......... .......... .......... 36% 42.7M 3s 2024-04-21T08:45:39.759 INFO:tasks.workunit.client.0.smithi007.stderr: 49800K .......... .......... .......... .......... .......... 36% 22.8M 3s 2024-04-21T08:45:39.760 INFO:tasks.workunit.client.0.smithi007.stderr: 49850K .......... .......... .......... .......... .......... 36% 37.7M 3s 2024-04-21T08:45:39.762 INFO:tasks.workunit.client.0.smithi007.stderr: 49900K .......... .......... .......... .......... .......... 36% 25.1M 3s 2024-04-21T08:45:39.765 INFO:tasks.workunit.client.0.smithi007.stderr: 49950K .......... .......... .......... .......... .......... 36% 22.5M 3s 2024-04-21T08:45:39.765 INFO:tasks.workunit.client.0.smithi007.stderr: 50000K .......... .......... .......... .......... .......... 36% 51.9M 3s 2024-04-21T08:45:39.768 INFO:tasks.workunit.client.0.smithi007.stderr: 50050K .......... .......... .......... .......... .......... 36% 20.9M 3s 2024-04-21T08:45:39.770 INFO:tasks.workunit.client.0.smithi007.stderr: 50100K .......... .......... .......... .......... .......... 36% 38.3M 3s 2024-04-21T08:45:39.771 INFO:tasks.workunit.client.0.smithi007.stderr: 50150K .......... .......... .......... .......... .......... 36% 22.8M 3s 2024-04-21T08:45:39.773 INFO:tasks.workunit.client.0.smithi007.stderr: 50200K .......... .......... .......... .......... .......... 37% 22.4M 3s 2024-04-21T08:45:39.774 INFO:tasks.workunit.client.0.smithi007.stderr: 50250K .......... .......... .......... .......... .......... 37% 54.5M 3s 2024-04-21T08:45:39.776 INFO:tasks.workunit.client.0.smithi007.stderr: 50300K .......... .......... .......... .......... .......... 37% 22.8M 3s 2024-04-21T08:45:39.779 INFO:tasks.workunit.client.0.smithi007.stderr: 50350K .......... .......... .......... .......... .......... 37% 40.6M 3s 2024-04-21T08:45:39.780 INFO:tasks.workunit.client.0.smithi007.stderr: 50400K .......... .......... .......... .......... .......... 37% 22.8M 3s 2024-04-21T08:45:39.782 INFO:tasks.workunit.client.0.smithi007.stderr: 50450K .......... .......... .......... .......... .......... 37% 24.7M 3s 2024-04-21T08:45:39.783 INFO:tasks.workunit.client.0.smithi007.stderr: 50500K .......... .......... .......... .......... .......... 37% 48.4M 3s 2024-04-21T08:45:39.785 INFO:tasks.workunit.client.0.smithi007.stderr: 50550K .......... .......... .......... .......... .......... 37% 23.0M 3s 2024-04-21T08:45:39.787 INFO:tasks.workunit.client.0.smithi007.stderr: 50600K .......... .......... .......... .......... .......... 37% 24.6M 3s 2024-04-21T08:45:39.789 INFO:tasks.workunit.client.0.smithi007.stderr: 50650K .......... .......... .......... .......... .......... 37% 41.0M 3s 2024-04-21T08:45:39.790 INFO:tasks.workunit.client.0.smithi007.stderr: 50700K .......... .......... .......... .......... .......... 37% 24.5M 3s 2024-04-21T08:45:39.791 INFO:tasks.workunit.client.0.smithi007.stderr: 50750K .......... .......... .......... .......... .......... 37% 50.3M 3s 2024-04-21T08:45:39.793 INFO:tasks.workunit.client.0.smithi007.stderr: 50800K .......... .......... .......... .......... .......... 37% 21.4M 3s 2024-04-21T08:45:39.795 INFO:tasks.workunit.client.0.smithi007.stderr: 50850K .......... .......... .......... .......... .......... 37% 22.5M 3s 2024-04-21T08:45:39.797 INFO:tasks.workunit.client.0.smithi007.stderr: 50900K .......... .......... .......... .......... .......... 37% 38.9M 3s 2024-04-21T08:45:39.799 INFO:tasks.workunit.client.0.smithi007.stderr: 50950K .......... .......... .......... .......... .......... 37% 24.3M 3s 2024-04-21T08:45:39.800 INFO:tasks.workunit.client.0.smithi007.stderr: 51000K .......... .......... .......... .......... .......... 37% 50.4M 3s 2024-04-21T08:45:39.802 INFO:tasks.workunit.client.0.smithi007.stderr: 51050K .......... .......... .......... .......... .......... 37% 23.8M 3s 2024-04-21T08:45:39.804 INFO:tasks.workunit.client.0.smithi007.stderr: 51100K .......... .......... .......... .......... .......... 37% 23.7M 3s 2024-04-21T08:45:39.805 INFO:tasks.workunit.client.0.smithi007.stderr: 51150K .......... .......... .......... .......... .......... 37% 39.6M 3s 2024-04-21T08:45:39.807 INFO:tasks.workunit.client.0.smithi007.stderr: 51200K .......... .......... .......... .......... .......... 37% 23.3M 3s 2024-04-21T08:45:39.808 INFO:tasks.workunit.client.0.smithi007.stderr: 51250K .......... .......... .......... .......... .......... 37% 47.9M 3s 2024-04-21T08:45:39.810 INFO:tasks.workunit.client.0.smithi007.stderr: 51300K .......... .......... .......... .......... .......... 37% 24.3M 3s 2024-04-21T08:45:39.812 INFO:tasks.workunit.client.0.smithi007.stderr: 51350K .......... .......... .......... .......... .......... 37% 23.6M 3s 2024-04-21T08:45:39.813 INFO:tasks.workunit.client.0.smithi007.stderr: 51400K .......... .......... .......... .......... .......... 37% 40.9M 3s 2024-04-21T08:45:39.816 INFO:tasks.workunit.client.0.smithi007.stderr: 51450K .......... .......... .......... .......... .......... 37% 23.0M 3s 2024-04-21T08:45:39.818 INFO:tasks.workunit.client.0.smithi007.stderr: 51500K .......... .......... .......... .......... .......... 37% 53.2M 3s 2024-04-21T08:45:39.819 INFO:tasks.workunit.client.0.smithi007.stderr: 51550K .......... .......... .......... .......... .......... 38% 23.2M 3s 2024-04-21T08:45:39.821 INFO:tasks.workunit.client.0.smithi007.stderr: 51600K .......... .......... .......... .......... .......... 38% 23.2M 3s 2024-04-21T08:45:39.822 INFO:tasks.workunit.client.0.smithi007.stderr: 51650K .......... .......... .......... .......... .......... 38% 42.7M 3s 2024-04-21T08:45:39.824 INFO:tasks.workunit.client.0.smithi007.stderr: 51700K .......... .......... .......... .......... .......... 38% 22.5M 3s 2024-04-21T08:45:39.826 INFO:tasks.workunit.client.0.smithi007.stderr: 51750K .......... .......... .......... .......... .......... 38% 54.6M 3s 2024-04-21T08:45:39.827 INFO:tasks.workunit.client.0.smithi007.stderr: 51800K .......... .......... .......... .......... .......... 38% 22.4M 3s 2024-04-21T08:45:39.829 INFO:tasks.workunit.client.0.smithi007.stderr: 51850K .......... .......... .......... .......... .......... 38% 23.0M 3s 2024-04-21T08:45:39.830 INFO:tasks.workunit.client.0.smithi007.stderr: 51900K .......... .......... .......... .......... .......... 38% 52.9M 3s 2024-04-21T08:45:39.833 INFO:tasks.workunit.client.0.smithi007.stderr: 51950K .......... .......... .......... .......... .......... 38% 19.6M 3s 2024-04-21T08:45:39.833 INFO:tasks.workunit.client.0.smithi007.stderr: 52000K .......... .......... .......... .......... .......... 38% 51.5M 3s 2024-04-21T08:45:39.835 INFO:tasks.workunit.client.0.smithi007.stderr: 52050K .......... .......... .......... .......... .......... 38% 24.9M 3s 2024-04-21T08:45:39.838 INFO:tasks.workunit.client.0.smithi007.stderr: 52100K .......... .......... .......... .......... .......... 38% 24.2M 3s 2024-04-21T08:45:39.839 INFO:tasks.workunit.client.0.smithi007.stderr: 52150K .......... .......... .......... .......... .......... 38% 47.7M 3s 2024-04-21T08:45:39.841 INFO:tasks.workunit.client.0.smithi007.stderr: 52200K .......... .......... .......... .......... .......... 38% 19.8M 3s 2024-04-21T08:45:39.843 INFO:tasks.workunit.client.0.smithi007.stderr: 52250K .......... .......... .......... .......... .......... 38% 25.6M 3s 2024-04-21T08:45:39.844 INFO:tasks.workunit.client.0.smithi007.stderr: 52300K .......... .......... .......... .......... .......... 38% 50.6M 3s 2024-04-21T08:45:39.846 INFO:tasks.workunit.client.0.smithi007.stderr: 52350K .......... .......... .......... .......... .......... 38% 23.7M 3s 2024-04-21T08:45:39.847 INFO:tasks.workunit.client.0.smithi007.stderr: 52400K .......... .......... .......... .......... .......... 38% 41.5M 3s 2024-04-21T08:45:39.849 INFO:tasks.workunit.client.0.smithi007.stderr: 52450K .......... .......... .......... .......... .......... 38% 20.9M 3s 2024-04-21T08:45:39.850 INFO:tasks.workunit.client.0.smithi007.stderr: 52500K .......... .......... .......... .......... .......... 38% 47.9M 3s 2024-04-21T08:45:39.852 INFO:tasks.workunit.client.0.smithi007.stderr: 52550K .......... .......... .......... .......... .......... 38% 27.6M 3s 2024-04-21T08:45:39.854 INFO:tasks.workunit.client.0.smithi007.stderr: 52600K .......... .......... .......... .......... .......... 38% 29.9M 3s 2024-04-21T08:45:39.855 INFO:tasks.workunit.client.0.smithi007.stderr: 52650K .......... .......... .......... .......... .......... 38% 34.9M 3s 2024-04-21T08:45:39.858 INFO:tasks.workunit.client.0.smithi007.stderr: 52700K .......... .......... .......... .......... .......... 38% 21.1M 3s 2024-04-21T08:45:39.860 INFO:tasks.workunit.client.0.smithi007.stderr: 52750K .......... .......... .......... .......... .......... 38% 24.3M 3s 2024-04-21T08:45:39.861 INFO:tasks.workunit.client.0.smithi007.stderr: 52800K .......... .......... .......... .......... .......... 38% 50.3M 3s 2024-04-21T08:45:39.863 INFO:tasks.workunit.client.0.smithi007.stderr: 52850K .......... .......... .......... .......... .......... 38% 21.8M 3s 2024-04-21T08:45:39.864 INFO:tasks.workunit.client.0.smithi007.stderr: 52900K .......... .......... .......... .......... .......... 39% 36.9M 3s 2024-04-21T08:45:39.866 INFO:tasks.workunit.client.0.smithi007.stderr: 52950K .......... .......... .......... .......... .......... 39% 27.9M 3s 2024-04-21T08:45:39.867 INFO:tasks.workunit.client.0.smithi007.stderr: 53000K .......... .......... .......... .......... .......... 39% 39.8M 3s 2024-04-21T08:45:39.869 INFO:tasks.workunit.client.0.smithi007.stderr: 53050K .......... .......... .......... .......... .......... 39% 25.3M 3s 2024-04-21T08:45:39.871 INFO:tasks.workunit.client.0.smithi007.stderr: 53100K .......... .......... .......... .......... .......... 39% 24.6M 3s 2024-04-21T08:45:39.872 INFO:tasks.workunit.client.0.smithi007.stderr: 53150K .......... .......... .......... .......... .......... 39% 37.5M 3s 2024-04-21T08:45:39.875 INFO:tasks.workunit.client.0.smithi007.stderr: 53200K .......... .......... .......... .......... .......... 39% 20.9M 3s 2024-04-21T08:45:39.876 INFO:tasks.workunit.client.0.smithi007.stderr: 53250K .......... .......... .......... .......... .......... 39% 28.4M 3s 2024-04-21T08:45:39.877 INFO:tasks.workunit.client.0.smithi007.stderr: 53300K .......... .......... .......... .......... .......... 39% 52.9M 3s 2024-04-21T08:45:39.880 INFO:tasks.workunit.client.0.smithi007.stderr: 53350K .......... .......... .......... .......... .......... 39% 21.0M 3s 2024-04-21T08:45:39.881 INFO:tasks.workunit.client.0.smithi007.stderr: 53400K .......... .......... .......... .......... .......... 39% 38.7M 3s 2024-04-21T08:45:39.883 INFO:tasks.workunit.client.0.smithi007.stderr: 53450K .......... .......... .......... .......... .......... 39% 22.6M 3s 2024-04-21T08:45:39.885 INFO:tasks.workunit.client.0.smithi007.stderr: 53500K .......... .......... .......... .......... .......... 39% 24.7M 3s 2024-04-21T08:45:39.886 INFO:tasks.workunit.client.0.smithi007.stderr: 53550K .......... .......... .......... .......... .......... 39% 48.4M 3s 2024-04-21T08:45:39.889 INFO:tasks.workunit.client.0.smithi007.stderr: 53600K .......... .......... .......... .......... .......... 39% 19.1M 3s 2024-04-21T08:45:39.890 INFO:tasks.workunit.client.0.smithi007.stderr: 53650K .......... .......... .......... .......... .......... 39% 42.0M 3s 2024-04-21T08:45:39.891 INFO:tasks.workunit.client.0.smithi007.stderr: 53700K .......... .......... .......... .......... .......... 39% 31.4M 3s 2024-04-21T08:45:39.894 INFO:tasks.workunit.client.0.smithi007.stderr: 53750K .......... .......... .......... .......... .......... 39% 16.9M 3s 2024-04-21T08:45:39.895 INFO:tasks.workunit.client.0.smithi007.stderr: 53800K .......... .......... .......... .......... .......... 39% 39.5M 3s 2024-04-21T08:45:39.898 INFO:tasks.workunit.client.0.smithi007.stderr: 53850K .......... .......... .......... .......... .......... 39% 23.4M 3s 2024-04-21T08:45:39.899 INFO:tasks.workunit.client.0.smithi007.stderr: 53900K .......... .......... .......... .......... .......... 39% 34.0M 3s 2024-04-21T08:45:39.901 INFO:tasks.workunit.client.0.smithi007.stderr: 53950K .......... .......... .......... .......... .......... 39% 23.0M 3s 2024-04-21T08:45:39.903 INFO:tasks.workunit.client.0.smithi007.stderr: 54000K .......... .......... .......... .......... .......... 39% 25.5M 3s 2024-04-21T08:45:39.904 INFO:tasks.workunit.client.0.smithi007.stderr: 54050K .......... .......... .......... .......... .......... 39% 49.0M 3s 2024-04-21T08:45:39.906 INFO:tasks.workunit.client.0.smithi007.stderr: 54100K .......... .......... .......... .......... .......... 39% 24.2M 3s 2024-04-21T08:45:39.907 INFO:tasks.workunit.client.0.smithi007.stderr: 54150K .......... .......... .......... .......... .......... 39% 39.3M 3s 2024-04-21T08:45:39.908 INFO:tasks.workunit.client.0.smithi007.stderr: 54200K .......... .......... .......... .......... .......... 39% 32.8M 3s 2024-04-21T08:45:39.911 INFO:tasks.workunit.client.0.smithi007.stderr: 54250K .......... .......... .......... .......... .......... 39% 21.2M 3s 2024-04-21T08:45:39.914 INFO:tasks.workunit.client.0.smithi007.stderr: 54300K .......... .......... .......... .......... .......... 40% 43.1M 3s 2024-04-21T08:45:39.914 INFO:tasks.workunit.client.0.smithi007.stderr: 54350K .......... .......... .......... .......... .......... 40% 21.6M 3s 2024-04-21T08:45:39.916 INFO:tasks.workunit.client.0.smithi007.stderr: 54400K .......... .......... .......... .......... .......... 40% 43.5M 3s 2024-04-21T08:45:39.918 INFO:tasks.workunit.client.0.smithi007.stderr: 54450K .......... .......... .......... .......... .......... 40% 23.3M 3s 2024-04-21T08:45:39.920 INFO:tasks.workunit.client.0.smithi007.stderr: 54500K .......... .......... .......... .......... .......... 40% 22.7M 3s 2024-04-21T08:45:39.921 INFO:tasks.workunit.client.0.smithi007.stderr: 54550K .......... .......... .......... .......... .......... 40% 60.9M 3s 2024-04-21T08:45:39.923 INFO:tasks.workunit.client.0.smithi007.stderr: 54600K .......... .......... .......... .......... .......... 40% 21.5M 3s 2024-04-21T08:45:39.924 INFO:tasks.workunit.client.0.smithi007.stderr: 54650K .......... .......... .......... .......... .......... 40% 40.5M 3s 2024-04-21T08:45:39.926 INFO:tasks.workunit.client.0.smithi007.stderr: 54700K .......... .......... .......... .......... .......... 40% 24.7M 3s 2024-04-21T08:45:39.928 INFO:tasks.workunit.client.0.smithi007.stderr: 54750K .......... .......... .......... .......... .......... 40% 25.7M 3s 2024-04-21T08:45:39.929 INFO:tasks.workunit.client.0.smithi007.stderr: 54800K .......... .......... .......... .......... .......... 40% 43.6M 3s 2024-04-21T08:45:39.931 INFO:tasks.workunit.client.0.smithi007.stderr: 54850K .......... .......... .......... .......... .......... 40% 23.8M 3s 2024-04-21T08:45:39.932 INFO:tasks.workunit.client.0.smithi007.stderr: 54900K .......... .......... .......... .......... .......... 40% 39.5M 3s 2024-04-21T08:45:39.934 INFO:tasks.workunit.client.0.smithi007.stderr: 54950K .......... .......... .......... .......... .......... 40% 27.4M 3s 2024-04-21T08:45:39.937 INFO:tasks.workunit.client.0.smithi007.stderr: 55000K .......... .......... .......... .......... .......... 40% 19.8M 3s 2024-04-21T08:45:39.937 INFO:tasks.workunit.client.0.smithi007.stderr: 55050K .......... .......... .......... .......... .......... 40% 52.6M 3s 2024-04-21T08:45:39.940 INFO:tasks.workunit.client.0.smithi007.stderr: 55100K .......... .......... .......... .......... .......... 40% 23.3M 3s 2024-04-21T08:45:39.941 INFO:tasks.workunit.client.0.smithi007.stderr: 55150K .......... .......... .......... .......... .......... 40% 35.5M 3s 2024-04-21T08:45:39.943 INFO:tasks.workunit.client.0.smithi007.stderr: 55200K .......... .......... .......... .......... .......... 40% 29.1M 3s 2024-04-21T08:45:39.945 INFO:tasks.workunit.client.0.smithi007.stderr: 55250K .......... .......... .......... .......... .......... 40% 20.2M 3s 2024-04-21T08:45:39.946 INFO:tasks.workunit.client.0.smithi007.stderr: 55300K .......... .......... .......... .......... .......... 40% 43.4M 3s 2024-04-21T08:45:39.948 INFO:tasks.workunit.client.0.smithi007.stderr: 55350K .......... .......... .......... .......... .......... 40% 24.8M 3s 2024-04-21T08:45:39.950 INFO:tasks.workunit.client.0.smithi007.stderr: 55400K .......... .......... .......... .......... .......... 40% 23.9M 3s 2024-04-21T08:45:39.952 INFO:tasks.workunit.client.0.smithi007.stderr: 55450K .......... .......... .......... .......... .......... 40% 38.4M 3s 2024-04-21T08:45:39.954 INFO:tasks.workunit.client.0.smithi007.stderr: 55500K .......... .......... .......... .......... .......... 40% 19.4M 3s 2024-04-21T08:45:39.955 INFO:tasks.workunit.client.0.smithi007.stderr: 55550K .......... .......... .......... .......... .......... 40% 47.8M 3s 2024-04-21T08:45:39.957 INFO:tasks.workunit.client.0.smithi007.stderr: 55600K .......... .......... .......... .......... .......... 40% 21.1M 3s 2024-04-21T08:45:39.960 INFO:tasks.workunit.client.0.smithi007.stderr: 55650K .......... .......... .......... .......... .......... 41% 39.4M 3s 2024-04-21T08:45:39.961 INFO:tasks.workunit.client.0.smithi007.stderr: 55700K .......... .......... .......... .......... .......... 41% 21.1M 3s 2024-04-21T08:45:39.963 INFO:tasks.workunit.client.0.smithi007.stderr: 55750K .......... .......... .......... .......... .......... 41% 21.6M 3s 2024-04-21T08:45:39.964 INFO:tasks.workunit.client.0.smithi007.stderr: 55800K .......... .......... .......... .......... .......... 41% 45.6M 3s 2024-04-21T08:45:39.966 INFO:tasks.workunit.client.0.smithi007.stderr: 55850K .......... .......... .......... .......... .......... 41% 24.8M 3s 2024-04-21T08:45:39.968 INFO:tasks.workunit.client.0.smithi007.stderr: 55900K .......... .......... .......... .......... .......... 41% 23.5M 3s 2024-04-21T08:45:39.970 INFO:tasks.workunit.client.0.smithi007.stderr: 55950K .......... .......... .......... .......... .......... 41% 36.7M 3s 2024-04-21T08:45:39.972 INFO:tasks.workunit.client.0.smithi007.stderr: 56000K .......... .......... .......... .......... .......... 41% 28.1M 3s 2024-04-21T08:45:39.973 INFO:tasks.workunit.client.0.smithi007.stderr: 56050K .......... .......... .......... .......... .......... 41% 38.2M 3s 2024-04-21T08:45:39.974 INFO:tasks.workunit.client.0.smithi007.stderr: 56100K .......... .......... .......... .......... .......... 41% 25.0M 3s 2024-04-21T08:45:39.976 INFO:tasks.workunit.client.0.smithi007.stderr: 56150K .......... .......... .......... .......... .......... 41% 26.0M 3s 2024-04-21T08:45:39.978 INFO:tasks.workunit.client.0.smithi007.stderr: 56200K .......... .......... .......... .......... .......... 41% 33.6M 3s 2024-04-21T08:45:39.980 INFO:tasks.workunit.client.0.smithi007.stderr: 56250K .......... .......... .......... .......... .......... 41% 22.9M 3s 2024-04-21T08:45:39.981 INFO:tasks.workunit.client.0.smithi007.stderr: 56300K .......... .......... .......... .......... .......... 41% 58.4M 3s 2024-04-21T08:45:39.983 INFO:tasks.workunit.client.0.smithi007.stderr: 56350K .......... .......... .......... .......... .......... 41% 23.6M 3s 2024-04-21T08:45:39.985 INFO:tasks.workunit.client.0.smithi007.stderr: 56400K .......... .......... .......... .......... .......... 41% 23.1M 3s 2024-04-21T08:45:39.986 INFO:tasks.workunit.client.0.smithi007.stderr: 56450K .......... .......... .......... .......... .......... 41% 41.7M 3s 2024-04-21T08:45:39.989 INFO:tasks.workunit.client.0.smithi007.stderr: 56500K .......... .......... .......... .......... .......... 41% 16.6M 3s 2024-04-21T08:45:39.990 INFO:tasks.workunit.client.0.smithi007.stderr: 56550K .......... .......... .......... .......... .......... 41% 54.6M 3s 2024-04-21T08:45:39.992 INFO:tasks.workunit.client.0.smithi007.stderr: 56600K .......... .......... .......... .......... .......... 41% 23.3M 3s 2024-04-21T08:45:39.994 INFO:tasks.workunit.client.0.smithi007.stderr: 56650K .......... .......... .......... .......... .......... 41% 23.0M 3s 2024-04-21T08:45:39.995 INFO:tasks.workunit.client.0.smithi007.stderr: 56700K .......... .......... .......... .......... .......... 41% 42.6M 3s 2024-04-21T08:45:39.998 INFO:tasks.workunit.client.0.smithi007.stderr: 56750K .......... .......... .......... .......... .......... 41% 22.3M 3s 2024-04-21T08:45:40.000 INFO:tasks.workunit.client.0.smithi007.stderr: 56800K .......... .......... .......... .......... .......... 41% 53.0M 3s 2024-04-21T08:45:40.000 INFO:tasks.workunit.client.0.smithi007.stderr: 56850K .......... .......... .......... .......... .......... 41% 24.7M 3s 2024-04-21T08:45:40.002 INFO:tasks.workunit.client.0.smithi007.stderr: 56900K .......... .......... .......... .......... .......... 41% 23.7M 3s 2024-04-21T08:45:40.004 INFO:tasks.workunit.client.0.smithi007.stderr: 56950K .......... .......... .......... .......... .......... 41% 39.1M 3s 2024-04-21T08:45:40.006 INFO:tasks.workunit.client.0.smithi007.stderr: 57000K .......... .......... .......... .......... .......... 42% 22.2M 3s 2024-04-21T08:45:40.008 INFO:tasks.workunit.client.0.smithi007.stderr: 57050K .......... .......... .......... .......... .......... 42% 47.4M 3s 2024-04-21T08:45:40.009 INFO:tasks.workunit.client.0.smithi007.stderr: 57100K .......... .......... .......... .......... .......... 42% 24.7M 3s 2024-04-21T08:45:40.012 INFO:tasks.workunit.client.0.smithi007.stderr: 57150K .......... .......... .......... .......... .......... 42% 23.6M 3s 2024-04-21T08:45:40.013 INFO:tasks.workunit.client.0.smithi007.stderr: 57200K .......... .......... .......... .......... .......... 42% 29.6M 3s 2024-04-21T08:45:40.015 INFO:tasks.workunit.client.0.smithi007.stderr: 57250K .......... .......... .......... .......... .......... 42% 24.8M 3s 2024-04-21T08:45:40.017 INFO:tasks.workunit.client.0.smithi007.stderr: 57300K .......... .......... .......... .......... .......... 42% 25.0M 3s 2024-04-21T08:45:40.018 INFO:tasks.workunit.client.0.smithi007.stderr: 57350K .......... .......... .......... .......... .......... 42% 30.9M 3s 2024-04-21T08:45:40.020 INFO:tasks.workunit.client.0.smithi007.stderr: 57400K .......... .......... .......... .......... .......... 42% 24.5M 3s 2024-04-21T08:45:40.022 INFO:tasks.workunit.client.0.smithi007.stderr: 57450K .......... .......... .......... .......... .......... 42% 36.1M 3s 2024-04-21T08:45:40.024 INFO:tasks.workunit.client.0.smithi007.stderr: 57500K .......... .......... .......... .......... .......... 42% 20.9M 3s 2024-04-21T08:45:40.026 INFO:tasks.workunit.client.0.smithi007.stderr: 57550K .......... .......... .......... .......... .......... 42% 29.9M 3s 2024-04-21T08:45:40.027 INFO:tasks.workunit.client.0.smithi007.stderr: 57600K .......... .......... .......... .......... .......... 42% 46.5M 3s 2024-04-21T08:45:40.029 INFO:tasks.workunit.client.0.smithi007.stderr: 57650K .......... .......... .......... .......... .......... 42% 24.1M 3s 2024-04-21T08:45:40.030 INFO:tasks.workunit.client.0.smithi007.stderr: 57700K .......... .......... .......... .......... .......... 42% 39.7M 3s 2024-04-21T08:45:40.032 INFO:tasks.workunit.client.0.smithi007.stderr: 57750K .......... .......... .......... .......... .......... 42% 22.2M 3s 2024-04-21T08:45:40.034 INFO:tasks.workunit.client.0.smithi007.stderr: 57800K .......... .......... .......... .......... .......... 42% 23.8M 3s 2024-04-21T08:45:40.035 INFO:tasks.workunit.client.0.smithi007.stderr: 57850K .......... .......... .......... .......... .......... 42% 52.0M 3s 2024-04-21T08:45:40.037 INFO:tasks.workunit.client.0.smithi007.stderr: 57900K .......... .......... .......... .......... .......... 42% 23.3M 3s 2024-04-21T08:45:40.038 INFO:tasks.workunit.client.0.smithi007.stderr: 57950K .......... .......... .......... .......... .......... 42% 40.0M 3s 2024-04-21T08:45:40.040 INFO:tasks.workunit.client.0.smithi007.stderr: 58000K .......... .......... .......... .......... .......... 42% 23.0M 3s 2024-04-21T08:45:40.043 INFO:tasks.workunit.client.0.smithi007.stderr: 58050K .......... .......... .......... .......... .......... 42% 23.5M 3s 2024-04-21T08:45:40.043 INFO:tasks.workunit.client.0.smithi007.stderr: 58100K .......... .......... .......... .......... .......... 42% 58.3M 3s 2024-04-21T08:45:40.045 INFO:tasks.workunit.client.0.smithi007.stderr: 58150K .......... .......... .......... .......... .......... 42% 23.5M 3s 2024-04-21T08:45:40.047 INFO:tasks.workunit.client.0.smithi007.stderr: 58200K .......... .......... .......... .......... .......... 42% 32.5M 3s 2024-04-21T08:45:40.049 INFO:tasks.workunit.client.0.smithi007.stderr: 58250K .......... .......... .......... .......... .......... 42% 22.8M 3s 2024-04-21T08:45:40.051 INFO:tasks.workunit.client.0.smithi007.stderr: 58300K .......... .......... .......... .......... .......... 42% 23.0M 3s 2024-04-21T08:45:40.052 INFO:tasks.workunit.client.0.smithi007.stderr: 58350K .......... .......... .......... .......... .......... 43% 46.0M 3s 2024-04-21T08:45:40.054 INFO:tasks.workunit.client.0.smithi007.stderr: 58400K .......... .......... .......... .......... .......... 43% 25.3M 3s 2024-04-21T08:45:40.056 INFO:tasks.workunit.client.0.smithi007.stderr: 58450K .......... .......... .......... .......... .......... 43% 37.3M 3s 2024-04-21T08:45:40.058 INFO:tasks.workunit.client.0.smithi007.stderr: 58500K .......... .......... .......... .......... .......... 43% 21.1M 3s 2024-04-21T08:45:40.060 INFO:tasks.workunit.client.0.smithi007.stderr: 58550K .......... .......... .......... .......... .......... 43% 22.4M 3s 2024-04-21T08:45:40.061 INFO:tasks.workunit.client.0.smithi007.stderr: 58600K .......... .......... .......... .......... .......... 43% 49.0M 3s 2024-04-21T08:45:40.063 INFO:tasks.workunit.client.0.smithi007.stderr: 58650K .......... .......... .......... .......... .......... 43% 24.0M 3s 2024-04-21T08:45:40.064 INFO:tasks.workunit.client.0.smithi007.stderr: 58700K .......... .......... .......... .......... .......... 43% 36.6M 3s 2024-04-21T08:45:40.067 INFO:tasks.workunit.client.0.smithi007.stderr: 58750K .......... .......... .......... .......... .......... 43% 23.5M 3s 2024-04-21T08:45:40.068 INFO:tasks.workunit.client.0.smithi007.stderr: 58800K .......... .......... .......... .......... .......... 43% 24.4M 3s 2024-04-21T08:45:40.069 INFO:tasks.workunit.client.0.smithi007.stderr: 58850K .......... .......... .......... .......... .......... 43% 53.6M 3s 2024-04-21T08:45:40.072 INFO:tasks.workunit.client.0.smithi007.stderr: 58900K .......... .......... .......... .......... .......... 43% 23.0M 3s 2024-04-21T08:45:40.073 INFO:tasks.workunit.client.0.smithi007.stderr: 58950K .......... .......... .......... .......... .......... 43% 41.9M 3s 2024-04-21T08:45:40.075 INFO:tasks.workunit.client.0.smithi007.stderr: 59000K .......... .......... .......... .......... .......... 43% 22.5M 3s 2024-04-21T08:45:40.077 INFO:tasks.workunit.client.0.smithi007.stderr: 59050K .......... .......... .......... .......... .......... 43% 24.8M 3s 2024-04-21T08:45:40.078 INFO:tasks.workunit.client.0.smithi007.stderr: 59100K .......... .......... .......... .......... .......... 43% 47.7M 3s 2024-04-21T08:45:40.080 INFO:tasks.workunit.client.0.smithi007.stderr: 59150K .......... .......... .......... .......... .......... 43% 24.6M 3s 2024-04-21T08:45:40.081 INFO:tasks.workunit.client.0.smithi007.stderr: 59200K .......... .......... .......... .......... .......... 43% 42.6M 3s 2024-04-21T08:45:40.083 INFO:tasks.workunit.client.0.smithi007.stderr: 59250K .......... .......... .......... .......... .......... 43% 22.8M 3s 2024-04-21T08:45:40.085 INFO:tasks.workunit.client.0.smithi007.stderr: 59300K .......... .......... .......... .......... .......... 43% 22.3M 3s 2024-04-21T08:45:40.086 INFO:tasks.workunit.client.0.smithi007.stderr: 59350K .......... .......... .......... .......... .......... 43% 47.2M 3s 2024-04-21T08:45:40.088 INFO:tasks.workunit.client.0.smithi007.stderr: 59400K .......... .......... .......... .......... .......... 43% 22.4M 3s 2024-04-21T08:45:40.090 INFO:tasks.workunit.client.0.smithi007.stderr: 59450K .......... .......... .......... .......... .......... 43% 45.1M 3s 2024-04-21T08:45:40.092 INFO:tasks.workunit.client.0.smithi007.stderr: 59500K .......... .......... .......... .......... .......... 43% 22.2M 3s 2024-04-21T08:45:40.094 INFO:tasks.workunit.client.0.smithi007.stderr: 59550K .......... .......... .......... .......... .......... 43% 26.3M 3s 2024-04-21T08:45:40.095 INFO:tasks.workunit.client.0.smithi007.stderr: 59600K .......... .......... .......... .......... .......... 43% 37.3M 3s 2024-04-21T08:45:40.097 INFO:tasks.workunit.client.0.smithi007.stderr: 59650K .......... .......... .......... .......... .......... 43% 30.8M 3s 2024-04-21T08:45:40.098 INFO:tasks.workunit.client.0.smithi007.stderr: 59700K .......... .......... .......... .......... .......... 44% 30.1M 3s 2024-04-21T08:45:40.100 INFO:tasks.workunit.client.0.smithi007.stderr: 59750K .......... .......... .......... .......... .......... 44% 26.2M 3s 2024-04-21T08:45:40.103 INFO:tasks.workunit.client.0.smithi007.stderr: 59800K .......... .......... .......... .......... .......... 44% 23.8M 3s 2024-04-21T08:45:40.103 INFO:tasks.workunit.client.0.smithi007.stderr: 59850K .......... .......... .......... .......... .......... 44% 54.2M 3s 2024-04-21T08:45:40.105 INFO:tasks.workunit.client.0.smithi007.stderr: 59900K .......... .......... .......... .......... .......... 44% 22.6M 3s 2024-04-21T08:45:40.106 INFO:tasks.workunit.client.0.smithi007.stderr: 59950K .......... .......... .......... .......... .......... 44% 39.0M 3s 2024-04-21T08:45:40.108 INFO:tasks.workunit.client.0.smithi007.stderr: 60000K .......... .......... .......... .......... .......... 44% 23.1M 3s 2024-04-21T08:45:40.111 INFO:tasks.workunit.client.0.smithi007.stderr: 60050K .......... .......... .......... .......... .......... 44% 23.0M 3s 2024-04-21T08:45:40.111 INFO:tasks.workunit.client.0.smithi007.stderr: 60100K .......... .......... .......... .......... .......... 44% 55.2M 3s 2024-04-21T08:45:40.114 INFO:tasks.workunit.client.0.smithi007.stderr: 60150K .......... .......... .......... .......... .......... 44% 23.7M 3s 2024-04-21T08:45:40.115 INFO:tasks.workunit.client.0.smithi007.stderr: 60200K .......... .......... .......... .......... .......... 44% 40.5M 3s 2024-04-21T08:45:40.117 INFO:tasks.workunit.client.0.smithi007.stderr: 60250K .......... .......... .......... .......... .......... 44% 22.3M 3s 2024-04-21T08:45:40.119 INFO:tasks.workunit.client.0.smithi007.stderr: 60300K .......... .......... .......... .......... .......... 44% 24.7M 3s 2024-04-21T08:45:40.120 INFO:tasks.workunit.client.0.smithi007.stderr: 60350K .......... .......... .......... .......... .......... 44% 51.4M 3s 2024-04-21T08:45:40.122 INFO:tasks.workunit.client.0.smithi007.stderr: 60400K .......... .......... .......... .......... .......... 44% 22.5M 3s 2024-04-21T08:45:40.123 INFO:tasks.workunit.client.0.smithi007.stderr: 60450K .......... .......... .......... .......... .......... 44% 41.5M 3s 2024-04-21T08:45:40.125 INFO:tasks.workunit.client.0.smithi007.stderr: 60500K .......... .......... .......... .......... .......... 44% 23.6M 3s 2024-04-21T08:45:40.127 INFO:tasks.workunit.client.0.smithi007.stderr: 60550K .......... .......... .......... .......... .......... 44% 24.5M 3s 2024-04-21T08:45:40.128 INFO:tasks.workunit.client.0.smithi007.stderr: 60600K .......... .......... .......... .......... .......... 44% 43.5M 3s 2024-04-21T08:45:40.130 INFO:tasks.workunit.client.0.smithi007.stderr: 60650K .......... .......... .......... .......... .......... 44% 25.9M 3s 2024-04-21T08:45:40.133 INFO:tasks.workunit.client.0.smithi007.stderr: 60700K .......... .......... .......... .......... .......... 44% 40.9M 3s 2024-04-21T08:45:40.134 INFO:tasks.workunit.client.0.smithi007.stderr: 60750K .......... .......... .......... .......... .......... 44% 23.6M 3s 2024-04-21T08:45:40.136 INFO:tasks.workunit.client.0.smithi007.stderr: 60800K .......... .......... .......... .......... .......... 44% 22.6M 3s 2024-04-21T08:45:40.137 INFO:tasks.workunit.client.0.smithi007.stderr: 60850K .......... .......... .......... .......... .......... 44% 57.4M 3s 2024-04-21T08:45:40.139 INFO:tasks.workunit.client.0.smithi007.stderr: 60900K .......... .......... .......... .......... .......... 44% 23.0M 3s 2024-04-21T08:45:40.141 INFO:tasks.workunit.client.0.smithi007.stderr: 60950K .......... .......... .......... .......... .......... 44% 24.2M 3s 2024-04-21T08:45:40.142 INFO:tasks.workunit.client.0.smithi007.stderr: 61000K .......... .......... .......... .......... .......... 44% 37.3M 3s 2024-04-21T08:45:40.144 INFO:tasks.workunit.client.0.smithi007.stderr: 61050K .......... .......... .......... .......... .......... 45% 24.0M 3s 2024-04-21T08:45:40.145 INFO:tasks.workunit.client.0.smithi007.stderr: 61100K .......... .......... .......... .......... .......... 45% 63.0M 3s 2024-04-21T08:45:40.147 INFO:tasks.workunit.client.0.smithi007.stderr: 61150K .......... .......... .......... .......... .......... 45% 23.9M 3s 2024-04-21T08:45:40.149 INFO:tasks.workunit.client.0.smithi007.stderr: 61200K .......... .......... .......... .......... .......... 45% 21.5M 3s 2024-04-21T08:45:40.150 INFO:tasks.workunit.client.0.smithi007.stderr: 61250K .......... .......... .......... .......... .......... 45% 38.8M 3s 2024-04-21T08:45:40.153 INFO:tasks.workunit.client.0.smithi007.stderr: 61300K .......... .......... .......... .......... .......... 45% 23.6M 3s 2024-04-21T08:45:40.153 INFO:tasks.workunit.client.0.smithi007.stderr: 61350K .......... .......... .......... .......... .......... 45% 48.5M 3s 2024-04-21T08:45:40.155 INFO:tasks.workunit.client.0.smithi007.stderr: 61400K .......... .......... .......... .......... .......... 45% 24.5M 3s 2024-04-21T08:45:40.158 INFO:tasks.workunit.client.0.smithi007.stderr: 61450K .......... .......... .......... .......... .......... 45% 22.9M 3s 2024-04-21T08:45:40.159 INFO:tasks.workunit.client.0.smithi007.stderr: 61500K .......... .......... .......... .......... .......... 45% 42.7M 3s 2024-04-21T08:45:40.161 INFO:tasks.workunit.client.0.smithi007.stderr: 61550K .......... .......... .......... .......... .......... 45% 22.2M 3s 2024-04-21T08:45:40.162 INFO:tasks.workunit.client.0.smithi007.stderr: 61600K .......... .......... .......... .......... .......... 45% 48.6M 3s 2024-04-21T08:45:40.164 INFO:tasks.workunit.client.0.smithi007.stderr: 61650K .......... .......... .......... .......... .......... 45% 24.1M 3s 2024-04-21T08:45:40.166 INFO:tasks.workunit.client.0.smithi007.stderr: 61700K .......... .......... .......... .......... .......... 45% 23.8M 3s 2024-04-21T08:45:40.167 INFO:tasks.workunit.client.0.smithi007.stderr: 61750K .......... .......... .......... .......... .......... 45% 40.7M 3s 2024-04-21T08:45:40.170 INFO:tasks.workunit.client.0.smithi007.stderr: 61800K .......... .......... .......... .......... .......... 45% 20.8M 3s 2024-04-21T08:45:40.170 INFO:tasks.workunit.client.0.smithi007.stderr: 61850K .......... .......... .......... .......... .......... 45% 56.4M 3s 2024-04-21T08:45:40.173 INFO:tasks.workunit.client.0.smithi007.stderr: 61900K .......... .......... .......... .......... .......... 45% 22.3M 3s 2024-04-21T08:45:40.175 INFO:tasks.workunit.client.0.smithi007.stderr: 61950K .......... .......... .......... .......... .......... 45% 23.9M 3s 2024-04-21T08:45:40.176 INFO:tasks.workunit.client.0.smithi007.stderr: 62000K .......... .......... .......... .......... .......... 45% 37.2M 3s 2024-04-21T08:45:40.178 INFO:tasks.workunit.client.0.smithi007.stderr: 62050K .......... .......... .......... .......... .......... 45% 20.6M 3s 2024-04-21T08:45:40.180 INFO:tasks.workunit.client.0.smithi007.stderr: 62100K .......... .......... .......... .......... .......... 45% 45.3M 3s 2024-04-21T08:45:40.185 INFO:tasks.workunit.client.0.smithi007.stderr: 62150K .......... .......... .......... .......... .......... 45% 11.7M 3s 2024-04-21T08:45:40.185 INFO:tasks.workunit.client.0.smithi007.stderr: 62200K .......... .......... .......... .......... .......... 45% 263M 3s 2024-04-21T08:45:40.186 INFO:tasks.workunit.client.0.smithi007.stderr: 62250K .......... .......... .......... .......... .......... 45% 31.8M 3s 2024-04-21T08:45:40.187 INFO:tasks.workunit.client.0.smithi007.stderr: 62300K .......... .......... .......... .......... .......... 45% 23.8M 3s 2024-04-21T08:45:40.190 INFO:tasks.workunit.client.0.smithi007.stderr: 62350K .......... .......... .......... .......... .......... 45% 41.0M 3s 2024-04-21T08:45:40.191 INFO:tasks.workunit.client.0.smithi007.stderr: 62400K .......... .......... .......... .......... .......... 46% 23.5M 3s 2024-04-21T08:45:40.193 INFO:tasks.workunit.client.0.smithi007.stderr: 62450K .......... .......... .......... .......... .......... 46% 23.1M 3s 2024-04-21T08:45:40.194 INFO:tasks.workunit.client.0.smithi007.stderr: 62500K .......... .......... .......... .......... .......... 46% 35.8M 3s 2024-04-21T08:45:40.196 INFO:tasks.workunit.client.0.smithi007.stderr: 62550K .......... .......... .......... .......... .......... 46% 25.5M 3s 2024-04-21T08:45:40.198 INFO:tasks.workunit.client.0.smithi007.stderr: 62600K .......... .......... .......... .......... .......... 46% 23.7M 3s 2024-04-21T08:45:40.199 INFO:tasks.workunit.client.0.smithi007.stderr: 62650K .......... .......... .......... .......... .......... 46% 53.0M 3s 2024-04-21T08:45:40.201 INFO:tasks.workunit.client.0.smithi007.stderr: 62700K .......... .......... .......... .......... .......... 46% 23.8M 3s 2024-04-21T08:45:40.202 INFO:tasks.workunit.client.0.smithi007.stderr: 62750K .......... .......... .......... .......... .......... 46% 39.4M 3s 2024-04-21T08:45:40.205 INFO:tasks.workunit.client.0.smithi007.stderr: 62800K .......... .......... .......... .......... .......... 46% 22.8M 3s 2024-04-21T08:45:40.207 INFO:tasks.workunit.client.0.smithi007.stderr: 62850K .......... .......... .......... .......... .......... 46% 24.6M 3s 2024-04-21T08:45:40.207 INFO:tasks.workunit.client.0.smithi007.stderr: 62900K .......... .......... .......... .......... .......... 46% 52.3M 3s 2024-04-21T08:45:40.210 INFO:tasks.workunit.client.0.smithi007.stderr: 62950K .......... .......... .......... .......... .......... 46% 23.8M 3s 2024-04-21T08:45:40.211 INFO:tasks.workunit.client.0.smithi007.stderr: 63000K .......... .......... .......... .......... .......... 46% 40.2M 3s 2024-04-21T08:45:40.213 INFO:tasks.workunit.client.0.smithi007.stderr: 63050K .......... .......... .......... .......... .......... 46% 23.6M 3s 2024-04-21T08:45:40.215 INFO:tasks.workunit.client.0.smithi007.stderr: 63100K .......... .......... .......... .......... .......... 46% 23.6M 3s 2024-04-21T08:45:40.216 INFO:tasks.workunit.client.0.smithi007.stderr: 63150K .......... .......... .......... .......... .......... 46% 51.2M 3s 2024-04-21T08:45:40.218 INFO:tasks.workunit.client.0.smithi007.stderr: 63200K .......... .......... .......... .......... .......... 46% 22.8M 3s 2024-04-21T08:45:40.219 INFO:tasks.workunit.client.0.smithi007.stderr: 63250K .......... .......... .......... .......... .......... 46% 44.1M 3s 2024-04-21T08:45:40.221 INFO:tasks.workunit.client.0.smithi007.stderr: 63300K .......... .......... .......... .......... .......... 46% 24.2M 3s 2024-04-21T08:45:40.223 INFO:tasks.workunit.client.0.smithi007.stderr: 63350K .......... .......... .......... .......... .......... 46% 24.1M 3s 2024-04-21T08:45:40.224 INFO:tasks.workunit.client.0.smithi007.stderr: 63400K .......... .......... .......... .......... .......... 46% 47.4M 3s 2024-04-21T08:45:40.226 INFO:tasks.workunit.client.0.smithi007.stderr: 63450K .......... .......... .......... .......... .......... 46% 23.3M 3s 2024-04-21T08:45:40.227 INFO:tasks.workunit.client.0.smithi007.stderr: 63500K .......... .......... .......... .......... .......... 46% 41.1M 3s 2024-04-21T08:45:40.230 INFO:tasks.workunit.client.0.smithi007.stderr: 63550K .......... .......... .......... .......... .......... 46% 22.5M 3s 2024-04-21T08:45:40.231 INFO:tasks.workunit.client.0.smithi007.stderr: 63600K .......... .......... .......... .......... .......... 46% 42.4M 3s 2024-04-21T08:45:40.232 INFO:tasks.workunit.client.0.smithi007.stderr: 63650K .......... .......... .......... .......... .......... 46% 24.8M 3s 2024-04-21T08:45:40.235 INFO:tasks.workunit.client.0.smithi007.stderr: 63700K .......... .......... .......... .......... .......... 46% 24.4M 3s 2024-04-21T08:45:40.236 INFO:tasks.workunit.client.0.smithi007.stderr: 63750K .......... .......... .......... .......... .......... 46% 35.9M 3s 2024-04-21T08:45:40.238 INFO:tasks.workunit.client.0.smithi007.stderr: 63800K .......... .......... .......... .......... .......... 47% 23.9M 3s 2024-04-21T08:45:40.240 INFO:tasks.workunit.client.0.smithi007.stderr: 63850K .......... .......... .......... .......... .......... 47% 24.0M 3s 2024-04-21T08:45:40.241 INFO:tasks.workunit.client.0.smithi007.stderr: 63900K .......... .......... .......... .......... .......... 47% 52.0M 3s 2024-04-21T08:45:40.244 INFO:tasks.workunit.client.0.smithi007.stderr: 63950K .......... .......... .......... .......... .......... 47% 22.3M 3s 2024-04-21T08:45:40.245 INFO:tasks.workunit.client.0.smithi007.stderr: 64000K .......... .......... .......... .......... .......... 47% 29.8M 3s 2024-04-21T08:45:40.247 INFO:tasks.workunit.client.0.smithi007.stderr: 64050K .......... .......... .......... .......... .......... 47% 24.3M 3s 2024-04-21T08:45:40.250 INFO:tasks.workunit.client.0.smithi007.stderr: 64100K .......... .......... .......... .......... .......... 47% 18.6M 3s 2024-04-21T08:45:40.250 INFO:tasks.workunit.client.0.smithi007.stderr: 64150K .......... .......... .......... .......... .......... 47% 42.4M 3s 2024-04-21T08:45:40.253 INFO:tasks.workunit.client.0.smithi007.stderr: 64200K .......... .......... .......... .......... .......... 47% 22.6M 3s 2024-04-21T08:45:40.255 INFO:tasks.workunit.client.0.smithi007.stderr: 64250K .......... .......... .......... .......... .......... 47% 36.0M 3s 2024-04-21T08:45:40.256 INFO:tasks.workunit.client.0.smithi007.stderr: 64300K .......... .......... .......... .......... .......... 47% 24.3M 3s 2024-04-21T08:45:40.258 INFO:tasks.workunit.client.0.smithi007.stderr: 64350K .......... .......... .......... .......... .......... 47% 24.1M 3s 2024-04-21T08:45:40.259 INFO:tasks.workunit.client.0.smithi007.stderr: 64400K .......... .......... .......... .......... .......... 47% 53.2M 3s 2024-04-21T08:45:40.261 INFO:tasks.workunit.client.0.smithi007.stderr: 64450K .......... .......... .......... .......... .......... 47% 23.8M 3s 2024-04-21T08:45:40.263 INFO:tasks.workunit.client.0.smithi007.stderr: 64500K .......... .......... .......... .......... .......... 47% 39.9M 3s 2024-04-21T08:45:40.264 INFO:tasks.workunit.client.0.smithi007.stderr: 64550K .......... .......... .......... .......... .......... 47% 26.1M 3s 2024-04-21T08:45:40.267 INFO:tasks.workunit.client.0.smithi007.stderr: 64600K .......... .......... .......... .......... .......... 47% 23.8M 3s 2024-04-21T08:45:40.267 INFO:tasks.workunit.client.0.smithi007.stderr: 64650K .......... .......... .......... .......... .......... 47% 41.1M 3s 2024-04-21T08:45:40.270 INFO:tasks.workunit.client.0.smithi007.stderr: 64700K .......... .......... .......... .......... .......... 47% 22.9M 3s 2024-04-21T08:45:40.271 INFO:tasks.workunit.client.0.smithi007.stderr: 64750K .......... .......... .......... .......... .......... 47% 37.2M 3s 2024-04-21T08:45:40.273 INFO:tasks.workunit.client.0.smithi007.stderr: 64800K .......... .......... .......... .......... .......... 47% 23.6M 3s 2024-04-21T08:45:40.275 INFO:tasks.workunit.client.0.smithi007.stderr: 64850K .......... .......... .......... .......... .......... 47% 25.3M 3s 2024-04-21T08:45:40.276 INFO:tasks.workunit.client.0.smithi007.stderr: 64900K .......... .......... .......... .......... .......... 47% 45.4M 3s 2024-04-21T08:45:40.279 INFO:tasks.workunit.client.0.smithi007.stderr: 64950K .......... .......... .......... .......... .......... 47% 20.3M 3s 2024-04-21T08:45:40.280 INFO:tasks.workunit.client.0.smithi007.stderr: 65000K .......... .......... .......... .......... .......... 47% 30.0M 3s 2024-04-21T08:45:40.283 INFO:tasks.workunit.client.0.smithi007.stderr: 65050K .......... .......... .......... .......... .......... 47% 33.1M 3s 2024-04-21T08:45:40.284 INFO:tasks.workunit.client.0.smithi007.stderr: 65100K .......... .......... .......... .......... .......... 47% 18.8M 3s 2024-04-21T08:45:40.285 INFO:tasks.workunit.client.0.smithi007.stderr: 65150K .......... .......... .......... .......... .......... 48% 49.4M 3s 2024-04-21T08:45:40.287 INFO:tasks.workunit.client.0.smithi007.stderr: 65200K .......... .......... .......... .......... .......... 48% 22.4M 3s 2024-04-21T08:45:40.289 INFO:tasks.workunit.client.0.smithi007.stderr: 65250K .......... .......... .......... .......... .......... 48% 39.5M 3s 2024-04-21T08:45:40.291 INFO:tasks.workunit.client.0.smithi007.stderr: 65300K .......... .......... .......... .......... .......... 48% 23.7M 3s 2024-04-21T08:45:40.292 INFO:tasks.workunit.client.0.smithi007.stderr: 65350K .......... .......... .......... .......... .......... 48% 24.5M 3s 2024-04-21T08:45:40.293 INFO:tasks.workunit.client.0.smithi007.stderr: 65400K .......... .......... .......... .......... .......... 48% 54.4M 3s 2024-04-21T08:45:40.296 INFO:tasks.workunit.client.0.smithi007.stderr: 65450K .......... .......... .......... .......... .......... 48% 22.6M 3s 2024-04-21T08:45:40.297 INFO:tasks.workunit.client.0.smithi007.stderr: 65500K .......... .......... .......... .......... .......... 48% 41.8M 3s 2024-04-21T08:45:40.299 INFO:tasks.workunit.client.0.smithi007.stderr: 65550K .......... .......... .......... .......... .......... 48% 23.0M 3s 2024-04-21T08:45:40.300 INFO:tasks.workunit.client.0.smithi007.stderr: 65600K .......... .......... .......... .......... .......... 48% 30.4M 2s 2024-04-21T08:45:40.302 INFO:tasks.workunit.client.0.smithi007.stderr: 65650K .......... .......... .......... .......... .......... 48% 38.8M 2s 2024-04-21T08:45:40.304 INFO:tasks.workunit.client.0.smithi007.stderr: 65700K .......... .......... .......... .......... .......... 48% 21.9M 2s 2024-04-21T08:45:40.306 INFO:tasks.workunit.client.0.smithi007.stderr: 65750K .......... .......... .......... .......... .......... 48% 34.5M 2s 2024-04-21T08:45:40.308 INFO:tasks.workunit.client.0.smithi007.stderr: 65800K .......... .......... .......... .......... .......... 48% 22.6M 2s 2024-04-21T08:45:40.310 INFO:tasks.workunit.client.0.smithi007.stderr: 65850K .......... .......... .......... .......... .......... 48% 17.7M 2s 2024-04-21T08:45:40.311 INFO:tasks.workunit.client.0.smithi007.stderr: 65900K .......... .......... .......... .......... .......... 48% 42.4M 2s 2024-04-21T08:45:40.313 INFO:tasks.workunit.client.0.smithi007.stderr: 65950K .......... .......... .......... .......... .......... 48% 26.0M 2s 2024-04-21T08:45:40.316 INFO:tasks.workunit.client.0.smithi007.stderr: 66000K .......... .......... .......... .......... .......... 48% 34.4M 2s 2024-04-21T08:45:40.316 INFO:tasks.workunit.client.0.smithi007.stderr: 66050K .......... .......... .......... .......... .......... 48% 26.4M 2s 2024-04-21T08:45:40.319 INFO:tasks.workunit.client.0.smithi007.stderr: 66100K .......... .......... .......... .......... .......... 48% 23.6M 2s 2024-04-21T08:45:40.320 INFO:tasks.workunit.client.0.smithi007.stderr: 66150K .......... .......... .......... .......... .......... 48% 40.0M 2s 2024-04-21T08:45:40.322 INFO:tasks.workunit.client.0.smithi007.stderr: 66200K .......... .......... .......... .......... .......... 48% 24.7M 2s 2024-04-21T08:45:40.326 INFO:tasks.workunit.client.0.smithi007.stderr: 66250K .......... .......... .......... .......... .......... 48% 24.9M 2s 2024-04-21T08:45:40.326 INFO:tasks.workunit.client.0.smithi007.stderr: 66300K .......... .......... .......... .......... .......... 48% 42.0M 2s 2024-04-21T08:45:40.327 INFO:tasks.workunit.client.0.smithi007.stderr: 66350K .......... .......... .......... .......... .......... 48% 22.3M 2s 2024-04-21T08:45:40.328 INFO:tasks.workunit.client.0.smithi007.stderr: 66400K .......... .......... .......... .......... .......... 48% 48.0M 2s 2024-04-21T08:45:40.330 INFO:tasks.workunit.client.0.smithi007.stderr: 66450K .......... .......... .......... .......... .......... 48% 25.3M 2s 2024-04-21T08:45:40.332 INFO:tasks.workunit.client.0.smithi007.stderr: 66500K .......... .......... .......... .......... .......... 49% 22.8M 2s 2024-04-21T08:45:40.334 INFO:tasks.workunit.client.0.smithi007.stderr: 66550K .......... .......... .......... .......... .......... 49% 38.7M 2s 2024-04-21T08:45:40.336 INFO:tasks.workunit.client.0.smithi007.stderr: 66600K .......... .......... .......... .......... .......... 49% 24.5M 2s 2024-04-21T08:45:40.336 INFO:tasks.workunit.client.0.smithi007.stderr: 66650K .......... .......... .......... .......... .......... 49% 49.7M 2s 2024-04-21T08:45:40.339 INFO:tasks.workunit.client.0.smithi007.stderr: 66700K .......... .......... .......... .......... .......... 49% 22.1M 2s 2024-04-21T08:45:40.341 INFO:tasks.workunit.client.0.smithi007.stderr: 66750K .......... .......... .......... .......... .......... 49% 23.2M 2s 2024-04-21T08:45:40.342 INFO:tasks.workunit.client.0.smithi007.stderr: 66800K .......... .......... .......... .......... .......... 49% 45.0M 2s 2024-04-21T08:45:40.344 INFO:tasks.workunit.client.0.smithi007.stderr: 66850K .......... .......... .......... .......... .......... 49% 23.5M 2s 2024-04-21T08:45:40.345 INFO:tasks.workunit.client.0.smithi007.stderr: 66900K .......... .......... .......... .......... .......... 49% 51.3M 2s 2024-04-21T08:45:40.347 INFO:tasks.workunit.client.0.smithi007.stderr: 66950K .......... .......... .......... .......... .......... 49% 21.7M 2s 2024-04-21T08:45:40.349 INFO:tasks.workunit.client.0.smithi007.stderr: 67000K .......... .......... .......... .......... .......... 49% 26.5M 2s 2024-04-21T08:45:40.351 INFO:tasks.workunit.client.0.smithi007.stderr: 67050K .......... .......... .......... .......... .......... 49% 29.7M 2s 2024-04-21T08:45:40.353 INFO:tasks.workunit.client.0.smithi007.stderr: 67100K .......... .......... .......... .......... .......... 49% 21.2M 2s 2024-04-21T08:45:40.354 INFO:tasks.workunit.client.0.smithi007.stderr: 67150K .......... .......... .......... .......... .......... 49% 53.3M 2s 2024-04-21T08:45:40.356 INFO:tasks.workunit.client.0.smithi007.stderr: 67200K .......... .......... .......... .......... .......... 49% 22.6M 2s 2024-04-21T08:45:40.358 INFO:tasks.workunit.client.0.smithi007.stderr: 67250K .......... .......... .......... .......... .......... 49% 24.8M 2s 2024-04-21T08:45:40.359 INFO:tasks.workunit.client.0.smithi007.stderr: 67300K .......... .......... .......... .......... .......... 49% 37.9M 2s 2024-04-21T08:45:40.362 INFO:tasks.workunit.client.0.smithi007.stderr: 67350K .......... .......... .......... .......... .......... 49% 22.9M 2s 2024-04-21T08:45:40.364 INFO:tasks.workunit.client.0.smithi007.stderr: 67400K .......... .......... .......... .......... .......... 49% 54.9M 2s 2024-04-21T08:45:40.364 INFO:tasks.workunit.client.0.smithi007.stderr: 67450K .......... .......... .......... .......... .......... 49% 23.0M 2s 2024-04-21T08:45:40.366 INFO:tasks.workunit.client.0.smithi007.stderr: 67500K .......... .......... .......... .......... .......... 49% 24.6M 2s 2024-04-21T08:45:40.368 INFO:tasks.workunit.client.0.smithi007.stderr: 67550K .......... .......... .......... .......... .......... 49% 35.9M 2s 2024-04-21T08:45:40.370 INFO:tasks.workunit.client.0.smithi007.stderr: 67600K .......... .......... .......... .......... .......... 49% 24.1M 2s 2024-04-21T08:45:40.372 INFO:tasks.workunit.client.0.smithi007.stderr: 67650K .......... .......... .......... .......... .......... 49% 23.9M 2s 2024-04-21T08:45:40.373 INFO:tasks.workunit.client.0.smithi007.stderr: 67700K .......... .......... .......... .......... .......... 49% 47.1M 2s 2024-04-21T08:45:40.375 INFO:tasks.workunit.client.0.smithi007.stderr: 67750K .......... .......... .......... .......... .......... 49% 22.5M 2s 2024-04-21T08:45:40.376 INFO:tasks.workunit.client.0.smithi007.stderr: 67800K .......... .......... .......... .......... .......... 49% 39.6M 2s 2024-04-21T08:45:40.378 INFO:tasks.workunit.client.0.smithi007.stderr: 67850K .......... .......... .......... .......... .......... 50% 25.0M 2s 2024-04-21T08:45:40.381 INFO:tasks.workunit.client.0.smithi007.stderr: 67900K .......... .......... .......... .......... .......... 50% 22.5M 2s 2024-04-21T08:45:40.381 INFO:tasks.workunit.client.0.smithi007.stderr: 67950K .......... .......... .......... .......... .......... 50% 49.8M 2s 2024-04-21T08:45:40.383 INFO:tasks.workunit.client.0.smithi007.stderr: 68000K .......... .......... .......... .......... .......... 50% 23.9M 2s 2024-04-21T08:45:40.385 INFO:tasks.workunit.client.0.smithi007.stderr: 68050K .......... .......... .......... .......... .......... 50% 40.6M 2s 2024-04-21T08:45:40.387 INFO:tasks.workunit.client.0.smithi007.stderr: 68100K .......... .......... .......... .......... .......... 50% 23.1M 2s 2024-04-21T08:45:40.389 INFO:tasks.workunit.client.0.smithi007.stderr: 68150K .......... .......... .......... .......... .......... 50% 24.7M 2s 2024-04-21T08:45:40.390 INFO:tasks.workunit.client.0.smithi007.stderr: 68200K .......... .......... .......... .......... .......... 50% 38.7M 2s 2024-04-21T08:45:40.392 INFO:tasks.workunit.client.0.smithi007.stderr: 68250K .......... .......... .......... .......... .......... 50% 26.1M 2s 2024-04-21T08:45:40.393 INFO:tasks.workunit.client.0.smithi007.stderr: 68300K .......... .......... .......... .......... .......... 50% 37.8M 2s 2024-04-21T08:45:40.395 INFO:tasks.workunit.client.0.smithi007.stderr: 68350K .......... .......... .......... .......... .......... 50% 23.9M 2s 2024-04-21T08:45:40.397 INFO:tasks.workunit.client.0.smithi007.stderr: 68400K .......... .......... .......... .......... .......... 50% 26.5M 2s 2024-04-21T08:45:40.398 INFO:tasks.workunit.client.0.smithi007.stderr: 68450K .......... .......... .......... .......... .......... 50% 38.7M 2s 2024-04-21T08:45:40.400 INFO:tasks.workunit.client.0.smithi007.stderr: 68500K .......... .......... .......... .......... .......... 50% 23.6M 2s 2024-04-21T08:45:40.402 INFO:tasks.workunit.client.0.smithi007.stderr: 68550K .......... .......... .......... .......... .......... 50% 40.0M 2s 2024-04-21T08:45:40.404 INFO:tasks.workunit.client.0.smithi007.stderr: 68600K .......... .......... .......... .......... .......... 50% 23.6M 2s 2024-04-21T08:45:40.406 INFO:tasks.workunit.client.0.smithi007.stderr: 68650K .......... .......... .......... .......... .......... 50% 24.5M 2s 2024-04-21T08:45:40.407 INFO:tasks.workunit.client.0.smithi007.stderr: 68700K .......... .......... .......... .......... .......... 50% 52.1M 2s 2024-04-21T08:45:40.409 INFO:tasks.workunit.client.0.smithi007.stderr: 68750K .......... .......... .......... .......... .......... 50% 23.0M 2s 2024-04-21T08:45:40.411 INFO:tasks.workunit.client.0.smithi007.stderr: 68800K .......... .......... .......... .......... .......... 50% 27.1M 2s 2024-04-21T08:45:40.412 INFO:tasks.workunit.client.0.smithi007.stderr: 68850K .......... .......... .......... .......... .......... 50% 32.7M 2s 2024-04-21T08:45:40.414 INFO:tasks.workunit.client.0.smithi007.stderr: 68900K .......... .......... .......... .......... .......... 50% 26.4M 2s 2024-04-21T08:45:40.415 INFO:tasks.workunit.client.0.smithi007.stderr: 68950K .......... .......... .......... .......... .......... 50% 42.5M 2s 2024-04-21T08:45:40.417 INFO:tasks.workunit.client.0.smithi007.stderr: 69000K .......... .......... .......... .......... .......... 50% 23.6M 2s 2024-04-21T08:45:40.419 INFO:tasks.workunit.client.0.smithi007.stderr: 69050K .......... .......... .......... .......... .......... 50% 35.6M 2s 2024-04-21T08:45:40.420 INFO:tasks.workunit.client.0.smithi007.stderr: 69100K .......... .......... .......... .......... .......... 50% 24.6M 2s 2024-04-21T08:45:40.423 INFO:tasks.workunit.client.0.smithi007.stderr: 69150K .......... .......... .......... .......... .......... 50% 26.1M 2s 2024-04-21T08:45:40.423 INFO:tasks.workunit.client.0.smithi007.stderr: 69200K .......... .......... .......... .......... .......... 51% 44.0M 2s 2024-04-21T08:45:40.426 INFO:tasks.workunit.client.0.smithi007.stderr: 69250K .......... .......... .......... .......... .......... 51% 24.1M 2s 2024-04-21T08:45:40.427 INFO:tasks.workunit.client.0.smithi007.stderr: 69300K .......... .......... .......... .......... .......... 51% 40.7M 2s 2024-04-21T08:45:40.429 INFO:tasks.workunit.client.0.smithi007.stderr: 69350K .......... .......... .......... .......... .......... 51% 22.3M 2s 2024-04-21T08:45:40.431 INFO:tasks.workunit.client.0.smithi007.stderr: 69400K .......... .......... .......... .......... .......... 51% 25.2M 2s 2024-04-21T08:45:40.432 INFO:tasks.workunit.client.0.smithi007.stderr: 69450K .......... .......... .......... .......... .......... 51% 43.2M 2s 2024-04-21T08:45:40.434 INFO:tasks.workunit.client.0.smithi007.stderr: 69500K .......... .......... .......... .......... .......... 51% 25.0M 2s 2024-04-21T08:45:40.435 INFO:tasks.workunit.client.0.smithi007.stderr: 69550K .......... .......... .......... .......... .......... 51% 40.9M 2s 2024-04-21T08:45:40.437 INFO:tasks.workunit.client.0.smithi007.stderr: 69600K .......... .......... .......... .......... .......... 51% 22.8M 2s 2024-04-21T08:45:40.439 INFO:tasks.workunit.client.0.smithi007.stderr: 69650K .......... .......... .......... .......... .......... 51% 24.8M 2s 2024-04-21T08:45:40.440 INFO:tasks.workunit.client.0.smithi007.stderr: 69700K .......... .......... .......... .......... .......... 51% 41.2M 2s 2024-04-21T08:45:40.442 INFO:tasks.workunit.client.0.smithi007.stderr: 69750K .......... .......... .......... .......... .......... 51% 27.2M 2s 2024-04-21T08:45:40.443 INFO:tasks.workunit.client.0.smithi007.stderr: 69800K .......... .......... .......... .......... .......... 51% 35.5M 2s 2024-04-21T08:45:40.446 INFO:tasks.workunit.client.0.smithi007.stderr: 69850K .......... .......... .......... .......... .......... 51% 24.4M 2s 2024-04-21T08:45:40.448 INFO:tasks.workunit.client.0.smithi007.stderr: 69900K .......... .......... .......... .......... .......... 51% 23.7M 2s 2024-04-21T08:45:40.449 INFO:tasks.workunit.client.0.smithi007.stderr: 69950K .......... .......... .......... .......... .......... 51% 45.2M 2s 2024-04-21T08:45:40.451 INFO:tasks.workunit.client.0.smithi007.stderr: 70000K .......... .......... .......... .......... .......... 51% 28.5M 2s 2024-04-21T08:45:40.452 INFO:tasks.workunit.client.0.smithi007.stderr: 70050K .......... .......... .......... .......... .......... 51% 30.7M 2s 2024-04-21T08:45:40.454 INFO:tasks.workunit.client.0.smithi007.stderr: 70100K .......... .......... .......... .......... .......... 51% 25.1M 2s 2024-04-21T08:45:40.456 INFO:tasks.workunit.client.0.smithi007.stderr: 70150K .......... .......... .......... .......... .......... 51% 22.6M 2s 2024-04-21T08:45:40.457 INFO:tasks.workunit.client.0.smithi007.stderr: 70200K .......... .......... .......... .......... .......... 51% 46.8M 2s 2024-04-21T08:45:40.459 INFO:tasks.workunit.client.0.smithi007.stderr: 70250K .......... .......... .......... .......... .......... 51% 22.7M 2s 2024-04-21T08:45:40.460 INFO:tasks.workunit.client.0.smithi007.stderr: 70300K .......... .......... .......... .......... .......... 51% 39.1M 2s 2024-04-21T08:45:40.463 INFO:tasks.workunit.client.0.smithi007.stderr: 70350K .......... .......... .......... .......... .......... 51% 23.0M 2s 2024-04-21T08:45:40.465 INFO:tasks.workunit.client.0.smithi007.stderr: 70400K .......... .......... .......... .......... .......... 51% 24.6M 2s 2024-04-21T08:45:40.466 INFO:tasks.workunit.client.0.smithi007.stderr: 70450K .......... .......... .......... .......... .......... 51% 51.1M 2s 2024-04-21T08:45:40.468 INFO:tasks.workunit.client.0.smithi007.stderr: 70500K .......... .......... .......... .......... .......... 51% 18.3M 2s 2024-04-21T08:45:40.470 INFO:tasks.workunit.client.0.smithi007.stderr: 70550K .......... .......... .......... .......... .......... 52% 40.6M 2s 2024-04-21T08:45:40.471 INFO:tasks.workunit.client.0.smithi007.stderr: 70600K .......... .......... .......... .......... .......... 52% 24.8M 2s 2024-04-21T08:45:40.474 INFO:tasks.workunit.client.0.smithi007.stderr: 70650K .......... .......... .......... .......... .......... 52% 19.6M 2s 2024-04-21T08:45:40.475 INFO:tasks.workunit.client.0.smithi007.stderr: 70700K .......... .......... .......... .......... .......... 52% 47.4M 2s 2024-04-21T08:45:40.477 INFO:tasks.workunit.client.0.smithi007.stderr: 70750K .......... .......... .......... .......... .......... 52% 22.0M 2s 2024-04-21T08:45:40.478 INFO:tasks.workunit.client.0.smithi007.stderr: 70800K .......... .......... .......... .......... .......... 52% 43.5M 2s 2024-04-21T08:45:40.481 INFO:tasks.workunit.client.0.smithi007.stderr: 70850K .......... .......... .......... .......... .......... 52% 20.4M 2s 2024-04-21T08:45:40.483 INFO:tasks.workunit.client.0.smithi007.stderr: 70900K .......... .......... .......... .......... .......... 52% 23.4M 2s 2024-04-21T08:45:40.483 INFO:tasks.workunit.client.0.smithi007.stderr: 70950K .......... .......... .......... .......... .......... 52% 55.6M 2s 2024-04-21T08:45:40.486 INFO:tasks.workunit.client.0.smithi007.stderr: 71000K .......... .......... .......... .......... .......... 52% 22.6M 2s 2024-04-21T08:45:40.488 INFO:tasks.workunit.client.0.smithi007.stderr: 71050K .......... .......... .......... .......... .......... 52% 39.2M 2s 2024-04-21T08:45:40.489 INFO:tasks.workunit.client.0.smithi007.stderr: 71100K .......... .......... .......... .......... .......... 52% 25.8M 2s 2024-04-21T08:45:40.491 INFO:tasks.workunit.client.0.smithi007.stderr: 71150K .......... .......... .......... .......... .......... 52% 23.0M 2s 2024-04-21T08:45:40.492 INFO:tasks.workunit.client.0.smithi007.stderr: 71200K .......... .......... .......... .......... .......... 52% 44.2M 2s 2024-04-21T08:45:40.494 INFO:tasks.workunit.client.0.smithi007.stderr: 71250K .......... .......... .......... .......... .......... 52% 23.7M 2s 2024-04-21T08:45:40.496 INFO:tasks.workunit.client.0.smithi007.stderr: 71300K .......... .......... .......... .......... .......... 52% 38.7M 2s 2024-04-21T08:45:40.498 INFO:tasks.workunit.client.0.smithi007.stderr: 71350K .......... .......... .......... .......... .......... 52% 23.5M 2s 2024-04-21T08:45:40.500 INFO:tasks.workunit.client.0.smithi007.stderr: 71400K .......... .......... .......... .......... .......... 52% 23.6M 2s 2024-04-21T08:45:40.501 INFO:tasks.workunit.client.0.smithi007.stderr: 71450K .......... .......... .......... .......... .......... 52% 48.6M 2s 2024-04-21T08:45:40.503 INFO:tasks.workunit.client.0.smithi007.stderr: 71500K .......... .......... .......... .......... .......... 52% 23.4M 2s 2024-04-21T08:45:40.505 INFO:tasks.workunit.client.0.smithi007.stderr: 71550K .......... .......... .......... .......... .......... 52% 24.0M 2s 2024-04-21T08:45:40.506 INFO:tasks.workunit.client.0.smithi007.stderr: 71600K .......... .......... .......... .......... .......... 52% 41.6M 2s 2024-04-21T08:45:40.508 INFO:tasks.workunit.client.0.smithi007.stderr: 71650K .......... .......... .......... .......... .......... 52% 23.4M 2s 2024-04-21T08:45:40.509 INFO:tasks.workunit.client.0.smithi007.stderr: 71700K .......... .......... .......... .......... .......... 52% 45.0M 2s 2024-04-21T08:45:40.511 INFO:tasks.workunit.client.0.smithi007.stderr: 71750K .......... .......... .......... .......... .......... 52% 24.6M 2s 2024-04-21T08:45:40.513 INFO:tasks.workunit.client.0.smithi007.stderr: 71800K .......... .......... .......... .......... .......... 52% 28.2M 2s 2024-04-21T08:45:40.514 INFO:tasks.workunit.client.0.smithi007.stderr: 71850K .......... .......... .......... .......... .......... 52% 34.3M 2s 2024-04-21T08:45:40.517 INFO:tasks.workunit.client.0.smithi007.stderr: 71900K .......... .......... .......... .......... .......... 53% 22.7M 2s 2024-04-21T08:45:40.517 INFO:tasks.workunit.client.0.smithi007.stderr: 71950K .......... .......... .......... .......... .......... 53% 53.3M 2s 2024-04-21T08:45:40.519 INFO:tasks.workunit.client.0.smithi007.stderr: 72000K .......... .......... .......... .......... .......... 53% 23.6M 2s 2024-04-21T08:45:40.522 INFO:tasks.workunit.client.0.smithi007.stderr: 72050K .......... .......... .......... .......... .......... 53% 23.1M 2s 2024-04-21T08:45:40.523 INFO:tasks.workunit.client.0.smithi007.stderr: 72100K .......... .......... .......... .......... .......... 53% 42.2M 2s 2024-04-21T08:45:40.525 INFO:tasks.workunit.client.0.smithi007.stderr: 72150K .......... .......... .......... .......... .......... 53% 20.3M 2s 2024-04-21T08:45:40.526 INFO:tasks.workunit.client.0.smithi007.stderr: 72200K .......... .......... .......... .......... .......... 53% 48.3M 2s 2024-04-21T08:45:40.528 INFO:tasks.workunit.client.0.smithi007.stderr: 72250K .......... .......... .......... .......... .......... 53% 24.2M 2s 2024-04-21T08:45:40.530 INFO:tasks.workunit.client.0.smithi007.stderr: 72300K .......... .......... .......... .......... .......... 53% 24.9M 2s 2024-04-21T08:45:40.532 INFO:tasks.workunit.client.0.smithi007.stderr: 72350K .......... .......... .......... .......... .......... 53% 41.0M 2s 2024-04-21T08:45:40.534 INFO:tasks.workunit.client.0.smithi007.stderr: 72400K .......... .......... .......... .......... .......... 53% 21.7M 2s 2024-04-21T08:45:40.535 INFO:tasks.workunit.client.0.smithi007.stderr: 72450K .......... .......... .......... .......... .......... 53% 39.5M 2s 2024-04-21T08:45:40.537 INFO:tasks.workunit.client.0.smithi007.stderr: 72500K .......... .......... .......... .......... .......... 53% 23.6M 2s 2024-04-21T08:45:40.539 INFO:tasks.workunit.client.0.smithi007.stderr: 72550K .......... .......... .......... .......... .......... 53% 20.1M 2s 2024-04-21T08:45:40.541 INFO:tasks.workunit.client.0.smithi007.stderr: 72600K .......... .......... .......... .......... .......... 53% 31.4M 2s 2024-04-21T08:45:40.543 INFO:tasks.workunit.client.0.smithi007.stderr: 72650K .......... .......... .......... .......... .......... 53% 22.6M 2s 2024-04-21T08:45:40.545 INFO:tasks.workunit.client.0.smithi007.stderr: 72700K .......... .......... .......... .......... .......... 53% 57.1M 2s 2024-04-21T08:45:40.546 INFO:tasks.workunit.client.0.smithi007.stderr: 72750K .......... .......... .......... .......... .......... 53% 20.5M 2s 2024-04-21T08:45:40.548 INFO:tasks.workunit.client.0.smithi007.stderr: 72800K .......... .......... .......... .......... .......... 53% 23.8M 2s 2024-04-21T08:45:40.550 INFO:tasks.workunit.client.0.smithi007.stderr: 72850K .......... .......... .......... .......... .......... 53% 36.8M 2s 2024-04-21T08:45:40.552 INFO:tasks.workunit.client.0.smithi007.stderr: 72900K .......... .......... .......... .......... .......... 53% 24.4M 2s 2024-04-21T08:45:40.554 INFO:tasks.workunit.client.0.smithi007.stderr: 72950K .......... .......... .......... .......... .......... 53% 25.3M 2s 2024-04-21T08:45:40.555 INFO:tasks.workunit.client.0.smithi007.stderr: 73000K .......... .......... .......... .......... .......... 53% 41.1M 2s 2024-04-21T08:45:40.556 INFO:tasks.workunit.client.0.smithi007.stderr: 73050K .......... .......... .......... .......... .......... 53% 27.7M 2s 2024-04-21T08:45:40.558 INFO:tasks.workunit.client.0.smithi007.stderr: 73100K .......... .......... .......... .......... .......... 53% 34.4M 2s 2024-04-21T08:45:40.560 INFO:tasks.workunit.client.0.smithi007.stderr: 73150K .......... .......... .......... .......... .......... 53% 23.9M 2s 2024-04-21T08:45:40.562 INFO:tasks.workunit.client.0.smithi007.stderr: 73200K .......... .......... .......... .......... .......... 53% 24.4M 2s 2024-04-21T08:45:40.563 INFO:tasks.workunit.client.0.smithi007.stderr: 73250K .......... .......... .......... .......... .......... 53% 47.1M 2s 2024-04-21T08:45:40.565 INFO:tasks.workunit.client.0.smithi007.stderr: 73300K .......... .......... .......... .......... .......... 54% 27.8M 2s 2024-04-21T08:45:40.566 INFO:tasks.workunit.client.0.smithi007.stderr: 73350K .......... .......... .......... .......... .......... 54% 34.9M 2s 2024-04-21T08:45:40.568 INFO:tasks.workunit.client.0.smithi007.stderr: 73400K .......... .......... .......... .......... .......... 54% 21.8M 2s 2024-04-21T08:45:40.571 INFO:tasks.workunit.client.0.smithi007.stderr: 73450K .......... .......... .......... .......... .......... 54% 20.1M 2s 2024-04-21T08:45:40.572 INFO:tasks.workunit.client.0.smithi007.stderr: 73500K .......... .......... .......... .......... .......... 54% 36.6M 2s 2024-04-21T08:45:40.574 INFO:tasks.workunit.client.0.smithi007.stderr: 73550K .......... .......... .......... .......... .......... 54% 23.5M 2s 2024-04-21T08:45:40.576 INFO:tasks.workunit.client.0.smithi007.stderr: 73600K .......... .......... .......... .......... .......... 54% 37.7M 2s 2024-04-21T08:45:40.577 INFO:tasks.workunit.client.0.smithi007.stderr: 73650K .......... .......... .......... .......... .......... 54% 25.3M 2s 2024-04-21T08:45:40.580 INFO:tasks.workunit.client.0.smithi007.stderr: 73700K .......... .......... .......... .......... .......... 54% 23.5M 2s 2024-04-21T08:45:40.581 INFO:tasks.workunit.client.0.smithi007.stderr: 73750K .......... .......... .......... .......... .......... 54% 49.8M 2s 2024-04-21T08:45:40.583 INFO:tasks.workunit.client.0.smithi007.stderr: 73800K .......... .......... .......... .......... .......... 54% 24.6M 2s 2024-04-21T08:45:40.584 INFO:tasks.workunit.client.0.smithi007.stderr: 73850K .......... .......... .......... .......... .......... 54% 34.0M 2s 2024-04-21T08:45:40.586 INFO:tasks.workunit.client.0.smithi007.stderr: 73900K .......... .......... .......... .......... .......... 54% 25.4M 2s 2024-04-21T08:45:40.588 INFO:tasks.workunit.client.0.smithi007.stderr: 73950K .......... .......... .......... .......... .......... 54% 24.1M 2s 2024-04-21T08:45:40.589 INFO:tasks.workunit.client.0.smithi007.stderr: 74000K .......... .......... .......... .......... .......... 54% 49.4M 2s 2024-04-21T08:45:40.591 INFO:tasks.workunit.client.0.smithi007.stderr: 74050K .......... .......... .......... .......... .......... 54% 23.5M 2s 2024-04-21T08:45:40.592 INFO:tasks.workunit.client.0.smithi007.stderr: 74100K .......... .......... .......... .......... .......... 54% 41.4M 2s 2024-04-21T08:45:40.594 INFO:tasks.workunit.client.0.smithi007.stderr: 74150K .......... .......... .......... .......... .......... 54% 23.6M 2s 2024-04-21T08:45:40.596 INFO:tasks.workunit.client.0.smithi007.stderr: 74200K .......... .......... .......... .......... .......... 54% 24.3M 2s 2024-04-21T08:45:40.597 INFO:tasks.workunit.client.0.smithi007.stderr: 74250K .......... .......... .......... .......... .......... 54% 37.2M 2s 2024-04-21T08:45:40.600 INFO:tasks.workunit.client.0.smithi007.stderr: 74300K .......... .......... .......... .......... .......... 54% 20.2M 2s 2024-04-21T08:45:40.601 INFO:tasks.workunit.client.0.smithi007.stderr: 74350K .......... .......... .......... .......... .......... 54% 43.1M 2s 2024-04-21T08:45:40.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:40 smithi077 ceph-mon[31615]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-21T08:45:40.604 INFO:tasks.workunit.client.0.smithi007.stderr: 74400K .......... .......... .......... .......... .......... 54% 17.3M 2s 2024-04-21T08:45:40.606 INFO:tasks.workunit.client.0.smithi007.stderr: 74450K .......... .......... .......... .......... .......... 54% 25.6M 2s 2024-04-21T08:45:40.607 INFO:tasks.workunit.client.0.smithi007.stderr: 74500K .......... .......... .......... .......... .......... 54% 35.0M 2s 2024-04-21T08:45:40.609 INFO:tasks.workunit.client.0.smithi007.stderr: 74550K .......... .......... .......... .......... .......... 54% 24.1M 2s 2024-04-21T08:45:40.611 INFO:tasks.workunit.client.0.smithi007.stderr: 74600K .......... .......... .......... .......... .......... 54% 39.3M 2s 2024-04-21T08:45:40.612 INFO:tasks.workunit.client.0.smithi007.stderr: 74650K .......... .......... .......... .......... .......... 55% 22.9M 2s 2024-04-21T08:45:40.615 INFO:tasks.workunit.client.0.smithi007.stderr: 74700K .......... .......... .......... .......... .......... 55% 25.5M 2s 2024-04-21T08:45:40.616 INFO:tasks.workunit.client.0.smithi007.stderr: 74750K .......... .......... .......... .......... .......... 55% 51.4M 2s 2024-04-21T08:45:40.618 INFO:tasks.workunit.client.0.smithi007.stderr: 74800K .......... .......... .......... .......... .......... 55% 22.8M 2s 2024-04-21T08:45:40.619 INFO:tasks.workunit.client.0.smithi007.stderr: 74850K .......... .......... .......... .......... .......... 55% 43.0M 2s 2024-04-21T08:45:40.621 INFO:tasks.workunit.client.0.smithi007.stderr: 74900K .......... .......... .......... .......... .......... 55% 22.8M 2s 2024-04-21T08:45:40.623 INFO:tasks.workunit.client.0.smithi007.stderr: 74950K .......... .......... .......... .......... .......... 55% 24.1M 2s 2024-04-21T08:45:40.624 INFO:tasks.workunit.client.0.smithi007.stderr: 75000K .......... .......... .......... .......... .......... 55% 44.4M 2s 2024-04-21T08:45:40.626 INFO:tasks.workunit.client.0.smithi007.stderr: 75050K .......... .......... .......... .......... .......... 55% 29.9M 2s 2024-04-21T08:45:40.627 INFO:tasks.workunit.client.0.smithi007.stderr: 75100K .......... .......... .......... .......... .......... 55% 32.3M 2s 2024-04-21T08:45:40.630 INFO:tasks.workunit.client.0.smithi007.stderr: 75150K .......... .......... .......... .......... .......... 55% 18.9M 2s 2024-04-21T08:45:40.632 INFO:tasks.workunit.client.0.smithi007.stderr: 75200K .......... .......... .......... .......... .......... 55% 28.2M 2s 2024-04-21T08:45:40.633 INFO:tasks.workunit.client.0.smithi007.stderr: 75250K .......... .......... .......... .......... .......... 55% 47.0M 2s 2024-04-21T08:45:40.634 INFO:tasks.workunit.client.0.smithi007.stderr: 75300K .......... .......... .......... .......... .......... 55% 32.5M 2s 2024-04-21T08:45:40.636 INFO:tasks.workunit.client.0.smithi007.stderr: 75350K .......... .......... .......... .......... .......... 55% 23.8M 2s 2024-04-21T08:45:40.638 INFO:tasks.workunit.client.0.smithi007.stderr: 75400K .......... .......... .......... .......... .......... 55% 25.1M 2s 2024-04-21T08:45:40.640 INFO:tasks.workunit.client.0.smithi007.stderr: 75450K .......... .......... .......... .......... .......... 55% 24.6M 2s 2024-04-21T08:45:40.641 INFO:tasks.workunit.client.0.smithi007.stderr: 75500K .......... .......... .......... .......... .......... 55% 41.9M 2s 2024-04-21T08:45:40.643 INFO:tasks.workunit.client.0.smithi007.stderr: 75550K .......... .......... .......... .......... .......... 55% 20.9M 2s 2024-04-21T08:45:40.645 INFO:tasks.workunit.client.0.smithi007.stderr: 75600K .......... .......... .......... .......... .......... 55% 39.1M 2s 2024-04-21T08:45:40.647 INFO:tasks.workunit.client.0.smithi007.stderr: 75650K .......... .......... .......... .......... .......... 55% 22.4M 2s 2024-04-21T08:45:40.648 INFO:tasks.workunit.client.0.smithi007.stderr: 75700K .......... .......... .......... .......... .......... 55% 32.6M 2s 2024-04-21T08:45:40.650 INFO:tasks.workunit.client.0.smithi007.stderr: 75750K .......... .......... .......... .......... .......... 55% 33.2M 2s 2024-04-21T08:45:40.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:40 smithi007 ceph-mon[28477]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-21T08:45:40.652 INFO:tasks.workunit.client.0.smithi007.stderr: 75800K .......... .......... .......... .......... .......... 55% 22.9M 2s 2024-04-21T08:45:40.653 INFO:tasks.workunit.client.0.smithi007.stderr: 75850K .......... .......... .......... .......... .......... 55% 33.8M 2s 2024-04-21T08:45:40.655 INFO:tasks.workunit.client.0.smithi007.stderr: 75900K .......... .......... .......... .......... .......... 55% 25.4M 2s 2024-04-21T08:45:40.658 INFO:tasks.workunit.client.0.smithi007.stderr: 75950K .......... .......... .......... .......... .......... 55% 45.7M 2s 2024-04-21T08:45:40.658 INFO:tasks.workunit.client.0.smithi007.stderr: 76000K .......... .......... .......... .......... .......... 56% 25.5M 2s 2024-04-21T08:45:40.660 INFO:tasks.workunit.client.0.smithi007.stderr: 76050K .......... .......... .......... .......... .......... 56% 24.9M 2s 2024-04-21T08:45:40.662 INFO:tasks.workunit.client.0.smithi007.stderr: 76100K .......... .......... .......... .......... .......... 56% 34.3M 2s 2024-04-21T08:45:40.664 INFO:tasks.workunit.client.0.smithi007.stderr: 76150K .......... .......... .......... .......... .......... 56% 21.2M 2s 2024-04-21T08:45:40.665 INFO:tasks.workunit.client.0.smithi007.stderr: 76200K .......... .......... .......... .......... .......... 56% 59.5M 2s 2024-04-21T08:45:40.667 INFO:tasks.workunit.client.0.smithi007.stderr: 76250K .......... .......... .......... .......... .......... 56% 24.6M 2s 2024-04-21T08:45:40.669 INFO:tasks.workunit.client.0.smithi007.stderr: 76300K .......... .......... .......... .......... .......... 56% 23.2M 2s 2024-04-21T08:45:40.670 INFO:tasks.workunit.client.0.smithi007.stderr: 76350K .......... .......... .......... .......... .......... 56% 31.7M 2s 2024-04-21T08:45:40.672 INFO:tasks.workunit.client.0.smithi007.stderr: 76400K .......... .......... .......... .......... .......... 56% 29.2M 2s 2024-04-21T08:45:40.675 INFO:tasks.workunit.client.0.smithi007.stderr: 76450K .......... .......... .......... .......... .......... 56% 22.6M 2s 2024-04-21T08:45:40.675 INFO:tasks.workunit.client.0.smithi007.stderr: 76500K .......... .......... .......... .......... .......... 56% 51.5M 2s 2024-04-21T08:45:40.677 INFO:tasks.workunit.client.0.smithi007.stderr: 76550K .......... .......... .......... .......... .......... 56% 23.9M 2s 2024-04-21T08:45:40.679 INFO:tasks.workunit.client.0.smithi007.stderr: 76600K .......... .......... .......... .......... .......... 56% 37.6M 2s 2024-04-21T08:45:40.681 INFO:tasks.workunit.client.0.smithi007.stderr: 76650K .......... .......... .......... .......... .......... 56% 22.9M 2s 2024-04-21T08:45:40.682 INFO:tasks.workunit.client.0.smithi007.stderr: 76700K .......... .......... .......... .......... .......... 56% 40.3M 2s 2024-04-21T08:45:40.683 INFO:tasks.workunit.client.0.smithi007.stderr: 76750K .......... .......... .......... .......... .......... 56% 29.5M 2s 2024-04-21T08:45:40.686 INFO:tasks.workunit.client.0.smithi007.stderr: 76800K .......... .......... .......... .......... .......... 56% 22.5M 2s 2024-04-21T08:45:40.688 INFO:tasks.workunit.client.0.smithi007.stderr: 76850K .......... .......... .......... .......... .......... 56% 27.2M 2s 2024-04-21T08:45:40.689 INFO:tasks.workunit.client.0.smithi007.stderr: 76900K .......... .......... .......... .......... .......... 56% 34.9M 2s 2024-04-21T08:45:40.690 INFO:tasks.workunit.client.0.smithi007.stderr: 76950K .......... .......... .......... .......... .......... 56% 29.9M 2s 2024-04-21T08:45:40.692 INFO:tasks.workunit.client.0.smithi007.stderr: 77000K .......... .......... .......... .......... .......... 56% 35.3M 2s 2024-04-21T08:45:40.694 INFO:tasks.workunit.client.0.smithi007.stderr: 77050K .......... .......... .......... .......... .......... 56% 22.9M 2s 2024-04-21T08:45:40.696 INFO:tasks.workunit.client.0.smithi007.stderr: 77100K .......... .......... .......... .......... .......... 56% 24.1M 2s 2024-04-21T08:45:40.698 INFO:tasks.workunit.client.0.smithi007.stderr: 77150K .......... .......... .......... .......... .......... 56% 40.0M 2s 2024-04-21T08:45:40.699 INFO:tasks.workunit.client.0.smithi007.stderr: 77200K .......... .......... .......... .......... .......... 56% 22.9M 2s 2024-04-21T08:45:40.700 INFO:tasks.workunit.client.0.smithi007.stderr: 77250K .......... .......... .......... .......... .......... 56% 51.2M 2s 2024-04-21T08:45:40.702 INFO:tasks.workunit.client.0.smithi007.stderr: 77300K .......... .......... .......... .......... .......... 56% 24.9M 2s 2024-04-21T08:45:40.704 INFO:tasks.workunit.client.0.smithi007.stderr: 77350K .......... .......... .......... .......... .......... 57% 23.8M 2s 2024-04-21T08:45:40.706 INFO:tasks.workunit.client.0.smithi007.stderr: 77400K .......... .......... .......... .......... .......... 57% 31.9M 2s 2024-04-21T08:45:40.708 INFO:tasks.workunit.client.0.smithi007.stderr: 77450K .......... .......... .......... .......... .......... 57% 25.6M 2s 2024-04-21T08:45:40.709 INFO:tasks.workunit.client.0.smithi007.stderr: 77500K .......... .......... .......... .......... .......... 57% 45.4M 2s 2024-04-21T08:45:40.711 INFO:tasks.workunit.client.0.smithi007.stderr: 77550K .......... .......... .......... .......... .......... 57% 25.7M 2s 2024-04-21T08:45:40.713 INFO:tasks.workunit.client.0.smithi007.stderr: 77600K .......... .......... .......... .......... .......... 57% 26.5M 2s 2024-04-21T08:45:40.714 INFO:tasks.workunit.client.0.smithi007.stderr: 77650K .......... .......... .......... .......... .......... 57% 31.5M 2s 2024-04-21T08:45:40.716 INFO:tasks.workunit.client.0.smithi007.stderr: 77700K .......... .......... .......... .......... .......... 57% 24.8M 2s 2024-04-21T08:45:40.718 INFO:tasks.workunit.client.0.smithi007.stderr: 77750K .......... .......... .......... .......... .......... 57% 51.1M 2s 2024-04-21T08:45:40.719 INFO:tasks.workunit.client.0.smithi007.stderr: 77800K .......... .......... .......... .......... .......... 57% 21.4M 2s 2024-04-21T08:45:40.722 INFO:tasks.workunit.client.0.smithi007.stderr: 77850K .......... .......... .......... .......... .......... 57% 22.9M 2s 2024-04-21T08:45:40.723 INFO:tasks.workunit.client.0.smithi007.stderr: 77900K .......... .......... .......... .......... .......... 57% 38.4M 2s 2024-04-21T08:45:40.725 INFO:tasks.workunit.client.0.smithi007.stderr: 77950K .......... .......... .......... .......... .......... 57% 23.9M 2s 2024-04-21T08:45:40.727 INFO:tasks.workunit.client.0.smithi007.stderr: 78000K .......... .......... .......... .......... .......... 57% 56.3M 2s 2024-04-21T08:45:40.728 INFO:tasks.workunit.client.0.smithi007.stderr: 78050K .......... .......... .......... .......... .......... 57% 23.0M 2s 2024-04-21T08:45:40.730 INFO:tasks.workunit.client.0.smithi007.stderr: 78100K .......... .......... .......... .......... .......... 57% 24.3M 2s 2024-04-21T08:45:40.731 INFO:tasks.workunit.client.0.smithi007.stderr: 78150K .......... .......... .......... .......... .......... 57% 38.2M 2s 2024-04-21T08:45:40.733 INFO:tasks.workunit.client.0.smithi007.stderr: 78200K .......... .......... .......... .......... .......... 57% 24.8M 2s 2024-04-21T08:45:40.735 INFO:tasks.workunit.client.0.smithi007.stderr: 78250K .......... .......... .......... .......... .......... 57% 26.9M 2s 2024-04-21T08:45:40.736 INFO:tasks.workunit.client.0.smithi007.stderr: 78300K .......... .......... .......... .......... .......... 57% 40.4M 2s 2024-04-21T08:45:40.739 INFO:tasks.workunit.client.0.smithi007.stderr: 78350K .......... .......... .......... .......... .......... 57% 22.8M 2s 2024-04-21T08:45:40.740 INFO:tasks.workunit.client.0.smithi007.stderr: 78400K .......... .......... .......... .......... .......... 57% 40.0M 2s 2024-04-21T08:45:40.742 INFO:tasks.workunit.client.0.smithi007.stderr: 78450K .......... .......... .......... .......... .......... 57% 22.1M 2s 2024-04-21T08:45:40.744 INFO:tasks.workunit.client.0.smithi007.stderr: 78500K .......... .......... .......... .......... .......... 57% 24.1M 2s 2024-04-21T08:45:40.745 INFO:tasks.workunit.client.0.smithi007.stderr: 78550K .......... .......... .......... .......... .......... 57% 45.1M 2s 2024-04-21T08:45:40.747 INFO:tasks.workunit.client.0.smithi007.stderr: 78600K .......... .......... .......... .......... .......... 57% 24.9M 2s 2024-04-21T08:45:40.748 INFO:tasks.workunit.client.0.smithi007.stderr: 78650K .......... .......... .......... .......... .......... 57% 42.2M 2s 2024-04-21T08:45:40.750 INFO:tasks.workunit.client.0.smithi007.stderr: 78700K .......... .......... .......... .......... .......... 58% 20.7M 2s 2024-04-21T08:45:40.752 INFO:tasks.workunit.client.0.smithi007.stderr: 78750K .......... .......... .......... .......... .......... 58% 25.3M 2s 2024-04-21T08:45:40.753 INFO:tasks.workunit.client.0.smithi007.stderr: 78800K .......... .......... .......... .......... .......... 58% 51.1M 2s 2024-04-21T08:45:40.756 INFO:tasks.workunit.client.0.smithi007.stderr: 78850K .......... .......... .......... .......... .......... 58% 23.3M 2s 2024-04-21T08:45:40.757 INFO:tasks.workunit.client.0.smithi007.stderr: 78900K .......... .......... .......... .......... .......... 58% 37.0M 2s 2024-04-21T08:45:40.759 INFO:tasks.workunit.client.0.smithi007.stderr: 78950K .......... .......... .......... .......... .......... 58% 25.0M 2s 2024-04-21T08:45:40.761 INFO:tasks.workunit.client.0.smithi007.stderr: 79000K .......... .......... .......... .......... .......... 58% 18.9M 2s 2024-04-21T08:45:40.762 INFO:tasks.workunit.client.0.smithi007.stderr: 79050K .......... .......... .......... .......... .......... 58% 55.7M 2s 2024-04-21T08:45:40.764 INFO:tasks.workunit.client.0.smithi007.stderr: 79100K .......... .......... .......... .......... .......... 58% 23.9M 2s 2024-04-21T08:45:40.766 INFO:tasks.workunit.client.0.smithi007.stderr: 79150K .......... .......... .......... .......... .......... 58% 28.7M 2s 2024-04-21T08:45:40.768 INFO:tasks.workunit.client.0.smithi007.stderr: 79200K .......... .......... .......... .......... .......... 58% 22.6M 2s 2024-04-21T08:45:40.770 INFO:tasks.workunit.client.0.smithi007.stderr: 79250K .......... .......... .......... .......... .......... 58% 21.7M 2s 2024-04-21T08:45:40.771 INFO:tasks.workunit.client.0.smithi007.stderr: 79300K .......... .......... .......... .......... .......... 58% 57.1M 2s 2024-04-21T08:45:40.774 INFO:tasks.workunit.client.0.smithi007.stderr: 79350K .......... .......... .......... .......... .......... 58% 19.6M 2s 2024-04-21T08:45:40.775 INFO:tasks.workunit.client.0.smithi007.stderr: 79400K .......... .......... .......... .......... .......... 58% 36.9M 2s 2024-04-21T08:45:40.777 INFO:tasks.workunit.client.0.smithi007.stderr: 79450K .......... .......... .......... .......... .......... 58% 25.1M 2s 2024-04-21T08:45:40.779 INFO:tasks.workunit.client.0.smithi007.stderr: 79500K .......... .......... .......... .......... .......... 58% 24.2M 2s 2024-04-21T08:45:40.780 INFO:tasks.workunit.client.0.smithi007.stderr: 79550K .......... .......... .......... .......... .......... 58% 43.5M 2s 2024-04-21T08:45:40.782 INFO:tasks.workunit.client.0.smithi007.stderr: 79600K .......... .......... .......... .......... .......... 58% 25.0M 2s 2024-04-21T08:45:40.783 INFO:tasks.workunit.client.0.smithi007.stderr: 79650K .......... .......... .......... .......... .......... 58% 39.2M 2s 2024-04-21T08:45:40.785 INFO:tasks.workunit.client.0.smithi007.stderr: 79700K .......... .......... .......... .......... .......... 58% 24.6M 2s 2024-04-21T08:45:40.787 INFO:tasks.workunit.client.0.smithi007.stderr: 79750K .......... .......... .......... .......... .......... 58% 24.7M 2s 2024-04-21T08:45:40.788 INFO:tasks.workunit.client.0.smithi007.stderr: 79800K .......... .......... .......... .......... .......... 58% 44.9M 2s 2024-04-21T08:45:40.790 INFO:tasks.workunit.client.0.smithi007.stderr: 79850K .......... .......... .......... .......... .......... 58% 23.1M 2s 2024-04-21T08:45:40.792 INFO:tasks.workunit.client.0.smithi007.stderr: 79900K .......... .......... .......... .......... .......... 58% 32.6M 2s 2024-04-21T08:45:40.794 INFO:tasks.workunit.client.0.smithi007.stderr: 79950K .......... .......... .......... .......... .......... 58% 25.9M 2s 2024-04-21T08:45:40.797 INFO:tasks.workunit.client.0.smithi007.stderr: 80000K .......... .......... .......... .......... .......... 58% 25.5M 2s 2024-04-21T08:45:40.797 INFO:tasks.workunit.client.0.smithi007.stderr: 80050K .......... .......... .......... .......... .......... 59% 52.5M 2s 2024-04-21T08:45:40.799 INFO:tasks.workunit.client.0.smithi007.stderr: 80100K .......... .......... .......... .......... .......... 59% 23.2M 2s 2024-04-21T08:45:40.800 INFO:tasks.workunit.client.0.smithi007.stderr: 80150K .......... .......... .......... .......... .......... 59% 40.7M 2s 2024-04-21T08:45:40.802 INFO:tasks.workunit.client.0.smithi007.stderr: 80200K .......... .......... .......... .......... .......... 59% 23.6M 2s 2024-04-21T08:45:40.804 INFO:tasks.workunit.client.0.smithi007.stderr: 80250K .......... .......... .......... .......... .......... 59% 24.3M 2s 2024-04-21T08:45:40.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:40 smithi016 ceph-mon[31259]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-21T08:45:40.805 INFO:tasks.workunit.client.0.smithi007.stderr: 80300K .......... .......... .......... .......... .......... 59% 44.6M 2s 2024-04-21T08:45:40.807 INFO:tasks.workunit.client.0.smithi007.stderr: 80350K .......... .......... .......... .......... .......... 59% 25.0M 2s 2024-04-21T08:45:40.808 INFO:tasks.workunit.client.0.smithi007.stderr: 80400K .......... .......... .......... .......... .......... 59% 35.3M 2s 2024-04-21T08:45:40.810 INFO:tasks.workunit.client.0.smithi007.stderr: 80450K .......... .......... .......... .......... .......... 59% 26.2M 2s 2024-04-21T08:45:40.813 INFO:tasks.workunit.client.0.smithi007.stderr: 80500K .......... .......... .......... .......... .......... 59% 22.8M 2s 2024-04-21T08:45:40.813 INFO:tasks.workunit.client.0.smithi007.stderr: 80550K .......... .......... .......... .......... .......... 59% 50.9M 2s 2024-04-21T08:45:40.815 INFO:tasks.workunit.client.0.smithi007.stderr: 80600K .......... .......... .......... .......... .......... 59% 23.3M 2s 2024-04-21T08:45:40.817 INFO:tasks.workunit.client.0.smithi007.stderr: 80650K .......... .......... .......... .......... .......... 59% 35.3M 2s 2024-04-21T08:45:40.819 INFO:tasks.workunit.client.0.smithi007.stderr: 80700K .......... .......... .......... .......... .......... 59% 23.9M 2s 2024-04-21T08:45:40.821 INFO:tasks.workunit.client.0.smithi007.stderr: 80750K .......... .......... .......... .......... .......... 59% 22.5M 2s 2024-04-21T08:45:40.822 INFO:tasks.workunit.client.0.smithi007.stderr: 80800K .......... .......... .......... .......... .......... 59% 40.0M 2s 2024-04-21T08:45:40.824 INFO:tasks.workunit.client.0.smithi007.stderr: 80850K .......... .......... .......... .......... .......... 59% 24.6M 2s 2024-04-21T08:45:40.826 INFO:tasks.workunit.client.0.smithi007.stderr: 80900K .......... .......... .......... .......... .......... 59% 36.9M 2s 2024-04-21T08:45:40.828 INFO:tasks.workunit.client.0.smithi007.stderr: 80950K .......... .......... .......... .......... .......... 59% 19.9M 2s 2024-04-21T08:45:40.830 INFO:tasks.workunit.client.0.smithi007.stderr: 81000K .......... .......... .......... .......... .......... 59% 31.5M 2s 2024-04-21T08:45:40.831 INFO:tasks.workunit.client.0.smithi007.stderr: 81050K .......... .......... .......... .......... .......... 59% 34.1M 2s 2024-04-21T08:45:40.833 INFO:tasks.workunit.client.0.smithi007.stderr: 81100K .......... .......... .......... .......... .......... 59% 20.4M 2s 2024-04-21T08:45:40.836 INFO:tasks.workunit.client.0.smithi007.stderr: 81150K .......... .......... .......... .......... .......... 59% 27.0M 2s 2024-04-21T08:45:40.838 INFO:tasks.workunit.client.0.smithi007.stderr: 81200K .......... .......... .......... .......... .......... 59% 20.5M 2s 2024-04-21T08:45:40.840 INFO:tasks.workunit.client.0.smithi007.stderr: 81250K .......... .......... .......... .......... .......... 59% 25.7M 2s 2024-04-21T08:45:40.840 INFO:tasks.workunit.client.0.smithi007.stderr: 81300K .......... .......... .......... .......... .......... 59% 51.8M 2s 2024-04-21T08:45:40.843 INFO:tasks.workunit.client.0.smithi007.stderr: 81350K .......... .......... .......... .......... .......... 59% 22.2M 2s 2024-04-21T08:45:40.845 INFO:tasks.workunit.client.0.smithi007.stderr: 81400K .......... .......... .......... .......... .......... 59% 25.3M 2s 2024-04-21T08:45:40.846 INFO:tasks.workunit.client.0.smithi007.stderr: 81450K .......... .......... .......... .......... .......... 60% 38.5M 2s 2024-04-21T08:45:40.849 INFO:tasks.workunit.client.0.smithi007.stderr: 81500K .......... .......... .......... .......... .......... 60% 24.9M 2s 2024-04-21T08:45:40.849 INFO:tasks.workunit.client.0.smithi007.stderr: 81550K .......... .......... .......... .......... .......... 60% 47.7M 2s 2024-04-21T08:45:40.851 INFO:tasks.workunit.client.0.smithi007.stderr: 81600K .......... .......... .......... .......... .......... 60% 23.2M 2s 2024-04-21T08:45:40.853 INFO:tasks.workunit.client.0.smithi007.stderr: 81650K .......... .......... .......... .......... .......... 60% 35.4M 2s 2024-04-21T08:45:40.854 INFO:tasks.workunit.client.0.smithi007.stderr: 81700K .......... .......... .......... .......... .......... 60% 25.9M 2s 2024-04-21T08:45:40.856 INFO:tasks.workunit.client.0.smithi007.stderr: 81750K .......... .......... .......... .......... .......... 60% 31.5M 2s 2024-04-21T08:45:40.858 INFO:tasks.workunit.client.0.smithi007.stderr: 81800K .......... .......... .......... .......... .......... 60% 34.3M 2s 2024-04-21T08:45:40.859 INFO:tasks.workunit.client.0.smithi007.stderr: 81850K .......... .......... .......... .......... .......... 60% 21.9M 2s 2024-04-21T08:45:40.860 INFO:tasks.workunit.client.0.smithi007.stderr: 81900K .......... .......... .......... .......... .......... 60% 46.3M 2s 2024-04-21T08:45:40.863 INFO:tasks.workunit.client.0.smithi007.stderr: 81950K .......... .......... .......... .......... .......... 60% 19.4M 2s 2024-04-21T08:45:40.865 INFO:tasks.workunit.client.0.smithi007.stderr: 82000K .......... .......... .......... .......... .......... 60% 24.2M 2s 2024-04-21T08:45:40.866 INFO:tasks.workunit.client.0.smithi007.stderr: 82050K .......... .......... .......... .......... .......... 60% 30.9M 2s 2024-04-21T08:45:40.868 INFO:tasks.workunit.client.0.smithi007.stderr: 82100K .......... .......... .......... .......... .......... 60% 25.6M 2s 2024-04-21T08:45:40.871 INFO:tasks.workunit.client.0.smithi007.stderr: 82150K .......... .......... .......... .......... .......... 60% 22.0M 2s 2024-04-21T08:45:40.872 INFO:tasks.workunit.client.0.smithi007.stderr: 82200K .......... .......... .......... .......... .......... 60% 38.2M 2s 2024-04-21T08:45:40.874 INFO:tasks.workunit.client.0.smithi007.stderr: 82250K .......... .......... .......... .......... .......... 60% 24.0M 2s 2024-04-21T08:45:40.875 INFO:tasks.workunit.client.0.smithi007.stderr: 82300K .......... .......... .......... .......... .......... 60% 50.7M 2s 2024-04-21T08:45:40.877 INFO:tasks.workunit.client.0.smithi007.stderr: 82350K .......... .......... .......... .......... .......... 60% 24.9M 2s 2024-04-21T08:45:40.879 INFO:tasks.workunit.client.0.smithi007.stderr: 82400K .......... .......... .......... .......... .......... 60% 23.2M 2s 2024-04-21T08:45:40.880 INFO:tasks.workunit.client.0.smithi007.stderr: 82450K .......... .......... .......... .......... .......... 60% 38.3M 2s 2024-04-21T08:45:40.883 INFO:tasks.workunit.client.0.smithi007.stderr: 82500K .......... .......... .......... .......... .......... 60% 24.7M 2s 2024-04-21T08:45:40.883 INFO:tasks.workunit.client.0.smithi007.stderr: 82550K .......... .......... .......... .......... .......... 60% 44.2M 2s 2024-04-21T08:45:40.886 INFO:tasks.workunit.client.0.smithi007.stderr: 82600K .......... .......... .......... .......... .......... 60% 24.7M 2s 2024-04-21T08:45:40.887 INFO:tasks.workunit.client.0.smithi007.stderr: 82650K .......... .......... .......... .......... .......... 60% 30.7M 2s 2024-04-21T08:45:40.889 INFO:tasks.workunit.client.0.smithi007.stderr: 82700K .......... .......... .......... .......... .......... 60% 27.0M 2s 2024-04-21T08:45:40.891 INFO:tasks.workunit.client.0.smithi007.stderr: 82750K .......... .......... .......... .......... .......... 60% 24.3M 2s 2024-04-21T08:45:40.892 INFO:tasks.workunit.client.0.smithi007.stderr: 82800K .......... .......... .......... .......... .......... 61% 59.9M 2s 2024-04-21T08:45:40.894 INFO:tasks.workunit.client.0.smithi007.stderr: 82850K .......... .......... .......... .......... .......... 61% 19.1M 2s 2024-04-21T08:45:40.897 INFO:tasks.workunit.client.0.smithi007.stderr: 82900K .......... .......... .......... .......... .......... 61% 21.1M 2s 2024-04-21T08:45:40.898 INFO:tasks.workunit.client.0.smithi007.stderr: 82950K .......... .......... .......... .......... .......... 61% 35.8M 2s 2024-04-21T08:45:40.901 INFO:tasks.workunit.client.0.smithi007.stderr: 83000K .......... .......... .......... .......... .......... 61% 18.2M 2s 2024-04-21T08:45:40.902 INFO:tasks.workunit.client.0.smithi007.stderr: 83050K .......... .......... .......... .......... .......... 61% 54.4M 2s 2024-04-21T08:45:40.903 INFO:tasks.workunit.client.0.smithi007.stderr: 83100K .......... .......... .......... .......... .......... 61% 29.8M 2s 2024-04-21T08:45:40.906 INFO:tasks.workunit.client.0.smithi007.stderr: 83150K .......... .......... .......... .......... .......... 61% 17.2M 2s 2024-04-21T08:45:40.907 INFO:tasks.workunit.client.0.smithi007.stderr: 83200K .......... .......... .......... .......... .......... 61% 36.0M 2s 2024-04-21T08:45:40.910 INFO:tasks.workunit.client.0.smithi007.stderr: 83250K .......... .......... .......... .......... .......... 61% 19.4M 2s 2024-04-21T08:45:40.911 INFO:tasks.workunit.client.0.smithi007.stderr: 83300K .......... .......... .......... .......... .......... 61% 110M 2s 2024-04-21T08:45:40.912 INFO:tasks.workunit.client.0.smithi007.stderr: 83350K .......... .......... .......... .......... .......... 61% 28.4M 2s 2024-04-21T08:45:40.914 INFO:tasks.workunit.client.0.smithi007.stderr: 83400K .......... .......... .......... .......... .......... 61% 23.1M 2s 2024-04-21T08:45:40.916 INFO:tasks.workunit.client.0.smithi007.stderr: 83450K .......... .......... .......... .......... .......... 61% 41.2M 2s 2024-04-21T08:45:40.917 INFO:tasks.workunit.client.0.smithi007.stderr: 83500K .......... .......... .......... .......... .......... 61% 24.4M 2s 2024-04-21T08:45:40.919 INFO:tasks.workunit.client.0.smithi007.stderr: 83550K .......... .......... .......... .......... .......... 61% 25.4M 2s 2024-04-21T08:45:40.920 INFO:tasks.workunit.client.0.smithi007.stderr: 83600K .......... .......... .......... .......... .......... 61% 52.2M 2s 2024-04-21T08:45:40.922 INFO:tasks.workunit.client.0.smithi007.stderr: 83650K .......... .......... .......... .......... .......... 61% 23.5M 2s 2024-04-21T08:45:40.923 INFO:tasks.workunit.client.0.smithi007.stderr: 83700K .......... .......... .......... .......... .......... 61% 42.5M 2s 2024-04-21T08:45:40.926 INFO:tasks.workunit.client.0.smithi007.stderr: 83750K .......... .......... .......... .......... .......... 61% 20.8M 2s 2024-04-21T08:45:40.928 INFO:tasks.workunit.client.0.smithi007.stderr: 83800K .......... .......... .......... .......... .......... 61% 23.7M 2s 2024-04-21T08:45:40.929 INFO:tasks.workunit.client.0.smithi007.stderr: 83850K .......... .......... .......... .......... .......... 61% 51.0M 2s 2024-04-21T08:45:40.931 INFO:tasks.workunit.client.0.smithi007.stderr: 83900K .......... .......... .......... .......... .......... 61% 23.6M 2s 2024-04-21T08:45:40.933 INFO:tasks.workunit.client.0.smithi007.stderr: 83950K .......... .......... .......... .......... .......... 61% 41.0M 2s 2024-04-21T08:45:40.935 INFO:tasks.workunit.client.0.smithi007.stderr: 84000K .......... .......... .......... .......... .......... 61% 23.1M 2s 2024-04-21T08:45:40.937 INFO:tasks.workunit.client.0.smithi007.stderr: 84050K .......... .......... .......... .......... .......... 61% 20.6M 2s 2024-04-21T08:45:40.938 INFO:tasks.workunit.client.0.smithi007.stderr: 84100K .......... .......... .......... .......... .......... 61% 42.7M 2s 2024-04-21T08:45:40.940 INFO:tasks.workunit.client.0.smithi007.stderr: 84150K .......... .......... .......... .......... .......... 62% 22.4M 2s 2024-04-21T08:45:40.941 INFO:tasks.workunit.client.0.smithi007.stderr: 84200K .......... .......... .......... .......... .......... 62% 43.2M 2s 2024-04-21T08:45:40.943 INFO:tasks.workunit.client.0.smithi007.stderr: 84250K .......... .......... .......... .......... .......... 62% 24.0M 2s 2024-04-21T08:45:40.945 INFO:tasks.workunit.client.0.smithi007.stderr: 84300K .......... .......... .......... .......... .......... 62% 23.5M 2s 2024-04-21T08:45:40.946 INFO:tasks.workunit.client.0.smithi007.stderr: 84350K .......... .......... .......... .......... .......... 62% 44.4M 2s 2024-04-21T08:45:40.948 INFO:tasks.workunit.client.0.smithi007.stderr: 84400K .......... .......... .......... .......... .......... 62% 30.7M 2s 2024-04-21T08:45:40.949 INFO:tasks.workunit.client.0.smithi007.stderr: 84450K .......... .......... .......... .......... .......... 62% 31.0M 2s 2024-04-21T08:45:40.952 INFO:tasks.workunit.client.0.smithi007.stderr: 84500K .......... .......... .......... .......... .......... 62% 21.8M 2s 2024-04-21T08:45:40.954 INFO:tasks.workunit.client.0.smithi007.stderr: 84550K .......... .......... .......... .......... .......... 62% 24.1M 2s 2024-04-21T08:45:40.954 INFO:tasks.workunit.client.0.smithi007.stderr: 84600K .......... .......... .......... .......... .......... 62% 47.2M 2s 2024-04-21T08:45:40.957 INFO:tasks.workunit.client.0.smithi007.stderr: 84650K .......... .......... .......... .......... .......... 62% 23.8M 2s 2024-04-21T08:45:40.958 INFO:tasks.workunit.client.0.smithi007.stderr: 84700K .......... .......... .......... .......... .......... 62% 42.9M 2s 2024-04-21T08:45:40.960 INFO:tasks.workunit.client.0.smithi007.stderr: 84750K .......... .......... .......... .......... .......... 62% 23.3M 2s 2024-04-21T08:45:40.962 INFO:tasks.workunit.client.0.smithi007.stderr: 84800K .......... .......... .......... .......... .......... 62% 21.6M 2s 2024-04-21T08:45:40.963 INFO:tasks.workunit.client.0.smithi007.stderr: 84850K .......... .......... .......... .......... .......... 62% 49.6M 2s 2024-04-21T08:45:40.965 INFO:tasks.workunit.client.0.smithi007.stderr: 84900K .......... .......... .......... .......... .......... 62% 23.6M 2s 2024-04-21T08:45:40.966 INFO:tasks.workunit.client.0.smithi007.stderr: 84950K .......... .......... .......... .......... .......... 62% 39.9M 2s 2024-04-21T08:45:40.969 INFO:tasks.workunit.client.0.smithi007.stderr: 85000K .......... .......... .......... .......... .......... 62% 24.7M 2s 2024-04-21T08:45:40.971 INFO:tasks.workunit.client.0.smithi007.stderr: 85050K .......... .......... .......... .......... .......... 62% 22.8M 2s 2024-04-21T08:45:40.972 INFO:tasks.workunit.client.0.smithi007.stderr: 85100K .......... .......... .......... .......... .......... 62% 43.5M 2s 2024-04-21T08:45:40.973 INFO:tasks.workunit.client.0.smithi007.stderr: 85150K .......... .......... .......... .......... .......... 62% 33.0M 2s 2024-04-21T08:45:40.976 INFO:tasks.workunit.client.0.smithi007.stderr: 85200K .......... .......... .......... .......... .......... 62% 27.1M 2s 2024-04-21T08:45:40.977 INFO:tasks.workunit.client.0.smithi007.stderr: 85250K .......... .......... .......... .......... .......... 62% 23.8M 2s 2024-04-21T08:45:40.979 INFO:tasks.workunit.client.0.smithi007.stderr: 85300K .......... .......... .......... .......... .......... 62% 23.3M 2s 2024-04-21T08:45:40.980 INFO:tasks.workunit.client.0.smithi007.stderr: 85350K .......... .......... .......... .......... .......... 62% 47.9M 2s 2024-04-21T08:45:40.982 INFO:tasks.workunit.client.0.smithi007.stderr: 85400K .......... .......... .......... .......... .......... 62% 24.9M 2s 2024-04-21T08:45:40.984 INFO:tasks.workunit.client.0.smithi007.stderr: 85450K .......... .......... .......... .......... .......... 62% 23.9M 2s 2024-04-21T08:45:40.985 INFO:tasks.workunit.client.0.smithi007.stderr: 85500K .......... .......... .......... .......... .......... 63% 38.9M 2s 2024-04-21T08:45:40.988 INFO:tasks.workunit.client.0.smithi007.stderr: 85550K .......... .......... .......... .......... .......... 63% 23.2M 2s 2024-04-21T08:45:40.988 INFO:tasks.workunit.client.0.smithi007.stderr: 85600K .......... .......... .......... .......... .......... 63% 48.5M 2s 2024-04-21T08:45:40.990 INFO:tasks.workunit.client.0.smithi007.stderr: 85650K .......... .......... .......... .......... .......... 63% 23.9M 2s 2024-04-21T08:45:40.992 INFO:tasks.workunit.client.0.smithi007.stderr: 85700K .......... .......... .......... .......... .......... 63% 39.3M 2s 2024-04-21T08:45:40.994 INFO:tasks.workunit.client.0.smithi007.stderr: 85750K .......... .......... .......... .......... .......... 63% 24.5M 2s 2024-04-21T08:45:40.996 INFO:tasks.workunit.client.0.smithi007.stderr: 85800K .......... .......... .......... .......... .......... 63% 20.2M 2s 2024-04-21T08:45:40.997 INFO:tasks.workunit.client.0.smithi007.stderr: 85850K .......... .......... .......... .......... .......... 63% 52.9M 2s 2024-04-21T08:45:40.999 INFO:tasks.workunit.client.0.smithi007.stderr: 85900K .......... .......... .......... .......... .......... 63% 21.6M 2s 2024-04-21T08:45:41.001 INFO:tasks.workunit.client.0.smithi007.stderr: 85950K .......... .......... .......... .......... .......... 63% 28.4M 2s 2024-04-21T08:45:41.002 INFO:tasks.workunit.client.0.smithi007.stderr: 86000K .......... .......... .......... .......... .......... 63% 37.0M 2s 2024-04-21T08:45:41.004 INFO:tasks.workunit.client.0.smithi007.stderr: 86050K .......... .......... .......... .......... .......... 63% 23.7M 2s 2024-04-21T08:45:41.006 INFO:tasks.workunit.client.0.smithi007.stderr: 86100K .......... .......... .......... .......... .......... 63% 47.6M 2s 2024-04-21T08:45:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.007+0000 7f8a40a31640 1 -- 172.21.15.7:0/3143680644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a3c10e8f0 msgr2=0x7f8a3c110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.007+0000 7f8a40a31640 1 --2- 172.21.15.7:0/3143680644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a3c10e8f0 0x7f8a3c110ce0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009ca0 tx=0x7f8a3402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.007+0000 7f8a40a31640 1 -- 172.21.15.7:0/3143680644 shutdown_connections 2024-04-21T08:45:41.008 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.007+0000 7f8a40a31640 1 --2- 172.21.15.7:0/3143680644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a3c10e8f0 0x7f8a3c110ce0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.008 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.007+0000 7f8a40a31640 1 --2- 172.21.15.7:0/3143680644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a3c101660 0x7f8a3c10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.008 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.007+0000 7f8a40a31640 1 --2- 172.21.15.7:0/3143680644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a3c100d40 0x7f8a3c101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.008 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.007+0000 7f8a40a31640 1 -- 172.21.15.7:0/3143680644 >> 172.21.15.7:0/3143680644 conn(0x7f8a3c077f70 msgr2=0x7f8a3c078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:41.008 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.007+0000 7f8a40a31640 1 -- 172.21.15.7:0/3143680644 shutdown_connections 2024-04-21T08:45:41.008 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.007+0000 7f8a40a31640 1 -- 172.21.15.7:0/3143680644 wait complete. 2024-04-21T08:45:41.009 INFO:tasks.workunit.client.0.smithi007.stderr: 86150K .......... .......... .......... .......... .......... 63% 25.7M 2s 2024-04-21T08:45:41.010 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.007+0000 7f8a40a31640 1 Processor -- start 2024-04-21T08:45:41.010 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a40a31640 1 -- start start 2024-04-21T08:45:41.010 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a40a31640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a3c100d40 0x7f8a3c1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a40a31640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a3c101660 0x7f8a3c1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a40a31640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a3c10e8f0 0x7f8a3c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a40a31640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1143d0 con 0x7f8a3c100d40 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a40a31640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c114250 con 0x7f8a3c101660 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a40a31640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c114550 con 0x7f8a3c10e8f0 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a3a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a3c100d40 0x7f8a3c1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a3a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a3c100d40 0x7f8a3c1a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35510/0 (socket says 172.21.15.7:35510) 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a3a59c640 1 -- 172.21.15.7:0/3031575441 learned_addr learned my addr 172.21.15.7:0/3031575441 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a39d9b640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a3c101660 0x7f8a3c1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a3ad9d640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a3c10e8f0 0x7f8a3c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a3a59c640 1 -- 172.21.15.7:0/3031575441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a3c10e8f0 msgr2=0x7f8a3c19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a3c10e8f0 0x7f8a3c19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a3a59c640 1 -- 172.21.15.7:0/3031575441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a3c101660 msgr2=0x7f8a3c1a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:41.012 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a3c101660 0x7f8a3c1a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.012 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a3a59c640 1 -- 172.21.15.7:0/3031575441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3c19aff0 con 0x7f8a3c100d40 2024-04-21T08:45:41.012 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a3ad9d640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a3c10e8f0 0x7f8a3c19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:41.012 INFO:tasks.workunit.client.0.smithi007.stderr: 86200K .......... .......... .......... .......... .......... 63% 37.7M 2s 2024-04-21T08:45:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.008+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a3c100d40 0x7f8a3c1a0670 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c00e9f0 tx=0x7f8a2c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.009+0000 7f8a337fe640 1 -- 172.21.15.7:0/3031575441 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c00cd80 con 0x7f8a3c100d40 2024-04-21T08:45:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.009+0000 7f8a40a31640 1 -- 172.21.15.7:0/3031575441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a3c19b2e0 con 0x7f8a3c100d40 2024-04-21T08:45:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.009+0000 7f8a337fe640 1 -- 172.21.15.7:0/3031575441 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a2c0054d0 con 0x7f8a3c100d40 2024-04-21T08:45:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.009+0000 7f8a337fe640 1 -- 172.21.15.7:0/3031575441 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c010690 con 0x7f8a3c100d40 2024-04-21T08:45:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.009+0000 7f8a40a31640 1 -- 172.21.15.7:0/3031575441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a3c1a75f0 con 0x7f8a3c100d40 2024-04-21T08:45:41.015 INFO:tasks.workunit.client.0.smithi007.stderr: 86250K .......... .......... .......... .......... .......... 63% 23.1M 2s 2024-04-21T08:45:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.011+0000 7f8a337fe640 1 -- 172.21.15.7:0/3031575441 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f8a2c004e90 con 0x7f8a3c100d40 2024-04-21T08:45:41.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.011+0000 7f8a43626740 1 Processor -- start 2024-04-21T08:45:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.011+0000 7f8a43626740 1 -- start start 2024-04-21T08:45:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a337fe640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a280a3e30 0x7f8a280a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.017 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a43626740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d60620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a43626740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d60b60 0x5632b1d60fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a43626740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d6c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a337fe640 1 -- 172.21.15.7:0/3031575441 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f8a2c0d1340 con 0x7f8a3c100d40 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3ad9d640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a280a3e30 0x7f8a280a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a43626740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5632b1cc58d0 con 0x5632b1d618d0 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a43626740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5632b1c62e80 con 0x5632b1d6a0e0 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a43626740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5632b1cc1890 con 0x5632b1d60b60 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d60620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d60620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35512/0 (socket says 172.21.15.7:35512) 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3a59c640 1 -- 172.21.15.7:0/517481141 learned_addr learned my addr 172.21.15.7:0/517481141 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a39d9b640 1 --2- 172.21.15.7:0/517481141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d60b60 0x5632b1d60fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3ad9d640 1 --2- 172.21.15.7:0/517481141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d6c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3a59c640 1 -- 172.21.15.7:0/517481141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d60b60 msgr2=0x5632b1d60fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/517481141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d60b60 0x5632b1d60fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3a59c640 1 -- 172.21.15.7:0/517481141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 msgr2=0x5632b1d6c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/517481141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d6c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3a59c640 1 -- 172.21.15.7:0/517481141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632b1c84320 con 0x5632b1d618d0 2024-04-21T08:45:41.019 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3ad9d640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a280a3e30 0x7f8a280a62f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8a3402c700 tx=0x7f8a3402cf70 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:41.019 INFO:tasks.workunit.client.0.smithi007.stderr: 86300K .......... .......... .......... .......... .......... 63% 25.3M 2s 2024-04-21T08:45:41.020 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.012+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/517481141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d60620 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0ceca0 tx=0x7f8a2c09c600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c1db2cb5326a69 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:41.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a30ff9640 1 -- 172.21.15.7:0/517481141 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c0bbe60 con 0x5632b1d618d0 2024-04-21T08:45:41.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a30ff9640 1 -- 172.21.15.7:0/517481141 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a2c0d1160 con 0x5632b1d618d0 2024-04-21T08:45:41.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a30ff9640 1 -- 172.21.15.7:0/517481141 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c099310 con 0x5632b1d618d0 2024-04-21T08:45:41.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a43626740 1 -- 172.21.15.7:0/517481141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 msgr2=0x5632b1d60620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:41.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a43626740 1 --2- 172.21.15.7:0/517481141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d60620 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0ceca0 tx=0x7f8a2c09c600 comp rx=0 tx=0).stop 2024-04-21T08:45:41.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a43626740 1 -- 172.21.15.7:0/517481141 shutdown_connections 2024-04-21T08:45:41.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a43626740 1 --2- 172.21.15.7:0/517481141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d6c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a43626740 1 --2- 172.21.15.7:0/517481141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d60b60 0x5632b1d60fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a43626740 1 --2- 172.21.15.7:0/517481141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d60620 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a43626740 1 -- 172.21.15.7:0/517481141 >> 172.21.15.7:0/517481141 conn(0x5632b1d27c90 msgr2=0x5632b1d5b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:41.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a43626740 1 -- 172.21.15.7:0/517481141 shutdown_connections 2024-04-21T08:45:41.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.013+0000 7f8a43626740 1 -- 172.21.15.7:0/517481141 wait complete. 2024-04-21T08:45:41.023 INFO:tasks.workunit.client.0.smithi007.stderr: 86350K .......... .......... .......... .......... .......... 63% 43.8M 2s 2024-04-21T08:45:41.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a43626740 1 -- start start 2024-04-21T08:45:41.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a43626740 1 Processor -- start 2024-04-21T08:45:41.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a43626740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d69a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a43626740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d618d0 0x5632b1d66e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a43626740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d674b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a43626740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5632b1d6ce10 con 0x5632b1d60b60 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a43626740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5632b1c4deb0 con 0x5632b1d6a0e0 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a43626740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5632b1d6cf90 con 0x5632b1d618d0 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a43626740 10 client.0 _ll_register_callbacks cb 0x5632b1c7ab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a3a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d69a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a3a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d69a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35522/0 (socket says 172.21.15.7:35522) 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a3a59c640 1 -- 172.21.15.7:0/2662087369 learned_addr learned my addr 172.21.15.7:0/2662087369 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a39d9b640 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d618d0 0x5632b1d66e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a3a59c640 1 -- 172.21.15.7:0/2662087369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d618d0 msgr2=0x5632b1d66e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d618d0 0x5632b1d66e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a3a59c640 1 -- 172.21.15.7:0/2662087369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 msgr2=0x5632b1d674b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a3ad9d640 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d674b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d674b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.014+0000 7f8a3a59c640 1 -- 172.21.15.7:0/2662087369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632b19d8210 con 0x5632b1d60b60 2024-04-21T08:45:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d69a80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c09c9a0 tx=0x7f8a2c09d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78841e6871f6db3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a21ffb640 1 -- 172.21.15.7:0/2662087369 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c0bbe60 con 0x5632b1d60b60 2024-04-21T08:45:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a21ffb640 1 -- 172.21.15.7:0/2662087369 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f8a2c0046c0 con 0x5632b1d60b60 2024-04-21T08:45:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a43626740 1 -- 172.21.15.7:0/2662087369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a2c0d1160 con 0x5632b1d60b60 2024-04-21T08:45:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a21ffb640 1 -- 172.21.15.7:0/2662087369 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8a2c0a1050 con 0x5632b1d60b60 2024-04-21T08:45:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a43626740 10 client.15954 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a43626740 1 -- 172.21.15.7:0/2662087369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5632b1aa4620 con 0x5632b1d60b60 2024-04-21T08:45:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a21ffb640 1 -- 172.21.15.7:0/2662087369 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a2c0d1160 con 0x5632b1d60b60 2024-04-21T08:45:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a21ffb640 1 -- 172.21.15.7:0/2662087369 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8a2c061ce0 con 0x5632b1d60b60 2024-04-21T08:45:41.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a43626740 10 client.15954 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a43626740 10 client.15954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a43626740 20 client.15954 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a43626740 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5632b1d79fe0 0x5632b1d7c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a43626740 4 client.15954 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:41.030 INFO:tasks.workunit.client.0.smithi007.stderr: 86400K .......... .......... .......... .......... .......... 63% 24.9M 2s 2024-04-21T08:45:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.015+0000 7f8a43626740 1 -- 172.21.15.7:0/2662087369 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632b1cd9880 con 0x5632b1d79fe0 2024-04-21T08:45:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.016+0000 7f8a39d9b640 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5632b1d79fe0 0x5632b1d7c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.031 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.016+0000 7f8a39d9b640 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5632b1d79fe0 0x5632b1d7c3d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fca3d0eadb467a36 server_cookie=b8e682f167a63929 in_seq=0 out_seq=0 2024-04-21T08:45:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.016+0000 7f8a21ffb640 10 client.15954 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:41.032 INFO:tasks.workunit.client.0.smithi007.stderr: 86450K .......... .......... .......... .......... .......... 63% 39.1M 2s 2024-04-21T08:45:41.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a21ffb640 1 -- 172.21.15.7:0/2662087369 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632b1cd9880 con 0x5632b1d79fe0 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a21ffb640 10 client.15954 handle_command_reply: tid=0 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 1 client.15954 shutdown 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 1 -- 172.21.15.7:0/2662087369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 msgr2=0x5632b1d69a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d69a80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c09c9a0 tx=0x7f8a2c09d460 comp rx=0 tx=0).stop 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 1 -- 172.21.15.7:0/2662087369 shutdown_connections 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5632b1d79fe0 0x5632b1d7c3d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d674b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d618d0 0x5632b1d66e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a21ffb640 0 client.15954 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 1 --2- 172.21.15.7:0/2662087369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d69a80 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 1 -- 172.21.15.7:0/2662087369 >> 172.21.15.7:0/2662087369 conn(0x5632b1d27c90 msgr2=0x5632b1d5d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 1 -- 172.21.15.7:0/2662087369 shutdown_connections 2024-04-21T08:45:41.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 1 -- 172.21.15.7:0/2662087369 wait complete. 2024-04-21T08:45:41.035 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.017+0000 7f8a43626740 20 client.15954 trim_cache size 0 max 16384 2024-04-21T08:45:41.043 INFO:tasks.workunit.client.0.smithi007.stderr: 86500K .......... .......... .......... .......... .......... 63% 23.8M 2s 2024-04-21T08:45:41.043 INFO:tasks.workunit.client.0.smithi007.stderr: 86550K .......... .......... .......... .......... .......... 63% 22.3M 2s 2024-04-21T08:45:41.043 INFO:tasks.workunit.client.0.smithi007.stderr: 86600K .......... .......... .......... .......... .......... 63% 54.7M 2s 2024-04-21T08:45:41.043 INFO:tasks.workunit.client.0.smithi007.stderr: 86650K .......... .......... .......... .......... .......... 63% 23.6M 2s 2024-04-21T08:45:41.044 INFO:tasks.workunit.client.0.smithi007.stderr: 86700K .......... .......... .......... .......... .......... 63% 40.3M 2s 2024-04-21T08:45:41.044 INFO:tasks.workunit.client.0.smithi007.stderr: 86750K .......... .......... .......... .......... .......... 63% 23.8M 2s 2024-04-21T08:45:41.044 INFO:tasks.workunit.client.0.smithi007.stderr: 86800K .......... .......... .......... .......... .......... 63% 23.3M 2s 2024-04-21T08:45:41.044 INFO:tasks.workunit.client.0.smithi007.stderr: 86850K .......... .......... .......... .......... .......... 64% 50.6M 2s 2024-04-21T08:45:41.044 INFO:tasks.workunit.client.0.smithi007.stderr: 86900K .......... .......... .......... .......... .......... 64% 24.0M 2s 2024-04-21T08:45:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a43626740 1 Processor -- start 2024-04-21T08:45:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a43626740 1 -- start start 2024-04-21T08:45:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a43626740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d60b60 0x5632b1d695b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a43626740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d7f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a43626740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d7f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a43626740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1af660 con 0x5632b1d618d0 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a43626740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1b1ee0 con 0x5632b1d6a0e0 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a43626740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1afcd0 con 0x5632b1d60b60 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a39d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d7f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a39d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d7f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35538/0 (socket says 172.21.15.7:35538) 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a39d9b640 1 -- 172.21.15.7:0/4027253510 learned_addr learned my addr 172.21.15.7:0/4027253510 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a3ad9d640 1 --2- 172.21.15.7:0/4027253510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d7f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/4027253510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d60b60 0x5632b1d695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a39d9b640 1 -- 172.21.15.7:0/4027253510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d60b60 msgr2=0x5632b1d695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a39d9b640 1 --2- 172.21.15.7:0/4027253510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d60b60 0x5632b1d695b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a39d9b640 1 -- 172.21.15.7:0/4027253510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 msgr2=0x5632b1d7f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a39d9b640 1 --2- 172.21.15.7:0/4027253510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d7f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a39d9b640 1 -- 172.21.15.7:0/4027253510 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632b1d5fe30 con 0x5632b1d618d0 2024-04-21T08:45:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.033+0000 7f8a39d9b640 1 --2- 172.21.15.7:0/4027253510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d7f3f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x5632b1d6d000 tx=0x7f8a24005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc6617ebf417bb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a23fff640 1 -- 172.21.15.7:0/4027253510 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24025070 con 0x5632b1d618d0 2024-04-21T08:45:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a23fff640 1 -- 172.21.15.7:0/4027253510 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a24003d80 con 0x5632b1d618d0 2024-04-21T08:45:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a23fff640 1 -- 172.21.15.7:0/4027253510 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24010950 con 0x5632b1d618d0 2024-04-21T08:45:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a43626740 1 -- 172.21.15.7:0/4027253510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 msgr2=0x5632b1d7f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a43626740 1 --2- 172.21.15.7:0/4027253510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d7f3f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x5632b1d6d000 tx=0x7f8a24005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:41.049 INFO:tasks.workunit.client.0.smithi007.stderr: 86950K .......... .......... .......... .......... .......... 64% 37.5M 2s 2024-04-21T08:45:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a43626740 1 -- 172.21.15.7:0/4027253510 shutdown_connections 2024-04-21T08:45:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a43626740 1 --2- 172.21.15.7:0/4027253510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d6a0e0 0x5632b1d7f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a43626740 1 --2- 172.21.15.7:0/4027253510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d618d0 0x5632b1d7f3f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a43626740 1 --2- 172.21.15.7:0/4027253510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d60b60 0x5632b1d695b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a43626740 1 -- 172.21.15.7:0/4027253510 >> 172.21.15.7:0/4027253510 conn(0x5632b1d27c90 msgr2=0x5632b1d663c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a43626740 1 -- 172.21.15.7:0/4027253510 shutdown_connections 2024-04-21T08:45:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.034+0000 7f8a43626740 1 -- 172.21.15.7:0/4027253510 wait complete. 2024-04-21T08:45:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a43626740 1 -- start start 2024-04-21T08:45:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a43626740 1 Processor -- start 2024-04-21T08:45:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a43626740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d7f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a43626740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d618d0 0x5632b1d7cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a43626740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d6a0e0 0x5632b1d7d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a43626740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1b03a0 con 0x5632b1d60b60 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a43626740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1b1810 con 0x5632b1d618d0 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a43626740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c1b0a70 con 0x5632b1d6a0e0 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a43626740 10 client.0 _ll_register_callbacks cb 0x7f8a34047070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a3a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d7f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a3a59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d7f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35544/0 (socket says 172.21.15.7:35544) 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a3a59c640 1 -- 172.21.15.7:0/3688974269 learned_addr learned my addr 172.21.15.7:0/3688974269 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a3a59c640 1 -- 172.21.15.7:0/3688974269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d6a0e0 msgr2=0x5632b1d7d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/3688974269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d6a0e0 0x5632b1d7d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a3a59c640 1 -- 172.21.15.7:0/3688974269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d618d0 msgr2=0x5632b1d7cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/3688974269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d618d0 0x5632b1d7cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.035+0000 7f8a3a59c640 1 -- 172.21.15.7:0/3688974269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632b1d56f30 con 0x5632b1d60b60 2024-04-21T08:45:41.054 INFO:tasks.workunit.client.0.smithi007.stderr: 87000K .......... .......... .......... .......... .......... 64% 24.5M 2s 2024-04-21T08:45:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a3a59c640 1 --2- 172.21.15.7:0/3688974269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d7f8e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c099100 tx=0x7f8a2c09c600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d660d3378732b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a21ffb640 1 -- 172.21.15.7:0/3688974269 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c0bbe60 con 0x5632b1d60b60 2024-04-21T08:45:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a21ffb640 1 -- 172.21.15.7:0/3688974269 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f8a2c0a5bb0 con 0x5632b1d60b60 2024-04-21T08:45:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a43626740 1 -- 172.21.15.7:0/3688974269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a24003d80 con 0x5632b1d60b60 2024-04-21T08:45:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a21ffb640 1 -- 172.21.15.7:0/3688974269 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8a2c0ce460 con 0x5632b1d60b60 2024-04-21T08:45:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a43626740 10 client.15960 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a43626740 1 -- 172.21.15.7:0/3688974269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5632b1de01a0 con 0x5632b1d60b60 2024-04-21T08:45:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a21ffb640 1 -- 172.21.15.7:0/3688974269 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a24003d80 con 0x5632b1d60b60 2024-04-21T08:45:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a21ffb640 1 -- 172.21.15.7:0/3688974269 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8a2c061ce0 con 0x5632b1d60b60 2024-04-21T08:45:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a43626740 10 client.15960 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a43626740 10 client.15960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a43626740 20 client.15960 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a43626740 1 --2- 172.21.15.7:0/3688974269 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5632b1de06d0 0x5632b1de2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a43626740 4 client.15960 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:45:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.036+0000 7f8a43626740 1 -- 172.21.15.7:0/3688974269 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632b1c653d0 con 0x5632b1de06d0 2024-04-21T08:45:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.037+0000 7f8a39d9b640 1 --2- 172.21.15.7:0/3688974269 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5632b1de06d0 0x5632b1de2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.037+0000 7f8a39d9b640 1 --2- 172.21.15.7:0/3688974269 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5632b1de06d0 0x5632b1de2b40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d236319c7f1b587 server_cookie=d62bc60acf024c5b in_seq=0 out_seq=0 2024-04-21T08:45:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.037+0000 7f8a21ffb640 10 client.15960 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:41.063 INFO:tasks.workunit.client.0.smithi007.stderr: 87050K .......... .......... .......... .......... .......... 64% 22.7M 2s 2024-04-21T08:45:41.063 INFO:tasks.workunit.client.0.smithi007.stderr: 87100K .......... .......... .......... .......... .......... 64% 44.9M 2s 2024-04-21T08:45:41.064 INFO:tasks.workunit.client.0.smithi007.stderr: 87150K .......... .......... .......... .......... .......... 64% 25.1M 2s 2024-04-21T08:45:41.064 INFO:tasks.workunit.client.0.smithi007.stderr: 87200K .......... .......... .......... .......... .......... 64% 25.5M 2s 2024-04-21T08:45:41.064 INFO:tasks.workunit.client.0.smithi007.stderr: 87250K .......... .......... .......... .......... .......... 64% 31.3M 2s 2024-04-21T08:45:41.064 INFO:tasks.workunit.client.0.smithi007.stderr: 87300K .......... .......... .......... .......... .......... 64% 24.2M 2s 2024-04-21T08:45:41.064 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a21ffb640 1 -- 172.21.15.7:0/3688974269 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5632b1c653d0 con 0x5632b1de06d0 2024-04-21T08:45:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a21ffb640 10 client.15960 handle_command_reply: tid=0 2024-04-21T08:45:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 1 client.15960 shutdown 2024-04-21T08:45:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 1 -- 172.21.15.7:0/3688974269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 msgr2=0x5632b1d7f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 1 --2- 172.21.15.7:0/3688974269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d7f8e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c099100 tx=0x7f8a2c09c600 comp rx=0 tx=0).stop 2024-04-21T08:45:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 1 -- 172.21.15.7:0/3688974269 shutdown_connections 2024-04-21T08:45:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 1 --2- 172.21.15.7:0/3688974269 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5632b1de06d0 0x5632b1de2b40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 1 --2- 172.21.15.7:0/3688974269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5632b1d6a0e0 0x5632b1d7d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a21ffb640 0 client.15960 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 1 --2- 172.21.15.7:0/3688974269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5632b1d618d0 0x5632b1d7cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 1 --2- 172.21.15.7:0/3688974269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5632b1d60b60 0x5632b1d7f8e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 1 -- 172.21.15.7:0/3688974269 >> 172.21.15.7:0/3688974269 conn(0x5632b1d27c90 msgr2=0x5632b1d65710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 1 -- 172.21.15.7:0/3688974269 shutdown_connections 2024-04-21T08:45:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 1 -- 172.21.15.7:0/3688974269 wait complete. 2024-04-21T08:45:41.068 INFO:tasks.workunit.client.0.smithi007.stderr: 87350K .......... .......... .......... .......... .......... 64% 46.0M 2s 2024-04-21T08:45:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.047+0000 7f8a43626740 20 client.15960 trim_cache size 0 max 16384 2024-04-21T08:45:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:45:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:45:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:45:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:45:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 -- 172.21.15.7:0/3031575441 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a280a3e30 msgr2=0x7f8a280a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a280a3e30 0x7f8a280a62f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8a3402c700 tx=0x7f8a3402cf70 comp rx=0 tx=0).stop 2024-04-21T08:45:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 -- 172.21.15.7:0/3031575441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a3c100d40 msgr2=0x7f8a3c1a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a3c100d40 0x7f8a3c1a0670 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c00e9f0 tx=0x7f8a2c00edf0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 -- 172.21.15.7:0/3031575441 shutdown_connections 2024-04-21T08:45:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a280a3e30 0x7f8a280a62f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a3c10e8f0 0x7f8a3c19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a3c101660 0x7f8a3c1a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 --2- 172.21.15.7:0/3031575441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a3c100d40 0x7f8a3c1a0670 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 -- 172.21.15.7:0/3031575441 >> 172.21.15.7:0/3031575441 conn(0x7f8a3c077f70 msgr2=0x7f8a3c10fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 -- 172.21.15.7:0/3031575441 shutdown_connections 2024-04-21T08:45:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:41.048+0000 7f8a40a31640 1 -- 172.21.15.7:0/3031575441 wait complete. 2024-04-21T08:45:41.087 INFO:tasks.workunit.client.0.smithi007.stderr: 87400K .......... .......... .......... .......... .......... 64% 24.2M 2s 2024-04-21T08:45:41.087 INFO:tasks.workunit.client.0.smithi007.stderr: 87450K .......... .......... .......... .......... .......... 64% 23.9M 2s 2024-04-21T08:45:41.087 INFO:tasks.workunit.client.0.smithi007.stderr: 87500K .......... .......... .......... .......... .......... 64% 27.1M 2s 2024-04-21T08:45:41.087 INFO:tasks.workunit.client.0.smithi007.stderr: 87550K .......... .......... .......... .......... .......... 64% 25.3M 2s 2024-04-21T08:45:41.087 INFO:tasks.workunit.client.0.smithi007.stderr: 87600K .......... .......... .......... .......... .......... 64% 46.9M 2s 2024-04-21T08:45:41.087 INFO:tasks.workunit.client.0.smithi007.stderr: 87650K .......... .......... .......... .......... .......... 64% 18.8M 2s 2024-04-21T08:45:41.087 INFO:tasks.workunit.client.0.smithi007.stderr: 87700K .......... .......... .......... .......... .......... 64% 21.1M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 87750K .......... .......... .......... .......... .......... 64% 48.6M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 87800K .......... .......... .......... .......... .......... 64% 23.0M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 87850K .......... .......... .......... .......... .......... 64% 35.4M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 87900K .......... .......... .......... .......... .......... 64% 22.4M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 87950K .......... .......... .......... .......... .......... 64% 25.9M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 88000K .......... .......... .......... .......... .......... 64% 37.8M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 88050K .......... .......... .......... .......... .......... 64% 24.2M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 88100K .......... .......... .......... .......... .......... 64% 42.9M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 88150K .......... .......... .......... .......... .......... 64% 24.0M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 88200K .......... .......... .......... .......... .......... 65% 28.1M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 88250K .......... .......... .......... .......... .......... 65% 30.1M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 88300K .......... .......... .......... .......... .......... 65% 25.5M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 88350K .......... .......... .......... .......... .......... 65% 54.0M 2s 2024-04-21T08:45:41.088 INFO:tasks.workunit.client.0.smithi007.stderr: 88400K .......... .......... .......... .......... .......... 65% 23.0M 2s 2024-04-21T08:45:41.089 INFO:tasks.workunit.client.0.smithi007.stderr: 88450K .......... .......... .......... .......... .......... 65% 31.5M 2s 2024-04-21T08:45:41.091 INFO:tasks.workunit.client.0.smithi007.stderr: 88500K .......... .......... .......... .......... .......... 65% 26.9M 2s 2024-04-21T08:45:41.091 INFO:tasks.workunit.client.0.smithi007.stderr: 88550K .......... .......... .......... .......... .......... 65% 23.5M 2s 2024-04-21T08:45:41.091 INFO:tasks.workunit.client.0.smithi007.stderr: 88600K .......... .......... .......... .......... .......... 65% 54.6M 2s 2024-04-21T08:45:41.093 INFO:tasks.workunit.client.0.smithi007.stderr: 88650K .......... .......... .......... .......... .......... 65% 24.3M 2s 2024-04-21T08:45:41.095 INFO:tasks.workunit.client.0.smithi007.stderr: 88700K .......... .......... .......... .......... .......... 65% 31.2M 2s 2024-04-21T08:45:41.096 INFO:tasks.workunit.client.0.smithi007.stderr: 88750K .......... .......... .......... .......... .......... 65% 26.5M 2s 2024-04-21T08:45:41.099 INFO:tasks.workunit.client.0.smithi007.stderr: 88800K .......... .......... .......... .......... .......... 65% 24.0M 2s 2024-04-21T08:45:41.099 INFO:tasks.workunit.client.0.smithi007.stderr: 88850K .......... .......... .......... .......... .......... 65% 57.2M 2s 2024-04-21T08:45:41.101 INFO:tasks.workunit.client.0.smithi007.stderr: 88900K .......... .......... .......... .......... .......... 65% 24.0M 2s 2024-04-21T08:45:41.103 INFO:tasks.workunit.client.0.smithi007.stderr: 88950K .......... .......... .......... .......... .......... 65% 34.7M 2s 2024-04-21T08:45:41.106 INFO:tasks.workunit.client.0.smithi007.stderr: 89000K .......... .......... .......... .......... .......... 65% 20.1M 2s 2024-04-21T08:45:41.107 INFO:tasks.workunit.client.0.smithi007.stderr: 89050K .......... .......... .......... .......... .......... 65% 29.6M 2s 2024-04-21T08:45:41.108 INFO:tasks.workunit.client.0.smithi007.stderr: 89100K .......... .......... .......... .......... .......... 65% 50.9M 2s 2024-04-21T08:45:41.110 INFO:tasks.workunit.client.0.smithi007.stderr: 89150K .......... .......... .......... .......... .......... 65% 24.7M 2s 2024-04-21T08:45:41.112 INFO:tasks.workunit.client.0.smithi007.stderr: 89200K .......... .......... .......... .......... .......... 65% 20.3M 2s 2024-04-21T08:45:41.114 INFO:tasks.workunit.client.0.smithi007.stderr: 89250K .......... .......... .......... .......... .......... 65% 52.7M 2s 2024-04-21T08:45:41.115 INFO:tasks.workunit.client.0.smithi007.stderr: 89300K .......... .......... .......... .......... .......... 65% 24.7M 2s 2024-04-21T08:45:41.118 INFO:tasks.workunit.client.0.smithi007.stderr: 89350K .......... .......... .......... .......... .......... 65% 20.0M 2s 2024-04-21T08:45:41.119 INFO:tasks.workunit.client.0.smithi007.stderr: 89400K .......... .......... .......... .......... .......... 65% 52.7M 2s 2024-04-21T08:45:41.120 INFO:tasks.workunit.client.0.smithi007.stderr: 89450K .......... .......... .......... .......... .......... 65% 28.8M 2s 2024-04-21T08:45:41.122 INFO:tasks.workunit.client.0.smithi007.stderr: 89500K .......... .......... .......... .......... .......... 65% 28.0M 2s 2024-04-21T08:45:41.124 INFO:tasks.workunit.client.0.smithi007.stderr: 89550K .......... .......... .......... .......... .......... 66% 25.5M 2s 2024-04-21T08:45:41.125 INFO:tasks.workunit.client.0.smithi007.stderr: 89600K .......... .......... .......... .......... .......... 66% 26.7M 2s 2024-04-21T08:45:41.128 INFO:tasks.workunit.client.0.smithi007.stderr: 89650K .......... .......... .......... .......... .......... 66% 24.6M 2s 2024-04-21T08:45:41.131 INFO:tasks.workunit.client.0.smithi007.stderr: 89700K .......... .......... .......... .......... .......... 66% 19.4M 2s 2024-04-21T08:45:41.132 INFO:tasks.workunit.client.0.smithi007.stderr: 89750K .......... .......... .......... .......... .......... 66% 30.1M 2s 2024-04-21T08:45:41.133 INFO:tasks.workunit.client.0.smithi007.stderr: 89800K .......... .......... .......... .......... .......... 66% 29.0M 2s 2024-04-21T08:45:41.136 INFO:tasks.workunit.client.0.smithi007.stderr: 89850K .......... .......... .......... .......... .......... 66% 22.7M 2s 2024-04-21T08:45:41.137 INFO:tasks.workunit.client.0.smithi007.stderr: 89900K .......... .......... .......... .......... .......... 66% 50.9M 2s 2024-04-21T08:45:41.139 INFO:tasks.workunit.client.0.smithi007.stderr: 89950K .......... .......... .......... .......... .......... 66% 26.1M 2s 2024-04-21T08:45:41.141 INFO:tasks.workunit.client.0.smithi007.stderr: 90000K .......... .......... .......... .......... .......... 66% 23.2M 2s 2024-04-21T08:45:41.142 INFO:tasks.workunit.client.0.smithi007.stderr: 90050K .......... .......... .......... .......... .......... 66% 34.9M 2s 2024-04-21T08:45:41.144 INFO:tasks.workunit.client.0.smithi007.stderr: 90100K .......... .......... .......... .......... .......... 66% 22.9M 2s 2024-04-21T08:45:41.145 INFO:tasks.workunit.client.0.smithi007.stderr: 90150K .......... .......... .......... .......... .......... 66% 51.7M 2s 2024-04-21T08:45:41.147 INFO:tasks.workunit.client.0.smithi007.stderr: 90200K .......... .......... .......... .......... .......... 66% 25.1M 2s 2024-04-21T08:45:41.149 INFO:tasks.workunit.client.0.smithi007.stderr: 90250K .......... .......... .......... .......... .......... 66% 28.8M 2s 2024-04-21T08:45:41.150 INFO:tasks.workunit.client.0.smithi007.stderr: 90300K .......... .......... .......... .......... .......... 66% 30.7M 2s 2024-04-21T08:45:41.152 INFO:tasks.workunit.client.0.smithi007.stderr: 90350K .......... .......... .......... .......... .......... 66% 23.3M 2s 2024-04-21T08:45:41.154 INFO:tasks.workunit.client.0.smithi007.stderr: 90400K .......... .......... .......... .......... .......... 66% 39.0M 2s 2024-04-21T08:45:41.156 INFO:tasks.workunit.client.0.smithi007.stderr: 90450K .......... .......... .......... .......... .......... 66% 24.5M 2s 2024-04-21T08:45:41.158 INFO:tasks.workunit.client.0.smithi007.stderr: 90500K .......... .......... .......... .......... .......... 66% 21.5M 2s 2024-04-21T08:45:41.159 INFO:tasks.workunit.client.0.smithi007.stderr: 90550K .......... .......... .......... .......... .......... 66% 42.4M 2s 2024-04-21T08:45:41.161 INFO:tasks.workunit.client.0.smithi007.stderr: 90600K .......... .......... .......... .......... .......... 66% 20.8M 2s 2024-04-21T08:45:41.163 INFO:tasks.workunit.client.0.smithi007.stderr: 90650K .......... .......... .......... .......... .......... 66% 37.7M 2s 2024-04-21T08:45:41.165 INFO:tasks.workunit.client.0.smithi007.stderr: 90700K .......... .......... .......... .......... .......... 66% 25.3M 2s 2024-04-21T08:45:41.166 INFO:tasks.workunit.client.0.smithi007.stderr: 90750K .......... .......... .......... .......... .......... 66% 34.9M 2s 2024-04-21T08:45:41.168 INFO:tasks.workunit.client.0.smithi007.stderr: 90800K .......... .......... .......... .......... .......... 66% 24.3M 2s 2024-04-21T08:45:41.170 INFO:tasks.workunit.client.0.smithi007.stderr: 90850K .......... .......... .......... .......... .......... 66% 24.7M 2s 2024-04-21T08:45:41.171 INFO:tasks.workunit.client.0.smithi007.stderr: 90900K .......... .......... .......... .......... .......... 66% 42.4M 2s 2024-04-21T08:45:41.173 INFO:tasks.workunit.client.0.smithi007.stderr: 90950K .......... .......... .......... .......... .......... 67% 21.8M 2s 2024-04-21T08:45:41.176 INFO:tasks.workunit.client.0.smithi007.stderr: 91000K .......... .......... .......... .......... .......... 67% 24.3M 2s 2024-04-21T08:45:41.176 INFO:tasks.workunit.client.0.smithi007.stderr: 91050K .......... .......... .......... .......... .......... 67% 42.5M 2s 2024-04-21T08:45:41.179 INFO:tasks.workunit.client.0.smithi007.stderr: 91100K .......... .......... .......... .......... .......... 67% 22.4M 2s 2024-04-21T08:45:41.180 INFO:tasks.workunit.client.0.smithi007.stderr: 91150K .......... .......... .......... .......... .......... 67% 46.2M 2s 2024-04-21T08:45:41.186 INFO:tasks.workunit.client.0.smithi007.stderr: 91200K .......... .......... .......... .......... .......... 67% 10.8M 2s 2024-04-21T08:45:41.186 INFO:tasks.workunit.client.0.smithi007.stderr: 91250K .......... .......... .......... .......... .......... 67% 600M 2s 2024-04-21T08:45:41.186 INFO:tasks.workunit.client.0.smithi007.stderr: 91300K .......... .......... .......... .......... .......... 67% 35.0M 2s 2024-04-21T08:45:41.187 INFO:tasks.workunit.client.0.smithi007.stderr: 91350K .......... .......... .......... .......... .......... 67% 35.0M 2s 2024-04-21T08:45:41.188 INFO:tasks.workunit.client.0.smithi007.stderr: 91400K .......... .......... .......... .......... .......... 67% 50.2M 2s 2024-04-21T08:45:41.191 INFO:tasks.workunit.client.0.smithi007.stderr: 91450K .......... .......... .......... .......... .......... 67% 18.9M 2s 2024-04-21T08:45:41.192 INFO:tasks.workunit.client.0.smithi007.stderr: 91500K .......... .......... .......... .......... .......... 67% 41.4M 2s 2024-04-21T08:45:41.194 INFO:tasks.workunit.client.0.smithi007.stderr: 91550K .......... .......... .......... .......... .......... 67% 22.5M 2s 2024-04-21T08:45:41.197 INFO:tasks.workunit.client.0.smithi007.stderr: 91600K .......... .......... .......... .......... .......... 67% 21.3M 2s 2024-04-21T08:45:41.198 INFO:tasks.workunit.client.0.smithi007.stderr: 91650K .......... .......... .......... .......... .......... 67% 47.1M 2s 2024-04-21T08:45:41.200 INFO:tasks.workunit.client.0.smithi007.stderr: 91700K .......... .......... .......... .......... .......... 67% 23.6M 2s 2024-04-21T08:45:41.202 INFO:tasks.workunit.client.0.smithi007.stderr: 91750K .......... .......... .......... .......... .......... 67% 29.9M 2s 2024-04-21T08:45:41.203 INFO:tasks.workunit.client.0.smithi007.stderr: 91800K .......... .......... .......... .......... .......... 67% 29.2M 2s 2024-04-21T08:45:41.205 INFO:tasks.workunit.client.0.smithi007.stderr: 91850K .......... .......... .......... .......... .......... 67% 25.0M 2s 2024-04-21T08:45:41.206 INFO:tasks.workunit.client.0.smithi007.stderr: 91900K .......... .......... .......... .......... .......... 67% 48.2M 2s 2024-04-21T08:45:41.208 INFO:tasks.workunit.client.0.smithi007.stderr: 91950K .......... .......... .......... .......... .......... 67% 24.8M 2s 2024-04-21T08:45:41.210 INFO:tasks.workunit.client.0.smithi007.stderr: 92000K .......... .......... .......... .......... .......... 67% 38.5M 2s 2024-04-21T08:45:41.211 INFO:tasks.workunit.client.0.smithi007.stderr: 92050K .......... .......... .......... .......... .......... 67% 23.8M 2s 2024-04-21T08:45:41.213 INFO:tasks.workunit.client.0.smithi007.stderr: 92100K .......... .......... .......... .......... .......... 67% 26.0M 2s 2024-04-21T08:45:41.214 INFO:tasks.workunit.client.0.smithi007.stderr: 92150K .......... .......... .......... .......... .......... 67% 42.5M 2s 2024-04-21T08:45:41.216 INFO:tasks.workunit.client.0.smithi007.stderr: 92200K .......... .......... .......... .......... .......... 67% 23.1M 2s 2024-04-21T08:45:41.218 INFO:tasks.workunit.client.0.smithi007.stderr: 92250K .......... .......... .......... .......... .......... 67% 41.2M 2s 2024-04-21T08:45:41.220 INFO:tasks.workunit.client.0.smithi007.stderr: 92300K .......... .......... .......... .......... .......... 68% 23.0M 2s 2024-04-21T08:45:41.222 INFO:tasks.workunit.client.0.smithi007.stderr: 92350K .......... .......... .......... .......... .......... 68% 21.1M 2s 2024-04-21T08:45:41.223 INFO:tasks.workunit.client.0.smithi007.stderr: 92400K .......... .......... .......... .......... .......... 68% 39.6M 2s 2024-04-21T08:45:41.225 INFO:tasks.workunit.client.0.smithi007.stderr: 92450K .......... .......... .......... .......... .......... 68% 25.9M 2s 2024-04-21T08:45:41.227 INFO:tasks.workunit.client.0.smithi007.stderr: 92500K .......... .......... .......... .......... .......... 68% 24.4M 2s 2024-04-21T08:45:41.228 INFO:tasks.workunit.client.0.smithi007.stderr: 92550K .......... .......... .......... .......... .......... 68% 39.6M 2s 2024-04-21T08:45:41.230 INFO:tasks.workunit.client.0.smithi007.stderr: 92600K .......... .......... .......... .......... .......... 68% 24.7M 2s 2024-04-21T08:45:41.231 INFO:tasks.workunit.client.0.smithi007.stderr: 92650K .......... .......... .......... .......... .......... 68% 38.7M 2s 2024-04-21T08:45:41.234 INFO:tasks.workunit.client.0.smithi007.stderr: 92700K .......... .......... .......... .......... .......... 68% 18.4M 2s 2024-04-21T08:45:41.236 INFO:tasks.workunit.client.0.smithi007.stderr: 92750K .......... .......... .......... .......... .......... 68% 30.8M 2s 2024-04-21T08:45:41.237 INFO:tasks.workunit.client.0.smithi007.stderr: 92800K .......... .......... .......... .......... .......... 68% 41.5M 2s 2024-04-21T08:45:41.239 INFO:tasks.workunit.client.0.smithi007.stderr: 92850K .......... .......... .......... .......... .......... 68% 19.2M 2s 2024-04-21T08:45:41.240 INFO:tasks.workunit.client.0.smithi007.stderr: 92900K .......... .......... .......... .......... .......... 68% 57.8M 2s 2024-04-21T08:45:41.241 INFO:tasks.workunit.client.0.smithi007.stderr: 92950K .......... .......... .......... .......... .......... 68% 30.4M 2s 2024-04-21T08:45:41.244 INFO:tasks.workunit.client.0.smithi007.stderr: 93000K .......... .......... .......... .......... .......... 68% 18.9M 2s 2024-04-21T08:45:41.246 INFO:tasks.workunit.client.0.smithi007.stderr: 93050K .......... .......... .......... .......... .......... 68% 38.0M 2s 2024-04-21T08:45:41.248 INFO:tasks.workunit.client.0.smithi007.stderr: 93100K .......... .......... .......... .......... .......... 68% 24.8M 2s 2024-04-21T08:45:41.249 INFO:tasks.workunit.client.0.smithi007.stderr: 93150K .......... .......... .......... .......... .......... 68% 49.0M 2s 2024-04-21T08:45:41.251 INFO:tasks.workunit.client.0.smithi007.stderr: 93200K .......... .......... .......... .......... .......... 68% 25.1M 2s 2024-04-21T08:45:41.253 INFO:tasks.workunit.client.0.smithi007.stderr: 93250K .......... .......... .......... .......... .......... 68% 21.6M 1s 2024-04-21T08:45:41.254 INFO:tasks.workunit.client.0.smithi007.stderr: 93300K .......... .......... .......... .......... .......... 68% 42.7M 1s 2024-04-21T08:45:41.256 INFO:tasks.workunit.client.0.smithi007.stderr: 93350K .......... .......... .......... .......... .......... 68% 21.9M 1s 2024-04-21T08:45:41.257 INFO:tasks.workunit.client.0.smithi007.stderr: 93400K .......... .......... .......... .......... .......... 68% 51.8M 1s 2024-04-21T08:45:41.259 INFO:tasks.workunit.client.0.smithi007.stderr: 93450K .......... .......... .......... .......... .......... 68% 23.3M 1s 2024-04-21T08:45:41.261 INFO:tasks.workunit.client.0.smithi007.stderr: 93500K .......... .......... .......... .......... .......... 68% 24.2M 1s 2024-04-21T08:45:41.263 INFO:tasks.workunit.client.0.smithi007.stderr: 93550K .......... .......... .......... .......... .......... 68% 40.2M 1s 2024-04-21T08:45:41.265 INFO:tasks.workunit.client.0.smithi007.stderr: 93600K .......... .......... .......... .......... .......... 68% 22.4M 1s 2024-04-21T08:45:41.267 INFO:tasks.workunit.client.0.smithi007.stderr: 93650K .......... .......... .......... .......... .......... 69% 46.3M 1s 2024-04-21T08:45:41.268 INFO:tasks.workunit.client.0.smithi007.stderr: 93700K .......... .......... .......... .......... .......... 69% 25.7M 1s 2024-04-21T08:45:41.270 INFO:tasks.workunit.client.0.smithi007.stderr: 93750K .......... .......... .......... .......... .......... 69% 21.8M 1s 2024-04-21T08:45:41.271 INFO:tasks.workunit.client.0.smithi007.stderr: 93800K .......... .......... .......... .......... .......... 69% 44.2M 1s 2024-04-21T08:45:41.275 INFO:tasks.workunit.client.0.smithi007.stderr: 93850K .......... .......... .......... .......... .......... 69% 23.6M 1s 2024-04-21T08:45:41.276 INFO:tasks.workunit.client.0.smithi007.stderr: 93900K .......... .......... .......... .......... .......... 69% 23.0M 1s 2024-04-21T08:45:41.276 INFO:tasks.workunit.client.0.smithi007.stderr: 93950K .......... .......... .......... .......... .......... 69% 54.1M 1s 2024-04-21T08:45:41.278 INFO:tasks.workunit.client.0.smithi007.stderr: 94000K .......... .......... .......... .......... .......... 69% 23.2M 1s 2024-04-21T08:45:41.279 INFO:tasks.workunit.client.0.smithi007.stderr: 94050K .......... .......... .......... .......... .......... 69% 44.7M 1s 2024-04-21T08:45:41.282 INFO:tasks.workunit.client.0.smithi007.stderr: 94100K .......... .......... .......... .......... .......... 69% 21.9M 1s 2024-04-21T08:45:41.285 INFO:tasks.workunit.client.0.smithi007.stderr: 94150K .......... .......... .......... .......... .......... 69% 22.2M 1s 2024-04-21T08:45:41.285 INFO:tasks.workunit.client.0.smithi007.stderr: 94200K .......... .......... .......... .......... .......... 69% 72.1M 1s 2024-04-21T08:45:41.287 INFO:tasks.workunit.client.0.smithi007.stderr: 94250K .......... .......... .......... .......... .......... 69% 21.7M 1s 2024-04-21T08:45:41.288 INFO:tasks.workunit.client.0.smithi007.stderr: 94300K .......... .......... .......... .......... .......... 69% 42.6M 1s 2024-04-21T08:45:41.290 INFO:tasks.workunit.client.0.smithi007.stderr: 94350K .......... .......... .......... .......... .......... 69% 23.4M 1s 2024-04-21T08:45:41.292 INFO:tasks.workunit.client.0.smithi007.stderr: 94400K .......... .......... .......... .......... .......... 69% 25.5M 1s 2024-04-21T08:45:41.293 INFO:tasks.workunit.client.0.smithi007.stderr: 94450K .......... .......... .......... .......... .......... 69% 45.6M 1s 2024-04-21T08:45:41.295 INFO:tasks.workunit.client.0.smithi007.stderr: 94500K .......... .......... .......... .......... .......... 69% 30.0M 1s 2024-04-21T08:45:41.296 INFO:tasks.workunit.client.0.smithi007.stderr: 94550K .......... .......... .......... .......... .......... 69% 27.6M 1s 2024-04-21T08:45:41.298 INFO:tasks.workunit.client.0.smithi007.stderr: 94600K .......... .......... .......... .......... .......... 69% 27.3M 1s 2024-04-21T08:45:41.300 INFO:tasks.workunit.client.0.smithi007.stderr: 94650K .......... .......... .......... .......... .......... 69% 21.7M 1s 2024-04-21T08:45:41.301 INFO:tasks.workunit.client.0.smithi007.stderr: 94700K .......... .......... .......... .......... .......... 69% 48.1M 1s 2024-04-21T08:45:41.303 INFO:tasks.workunit.client.0.smithi007.stderr: 94750K .......... .......... .......... .......... .......... 69% 24.4M 1s 2024-04-21T08:45:41.304 INFO:tasks.workunit.client.0.smithi007.stderr: 94800K .......... .......... .......... .......... .......... 69% 43.9M 1s 2024-04-21T08:45:41.306 INFO:tasks.workunit.client.0.smithi007.stderr: 94850K .......... .......... .......... .......... .......... 69% 22.1M 1s 2024-04-21T08:45:41.309 INFO:tasks.workunit.client.0.smithi007.stderr: 94900K .......... .......... .......... .......... .......... 69% 23.9M 1s 2024-04-21T08:45:41.310 INFO:tasks.workunit.client.0.smithi007.stderr: 94950K .......... .......... .......... .......... .......... 69% 50.1M 1s 2024-04-21T08:45:41.312 INFO:tasks.workunit.client.0.smithi007.stderr: 95000K .......... .......... .......... .......... .......... 70% 23.6M 1s 2024-04-21T08:45:41.314 INFO:tasks.workunit.client.0.smithi007.stderr: 95050K .......... .......... .......... .......... .......... 70% 40.0M 1s 2024-04-21T08:45:41.315 INFO:tasks.workunit.client.0.smithi007.stderr: 95100K .......... .......... .......... .......... .......... 70% 23.9M 1s 2024-04-21T08:45:41.317 INFO:tasks.workunit.client.0.smithi007.stderr: 95150K .......... .......... .......... .......... .......... 70% 23.9M 1s 2024-04-21T08:45:41.318 INFO:tasks.workunit.client.0.smithi007.stderr: 95200K .......... .......... .......... .......... .......... 70% 47.7M 1s 2024-04-21T08:45:41.320 INFO:tasks.workunit.client.0.smithi007.stderr: 95250K .......... .......... .......... .......... .......... 70% 23.4M 1s 2024-04-21T08:45:41.321 INFO:tasks.workunit.client.0.smithi007.stderr: 95300K .......... .......... .......... .......... .......... 70% 41.8M 1s 2024-04-21T08:45:41.323 INFO:tasks.workunit.client.0.smithi007.stderr: 95350K .......... .......... .......... .......... .......... 70% 23.2M 1s 2024-04-21T08:45:41.326 INFO:tasks.workunit.client.0.smithi007.stderr: 95400K .......... .......... .......... .......... .......... 70% 22.2M 1s 2024-04-21T08:45:41.327 INFO:tasks.workunit.client.0.smithi007.stderr: 95450K .......... .......... .......... .......... .......... 70% 59.1M 1s 2024-04-21T08:45:41.329 INFO:tasks.workunit.client.0.smithi007.stderr: 95500K .......... .......... .......... .......... .......... 70% 23.4M 1s 2024-04-21T08:45:41.330 INFO:tasks.workunit.client.0.smithi007.stderr: 95550K .......... .......... .......... .......... .......... 70% 37.9M 1s 2024-04-21T08:45:41.332 INFO:tasks.workunit.client.0.smithi007.stderr: 95600K .......... .......... .......... .......... .......... 70% 26.1M 1s 2024-04-21T08:45:41.334 INFO:tasks.workunit.client.0.smithi007.stderr: 95650K .......... .......... .......... .......... .......... 70% 21.7M 1s 2024-04-21T08:45:41.335 INFO:tasks.workunit.client.0.smithi007.stderr: 95700K .......... .......... .......... .......... .......... 70% 46.4M 1s 2024-04-21T08:45:41.337 INFO:tasks.workunit.client.0.smithi007.stderr: 95750K .......... .......... .......... .......... .......... 70% 24.8M 1s 2024-04-21T08:45:41.338 INFO:tasks.workunit.client.0.smithi007.stderr: 95800K .......... .......... .......... .......... .......... 70% 34.2M 1s 2024-04-21T08:45:41.341 INFO:tasks.workunit.client.0.smithi007.stderr: 95850K .......... .......... .......... .......... .......... 70% 25.7M 1s 2024-04-21T08:45:41.343 INFO:tasks.workunit.client.0.smithi007.stderr: 95900K .......... .......... .......... .......... .......... 70% 24.2M 1s 2024-04-21T08:45:41.343 INFO:tasks.workunit.client.0.smithi007.stderr: 95950K .......... .......... .......... .......... .......... 70% 48.7M 1s 2024-04-21T08:45:41.345 INFO:tasks.workunit.client.0.smithi007.stderr: 96000K .......... .......... .......... .......... .......... 70% 23.2M 1s 2024-04-21T08:45:41.347 INFO:tasks.workunit.client.0.smithi007.stderr: 96050K .......... .......... .......... .......... .......... 70% 41.4M 1s 2024-04-21T08:45:41.349 INFO:tasks.workunit.client.0.smithi007.stderr: 96100K .......... .......... .......... .......... .......... 70% 19.7M 1s 2024-04-21T08:45:41.351 INFO:tasks.workunit.client.0.smithi007.stderr: 96150K .......... .......... .......... .......... .......... 70% 26.1M 1s 2024-04-21T08:45:41.352 INFO:tasks.workunit.client.0.smithi007.stderr: 96200K .......... .......... .......... .......... .......... 70% 42.7M 1s 2024-04-21T08:45:41.355 INFO:tasks.workunit.client.0.smithi007.stderr: 96250K .......... .......... .......... .......... .......... 70% 17.6M 1s 2024-04-21T08:45:41.356 INFO:tasks.workunit.client.0.smithi007.stderr: 96300K .......... .......... .......... .......... .......... 70% 40.4M 1s 2024-04-21T08:45:41.358 INFO:tasks.workunit.client.0.smithi007.stderr: 96350K .......... .......... .......... .......... .......... 71% 24.6M 1s 2024-04-21T08:45:41.361 INFO:tasks.workunit.client.0.smithi007.stderr: 96400K .......... .......... .......... .......... .......... 71% 23.1M 1s 2024-04-21T08:45:41.361 INFO:tasks.workunit.client.0.smithi007.stderr: 96450K .......... .......... .......... .......... .......... 71% 45.3M 1s 2024-04-21T08:45:41.364 INFO:tasks.workunit.client.0.smithi007.stderr: 96500K .......... .......... .......... .......... .......... 71% 22.1M 1s 2024-04-21T08:45:41.365 INFO:tasks.workunit.client.0.smithi007.stderr: 96550K .......... .......... .......... .......... .......... 71% 37.9M 1s 2024-04-21T08:45:41.368 INFO:tasks.workunit.client.0.smithi007.stderr: 96600K .......... .......... .......... .......... .......... 71% 21.0M 1s 2024-04-21T08:45:41.369 INFO:tasks.workunit.client.0.smithi007.stderr: 96650K .......... .......... .......... .......... .......... 71% 26.0M 1s 2024-04-21T08:45:41.370 INFO:tasks.workunit.client.0.smithi007.stderr: 96700K .......... .......... .......... .......... .......... 71% 43.6M 1s 2024-04-21T08:45:41.372 INFO:tasks.workunit.client.0.smithi007.stderr: 96750K .......... .......... .......... .......... .......... 71% 24.8M 1s 2024-04-21T08:45:41.373 INFO:tasks.workunit.client.0.smithi007.stderr: 96800K .......... .......... .......... .......... .......... 71% 36.0M 1s 2024-04-21T08:45:41.375 INFO:tasks.workunit.client.0.smithi007.stderr: 96850K .......... .......... .......... .......... .......... 71% 26.0M 1s 2024-04-21T08:45:41.377 INFO:tasks.workunit.client.0.smithi007.stderr: 96900K .......... .......... .......... .......... .......... 71% 23.0M 1s 2024-04-21T08:45:41.378 INFO:tasks.workunit.client.0.smithi007.stderr: 96950K .......... .......... .......... .......... .......... 71% 58.5M 1s 2024-04-21T08:45:41.380 INFO:tasks.workunit.client.0.smithi007.stderr: 97000K .......... .......... .......... .......... .......... 71% 24.5M 1s 2024-04-21T08:45:41.382 INFO:tasks.workunit.client.0.smithi007.stderr: 97050K .......... .......... .......... .......... .......... 71% 36.0M 1s 2024-04-21T08:45:41.384 INFO:tasks.workunit.client.0.smithi007.stderr: 97100K .......... .......... .......... .......... .......... 71% 23.3M 1s 2024-04-21T08:45:41.387 INFO:tasks.workunit.client.0.smithi007.stderr: 97150K .......... .......... .......... .......... .......... 71% 21.0M 1s 2024-04-21T08:45:41.387 INFO:tasks.workunit.client.0.smithi007.stderr: 97200K .......... .......... .......... .......... .......... 71% 59.2M 1s 2024-04-21T08:45:41.388 INFO:tasks.workunit.client.0.smithi007.stderr: 97250K .......... .......... .......... .......... .......... 71% 29.0M 1s 2024-04-21T08:45:41.391 INFO:tasks.workunit.client.0.smithi007.stderr: 97300K .......... .......... .......... .......... .......... 71% 27.8M 1s 2024-04-21T08:45:41.392 INFO:tasks.workunit.client.0.smithi007.stderr: 97350K .......... .......... .......... .......... .......... 71% 23.8M 1s 2024-04-21T08:45:41.395 INFO:tasks.workunit.client.0.smithi007.stderr: 97400K .......... .......... .......... .......... .......... 71% 23.8M 1s 2024-04-21T08:45:41.395 INFO:tasks.workunit.client.0.smithi007.stderr: 97450K .......... .......... .......... .......... .......... 71% 54.0M 1s 2024-04-21T08:45:41.397 INFO:tasks.workunit.client.0.smithi007.stderr: 97500K .......... .......... .......... .......... .......... 71% 23.2M 1s 2024-04-21T08:45:41.400 INFO:tasks.workunit.client.0.smithi007.stderr: 97550K .......... .......... .......... .......... .......... 71% 42.8M 1s 2024-04-21T08:45:41.401 INFO:tasks.workunit.client.0.smithi007.stderr: 97600K .......... .......... .......... .......... .......... 71% 21.5M 1s 2024-04-21T08:45:41.403 INFO:tasks.workunit.client.0.smithi007.stderr: 97650K .......... .......... .......... .......... .......... 71% 24.5M 1s 2024-04-21T08:45:41.404 INFO:tasks.workunit.client.0.smithi007.stderr: 97700K .......... .......... .......... .......... .......... 72% 51.1M 1s 2024-04-21T08:45:41.406 INFO:tasks.workunit.client.0.smithi007.stderr: 97750K .......... .......... .......... .......... .......... 72% 21.1M 1s 2024-04-21T08:45:41.408 INFO:tasks.workunit.client.0.smithi007.stderr: 97800K .......... .......... .......... .......... .......... 72% 26.0M 1s 2024-04-21T08:45:41.410 INFO:tasks.workunit.client.0.smithi007.stderr: 97850K .......... .......... .......... .......... .......... 72% 32.6M 1s 2024-04-21T08:45:41.412 INFO:tasks.workunit.client.0.smithi007.stderr: 97900K .......... .......... .......... .......... .......... 72% 22.5M 1s 2024-04-21T08:45:41.412 INFO:tasks.workunit.client.0.smithi007.stderr: 97950K .......... .......... .......... .......... .......... 72% 61.0M 1s 2024-04-21T08:45:41.415 INFO:tasks.workunit.client.0.smithi007.stderr: 98000K .......... .......... .......... .......... .......... 72% 22.1M 1s 2024-04-21T08:45:41.416 INFO:tasks.workunit.client.0.smithi007.stderr: 98050K .......... .......... .......... .......... .......... 72% 28.5M 1s 2024-04-21T08:45:41.418 INFO:tasks.workunit.client.0.smithi007.stderr: 98100K .......... .......... .......... .......... .......... 72% 33.6M 1s 2024-04-21T08:45:41.421 INFO:tasks.workunit.client.0.smithi007.stderr: 98150K .......... .......... .......... .......... .......... 72% 19.8M 1s 2024-04-21T08:45:41.421 INFO:tasks.workunit.client.0.smithi007.stderr: 98200K .......... .......... .......... .......... .......... 72% 48.6M 1s 2024-04-21T08:45:41.423 INFO:tasks.workunit.client.0.smithi007.stderr: 98250K .......... .......... .......... .......... .......... 72% 23.2M 1s 2024-04-21T08:45:41.426 INFO:tasks.workunit.client.0.smithi007.stderr: 98300K .......... .......... .......... .......... .......... 72% 24.2M 1s 2024-04-21T08:45:41.427 INFO:tasks.workunit.client.0.smithi007.stderr: 98350K .......... .......... .......... .......... .......... 72% 24.6M 1s 2024-04-21T08:45:41.429 INFO:tasks.workunit.client.0.smithi007.stderr: 98400K .......... .......... .......... .......... .......... 72% 25.8M 1s 2024-04-21T08:45:41.430 INFO:tasks.workunit.client.0.smithi007.stderr: 98450K .......... .......... .......... .......... .......... 72% 46.9M 1s 2024-04-21T08:45:41.433 INFO:tasks.workunit.client.0.smithi007.stderr: 98500K .......... .......... .......... .......... .......... 72% 20.1M 1s 2024-04-21T08:45:41.434 INFO:tasks.workunit.client.0.smithi007.stderr: 98550K .......... .......... .......... .......... .......... 72% 29.6M 1s 2024-04-21T08:45:41.436 INFO:tasks.workunit.client.0.smithi007.stderr: 98600K .......... .......... .......... .......... .......... 72% 32.2M 1s 2024-04-21T08:45:41.441 INFO:tasks.workunit.client.0.smithi007.stderr: 98650K .......... .......... .......... .......... .......... 72% 22.2M 1s 2024-04-21T08:45:41.443 INFO:tasks.workunit.client.0.smithi007.stderr: 98700K .......... .......... .......... .......... .......... 72% 62.4M 1s 2024-04-21T08:45:41.443 INFO:tasks.workunit.client.0.smithi007.stderr: 98750K .......... .......... .......... .......... .......... 72% 30.5M 1s 2024-04-21T08:45:41.443 INFO:tasks.workunit.client.0.smithi007.stderr: 98800K .......... .......... .......... .......... .......... 72% 22.7M 1s 2024-04-21T08:45:41.444 INFO:tasks.workunit.client.0.smithi007.stderr: 98850K .......... .......... .......... .......... .......... 72% 30.8M 1s 2024-04-21T08:45:41.447 INFO:tasks.workunit.client.0.smithi007.stderr: 98900K .......... .......... .......... .......... .......... 72% 21.9M 1s 2024-04-21T08:45:41.448 INFO:tasks.workunit.client.0.smithi007.stderr: 98950K .......... .......... .......... .......... .......... 72% 56.4M 1s 2024-04-21T08:45:41.449 INFO:tasks.workunit.client.0.smithi007.stderr: 99000K .......... .......... .......... .......... .......... 72% 23.9M 1s 2024-04-21T08:45:41.451 INFO:tasks.workunit.client.0.smithi007.stderr: 99050K .......... .......... .......... .......... .......... 73% 26.2M 1s 2024-04-21T08:45:41.452 INFO:tasks.workunit.client.0.smithi007.stderr: 99100K .......... .......... .......... .......... .......... 73% 33.8M 1s 2024-04-21T08:45:41.455 INFO:tasks.workunit.client.0.smithi007.stderr: 99150K .......... .......... .......... .......... .......... 73% 21.0M 1s 2024-04-21T08:45:41.457 INFO:tasks.workunit.client.0.smithi007.stderr: 99200K .......... .......... .......... .......... .......... 73% 24.9M 1s 2024-04-21T08:45:41.458 INFO:tasks.workunit.client.0.smithi007.stderr: 99250K .......... .......... .......... .......... .......... 73% 39.3M 1s 2024-04-21T08:45:41.460 INFO:tasks.workunit.client.0.smithi007.stderr: 99300K .......... .......... .......... .......... .......... 73% 26.5M 1s 2024-04-21T08:45:41.462 INFO:tasks.workunit.client.0.smithi007.stderr: 99350K .......... .......... .......... .......... .......... 73% 27.9M 1s 2024-04-21T08:45:41.464 INFO:tasks.workunit.client.0.smithi007.stderr: 99400K .......... .......... .......... .......... .......... 73% 24.0M 1s 2024-04-21T08:45:41.466 INFO:tasks.workunit.client.0.smithi007.stderr: 99450K .......... .......... .......... .......... .......... 73% 23.0M 1s 2024-04-21T08:45:41.467 INFO:tasks.workunit.client.0.smithi007.stderr: 99500K .......... .......... .......... .......... .......... 73% 42.4M 1s 2024-04-21T08:45:41.469 INFO:tasks.workunit.client.0.smithi007.stderr: 99550K .......... .......... .......... .......... .......... 73% 24.8M 1s 2024-04-21T08:45:41.470 INFO:tasks.workunit.client.0.smithi007.stderr: 99600K .......... .......... .......... .......... .......... 73% 37.0M 1s 2024-04-21T08:45:41.472 INFO:tasks.workunit.client.0.smithi007.stderr: 99650K .......... .......... .......... .......... .......... 73% 24.3M 1s 2024-04-21T08:45:41.475 INFO:tasks.workunit.client.0.smithi007.stderr: 99700K .......... .......... .......... .......... .......... 73% 24.1M 1s 2024-04-21T08:45:41.475 INFO:tasks.workunit.client.0.smithi007.stderr: 99750K .......... .......... .......... .......... .......... 73% 46.4M 1s 2024-04-21T08:45:41.477 INFO:tasks.workunit.client.0.smithi007.stderr: 99800K .......... .......... .......... .......... .......... 73% 28.5M 1s 2024-04-21T08:45:41.479 INFO:tasks.workunit.client.0.smithi007.stderr: 99850K .......... .......... .......... .......... .......... 73% 30.7M 1s 2024-04-21T08:45:41.481 INFO:tasks.workunit.client.0.smithi007.stderr: 99900K .......... .......... .......... .......... .......... 73% 24.5M 1s 2024-04-21T08:45:41.483 INFO:tasks.workunit.client.0.smithi007.stderr: 99950K .......... .......... .......... .......... .......... 73% 23.9M 1s 2024-04-21T08:45:41.484 INFO:tasks.workunit.client.0.smithi007.stderr:100000K .......... .......... .......... .......... .......... 73% 54.1M 1s 2024-04-21T08:45:41.486 INFO:tasks.workunit.client.0.smithi007.stderr:100050K .......... .......... .......... .......... .......... 73% 23.8M 1s 2024-04-21T08:45:41.487 INFO:tasks.workunit.client.0.smithi007.stderr:100100K .......... .......... .......... .......... .......... 73% 35.0M 1s 2024-04-21T08:45:41.489 INFO:tasks.workunit.client.0.smithi007.stderr:100150K .......... .......... .......... .......... .......... 73% 22.7M 1s 2024-04-21T08:45:41.491 INFO:tasks.workunit.client.0.smithi007.stderr:100200K .......... .......... .......... .......... .......... 73% 22.7M 1s 2024-04-21T08:45:41.493 INFO:tasks.workunit.client.0.smithi007.stderr:100250K .......... .......... .......... .......... .......... 73% 36.2M 1s 2024-04-21T08:45:41.495 INFO:tasks.workunit.client.0.smithi007.stderr:100300K .......... .......... .......... .......... .......... 73% 24.2M 1s 2024-04-21T08:45:41.496 INFO:tasks.workunit.client.0.smithi007.stderr:100350K .......... .......... .......... .......... .......... 73% 42.9M 1s 2024-04-21T08:45:41.497 INFO:tasks.workunit.client.0.smithi007.stderr:100400K .......... .......... .......... .......... .......... 73% 29.0M 1s 2024-04-21T08:45:41.500 INFO:tasks.workunit.client.0.smithi007.stderr:100450K .......... .......... .......... .......... .......... 74% 20.3M 1s 2024-04-21T08:45:41.501 INFO:tasks.workunit.client.0.smithi007.stderr:100500K .......... .......... .......... .......... .......... 74% 51.2M 1s 2024-04-21T08:45:41.503 INFO:tasks.workunit.client.0.smithi007.stderr:100550K .......... .......... .......... .......... .......... 74% 21.0M 1s 2024-04-21T08:45:41.505 INFO:tasks.workunit.client.0.smithi007.stderr:100600K .......... .......... .......... .......... .......... 74% 37.6M 1s 2024-04-21T08:45:41.507 INFO:tasks.workunit.client.0.smithi007.stderr:100650K .......... .......... .......... .......... .......... 74% 23.7M 1s 2024-04-21T08:45:41.509 INFO:tasks.workunit.client.0.smithi007.stderr:100700K .......... .......... .......... .......... .......... 74% 24.9M 1s 2024-04-21T08:45:41.510 INFO:tasks.workunit.client.0.smithi007.stderr:100750K .......... .......... .......... .......... .......... 74% 45.0M 1s 2024-04-21T08:45:41.512 INFO:tasks.workunit.client.0.smithi007.stderr:100800K .......... .......... .......... .......... .......... 74% 24.9M 1s 2024-04-21T08:45:41.513 INFO:tasks.workunit.client.0.smithi007.stderr:100850K .......... .......... .......... .......... .......... 74% 35.4M 1s 2024-04-21T08:45:41.515 INFO:tasks.workunit.client.0.smithi007.stderr:100900K .......... .......... .......... .......... .......... 74% 25.6M 1s 2024-04-21T08:45:41.517 INFO:tasks.workunit.client.0.smithi007.stderr:100950K .......... .......... .......... .......... .......... 74% 23.6M 1s 2024-04-21T08:45:41.518 INFO:tasks.workunit.client.0.smithi007.stderr:101000K .......... .......... .......... .......... .......... 74% 43.0M 1s 2024-04-21T08:45:41.520 INFO:tasks.workunit.client.0.smithi007.stderr:101050K .......... .......... .......... .......... .......... 74% 23.2M 1s 2024-04-21T08:45:41.521 INFO:tasks.workunit.client.0.smithi007.stderr:101100K .......... .......... .......... .......... .......... 74% 41.8M 1s 2024-04-21T08:45:41.524 INFO:tasks.workunit.client.0.smithi007.stderr:101150K .......... .......... .......... .......... .......... 74% 21.8M 1s 2024-04-21T08:45:41.526 INFO:tasks.workunit.client.0.smithi007.stderr:101200K .......... .......... .......... .......... .......... 74% 25.3M 1s 2024-04-21T08:45:41.526 INFO:tasks.workunit.client.0.smithi007.stderr:101250K .......... .......... .......... .......... .......... 74% 50.5M 1s 2024-04-21T08:45:41.529 INFO:tasks.workunit.client.0.smithi007.stderr:101300K .......... .......... .......... .......... .......... 74% 21.8M 1s 2024-04-21T08:45:41.530 INFO:tasks.workunit.client.0.smithi007.stderr:101350K .......... .......... .......... .......... .......... 74% 35.5M 1s 2024-04-21T08:45:41.532 INFO:tasks.workunit.client.0.smithi007.stderr:101400K .......... .......... .......... .......... .......... 74% 23.3M 1s 2024-04-21T08:45:41.534 INFO:tasks.workunit.client.0.smithi007.stderr:101450K .......... .......... .......... .......... .......... 74% 23.2M 1s 2024-04-21T08:45:41.535 INFO:tasks.workunit.client.0.smithi007.stderr:101500K .......... .......... .......... .......... .......... 74% 51.3M 1s 2024-04-21T08:45:41.537 INFO:tasks.workunit.client.0.smithi007.stderr:101550K .......... .......... .......... .......... .......... 74% 22.5M 1s 2024-04-21T08:45:41.539 INFO:tasks.workunit.client.0.smithi007.stderr:101600K .......... .......... .......... .......... .......... 74% 39.1M 1s 2024-04-21T08:45:41.541 INFO:tasks.workunit.client.0.smithi007.stderr:101650K .......... .......... .......... .......... .......... 74% 25.8M 1s 2024-04-21T08:45:41.543 INFO:tasks.workunit.client.0.smithi007.stderr:101700K .......... .......... .......... .......... .......... 74% 23.3M 1s 2024-04-21T08:45:41.544 INFO:tasks.workunit.client.0.smithi007.stderr:101750K .......... .......... .......... .......... .......... 74% 44.9M 1s 2024-04-21T08:45:41.546 INFO:tasks.workunit.client.0.smithi007.stderr:101800K .......... .......... .......... .......... .......... 75% 23.6M 1s 2024-04-21T08:45:41.547 INFO:tasks.workunit.client.0.smithi007.stderr:101850K .......... .......... .......... .......... .......... 75% 37.4M 1s 2024-04-21T08:45:41.549 INFO:tasks.workunit.client.0.smithi007.stderr:101900K .......... .......... .......... .......... .......... 75% 25.5M 1s 2024-04-21T08:45:41.551 INFO:tasks.workunit.client.0.smithi007.stderr:101950K .......... .......... .......... .......... .......... 75% 23.1M 1s 2024-04-21T08:45:41.552 INFO:tasks.workunit.client.0.smithi007.stderr:102000K .......... .......... .......... .......... .......... 75% 47.5M 1s 2024-04-21T08:45:41.554 INFO:tasks.workunit.client.0.smithi007.stderr:102050K .......... .......... .......... .......... .......... 75% 23.0M 1s 2024-04-21T08:45:41.556 INFO:tasks.workunit.client.0.smithi007.stderr:102100K .......... .......... .......... .......... .......... 75% 40.0M 1s 2024-04-21T08:45:41.558 INFO:tasks.workunit.client.0.smithi007.stderr:102150K .......... .......... .......... .......... .......... 75% 23.7M 1s 2024-04-21T08:45:41.560 INFO:tasks.workunit.client.0.smithi007.stderr:102200K .......... .......... .......... .......... .......... 75% 25.5M 1s 2024-04-21T08:45:41.560 INFO:tasks.workunit.client.0.smithi007.stderr:102250K .......... .......... .......... .......... .......... 75% 47.4M 1s 2024-04-21T08:45:41.563 INFO:tasks.workunit.client.0.smithi007.stderr:102300K .......... .......... .......... .......... .......... 75% 22.3M 1s 2024-04-21T08:45:41.564 INFO:tasks.workunit.client.0.smithi007.stderr:102350K .......... .......... .......... .......... .......... 75% 46.3M 1s 2024-04-21T08:45:41.566 INFO:tasks.workunit.client.0.smithi007.stderr:102400K .......... .......... .......... .......... .......... 75% 21.6M 1s 2024-04-21T08:45:41.568 INFO:tasks.workunit.client.0.smithi007.stderr:102450K .......... .......... .......... .......... .......... 75% 25.4M 1s 2024-04-21T08:45:41.569 INFO:tasks.workunit.client.0.smithi007.stderr:102500K .......... .......... .......... .......... .......... 75% 48.0M 1s 2024-04-21T08:45:41.571 INFO:tasks.workunit.client.0.smithi007.stderr:102550K .......... .......... .......... .......... .......... 75% 22.7M 1s 2024-04-21T08:45:41.573 INFO:tasks.workunit.client.0.smithi007.stderr:102600K .......... .......... .......... .......... .......... 75% 32.2M 1s 2024-04-21T08:45:41.574 INFO:tasks.workunit.client.0.smithi007.stderr:102650K .......... .......... .......... .......... .......... 75% 27.9M 1s 2024-04-21T08:45:41.577 INFO:tasks.workunit.client.0.smithi007.stderr:102700K .......... .......... .......... .......... .......... 75% 23.6M 1s 2024-04-21T08:45:41.577 INFO:tasks.workunit.client.0.smithi007.stderr:102750K .......... .......... .......... .......... .......... 75% 52.6M 1s 2024-04-21T08:45:41.579 INFO:tasks.workunit.client.0.smithi007.stderr:102800K .......... .......... .......... .......... .......... 75% 23.6M 1s 2024-04-21T08:45:41.581 INFO:tasks.workunit.client.0.smithi007.stderr:102850K .......... .......... .......... .......... .......... 75% 41.2M 1s 2024-04-21T08:45:41.584 INFO:tasks.workunit.client.0.smithi007.stderr:102900K .......... .......... .......... .......... .......... 75% 19.4M 1s 2024-04-21T08:45:41.585 INFO:tasks.workunit.client.0.smithi007.stderr:102950K .......... .......... .......... .......... .......... 75% 30.4M 1s 2024-04-21T08:45:41.586 INFO:tasks.workunit.client.0.smithi007.stderr:103000K .......... .......... .......... .......... .......... 75% 48.7M 1s 2024-04-21T08:45:41.588 INFO:tasks.workunit.client.0.smithi007.stderr:103050K .......... .......... .......... .......... .......... 75% 25.7M 1s 2024-04-21T08:45:41.589 INFO:tasks.workunit.client.0.smithi007.stderr:103100K .......... .......... .......... .......... .......... 75% 33.0M 1s 2024-04-21T08:45:41.591 INFO:tasks.workunit.client.0.smithi007.stderr:103150K .......... .......... .......... .......... .......... 76% 22.1M 1s 2024-04-21T08:45:41.593 INFO:tasks.workunit.client.0.smithi007.stderr:103200K .......... .......... .......... .......... .......... 76% 27.0M 1s 2024-04-21T08:45:41.594 INFO:tasks.workunit.client.0.smithi007.stderr:103250K .......... .......... .......... .......... .......... 76% 53.3M 1s 2024-04-21T08:45:41.596 INFO:tasks.workunit.client.0.smithi007.stderr:103300K .......... .......... .......... .......... .......... 76% 23.9M 1s 2024-04-21T08:45:41.598 INFO:tasks.workunit.client.0.smithi007.stderr:103350K .......... .......... .......... .......... .......... 76% 23.1M 1s 2024-04-21T08:45:41.599 INFO:tasks.workunit.client.0.smithi007.stderr:103400K .......... .......... .......... .......... .......... 76% 42.9M 1s 2024-04-21T08:45:41.602 INFO:tasks.workunit.client.0.smithi007.stderr:103450K .......... .......... .......... .......... .......... 76% 21.3M 1s 2024-04-21T08:45:41.602 INFO:tasks.workunit.client.0.smithi007.stderr:103500K .......... .......... .......... .......... .......... 76% 55.8M 1s 2024-04-21T08:45:41.605 INFO:tasks.workunit.client.0.smithi007.stderr:103550K .......... .......... .......... .......... .......... 76% 27.5M 1s 2024-04-21T08:45:41.607 INFO:tasks.workunit.client.0.smithi007.stderr:103600K .......... .......... .......... .......... .......... 76% 21.1M 1s 2024-04-21T08:45:41.608 INFO:tasks.workunit.client.0.smithi007.stderr:103650K .......... .......... .......... .......... .......... 76% 43.4M 1s 2024-04-21T08:45:41.610 INFO:tasks.workunit.client.0.smithi007.stderr:103700K .......... .......... .......... .......... .......... 76% 22.6M 1s 2024-04-21T08:45:41.611 INFO:tasks.workunit.client.0.smithi007.stderr:103750K .......... .......... .......... .......... .......... 76% 52.5M 1s 2024-04-21T08:45:41.613 INFO:tasks.workunit.client.0.smithi007.stderr:103800K .......... .......... .......... .......... .......... 76% 23.1M 1s 2024-04-21T08:45:41.615 INFO:tasks.workunit.client.0.smithi007.stderr:103850K .......... .......... .......... .......... .......... 76% 24.3M 1s 2024-04-21T08:45:41.616 INFO:tasks.workunit.client.0.smithi007.stderr:103900K .......... .......... .......... .......... .......... 76% 35.9M 1s 2024-04-21T08:45:41.619 INFO:tasks.workunit.client.0.smithi007.stderr:103950K .......... .......... .......... .......... .......... 76% 42.6M 1s 2024-04-21T08:45:41.619 INFO:tasks.workunit.client.0.smithi007.stderr:104000K .......... .......... .......... .......... .......... 76% 27.6M 1s 2024-04-21T08:45:41.621 INFO:tasks.workunit.client.0.smithi007.stderr:104050K .......... .......... .......... .......... .......... 76% 23.4M 1s 2024-04-21T08:45:41.625 INFO:tasks.workunit.client.0.smithi007.stderr:104100K .......... .......... .......... .......... .......... 76% 23.1M 1s 2024-04-21T08:45:41.625 INFO:tasks.workunit.client.0.smithi007.stderr:104150K .......... .......... .......... .......... .......... 76% 40.8M 1s 2024-04-21T08:45:41.627 INFO:tasks.workunit.client.0.smithi007.stderr:104200K .......... .......... .......... .......... .......... 76% 23.3M 1s 2024-04-21T08:45:41.629 INFO:tasks.workunit.client.0.smithi007.stderr:104250K .......... .......... .......... .......... .......... 76% 52.4M 1s 2024-04-21T08:45:41.630 INFO:tasks.workunit.client.0.smithi007.stderr:104300K .......... .......... .......... .......... .......... 76% 22.4M 1s 2024-04-21T08:45:41.632 INFO:tasks.workunit.client.0.smithi007.stderr:104350K .......... .......... .......... .......... .......... 76% 29.6M 1s 2024-04-21T08:45:41.633 INFO:tasks.workunit.client.0.smithi007.stderr:104400K .......... .......... .......... .......... .......... 76% 33.2M 1s 2024-04-21T08:45:41.635 INFO:tasks.workunit.client.0.smithi007.stderr:104450K .......... .......... .......... .......... .......... 76% 26.3M 1s 2024-04-21T08:45:41.637 INFO:tasks.workunit.client.0.smithi007.stderr:104500K .......... .......... .......... .......... .......... 77% 20.6M 1s 2024-04-21T08:45:41.639 INFO:tasks.workunit.client.0.smithi007.stderr:104550K .......... .......... .......... .......... .......... 77% 43.5M 1s 2024-04-21T08:45:41.640 INFO:tasks.workunit.client.0.smithi007.stderr:104600K .......... .......... .......... .......... .......... 77% 29.7M 1s 2024-04-21T08:45:41.643 INFO:tasks.workunit.client.0.smithi007.stderr:104650K .......... .......... .......... .......... .......... 77% 21.4M 1s 2024-04-21T08:45:41.644 INFO:tasks.workunit.client.0.smithi007.stderr:104700K .......... .......... .......... .......... .......... 77% 28.0M 1s 2024-04-21T08:45:41.646 INFO:tasks.workunit.client.0.smithi007.stderr:104750K .......... .......... .......... .......... .......... 77% 24.2M 1s 2024-04-21T08:45:41.647 INFO:tasks.workunit.client.0.smithi007.stderr:104800K .......... .......... .......... .......... .......... 77% 49.5M 1s 2024-04-21T08:45:41.649 INFO:tasks.workunit.client.0.smithi007.stderr:104850K .......... .......... .......... .......... .......... 77% 23.0M 1s 2024-04-21T08:45:41.651 INFO:tasks.workunit.client.0.smithi007.stderr:104900K .......... .......... .......... .......... .......... 77% 33.0M 1s 2024-04-21T08:45:41.653 INFO:tasks.workunit.client.0.smithi007.stderr:104950K .......... .......... .......... .......... .......... 77% 22.3M 1s 2024-04-21T08:45:41.655 INFO:tasks.workunit.client.0.smithi007.stderr:105000K .......... .......... .......... .......... .......... 77% 23.4M 1s 2024-04-21T08:45:41.656 INFO:tasks.workunit.client.0.smithi007.stderr:105050K .......... .......... .......... .......... .......... 77% 46.5M 1s 2024-04-21T08:45:41.658 INFO:tasks.workunit.client.0.smithi007.stderr:105100K .......... .......... .......... .......... .......... 77% 23.9M 1s 2024-04-21T08:45:41.659 INFO:tasks.workunit.client.0.smithi007.stderr:105150K .......... .......... .......... .......... .......... 77% 35.3M 1s 2024-04-21T08:45:41.661 INFO:tasks.workunit.client.0.smithi007.stderr:105200K .......... .......... .......... .......... .......... 77% 31.0M 1s 2024-04-21T08:45:41.664 INFO:tasks.workunit.client.0.smithi007.stderr:105250K .......... .......... .......... .......... .......... 77% 20.4M 1s 2024-04-21T08:45:41.664 INFO:tasks.workunit.client.0.smithi007.stderr:105300K .......... .......... .......... .......... .......... 77% 49.5M 1s 2024-04-21T08:45:41.667 INFO:tasks.workunit.client.0.smithi007.stderr:105350K .......... .......... .......... .......... .......... 77% 23.7M 1s 2024-04-21T08:45:41.669 INFO:tasks.workunit.client.0.smithi007.stderr:105400K .......... .......... .......... .......... .......... 77% 27.0M 1s 2024-04-21T08:45:41.670 INFO:tasks.workunit.client.0.smithi007.stderr:105450K .......... .......... .......... .......... .......... 77% 33.8M 1s 2024-04-21T08:45:41.672 INFO:tasks.workunit.client.0.smithi007.stderr:105500K .......... .......... .......... .......... .......... 77% 23.2M 1s 2024-04-21T08:45:41.673 INFO:tasks.workunit.client.0.smithi007.stderr:105550K .......... .......... .......... .......... .......... 77% 49.7M 1s 2024-04-21T08:45:41.675 INFO:tasks.workunit.client.0.smithi007.stderr:105600K .......... .......... .......... .......... .......... 77% 23.8M 1s 2024-04-21T08:45:41.677 INFO:tasks.workunit.client.0.smithi007.stderr:105650K .......... .......... .......... .......... .......... 77% 36.7M 1s 2024-04-21T08:45:41.679 INFO:tasks.workunit.client.0.smithi007.stderr:105700K .......... .......... .......... .......... .......... 77% 24.6M 1s 2024-04-21T08:45:41.680 INFO:tasks.workunit.client.0.smithi007.stderr:105750K .......... .......... .......... .......... .......... 77% 30.7M 1s 2024-04-21T08:45:41.681 INFO:tasks.workunit.client.0.smithi007.stderr:105800K .......... .......... .......... .......... .......... 77% 37.4M 1s 2024-04-21T08:45:41.683 INFO:tasks.workunit.client.0.smithi007.stderr:105850K .......... .......... .......... .......... .......... 78% 23.1M 1s 2024-04-21T08:45:41.684 INFO:tasks.workunit.client.0.smithi007.stderr:105900K .......... .......... .......... .......... .......... 78% 42.0M 1s 2024-04-21T08:45:41.687 INFO:tasks.workunit.client.0.smithi007.stderr:105950K .......... .......... .......... .......... .......... 78% 22.2M 1s 2024-04-21T08:45:41.689 INFO:tasks.workunit.client.0.smithi007.stderr:106000K .......... .......... .......... .......... .......... 78% 24.6M 1s 2024-04-21T08:45:41.689 INFO:tasks.workunit.client.0.smithi007.stderr:106050K .......... .......... .......... .......... .......... 78% 51.5M 1s 2024-04-21T08:45:41.692 INFO:tasks.workunit.client.0.smithi007.stderr:106100K .......... .......... .......... .......... .......... 78% 23.4M 1s 2024-04-21T08:45:41.693 INFO:tasks.workunit.client.0.smithi007.stderr:106150K .......... .......... .......... .......... .......... 78% 37.6M 1s 2024-04-21T08:45:41.695 INFO:tasks.workunit.client.0.smithi007.stderr:106200K .......... .......... .......... .......... .......... 78% 22.6M 1s 2024-04-21T08:45:41.697 INFO:tasks.workunit.client.0.smithi007.stderr:106250K .......... .......... .......... .......... .......... 78% 24.8M 1s 2024-04-21T08:45:41.698 INFO:tasks.workunit.client.0.smithi007.stderr:106300K .......... .......... .......... .......... .......... 78% 53.9M 1s 2024-04-21T08:45:41.700 INFO:tasks.workunit.client.0.smithi007.stderr:106350K .......... .......... .......... .......... .......... 78% 22.3M 1s 2024-04-21T08:45:41.702 INFO:tasks.workunit.client.0.smithi007.stderr:106400K .......... .......... .......... .......... .......... 78% 25.0M 1s 2024-04-21T08:45:41.703 INFO:tasks.workunit.client.0.smithi007.stderr:106450K .......... .......... .......... .......... .......... 78% 43.9M 1s 2024-04-21T08:45:41.706 INFO:tasks.workunit.client.0.smithi007.stderr:106500K .......... .......... .......... .......... .......... 78% 19.9M 1s 2024-04-21T08:45:41.707 INFO:tasks.workunit.client.0.smithi007.stderr:106550K .......... .......... .......... .......... .......... 78% 49.1M 1s 2024-04-21T08:45:41.708 INFO:tasks.workunit.client.0.smithi007.stderr:106600K .......... .......... .......... .......... .......... 78% 32.0M 1s 2024-04-21T08:45:41.711 INFO:tasks.workunit.client.0.smithi007.stderr:106650K .......... .......... .......... .......... .......... 78% 20.0M 1s 2024-04-21T08:45:41.712 INFO:tasks.workunit.client.0.smithi007.stderr:106700K .......... .......... .......... .......... .......... 78% 37.6M 1s 2024-04-21T08:45:41.714 INFO:tasks.workunit.client.0.smithi007.stderr:106750K .......... .......... .......... .......... .......... 78% 22.7M 1s 2024-04-21T08:45:41.717 INFO:tasks.workunit.client.0.smithi007.stderr:106800K .......... .......... .......... .......... .......... 78% 20.8M 1s 2024-04-21T08:45:41.717 INFO:tasks.workunit.client.0.smithi007.stderr:106850K .......... .......... .......... .......... .......... 78% 38.9M 1s 2024-04-21T08:45:41.719 INFO:tasks.workunit.client.0.smithi007.stderr:106900K .......... .......... .......... .......... .......... 78% 42.6M 1s 2024-04-21T08:45:41.721 INFO:tasks.workunit.client.0.smithi007.stderr:106950K .......... .......... .......... .......... .......... 78% 22.9M 1s 2024-04-21T08:45:41.723 INFO:tasks.workunit.client.0.smithi007.stderr:107000K .......... .......... .......... .......... .......... 78% 20.3M 1s 2024-04-21T08:45:41.724 INFO:tasks.workunit.client.0.smithi007.stderr:107050K .......... .......... .......... .......... .......... 78% 45.1M 1s 2024-04-21T08:45:41.727 INFO:tasks.workunit.client.0.smithi007.stderr:107100K .......... .......... .......... .......... .......... 78% 22.0M 1s 2024-04-21T08:45:41.728 INFO:tasks.workunit.client.0.smithi007.stderr:107150K .......... .......... .......... .......... .......... 78% 43.3M 1s 2024-04-21T08:45:41.730 INFO:tasks.workunit.client.0.smithi007.stderr:107200K .......... .......... .......... .......... .......... 79% 20.8M 1s 2024-04-21T08:45:41.732 INFO:tasks.workunit.client.0.smithi007.stderr:107250K .......... .......... .......... .......... .......... 79% 25.9M 1s 2024-04-21T08:45:41.733 INFO:tasks.workunit.client.0.smithi007.stderr:107300K .......... .......... .......... .......... .......... 79% 48.4M 1s 2024-04-21T08:45:41.735 INFO:tasks.workunit.client.0.smithi007.stderr:107350K .......... .......... .......... .......... .......... 79% 27.7M 1s 2024-04-21T08:45:41.736 INFO:tasks.workunit.client.0.smithi007.stderr:107400K .......... .......... .......... .......... .......... 79% 28.7M 1s 2024-04-21T08:45:41.738 INFO:tasks.workunit.client.0.smithi007.stderr:107450K .......... .......... .......... .......... .......... 79% 25.4M 1s 2024-04-21T08:45:41.741 INFO:tasks.workunit.client.0.smithi007.stderr:107500K .......... .......... .......... .......... .......... 79% 23.4M 1s 2024-04-21T08:45:41.741 INFO:tasks.workunit.client.0.smithi007.stderr:107550K .......... .......... .......... .......... .......... 79% 47.6M 1s 2024-04-21T08:45:41.743 INFO:tasks.workunit.client.0.smithi007.stderr:107600K .......... .......... .......... .......... .......... 79% 24.2M 1s 2024-04-21T08:45:41.745 INFO:tasks.workunit.client.0.smithi007.stderr:107650K .......... .......... .......... .......... .......... 79% 39.3M 1s 2024-04-21T08:45:41.747 INFO:tasks.workunit.client.0.smithi007.stderr:107700K .......... .......... .......... .......... .......... 79% 23.5M 1s 2024-04-21T08:45:41.748 INFO:tasks.workunit.client.0.smithi007.stderr:107750K .......... .......... .......... .......... .......... 79% 29.0M 1s 2024-04-21T08:45:41.750 INFO:tasks.workunit.client.0.smithi007.stderr:107800K .......... .......... .......... .......... .......... 79% 38.4M 1s 2024-04-21T08:45:41.752 INFO:tasks.workunit.client.0.smithi007.stderr:107850K .......... .......... .......... .......... .......... 79% 21.1M 1s 2024-04-21T08:45:41.754 INFO:tasks.workunit.client.0.smithi007.stderr:107900K .......... .......... .......... .......... .......... 79% 37.6M 1s 2024-04-21T08:45:41.756 INFO:tasks.workunit.client.0.smithi007.stderr:107950K .......... .......... .......... .......... .......... 79% 21.7M 1s 2024-04-21T08:45:41.758 INFO:tasks.workunit.client.0.smithi007.stderr:108000K .......... .......... .......... .......... .......... 79% 20.8M 1s 2024-04-21T08:45:41.759 INFO:tasks.workunit.client.0.smithi007.stderr:108050K .......... .......... .......... .......... .......... 79% 51.5M 1s 2024-04-21T08:45:41.761 INFO:tasks.workunit.client.0.smithi007.stderr:108100K .......... .......... .......... .......... .......... 79% 23.7M 1s 2024-04-21T08:45:41.763 INFO:tasks.workunit.client.0.smithi007.stderr:108150K .......... .......... .......... .......... .......... 79% 41.3M 1s 2024-04-21T08:45:41.765 INFO:tasks.workunit.client.0.smithi007.stderr:108200K .......... .......... .......... .......... .......... 79% 21.0M 1s 2024-04-21T08:45:41.767 INFO:tasks.workunit.client.0.smithi007.stderr:108250K .......... .......... .......... .......... .......... 79% 24.2M 1s 2024-04-21T08:45:41.767 INFO:tasks.workunit.client.0.smithi007.stderr:108300K .......... .......... .......... .......... .......... 79% 47.8M 1s 2024-04-21T08:45:41.770 INFO:tasks.workunit.client.0.smithi007.stderr:108350K .......... .......... .......... .......... .......... 79% 23.0M 1s 2024-04-21T08:45:41.771 INFO:tasks.workunit.client.0.smithi007.stderr:108400K .......... .......... .......... .......... .......... 79% 26.5M 1s 2024-04-21T08:45:41.773 INFO:tasks.workunit.client.0.smithi007.stderr:108450K .......... .......... .......... .......... .......... 79% 38.4M 1s 2024-04-21T08:45:41.775 INFO:tasks.workunit.client.0.smithi007.stderr:108500K .......... .......... .......... .......... .......... 79% 22.5M 1s 2024-04-21T08:45:41.776 INFO:tasks.workunit.client.0.smithi007.stderr:108550K .......... .......... .......... .......... .......... 79% 54.4M 1s 2024-04-21T08:45:41.778 INFO:tasks.workunit.client.0.smithi007.stderr:108600K .......... .......... .......... .......... .......... 80% 23.4M 1s 2024-04-21T08:45:41.780 INFO:tasks.workunit.client.0.smithi007.stderr:108650K .......... .......... .......... .......... .......... 80% 24.6M 1s 2024-04-21T08:45:41.781 INFO:tasks.workunit.client.0.smithi007.stderr:108700K .......... .......... .......... .......... .......... 80% 39.7M 1s 2024-04-21T08:45:41.783 INFO:tasks.workunit.client.0.smithi007.stderr:108750K .......... .......... .......... .......... .......... 80% 22.8M 1s 2024-04-21T08:45:41.784 INFO:tasks.workunit.client.0.smithi007.stderr:108800K .......... .......... .......... .......... .......... 80% 52.6M 1s 2024-04-21T08:45:41.787 INFO:tasks.workunit.client.0.smithi007.stderr:108850K .......... .......... .......... .......... .......... 80% 19.8M 1s 2024-04-21T08:45:41.789 INFO:tasks.workunit.client.0.smithi007.stderr:108900K .......... .......... .......... .......... .......... 80% 22.4M 1s 2024-04-21T08:45:41.790 INFO:tasks.workunit.client.0.smithi007.stderr:108950K .......... .......... .......... .......... .......... 80% 40.5M 1s 2024-04-21T08:45:41.792 INFO:tasks.workunit.client.0.smithi007.stderr:109000K .......... .......... .......... .......... .......... 80% 23.9M 1s 2024-04-21T08:45:41.793 INFO:tasks.workunit.client.0.smithi007.stderr:109050K .......... .......... .......... .......... .......... 80% 43.8M 1s 2024-04-21T08:45:41.795 INFO:tasks.workunit.client.0.smithi007.stderr:109100K .......... .......... .......... .......... .......... 80% 25.0M 1s 2024-04-21T08:45:41.797 INFO:tasks.workunit.client.0.smithi007.stderr:109150K .......... .......... .......... .......... .......... 80% 23.9M 1s 2024-04-21T08:45:41.798 INFO:tasks.workunit.client.0.smithi007.stderr:109200K .......... .......... .......... .......... .......... 80% 40.0M 1s 2024-04-21T08:45:41.801 INFO:tasks.workunit.client.0.smithi007.stderr:109250K .......... .......... .......... .......... .......... 80% 22.6M 1s 2024-04-21T08:45:41.803 INFO:tasks.workunit.client.0.smithi007.stderr:109300K .......... .......... .......... .......... .......... 80% 54.3M 1s 2024-04-21T08:45:41.804 INFO:tasks.workunit.client.0.smithi007.stderr:109350K .......... .......... .......... .......... .......... 80% 24.5M 1s 2024-04-21T08:45:41.805 INFO:tasks.workunit.client.0.smithi007.stderr:109400K .......... .......... .......... .......... .......... 80% 23.9M 1s 2024-04-21T08:45:41.807 INFO:tasks.workunit.client.0.smithi007.stderr:109450K .......... .......... .......... .......... .......... 80% 39.4M 1s 2024-04-21T08:45:41.809 INFO:tasks.workunit.client.0.smithi007.stderr:109500K .......... .......... .......... .......... .......... 80% 22.0M 1s 2024-04-21T08:45:41.811 INFO:tasks.workunit.client.0.smithi007.stderr:109550K .......... .......... .......... .......... .......... 80% 58.1M 1s 2024-04-21T08:45:41.812 INFO:tasks.workunit.client.0.smithi007.stderr:109600K .......... .......... .......... .......... .......... 80% 24.1M 1s 2024-04-21T08:45:41.814 INFO:tasks.workunit.client.0.smithi007.stderr:109650K .......... .......... .......... .......... .......... 80% 22.8M 1s 2024-04-21T08:45:41.815 INFO:tasks.workunit.client.0.smithi007.stderr:109700K .......... .......... .......... .......... .......... 80% 34.8M 1s 2024-04-21T08:45:41.818 INFO:tasks.workunit.client.0.smithi007.stderr:109750K .......... .......... .......... .......... .......... 80% 22.2M 1s 2024-04-21T08:45:41.820 INFO:tasks.workunit.client.0.smithi007.stderr:109800K .......... .......... .......... .......... .......... 80% 25.3M 1s 2024-04-21T08:45:41.820 INFO:tasks.workunit.client.0.smithi007.stderr:109850K .......... .......... .......... .......... .......... 80% 44.6M 1s 2024-04-21T08:45:41.823 INFO:tasks.workunit.client.0.smithi007.stderr:109900K .......... .......... .......... .......... .......... 80% 24.5M 1s 2024-04-21T08:45:41.824 INFO:tasks.workunit.client.0.smithi007.stderr:109950K .......... .......... .......... .......... .......... 81% 46.4M 1s 2024-04-21T08:45:41.826 INFO:tasks.workunit.client.0.smithi007.stderr:110000K .......... .......... .......... .......... .......... 81% 19.1M 1s 2024-04-21T08:45:41.828 INFO:tasks.workunit.client.0.smithi007.stderr:110050K .......... .......... .......... .......... .......... 81% 24.2M 1s 2024-04-21T08:45:41.829 INFO:tasks.workunit.client.0.smithi007.stderr:110100K .......... .......... .......... .......... .......... 81% 47.2M 1s 2024-04-21T08:45:41.831 INFO:tasks.workunit.client.0.smithi007.stderr:110150K .......... .......... .......... .......... .......... 81% 31.5M 1s 2024-04-21T08:45:41.833 INFO:tasks.workunit.client.0.smithi007.stderr:110200K .......... .......... .......... .......... .......... 81% 28.9M 1s 2024-04-21T08:45:41.835 INFO:tasks.workunit.client.0.smithi007.stderr:110250K .......... .......... .......... .......... .......... 81% 22.0M 1s 2024-04-21T08:45:41.836 INFO:tasks.workunit.client.0.smithi007.stderr:110300K .......... .......... .......... .......... .......... 81% 38.3M 1s 2024-04-21T08:45:41.838 INFO:tasks.workunit.client.0.smithi007.stderr:110350K .......... .......... .......... .......... .......... 81% 28.0M 1s 2024-04-21T08:45:41.840 INFO:tasks.workunit.client.0.smithi007.stderr:110400K .......... .......... .......... .......... .......... 81% 25.5M 1s 2024-04-21T08:45:41.841 INFO:tasks.workunit.client.0.smithi007.stderr:110450K .......... .......... .......... .......... .......... 81% 42.4M 1s 2024-04-21T08:45:41.843 INFO:tasks.workunit.client.0.smithi007.stderr:110500K .......... .......... .......... .......... .......... 81% 23.4M 1s 2024-04-21T08:45:41.845 INFO:tasks.workunit.client.0.smithi007.stderr:110550K .......... .......... .......... .......... .......... 81% 21.6M 1s 2024-04-21T08:45:41.846 INFO:tasks.workunit.client.0.smithi007.stderr:110600K .......... .......... .......... .......... .......... 81% 42.8M 1s 2024-04-21T08:45:41.848 INFO:tasks.workunit.client.0.smithi007.stderr:110650K .......... .......... .......... .......... .......... 81% 30.2M 1s 2024-04-21T08:45:41.849 INFO:tasks.workunit.client.0.smithi007.stderr:110700K .......... .......... .......... .......... .......... 81% 32.1M 1s 2024-04-21T08:45:41.852 INFO:tasks.workunit.client.0.smithi007.stderr:110750K .......... .......... .......... .......... .......... 81% 23.0M 1s 2024-04-21T08:45:41.854 INFO:tasks.workunit.client.0.smithi007.stderr:110800K .......... .......... .......... .......... .......... 81% 23.8M 1s 2024-04-21T08:45:41.855 INFO:tasks.workunit.client.0.smithi007.stderr:110850K .......... .......... .......... .......... .......... 81% 42.0M 1s 2024-04-21T08:45:41.857 INFO:tasks.workunit.client.0.smithi007.stderr:110900K .......... .......... .......... .......... .......... 81% 24.9M 1s 2024-04-21T08:45:41.858 INFO:tasks.workunit.client.0.smithi007.stderr:110950K .......... .......... .......... .......... .......... 81% 37.7M 1s 2024-04-21T08:45:41.860 INFO:tasks.workunit.client.0.smithi007.stderr:111000K .......... .......... .......... .......... .......... 81% 24.4M 1s 2024-04-21T08:45:41.862 INFO:tasks.workunit.client.0.smithi007.stderr:111050K .......... .......... .......... .......... .......... 81% 24.0M 1s 2024-04-21T08:45:41.863 INFO:tasks.workunit.client.0.smithi007.stderr:111100K .......... .......... .......... .......... .......... 81% 52.6M 1s 2024-04-21T08:45:41.865 INFO:tasks.workunit.client.0.smithi007.stderr:111150K .......... .......... .......... .......... .......... 81% 22.8M 1s 2024-04-21T08:45:41.866 INFO:tasks.workunit.client.0.smithi007.stderr:111200K .......... .......... .......... .......... .......... 81% 39.2M 1s 2024-04-21T08:45:41.868 INFO:tasks.workunit.client.0.smithi007.stderr:111250K .......... .......... .......... .......... .......... 81% 23.2M 1s 2024-04-21T08:45:41.870 INFO:tasks.workunit.client.0.smithi007.stderr:111300K .......... .......... .......... .......... .......... 82% 24.3M 1s 2024-04-21T08:45:41.872 INFO:tasks.workunit.client.0.smithi007.stderr:111350K .......... .......... .......... .......... .......... 82% 48.7M 1s 2024-04-21T08:45:41.873 INFO:tasks.workunit.client.0.smithi007.stderr:111400K .......... .......... .......... .......... .......... 82% 24.8M 1s 2024-04-21T08:45:41.875 INFO:tasks.workunit.client.0.smithi007.stderr:111450K .......... .......... .......... .......... .......... 82% 39.3M 1s 2024-04-21T08:45:41.877 INFO:tasks.workunit.client.0.smithi007.stderr:111500K .......... .......... .......... .......... .......... 82% 22.1M 1s 2024-04-21T08:45:41.879 INFO:tasks.workunit.client.0.smithi007.stderr:111550K .......... .......... .......... .......... .......... 82% 24.6M 1s 2024-04-21T08:45:41.880 INFO:tasks.workunit.client.0.smithi007.stderr:111600K .......... .......... .......... .......... .......... 82% 52.6M 1s 2024-04-21T08:45:41.882 INFO:tasks.workunit.client.0.smithi007.stderr:111650K .......... .......... .......... .......... .......... 82% 24.7M 1s 2024-04-21T08:45:41.883 INFO:tasks.workunit.client.0.smithi007.stderr:111700K .......... .......... .......... .......... .......... 82% 40.2M 1s 2024-04-21T08:45:41.885 INFO:tasks.workunit.client.0.smithi007.stderr:111750K .......... .......... .......... .......... .......... 82% 22.6M 1s 2024-04-21T08:45:41.887 INFO:tasks.workunit.client.0.smithi007.stderr:111800K .......... .......... .......... .......... .......... 82% 25.0M 1s 2024-04-21T08:45:41.888 INFO:tasks.workunit.client.0.smithi007.stderr:111850K .......... .......... .......... .......... .......... 82% 45.9M 1s 2024-04-21T08:45:41.890 INFO:tasks.workunit.client.0.smithi007.stderr:111900K .......... .......... .......... .......... .......... 82% 24.2M 1s 2024-04-21T08:45:41.891 INFO:tasks.workunit.client.0.smithi007.stderr:111950K .......... .......... .......... .......... .......... 82% 38.2M 1s 2024-04-21T08:45:41.894 INFO:tasks.workunit.client.0.smithi007.stderr:112000K .......... .......... .......... .......... .......... 82% 22.6M 1s 2024-04-21T08:45:41.896 INFO:tasks.workunit.client.0.smithi007.stderr:112050K .......... .......... .......... .......... .......... 82% 23.7M 1s 2024-04-21T08:45:41.897 INFO:tasks.workunit.client.0.smithi007.stderr:112100K .......... .......... .......... .......... .......... 82% 50.7M 1s 2024-04-21T08:45:41.898 INFO:tasks.workunit.client.0.smithi007.stderr:112150K .......... .......... .......... .......... .......... 82% 25.2M 1s 2024-04-21T08:45:41.900 INFO:tasks.workunit.client.0.smithi007.stderr:112200K .......... .......... .......... .......... .......... 82% 35.3M 1s 2024-04-21T08:45:41.903 INFO:tasks.workunit.client.0.smithi007.stderr:112250K .......... .......... .......... .......... .......... 82% 21.1M 1s 2024-04-21T08:45:41.904 INFO:tasks.workunit.client.0.smithi007.stderr:112300K .......... .......... .......... .......... .......... 82% 28.5M 1s 2024-04-21T08:45:41.905 INFO:tasks.workunit.client.0.smithi007.stderr:112350K .......... .......... .......... .......... .......... 82% 48.7M 1s 2024-04-21T08:45:41.907 INFO:tasks.workunit.client.0.smithi007.stderr:112400K .......... .......... .......... .......... .......... 82% 23.3M 1s 2024-04-21T08:45:41.908 INFO:tasks.workunit.client.0.smithi007.stderr:112450K .......... .......... .......... .......... .......... 82% 41.9M 1s 2024-04-21T08:45:41.911 INFO:tasks.workunit.client.0.smithi007.stderr:112500K .......... .......... .......... .......... .......... 82% 23.6M 1s 2024-04-21T08:45:41.912 INFO:tasks.workunit.client.0.smithi007.stderr:112550K .......... .......... .......... .......... .......... 82% 24.4M 1s 2024-04-21T08:45:41.913 INFO:tasks.workunit.client.0.smithi007.stderr:112600K .......... .......... .......... .......... .......... 82% 51.1M 1s 2024-04-21T08:45:41.915 INFO:tasks.workunit.client.0.smithi007.stderr:112650K .......... .......... .......... .......... .......... 83% 22.7M 1s 2024-04-21T08:45:41.917 INFO:tasks.workunit.client.0.smithi007.stderr:112700K .......... .......... .......... .......... .......... 83% 38.7M 1s 2024-04-21T08:45:41.919 INFO:tasks.workunit.client.0.smithi007.stderr:112750K .......... .......... .......... .......... .......... 83% 25.9M 1s 2024-04-21T08:45:41.921 INFO:tasks.workunit.client.0.smithi007.stderr:112800K .......... .......... .......... .......... .......... 83% 24.0M 1s 2024-04-21T08:45:41.921 INFO:tasks.workunit.client.0.smithi007.stderr:112850K .......... .......... .......... .......... .......... 83% 53.5M 1s 2024-04-21T08:45:41.924 INFO:tasks.workunit.client.0.smithi007.stderr:112900K .......... .......... .......... .......... .......... 83% 22.6M 1s 2024-04-21T08:45:41.926 INFO:tasks.workunit.client.0.smithi007.stderr:112950K .......... .......... .......... .......... .......... 83% 39.7M 1s 2024-04-21T08:45:41.928 INFO:tasks.workunit.client.0.smithi007.stderr:113000K .......... .......... .......... .......... .......... 83% 23.8M 1s 2024-04-21T08:45:41.928 INFO:tasks.workunit.client.0.smithi007.stderr:113050K .......... .......... .......... .......... .......... 83% 35.3M 1s 2024-04-21T08:45:41.930 INFO:tasks.workunit.client.0.smithi007.stderr:113100K .......... .......... .......... .......... .......... 83% 27.7M 1s 2024-04-21T08:45:41.932 INFO:tasks.workunit.client.0.smithi007.stderr:113150K .......... .......... .......... .......... .......... 83% 31.3M 1s 2024-04-21T08:45:41.935 INFO:tasks.workunit.client.0.smithi007.stderr:113200K .......... .......... .......... .......... .......... 83% 27.7M 1s 2024-04-21T08:45:41.935 INFO:tasks.workunit.client.0.smithi007.stderr:113250K .......... .......... .......... .......... .......... 83% 24.5M 1s 2024-04-21T08:45:41.938 INFO:tasks.workunit.client.0.smithi007.stderr:113300K .......... .......... .......... .......... .......... 83% 22.0M 1s 2024-04-21T08:45:41.939 INFO:tasks.workunit.client.0.smithi007.stderr:113350K .......... .......... .......... .......... .......... 83% 46.4M 1s 2024-04-21T08:45:41.941 INFO:tasks.workunit.client.0.smithi007.stderr:113400K .......... .......... .......... .......... .......... 83% 19.9M 1s 2024-04-21T08:45:41.943 INFO:tasks.workunit.client.0.smithi007.stderr:113450K .......... .......... .......... .......... .......... 83% 25.5M 1s 2024-04-21T08:45:41.944 INFO:tasks.workunit.client.0.smithi007.stderr:113500K .......... .......... .......... .......... .......... 83% 42.1M 1s 2024-04-21T08:45:41.947 INFO:tasks.workunit.client.0.smithi007.stderr:113550K .......... .......... .......... .......... .......... 83% 22.1M 1s 2024-04-21T08:45:41.948 INFO:tasks.workunit.client.0.smithi007.stderr:113600K .......... .......... .......... .......... .......... 83% 42.7M 1s 2024-04-21T08:45:41.950 INFO:tasks.workunit.client.0.smithi007.stderr:113650K .......... .......... .......... .......... .......... 83% 22.4M 1s 2024-04-21T08:45:41.952 INFO:tasks.workunit.client.0.smithi007.stderr:113700K .......... .......... .......... .......... .......... 83% 23.1M 1s 2024-04-21T08:45:41.953 INFO:tasks.workunit.client.0.smithi007.stderr:113750K .......... .......... .......... .......... .......... 83% 38.2M 1s 2024-04-21T08:45:41.955 INFO:tasks.workunit.client.0.smithi007.stderr:113800K .......... .......... .......... .......... .......... 83% 23.4M 1s 2024-04-21T08:45:41.957 INFO:tasks.workunit.client.0.smithi007.stderr:113850K .......... .......... .......... .......... .......... 83% 49.0M 1s 2024-04-21T08:45:41.958 INFO:tasks.workunit.client.0.smithi007.stderr:113900K .......... .......... .......... .......... .......... 83% 22.9M 1s 2024-04-21T08:45:41.960 INFO:tasks.workunit.client.0.smithi007.stderr:113950K .......... .......... .......... .......... .......... 83% 31.2M 1s 2024-04-21T08:45:41.961 INFO:tasks.workunit.client.0.smithi007.stderr:114000K .......... .......... .......... .......... .......... 84% 33.1M 1s 2024-04-21T08:45:41.964 INFO:tasks.workunit.client.0.smithi007.stderr:114050K .......... .......... .......... .......... .......... 84% 22.1M 1s 2024-04-21T08:45:41.965 INFO:tasks.workunit.client.0.smithi007.stderr:114100K .......... .......... .......... .......... .......... 84% 46.4M 1s 2024-04-21T08:45:41.966 INFO:tasks.workunit.client.0.smithi007.stderr:114150K .......... .......... .......... .......... .......... 84% 24.6M 1s 2024-04-21T08:45:41.969 INFO:tasks.workunit.client.0.smithi007.stderr:114200K .......... .......... .......... .......... .......... 84% 23.1M 1s 2024-04-21T08:45:41.970 INFO:tasks.workunit.client.0.smithi007.stderr:114250K .......... .......... .......... .......... .......... 84% 38.7M 1s 2024-04-21T08:45:41.972 INFO:tasks.workunit.client.0.smithi007.stderr:114300K .......... .......... .......... .......... .......... 84% 23.9M 1s 2024-04-21T08:45:41.973 INFO:tasks.workunit.client.0.smithi007.stderr:114350K .......... .......... .......... .......... .......... 84% 54.0M 1s 2024-04-21T08:45:41.975 INFO:tasks.workunit.client.0.smithi007.stderr:114400K .......... .......... .......... .......... .......... 84% 24.1M 1s 2024-04-21T08:45:41.977 INFO:tasks.workunit.client.0.smithi007.stderr:114450K .......... .......... .......... .......... .......... 84% 23.9M 1s 2024-04-21T08:45:41.978 INFO:tasks.workunit.client.0.smithi007.stderr:114500K .......... .......... .......... .......... .......... 84% 39.7M 1s 2024-04-21T08:45:41.980 INFO:tasks.workunit.client.0.smithi007.stderr:114550K .......... .......... .......... .......... .......... 84% 22.9M 1s 2024-04-21T08:45:41.983 INFO:tasks.workunit.client.0.smithi007.stderr:114600K .......... .......... .......... .......... .......... 84% 59.0M 1s 2024-04-21T08:45:41.983 INFO:tasks.workunit.client.0.smithi007.stderr:114650K .......... .......... .......... .......... .......... 84% 27.1M 1s 2024-04-21T08:45:41.986 INFO:tasks.workunit.client.0.smithi007.stderr:114700K .......... .......... .......... .......... .......... 84% 19.6M 1s 2024-04-21T08:45:41.987 INFO:tasks.workunit.client.0.smithi007.stderr:114750K .......... .......... .......... .......... .......... 84% 41.7M 1s 2024-04-21T08:45:41.989 INFO:tasks.workunit.client.0.smithi007.stderr:114800K .......... .......... .......... .......... .......... 84% 23.5M 1s 2024-04-21T08:45:41.991 INFO:tasks.workunit.client.0.smithi007.stderr:114850K .......... .......... .......... .......... .......... 84% 48.4M 1s 2024-04-21T08:45:41.992 INFO:tasks.workunit.client.0.smithi007.stderr:114900K .......... .......... .......... .......... .......... 84% 24.4M 1s 2024-04-21T08:45:41.994 INFO:tasks.workunit.client.0.smithi007.stderr:114950K .......... .......... .......... .......... .......... 84% 26.7M 1s 2024-04-21T08:45:41.995 INFO:tasks.workunit.client.0.smithi007.stderr:115000K .......... .......... .......... .......... .......... 84% 32.1M 1s 2024-04-21T08:45:41.998 INFO:tasks.workunit.client.0.smithi007.stderr:115050K .......... .......... .......... .......... .......... 84% 24.9M 1s 2024-04-21T08:45:41.999 INFO:tasks.workunit.client.0.smithi007.stderr:115100K .......... .......... .......... .......... .......... 84% 23.7M 1s 2024-04-21T08:45:42.000 INFO:tasks.workunit.client.0.smithi007.stderr:115150K .......... .......... .......... .......... .......... 84% 80.1M 1s 2024-04-21T08:45:42.003 INFO:tasks.workunit.client.0.smithi007.stderr:115200K .......... .......... .......... .......... .......... 84% 16.9M 1s 2024-04-21T08:45:42.004 INFO:tasks.workunit.client.0.smithi007.stderr:115250K .......... .......... .......... .......... .......... 84% 42.4M 1s 2024-04-21T08:45:42.007 INFO:tasks.workunit.client.0.smithi007.stderr:115300K .......... .......... .......... .......... .......... 84% 22.8M 1s 2024-04-21T08:45:42.009 INFO:tasks.workunit.client.0.smithi007.stderr:115350K .......... .......... .......... .......... .......... 85% 24.1M 1s 2024-04-21T08:45:42.009 INFO:tasks.workunit.client.0.smithi007.stderr:115400K .......... .......... .......... .......... .......... 85% 77.9M 1s 2024-04-21T08:45:42.011 INFO:tasks.workunit.client.0.smithi007.stderr:115450K .......... .......... .......... .......... .......... 85% 20.3M 1s 2024-04-21T08:45:42.012 INFO:tasks.workunit.client.0.smithi007.stderr:115500K .......... .......... .......... .......... .......... 85% 44.2M 1s 2024-04-21T08:45:42.015 INFO:tasks.workunit.client.0.smithi007.stderr:115550K .......... .......... .......... .......... .......... 85% 21.1M 1s 2024-04-21T08:45:42.017 INFO:tasks.workunit.client.0.smithi007.stderr:115600K .......... .......... .......... .......... .......... 85% 21.1M 1s 2024-04-21T08:45:42.018 INFO:tasks.workunit.client.0.smithi007.stderr:115650K .......... .......... .......... .......... .......... 85% 57.0M 1s 2024-04-21T08:45:42.020 INFO:tasks.workunit.client.0.smithi007.stderr:115700K .......... .......... .......... .......... .......... 85% 20.8M 1s 2024-04-21T08:45:42.021 INFO:tasks.workunit.client.0.smithi007.stderr:115750K .......... .......... .......... .......... .......... 85% 41.0M 1s 2024-04-21T08:45:42.024 INFO:tasks.workunit.client.0.smithi007.stderr:115800K .......... .......... .......... .......... .......... 85% 19.4M 1s 2024-04-21T08:45:42.026 INFO:tasks.workunit.client.0.smithi007.stderr:115850K .......... .......... .......... .......... .......... 85% 23.6M 1s 2024-04-21T08:45:42.027 INFO:tasks.workunit.client.0.smithi007.stderr:115900K .......... .......... .......... .......... .......... 85% 61.7M 1s 2024-04-21T08:45:42.029 INFO:tasks.workunit.client.0.smithi007.stderr:115950K .......... .......... .......... .......... .......... 85% 20.3M 1s 2024-04-21T08:45:42.030 INFO:tasks.workunit.client.0.smithi007.stderr:116000K .......... .......... .......... .......... .......... 85% 44.5M 1s 2024-04-21T08:45:42.032 INFO:tasks.workunit.client.0.smithi007.stderr:116050K .......... .......... .......... .......... .......... 85% 22.6M 1s 2024-04-21T08:45:42.034 INFO:tasks.workunit.client.0.smithi007.stderr:116100K .......... .......... .......... .......... .......... 85% 23.4M 1s 2024-04-21T08:45:42.035 INFO:tasks.workunit.client.0.smithi007.stderr:116150K .......... .......... .......... .......... .......... 85% 52.7M 1s 2024-04-21T08:45:42.037 INFO:tasks.workunit.client.0.smithi007.stderr:116200K .......... .......... .......... .......... .......... 85% 24.2M 1s 2024-04-21T08:45:42.038 INFO:tasks.workunit.client.0.smithi007.stderr:116250K .......... .......... .......... .......... .......... 85% 38.3M 1s 2024-04-21T08:45:42.041 INFO:tasks.workunit.client.0.smithi007.stderr:116300K .......... .......... .......... .......... .......... 85% 23.1M 1s 2024-04-21T08:45:42.043 INFO:tasks.workunit.client.0.smithi007.stderr:116350K .......... .......... .......... .......... .......... 85% 23.1M 1s 2024-04-21T08:45:42.044 INFO:tasks.workunit.client.0.smithi007.stderr:116400K .......... .......... .......... .......... .......... 85% 50.4M 1s 2024-04-21T08:45:42.046 INFO:tasks.workunit.client.0.smithi007.stderr:116450K .......... .......... .......... .......... .......... 85% 24.9M 1s 2024-04-21T08:45:42.047 INFO:tasks.workunit.client.0.smithi007.stderr:116500K .......... .......... .......... .......... .......... 85% 35.7M 1s 2024-04-21T08:45:42.049 INFO:tasks.workunit.client.0.smithi007.stderr:116550K .......... .......... .......... .......... .......... 85% 22.4M 1s 2024-04-21T08:45:42.051 INFO:tasks.workunit.client.0.smithi007.stderr:116600K .......... .......... .......... .......... .......... 85% 22.8M 1s 2024-04-21T08:45:42.052 INFO:tasks.workunit.client.0.smithi007.stderr:116650K .......... .......... .......... .......... .......... 85% 45.1M 1s 2024-04-21T08:45:42.055 INFO:tasks.workunit.client.0.smithi007.stderr:116700K .......... .......... .......... .......... .......... 86% 21.1M 1s 2024-04-21T08:45:42.056 INFO:tasks.workunit.client.0.smithi007.stderr:116750K .......... .......... .......... .......... .......... 86% 37.6M 1s 2024-04-21T08:45:42.058 INFO:tasks.workunit.client.0.smithi007.stderr:116800K .......... .......... .......... .......... .......... 86% 22.3M 1s 2024-04-21T08:45:42.060 INFO:tasks.workunit.client.0.smithi007.stderr:116850K .......... .......... .......... .......... .......... 86% 23.9M 1s 2024-04-21T08:45:42.061 INFO:tasks.workunit.client.0.smithi007.stderr:116900K .......... .......... .......... .......... .......... 86% 44.3M 1s 2024-04-21T08:45:42.063 INFO:tasks.workunit.client.0.smithi007.stderr:116950K .......... .......... .......... .......... .......... 86% 23.4M 1s 2024-04-21T08:45:42.065 INFO:tasks.workunit.client.0.smithi007.stderr:117000K .......... .......... .......... .......... .......... 86% 42.3M 1s 2024-04-21T08:45:42.067 INFO:tasks.workunit.client.0.smithi007.stderr:117050K .......... .......... .......... .......... .......... 86% 22.2M 1s 2024-04-21T08:45:42.069 INFO:tasks.workunit.client.0.smithi007.stderr:117100K .......... .......... .......... .......... .......... 86% 25.4M 1s 2024-04-21T08:45:42.070 INFO:tasks.workunit.client.0.smithi007.stderr:117150K .......... .......... .......... .......... .......... 86% 45.6M 1s 2024-04-21T08:45:42.072 INFO:tasks.workunit.client.0.smithi007.stderr:117200K .......... .......... .......... .......... .......... 86% 23.6M 1s 2024-04-21T08:45:42.073 INFO:tasks.workunit.client.0.smithi007.stderr:117250K .......... .......... .......... .......... .......... 86% 45.1M 1s 2024-04-21T08:45:42.075 INFO:tasks.workunit.client.0.smithi007.stderr:117300K .......... .......... .......... .......... .......... 86% 23.7M 1s 2024-04-21T08:45:42.077 INFO:tasks.workunit.client.0.smithi007.stderr:117350K .......... .......... .......... .......... .......... 86% 22.8M 1s 2024-04-21T08:45:42.078 INFO:tasks.workunit.client.0.smithi007.stderr:117400K .......... .......... .......... .......... .......... 86% 50.0M 1s 2024-04-21T08:45:42.080 INFO:tasks.workunit.client.0.smithi007.stderr:117450K .......... .......... .......... .......... .......... 86% 23.6M 1s 2024-04-21T08:45:42.081 INFO:tasks.workunit.client.0.smithi007.stderr:117500K .......... .......... .......... .......... .......... 86% 41.2M 1s 2024-04-21T08:45:42.083 INFO:tasks.workunit.client.0.smithi007.stderr:117550K .......... .......... .......... .......... .......... 86% 23.2M 1s 2024-04-21T08:45:42.086 INFO:tasks.workunit.client.0.smithi007.stderr:117600K .......... .......... .......... .......... .......... 86% 20.2M 1s 2024-04-21T08:45:42.087 INFO:tasks.workunit.client.0.smithi007.stderr:117650K .......... .......... .......... .......... .......... 86% 53.8M 1s 2024-04-21T08:45:42.089 INFO:tasks.workunit.client.0.smithi007.stderr:117700K .......... .......... .......... .......... .......... 86% 21.9M 1s 2024-04-21T08:45:42.090 INFO:tasks.workunit.client.0.smithi007.stderr:117750K .......... .......... .......... .......... .......... 86% 36.5M 1s 2024-04-21T08:45:42.093 INFO:tasks.workunit.client.0.smithi007.stderr:117800K .......... .......... .......... .......... .......... 86% 22.0M 1s 2024-04-21T08:45:42.095 INFO:tasks.workunit.client.0.smithi007.stderr:117850K .......... .......... .......... .......... .......... 86% 25.7M 1s 2024-04-21T08:45:42.095 INFO:tasks.workunit.client.0.smithi007.stderr:117900K .......... .......... .......... .......... .......... 86% 45.7M 1s 2024-04-21T08:45:42.097 INFO:tasks.workunit.client.0.smithi007.stderr:117950K .......... .......... .......... .......... .......... 86% 25.1M 1s 2024-04-21T08:45:42.099 INFO:tasks.workunit.client.0.smithi007.stderr:118000K .......... .......... .......... .......... .......... 86% 35.6M 1s 2024-04-21T08:45:42.101 INFO:tasks.workunit.client.0.smithi007.stderr:118050K .......... .......... .......... .......... .......... 86% 24.8M 1s 2024-04-21T08:45:42.103 INFO:tasks.workunit.client.0.smithi007.stderr:118100K .......... .......... .......... .......... .......... 87% 23.9M 1s 2024-04-21T08:45:42.104 INFO:tasks.workunit.client.0.smithi007.stderr:118150K .......... .......... .......... .......... .......... 87% 46.1M 1s 2024-04-21T08:45:42.106 INFO:tasks.workunit.client.0.smithi007.stderr:118200K .......... .......... .......... .......... .......... 87% 23.6M 1s 2024-04-21T08:45:42.108 INFO:tasks.workunit.client.0.smithi007.stderr:118250K .......... .......... .......... .......... .......... 87% 39.5M 1s 2024-04-21T08:45:42.109 INFO:tasks.workunit.client.0.smithi007.stderr:118300K .......... .......... .......... .......... .......... 87% 25.6M 1s 2024-04-21T08:45:42.111 INFO:tasks.workunit.client.0.smithi007.stderr:118350K .......... .......... .......... .......... .......... 87% 23.3M 1s 2024-04-21T08:45:42.112 INFO:tasks.workunit.client.0.smithi007.stderr:118400K .......... .......... .......... .......... .......... 87% 51.0M 1s 2024-04-21T08:45:42.114 INFO:tasks.workunit.client.0.smithi007.stderr:118450K .......... .......... .......... .......... .......... 87% 21.3M 1s 2024-04-21T08:45:42.116 INFO:tasks.workunit.client.0.smithi007.stderr:118500K .......... .......... .......... .......... .......... 87% 39.0M 1s 2024-04-21T08:45:42.118 INFO:tasks.workunit.client.0.smithi007.stderr:118550K .......... .......... .......... .......... .......... 87% 23.4M 1s 2024-04-21T08:45:42.120 INFO:tasks.workunit.client.0.smithi007.stderr:118600K .......... .......... .......... .......... .......... 87% 23.1M 1s 2024-04-21T08:45:42.121 INFO:tasks.workunit.client.0.smithi007.stderr:118650K .......... .......... .......... .......... .......... 87% 46.1M 1s 2024-04-21T08:45:42.123 INFO:tasks.workunit.client.0.smithi007.stderr:118700K .......... .......... .......... .......... .......... 87% 25.7M 1s 2024-04-21T08:45:42.125 INFO:tasks.workunit.client.0.smithi007.stderr:118750K .......... .......... .......... .......... .......... 87% 22.4M 1s 2024-04-21T08:45:42.127 INFO:tasks.workunit.client.0.smithi007.stderr:118800K .......... .......... .......... .......... .......... 87% 31.8M 1s 2024-04-21T08:45:42.129 INFO:tasks.workunit.client.0.smithi007.stderr:118850K .......... .......... .......... .......... .......... 87% 25.0M 1s 2024-04-21T08:45:42.130 INFO:tasks.workunit.client.0.smithi007.stderr:118900K .......... .......... .......... .......... .......... 87% 43.3M 1s 2024-04-21T08:45:42.132 INFO:tasks.workunit.client.0.smithi007.stderr:118950K .......... .......... .......... .......... .......... 87% 25.1M 1s 2024-04-21T08:45:42.134 INFO:tasks.workunit.client.0.smithi007.stderr:119000K .......... .......... .......... .......... .......... 87% 24.4M 1s 2024-04-21T08:45:42.135 INFO:tasks.workunit.client.0.smithi007.stderr:119050K .......... .......... .......... .......... .......... 87% 51.6M 1s 2024-04-21T08:45:42.137 INFO:tasks.workunit.client.0.smithi007.stderr:119100K .......... .......... .......... .......... .......... 87% 21.9M 1s 2024-04-21T08:45:42.138 INFO:tasks.workunit.client.0.smithi007.stderr:119150K .......... .......... .......... .......... .......... 87% 47.0M 1s 2024-04-21T08:45:42.140 INFO:tasks.workunit.client.0.smithi007.stderr:119200K .......... .......... .......... .......... .......... 87% 23.3M 1s 2024-04-21T08:45:42.142 INFO:tasks.workunit.client.0.smithi007.stderr:119250K .......... .......... .......... .......... .......... 87% 23.4M 1s 2024-04-21T08:45:42.144 INFO:tasks.workunit.client.0.smithi007.stderr:119300K .......... .......... .......... .......... .......... 87% 30.3M 1s 2024-04-21T08:45:42.145 INFO:tasks.workunit.client.0.smithi007.stderr:119350K .......... .......... .......... .......... .......... 87% 33.7M 1s 2024-04-21T08:45:42.147 INFO:tasks.workunit.client.0.smithi007.stderr:119400K .......... .......... .......... .......... .......... 87% 38.2M 1s 2024-04-21T08:45:42.149 INFO:tasks.workunit.client.0.smithi007.stderr:119450K .......... .......... .......... .......... .......... 88% 23.4M 1s 2024-04-21T08:45:42.150 INFO:tasks.workunit.client.0.smithi007.stderr:119500K .......... .......... .......... .......... .......... 88% 32.2M 1s 2024-04-21T08:45:42.152 INFO:tasks.workunit.client.0.smithi007.stderr:119550K .......... .......... .......... .......... .......... 88% 23.3M 1s 2024-04-21T08:45:42.154 INFO:tasks.workunit.client.0.smithi007.stderr:119600K .......... .......... .......... .......... .......... 88% 25.3M 1s 2024-04-21T08:45:42.155 INFO:tasks.workunit.client.0.smithi007.stderr:119650K .......... .......... .......... .......... .......... 88% 49.0M 1s 2024-04-21T08:45:42.157 INFO:tasks.workunit.client.0.smithi007.stderr:119700K .......... .......... .......... .......... .......... 88% 24.0M 1s 2024-04-21T08:45:42.159 INFO:tasks.workunit.client.0.smithi007.stderr:119750K .......... .......... .......... .......... .......... 88% 23.5M 1s 2024-04-21T08:45:42.161 INFO:tasks.workunit.client.0.smithi007.stderr:119800K .......... .......... .......... .......... .......... 88% 26.3M 1s 2024-04-21T08:45:42.163 INFO:tasks.workunit.client.0.smithi007.stderr:119850K .......... .......... .......... .......... .......... 88% 30.9M 1s 2024-04-21T08:45:42.164 INFO:tasks.workunit.client.0.smithi007.stderr:119900K .......... .......... .......... .......... .......... 88% 66.0M 1s 2024-04-21T08:45:42.165 INFO:tasks.workunit.client.0.smithi007.stderr:119950K .......... .......... .......... .......... .......... 88% 24.7M 1s 2024-04-21T08:45:42.170 INFO:tasks.workunit.client.0.smithi007.stderr:120000K .......... .......... .......... .......... .......... 88% 21.5M 1s 2024-04-21T08:45:42.171 INFO:tasks.workunit.client.0.smithi007.stderr:120050K .......... .......... .......... .......... .......... 88% 47.3M 1s 2024-04-21T08:45:42.171 INFO:tasks.workunit.client.0.smithi007.stderr:120100K .......... .......... .......... .......... .......... 88% 31.4M 1s 2024-04-21T08:45:42.178 INFO:tasks.workunit.client.0.smithi007.stderr:120150K .......... .......... .......... .......... .......... 88% 19.2M 1s 2024-04-21T08:45:42.178 INFO:tasks.workunit.client.0.smithi007.stderr:120200K .......... .......... .......... .......... .......... 88% 39.6M 1s 2024-04-21T08:45:42.178 INFO:tasks.workunit.client.0.smithi007.stderr:120250K .......... .......... .......... .......... .......... 88% 24.2M 1s 2024-04-21T08:45:42.178 INFO:tasks.workunit.client.0.smithi007.stderr:120300K .......... .......... .......... .......... .......... 88% 37.3M 1s 2024-04-21T08:45:42.179 INFO:tasks.workunit.client.0.smithi007.stderr:120350K .......... .......... .......... .......... .......... 88% 25.3M 1s 2024-04-21T08:45:42.187 INFO:tasks.workunit.client.0.smithi007.stderr:120400K .......... .......... .......... .......... .......... 88% 9.85M 1s 2024-04-21T08:45:42.188 INFO:tasks.workunit.client.0.smithi007.stderr:120450K .......... .......... .......... .......... .......... 88% 94.8M 1s 2024-04-21T08:45:42.188 INFO:tasks.workunit.client.0.smithi007.stderr:120500K .......... .......... .......... .......... .......... 88% 72.5M 1s 2024-04-21T08:45:42.188 INFO:tasks.workunit.client.0.smithi007.stderr:120550K .......... .......... .......... .......... .......... 88% 41.1M 1s 2024-04-21T08:45:42.188 INFO:tasks.workunit.client.0.smithi007.stderr:120600K .......... .......... .......... .......... .......... 88% 44.9M 1s 2024-04-21T08:45:42.190 INFO:tasks.workunit.client.0.smithi007.stderr:120650K .......... .......... .......... .......... .......... 88% 24.3M 1s 2024-04-21T08:45:42.191 INFO:tasks.workunit.client.0.smithi007.stderr:120700K .......... .......... .......... .......... .......... 88% 46.1M 1s 2024-04-21T08:45:42.193 INFO:tasks.workunit.client.0.smithi007.stderr:120750K .......... .......... .......... .......... .......... 88% 25.5M 1s 2024-04-21T08:45:42.194 INFO:tasks.workunit.client.0.smithi007.stderr:120800K .......... .......... .......... .......... .......... 89% 33.4M 1s 2024-04-21T08:45:42.196 INFO:tasks.workunit.client.0.smithi007.stderr:120850K .......... .......... .......... .......... .......... 89% 27.8M 1s 2024-04-21T08:45:42.198 INFO:tasks.workunit.client.0.smithi007.stderr:120900K .......... .......... .......... .......... .......... 89% 22.8M 1s 2024-04-21T08:45:42.199 INFO:tasks.workunit.client.0.smithi007.stderr:120950K .......... .......... .......... .......... .......... 89% 42.4M 1s 2024-04-21T08:45:42.201 INFO:tasks.workunit.client.0.smithi007.stderr:121000K .......... .......... .......... .......... .......... 89% 25.6M 1s 2024-04-21T08:45:42.202 INFO:tasks.workunit.client.0.smithi007.stderr:121050K .......... .......... .......... .......... .......... 89% 43.0M 1s 2024-04-21T08:45:42.204 INFO:tasks.workunit.client.0.smithi007.stderr:121100K .......... .......... .......... .......... .......... 89% 23.5M 1s 2024-04-21T08:45:42.206 INFO:tasks.workunit.client.0.smithi007.stderr:121150K .......... .......... .......... .......... .......... 89% 24.0M 1s 2024-04-21T08:45:42.208 INFO:tasks.workunit.client.0.smithi007.stderr:121200K .......... .......... .......... .......... .......... 89% 49.3M 1s 2024-04-21T08:45:42.209 INFO:tasks.workunit.client.0.smithi007.stderr:121250K .......... .......... .......... .......... .......... 89% 23.0M 1s 2024-04-21T08:45:42.211 INFO:tasks.workunit.client.0.smithi007.stderr:121300K .......... .......... .......... .......... .......... 89% 41.0M 1s 2024-04-21T08:45:42.213 INFO:tasks.workunit.client.0.smithi007.stderr:121350K .......... .......... .......... .......... .......... 89% 20.6M 1s 2024-04-21T08:45:42.214 INFO:tasks.workunit.client.0.smithi007.stderr:121400K .......... .......... .......... .......... .......... 89% 51.2M 1s 2024-04-21T08:45:42.215 INFO:tasks.workunit.client.0.smithi007.stderr:121450K .......... .......... .......... .......... .......... 89% 27.5M 0s 2024-04-21T08:45:42.218 INFO:tasks.workunit.client.0.smithi007.stderr:121500K .......... .......... .......... .......... .......... 89% 25.2M 0s 2024-04-21T08:45:42.219 INFO:tasks.workunit.client.0.smithi007.stderr:121550K .......... .......... .......... .......... .......... 89% 38.5M 0s 2024-04-21T08:45:42.221 INFO:tasks.workunit.client.0.smithi007.stderr:121600K .......... .......... .......... .......... .......... 89% 22.7M 0s 2024-04-21T08:45:42.223 INFO:tasks.workunit.client.0.smithi007.stderr:121650K .......... .......... .......... .......... .......... 89% 25.4M 0s 2024-04-21T08:45:42.224 INFO:tasks.workunit.client.0.smithi007.stderr:121700K .......... .......... .......... .......... .......... 89% 45.3M 0s 2024-04-21T08:45:42.226 INFO:tasks.workunit.client.0.smithi007.stderr:121750K .......... .......... .......... .......... .......... 89% 23.4M 0s 2024-04-21T08:45:42.227 INFO:tasks.workunit.client.0.smithi007.stderr:121800K .......... .......... .......... .......... .......... 89% 33.8M 0s 2024-04-21T08:45:42.230 INFO:tasks.workunit.client.0.smithi007.stderr:121850K .......... .......... .......... .......... .......... 89% 23.4M 0s 2024-04-21T08:45:42.232 INFO:tasks.workunit.client.0.smithi007.stderr:121900K .......... .......... .......... .......... .......... 89% 23.8M 0s 2024-04-21T08:45:42.233 INFO:tasks.workunit.client.0.smithi007.stderr:121950K .......... .......... .......... .......... .......... 89% 51.5M 0s 2024-04-21T08:45:42.235 INFO:tasks.workunit.client.0.smithi007.stderr:122000K .......... .......... .......... .......... .......... 89% 23.2M 0s 2024-04-21T08:45:42.237 INFO:tasks.workunit.client.0.smithi007.stderr:122050K .......... .......... .......... .......... .......... 89% 41.2M 0s 2024-04-21T08:45:42.239 INFO:tasks.workunit.client.0.smithi007.stderr:122100K .......... .......... .......... .......... .......... 89% 18.7M 0s 2024-04-21T08:45:42.241 INFO:tasks.workunit.client.0.smithi007.stderr:122150K .......... .......... .......... .......... .......... 90% 24.1M 0s 2024-04-21T08:45:42.242 INFO:tasks.workunit.client.0.smithi007.stderr:122200K .......... .......... .......... .......... .......... 90% 54.9M 0s 2024-04-21T08:45:42.243 INFO:tasks.workunit.client.0.smithi007.stderr:122250K .......... .......... .......... .......... .......... 90% 28.3M 0s 2024-04-21T08:45:42.245 INFO:tasks.workunit.client.0.smithi007.stderr:122300K .......... .......... .......... .......... .......... 90% 26.0M 0s 2024-04-21T08:45:42.247 INFO:tasks.workunit.client.0.smithi007.stderr:122350K .......... .......... .......... .......... .......... 90% 28.9M 0s 2024-04-21T08:45:42.250 INFO:tasks.workunit.client.0.smithi007.stderr:122400K .......... .......... .......... .......... .......... 90% 18.8M 0s 2024-04-21T08:45:42.251 INFO:tasks.workunit.client.0.smithi007.stderr:122450K .......... .......... .......... .......... .......... 90% 46.2M 0s 2024-04-21T08:45:42.252 INFO:tasks.workunit.client.0.smithi007.stderr:122500K .......... .......... .......... .......... .......... 90% 23.6M 0s 2024-04-21T08:45:42.254 INFO:tasks.workunit.client.0.smithi007.stderr:122550K .......... .......... .......... .......... .......... 90% 41.8M 0s 2024-04-21T08:45:42.256 INFO:tasks.workunit.client.0.smithi007.stderr:122600K .......... .......... .......... .......... .......... 90% 22.8M 0s 2024-04-21T08:45:42.258 INFO:tasks.workunit.client.0.smithi007.stderr:122650K .......... .......... .......... .......... .......... 90% 24.8M 0s 2024-04-21T08:45:42.259 INFO:tasks.workunit.client.0.smithi007.stderr:122700K .......... .......... .......... .......... .......... 90% 50.6M 0s 2024-04-21T08:45:42.261 INFO:tasks.workunit.client.0.smithi007.stderr:122750K .......... .......... .......... .......... .......... 90% 24.1M 0s 2024-04-21T08:45:42.262 INFO:tasks.workunit.client.0.smithi007.stderr:122800K .......... .......... .......... .......... .......... 90% 40.4M 0s 2024-04-21T08:45:42.264 INFO:tasks.workunit.client.0.smithi007.stderr:122850K .......... .......... .......... .......... .......... 90% 26.4M 0s 2024-04-21T08:45:42.266 INFO:tasks.workunit.client.0.smithi007.stderr:122900K .......... .......... .......... .......... .......... 90% 21.9M 0s 2024-04-21T08:45:42.267 INFO:tasks.workunit.client.0.smithi007.stderr:122950K .......... .......... .......... .......... .......... 90% 47.3M 0s 2024-04-21T08:45:42.269 INFO:tasks.workunit.client.0.smithi007.stderr:123000K .......... .......... .......... .......... .......... 90% 23.8M 0s 2024-04-21T08:45:42.270 INFO:tasks.workunit.client.0.smithi007.stderr:123050K .......... .......... .......... .......... .......... 90% 41.4M 0s 2024-04-21T08:45:42.273 INFO:tasks.workunit.client.0.smithi007.stderr:123100K .......... .......... .......... .......... .......... 90% 23.9M 0s 2024-04-21T08:45:42.277 INFO:tasks.workunit.client.0.smithi007.stderr:123150K .......... .......... .......... .......... .......... 90% 26.9M 0s 2024-04-21T08:45:42.277 INFO:tasks.workunit.client.0.smithi007.stderr:123200K .......... .......... .......... .......... .......... 90% 41.3M 0s 2024-04-21T08:45:42.279 INFO:tasks.workunit.client.0.smithi007.stderr:123250K .......... .......... .......... .......... .......... 90% 23.0M 0s 2024-04-21T08:45:42.279 INFO:tasks.workunit.client.0.smithi007.stderr:123300K .......... .......... .......... .......... .......... 90% 40.7M 0s 2024-04-21T08:45:42.281 INFO:tasks.workunit.client.0.smithi007.stderr:123350K .......... .......... .......... .......... .......... 90% 21.7M 0s 2024-04-21T08:45:42.283 INFO:tasks.workunit.client.0.smithi007.stderr:123400K .......... .......... .......... .......... .......... 90% 24.2M 0s 2024-04-21T08:45:42.284 INFO:tasks.workunit.client.0.smithi007.stderr:123450K .......... .......... .......... .......... .......... 90% 48.7M 0s 2024-04-21T08:45:42.286 INFO:tasks.workunit.client.0.smithi007.stderr:123500K .......... .......... .......... .......... .......... 91% 25.5M 0s 2024-04-21T08:45:42.288 INFO:tasks.workunit.client.0.smithi007.stderr:123550K .......... .......... .......... .......... .......... 91% 41.5M 0s 2024-04-21T08:45:42.289 INFO:tasks.workunit.client.0.smithi007.stderr:123600K .......... .......... .......... .......... .......... 91% 23.5M 0s 2024-04-21T08:45:42.291 INFO:tasks.workunit.client.0.smithi007.stderr:123650K .......... .......... .......... .......... .......... 91% 22.4M 0s 2024-04-21T08:45:42.293 INFO:tasks.workunit.client.0.smithi007.stderr:123700K .......... .......... .......... .......... .......... 91% 56.0M 0s 2024-04-21T08:45:42.295 INFO:tasks.workunit.client.0.smithi007.stderr:123750K .......... .......... .......... .......... .......... 91% 25.1M 0s 2024-04-21T08:45:42.297 INFO:tasks.workunit.client.0.smithi007.stderr:123800K .......... .......... .......... .......... .......... 91% 20.3M 0s 2024-04-21T08:45:42.298 INFO:tasks.workunit.client.0.smithi007.stderr:123850K .......... .......... .......... .......... .......... 91% 38.4M 0s 2024-04-21T08:45:42.300 INFO:tasks.workunit.client.0.smithi007.stderr:123900K .......... .......... .......... .......... .......... 91% 22.8M 0s 2024-04-21T08:45:42.301 INFO:tasks.workunit.client.0.smithi007.stderr:123950K .......... .......... .......... .......... .......... 91% 35.7M 0s 2024-04-21T08:45:42.303 INFO:tasks.workunit.client.0.smithi007.stderr:124000K .......... .......... .......... .......... .......... 91% 24.0M 0s 2024-04-21T08:45:42.306 INFO:tasks.workunit.client.0.smithi007.stderr:124050K .......... .......... .......... .......... .......... 91% 23.8M 0s 2024-04-21T08:45:42.308 INFO:tasks.workunit.client.0.smithi007.stderr:124100K .......... .......... .......... .......... .......... 91% 35.1M 0s 2024-04-21T08:45:42.309 INFO:tasks.workunit.client.0.smithi007.stderr:124150K .......... .......... .......... .......... .......... 91% 24.2M 0s 2024-04-21T08:45:42.310 INFO:tasks.workunit.client.0.smithi007.stderr:124200K .......... .......... .......... .......... .......... 91% 48.0M 0s 2024-04-21T08:45:42.312 INFO:tasks.workunit.client.0.smithi007.stderr:124250K .......... .......... .......... .......... .......... 91% 24.9M 0s 2024-04-21T08:45:42.314 INFO:tasks.workunit.client.0.smithi007.stderr:124300K .......... .......... .......... .......... .......... 91% 20.2M 0s 2024-04-21T08:45:42.316 INFO:tasks.workunit.client.0.smithi007.stderr:124350K .......... .......... .......... .......... .......... 91% 38.2M 0s 2024-04-21T08:45:42.318 INFO:tasks.workunit.client.0.smithi007.stderr:124400K .......... .......... .......... .......... .......... 91% 23.6M 0s 2024-04-21T08:45:42.318 INFO:tasks.workunit.client.0.smithi007.stderr:124450K .......... .......... .......... .......... .......... 91% 47.3M 0s 2024-04-21T08:45:42.321 INFO:tasks.workunit.client.0.smithi007.stderr:124500K .......... .......... .......... .......... .......... 91% 21.4M 0s 2024-04-21T08:45:42.323 INFO:tasks.workunit.client.0.smithi007.stderr:124550K .......... .......... .......... .......... .......... 91% 21.4M 0s 2024-04-21T08:45:42.324 INFO:tasks.workunit.client.0.smithi007.stderr:124600K .......... .......... .......... .......... .......... 91% 40.7M 0s 2024-04-21T08:45:42.327 INFO:tasks.workunit.client.0.smithi007.stderr:124650K .......... .......... .......... .......... .......... 91% 20.3M 0s 2024-04-21T08:45:42.327 INFO:tasks.workunit.client.0.smithi007.stderr:124700K .......... .......... .......... .......... .......... 91% 59.5M 0s 2024-04-21T08:45:42.330 INFO:tasks.workunit.client.0.smithi007.stderr:124750K .......... .......... .......... .......... .......... 91% 22.7M 0s 2024-04-21T08:45:42.332 INFO:tasks.workunit.client.0.smithi007.stderr:124800K .......... .......... .......... .......... .......... 91% 24.5M 0s 2024-04-21T08:45:42.333 INFO:tasks.workunit.client.0.smithi007.stderr:124850K .......... .......... .......... .......... .......... 92% 37.0M 0s 2024-04-21T08:45:42.336 INFO:tasks.workunit.client.0.smithi007.stderr:124900K .......... .......... .......... .......... .......... 92% 24.9M 0s 2024-04-21T08:45:42.336 INFO:tasks.workunit.client.0.smithi007.stderr:124950K .......... .......... .......... .......... .......... 92% 53.8M 0s 2024-04-21T08:45:42.338 INFO:tasks.workunit.client.0.smithi007.stderr:125000K .......... .......... .......... .......... .......... 92% 23.2M 0s 2024-04-21T08:45:42.340 INFO:tasks.workunit.client.0.smithi007.stderr:125050K .......... .......... .......... .......... .......... 92% 23.8M 0s 2024-04-21T08:45:42.341 INFO:tasks.workunit.client.0.smithi007.stderr:125100K .......... .......... .......... .......... .......... 92% 38.4M 0s 2024-04-21T08:45:42.343 INFO:tasks.workunit.client.0.smithi007.stderr:125150K .......... .......... .......... .......... .......... 92% 26.6M 0s 2024-04-21T08:45:42.345 INFO:tasks.workunit.client.0.smithi007.stderr:125200K .......... .......... .......... .......... .......... 92% 35.6M 0s 2024-04-21T08:45:42.347 INFO:tasks.workunit.client.0.smithi007.stderr:125250K .......... .......... .......... .......... .......... 92% 23.2M 0s 2024-04-21T08:45:42.349 INFO:tasks.workunit.client.0.smithi007.stderr:125300K .......... .......... .......... .......... .......... 92% 21.5M 0s 2024-04-21T08:45:42.351 INFO:tasks.workunit.client.0.smithi007.stderr:125350K .......... .......... .......... .......... .......... 92% 38.9M 0s 2024-04-21T08:45:42.353 INFO:tasks.workunit.client.0.smithi007.stderr:125400K .......... .......... .......... .......... .......... 92% 24.6M 0s 2024-04-21T08:45:42.355 INFO:tasks.workunit.client.0.smithi007.stderr:125450K .......... .......... .......... .......... .......... 92% 19.6M 0s 2024-04-21T08:45:42.356 INFO:tasks.workunit.client.0.smithi007.stderr:125500K .......... .......... .......... .......... .......... 92% 54.3M 0s 2024-04-21T08:45:42.357 INFO:tasks.workunit.client.0.smithi007.stderr:125550K .......... .......... .......... .......... .......... 92% 23.1M 0s 2024-04-21T08:45:42.359 INFO:tasks.workunit.client.0.smithi007.stderr:125600K .......... .......... .......... .......... .......... 92% 34.9M 0s 2024-04-21T08:45:42.361 INFO:tasks.workunit.client.0.smithi007.stderr:125650K .......... .......... .......... .......... .......... 92% 25.4M 0s 2024-04-21T08:45:42.363 INFO:tasks.workunit.client.0.smithi007.stderr:125700K .......... .......... .......... .......... .......... 92% 23.4M 0s 2024-04-21T08:45:42.364 INFO:tasks.workunit.client.0.smithi007.stderr:125750K .......... .......... .......... .......... .......... 92% 47.0M 0s 2024-04-21T08:45:42.366 INFO:tasks.workunit.client.0.smithi007.stderr:125800K .......... .......... .......... .......... .......... 92% 25.1M 0s 2024-04-21T08:45:42.367 INFO:tasks.workunit.client.0.smithi007.stderr:125850K .......... .......... .......... .......... .......... 92% 45.7M 0s 2024-04-21T08:45:42.370 INFO:tasks.workunit.client.0.smithi007.stderr:125900K .......... .......... .......... .......... .......... 92% 22.1M 0s 2024-04-21T08:45:42.371 INFO:tasks.workunit.client.0.smithi007.stderr:125950K .......... .......... .......... .......... .......... 92% 23.0M 0s 2024-04-21T08:45:42.372 INFO:tasks.workunit.client.0.smithi007.stderr:126000K .......... .......... .......... .......... .......... 92% 46.1M 0s 2024-04-21T08:45:42.375 INFO:tasks.workunit.client.0.smithi007.stderr:126050K .......... .......... .......... .......... .......... 92% 24.1M 0s 2024-04-21T08:45:42.376 INFO:tasks.workunit.client.0.smithi007.stderr:126100K .......... .......... .......... .......... .......... 92% 41.1M 0s 2024-04-21T08:45:42.378 INFO:tasks.workunit.client.0.smithi007.stderr:126150K .......... .......... .......... .......... .......... 92% 23.1M 0s 2024-04-21T08:45:42.380 INFO:tasks.workunit.client.0.smithi007.stderr:126200K .......... .......... .......... .......... .......... 92% 24.3M 0s 2024-04-21T08:45:42.381 INFO:tasks.workunit.client.0.smithi007.stderr:126250K .......... .......... .......... .......... .......... 93% 54.3M 0s 2024-04-21T08:45:42.383 INFO:tasks.workunit.client.0.smithi007.stderr:126300K .......... .......... .......... .......... .......... 93% 23.2M 0s 2024-04-21T08:45:42.384 INFO:tasks.workunit.client.0.smithi007.stderr:126350K .......... .......... .......... .......... .......... 93% 33.5M 0s 2024-04-21T08:45:42.386 INFO:tasks.workunit.client.0.smithi007.stderr:126400K .......... .......... .......... .......... .......... 93% 21.7M 0s 2024-04-21T08:45:42.389 INFO:tasks.workunit.client.0.smithi007.stderr:126450K .......... .......... .......... .......... .......... 93% 21.8M 0s 2024-04-21T08:45:42.390 INFO:tasks.workunit.client.0.smithi007.stderr:126500K .......... .......... .......... .......... .......... 93% 42.0M 0s 2024-04-21T08:45:42.392 INFO:tasks.workunit.client.0.smithi007.stderr:126550K .......... .......... .......... .......... .......... 93% 24.3M 0s 2024-04-21T08:45:42.393 INFO:tasks.workunit.client.0.smithi007.stderr:126600K .......... .......... .......... .......... .......... 93% 43.6M 0s 2024-04-21T08:45:42.395 INFO:tasks.workunit.client.0.smithi007.stderr:126650K .......... .......... .......... .......... .......... 93% 23.1M 0s 2024-04-21T08:45:42.397 INFO:tasks.workunit.client.0.smithi007.stderr:126700K .......... .......... .......... .......... .......... 93% 24.5M 0s 2024-04-21T08:45:42.398 INFO:tasks.workunit.client.0.smithi007.stderr:126750K .......... .......... .......... .......... .......... 93% 45.2M 0s 2024-04-21T08:45:42.401 INFO:tasks.workunit.client.0.smithi007.stderr:126800K .......... .......... .......... .......... .......... 93% 23.1M 0s 2024-04-21T08:45:42.402 INFO:tasks.workunit.client.0.smithi007.stderr:126850K .......... .......... .......... .......... .......... 93% 39.8M 0s 2024-04-21T08:45:42.404 INFO:tasks.workunit.client.0.smithi007.stderr:126900K .......... .......... .......... .......... .......... 93% 24.1M 0s 2024-04-21T08:45:42.406 INFO:tasks.workunit.client.0.smithi007.stderr:126950K .......... .......... .......... .......... .......... 93% 23.8M 0s 2024-04-21T08:45:42.406 INFO:tasks.workunit.client.0.smithi007.stderr:127000K .......... .......... .......... .......... .......... 93% 49.1M 0s 2024-04-21T08:45:42.409 INFO:tasks.workunit.client.0.smithi007.stderr:127050K .......... .......... .......... .......... .......... 93% 24.0M 0s 2024-04-21T08:45:42.410 INFO:tasks.workunit.client.0.smithi007.stderr:127100K .......... .......... .......... .......... .......... 93% 37.8M 0s 2024-04-21T08:45:42.412 INFO:tasks.workunit.client.0.smithi007.stderr:127150K .......... .......... .......... .......... .......... 93% 20.9M 0s 2024-04-21T08:45:42.414 INFO:tasks.workunit.client.0.smithi007.stderr:127200K .......... .......... .......... .......... .......... 93% 27.4M 0s 2024-04-21T08:45:42.415 INFO:tasks.workunit.client.0.smithi007.stderr:127250K .......... .......... .......... .......... .......... 93% 37.9M 0s 2024-04-21T08:45:42.418 INFO:tasks.workunit.client.0.smithi007.stderr:127300K .......... .......... .......... .......... .......... 93% 23.3M 0s 2024-04-21T08:45:42.419 INFO:tasks.workunit.client.0.smithi007.stderr:127350K .......... .......... .......... .......... .......... 93% 41.5M 0s 2024-04-21T08:45:42.421 INFO:tasks.workunit.client.0.smithi007.stderr:127400K .......... .......... .......... .......... .......... 93% 22.6M 0s 2024-04-21T08:45:42.423 INFO:tasks.workunit.client.0.smithi007.stderr:127450K .......... .......... .......... .......... .......... 93% 25.1M 0s 2024-04-21T08:45:42.424 INFO:tasks.workunit.client.0.smithi007.stderr:127500K .......... .......... .......... .......... .......... 93% 40.2M 0s 2024-04-21T08:45:42.426 INFO:tasks.workunit.client.0.smithi007.stderr:127550K .......... .......... .......... .......... .......... 93% 24.6M 0s 2024-04-21T08:45:42.428 INFO:tasks.workunit.client.0.smithi007.stderr:127600K .......... .......... .......... .......... .......... 94% 31.6M 0s 2024-04-21T08:45:42.429 INFO:tasks.workunit.client.0.smithi007.stderr:127650K .......... .......... .......... .......... .......... 94% 29.0M 0s 2024-04-21T08:45:42.432 INFO:tasks.workunit.client.0.smithi007.stderr:127700K .......... .......... .......... .......... .......... 94% 20.5M 0s 2024-04-21T08:45:42.433 INFO:tasks.workunit.client.0.smithi007.stderr:127750K .......... .......... .......... .......... .......... 94% 50.1M 0s 2024-04-21T08:45:42.435 INFO:tasks.workunit.client.0.smithi007.stderr:127800K .......... .......... .......... .......... .......... 94% 22.9M 0s 2024-04-21T08:45:42.437 INFO:tasks.workunit.client.0.smithi007.stderr:127850K .......... .......... .......... .......... .......... 94% 37.6M 0s 2024-04-21T08:45:42.437 INFO:tasks.workunit.client.0.smithi007.stderr:127900K .......... .......... .......... .......... .......... 94% 32.0M 0s 2024-04-21T08:45:42.440 INFO:tasks.workunit.client.0.smithi007.stderr:127950K .......... .......... .......... .......... .......... 94% 19.3M 0s 2024-04-21T08:45:42.441 INFO:tasks.workunit.client.0.smithi007.stderr:128000K .......... .......... .......... .......... .......... 94% 52.7M 0s 2024-04-21T08:45:42.443 INFO:tasks.workunit.client.0.smithi007.stderr:128050K .......... .......... .......... .......... .......... 94% 21.3M 0s 2024-04-21T08:45:42.444 INFO:tasks.workunit.client.0.smithi007.stderr:128100K .......... .......... .......... .......... .......... 94% 45.1M 0s 2024-04-21T08:45:42.446 INFO:tasks.workunit.client.0.smithi007.stderr:128150K .......... .......... .......... .......... .......... 94% 24.5M 0s 2024-04-21T08:45:42.449 INFO:tasks.workunit.client.0.smithi007.stderr:128200K .......... .......... .......... .......... .......... 94% 22.0M 0s 2024-04-21T08:45:42.450 INFO:tasks.workunit.client.0.smithi007.stderr:128250K .......... .......... .......... .......... .......... 94% 44.6M 0s 2024-04-21T08:45:42.452 INFO:tasks.workunit.client.0.smithi007.stderr:128300K .......... .......... .......... .......... .......... 94% 24.7M 0s 2024-04-21T08:45:42.453 INFO:tasks.workunit.client.0.smithi007.stderr:128350K .......... .......... .......... .......... .......... 94% 42.2M 0s 2024-04-21T08:45:42.455 INFO:tasks.workunit.client.0.smithi007.stderr:128400K .......... .......... .......... .......... .......... 94% 23.3M 0s 2024-04-21T08:45:42.457 INFO:tasks.workunit.client.0.smithi007.stderr:128450K .......... .......... .......... .......... .......... 94% 23.5M 0s 2024-04-21T08:45:42.458 INFO:tasks.workunit.client.0.smithi007.stderr:128500K .......... .......... .......... .......... .......... 94% 47.1M 0s 2024-04-21T08:45:42.460 INFO:tasks.workunit.client.0.smithi007.stderr:128550K .......... .......... .......... .......... .......... 94% 24.5M 0s 2024-04-21T08:45:42.461 INFO:tasks.workunit.client.0.smithi007.stderr:128600K .......... .......... .......... .......... .......... 94% 42.6M 0s 2024-04-21T08:45:42.463 INFO:tasks.workunit.client.0.smithi007.stderr:128650K .......... .......... .......... .......... .......... 94% 25.5M 0s 2024-04-21T08:45:42.466 INFO:tasks.workunit.client.0.smithi007.stderr:128700K .......... .......... .......... .......... .......... 94% 22.2M 0s 2024-04-21T08:45:42.466 INFO:tasks.workunit.client.0.smithi007.stderr:128750K .......... .......... .......... .......... .......... 94% 47.8M 0s 2024-04-21T08:45:42.468 INFO:tasks.workunit.client.0.smithi007.stderr:128800K .......... .......... .......... .......... .......... 94% 23.2M 0s 2024-04-21T08:45:42.470 INFO:tasks.workunit.client.0.smithi007.stderr:128850K .......... .......... .......... .......... .......... 94% 42.2M 0s 2024-04-21T08:45:42.472 INFO:tasks.workunit.client.0.smithi007.stderr:128900K .......... .......... .......... .......... .......... 94% 23.5M 0s 2024-04-21T08:45:42.474 INFO:tasks.workunit.client.0.smithi007.stderr:128950K .......... .......... .......... .......... .......... 95% 23.3M 0s 2024-04-21T08:45:42.475 INFO:tasks.workunit.client.0.smithi007.stderr:129000K .......... .......... .......... .......... .......... 95% 54.7M 0s 2024-04-21T08:45:42.477 INFO:tasks.workunit.client.0.smithi007.stderr:129050K .......... .......... .......... .......... .......... 95% 21.7M 0s 2024-04-21T08:45:42.479 INFO:tasks.workunit.client.0.smithi007.stderr:129100K .......... .......... .......... .......... .......... 95% 41.9M 0s 2024-04-21T08:45:42.480 INFO:tasks.workunit.client.0.smithi007.stderr:129150K .......... .......... .......... .......... .......... 95% 24.6M 0s 2024-04-21T08:45:42.482 INFO:tasks.workunit.client.0.smithi007.stderr:129200K .......... .......... .......... .......... .......... 95% 24.9M 0s 2024-04-21T08:45:42.483 INFO:tasks.workunit.client.0.smithi007.stderr:129250K .......... .......... .......... .......... .......... 95% 53.1M 0s 2024-04-21T08:45:42.485 INFO:tasks.workunit.client.0.smithi007.stderr:129300K .......... .......... .......... .......... .......... 95% 21.8M 0s 2024-04-21T08:45:42.487 INFO:tasks.workunit.client.0.smithi007.stderr:129350K .......... .......... .......... .......... .......... 95% 24.7M 0s 2024-04-21T08:45:42.489 INFO:tasks.workunit.client.0.smithi007.stderr:129400K .......... .......... .......... .......... .......... 95% 40.7M 0s 2024-04-21T08:45:42.490 INFO:tasks.workunit.client.0.smithi007.stderr:129450K .......... .......... .......... .......... .......... 95% 29.7M 0s 2024-04-21T08:45:42.491 INFO:tasks.workunit.client.0.smithi007.stderr:129500K .......... .......... .......... .......... .......... 95% 36.2M 0s 2024-04-21T08:45:42.493 INFO:tasks.workunit.client.0.smithi007.stderr:129550K .......... .......... .......... .......... .......... 95% 22.8M 0s 2024-04-21T08:45:42.495 INFO:tasks.workunit.client.0.smithi007.stderr:129600K .......... .......... .......... .......... .......... 95% 27.9M 0s 2024-04-21T08:45:42.497 INFO:tasks.workunit.client.0.smithi007.stderr:129650K .......... .......... .......... .......... .......... 95% 29.7M 0s 2024-04-21T08:45:42.499 INFO:tasks.workunit.client.0.smithi007.stderr:129700K .......... .......... .......... .......... .......... 95% 25.6M 0s 2024-04-21T08:45:42.500 INFO:tasks.workunit.client.0.smithi007.stderr:129750K .......... .......... .......... .......... .......... 95% 52.2M 0s 2024-04-21T08:45:42.502 INFO:tasks.workunit.client.0.smithi007.stderr:129800K .......... .......... .......... .......... .......... 95% 21.9M 0s 2024-04-21T08:45:42.504 INFO:tasks.workunit.client.0.smithi007.stderr:129850K .......... .......... .......... .......... .......... 95% 25.3M 0s 2024-04-21T08:45:42.505 INFO:tasks.workunit.client.0.smithi007.stderr:129900K .......... .......... .......... .......... .......... 95% 39.8M 0s 2024-04-21T08:45:42.507 INFO:tasks.workunit.client.0.smithi007.stderr:129950K .......... .......... .......... .......... .......... 95% 23.7M 0s 2024-04-21T08:45:42.508 INFO:tasks.workunit.client.0.smithi007.stderr:130000K .......... .......... .......... .......... .......... 95% 47.3M 0s 2024-04-21T08:45:42.510 INFO:tasks.workunit.client.0.smithi007.stderr:130050K .......... .......... .......... .......... .......... 95% 24.3M 0s 2024-04-21T08:45:42.512 INFO:tasks.workunit.client.0.smithi007.stderr:130100K .......... .......... .......... .......... .......... 95% 23.9M 0s 2024-04-21T08:45:42.514 INFO:tasks.workunit.client.0.smithi007.stderr:130150K .......... .......... .......... .......... .......... 95% 37.0M 0s 2024-04-21T08:45:42.516 INFO:tasks.workunit.client.0.smithi007.stderr:130200K .......... .......... .......... .......... .......... 95% 24.7M 0s 2024-04-21T08:45:42.518 INFO:tasks.workunit.client.0.smithi007.stderr:130250K .......... .......... .......... .......... .......... 95% 58.4M 0s 2024-04-21T08:45:42.519 INFO:tasks.workunit.client.0.smithi007.stderr:130300K .......... .......... .......... .......... .......... 96% 23.1M 0s 2024-04-21T08:45:42.521 INFO:tasks.workunit.client.0.smithi007.stderr:130350K .......... .......... .......... .......... .......... 96% 23.4M 0s 2024-04-21T08:45:42.522 INFO:tasks.workunit.client.0.smithi007.stderr:130400K .......... .......... .......... .......... .......... 96% 39.6M 0s 2024-04-21T08:45:42.524 INFO:tasks.workunit.client.0.smithi007.stderr:130450K .......... .......... .......... .......... .......... 96% 22.0M 0s 2024-04-21T08:45:42.527 INFO:tasks.workunit.client.0.smithi007.stderr:130500K .......... .......... .......... .......... .......... 96% 54.2M 0s 2024-04-21T08:45:42.527 INFO:tasks.workunit.client.0.smithi007.stderr:130550K .......... .......... .......... .......... .......... 96% 27.5M 0s 2024-04-21T08:45:42.530 INFO:tasks.workunit.client.0.smithi007.stderr:130600K .......... .......... .......... .......... .......... 96% 18.2M 0s 2024-04-21T08:45:42.531 INFO:tasks.workunit.client.0.smithi007.stderr:130650K .......... .......... .......... .......... .......... 96% 45.4M 0s 2024-04-21T08:45:42.533 INFO:tasks.workunit.client.0.smithi007.stderr:130700K .......... .......... .......... .......... .......... 96% 23.1M 0s 2024-04-21T08:45:42.535 INFO:tasks.workunit.client.0.smithi007.stderr:130750K .......... .......... .......... .......... .......... 96% 21.9M 0s 2024-04-21T08:45:42.536 INFO:tasks.workunit.client.0.smithi007.stderr:130800K .......... .......... .......... .......... .......... 96% 58.1M 0s 2024-04-21T08:45:42.538 INFO:tasks.workunit.client.0.smithi007.stderr:130850K .......... .......... .......... .......... .......... 96% 25.1M 0s 2024-04-21T08:45:42.540 INFO:tasks.workunit.client.0.smithi007.stderr:130900K .......... .......... .......... .......... .......... 96% 25.7M 0s 2024-04-21T08:45:42.541 INFO:tasks.workunit.client.0.smithi007.stderr:130950K .......... .......... .......... .......... .......... 96% 24.6M 0s 2024-04-21T08:45:42.544 INFO:tasks.workunit.client.0.smithi007.stderr:131000K .......... .......... .......... .......... .......... 96% 21.9M 0s 2024-04-21T08:45:42.545 INFO:tasks.workunit.client.0.smithi007.stderr:131050K .......... .......... .......... .......... .......... 96% 52.1M 0s 2024-04-21T08:45:42.547 INFO:tasks.workunit.client.0.smithi007.stderr:131100K .......... .......... .......... .......... .......... 96% 21.3M 0s 2024-04-21T08:45:42.549 INFO:tasks.workunit.client.0.smithi007.stderr:131150K .......... .......... .......... .......... .......... 96% 42.9M 0s 2024-04-21T08:45:42.550 INFO:tasks.workunit.client.0.smithi007.stderr:131200K .......... .......... .......... .......... .......... 96% 22.8M 0s 2024-04-21T08:45:42.553 INFO:tasks.workunit.client.0.smithi007.stderr:131250K .......... .......... .......... .......... .......... 96% 23.3M 0s 2024-04-21T08:45:42.553 INFO:tasks.workunit.client.0.smithi007.stderr:131300K .......... .......... .......... .......... .......... 96% 49.4M 0s 2024-04-21T08:45:42.556 INFO:tasks.workunit.client.0.smithi007.stderr:131350K .......... .......... .......... .......... .......... 96% 22.9M 0s 2024-04-21T08:45:42.557 INFO:tasks.workunit.client.0.smithi007.stderr:131400K .......... .......... .......... .......... .......... 96% 37.1M 0s 2024-04-21T08:45:42.559 INFO:tasks.workunit.client.0.smithi007.stderr:131450K .......... .......... .......... .......... .......... 96% 25.3M 0s 2024-04-21T08:45:42.562 INFO:tasks.workunit.client.0.smithi007.stderr:131500K .......... .......... .......... .......... .......... 96% 23.6M 0s 2024-04-21T08:45:42.562 INFO:tasks.workunit.client.0.smithi007.stderr:131550K .......... .......... .......... .......... .......... 96% 49.7M 0s 2024-04-21T08:45:42.563 INFO:tasks.workunit.client.0.smithi007.stderr:131600K .......... .......... .......... .......... .......... 96% 32.9M 0s 2024-04-21T08:45:42.565 INFO:tasks.workunit.client.0.smithi007.stderr:131650K .......... .......... .......... .......... .......... 97% 26.3M 0s 2024-04-21T08:45:42.566 INFO:tasks.workunit.client.0.smithi007.stderr:131700K .......... .......... .......... .......... .......... 97% 31.7M 0s 2024-04-21T08:45:42.569 INFO:tasks.workunit.client.0.smithi007.stderr:131750K .......... .......... .......... .......... .......... 97% 19.6M 0s 2024-04-21T08:45:42.570 INFO:tasks.workunit.client.0.smithi007.stderr:131800K .......... .......... .......... .......... .......... 97% 45.0M 0s 2024-04-21T08:45:42.572 INFO:tasks.workunit.client.0.smithi007.stderr:131850K .......... .......... .......... .......... .......... 97% 25.8M 0s 2024-04-21T08:45:42.573 INFO:tasks.workunit.client.0.smithi007.stderr:131900K .......... .......... .......... .......... .......... 97% 39.5M 0s 2024-04-21T08:45:42.576 INFO:tasks.workunit.client.0.smithi007.stderr:131950K .......... .......... .......... .......... .......... 97% 23.8M 0s 2024-04-21T08:45:42.578 INFO:tasks.workunit.client.0.smithi007.stderr:132000K .......... .......... .......... .......... .......... 97% 22.6M 0s 2024-04-21T08:45:42.579 INFO:tasks.workunit.client.0.smithi007.stderr:132050K .......... .......... .......... .......... .......... 97% 53.2M 0s 2024-04-21T08:45:42.581 INFO:tasks.workunit.client.0.smithi007.stderr:132100K .......... .......... .......... .......... .......... 97% 31.7M 0s 2024-04-21T08:45:42.582 INFO:tasks.workunit.client.0.smithi007.stderr:132150K .......... .......... .......... .......... .......... 97% 26.8M 0s 2024-04-21T08:45:42.584 INFO:tasks.workunit.client.0.smithi007.stderr:132200K .......... .......... .......... .......... .......... 97% 23.9M 0s 2024-04-21T08:45:42.587 INFO:tasks.workunit.client.0.smithi007.stderr:132250K .......... .......... .......... .......... .......... 97% 24.1M 0s 2024-04-21T08:45:42.587 INFO:tasks.workunit.client.0.smithi007.stderr:132300K .......... .......... .......... .......... .......... 97% 51.0M 0s 2024-04-21T08:45:42.589 INFO:tasks.workunit.client.0.smithi007.stderr:132350K .......... .......... .......... .......... .......... 97% 23.4M 0s 2024-04-21T08:45:42.591 INFO:tasks.workunit.client.0.smithi007.stderr:132400K .......... .......... .......... .......... .......... 97% 38.5M 0s 2024-04-21T08:45:42.592 INFO:tasks.workunit.client.0.smithi007.stderr:132450K .......... .......... .......... .......... .......... 97% 25.1M 0s 2024-04-21T08:45:42.595 INFO:tasks.workunit.client.0.smithi007.stderr:132500K .......... .......... .......... .......... .......... 97% 24.5M 0s 2024-04-21T08:45:42.595 INFO:tasks.workunit.client.0.smithi007.stderr:132550K .......... .......... .......... .......... .......... 97% 50.8M 0s 2024-04-21T08:45:42.597 INFO:tasks.workunit.client.0.smithi007.stderr:132600K .......... .......... .......... .......... .......... 97% 31.2M 0s 2024-04-21T08:45:42.599 INFO:tasks.workunit.client.0.smithi007.stderr:132650K .......... .......... .......... .......... .......... 97% 25.7M 0s 2024-04-21T08:45:42.601 INFO:tasks.workunit.client.0.smithi007.stderr:132700K .......... .......... .......... .......... .......... 97% 20.7M 0s 2024-04-21T08:45:42.603 INFO:tasks.workunit.client.0.smithi007.stderr:132750K .......... .......... .......... .......... .......... 97% 23.0M 0s 2024-04-21T08:45:42.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:42 smithi077 ceph-mon[31615]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-21T08:45:42.604 INFO:tasks.workunit.client.0.smithi007.stderr:132800K .......... .......... .......... .......... .......... 97% 48.1M 0s 2024-04-21T08:45:42.606 INFO:tasks.workunit.client.0.smithi007.stderr:132850K .......... .......... .......... .......... .......... 97% 23.5M 0s 2024-04-21T08:45:42.607 INFO:tasks.workunit.client.0.smithi007.stderr:132900K .......... .......... .......... .......... .......... 97% 40.0M 0s 2024-04-21T08:45:42.610 INFO:tasks.workunit.client.0.smithi007.stderr:132950K .......... .......... .......... .......... .......... 97% 24.2M 0s 2024-04-21T08:45:42.612 INFO:tasks.workunit.client.0.smithi007.stderr:133000K .......... .......... .......... .......... .......... 98% 22.7M 0s 2024-04-21T08:45:42.613 INFO:tasks.workunit.client.0.smithi007.stderr:133050K .......... .......... .......... .......... .......... 98% 44.9M 0s 2024-04-21T08:45:42.615 INFO:tasks.workunit.client.0.smithi007.stderr:133100K .......... .......... .......... .......... .......... 98% 21.4M 0s 2024-04-21T08:45:42.616 INFO:tasks.workunit.client.0.smithi007.stderr:133150K .......... .......... .......... .......... .......... 98% 42.2M 0s 2024-04-21T08:45:42.618 INFO:tasks.workunit.client.0.smithi007.stderr:133200K .......... .......... .......... .......... .......... 98% 22.9M 0s 2024-04-21T08:45:42.621 INFO:tasks.workunit.client.0.smithi007.stderr:133250K .......... .......... .......... .......... .......... 98% 20.9M 0s 2024-04-21T08:45:42.622 INFO:tasks.workunit.client.0.smithi007.stderr:133300K .......... .......... .......... .......... .......... 98% 37.1M 0s 2024-04-21T08:45:42.624 INFO:tasks.workunit.client.0.smithi007.stderr:133350K .......... .......... .......... .......... .......... 98% 29.6M 0s 2024-04-21T08:45:42.625 INFO:tasks.workunit.client.0.smithi007.stderr:133400K .......... .......... .......... .......... .......... 98% 31.3M 0s 2024-04-21T08:45:42.627 INFO:tasks.workunit.client.0.smithi007.stderr:133450K .......... .......... .......... .......... .......... 98% 22.2M 0s 2024-04-21T08:45:42.629 INFO:tasks.workunit.client.0.smithi007.stderr:133500K .......... .......... .......... .......... .......... 98% 23.9M 0s 2024-04-21T08:45:42.630 INFO:tasks.workunit.client.0.smithi007.stderr:133550K .......... .......... .......... .......... .......... 98% 48.2M 0s 2024-04-21T08:45:42.633 INFO:tasks.workunit.client.0.smithi007.stderr:133600K .......... .......... .......... .......... .......... 98% 22.5M 0s 2024-04-21T08:45:42.634 INFO:tasks.workunit.client.0.smithi007.stderr:133650K .......... .......... .......... .......... .......... 98% 48.5M 0s 2024-04-21T08:45:42.636 INFO:tasks.workunit.client.0.smithi007.stderr:133700K .......... .......... .......... .......... .......... 98% 21.6M 0s 2024-04-21T08:45:42.638 INFO:tasks.workunit.client.0.smithi007.stderr:133750K .......... .......... .......... .......... .......... 98% 23.8M 0s 2024-04-21T08:45:42.639 INFO:tasks.workunit.client.0.smithi007.stderr:133800K .......... .......... .......... .......... .......... 98% 54.7M 0s 2024-04-21T08:45:42.641 INFO:tasks.workunit.client.0.smithi007.stderr:133850K .......... .......... .......... .......... .......... 98% 23.2M 0s 2024-04-21T08:45:42.642 INFO:tasks.workunit.client.0.smithi007.stderr:133900K .......... .......... .......... .......... .......... 98% 36.2M 0s 2024-04-21T08:45:42.644 INFO:tasks.workunit.client.0.smithi007.stderr:133950K .......... .......... .......... .......... .......... 98% 23.4M 0s 2024-04-21T08:45:42.646 INFO:tasks.workunit.client.0.smithi007.stderr:134000K .......... .......... .......... .......... .......... 98% 24.8M 0s 2024-04-21T08:45:42.647 INFO:tasks.workunit.client.0.smithi007.stderr:134050K .......... .......... .......... .......... .......... 98% 43.6M 0s 2024-04-21T08:45:42.649 INFO:tasks.workunit.client.0.smithi007.stderr:134100K .......... .......... .......... .......... .......... 98% 23.6M 0s 2024-04-21T08:45:42.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:42 smithi007 ceph-mon[28477]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-21T08:45:42.652 INFO:tasks.workunit.client.0.smithi007.stderr:134150K .......... .......... .......... .......... .......... 98% 30.4M 0s 2024-04-21T08:45:42.653 INFO:tasks.workunit.client.0.smithi007.stderr:134200K .......... .......... .......... .......... .......... 98% 26.2M 0s 2024-04-21T08:45:42.655 INFO:tasks.workunit.client.0.smithi007.stderr:134250K .......... .......... .......... .......... .......... 98% 19.9M 0s 2024-04-21T08:45:42.656 INFO:tasks.workunit.client.0.smithi007.stderr:134300K .......... .......... .......... .......... .......... 98% 42.8M 0s 2024-04-21T08:45:42.659 INFO:tasks.workunit.client.0.smithi007.stderr:134350K .......... .......... .......... .......... .......... 99% 22.8M 0s 2024-04-21T08:45:42.661 INFO:tasks.workunit.client.0.smithi007.stderr:134400K .......... .......... .......... .......... .......... 99% 34.6M 0s 2024-04-21T08:45:42.662 INFO:tasks.workunit.client.0.smithi007.stderr:134450K .......... .......... .......... .......... .......... 99% 24.3M 0s 2024-04-21T08:45:42.664 INFO:tasks.workunit.client.0.smithi007.stderr:134500K .......... .......... .......... .......... .......... 99% 23.0M 0s 2024-04-21T08:45:42.665 INFO:tasks.workunit.client.0.smithi007.stderr:134550K .......... .......... .......... .......... .......... 99% 53.5M 0s 2024-04-21T08:45:42.667 INFO:tasks.workunit.client.0.smithi007.stderr:134600K .......... .......... .......... .......... .......... 99% 23.2M 0s 2024-04-21T08:45:42.669 INFO:tasks.workunit.client.0.smithi007.stderr:134650K .......... .......... .......... .......... .......... 99% 24.8M 0s 2024-04-21T08:45:42.671 INFO:tasks.workunit.client.0.smithi007.stderr:134700K .......... .......... .......... .......... .......... 99% 40.4M 0s 2024-04-21T08:45:42.673 INFO:tasks.workunit.client.0.smithi007.stderr:134750K .......... .......... .......... .......... .......... 99% 23.1M 0s 2024-04-21T08:45:42.673 INFO:tasks.workunit.client.0.smithi007.stderr:134800K .......... .......... .......... .......... .......... 99% 58.0M 0s 2024-04-21T08:45:42.676 INFO:tasks.workunit.client.0.smithi007.stderr:134850K .......... .......... .......... .......... .......... 99% 22.3M 0s 2024-04-21T08:45:42.678 INFO:tasks.workunit.client.0.smithi007.stderr:134900K .......... .......... .......... .......... .......... 99% 23.4M 0s 2024-04-21T08:45:42.680 INFO:tasks.workunit.client.0.smithi007.stderr:134950K .......... .......... .......... .......... .......... 99% 36.4M 0s 2024-04-21T08:45:42.681 INFO:tasks.workunit.client.0.smithi007.stderr:135000K .......... .......... .......... .......... .......... 99% 26.2M 0s 2024-04-21T08:45:42.682 INFO:tasks.workunit.client.0.smithi007.stderr:135050K .......... .......... .......... .......... .......... 99% 47.8M 0s 2024-04-21T08:45:42.684 INFO:tasks.workunit.client.0.smithi007.stderr:135100K .......... .......... .......... .......... .......... 99% 22.4M 0s 2024-04-21T08:45:42.686 INFO:tasks.workunit.client.0.smithi007.stderr:135150K .......... .......... .......... .......... .......... 99% 20.6M 0s 2024-04-21T08:45:42.688 INFO:tasks.workunit.client.0.smithi007.stderr:135200K .......... .......... .......... .......... .......... 99% 41.9M 0s 2024-04-21T08:45:42.690 INFO:tasks.workunit.client.0.smithi007.stderr:135250K .......... .......... .......... .......... .......... 99% 20.5M 0s 2024-04-21T08:45:42.691 INFO:tasks.workunit.client.0.smithi007.stderr:135300K .......... .......... .......... .......... .......... 99% 47.7M 0s 2024-04-21T08:45:42.693 INFO:tasks.workunit.client.0.smithi007.stderr:135350K .......... .......... .......... .......... .......... 99% 24.0M 0s 2024-04-21T08:45:42.695 INFO:tasks.workunit.client.0.smithi007.stderr:135400K .......... .......... .......... .......... .......... 99% 23.6M 0s 2024-04-21T08:45:42.697 INFO:tasks.workunit.client.0.smithi007.stderr:135450K .......... .......... .......... .......... .......... 99% 35.5M 0s 2024-04-21T08:45:42.699 INFO:tasks.workunit.client.0.smithi007.stderr:135500K .......... .......... .......... .......... .......... 99% 24.5M 0s 2024-04-21T08:45:42.700 INFO:tasks.workunit.client.0.smithi007.stderr:135550K .......... .......... .......... .......... .......... 99% 45.3M 0s 2024-04-21T08:45:42.701 INFO:tasks.workunit.client.0.smithi007.stderr:135600K .......... .......... .......... .......... .......... 99% 25.6M 0s 2024-04-21T08:45:42.704 INFO:tasks.workunit.client.0.smithi007.stderr:135650K .......... .......... .......... .......... .......... 99% 24.1M 0s 2024-04-21T08:45:42.705 INFO:tasks.workunit.client.0.smithi007.stderr:135700K .......... .......... .......... .......... .......... 99% 43.1M 0s 2024-04-21T08:45:42.705 INFO:tasks.workunit.client.0.smithi007.stderr:135750K .. 100% 5.30T=4.7s 2024-04-21T08:45:42.705 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-04-21T08:45:42.705 INFO:tasks.workunit.client.0.smithi007.stderr:2024-04-21 08:45:42 (27.9 MB/s) - ‘linux.tar.xz’ saved [139010916/139010916] 2024-04-21T08:45:42.705 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-04-21T08:45:42.705 INFO:tasks.workunit.client.0.smithi007.stderr:+ mkdir t 2024-04-21T08:45:42.707 INFO:tasks.workunit.client.0.smithi007.stderr:+ cd t 2024-04-21T08:45:42.708 INFO:tasks.workunit.client.0.smithi007.stderr:+ tar xJf ../linux.tar.xz 2024-04-21T08:45:42.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:42 smithi016 ceph-mon[31259]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-21T08:45:44.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:44 smithi007 ceph-mon[28477]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.8 KiB/s wr, 7 op/s 2024-04-21T08:45:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:44 smithi016 ceph-mon[31259]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.8 KiB/s wr, 7 op/s 2024-04-21T08:45:44.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:44 smithi077 ceph-mon[31615]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.8 KiB/s wr, 7 op/s 2024-04-21T08:45:46.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:46 smithi007 ceph-mon[28477]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2024-04-21T08:45:46.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:46 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:46.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:46 smithi016 ceph-mon[31259]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2024-04-21T08:45:46.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:46 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:46 smithi077 ceph-mon[31615]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2024-04-21T08:45:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:46 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:48.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:47 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:47 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:47 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:47 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:47 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:47 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:47 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:48.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:47 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:48.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:47 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:47 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:47 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:47 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:47 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:47 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:47 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:47 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:48.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:47 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:48.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:47 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:47 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:47 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:47 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:47 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:47 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:47 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:48.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:47 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:48.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:47 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:48.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:47 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:45:49.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:48 smithi077 ceph-mon[31615]: pgmap v36: 129 pgs: 129 active+clean; 103 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 8.5 MiB/s wr, 27 op/s 2024-04-21T08:45:49.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:48 smithi007 ceph-mon[28477]: pgmap v36: 129 pgs: 129 active+clean; 103 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 8.5 MiB/s wr, 27 op/s 2024-04-21T08:45:49.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:45:49.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:45:49.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.287+0000 7f06e248c640 1 -- 172.21.15.7:0/496750377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc069f20 msgr2=0x7f06dc106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.287+0000 7f06e248c640 1 --2- 172.21.15.7:0/496750377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc069f20 0x7f06dc106b50 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009900 tx=0x7f06d002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.287+0000 7f06e248c640 1 -- 172.21.15.7:0/496750377 shutdown_connections 2024-04-21T08:45:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.287+0000 7f06e248c640 1 --2- 172.21.15.7:0/496750377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06dc107230 0x7f06dc10ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.287+0000 7f06e248c640 1 --2- 172.21.15.7:0/496750377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc069f20 0x7f06dc106b50 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.287+0000 7f06e248c640 1 --2- 172.21.15.7:0/496750377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06dc069570 0x7f06dc069950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.287+0000 7f06e248c640 1 -- 172.21.15.7:0/496750377 >> 172.21.15.7:0/496750377 conn(0x7f06dc077fa0 msgr2=0x7f06dc0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.287+0000 7f06e248c640 1 -- 172.21.15.7:0/496750377 shutdown_connections 2024-04-21T08:45:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.287+0000 7f06e248c640 1 -- 172.21.15.7:0/496750377 wait complete. 2024-04-21T08:45:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.287+0000 7f06e248c640 1 Processor -- start 2024-04-21T08:45:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e248c640 1 -- start start 2024-04-21T08:45:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e248c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06dc069570 0x7f06dc1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e248c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06dc107230 0x7f06dc198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e248c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc19d020 0x7f06dc19d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e248c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc112170 con 0x7f06dc19d020 2024-04-21T08:45:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e248c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc111ff0 con 0x7f06dc107230 2024-04-21T08:45:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e248c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1122f0 con 0x7f06dc069570 2024-04-21T08:45:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e0a29640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc19d020 0x7f06dc19d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06db7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06dc107230 0x7f06dc198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e0a29640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc19d020 0x7f06dc19d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42646/0 (socket says 172.21.15.7:42646) 2024-04-21T08:45:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e0a29640 1 -- 172.21.15.7:0/803914730 learned_addr learned my addr 172.21.15.7:0/803914730 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06dbfff640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06dc069570 0x7f06dc1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e0a29640 1 -- 172.21.15.7:0/803914730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06dc069570 msgr2=0x7f06dc1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e0a29640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06dc069570 0x7f06dc1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e0a29640 1 -- 172.21.15.7:0/803914730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06dc107230 msgr2=0x7f06dc198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e0a29640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06dc107230 0x7f06dc198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06e0a29640 1 -- 172.21.15.7:0/803914730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06dc19db10 con 0x7f06dc19d020 2024-04-21T08:45:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06db7fe640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06dc107230 0x7f06dc198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.288+0000 7f06dbfff640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06dc069570 0x7f06dc1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.289+0000 7f06e0a29640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc19d020 0x7f06dc19d4d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f06d400efd0 tx=0x7f06d400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.289+0000 7f06d97fa640 1 -- 172.21.15.7:0/803914730 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4003b70 con 0x7f06dc19d020 2024-04-21T08:45:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.289+0000 7f06d97fa640 1 -- 172.21.15.7:0/803914730 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d4003d10 con 0x7f06dc19d020 2024-04-21T08:45:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.289+0000 7f06d97fa640 1 -- 172.21.15.7:0/803914730 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d40108a0 con 0x7f06dc19d020 2024-04-21T08:45:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.289+0000 7f06e248c640 1 -- 172.21.15.7:0/803914730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06dc19de00 con 0x7f06dc19d020 2024-04-21T08:45:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.289+0000 7f06e248c640 1 -- 172.21.15.7:0/803914730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06dc1a55c0 con 0x7f06dc19d020 2024-04-21T08:45:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.291+0000 7f06d97fa640 1 -- 172.21.15.7:0/803914730 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f06d4004020 con 0x7f06dc19d020 2024-04-21T08:45:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e5082740 1 Processor -- start 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e5082740 1 -- start start 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e5082740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aecc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e5082740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aeccbd0 0x55b01aed9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e5082740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 0x55b01aedc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06d97fa640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f06c40a3ec0 0x7f06c40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e5082740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b01ae311e0 con 0x55b01aecc2b0 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e5082740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b01adce390 con 0x55b01aed9e90 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e5082740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b01ae2d1a0 con 0x55b01aeccbd0 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06db7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aecc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06db7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aecc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42658/0 (socket says 172.21.15.7:42658) 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e0a29640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 0x55b01aedc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06db7fe640 1 -- 172.21.15.7:0/1107281828 learned_addr learned my addr 172.21.15.7:0/1107281828 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06db7fe640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f06c40a3ec0 0x7f06c40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06d97fa640 1 -- 172.21.15.7:0/803914730 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f06d40d1310 con 0x7f06dc19d020 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06dbfff640 1 --2- 172.21.15.7:0/1107281828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aeccbd0 0x55b01aed9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e0a29640 1 -- 172.21.15.7:0/1107281828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aeccbd0 msgr2=0x55b01aed9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e0a29640 1 --2- 172.21.15.7:0/1107281828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aeccbd0 0x55b01aed9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e0a29640 1 -- 172.21.15.7:0/1107281828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 msgr2=0x55b01aecc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e0a29640 1 --2- 172.21.15.7:0/1107281828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aecc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06e0a29640 1 -- 172.21.15.7:0/1107281828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b01ac0d7d0 con 0x55b01aed9e90 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.292+0000 7f06db7fe640 1 --2- 172.21.15.7:0/1107281828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aecc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06e0a29640 1 --2- 172.21.15.7:0/1107281828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 0x55b01aedc280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f06d40ce490 tx=0x7f06d4098fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a21b90f7f7294183 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06db7fe640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f06c40a3ec0 0x7f06c40a6380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f06d0005ec0 tx=0x7f06d0005fd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06c27fc640 1 -- 172.21.15.7:0/1107281828 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4009820 con 0x55b01aed9e90 2024-04-21T08:45:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06c27fc640 1 -- 172.21.15.7:0/1107281828 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d409d310 con 0x55b01aed9e90 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06c27fc640 1 -- 172.21.15.7:0/1107281828 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d409afc0 con 0x55b01aed9e90 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06e5082740 1 -- 172.21.15.7:0/1107281828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 msgr2=0x55b01aedc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06e5082740 1 --2- 172.21.15.7:0/1107281828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 0x55b01aedc280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f06d40ce490 tx=0x7f06d4098fe0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06e5082740 1 -- 172.21.15.7:0/1107281828 shutdown_connections 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06e5082740 1 --2- 172.21.15.7:0/1107281828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 0x55b01aedc280 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06e5082740 1 --2- 172.21.15.7:0/1107281828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aeccbd0 0x55b01aed9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06e5082740 1 --2- 172.21.15.7:0/1107281828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aecc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.293+0000 7f06e5082740 1 -- 172.21.15.7:0/1107281828 >> 172.21.15.7:0/1107281828 conn(0x55b01aec6cb0 msgr2=0x55b01aec7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06e5082740 1 -- 172.21.15.7:0/1107281828 shutdown_connections 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06e5082740 1 -- 172.21.15.7:0/1107281828 wait complete. 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06e5082740 1 -- start start 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06e5082740 1 Processor -- start 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06e5082740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aecc2b0 0x55b01aed6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06e5082740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aeccbd0 0x55b01aed7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06e5082740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 0x55b01aed79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06e5082740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b01aedcbf0 con 0x55b01aeccbd0 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06e5082740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b01adea2f0 con 0x55b01aed9e90 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06e5082740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b01aedcd70 con 0x55b01aecc2b0 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06e5082740 10 client.0 _ll_register_callbacks cb 0x55b01adf7580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.294+0000 7f06db7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aeccbd0 0x55b01aed7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06db7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aeccbd0 0x55b01aed7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42666/0 (socket says 172.21.15.7:42666) 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06db7fe640 1 -- 172.21.15.7:0/203982652 learned_addr learned my addr 172.21.15.7:0/203982652 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06e0a29640 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 0x55b01aed79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06dbfff640 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aecc2b0 0x55b01aed6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06db7fe640 1 -- 172.21.15.7:0/203982652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aecc2b0 msgr2=0x55b01aed6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06db7fe640 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aecc2b0 0x55b01aed6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06db7fe640 1 -- 172.21.15.7:0/203982652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 msgr2=0x55b01aed79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06db7fe640 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 0x55b01aed79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06db7fe640 1 -- 172.21.15.7:0/203982652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b01ab433f0 con 0x55b01aeccbd0 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06e0a29640 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 0x55b01aed79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06dbfff640 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aecc2b0 0x55b01aed6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06db7fe640 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aeccbd0 0x55b01aed7470 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f06d0036040 tx=0x7f06d0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e039f8aafb04d8eb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06bb7fe640 1 -- 172.21.15.7:0/203982652 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0003d50 con 0x55b01aeccbd0 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06bb7fe640 1 -- 172.21.15.7:0/203982652 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f06d0050460 con 0x55b01aeccbd0 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06e5082740 1 -- 172.21.15.7:0/203982652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d409d310 con 0x55b01aeccbd0 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06bb7fe640 1 -- 172.21.15.7:0/203982652 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f06d005e050 con 0x55b01aeccbd0 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06e5082740 10 client.15972 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.295+0000 7f06e5082740 1 -- 172.21.15.7:0/203982652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b01ab030f0 con 0x55b01aeccbd0 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.296+0000 7f06bb7fe640 1 -- 172.21.15.7:0/203982652 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06d409d310 con 0x55b01aeccbd0 2024-04-21T08:45:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.296+0000 7f06bb7fe640 1 -- 172.21.15.7:0/203982652 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f06d0043570 con 0x55b01aeccbd0 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.296+0000 7f06e5082740 10 client.15972 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.296+0000 7f06e5082740 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.296+0000 7f06e5082740 20 client.15972 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.296+0000 7f06e5082740 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b01aee9e90 0x55b01aeec2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.296+0000 7f06e5082740 4 client.15972 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.296+0000 7f06e5082740 1 -- 172.21.15.7:0/203982652 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b01ae3ec00 con 0x55b01aee9e90 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.296+0000 7f06dbfff640 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b01aee9e90 0x55b01aeec2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.296+0000 7f06dbfff640 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b01aee9e90 0x55b01aeec2e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7421f20270a8fb97 server_cookie=c52f577a39761204 in_seq=0 out_seq=0 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.296+0000 7f06bb7fe640 10 client.15972 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.297+0000 7f06bb7fe640 1 -- 172.21.15.7:0/203982652 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b01ae3ec00 con 0x55b01aee9e90 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.297+0000 7f06bb7fe640 10 client.15972 handle_command_reply: tid=0 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 1 client.15972 shutdown 2024-04-21T08:45:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 1 -- 172.21.15.7:0/203982652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aeccbd0 msgr2=0x55b01aed7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aeccbd0 0x55b01aed7470 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f06d0036040 tx=0x7f06d0009260 comp rx=0 tx=0).stop 2024-04-21T08:45:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 1 -- 172.21.15.7:0/203982652 shutdown_connections 2024-04-21T08:45:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b01aee9e90 0x55b01aeec2e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aed9e90 0x55b01aed79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aeccbd0 0x55b01aed7470 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06bb7fe640 0 client.15972 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 1 --2- 172.21.15.7:0/203982652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aecc2b0 0x55b01aed6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 1 -- 172.21.15.7:0/203982652 >> 172.21.15.7:0/203982652 conn(0x55b01aec6cb0 msgr2=0x55b01aec8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 1 -- 172.21.15.7:0/203982652 shutdown_connections 2024-04-21T08:45:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 1 -- 172.21.15.7:0/203982652 wait complete. 2024-04-21T08:45:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.298+0000 7f06e5082740 20 client.15972 trim_cache size 0 max 16384 2024-04-21T08:45:49.307 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:48 smithi016 ceph-mon[31259]: pgmap v36: 129 pgs: 129 active+clean; 103 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 8.5 MiB/s wr, 27 op/s 2024-04-21T08:45:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.316+0000 7f06e5082740 1 Processor -- start 2024-04-21T08:45:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.316+0000 7f06e5082740 1 -- start start 2024-04-21T08:45:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06e5082740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aed9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06e5082740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aeccbd0 0x55b01aed5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06e5082740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aed9e90 0x55b01aee5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06e5082740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1ab1f0 con 0x55b01aecc2b0 2024-04-21T08:45:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06e5082740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1ada70 con 0x55b01aeccbd0 2024-04-21T08:45:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06e5082740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1ab860 con 0x55b01aed9e90 2024-04-21T08:45:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06dbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aed9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06dbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aed9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42680/0 (socket says 172.21.15.7:42680) 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06dbfff640 1 -- 172.21.15.7:0/3621107200 learned_addr learned my addr 172.21.15.7:0/3621107200 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06db7fe640 1 --2- 172.21.15.7:0/3621107200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aeccbd0 0x55b01aed5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06dbfff640 1 -- 172.21.15.7:0/3621107200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aed9e90 msgr2=0x55b01aee5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06dbfff640 1 --2- 172.21.15.7:0/3621107200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aed9e90 0x55b01aee5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06dbfff640 1 -- 172.21.15.7:0/3621107200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aeccbd0 msgr2=0x55b01aed5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06dbfff640 1 --2- 172.21.15.7:0/3621107200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aeccbd0 0x55b01aed5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06dbfff640 1 -- 172.21.15.7:0/3621107200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b01aedcb30 con 0x55b01aecc2b0 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06dbfff640 1 --2- 172.21.15.7:0/3621107200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aed9660 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x55b01aeecb50 tx=0x7f06cc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67fea5695e7bf13 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06c1ffb640 1 -- 172.21.15.7:0/3621107200 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06cc0140d0 con 0x55b01aecc2b0 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06c1ffb640 1 -- 172.21.15.7:0/3621107200 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06cc014270 con 0x55b01aecc2b0 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.317+0000 7f06c1ffb640 1 -- 172.21.15.7:0/3621107200 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06cc012b80 con 0x55b01aecc2b0 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.318+0000 7f06e5082740 1 -- 172.21.15.7:0/3621107200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 msgr2=0x55b01aed9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.318+0000 7f06e5082740 1 --2- 172.21.15.7:0/3621107200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aed9660 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x55b01aeecb50 tx=0x7f06cc005850 comp rx=0 tx=0).stop 2024-04-21T08:45:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.318+0000 7f06e5082740 1 -- 172.21.15.7:0/3621107200 shutdown_connections 2024-04-21T08:45:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.318+0000 7f06e5082740 1 --2- 172.21.15.7:0/3621107200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aed9e90 0x55b01aee5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.318+0000 7f06e5082740 1 --2- 172.21.15.7:0/3621107200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aeccbd0 0x55b01aed5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.318+0000 7f06e5082740 1 --2- 172.21.15.7:0/3621107200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aed9660 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.318+0000 7f06e5082740 1 -- 172.21.15.7:0/3621107200 >> 172.21.15.7:0/3621107200 conn(0x55b01aec6cb0 msgr2=0x55b01aec8990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.318+0000 7f06e5082740 1 -- 172.21.15.7:0/3621107200 shutdown_connections 2024-04-21T08:45:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.318+0000 7f06e5082740 1 -- 172.21.15.7:0/3621107200 wait complete. 2024-04-21T08:45:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.318+0000 7f06e5082740 1 -- start start 2024-04-21T08:45:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06e5082740 1 Processor -- start 2024-04-21T08:45:49.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06e5082740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aedda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06e5082740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aeccbd0 0x55b01aeddfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06e5082740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aed9e90 0x55b01aede4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06e5082740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1abf30 con 0x55b01aecc2b0 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06e5082740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1ad3a0 con 0x55b01aeccbd0 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06e5082740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc1ac600 con 0x55b01aed9e90 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06e5082740 10 client.0 _ll_register_callbacks cb 0x7f06d0004150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06dbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aedda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06dbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aedda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42688/0 (socket says 172.21.15.7:42688) 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06dbfff640 1 -- 172.21.15.7:0/1151767079 learned_addr learned my addr 172.21.15.7:0/1151767079 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06db7fe640 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aeccbd0 0x55b01aeddfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06dbfff640 1 -- 172.21.15.7:0/1151767079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aed9e90 msgr2=0x55b01aede4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06e0a29640 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aed9e90 0x55b01aede4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06dbfff640 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aed9e90 0x55b01aede4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06dbfff640 1 -- 172.21.15.7:0/1151767079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aeccbd0 msgr2=0x55b01aeddfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06dbfff640 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aeccbd0 0x55b01aeddfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06dbfff640 1 -- 172.21.15.7:0/1151767079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b01af4f470 con 0x55b01aecc2b0 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06dbfff640 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aedda60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f06cc00b630 tx=0x7f06cc014d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221c31d649a8138 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.319+0000 7f06bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06bb7fe640 1 -- 172.21.15.7:0/1151767079 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06cc032060 con 0x55b01aecc2b0 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06bb7fe640 1 -- 172.21.15.7:0/1151767079 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f06cc01b330 con 0x55b01aecc2b0 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06e5082740 1 -- 172.21.15.7:0/1151767079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06cc014270 con 0x55b01aecc2b0 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06bb7fe640 1 -- 172.21.15.7:0/1151767079 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f06cc039050 con 0x55b01aecc2b0 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06e5082740 10 client.15978 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06e5082740 1 -- 172.21.15.7:0/1151767079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b01af50760 con 0x55b01aecc2b0 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06bb7fe640 1 -- 172.21.15.7:0/1151767079 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06cc014270 con 0x55b01aecc2b0 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06bb7fe640 1 -- 172.21.15.7:0/1151767079 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f06cc012a70 con 0x55b01aecc2b0 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06e5082740 10 client.15978 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:49.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06e5082740 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:49.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06e5082740 20 client.15978 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:49.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06e5082740 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b01af50df0 0x55b01af53210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06e5082740 4 client.15978 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:45:49.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06e5082740 1 -- 172.21.15.7:0/1151767079 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b01ae38a70 con 0x55b01af50df0 2024-04-21T08:45:49.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.320+0000 7f06db7fe640 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b01af50df0 0x55b01af53210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06db7fe640 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b01af50df0 0x55b01af53210 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3331eef859b8f525 server_cookie=34b1f43acb6b8368 in_seq=0 out_seq=0 2024-04-21T08:45:49.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06bb7fe640 10 client.15978 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:49.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06bb7fe640 1 -- 172.21.15.7:0/1151767079 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b01ae38a70 con 0x55b01af50df0 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06bb7fe640 10 client.15978 handle_command_reply: tid=0 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06e5082740 1 client.15978 shutdown 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06e5082740 1 -- 172.21.15.7:0/1151767079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 msgr2=0x55b01aedda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06e5082740 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aedda60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f06cc00b630 tx=0x7f06cc014d30 comp rx=0 tx=0).stop 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06e5082740 1 -- 172.21.15.7:0/1151767079 shutdown_connections 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06e5082740 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b01af50df0 0x55b01af53210 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06e5082740 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b01aed9e90 0x55b01aede4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06e5082740 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b01aeccbd0 0x55b01aeddfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06bb7fe640 0 client.15978 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06e5082740 1 --2- 172.21.15.7:0/1151767079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b01aecc2b0 0x55b01aedda60 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06e5082740 1 -- 172.21.15.7:0/1151767079 >> 172.21.15.7:0/1151767079 conn(0x55b01aec6cb0 msgr2=0x55b01aed6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06e5082740 1 -- 172.21.15.7:0/1151767079 shutdown_connections 2024-04-21T08:45:49.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.321+0000 7f06e5082740 1 -- 172.21.15.7:0/1151767079 wait complete. 2024-04-21T08:45:49.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e5082740 20 client.15978 trim_cache size 0 max 16384 2024-04-21T08:45:49.331 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:45:49.331 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:45:49.331 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:45:49.331 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:45:49.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 -- 172.21.15.7:0/803914730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f06c40a3ec0 msgr2=0x7f06c40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f06c40a3ec0 0x7f06c40a6380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f06d0005ec0 tx=0x7f06d0005fd0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 -- 172.21.15.7:0/803914730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc19d020 msgr2=0x7f06dc19d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc19d020 0x7f06dc19d4d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f06d400efd0 tx=0x7f06d400c6a0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 -- 172.21.15.7:0/803914730 shutdown_connections 2024-04-21T08:45:49.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc19d020 0x7f06dc19d4d0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f06dc107230 0x7f06dc198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f06c40a3ec0 0x7f06c40a6380 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 --2- 172.21.15.7:0/803914730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f06dc069570 0x7f06dc1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 -- 172.21.15.7:0/803914730 >> 172.21.15.7:0/803914730 conn(0x7f06dc077fa0 msgr2=0x7f06dc0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 -- 172.21.15.7:0/803914730 shutdown_connections 2024-04-21T08:45:49.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.322+0000 7f06e248c640 1 -- 172.21.15.7:0/803914730 wait complete. 2024-04-21T08:45:49.334 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:45:49.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:45:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.436+0000 7f8a95a74640 1 -- 172.21.15.7:0/3080055671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a90107260 msgr2=0x7f8a9010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.436+0000 7f8a95a74640 1 --2- 172.21.15.7:0/3080055671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a90107260 0x7f8a9010ee30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8a88009ca0 tx=0x7f8a8802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.436+0000 7f8a95a74640 1 -- 172.21.15.7:0/3080055671 shutdown_connections 2024-04-21T08:45:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.436+0000 7f8a95a74640 1 --2- 172.21.15.7:0/3080055671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a90107260 0x7f8a9010ee30 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.436+0000 7f8a95a74640 1 --2- 172.21.15.7:0/3080055671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90102b40 0x7f8a90106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.436+0000 7f8a95a74640 1 --2- 172.21.15.7:0/3080055671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a90102190 0x7f8a90102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.436+0000 7f8a95a74640 1 -- 172.21.15.7:0/3080055671 >> 172.21.15.7:0/3080055671 conn(0x7f8a900793e0 msgr2=0x7f8a900797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.436+0000 7f8a95a74640 1 -- 172.21.15.7:0/3080055671 shutdown_connections 2024-04-21T08:45:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.436+0000 7f8a95a74640 1 -- 172.21.15.7:0/3080055671 wait complete. 2024-04-21T08:45:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.437+0000 7f8a95a74640 1 Processor -- start 2024-04-21T08:45:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.437+0000 7f8a95a74640 1 -- start start 2024-04-21T08:45:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.437+0000 7f8a95a74640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a90102190 0x7f8a90074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.437+0000 7f8a95a74640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a90107260 0x7f8a90074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.437+0000 7f8a95a74640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90074cd0 0x7f8a901a6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.437+0000 7f8a95a74640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8a901121b0 con 0x7f8a90102190 2024-04-21T08:45:49.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.437+0000 7f8a95a74640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a90112030 con 0x7f8a90074cd0 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.437+0000 7f8a95a74640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8a90112330 con 0x7f8a90107260 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a90102190 0x7f8a90074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a90102190 0x7f8a90074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42706/0 (socket says 172.21.15.7:42706) 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8f7fe640 1 -- 172.21.15.7:0/1234607914 learned_addr learned my addr 172.21.15.7:0/1234607914 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8ffff640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90074cd0 0x7f8a901a6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8effd640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a90107260 0x7f8a90074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8ffff640 1 -- 172.21.15.7:0/1234607914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a90107260 msgr2=0x7f8a90074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8ffff640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a90107260 0x7f8a90074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8ffff640 1 -- 172.21.15.7:0/1234607914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a90102190 msgr2=0x7f8a90074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8ffff640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a90102190 0x7f8a90074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8ffff640 1 -- 172.21.15.7:0/1234607914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a901a6fa0 con 0x7f8a90074cd0 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8f7fe640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a90102190 0x7f8a90074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8ffff640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90074cd0 0x7f8a901a6900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8a8802c700 tx=0x7f8a880346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8cff9640 1 -- 172.21.15.7:0/1234607914 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88041020 con 0x7f8a90074cd0 2024-04-21T08:45:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a95a74640 1 -- 172.21.15.7:0/1234607914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a901a7230 con 0x7f8a90074cd0 2024-04-21T08:45:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8cff9640 1 -- 172.21.15.7:0/1234607914 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a88032970 con 0x7f8a90074cd0 2024-04-21T08:45:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a8cff9640 1 -- 172.21.15.7:0/1234607914 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88039040 con 0x7f8a90074cd0 2024-04-21T08:45:49.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.438+0000 7f8a95a74640 1 -- 172.21.15.7:0/1234607914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a901a7740 con 0x7f8a90074cd0 2024-04-21T08:45:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.439+0000 7f8a95a74640 1 -- 172.21.15.7:0/1234607914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a90068570 con 0x7f8a90074cd0 2024-04-21T08:45:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.441+0000 7f8a8cff9640 1 -- 172.21.15.7:0/1234607914 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f8a88044020 con 0x7f8a90074cd0 2024-04-21T08:45:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.441+0000 7f8a8cff9640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a780a3e40 0x7f8a780a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.441+0000 7f8a8f7fe640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a780a3e40 0x7f8a780a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.442+0000 7f8a8cff9640 1 -- 172.21.15.7:0/1234607914 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f8a880f2e20 con 0x7f8a90074cd0 2024-04-21T08:45:49.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.442+0000 7f8a8f7fe640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a780a3e40 0x7f8a780a6300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a80011550 tx=0x7f8a80005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.450+0000 7f8a8cff9640 1 -- 172.21.15.7:0/1234607914 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a8804b020 con 0x7f8a90074cd0 2024-04-21T08:45:49.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.682+0000 7f8a95a74640 1 -- 172.21.15.7:0/1234607914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a9010a4f0 con 0x7f8a90074cd0 2024-04-21T08:45:49.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.684+0000 7f8a8cff9640 1 -- 172.21.15.7:0/1234607914 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11782 (secure 0 0 0) 0x7f8a88003970 con 0x7f8a90074cd0 2024-04-21T08:45:49.684 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:45:49.684 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:42:42.080753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/516970569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":516970569},{"type":"v1","addr":"172.21.15.77:6839","nonce":516970569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:45:49.685 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 -- 172.21.15.7:0/1234607914 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a780a3e40 msgr2=0x7f8a780a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a780a3e40 0x7f8a780a6300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a80011550 tx=0x7f8a80005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 -- 172.21.15.7:0/1234607914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90074cd0 msgr2=0x7f8a901a6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90074cd0 0x7f8a901a6900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8a8802c700 tx=0x7f8a880346f0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 -- 172.21.15.7:0/1234607914 shutdown_connections 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a90074cd0 0x7f8a901a6900 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8a780a3e40 0x7f8a780a6300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a90107260 0x7f8a90074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 --2- 172.21.15.7:0/1234607914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a90102190 0x7f8a90074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 -- 172.21.15.7:0/1234607914 >> 172.21.15.7:0/1234607914 conn(0x7f8a900793e0 msgr2=0x7f8a900ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 -- 172.21.15.7:0/1234607914 shutdown_connections 2024-04-21T08:45:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.686+0000 7f8a95a74640 1 -- 172.21.15.7:0/1234607914 wait complete. 2024-04-21T08:45:49.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:45:49.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.804+0000 7f122ddce640 1 -- 172.21.15.7:0/391383109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f122810c500 msgr2=0x7f122810e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.804+0000 7f122ddce640 1 --2- 172.21.15.7:0/391383109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f122810c500 0x7f122810e9a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f1220009ca0 tx=0x7f122002c180 comp rx=0 tx=0).stop 2024-04-21T08:45:49.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.804+0000 7f122ddce640 1 -- 172.21.15.7:0/391383109 shutdown_connections 2024-04-21T08:45:49.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.804+0000 7f122ddce640 1 --2- 172.21.15.7:0/391383109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f122810c500 0x7f122810e9a0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.804+0000 7f122ddce640 1 --2- 172.21.15.7:0/391383109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1228103ae0 0x7f122810bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.804+0000 7f122ddce640 1 --2- 172.21.15.7:0/391383109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12281031c0 0x7f12281035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.804+0000 7f122ddce640 1 -- 172.21.15.7:0/391383109 >> 172.21.15.7:0/391383109 conn(0x7f1228078000 msgr2=0x7f1228078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.804+0000 7f122ddce640 1 -- 172.21.15.7:0/391383109 shutdown_connections 2024-04-21T08:45:49.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.804+0000 7f122ddce640 1 -- 172.21.15.7:0/391383109 wait complete. 2024-04-21T08:45:49.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f122ddce640 1 Processor -- start 2024-04-21T08:45:49.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f122ddce640 1 -- start start 2024-04-21T08:45:49.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f122ddce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12281031c0 0x7f1228198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f122ddce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1228103ae0 0x7f1228198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f122ddce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f122810c500 0x7f122819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f122ddce640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1228112190 con 0x7f122810c500 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f122ddce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1228112010 con 0x7f12281031c0 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f122ddce640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1228112310 con 0x7f1228103ae0 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f1227fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f122810c500 0x7f122819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f12277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12281031c0 0x7f1228198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f12277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12281031c0 0x7f1228198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:37122/0 (socket says 172.21.15.7:37122) 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f12277fe640 1 -- 172.21.15.7:0/188720261 learned_addr learned my addr 172.21.15.7:0/188720261 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.805+0000 7f1227fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f122810c500 0x7f122819cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42718/0 (socket says 172.21.15.7:42718) 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f1226ffd640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1228103ae0 0x7f1228198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f12277fe640 1 -- 172.21.15.7:0/188720261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1228103ae0 msgr2=0x7f1228198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f12277fe640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1228103ae0 0x7f1228198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f12277fe640 1 -- 172.21.15.7:0/188720261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f122810c500 msgr2=0x7f122819cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f12277fe640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f122810c500 0x7f122819cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f12277fe640 1 -- 172.21.15.7:0/188720261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122819d6d0 con 0x7f12281031c0 2024-04-21T08:45:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f1227fff640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f122810c500 0x7f122819cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f12277fe640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12281031c0 0x7f1228198730 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f121800e9f0 tx=0x7f121800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f1224ff9640 1 -- 172.21.15.7:0/188720261 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121800cd80 con 0x7f12281031c0 2024-04-21T08:45:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f122ddce640 1 -- 172.21.15.7:0/188720261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f122819d9c0 con 0x7f12281031c0 2024-04-21T08:45:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f1224ff9640 1 -- 172.21.15.7:0/188720261 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12180054d0 con 0x7f12281031c0 2024-04-21T08:45:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f1224ff9640 1 -- 172.21.15.7:0/188720261 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218010690 con 0x7f12281031c0 2024-04-21T08:45:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.806+0000 7f122ddce640 1 -- 172.21.15.7:0/188720261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12281a5250 con 0x7f12281031c0 2024-04-21T08:45:49.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.808+0000 7f12309bf740 1 Processor -- start 2024-04-21T08:45:49.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.808+0000 7f1224ff9640 1 -- 172.21.15.7:0/188720261 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f1218004e90 con 0x7f12281031c0 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f12309bf740 1 -- start start 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f12309bf740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f4258b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f12309bf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f425e80 0x55564f42e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f12309bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f42ea00 0x55564f430e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f12309bf740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55564f38a1f0 con 0x55564f4254d0 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f12309bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55564f3274b0 con 0x55564f42ea00 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f12309bf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55564f3861b0 con 0x55564f425e80 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1227fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f4258b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1227fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f4258b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42726/0 (socket says 172.21.15.7:42726) 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1227fff640 1 -- 172.21.15.7:0/4016528890 learned_addr learned my addr 172.21.15.7:0/4016528890 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1226ffd640 1 --2- 172.21.15.7:0/4016528890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f42ea00 0x55564f430e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f12277fe640 1 --2- 172.21.15.7:0/4016528890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f425e80 0x55564f42e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1224ff9640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f12100a3e40 0x7f12100a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1227fff640 1 -- 172.21.15.7:0/4016528890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f425e80 msgr2=0x55564f42e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1227fff640 1 --2- 172.21.15.7:0/4016528890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f425e80 0x55564f42e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1227fff640 1 -- 172.21.15.7:0/4016528890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f42ea00 msgr2=0x55564f430e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1227fff640 1 --2- 172.21.15.7:0/4016528890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f42ea00 0x55564f430e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1227fff640 1 -- 172.21.15.7:0/4016528890 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55564f3191b0 con 0x55564f4254d0 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f12277fe640 1 --2- 172.21.15.7:0/4016528890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f425e80 0x55564f42e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1224ff9640 1 -- 172.21.15.7:0/188720261 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f12180d1340 con 0x7f12281031c0 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.809+0000 7f1227fff640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f12100a3e40 0x7f12100a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f1227fff640 1 --2- 172.21.15.7:0/4016528890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f4258b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1220009ca0 tx=0x7f1220009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff301b57620be2a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f120dffb640 1 -- 172.21.15.7:0/4016528890 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220039040 con 0x55564f4254d0 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f120dffb640 1 -- 172.21.15.7:0/4016528890 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1220043030 con 0x55564f4254d0 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f120dffb640 1 -- 172.21.15.7:0/4016528890 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220004800 con 0x55564f4254d0 2024-04-21T08:45:49.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f12309bf740 1 -- 172.21.15.7:0/4016528890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 msgr2=0x55564f4258b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f1227fff640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f12100a3e40 0x7f12100a6300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1220047000 tx=0x7f12200325e0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f12309bf740 1 --2- 172.21.15.7:0/4016528890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f4258b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1220009ca0 tx=0x7f1220009900 comp rx=0 tx=0).stop 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f12309bf740 1 -- 172.21.15.7:0/4016528890 shutdown_connections 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f12309bf740 1 --2- 172.21.15.7:0/4016528890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f42ea00 0x55564f430e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f12309bf740 1 --2- 172.21.15.7:0/4016528890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f425e80 0x55564f42e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f12309bf740 1 --2- 172.21.15.7:0/4016528890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f4258b0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.810+0000 7f12309bf740 1 -- 172.21.15.7:0/4016528890 >> 172.21.15.7:0/4016528890 conn(0x55564f41fba0 msgr2=0x55564f41ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f12309bf740 1 -- 172.21.15.7:0/4016528890 shutdown_connections 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f12309bf740 1 -- 172.21.15.7:0/4016528890 wait complete. 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f12309bf740 1 -- start start 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f12309bf740 1 Processor -- start 2024-04-21T08:45:49.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f12309bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f4254d0 0x55564f42e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f12309bf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f425e80 0x55564f42b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f12309bf740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f42ea00 0x55564f42bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f12309bf740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55564f431970 con 0x55564f42ea00 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f12309bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55564f343500 con 0x55564f4254d0 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f12309bf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55564f431af0 con 0x55564f425e80 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f12309bf740 10 client.0 _ll_register_callbacks cb 0x55564f311b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.811+0000 7f1227fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f42ea00 0x55564f42bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1227fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f42ea00 0x55564f42bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42730/0 (socket says 172.21.15.7:42730) 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1227fff640 1 -- 172.21.15.7:0/2076284493 learned_addr learned my addr 172.21.15.7:0/2076284493 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f12277fe640 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f4254d0 0x55564f42e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1226ffd640 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f425e80 0x55564f42b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1227fff640 1 -- 172.21.15.7:0/2076284493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f425e80 msgr2=0x55564f42b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1227fff640 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f425e80 0x55564f42b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1227fff640 1 -- 172.21.15.7:0/2076284493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f4254d0 msgr2=0x55564f42e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1227fff640 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f4254d0 0x55564f42e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1227fff640 1 -- 172.21.15.7:0/2076284493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55564f09c3f0 con 0x55564f42ea00 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1227fff640 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f42ea00 0x55564f42bb70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f122002c1c0 tx=0x7f1220041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2bc0551e126b46 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1206ffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1206ffd640 1 -- 172.21.15.7:0/2076284493 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220039030 con 0x55564f42ea00 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1206ffd640 1 -- 172.21.15.7:0/2076284493 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f1220004000 con 0x55564f42ea00 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f12309bf740 1 -- 172.21.15.7:0/2076284493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1220043030 con 0x55564f42ea00 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f1206ffd640 1 -- 172.21.15.7:0/2076284493 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1220032340 con 0x55564f42ea00 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f12309bf740 10 client.15996 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.812+0000 7f12309bf740 1 -- 172.21.15.7:0/2076284493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55564f1667d0 con 0x55564f42ea00 2024-04-21T08:45:49.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.813+0000 7f1206ffd640 1 -- 172.21.15.7:0/2076284493 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1220043030 con 0x55564f42ea00 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.813+0000 7f1206ffd640 1 -- 172.21.15.7:0/2076284493 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f12200347f0 con 0x55564f42ea00 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.813+0000 7f12309bf740 10 client.15996 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.813+0000 7f12309bf740 10 client.15996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.813+0000 7f12309bf740 20 client.15996 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.813+0000 7f12309bf740 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55564f43ea40 0x55564f440e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.813+0000 7f12309bf740 4 client.15996 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.813+0000 7f12309bf740 1 -- 172.21.15.7:0/2076284493 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55564f329930 con 0x55564f43ea40 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.813+0000 7f12277fe640 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55564f43ea40 0x55564f440e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.813+0000 7f12277fe640 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55564f43ea40 0x55564f440e30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc46918248df49a9 server_cookie=b9a56b974fe26be9 in_seq=0 out_seq=0 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.813+0000 7f1206ffd640 10 client.15996 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f1206ffd640 1 -- 172.21.15.7:0/2076284493 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55564f329930 con 0x55564f43ea40 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f1206ffd640 10 client.15996 handle_command_reply: tid=0 2024-04-21T08:45:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 1 client.15996 shutdown 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 1 -- 172.21.15.7:0/2076284493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f42ea00 msgr2=0x55564f42bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f42ea00 0x55564f42bb70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f122002c1c0 tx=0x7f1220041000 comp rx=0 tx=0).stop 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 1 -- 172.21.15.7:0/2076284493 shutdown_connections 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55564f43ea40 0x55564f440e30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f42ea00 0x55564f42bb70 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f425e80 0x55564f42b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f1206ffd640 0 client.15996 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 1 --2- 172.21.15.7:0/2076284493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f4254d0 0x55564f42e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 1 -- 172.21.15.7:0/2076284493 >> 172.21.15.7:0/2076284493 conn(0x55564f41fba0 msgr2=0x55564f4218f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 1 -- 172.21.15.7:0/2076284493 shutdown_connections 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 1 -- 172.21.15.7:0/2076284493 wait complete. 2024-04-21T08:45:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.815+0000 7f12309bf740 20 client.15996 trim_cache size 0 max 16384 2024-04-21T08:45:49.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.828+0000 7f12309bf740 1 Processor -- start 2024-04-21T08:45:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f12309bf740 1 -- start start 2024-04-21T08:45:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f12309bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f4254d0 0x55564f426bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f12309bf740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f425e80 0x55564f421680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f12309bf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f42ea00 0x55564f42dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f12309bf740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f12281ad450 con 0x55564f425e80 2024-04-21T08:45:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f12309bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f12281afcd0 con 0x55564f4254d0 2024-04-21T08:45:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f12309bf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f12281adac0 con 0x55564f42ea00 2024-04-21T08:45:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f1226ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f425e80 0x55564f421680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f1226ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f425e80 0x55564f421680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42742/0 (socket says 172.21.15.7:42742) 2024-04-21T08:45:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f1226ffd640 1 -- 172.21.15.7:0/4273831056 learned_addr learned my addr 172.21.15.7:0/4273831056 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f12277fe640 1 --2- 172.21.15.7:0/4273831056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f4254d0 0x55564f426bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f1227fff640 1 --2- 172.21.15.7:0/4273831056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f42ea00 0x55564f42dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f1226ffd640 1 -- 172.21.15.7:0/4273831056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f42ea00 msgr2=0x55564f42dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f1226ffd640 1 --2- 172.21.15.7:0/4273831056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f42ea00 0x55564f42dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f1226ffd640 1 -- 172.21.15.7:0/4273831056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f4254d0 msgr2=0x55564f426bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f1226ffd640 1 --2- 172.21.15.7:0/4273831056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f4254d0 0x55564f426bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f1226ffd640 1 -- 172.21.15.7:0/4273831056 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55564f429b80 con 0x55564f425e80 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f1226ffd640 1 --2- 172.21.15.7:0/4273831056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f425e80 0x55564f421680 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f121c009b60 tx=0x7f121c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a69e08c50903ef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f120d7fa640 1 -- 172.21.15.7:0/4273831056 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c009d10 con 0x55564f425e80 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.829+0000 7f120d7fa640 1 -- 172.21.15.7:0/4273831056 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f121c0035f0 con 0x55564f425e80 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.830+0000 7f120d7fa640 1 -- 172.21.15.7:0/4273831056 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c00f680 con 0x55564f425e80 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.830+0000 7f12309bf740 1 -- 172.21.15.7:0/4273831056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f425e80 msgr2=0x55564f421680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.830+0000 7f12309bf740 1 --2- 172.21.15.7:0/4273831056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f425e80 0x55564f421680 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f121c009b60 tx=0x7f121c005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.830+0000 7f12309bf740 1 -- 172.21.15.7:0/4273831056 shutdown_connections 2024-04-21T08:45:49.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.830+0000 7f12309bf740 1 --2- 172.21.15.7:0/4273831056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f42ea00 0x55564f42dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.830+0000 7f12309bf740 1 --2- 172.21.15.7:0/4273831056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f425e80 0x55564f421680 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.830+0000 7f12309bf740 1 --2- 172.21.15.7:0/4273831056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f4254d0 0x55564f426bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.830+0000 7f12309bf740 1 -- 172.21.15.7:0/4273831056 >> 172.21.15.7:0/4273831056 conn(0x55564f41fba0 msgr2=0x55564f420f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.830+0000 7f12309bf740 1 -- 172.21.15.7:0/4273831056 shutdown_connections 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.830+0000 7f12309bf740 1 -- 172.21.15.7:0/4273831056 wait complete. 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.830+0000 7f12309bf740 1 -- start start 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12309bf740 1 Processor -- start 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12309bf740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f49d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12309bf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f425e80 0x55564f49d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12309bf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f42ea00 0x55564f433030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12309bf740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f12281ae190 con 0x55564f4254d0 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12309bf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f12281af600 con 0x55564f425e80 2024-04-21T08:45:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12309bf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f12281ae860 con 0x55564f42ea00 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12309bf740 10 client.0 _ll_register_callbacks cb 0x7f1218005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12277fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f49d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f1226ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f425e80 0x55564f49d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12277fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f49d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42750/0 (socket says 172.21.15.7:42750) 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12277fe640 1 -- 172.21.15.7:0/1641451794 learned_addr learned my addr 172.21.15.7:0/1641451794 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f1227fff640 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f42ea00 0x55564f433030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12277fe640 1 -- 172.21.15.7:0/1641451794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f42ea00 msgr2=0x55564f433030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12277fe640 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f42ea00 0x55564f433030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12277fe640 1 -- 172.21.15.7:0/1641451794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f425e80 msgr2=0x55564f49d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12277fe640 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f425e80 0x55564f49d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.831+0000 7f12277fe640 1 -- 172.21.15.7:0/1641451794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55564f433830 con 0x55564f4254d0 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f12277fe640 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f49d040 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1218099100 tx=0x7f1218004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597060095ba20196 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f1206ffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f1206ffd640 1 -- 172.21.15.7:0/1641451794 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12180bbe60 con 0x55564f4254d0 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f1206ffd640 1 -- 172.21.15.7:0/1641451794 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f121809afa0 con 0x55564f4254d0 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f12309bf740 1 -- 172.21.15.7:0/1641451794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f121c0035f0 con 0x55564f4254d0 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f1206ffd640 1 -- 172.21.15.7:0/1641451794 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f12180b6200 con 0x55564f4254d0 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f12309bf740 10 client.16002 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f12309bf740 1 -- 172.21.15.7:0/1641451794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55564f4a58d0 con 0x55564f4254d0 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f1206ffd640 1 -- 172.21.15.7:0/1641451794 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f121c0035f0 con 0x55564f4254d0 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f1206ffd640 1 -- 172.21.15.7:0/1641451794 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1218099310 con 0x55564f4254d0 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f12309bf740 10 client.16002 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f12309bf740 10 client.16002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f12309bf740 20 client.16002 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f12309bf740 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55564f4a5fa0 0x55564f4a8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f12309bf740 4 client.16002 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:45:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f12309bf740 1 -- 172.21.15.7:0/1641451794 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55564f397c10 con 0x55564f4a5fa0 2024-04-21T08:45:49.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.832+0000 7f1226ffd640 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55564f4a5fa0 0x55564f4a8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:49.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.833+0000 7f1226ffd640 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55564f4a5fa0 0x55564f4a8390 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4bb4ddff700d8ea server_cookie=9f3a6b58ebd5cd3 in_seq=0 out_seq=0 2024-04-21T08:45:49.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.833+0000 7f1206ffd640 10 client.16002 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f1206ffd640 1 -- 172.21.15.7:0/1641451794 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55564f397c10 con 0x55564f4a5fa0 2024-04-21T08:45:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f1206ffd640 10 client.16002 handle_command_reply: tid=0 2024-04-21T08:45:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 1 client.16002 shutdown 2024-04-21T08:45:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 1 -- 172.21.15.7:0/1641451794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 msgr2=0x55564f49d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f49d040 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1218099100 tx=0x7f1218004a90 comp rx=0 tx=0).stop 2024-04-21T08:45:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 1 -- 172.21.15.7:0/1641451794 shutdown_connections 2024-04-21T08:45:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55564f4a5fa0 0x55564f4a8390 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55564f42ea00 0x55564f433030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55564f425e80 0x55564f49d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f1206ffd640 0 client.16002 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:49.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 1 --2- 172.21.15.7:0/1641451794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55564f4254d0 0x55564f49d040 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 1 -- 172.21.15.7:0/1641451794 >> 172.21.15.7:0/1641451794 conn(0x55564f41fba0 msgr2=0x55564f41e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 1 -- 172.21.15.7:0/1641451794 shutdown_connections 2024-04-21T08:45:49.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 1 -- 172.21.15.7:0/1641451794 wait complete. 2024-04-21T08:45:49.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.834+0000 7f12309bf740 20 client.16002 trim_cache size 0 max 16384 2024-04-21T08:45:49.841 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:45:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 -- 172.21.15.7:0/188720261 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f12100a3e40 msgr2=0x7f12100a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f12100a3e40 0x7f12100a6300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1220047000 tx=0x7f12200325e0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 -- 172.21.15.7:0/188720261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12281031c0 msgr2=0x7f1228198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12281031c0 0x7f1228198730 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f121800e9f0 tx=0x7f121800edf0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 -- 172.21.15.7:0/188720261 shutdown_connections 2024-04-21T08:45:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f122810c500 0x7f122819cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f12100a3e40 0x7f12100a6300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1228103ae0 0x7f1228198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 --2- 172.21.15.7:0/188720261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12281031c0 0x7f1228198730 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 -- 172.21.15.7:0/188720261 >> 172.21.15.7:0/188720261 conn(0x7f1228078000 msgr2=0x7f12280fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 -- 172.21.15.7:0/188720261 shutdown_connections 2024-04-21T08:45:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:49.835+0000 7f122ddce640 1 -- 172.21.15.7:0/188720261 wait complete. 2024-04-21T08:45:50.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:49 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1234607914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:50.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:49 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1234607914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:50.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:49 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1234607914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:50.848 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:45:50.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:50.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:45:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.956+0000 7f7eef766640 1 -- 172.21.15.7:0/1693536044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ee8102180 msgr2=0x7f7ee8102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.956+0000 7f7eef766640 1 --2- 172.21.15.7:0/1693536044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ee8102180 0x7f7ee8102560 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009900 tx=0x7f7ee402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.956+0000 7f7eef766640 1 -- 172.21.15.7:0/1693536044 shutdown_connections 2024-04-21T08:45:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.956+0000 7f7eef766640 1 --2- 172.21.15.7:0/1693536044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee8107250 0x7f7ee810ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.956+0000 7f7eef766640 1 --2- 172.21.15.7:0/1693536044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ee8102b30 0x7f7ee8106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.956+0000 7f7eef766640 1 --2- 172.21.15.7:0/1693536044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ee8102180 0x7f7ee8102560 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.956+0000 7f7eef766640 1 -- 172.21.15.7:0/1693536044 >> 172.21.15.7:0/1693536044 conn(0x7f7ee80fda00 msgr2=0x7f7ee80ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.956+0000 7f7eef766640 1 -- 172.21.15.7:0/1693536044 shutdown_connections 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.956+0000 7f7eef766640 1 -- 172.21.15.7:0/1693536044 wait complete. 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eef766640 1 Processor -- start 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eef766640 1 -- start start 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eef766640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ee8102180 0x7f7ee819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eef766640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee8107250 0x7f7ee819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eef766640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ee8198590 0x7f7ee8198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eef766640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81121b0 con 0x7f7ee8198590 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eef766640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee8112030 con 0x7f7ee8107250 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eef766640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee8112330 con 0x7f7ee8102180 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eedd03640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ee8198590 0x7f7ee8198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eecd01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee8107250 0x7f7ee819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eedd03640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ee8198590 0x7f7ee8198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42762/0 (socket says 172.21.15.7:42762) 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eedd03640 1 -- 172.21.15.7:0/4098914936 learned_addr learned my addr 172.21.15.7:0/4098914936 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eed502640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ee8102180 0x7f7ee819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eedd03640 1 -- 172.21.15.7:0/4098914936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ee8102180 msgr2=0x7f7ee819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eedd03640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ee8102180 0x7f7ee819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.957+0000 7f7eedd03640 1 -- 172.21.15.7:0/4098914936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee8107250 msgr2=0x7f7ee819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.958+0000 7f7eedd03640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee8107250 0x7f7ee819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.958+0000 7f7eedd03640 1 -- 172.21.15.7:0/4098914936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee81990b0 con 0x7f7ee8198590 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.958+0000 7f7eecd01640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee8107250 0x7f7ee819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.958+0000 7f7eedd03640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ee8198590 0x7f7ee8198a40 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7ee000eb10 tx=0x7f7ee000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.958+0000 7f7ede7fc640 1 -- 172.21.15.7:0/4098914936 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0004b20 con 0x7f7ee8198590 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.958+0000 7f7eef766640 1 -- 172.21.15.7:0/4098914936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee81993a0 con 0x7f7ee8198590 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.958+0000 7f7ede7fc640 1 -- 172.21.15.7:0/4098914936 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ee0004cc0 con 0x7f7ee8198590 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.958+0000 7f7ede7fc640 1 -- 172.21.15.7:0/4098914936 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0010670 con 0x7f7ee8198590 2024-04-21T08:45:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.958+0000 7f7eef766640 1 -- 172.21.15.7:0/4098914936 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee81a55f0 con 0x7f7ee8198590 2024-04-21T08:45:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.960+0000 7f7ede7fc640 1 -- 172.21.15.7:0/4098914936 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f7ee0005020 con 0x7f7ee8198590 2024-04-21T08:45:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7ef2430740 1 Processor -- start 2024-04-21T08:45:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7ef2430740 1 -- start start 2024-04-21T08:45:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162b7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 0x5623162ba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7ef2430740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562316212f00 con 0x5623162af290 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7ef2430740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5623161b0590 con 0x5623162ae8e0 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7ef2430740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56231620eec0 con 0x5623162b7e10 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eecd01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eecd01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162aecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:37148/0 (socket says 172.21.15.7:37148) 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eed502640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162b7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eecd01640 1 -- 172.21.15.7:0/2954277389 learned_addr learned my addr 172.21.15.7:0/2954277389 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eecd01640 1 --2- 172.21.15.7:0/2954277389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 0x5623162ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7ede7fc640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ed40a3e40 0x7f7ed40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eecd01640 1 -- 172.21.15.7:0/2954277389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 msgr2=0x5623162ba260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eecd01640 1 --2- 172.21.15.7:0/2954277389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 0x5623162ba260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eecd01640 1 -- 172.21.15.7:0/2954277389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 msgr2=0x5623162b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eecd01640 1 --2- 172.21.15.7:0/2954277389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162b7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eecd01640 1 -- 172.21.15.7:0/2954277389 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562315fed7e0 con 0x5623162ae8e0 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eedd03640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ed40a3e40 0x7f7ed40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7ede7fc640 1 -- 172.21.15.7:0/4098914936 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f7ee00d13a0 con 0x7f7ee8198590 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.961+0000 7f7eecd01640 1 --2- 172.21.15.7:0/2954277389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162aecc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8012b20 tx=0x7f7ed8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52391f817fa3badf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.962+0000 7f7ed37fe640 1 -- 172.21.15.7:0/2954277389 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8004030 con 0x5623162ae8e0 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.962+0000 7f7ed37fe640 1 -- 172.21.15.7:0/2954277389 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ed80054d0 con 0x5623162ae8e0 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.962+0000 7f7ed37fe640 1 -- 172.21.15.7:0/2954277389 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8010c50 con 0x5623162ae8e0 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.962+0000 7f7ef2430740 1 -- 172.21.15.7:0/2954277389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 msgr2=0x5623162aecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.962+0000 7f7ef2430740 1 --2- 172.21.15.7:0/2954277389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162aecc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8012b20 tx=0x7f7ed8004730 comp rx=0 tx=0).stop 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.962+0000 7f7eedd03640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ed40a3e40 0x7f7ed40a6300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7ee00ce5e0 tx=0x7f7ee00bd040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.962+0000 7f7ef2430740 1 -- 172.21.15.7:0/2954277389 shutdown_connections 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.962+0000 7f7ef2430740 1 --2- 172.21.15.7:0/2954277389 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 0x5623162ba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.962+0000 7f7ef2430740 1 --2- 172.21.15.7:0/2954277389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162b7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.962+0000 7f7ef2430740 1 --2- 172.21.15.7:0/2954277389 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162aecc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.962+0000 7f7ef2430740 1 -- 172.21.15.7:0/2954277389 >> 172.21.15.7:0/2954277389 conn(0x5623162a8fb0 msgr2=0x5623162a9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7ef2430740 1 -- 172.21.15.7:0/2954277389 shutdown_connections 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7ef2430740 1 -- 172.21.15.7:0/2954277389 wait complete. 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7ef2430740 1 -- start start 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7ef2430740 1 Processor -- start 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 0x5623162b5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7ef2430740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5623162bad80 con 0x5623162af290 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7ef2430740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5623161cc580 con 0x5623162ae8e0 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7ef2430740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5623162baf00 con 0x5623162b7e10 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7ef2430740 10 client.0 _ll_register_callbacks cb 0x5623161d2650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7eed502640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7eed502640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162b53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42782/0 (socket says 172.21.15.7:42782) 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.963+0000 7f7eed502640 1 -- 172.21.15.7:0/170959581 learned_addr learned my addr 172.21.15.7:0/170959581 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7eecd01640 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7eecd01640 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 0x5623162b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7eed502640 1 -- 172.21.15.7:0/170959581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 msgr2=0x5623162b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7eed502640 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 0x5623162b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7eed502640 1 -- 172.21.15.7:0/170959581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 msgr2=0x5623162b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7eed502640 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7eed502640 1 -- 172.21.15.7:0/170959581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56231604df70 con 0x5623162af290 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7eecd01640 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7eed502640 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162b53d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009900 tx=0x7f7ee402c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350029b667652beb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7ed0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/170959581 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee402c610 con 0x5623162af290 2024-04-21T08:45:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7ef2430740 1 -- 172.21.15.7:0/170959581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ed80054d0 con 0x5623162af290 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/170959581 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f7ee4005030 con 0x5623162af290 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/170959581 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7ee4005230 con 0x5623162af290 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7ef2430740 10 client.16014 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.964+0000 7f7ef2430740 1 -- 172.21.15.7:0/170959581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562315ee5170 con 0x5623162af290 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.965+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/170959581 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ed80054d0 con 0x5623162af290 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.965+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/170959581 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7ee4003710 con 0x5623162af290 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.965+0000 7f7ef2430740 10 client.16014 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.965+0000 7f7ef2430740 10 client.16014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.965+0000 7f7ef2430740 20 client.16014 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.965+0000 7f7ef2430740 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5623162c8390 0x5623162ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.965+0000 7f7ef2430740 4 client.16014 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.965+0000 7f7ef2430740 1 -- 172.21.15.7:0/170959581 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562316220920 con 0x5623162c8390 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.965+0000 7f7eecd01640 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5623162c8390 0x5623162ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.965+0000 7f7eecd01640 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5623162c8390 0x5623162ca7c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c3be5c1efe360c0 server_cookie=57299d147d6a456e in_seq=0 out_seq=0 2024-04-21T08:45:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.965+0000 7f7ed0ff9640 10 client.16014 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/170959581 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562316220920 con 0x5623162c8390 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ed0ff9640 10 client.16014 handle_command_reply: tid=0 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 1 client.16014 shutdown 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 1 -- 172.21.15.7:0/170959581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 msgr2=0x5623162b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162b53d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009900 tx=0x7f7ee402c9f0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 1 -- 172.21.15.7:0/170959581 shutdown_connections 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5623162c8390 0x5623162ca7c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 0x5623162b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ed0ff9640 0 client.16014 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162b53d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 1 --2- 172.21.15.7:0/170959581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 1 -- 172.21.15.7:0/170959581 >> 172.21.15.7:0/170959581 conn(0x5623162a8fb0 msgr2=0x5623162aad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 1 -- 172.21.15.7:0/170959581 shutdown_connections 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 1 -- 172.21.15.7:0/170959581 wait complete. 2024-04-21T08:45:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.966+0000 7f7ef2430740 20 client.16014 trim_cache size 0 max 16384 2024-04-21T08:45:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7ef2430740 1 Processor -- start 2024-04-21T08:45:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7ef2430740 1 -- start start 2024-04-21T08:45:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162b75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 0x5623162b5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7ef2430740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81ad4d0 con 0x5623162af290 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7ef2430740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81afd50 con 0x5623162ae8e0 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7ef2430740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81adb40 con 0x5623162b7e10 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7eed502640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7eecd01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162b75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7eecd01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162b75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:37174/0 (socket says 172.21.15.7:37174) 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7eecd01640 1 -- 172.21.15.7:0/1611044277 learned_addr learned my addr 172.21.15.7:0/1611044277 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7eed502640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162c3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42786/0 (socket says 172.21.15.7:42786) 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7eecd01640 1 -- 172.21.15.7:0/1611044277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 msgr2=0x5623162b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7eecd01640 1 --2- 172.21.15.7:0/1611044277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 0x5623162b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7eecd01640 1 -- 172.21.15.7:0/1611044277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 msgr2=0x5623162c3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7eecd01640 1 --2- 172.21.15.7:0/1611044277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162c3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.984+0000 7f7eecd01640 1 -- 172.21.15.7:0/1611044277 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623162b0a50 con 0x5623162ae8e0 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7eecd01640 1 --2- 172.21.15.7:0/1611044277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162b75c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800ba60 tx=0x7f7ed8017040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5f83770fe219119 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7ed2ffd640 1 -- 172.21.15.7:0/1611044277 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8010c50 con 0x5623162ae8e0 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7ed2ffd640 1 -- 172.21.15.7:0/1611044277 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ed8032070 con 0x5623162ae8e0 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7ed2ffd640 1 -- 172.21.15.7:0/1611044277 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed801e790 con 0x5623162ae8e0 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7ef2430740 1 -- 172.21.15.7:0/1611044277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 msgr2=0x5623162b75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7ef2430740 1 --2- 172.21.15.7:0/1611044277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162b75c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800ba60 tx=0x7f7ed8017040 comp rx=0 tx=0).stop 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7ef2430740 1 -- 172.21.15.7:0/1611044277 shutdown_connections 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7ef2430740 1 --2- 172.21.15.7:0/1611044277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162b7e10 0x5623162b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7ef2430740 1 --2- 172.21.15.7:0/1611044277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162c3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7ef2430740 1 --2- 172.21.15.7:0/1611044277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162ae8e0 0x5623162b75c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7ef2430740 1 -- 172.21.15.7:0/1611044277 >> 172.21.15.7:0/1611044277 conn(0x5623162a8fb0 msgr2=0x5623162b4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.985+0000 7f7ef2430740 1 -- 172.21.15.7:0/1611044277 shutdown_connections 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7ef2430740 1 -- 172.21.15.7:0/1611044277 wait complete. 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7ef2430740 1 -- start start 2024-04-21T08:45:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7ef2430740 1 Processor -- start 2024-04-21T08:45:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162ae8e0 0x5623162bbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162bc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7ef2430740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162b7e10 0x5623162bc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7ef2430740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81ae210 con 0x5623162af290 2024-04-21T08:45:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7ef2430740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81af680 con 0x5623162b7e10 2024-04-21T08:45:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7ef2430740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81ae8e0 con 0x5623162ae8e0 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7ef2430740 10 client.0 _ll_register_callbacks cb 0x7f7ee4009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7eed502640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162bc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7eed502640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162bc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42798/0 (socket says 172.21.15.7:42798) 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7eed502640 1 -- 172.21.15.7:0/3085925712 learned_addr learned my addr 172.21.15.7:0/3085925712 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.986+0000 7f7eecd01640 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162b7e10 0x5623162bc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7eecd01640 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162ae8e0 0x5623162bbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7eed502640 1 -- 172.21.15.7:0/3085925712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162ae8e0 msgr2=0x5623162bbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7eed502640 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162ae8e0 0x5623162bbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7eed502640 1 -- 172.21.15.7:0/3085925712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162b7e10 msgr2=0x5623162bc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7eed502640 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162b7e10 0x5623162bc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7eed502640 1 -- 172.21.15.7:0/3085925712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623163361e0 con 0x5623162af290 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7eecd01640 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162b7e10 0x5623162bc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7eed502640 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162bc180 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4003cf0 tx=0x7f7ee402caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e2e968af0b5041 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7ed0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/3085925712 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee405a020 con 0x5623162af290 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/3085925712 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f7ee40037f0 con 0x5623162af290 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7ef2430740 1 -- 172.21.15.7:0/3085925712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ed8032070 con 0x5623162af290 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/3085925712 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7ee4065050 con 0x5623162af290 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7ef2430740 10 client.16020 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.987+0000 7f7ef2430740 1 -- 172.21.15.7:0/3085925712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562316337550 con 0x5623162af290 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.988+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/3085925712 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ed8032070 con 0x5623162af290 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.988+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/3085925712 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7ee40046c0 con 0x5623162af290 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.988+0000 7f7ef2430740 10 client.16020 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.988+0000 7f7ef2430740 10 client.16020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.988+0000 7f7ef2430740 20 client.16020 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.988+0000 7f7ef2430740 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562316337d50 0x56231633a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.988+0000 7f7ef2430740 4 client.16020 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.988+0000 7f7ef2430740 1 -- 172.21.15.7:0/3085925712 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56231621a790 con 0x562316337d50 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.988+0000 7f7eecd01640 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562316337d50 0x56231633a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.988+0000 7f7eecd01640 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562316337d50 0x56231633a170 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5502b735f68af568 server_cookie=14c30f15511ab3aa in_seq=0 out_seq=0 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.988+0000 7f7ed0ff9640 10 client.16020 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ed0ff9640 1 -- 172.21.15.7:0/3085925712 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56231621a790 con 0x562316337d50 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ed0ff9640 10 client.16020 handle_command_reply: tid=0 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ef2430740 1 client.16020 shutdown 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ef2430740 1 -- 172.21.15.7:0/3085925712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 msgr2=0x5623162bc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ef2430740 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162bc180 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4003cf0 tx=0x7f7ee402caf0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ef2430740 1 -- 172.21.15.7:0/3085925712 shutdown_connections 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ef2430740 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562316337d50 0x56231633a170 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ef2430740 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5623162b7e10 0x5623162bc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ef2430740 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5623162af290 0x5623162bc180 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ef2430740 1 --2- 172.21.15.7:0/3085925712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5623162ae8e0 0x5623162bbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ef2430740 1 -- 172.21.15.7:0/3085925712 >> 172.21.15.7:0/3085925712 conn(0x5623162a8fb0 msgr2=0x5623162a8010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.989+0000 7f7ed0ff9640 0 client.16020 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7ef2430740 1 -- 172.21.15.7:0/3085925712 shutdown_connections 2024-04-21T08:45:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7ef2430740 1 -- 172.21.15.7:0/3085925712 wait complete. 2024-04-21T08:45:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7ef2430740 20 client.16020 trim_cache size 0 max 16384 2024-04-21T08:45:50.995 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:45:50.995 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:45:50.995 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "f310a520-149c-4b3d-a4b6-cd6daac1fcfc", 2024-04-21T08:45:50.995 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:45:50.996 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:45:50.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 -- 172.21.15.7:0/4098914936 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ed40a3e40 msgr2=0x7f7ed40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:50.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ed40a3e40 0x7f7ed40a6300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7ee00ce5e0 tx=0x7f7ee00bd040 comp rx=0 tx=0).stop 2024-04-21T08:45:50.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 -- 172.21.15.7:0/4098914936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ee8198590 msgr2=0x7f7ee8198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:50.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ee8198590 0x7f7ee8198a40 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7ee000eb10 tx=0x7f7ee000c2d0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 -- 172.21.15.7:0/4098914936 shutdown_connections 2024-04-21T08:45:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ee8198590 0x7f7ee8198a40 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee8107250 0x7f7ee819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ed40a3e40 0x7f7ed40a6300 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 --2- 172.21.15.7:0/4098914936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ee8102180 0x7f7ee819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 -- 172.21.15.7:0/4098914936 >> 172.21.15.7:0/4098914936 conn(0x7f7ee80fda00 msgr2=0x7f7ee80fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 -- 172.21.15.7:0/4098914936 shutdown_connections 2024-04-21T08:45:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:50.990+0000 7f7eef766640 1 -- 172.21.15.7:0/4098914936 wait complete. 2024-04-21T08:45:51.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:45:51.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:45:51.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:50 smithi077 ceph-mon[31615]: pgmap v37: 129 pgs: 129 active+clean; 139 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 12 MiB/s wr, 33 op/s 2024-04-21T08:45:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.111+0000 7fef08e5c640 1 -- 172.21.15.7:0/2243468352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef04101240 msgr2=0x7fef0410d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.111+0000 7fef08e5c640 1 --2- 172.21.15.7:0/2243468352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef04101240 0x7fef0410d530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feefc009ca0 tx=0x7feefc02c180 comp rx=0 tx=0).stop 2024-04-21T08:45:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.112+0000 7fef08e5c640 1 -- 172.21.15.7:0/2243468352 shutdown_connections 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.112+0000 7fef08e5c640 1 --2- 172.21.15.7:0/2243468352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef04101240 0x7fef0410d530 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.112+0000 7fef08e5c640 1 --2- 172.21.15.7:0/2243468352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04100880 0x7fef04100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.112+0000 7fef08e5c640 1 --2- 172.21.15.7:0/2243468352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef04102510 0x7fef041028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.112+0000 7fef08e5c640 1 -- 172.21.15.7:0/2243468352 >> 172.21.15.7:0/2243468352 conn(0x7fef04076160 msgr2=0x7fef04076570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.112+0000 7fef08e5c640 1 -- 172.21.15.7:0/2243468352 shutdown_connections 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.112+0000 7fef08e5c640 1 -- 172.21.15.7:0/2243468352 wait complete. 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.112+0000 7fef08e5c640 1 Processor -- start 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.112+0000 7fef08e5c640 1 -- start start 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.112+0000 7fef08e5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04100880 0x7fef04198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef08e5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef04101240 0x7fef04198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef08e5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef04102510 0x7fef0419d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef08e5c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fef041122e0 con 0x7fef04100880 2024-04-21T08:45:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef08e5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fef04112160 con 0x7fef04101240 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef08e5c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fef04112460 con 0x7fef04102510 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef0259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04100880 0x7fef04198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef0259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04100880 0x7fef04198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42822/0 (socket says 172.21.15.7:42822) 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef0259c640 1 -- 172.21.15.7:0/2939573783 learned_addr learned my addr 172.21.15.7:0/2939573783 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef01d9b640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef04101240 0x7fef04198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef02d9d640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef04102510 0x7fef0419d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef0259c640 1 -- 172.21.15.7:0/2939573783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef04102510 msgr2=0x7fef0419d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef0259c640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef04102510 0x7fef0419d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef0259c640 1 -- 172.21.15.7:0/2939573783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef04101240 msgr2=0x7fef04198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef0259c640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef04101240 0x7fef04198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef0259c640 1 -- 172.21.15.7:0/2939573783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef0419d750 con 0x7fef04100880 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef0259c640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04100880 0x7fef04198760 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7feef400e9f0 tx=0x7feef400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7feef37fe640 1 -- 172.21.15.7:0/2939573783 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef400cd80 con 0x7fef04100880 2024-04-21T08:45:51.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef08e5c640 1 -- 172.21.15.7:0/2939573783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef0419da40 con 0x7fef04100880 2024-04-21T08:45:51.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7fef08e5c640 1 -- 172.21.15.7:0/2939573783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef041a52d0 con 0x7fef04100880 2024-04-21T08:45:51.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7feef37fe640 1 -- 172.21.15.7:0/2939573783 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef40218f0 con 0x7fef04100880 2024-04-21T08:45:51.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.113+0000 7feef37fe640 1 -- 172.21.15.7:0/2939573783 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef40108c0 con 0x7fef04100880 2024-04-21T08:45:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.115+0000 7feef37fe640 1 -- 172.21.15.7:0/2939573783 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7feef4021440 con 0x7fef04100880 2024-04-21T08:45:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.116+0000 7fef0ba52740 1 Processor -- start 2024-04-21T08:45:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.116+0000 7fef0ba52740 1 -- start start 2024-04-21T08:45:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.116+0000 7feef37fe640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7feeec0a3e40 0x7feeec0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.116+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27580 0x5626e6e27960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.116+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27ea0 0x5626e6e30540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.116+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e32e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.116+0000 7fef0ba52740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5626e6d8c1e0 con 0x5626e6e27580 2024-04-21T08:45:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.116+0000 7fef0ba52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5626e6d29390 con 0x5626e6e30a80 2024-04-21T08:45:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.116+0000 7fef0ba52740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5626e6d881a0 con 0x5626e6e27ea0 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.116+0000 7fef02d9d640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7feeec0a3e40 0x7feeec0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.116+0000 7fef0259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27580 0x5626e6e27960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7fef0259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27580 0x5626e6e27960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42834/0 (socket says 172.21.15.7:42834) 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7fef0259c640 1 -- 172.21.15.7:0/3381526511 learned_addr learned my addr 172.21.15.7:0/3381526511 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7fef02d9d640 1 --2- 172.21.15.7:0/3381526511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e32e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7feef37fe640 1 -- 172.21.15.7:0/2939573783 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7feef40d1210 con 0x7fef04100880 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7fef01d9b640 1 --2- 172.21.15.7:0/3381526511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27ea0 0x5626e6e30540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7fef0259c640 1 -- 172.21.15.7:0/3381526511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27ea0 msgr2=0x5626e6e30540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7fef0259c640 1 --2- 172.21.15.7:0/3381526511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27ea0 0x5626e6e30540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7fef0259c640 1 -- 172.21.15.7:0/3381526511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 msgr2=0x5626e6e32e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7fef0259c640 1 --2- 172.21.15.7:0/3381526511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e32e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7fef0259c640 1 -- 172.21.15.7:0/3381526511 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e6b687d0 con 0x5626e6e27580 2024-04-21T08:45:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7fef02d9d640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7feeec0a3e40 0x7feeec0a6300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7feefc005e30 tx=0x7feefc0099d0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7fef0259c640 1 --2- 172.21.15.7:0/3381526511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27580 0x5626e6e27960 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7feef40ce330 tx=0x7feef4013b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf0f9bc6d756e64 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.117+0000 7feef0ff9640 1 -- 172.21.15.7:0/3381526511 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef40a2650 con 0x5626e6e27580 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.118+0000 7feef0ff9640 1 -- 172.21.15.7:0/3381526511 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef4021730 con 0x5626e6e27580 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.118+0000 7feef0ff9640 1 -- 172.21.15.7:0/3381526511 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef40a2050 con 0x5626e6e27580 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.118+0000 7fef0ba52740 1 -- 172.21.15.7:0/3381526511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27580 msgr2=0x5626e6e27960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.118+0000 7fef0ba52740 1 --2- 172.21.15.7:0/3381526511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27580 0x5626e6e27960 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7feef40ce330 tx=0x7feef4013b60 comp rx=0 tx=0).stop 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.118+0000 7fef0ba52740 1 -- 172.21.15.7:0/3381526511 shutdown_connections 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.118+0000 7fef0ba52740 1 --2- 172.21.15.7:0/3381526511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e32e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.118+0000 7fef0ba52740 1 --2- 172.21.15.7:0/3381526511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27ea0 0x5626e6e30540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.118+0000 7fef0ba52740 1 --2- 172.21.15.7:0/3381526511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27580 0x5626e6e27960 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.118+0000 7fef0ba52740 1 -- 172.21.15.7:0/3381526511 >> 172.21.15.7:0/3381526511 conn(0x5626e6e21cb0 msgr2=0x5626e6e22090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.118+0000 7fef0ba52740 1 -- 172.21.15.7:0/3381526511 shutdown_connections 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.118+0000 7fef0ba52740 1 -- 172.21.15.7:0/3381526511 wait complete. 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef0ba52740 1 -- start start 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef0ba52740 1 Processor -- start 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 0x5626e6e2da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e2dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e2e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef0ba52740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5626e6e337b0 con 0x5626e6e27ea0 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef0ba52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5626e6d452f0 con 0x5626e6e30a80 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef0ba52740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5626e6e33930 con 0x5626e6e27580 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef0ba52740 10 client.0 _ll_register_callbacks cb 0x5626e6d52580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef01d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e2dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef0259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 0x5626e6e2da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef02d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e2e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef01d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e2dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42850/0 (socket says 172.21.15.7:42850) 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.119+0000 7fef01d9b640 1 -- 172.21.15.7:0/642416196 learned_addr learned my addr 172.21.15.7:0/642416196 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7fef0259c640 1 -- 172.21.15.7:0/642416196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 msgr2=0x5626e6e2e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7fef0259c640 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e2e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7fef0259c640 1 -- 172.21.15.7:0/642416196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 msgr2=0x5626e6e2dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7fef0259c640 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e2dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7fef0259c640 1 -- 172.21.15.7:0/642416196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626e6a9e3f0 con 0x5626e6e27580 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7fef02d9d640 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e2e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7fef0259c640 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 0x5626e6e2da90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7feef40ce330 tx=0x7feef40a2650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14863b959f4b106b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7feee9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7feee9ffb640 1 -- 172.21.15.7:0/642416196 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef40a54f0 con 0x5626e6e27580 2024-04-21T08:45:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7fef0ba52740 1 -- 172.21.15.7:0/642416196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5626e6d99c00 con 0x5626e6e27580 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7feee9ffb640 1 -- 172.21.15.7:0/642416196 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7feef4013bd0 con 0x5626e6e27580 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7feee9ffb640 1 -- 172.21.15.7:0/642416196 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7feef40a0ec0 con 0x5626e6e27580 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7fef0ba52740 10 client.25636 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.120+0000 7fef0ba52740 1 -- 172.21.15.7:0/642416196 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5626e6a5e0f0 con 0x5626e6e27580 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.121+0000 7feee9ffb640 1 -- 172.21.15.7:0/642416196 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5626e6d99c00 con 0x5626e6e27580 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.121+0000 7feee9ffb640 1 -- 172.21.15.7:0/642416196 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7feef40b8070 con 0x5626e6e27580 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.121+0000 7fef0ba52740 10 client.25636 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.121+0000 7fef0ba52740 10 client.25636 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.121+0000 7fef0ba52740 20 client.25636 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.121+0000 7fef0ba52740 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5626e6e408b0 0x5626e6e42ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.121+0000 7fef0ba52740 4 client.25636 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.121+0000 7fef0ba52740 1 -- 172.21.15.7:0/642416196 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626e6d93a70 con 0x5626e6e408b0 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.121+0000 7fef01d9b640 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5626e6e408b0 0x5626e6e42ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.121+0000 7fef01d9b640 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5626e6e408b0 0x5626e6e42ca0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36baa4b1d43b6ce4 server_cookie=875afa0aae41fceb in_seq=0 out_seq=0 2024-04-21T08:45:51.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.121+0000 7feee9ffb640 10 client.25636 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.122+0000 7feee9ffb640 1 -- 172.21.15.7:0/642416196 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5626e6d93a70 con 0x5626e6e408b0 2024-04-21T08:45:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.122+0000 7feee9ffb640 10 client.25636 handle_command_reply: tid=0 2024-04-21T08:45:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.122+0000 7fef0ba52740 1 client.25636 shutdown 2024-04-21T08:45:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.122+0000 7fef0ba52740 1 -- 172.21.15.7:0/642416196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 msgr2=0x5626e6e2da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.123+0000 7fef0ba52740 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 0x5626e6e2da90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7feef40ce330 tx=0x7feef40a2650 comp rx=0 tx=0).stop 2024-04-21T08:45:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.123+0000 7fef0ba52740 1 -- 172.21.15.7:0/642416196 shutdown_connections 2024-04-21T08:45:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.123+0000 7fef0ba52740 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5626e6e408b0 0x5626e6e42ca0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.123+0000 7fef0ba52740 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e2e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.123+0000 7fef0ba52740 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e2dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.123+0000 7fef0ba52740 1 --2- 172.21.15.7:0/642416196 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 0x5626e6e2da90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.123+0000 7feee9ffb640 0 client.25636 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:51.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.123+0000 7fef0ba52740 1 -- 172.21.15.7:0/642416196 >> 172.21.15.7:0/642416196 conn(0x5626e6e21cb0 msgr2=0x5626e6e23a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:51.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.123+0000 7fef0ba52740 1 -- 172.21.15.7:0/642416196 shutdown_connections 2024-04-21T08:45:51.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.123+0000 7fef0ba52740 1 -- 172.21.15.7:0/642416196 wait complete. 2024-04-21T08:45:51.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.123+0000 7fef0ba52740 20 client.25636 trim_cache size 0 max 16384 2024-04-21T08:45:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.142+0000 7fef0ba52740 1 Processor -- start 2024-04-21T08:45:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.142+0000 7fef0ba52740 1 -- start start 2024-04-21T08:45:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.142+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 0x5626e6e301c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.142+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e3c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.142+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e2cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.142+0000 7fef0ba52740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7feee400e050 con 0x5626e6e27ea0 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.142+0000 7fef0ba52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feee40108d0 con 0x5626e6e30a80 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.142+0000 7fef0ba52740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feee400e6c0 con 0x5626e6e27580 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef01d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e3c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef0259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 0x5626e6e301c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef01d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e3c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42854/0 (socket says 172.21.15.7:42854) 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef0259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 0x5626e6e301c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:41816/0 (socket says 172.21.15.7:41816) 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef01d9b640 1 -- 172.21.15.7:0/2589509657 learned_addr learned my addr 172.21.15.7:0/2589509657 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef02d9d640 1 --2- 172.21.15.7:0/2589509657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e2cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef01d9b640 1 -- 172.21.15.7:0/2589509657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 msgr2=0x5626e6e301c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef01d9b640 1 --2- 172.21.15.7:0/2589509657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 0x5626e6e301c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef01d9b640 1 -- 172.21.15.7:0/2589509657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 msgr2=0x5626e6e2cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef01d9b640 1 --2- 172.21.15.7:0/2589509657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e2cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef01d9b640 1 -- 172.21.15.7:0/2589509657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e6e21ae0 con 0x5626e6e27ea0 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef0259c640 1 --2- 172.21.15.7:0/2589509657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 0x5626e6e301c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7fef01d9b640 1 --2- 172.21.15.7:0/2589509657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e3c700 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7feef8000c00 tx=0x7feef8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b840096d87588c56 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7feeebfff640 1 -- 172.21.15.7:0/2589509657 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef800fad0 con 0x5626e6e27ea0 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7feeebfff640 1 -- 172.21.15.7:0/2589509657 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef80054d0 con 0x5626e6e27ea0 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.143+0000 7feeebfff640 1 -- 172.21.15.7:0/2589509657 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8004c00 con 0x5626e6e27ea0 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.144+0000 7fef0ba52740 1 -- 172.21.15.7:0/2589509657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 msgr2=0x5626e6e3c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.144+0000 7fef0ba52740 1 --2- 172.21.15.7:0/2589509657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e3c700 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7feef8000c00 tx=0x7feef8005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.144+0000 7fef0ba52740 1 -- 172.21.15.7:0/2589509657 shutdown_connections 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.144+0000 7fef0ba52740 1 --2- 172.21.15.7:0/2589509657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e30a80 0x5626e6e2cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.144+0000 7fef0ba52740 1 --2- 172.21.15.7:0/2589509657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e3c700 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.144+0000 7fef0ba52740 1 --2- 172.21.15.7:0/2589509657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e27580 0x5626e6e301c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.144+0000 7fef0ba52740 1 -- 172.21.15.7:0/2589509657 >> 172.21.15.7:0/2589509657 conn(0x5626e6e21cb0 msgr2=0x5626e6e204e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.144+0000 7fef0ba52740 1 -- 172.21.15.7:0/2589509657 shutdown_connections 2024-04-21T08:45:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.144+0000 7fef0ba52740 1 -- 172.21.15.7:0/2589509657 wait complete. 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef0ba52740 1 -- start start 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef0ba52740 1 Processor -- start 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e27580 0x5626e6e39070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e2a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef0ba52740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e30a80 0x5626e6e2aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef0ba52740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7feee400ed90 con 0x5626e6e27ea0 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef0ba52740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feee4010200 con 0x5626e6e27580 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef0ba52740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feee400f460 con 0x5626e6e30a80 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef0ba52740 10 client.0 _ll_register_callbacks cb 0x7feef8009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef01d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e2a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef01d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e2a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42866/0 (socket says 172.21.15.7:42866) 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef0259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e27580 0x5626e6e39070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef01d9b640 1 -- 172.21.15.7:0/1481446544 learned_addr learned my addr 172.21.15.7:0/1481446544 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef02d9d640 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e30a80 0x5626e6e2aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef01d9b640 1 -- 172.21.15.7:0/1481446544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e30a80 msgr2=0x5626e6e2aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef01d9b640 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e30a80 0x5626e6e2aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef01d9b640 1 -- 172.21.15.7:0/1481446544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e27580 msgr2=0x5626e6e39070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef01d9b640 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e27580 0x5626e6e39070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef01d9b640 1 -- 172.21.15.7:0/1481446544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626e6e2b230 con 0x5626e6e27ea0 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.145+0000 7fef0259c640 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e27580 0x5626e6e39070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7fef01d9b640 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e2a480 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7feef8009d10 tx=0x7feef8005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c512e7a424033c0b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7feee9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7feee9ffb640 1 -- 172.21.15.7:0/1481446544 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef800fad0 con 0x5626e6e27ea0 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7feee9ffb640 1 -- 172.21.15.7:0/1481446544 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7feef800fc70 con 0x5626e6e27ea0 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7fef0ba52740 1 -- 172.21.15.7:0/1481446544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feef80054d0 con 0x5626e6e27ea0 2024-04-21T08:45:51.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7feee9ffb640 1 -- 172.21.15.7:0/1481446544 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7feef800fe70 con 0x5626e6e27ea0 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7fef0ba52740 10 client.16038 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7fef0ba52740 1 -- 172.21.15.7:0/1481446544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5626e6ea6b90 con 0x5626e6e27ea0 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7feee9ffb640 1 -- 172.21.15.7:0/1481446544 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef80054d0 con 0x5626e6e27ea0 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7feee9ffb640 1 -- 172.21.15.7:0/1481446544 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7feef8004b70 con 0x5626e6e27ea0 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7fef0ba52740 10 client.16038 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.146+0000 7fef0ba52740 10 client.16038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.147+0000 7fef0ba52740 20 client.16038 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.147+0000 7fef0ba52740 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5626e6ea7250 0x5626e6ea9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.147+0000 7fef0ba52740 4 client.16038 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.147+0000 7fef0ba52740 1 -- 172.21.15.7:0/1481446544 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5626e6d0f590 con 0x5626e6ea7250 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.147+0000 7fef0259c640 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5626e6ea7250 0x5626e6ea9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.147+0000 7fef0259c640 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5626e6ea7250 0x5626e6ea9670 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=456550b1e7957596 server_cookie=267d44253a16532 in_seq=0 out_seq=0 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.147+0000 7feee9ffb640 10 client.16038 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.147+0000 7feee9ffb640 1 -- 172.21.15.7:0/1481446544 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x5626e6d0f590 con 0x5626e6ea7250 2024-04-21T08:45:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.147+0000 7feee9ffb640 10 client.16038 handle_command_reply: tid=0 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.147+0000 7fef0ba52740 1 client.16038 shutdown 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef0ba52740 1 -- 172.21.15.7:0/1481446544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 msgr2=0x5626e6e2a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef0ba52740 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e2a480 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7feef8009d10 tx=0x7feef8005750 comp rx=0 tx=0).stop 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef0ba52740 1 -- 172.21.15.7:0/1481446544 shutdown_connections 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef0ba52740 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5626e6ea7250 0x5626e6ea9670 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef0ba52740 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e6e30a80 0x5626e6e2aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef0ba52740 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5626e6e27ea0 0x5626e6e2a480 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef0ba52740 1 --2- 172.21.15.7:0/1481446544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5626e6e27580 0x5626e6e39070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef0ba52740 1 -- 172.21.15.7:0/1481446544 >> 172.21.15.7:0/1481446544 conn(0x5626e6e21cb0 msgr2=0x5626e6e20d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7feee9ffb640 0 client.16038 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef0ba52740 1 -- 172.21.15.7:0/1481446544 shutdown_connections 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef0ba52740 1 -- 172.21.15.7:0/1481446544 wait complete. 2024-04-21T08:45:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef0ba52740 20 client.16038 trim_cache size 0 max 16384 2024-04-21T08:45:51.154 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:45:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (2883 inodes in the stack)", 2024-04-21T08:45:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:45:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "f310a520-149c-4b3d-a4b6-cd6daac1fcfc": { 2024-04-21T08:45:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:45:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "f310a520-149c-4b3d-a4b6-cd6daac1fcfc", 2024-04-21T08:45:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:45:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:45:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:45:51.154 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:45:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef08e5c640 1 -- 172.21.15.7:0/2939573783 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7feeec0a3e40 msgr2=0x7feeec0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef08e5c640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7feeec0a3e40 0x7feeec0a6300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7feefc005e30 tx=0x7feefc0099d0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef08e5c640 1 -- 172.21.15.7:0/2939573783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04100880 msgr2=0x7fef04198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.148+0000 7fef08e5c640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04100880 0x7fef04198760 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7feef400e9f0 tx=0x7feef400edf0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.149+0000 7fef08e5c640 1 -- 172.21.15.7:0/2939573783 shutdown_connections 2024-04-21T08:45:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.149+0000 7fef08e5c640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7feeec0a3e40 0x7feeec0a6300 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.149+0000 7fef08e5c640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef04102510 0x7fef0419d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.149+0000 7fef08e5c640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef04101240 0x7fef04198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.149+0000 7fef08e5c640 1 --2- 172.21.15.7:0/2939573783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04100880 0x7fef04198760 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.149+0000 7fef08e5c640 1 -- 172.21.15.7:0/2939573783 >> 172.21.15.7:0/2939573783 conn(0x7fef04076160 msgr2=0x7fef040fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.149+0000 7fef08e5c640 1 -- 172.21.15.7:0/2939573783 shutdown_connections 2024-04-21T08:45:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:45:51.149+0000 7fef08e5c640 1 -- 172.21.15.7:0/2939573783 wait complete. 2024-04-21T08:45:51.156 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:50 smithi007 ceph-mon[28477]: pgmap v37: 129 pgs: 129 active+clean; 139 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 12 MiB/s wr, 33 op/s 2024-04-21T08:45:51.160 INFO:tasks.cephfs.filesystem:scrub status for tag:f310a520-149c-4b3d-a4b6-cd6daac1fcfc - {'path': '/', 'tag': 'f310a520-149c-4b3d-a4b6-cd6daac1fcfc', 'options': 'recursive,force'} 2024-04-21T08:45:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:50 smithi016 ceph-mon[31259]: pgmap v37: 129 pgs: 129 active+clean; 139 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 12 MiB/s wr, 33 op/s 2024-04-21T08:45:52.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:51 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:45:52.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:51 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:52.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:51 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:45:52.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:51 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:45:52.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:51 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:52.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:51 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:45:52.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:51 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:45:52.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:51 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:52.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:51 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:45:53.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:52 smithi077 ceph-mon[31615]: pgmap v38: 129 pgs: 129 active+clean; 139 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 12 MiB/s wr, 33 op/s 2024-04-21T08:45:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:52 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:53.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:52 smithi007 ceph-mon[28477]: pgmap v38: 129 pgs: 129 active+clean; 139 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 12 MiB/s wr, 33 op/s 2024-04-21T08:45:53.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:52 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:52 smithi016 ceph-mon[31259]: pgmap v38: 129 pgs: 129 active+clean; 139 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 12 MiB/s wr, 33 op/s 2024-04-21T08:45:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:52 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.146+0000 7fa18d4eb640 1 -- 172.21.15.7:0/3644957516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa188106170 msgr2=0x7fa18810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.146+0000 7fa18d4eb640 1 --2- 172.21.15.7:0/3644957516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa188106170 0x7fa18810dd40 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa180009ca0 tx=0x7fa18002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.146+0000 7fa18d4eb640 1 -- 172.21.15.7:0/3644957516 shutdown_connections 2024-04-21T08:45:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.146+0000 7fa18d4eb640 1 --2- 172.21.15.7:0/3644957516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa188106170 0x7fa18810dd40 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.146+0000 7fa18d4eb640 1 --2- 172.21.15.7:0/3644957516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa188101a50 0x7fa188105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.146+0000 7fa18d4eb640 1 --2- 172.21.15.7:0/3644957516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1881010a0 0x7fa188101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.146+0000 7fa18d4eb640 1 -- 172.21.15.7:0/3644957516 >> 172.21.15.7:0/3644957516 conn(0x7fa188077f70 msgr2=0x7fa188078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.146+0000 7fa18d4eb640 1 -- 172.21.15.7:0/3644957516 shutdown_connections 2024-04-21T08:45:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.146+0000 7fa18d4eb640 1 -- 172.21.15.7:0/3644957516 wait complete. 2024-04-21T08:45:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.146+0000 7fa18d4eb640 1 Processor -- start 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa18d4eb640 1 -- start start 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa18d4eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1881010a0 0x7fa188198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa18d4eb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa188106170 0x7fa188198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa18d4eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa18819cfc0 0x7fa18819d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa18d4eb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa1881110c0 con 0x7fa188106170 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa18d4eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa188110f40 con 0x7fa18819cfc0 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa18d4eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa188111240 con 0x7fa1881010a0 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa17e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa188106170 0x7fa188198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa17e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa188106170 0x7fa188198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42880/0 (socket says 172.21.15.7:42880) 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa17e7fc640 1 -- 172.21.15.7:0/1521137317 learned_addr learned my addr 172.21.15.7:0/1521137317 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa1877fe640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa18819cfc0 0x7fa18819d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa186ffd640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1881010a0 0x7fa188198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa17e7fc640 1 -- 172.21.15.7:0/1521137317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1881010a0 msgr2=0x7fa188198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1881010a0 0x7fa188198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa17e7fc640 1 -- 172.21.15.7:0/1521137317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa18819cfc0 msgr2=0x7fa18819d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa18819cfc0 0x7fa18819d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa17e7fc640 1 -- 172.21.15.7:0/1521137317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18819dab0 con 0x7fa188106170 2024-04-21T08:45:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.147+0000 7fa1877fe640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa18819cfc0 0x7fa18819d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.148+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa188106170 0x7fa188198c40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa170005fd0 tx=0x7fa17000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.148+0000 7fa184ff9640 1 -- 172.21.15.7:0/1521137317 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170017070 con 0x7fa188106170 2024-04-21T08:45:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.148+0000 7fa184ff9640 1 -- 172.21.15.7:0/1521137317 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa170009c30 con 0x7fa188106170 2024-04-21T08:45:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.148+0000 7fa18d4eb640 1 -- 172.21.15.7:0/1521137317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa18819dda0 con 0x7fa188106170 2024-04-21T08:45:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.148+0000 7fa184ff9640 1 -- 172.21.15.7:0/1521137317 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170003d30 con 0x7fa188106170 2024-04-21T08:45:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.148+0000 7fa18d4eb640 1 -- 172.21.15.7:0/1521137317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1881a5590 con 0x7fa188106170 2024-04-21T08:45:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.150+0000 7fa184ff9640 1 -- 172.21.15.7:0/1521137317 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7fa170003ed0 con 0x7fa188106170 2024-04-21T08:45:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.150+0000 7fa1901bc740 1 Processor -- start 2024-04-21T08:45:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.150+0000 7fa1901bc740 1 -- start start 2024-04-21T08:45:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x555902249620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x555902249fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 0x5559022554d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1901bc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5559021ae8d0 con 0x55590224a8d0 2024-04-21T08:45:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1901bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55590214be80 con 0x555902249b60 2024-04-21T08:45:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1901bc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5559021aa890 con 0x5559022530e0 2024-04-21T08:45:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x555902249620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x555902249620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42888/0 (socket says 172.21.15.7:42888) 2024-04-21T08:45:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1877fe640 1 -- 172.21.15.7:0/1827971346 learned_addr learned my addr 172.21.15.7:0/1827971346 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa186ffd640 1 --2- 172.21.15.7:0/1827971346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x555902249fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/1827971346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 0x5559022554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa184ff9640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1740a3e30 0x7fa1740a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1877fe640 1 -- 172.21.15.7:0/1827971346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 msgr2=0x5559022554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1877fe640 1 --2- 172.21.15.7:0/1827971346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 0x5559022554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1877fe640 1 -- 172.21.15.7:0/1827971346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 msgr2=0x555902249fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1877fe640 1 --2- 172.21.15.7:0/1827971346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x555902249fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1877fe640 1 -- 172.21.15.7:0/1827971346 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55590216d320 con 0x55590224a8d0 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa186ffd640 1 --2- 172.21.15.7:0/1827971346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x555902249fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/1827971346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 0x5559022554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa1877fe640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1740a3e30 0x7fa1740a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.151+0000 7fa184ff9640 1 -- 172.21.15.7:0/1521137317 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fa1700d09d0 con 0x7fa188106170 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.152+0000 7fa1877fe640 1 --2- 172.21.15.7:0/1827971346 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x555902249620 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa18002c700 tx=0x7fa180032410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33dbad4f67307ff9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.152+0000 7fa17d7fa640 1 -- 172.21.15.7:0/1827971346 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa180039040 con 0x55590224a8d0 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.152+0000 7fa17d7fa640 1 -- 172.21.15.7:0/1827971346 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa180003b70 con 0x55590224a8d0 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.152+0000 7fa17d7fa640 1 -- 172.21.15.7:0/1827971346 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18004a6b0 con 0x55590224a8d0 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 -- 172.21.15.7:0/1827971346 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 msgr2=0x555902249620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1877fe640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1740a3e30 0x7fa1740a62f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa18004c000 tx=0x7fa180045000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 --2- 172.21.15.7:0/1827971346 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x555902249620 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa18002c700 tx=0x7fa180032410 comp rx=0 tx=0).stop 2024-04-21T08:45:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 -- 172.21.15.7:0/1827971346 shutdown_connections 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 --2- 172.21.15.7:0/1827971346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 0x5559022554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 --2- 172.21.15.7:0/1827971346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x555902249fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 --2- 172.21.15.7:0/1827971346 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x555902249620 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 -- 172.21.15.7:0/1827971346 >> 172.21.15.7:0/1827971346 conn(0x555902210c90 msgr2=0x555902244850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 -- 172.21.15.7:0/1827971346 shutdown_connections 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 -- 172.21.15.7:0/1827971346 wait complete. 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 -- start start 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 Processor -- start 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555902249b60 0x555902252a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590224fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559022530e0 0x555902250380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x555902255e10 con 0x55590224a8d0 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555902136eb0 con 0x5559022530e0 2024-04-21T08:45:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.153+0000 7fa1901bc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555902255f90 con 0x555902249b60 2024-04-21T08:45:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa1901bc740 10 client.0 _ll_register_callbacks cb 0x555902163b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa17e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590224fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa17e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590224fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42890/0 (socket says 172.21.15.7:42890) 2024-04-21T08:45:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa17e7fc640 1 -- 172.21.15.7:0/3319923866 learned_addr learned my addr 172.21.15.7:0/3319923866 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa1877fe640 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559022530e0 0x555902250380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa186ffd640 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555902249b60 0x555902252a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa17e7fc640 1 -- 172.21.15.7:0/3319923866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555902249b60 msgr2=0x555902252a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555902249b60 0x555902252a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa17e7fc640 1 -- 172.21.15.7:0/3319923866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559022530e0 msgr2=0x555902250380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559022530e0 0x555902250380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa17e7fc640 1 -- 172.21.15.7:0/3319923866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555901ec1210 con 0x55590224a8d0 2024-04-21T08:45:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa1877fe640 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559022530e0 0x555902250380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590224fe00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa170098790 tx=0x7fa1700a5060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd3ca6053c9acc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa16e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa16e7fc640 1 -- 172.21.15.7:0/3319923866 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17009c330 con 0x55590224a8d0 2024-04-21T08:45:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa16e7fc640 1 -- 172.21.15.7:0/3319923866 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fa1700cde90 con 0x55590224a8d0 2024-04-21T08:45:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa1901bc740 1 -- 172.21.15.7:0/3319923866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa180003b70 con 0x55590224a8d0 2024-04-21T08:45:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa16e7fc640 1 -- 172.21.15.7:0/3319923866 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa1700a06e0 con 0x55590224a8d0 2024-04-21T08:45:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa1901bc740 10 client.16050 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.154+0000 7fa1901bc740 1 -- 172.21.15.7:0/3319923866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555901f8d620 con 0x55590224a8d0 2024-04-21T08:45:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.155+0000 7fa16e7fc640 1 -- 172.21.15.7:0/3319923866 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa180003b70 con 0x55590224a8d0 2024-04-21T08:45:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.155+0000 7fa16e7fc640 1 -- 172.21.15.7:0/3319923866 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fa170098080 con 0x55590224a8d0 2024-04-21T08:45:54.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.155+0000 7fa1901bc740 10 client.16050 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:54.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.155+0000 7fa1901bc740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:54.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.155+0000 7fa1901bc740 20 client.16050 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:54.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.155+0000 7fa1901bc740 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555902263170 0x555902265560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.155+0000 7fa1901bc740 4 client.16050 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:54.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.155+0000 7fa1901bc740 1 -- 172.21.15.7:0/3319923866 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559021c2880 con 0x555902263170 2024-04-21T08:45:54.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.155+0000 7fa186ffd640 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555902263170 0x555902265560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.155+0000 7fa186ffd640 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555902263170 0x555902265560 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a331c38da376a3fd server_cookie=b9fa6d3be84c134d in_seq=0 out_seq=0 2024-04-21T08:45:54.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.155+0000 7fa16e7fc640 10 client.16050 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa16e7fc640 1 -- 172.21.15.7:0/3319923866 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5559021c2880 con 0x555902263170 2024-04-21T08:45:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa16e7fc640 10 client.16050 handle_command_reply: tid=0 2024-04-21T08:45:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 1 client.16050 shutdown 2024-04-21T08:45:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 1 -- 172.21.15.7:0/3319923866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 msgr2=0x55590224fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590224fe00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa170098790 tx=0x7fa1700a5060 comp rx=0 tx=0).stop 2024-04-21T08:45:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 1 -- 172.21.15.7:0/3319923866 shutdown_connections 2024-04-21T08:45:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555902263170 0x555902265560 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5559022530e0 0x555902250380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590224fe00 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 1 --2- 172.21.15.7:0/3319923866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555902249b60 0x555902252a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa16e7fc640 0 client.16050 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 1 -- 172.21.15.7:0/3319923866 >> 172.21.15.7:0/3319923866 conn(0x555902210c90 msgr2=0x5559022465a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 1 -- 172.21.15.7:0/3319923866 shutdown_connections 2024-04-21T08:45:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 1 -- 172.21.15.7:0/3319923866 wait complete. 2024-04-21T08:45:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.158+0000 7fa1901bc740 20 client.16050 trim_cache size 0 max 16384 2024-04-21T08:45:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.173+0000 7fa1901bc740 1 Processor -- start 2024-04-21T08:45:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.173+0000 7fa1901bc740 1 -- start start 2024-04-21T08:45:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.173+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x55590225ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.173+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590223f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.173+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 0x555902252530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.173+0000 7fa1901bc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa1881ad480 con 0x55590224a8d0 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.173+0000 7fa1901bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1881afd00 con 0x555902249b60 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.173+0000 7fa1901bc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa1881adaf0 con 0x5559022530e0 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590223f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590223f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42892/0 (socket says 172.21.15.7:42892) 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17e7fc640 1 -- 172.21.15.7:0/2453871730 learned_addr learned my addr 172.21.15.7:0/2453871730 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa186ffd640 1 --2- 172.21.15.7:0/2453871730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x55590225ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa1877fe640 1 --2- 172.21.15.7:0/2453871730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 0x555902252530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17e7fc640 1 -- 172.21.15.7:0/2453871730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 msgr2=0x555902252530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/2453871730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 0x555902252530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17e7fc640 1 -- 172.21.15.7:0/2453871730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 msgr2=0x55590225ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/2453871730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x55590225ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17e7fc640 1 -- 172.21.15.7:0/2453871730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55590223ff30 con 0x55590224a8d0 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa1877fe640 1 --2- 172.21.15.7:0/2453871730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 0x555902252530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa186ffd640 1 --2- 172.21.15.7:0/2453871730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x55590225ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/2453871730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590223f760 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa17009c030 tx=0x7fa1700a50d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61d9802a959efce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17cff9640 1 -- 172.21.15.7:0/2453871730 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1700d4070 con 0x55590224a8d0 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17cff9640 1 -- 172.21.15.7:0/2453871730 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1700989a0 con 0x55590224a8d0 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.174+0000 7fa17cff9640 1 -- 172.21.15.7:0/2453871730 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170004420 con 0x55590224a8d0 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.175+0000 7fa1901bc740 1 -- 172.21.15.7:0/2453871730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 msgr2=0x55590223f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.175+0000 7fa1901bc740 1 --2- 172.21.15.7:0/2453871730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590223f760 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa17009c030 tx=0x7fa1700a50d0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.175+0000 7fa1901bc740 1 -- 172.21.15.7:0/2453871730 shutdown_connections 2024-04-21T08:45:54.179 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.175+0000 7fa1901bc740 1 --2- 172.21.15.7:0/2453871730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5559022530e0 0x555902252530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.175+0000 7fa1901bc740 1 --2- 172.21.15.7:0/2453871730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55590224a8d0 0x55590223f760 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.175+0000 7fa1901bc740 1 --2- 172.21.15.7:0/2453871730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x55590225ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.175+0000 7fa1901bc740 1 -- 172.21.15.7:0/2453871730 >> 172.21.15.7:0/2453871730 conn(0x555902210c90 msgr2=0x55590224e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.175+0000 7fa1901bc740 1 -- 172.21.15.7:0/2453871730 shutdown_connections 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.175+0000 7fa1901bc740 1 -- 172.21.15.7:0/2453871730 wait complete. 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.175+0000 7fa1901bc740 1 -- start start 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.175+0000 7fa1901bc740 1 Processor -- start 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x55590225b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55590224a8d0 0x55590224c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1901bc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5559022530e0 0x55590224cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1901bc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa1881ae1c0 con 0x5559022530e0 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1901bc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1881af630 con 0x555902249b60 2024-04-21T08:45:54.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1901bc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa1881ae890 con 0x55590224a8d0 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1901bc740 10 client.0 _ll_register_callbacks cb 0x7fa170005bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa186ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x55590225b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5559022530e0 0x55590224cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5559022530e0 0x55590224cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42904/0 (socket says 172.21.15.7:42904) 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1877fe640 1 -- 172.21.15.7:0/1799277237 learned_addr learned my addr 172.21.15.7:0/1799277237 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa17e7fc640 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55590224a8d0 0x55590224c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1877fe640 1 -- 172.21.15.7:0/1799277237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55590224a8d0 msgr2=0x55590224c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1877fe640 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55590224a8d0 0x55590224c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1877fe640 1 -- 172.21.15.7:0/1799277237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 msgr2=0x55590225b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1877fe640 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x55590225b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1877fe640 1 -- 172.21.15.7:0/1799277237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55590224d670 con 0x5559022530e0 2024-04-21T08:45:54.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa186ffd640 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x55590225b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa1877fe640 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5559022530e0 0x55590224cdf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa18002c630 tx=0x7fa1800095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f3c22ff2d71a4f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa16e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.176+0000 7fa16e7fc640 1 -- 172.21.15.7:0/1799277237 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18005d060 con 0x5559022530e0 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa1901bc740 1 -- 172.21.15.7:0/1799277237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1700989a0 con 0x5559022530e0 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa16e7fc640 1 -- 172.21.15.7:0/1799277237 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7fa180003d80 con 0x5559022530e0 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa16e7fc640 1 -- 172.21.15.7:0/1799277237 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa180064050 con 0x5559022530e0 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa1901bc740 10 client.16056 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa1901bc740 1 -- 172.21.15.7:0/1799277237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5559022c9730 con 0x5559022530e0 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa16e7fc640 1 -- 172.21.15.7:0/1799277237 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1700989a0 con 0x5559022530e0 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa16e7fc640 1 -- 172.21.15.7:0/1799277237 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fa18003f040 con 0x5559022530e0 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa1901bc740 10 client.16056 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa1901bc740 10 client.16056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:54.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa1901bc740 20 client.16056 populate_metadata read hostname 'smithi007' 2024-04-21T08:45:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa1901bc740 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5559022c9a60 0x5559022cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa1901bc740 4 client.16056 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:45:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa1901bc740 1 -- 172.21.15.7:0/1799277237 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55590214e3d0 con 0x5559022c9a60 2024-04-21T08:45:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.177+0000 7fa186ffd640 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5559022c9a60 0x5559022cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.178+0000 7fa16e7fc640 10 client.16056 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:54.178+0000 7fa186ffd640 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5559022c9a60 0x5559022cbe50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbe5e1af92f36cde server_cookie=4199810703672ae9 in_seq=0 out_seq=0 2024-04-21T08:45:55.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:54 smithi077 ceph-mon[31615]: pgmap v39: 129 pgs: 129 active+clean; 146 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 12 MiB/s wr, 48 op/s 2024-04-21T08:45:55.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:54 smithi007 ceph-mon[28477]: pgmap v39: 129 pgs: 129 active+clean; 146 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 12 MiB/s wr, 48 op/s 2024-04-21T08:45:55.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:54 smithi016 ceph-mon[31259]: pgmap v39: 129 pgs: 129 active+clean; 146 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 12 MiB/s wr, 48 op/s 2024-04-21T08:45:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa16e7fc640 1 -- 172.21.15.7:0/1799277237 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55590214e3d0 con 0x5559022c9a60 2024-04-21T08:45:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa16e7fc640 10 client.16056 handle_command_reply: tid=0 2024-04-21T08:45:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 1 client.16056 shutdown 2024-04-21T08:45:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 1 -- 172.21.15.7:0/1799277237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5559022530e0 msgr2=0x55590224cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5559022530e0 0x55590224cdf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa18002c630 tx=0x7fa1800095c0 comp rx=0 tx=0).stop 2024-04-21T08:45:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 1 -- 172.21.15.7:0/1799277237 shutdown_connections 2024-04-21T08:45:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5559022c9a60 0x5559022cbe50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5559022530e0 0x55590224cdf0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55590224a8d0 0x55590224c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 1 --2- 172.21.15.7:0/1799277237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555902249b60 0x55590225b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa16e7fc640 0 client.16056 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:45:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 1 -- 172.21.15.7:0/1799277237 >> 172.21.15.7:0/1799277237 conn(0x555902210c90 msgr2=0x555902245be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 1 -- 172.21.15.7:0/1799277237 shutdown_connections 2024-04-21T08:45:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 1 -- 172.21.15.7:0/1799277237 wait complete. 2024-04-21T08:45:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.592+0000 7fa1901bc740 20 client.16056 trim_cache size 0 max 16384 2024-04-21T08:45:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:45:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:45:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:45:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:45:56.595 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 -- 172.21.15.7:0/1521137317 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1740a3e30 msgr2=0x7fa1740a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:56.595 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1740a3e30 0x7fa1740a62f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa18004c000 tx=0x7fa180045000 comp rx=0 tx=0).stop 2024-04-21T08:45:56.595 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 -- 172.21.15.7:0/1521137317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa188106170 msgr2=0x7fa188198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:56.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa188106170 0x7fa188198c40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa170005fd0 tx=0x7fa17000dde0 comp rx=0 tx=0).stop 2024-04-21T08:45:56.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 -- 172.21.15.7:0/1521137317 shutdown_connections 2024-04-21T08:45:56.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa18819cfc0 0x7fa18819d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:56.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa188106170 0x7fa188198c40 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:56.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1740a3e30 0x7fa1740a62f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:56.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 --2- 172.21.15.7:0/1521137317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1881010a0 0x7fa188198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:56.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 -- 172.21.15.7:0/1521137317 >> 172.21.15.7:0/1521137317 conn(0x7fa188077f70 msgr2=0x7fa188069aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:56.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 -- 172.21.15.7:0/1521137317 shutdown_connections 2024-04-21T08:45:56.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:45:56.593+0000 7fa18d4eb640 1 -- 172.21.15.7:0/1521137317 wait complete. 2024-04-21T08:45:57.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:56 smithi077 ceph-mon[31615]: pgmap v40: 129 pgs: 129 active+clean; 146 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 12 MiB/s wr, 43 op/s 2024-04-21T08:45:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:56 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:45:57.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:56 smithi007 ceph-mon[28477]: pgmap v40: 129 pgs: 129 active+clean; 146 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 12 MiB/s wr, 43 op/s 2024-04-21T08:45:57.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:56 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:45:57.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:56 smithi016 ceph-mon[31259]: pgmap v40: 129 pgs: 129 active+clean; 146 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 12 MiB/s wr, 43 op/s 2024-04-21T08:45:57.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:56 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:45:58.354 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:45:58 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i[62672]: *** Caught signal (Segmentation fault) ** 2024-04-21T08:45:58.354 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:45:58 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i[62672]: in thread 7f4372fcc700 thread_name:MR_Finisher 2024-04-21T08:45:58.354 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:45:58 smithi077 podman[65767]: 2024-04-21 08:45:58.126761441 +0000 UTC m=+0.023067187 container died 92989efcacbcc14b2ff5d02b9b55dce83b21cd87f391f46eddac1f592c09f7eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-21T08:45:59.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:45:58 smithi077 ceph-mon[31615]: pgmap v41: 129 pgs: 129 active+clean; 136 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 13 MiB/s wr, 1.66k op/s 2024-04-21T08:45:59.104 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:45:58 smithi077 podman[65767]: 2024-04-21 08:45:58.831178509 +0000 UTC m=+0.727484302 container remove 92989efcacbcc14b2ff5d02b9b55dce83b21cd87f391f46eddac1f592c09f7eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-04-21T08:45:59.104 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:45:58 smithi077 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.i.service: Main process exited, code=exited, status=139/n/a 2024-04-21T08:45:59.104 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:45:59 smithi077 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-21T08:45:59.104 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:45:59 smithi077 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.i.service: Consumed 2.145s CPU time. 2024-04-21T08:45:59.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:45:58 smithi007 ceph-mon[28477]: pgmap v41: 129 pgs: 129 active+clean; 136 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 13 MiB/s wr, 1.66k op/s 2024-04-21T08:45:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:45:58 smithi016 ceph-mon[31259]: pgmap v41: 129 pgs: 129 active+clean; 136 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 13 MiB/s wr, 1.66k op/s 2024-04-21T08:46:01.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:00 smithi007 ceph-mon[28477]: pgmap v42: 129 pgs: 129 active+clean; 136 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 4.2 MiB/s wr, 1.64k op/s 2024-04-21T08:46:01.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:00 smithi016 ceph-mon[31259]: pgmap v42: 129 pgs: 129 active+clean; 136 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 4.2 MiB/s wr, 1.64k op/s 2024-04-21T08:46:01.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:00 smithi077 ceph-mon[31615]: pgmap v42: 129 pgs: 129 active+clean; 136 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 4.2 MiB/s wr, 1.64k op/s 2024-04-21T08:46:03.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:02 smithi007 ceph-mon[28477]: pgmap v43: 129 pgs: 129 active+clean; 136 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 1.1 MiB/s wr, 1.64k op/s 2024-04-21T08:46:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:02 smithi016 ceph-mon[31259]: pgmap v43: 129 pgs: 129 active+clean; 136 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 1.1 MiB/s wr, 1.64k op/s 2024-04-21T08:46:03.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:02 smithi077 ceph-mon[31615]: pgmap v43: 129 pgs: 129 active+clean; 136 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 1.1 MiB/s wr, 1.64k op/s 2024-04-21T08:46:05.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:04 smithi007 ceph-mon[28477]: pgmap v44: 129 pgs: 129 active+clean; 143 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 1.7 MiB/s wr, 1.66k op/s 2024-04-21T08:46:05.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:04 smithi016 ceph-mon[31259]: pgmap v44: 129 pgs: 129 active+clean; 143 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 1.7 MiB/s wr, 1.66k op/s 2024-04-21T08:46:05.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:04 smithi077 ceph-mon[31615]: pgmap v44: 129 pgs: 129 active+clean; 143 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 1.7 MiB/s wr, 1.66k op/s 2024-04-21T08:46:07.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:06 smithi007 ceph-mon[28477]: pgmap v45: 129 pgs: 129 active+clean; 143 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.1 MiB/s wr, 1.64k op/s 2024-04-21T08:46:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:06 smithi016 ceph-mon[31259]: pgmap v45: 129 pgs: 129 active+clean; 143 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.1 MiB/s wr, 1.64k op/s 2024-04-21T08:46:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:06 smithi077 ceph-mon[31615]: pgmap v45: 129 pgs: 129 active+clean; 143 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 1.1 MiB/s wr, 1.64k op/s 2024-04-21T08:46:09.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:08 smithi007 ceph-mon[28477]: pgmap v46: 129 pgs: 129 active+clean; 177 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 4.3 MiB/s wr, 2.21k op/s 2024-04-21T08:46:09.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:08 smithi016 ceph-mon[31259]: pgmap v46: 129 pgs: 129 active+clean; 177 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 4.3 MiB/s wr, 2.21k op/s 2024-04-21T08:46:09.338 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:08 smithi077 ceph-mon[31615]: pgmap v46: 129 pgs: 129 active+clean; 177 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 4.3 MiB/s wr, 2.21k op/s 2024-04-21T08:46:09.338 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:46:09 smithi077 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-04-21T08:46:09.339 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:46:09 smithi077 systemd[1]: Stopped Ceph mds.i for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:46:09.339 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:46:09 smithi077 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.i.service: Consumed 2.145s CPU time. 2024-04-21T08:46:09.339 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:46:09 smithi077 systemd[1]: Starting Ceph mds.i for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:46:09.603 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:46:09 smithi077 podman[65861]: 2024-04-21 08:46:09.242085125 +0000 UTC m=+0.018172674 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:46:09.604 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:46:09 smithi077 podman[65861]: 2024-04-21 08:46:09.41593505 +0000 UTC m=+0.192022598 container create ea868477292cecddd08fc1fc49a28a8531c0129f41958ba23e60fd9c0d5c430b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-21T08:46:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.759+0000 7f782a915640 1 -- 172.21.15.7:0/3207505494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f782411b740 msgr2=0x7f782411db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:09.760 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.759+0000 7f782a915640 1 --2- 172.21.15.7:0/3207505494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f782411b740 0x7f782411db30 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7814009860 tx=0x7f781402bbb0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.763+0000 7f782a915640 1 -- 172.21.15.7:0/3207505494 shutdown_connections 2024-04-21T08:46:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.763+0000 7f782a915640 1 --2- 172.21.15.7:0/3207505494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f782411b740 0x7f782411db30 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.763+0000 7f782a915640 1 --2- 172.21.15.7:0/3207505494 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7824109520 0x7f7824109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.763+0000 7f782a915640 1 --2- 172.21.15.7:0/3207505494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782410c6a0 0x7f7824108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.763+0000 7f782a915640 1 -- 172.21.15.7:0/3207505494 >> 172.21.15.7:0/3207505494 conn(0x7f782406d4c0 msgr2=0x7f782406d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.766+0000 7f782a915640 1 -- 172.21.15.7:0/3207505494 shutdown_connections 2024-04-21T08:46:09.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.767+0000 7f782a915640 1 -- 172.21.15.7:0/3207505494 wait complete. 2024-04-21T08:46:09.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.767+0000 7f782a915640 1 Processor -- start 2024-04-21T08:46:09.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.768+0000 7f782a915640 1 -- start start 2024-04-21T08:46:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.768+0000 7f782a915640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7824109520 0x7f78241a7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.769+0000 7f782a915640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f782410c6a0 0x7f78241a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.769+0000 7f782a915640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782411b740 0x7f78241abf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.769+0000 7f782a915640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7824121030 con 0x7f7824109520 2024-04-21T08:46:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.769+0000 7f782a915640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7824120eb0 con 0x7f782410c6a0 2024-04-21T08:46:09.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.769+0000 7f782a915640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f78241211b0 con 0x7f782411b740 2024-04-21T08:46:09.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.769+0000 7f7823fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7824109520 0x7f78241a7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.769+0000 7f7823fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7824109520 0x7f78241a7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47610/0 (socket says 172.21.15.7:47610) 2024-04-21T08:46:09.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.769+0000 7f7823fff640 1 -- 172.21.15.7:0/1624800711 learned_addr learned my addr 172.21.15.7:0/1624800711 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:09.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.769+0000 7f78237fe640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f782410c6a0 0x7f78241a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.771+0000 7f7828eb2640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782411b740 0x7f78241abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.771 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.771+0000 7f78237fe640 1 -- 172.21.15.7:0/1624800711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782411b740 msgr2=0x7f78241abf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:09.771 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.771+0000 7f78237fe640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782411b740 0x7f78241abf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.771 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.771+0000 7f78237fe640 1 -- 172.21.15.7:0/1624800711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7824109520 msgr2=0x7f78241a7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:09.771 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.771+0000 7f78237fe640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7824109520 0x7f78241a7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.771 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.772+0000 7f78237fe640 1 -- 172.21.15.7:0/1624800711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78241ac660 con 0x7f782410c6a0 2024-04-21T08:46:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.773+0000 7f78237fe640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f782410c6a0 0x7f78241a7bb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7818009fd0 tx=0x7f781800b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.773+0000 7f78217fa640 1 -- 172.21.15.7:0/1624800711 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818015020 con 0x7f782410c6a0 2024-04-21T08:46:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.773+0000 7f78217fa640 1 -- 172.21.15.7:0/1624800711 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7818011ce0 con 0x7f782410c6a0 2024-04-21T08:46:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.773+0000 7f78217fa640 1 -- 172.21.15.7:0/1624800711 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818010de0 con 0x7f782410c6a0 2024-04-21T08:46:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.773+0000 7f782a915640 1 -- 172.21.15.7:0/1624800711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78241ac950 con 0x7f782410c6a0 2024-04-21T08:46:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.774+0000 7f782a915640 1 -- 172.21.15.7:0/1624800711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78241b41e0 con 0x7f782410c6a0 2024-04-21T08:46:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.777+0000 7f782d5e7740 1 Processor -- start 2024-04-21T08:46:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.778+0000 7f782d5e7740 1 -- start start 2024-04-21T08:46:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.778+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.778+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 0x55a8b1de0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.778+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1dea0e0 0x55a8b1dec4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.778+0000 7f782d5e7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a8b1d458d0 con 0x55a8b1de0b60 2024-04-21T08:46:09.779 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.778+0000 7f782d5e7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a8b1ce2e80 con 0x55a8b1dea0e0 2024-04-21T08:46:09.779 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.778+0000 7f782d5e7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a8b1d41890 con 0x55a8b1de18d0 2024-04-21T08:46:09.779 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.779+0000 7f7823fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 0x55a8b1de0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.779 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.779+0000 7f7823fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 0x55a8b1de0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47622/0 (socket says 172.21.15.7:47622) 2024-04-21T08:46:09.779 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.779+0000 7f7823fff640 1 -- 172.21.15.7:0/3514780919 learned_addr learned my addr 172.21.15.7:0/3514780919 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:09.779 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.779+0000 7f7823fff640 1 -- 172.21.15.7:0/3514780919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 msgr2=0x55a8b1de0620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:46:09.779 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.779+0000 7f7823fff640 1 --2- 172.21.15.7:0/3514780919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de0620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.779 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.779+0000 7f7823fff640 1 -- 172.21.15.7:0/3514780919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1dea0e0 msgr2=0x55a8b1dec4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:46:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.779+0000 7f7823fff640 1 --2- 172.21.15.7:0/3514780919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1dea0e0 0x55a8b1dec4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.779+0000 7f7823fff640 1 -- 172.21.15.7:0/3514780919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8b1d04320 con 0x55a8b1de0b60 2024-04-21T08:46:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.780+0000 7f7823fff640 1 --2- 172.21.15.7:0/3514780919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 0x55a8b1de0fe0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f781c00e600 tx=0x7f781c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6046b6ec04664c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.780+0000 7f78217fa640 1 -- 172.21.15.7:0/1624800711 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f7818018020 con 0x7f782410c6a0 2024-04-21T08:46:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.781+0000 7f78217fa640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f780c0a4100 0x7f780c0a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.782+0000 7f780a7fc640 1 -- 172.21.15.7:0/3514780919 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c00da10 con 0x55a8b1de0b60 2024-04-21T08:46:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.782+0000 7f78217fa640 1 -- 172.21.15.7:0/1624800711 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f7818010b80 con 0x7f782410c6a0 2024-04-21T08:46:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.782+0000 7f780a7fc640 1 -- 172.21.15.7:0/3514780919 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f781c01c070 con 0x55a8b1de0b60 2024-04-21T08:46:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.782+0000 7f780a7fc640 1 -- 172.21.15.7:0/3514780919 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c016380 con 0x55a8b1de0b60 2024-04-21T08:46:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.783+0000 7f782d5e7740 1 -- 172.21.15.7:0/3514780919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 msgr2=0x55a8b1de0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.783+0000 7f782d5e7740 1 --2- 172.21.15.7:0/3514780919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 0x55a8b1de0fe0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f781c00e600 tx=0x7f781c0095c0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.784+0000 7f782d5e7740 1 -- 172.21.15.7:0/3514780919 shutdown_connections 2024-04-21T08:46:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.784+0000 7f782d5e7740 1 --2- 172.21.15.7:0/3514780919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1dea0e0 0x55a8b1dec4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.784+0000 7f782d5e7740 1 --2- 172.21.15.7:0/3514780919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 0x55a8b1de0fe0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.784+0000 7f782d5e7740 1 --2- 172.21.15.7:0/3514780919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.784+0000 7f782d5e7740 1 -- 172.21.15.7:0/3514780919 >> 172.21.15.7:0/3514780919 conn(0x55a8b1da7c90 msgr2=0x55a8b1ddb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:09.785 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.785+0000 7f7828eb2640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f780c0a4100 0x7f780c0a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.787+0000 7f782d5e7740 1 -- 172.21.15.7:0/3514780919 shutdown_connections 2024-04-21T08:46:09.790 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.790+0000 7f782d5e7740 1 -- 172.21.15.7:0/3514780919 wait complete. 2024-04-21T08:46:09.790 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.791+0000 7f782d5e7740 1 -- start start 2024-04-21T08:46:09.790 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.791+0000 7f782d5e7740 1 Processor -- start 2024-04-21T08:46:09.794 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.794+0000 7f7828eb2640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f780c0a4100 0x7f780c0a65c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f78140058c0 tx=0x7f7814005850 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:09.794 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.795+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1de0b60 0x55a8b1de9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.794 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.795+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.794 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.795+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1dea0e0 0x55a8b1de74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.794 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.795+0000 7f782d5e7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a8b1dfde20 con 0x55a8b1dea0e0 2024-04-21T08:46:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.795+0000 7f782d5e7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a8b1ccdeb0 con 0x55a8b1de0b60 2024-04-21T08:46:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.795+0000 7f782d5e7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a8b1dfdfa0 con 0x55a8b1de18d0 2024-04-21T08:46:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.795+0000 7f782d5e7740 10 client.0 _ll_register_callbacks cb 0x55a8b1cfab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.795+0000 7f78237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.795+0000 7f78237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:58996/0 (socket says 172.21.15.7:58996) 2024-04-21T08:46:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.795+0000 7f78237fe640 1 -- 172.21.15.7:0/1028932476 learned_addr learned my addr 172.21.15.7:0/1028932476 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:09.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.795+0000 7f7823fff640 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1de0b60 0x55a8b1de9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.796 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.797+0000 7f78237fe640 1 -- 172.21.15.7:0/1028932476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1de0b60 msgr2=0x55a8b1de9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:09.796 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.797+0000 7f78237fe640 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1de0b60 0x55a8b1de9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.796 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.797+0000 7f78237fe640 1 -- 172.21.15.7:0/1028932476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1dea0e0 msgr2=0x55a8b1de74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.797+0000 7f78237fe640 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1dea0e0 0x55a8b1de74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.797 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.797+0000 7f78237fe640 1 -- 172.21.15.7:0/1028932476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8b1a58210 con 0x55a8b1de18d0 2024-04-21T08:46:09.798 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.799+0000 7f78237fe640 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de6e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f781809ba50 tx=0x7f781809b770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=360ec09f2cd6075b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:09.798 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.799+0000 7f78037fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:46:09.800 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.801+0000 7f78037fe640 1 -- 172.21.15.7:0/1028932476 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78180a49c0 con 0x55a8b1de18d0 2024-04-21T08:46:09.800 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.801+0000 7f78037fe640 1 -- 172.21.15.7:0/1028932476 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9524+0+0 (secure 0 0 0) 0x7f78180104a0 con 0x55a8b1de18d0 2024-04-21T08:46:09.804 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.804+0000 7f782d5e7740 1 -- 172.21.15.7:0/1028932476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f781c01c070 con 0x55a8b1de18d0 2024-04-21T08:46:09.805 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.806+0000 7f78037fe640 1 -- 172.21.15.7:0/1028932476 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f78180b5200 con 0x55a8b1de18d0 2024-04-21T08:46:09.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.808+0000 7f782d5e7740 10 client.25672 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:09.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.808+0000 7f782d5e7740 1 -- 172.21.15.7:0/1028932476 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a8b1b24620 con 0x55a8b1de18d0 2024-04-21T08:46:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.810+0000 7f78037fe640 1 -- 172.21.15.7:0/1028932476 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f781c01c070 con 0x55a8b1de18d0 2024-04-21T08:46:09.810 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.810+0000 7f78037fe640 1 -- 172.21.15.7:0/1028932476 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f78180cd7a0 con 0x55a8b1de18d0 2024-04-21T08:46:09.813 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.812+0000 7f782d5e7740 10 client.25672 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:09.813 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.812+0000 7f782d5e7740 10 client.25672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:09.813 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.812+0000 7f782d5e7740 20 client.25672 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:09.813 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.812+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8b1e0b130 0x55a8b1e0d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.813 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.812+0000 7f782d5e7740 4 client.25672 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:09.813 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.812+0000 7f782d5e7740 1 -- 172.21.15.7:0/1028932476 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8b1d59880 con 0x55a8b1e0b130 2024-04-21T08:46:09.816 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.816+0000 7f7823fff640 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8b1e0b130 0x55a8b1e0d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.829+0000 7f7823fff640 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8b1e0b130 0x55a8b1e0d520 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4e256e29be6d24c server_cookie=44dcfcebfd4f2d74 in_seq=0 out_seq=0 2024-04-21T08:46:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.830+0000 7f78037fe640 10 client.25672 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.833+0000 7f78037fe640 1 -- 172.21.15.7:0/1028932476 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8b1d59880 con 0x55a8b1e0b130 2024-04-21T08:46:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.833+0000 7f78037fe640 10 client.25672 handle_command_reply: tid=0 2024-04-21T08:46:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.833+0000 7f782d5e7740 1 client.25672 shutdown 2024-04-21T08:46:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.834+0000 7f782d5e7740 1 -- 172.21.15.7:0/1028932476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 msgr2=0x55a8b1de6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.834+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de6e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f781809ba50 tx=0x7f781809b770 comp rx=0 tx=0).stop 2024-04-21T08:46:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.834+0000 7f782d5e7740 1 -- 172.21.15.7:0/1028932476 shutdown_connections 2024-04-21T08:46:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.834+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8b1e0b130 0x55a8b1e0d520 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.834+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1dea0e0 0x55a8b1de74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.834+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de6e90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.834+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1028932476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1de0b60 0x55a8b1de9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.834+0000 7f782d5e7740 1 -- 172.21.15.7:0/1028932476 >> 172.21.15.7:0/1028932476 conn(0x55a8b1da7c90 msgr2=0x55a8b1ddd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.834+0000 7f78037fe640 0 client.25672 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.838+0000 7f782d5e7740 1 -- 172.21.15.7:0/1028932476 shutdown_connections 2024-04-21T08:46:09.838 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.838+0000 7f782d5e7740 1 -- 172.21.15.7:0/1028932476 wait complete. 2024-04-21T08:46:09.838 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.838+0000 7f782d5e7740 20 client.25672 trim_cache size 0 max 16384 2024-04-21T08:46:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.875+0000 7f782d5e7740 1 Processor -- start 2024-04-21T08:46:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.875+0000 7f782d5e7740 1 -- start start 2024-04-21T08:46:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.875+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 0x55a8b1e06d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.875+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.876+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1dea0e0 0x55a8b1de9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.876+0000 7f78237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.876+0000 7f78237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:59012/0 (socket says 172.21.15.7:59012) 2024-04-21T08:46:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.876+0000 7f78237fe640 1 -- 172.21.15.7:0/3197302663 learned_addr learned my addr 172.21.15.7:0/3197302663 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.876+0000 7f782d5e7740 1 -- 172.21.15.7:0/3197302663 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f78241a89c0 con 0x55a8b1de0b60 2024-04-21T08:46:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.876+0000 7f782d5e7740 1 -- 172.21.15.7:0/3197302663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78241b6530 con 0x55a8b1dea0e0 2024-04-21T08:46:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.876+0000 7f782d5e7740 1 -- 172.21.15.7:0/3197302663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f78241b43b0 con 0x55a8b1de18d0 2024-04-21T08:46:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.877+0000 7f7823fff640 1 --2- 172.21.15.7:0/3197302663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 0x55a8b1e06d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.877+0000 7f7823fff640 1 -- 172.21.15.7:0/3197302663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 msgr2=0x55a8b1de95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.877+0000 7f7823fff640 1 --2- 172.21.15.7:0/3197302663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.877+0000 7f7823fff640 1 -- 172.21.15.7:0/3197302663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1dea0e0 msgr2=0x55a8b1de9b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:46:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.877+0000 7f7823fff640 1 --2- 172.21.15.7:0/3197302663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1dea0e0 0x55a8b1de9b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.877+0000 7f7823fff640 1 -- 172.21.15.7:0/3197302663 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8b1ddfe30 con 0x55a8b1de0b60 2024-04-21T08:46:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.878+0000 7f7823fff640 1 --2- 172.21.15.7:0/3197302663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 0x55a8b1e06d20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f781c000f80 tx=0x7f781c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede41567fae62e37 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.879+0000 7f7809ffb640 1 -- 172.21.15.7:0/3197302663 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c030070 con 0x55a8b1de0b60 2024-04-21T08:46:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.879+0000 7f7809ffb640 1 -- 172.21.15.7:0/3197302663 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f781c01c030 con 0x55a8b1de0b60 2024-04-21T08:46:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.880+0000 7f782d5e7740 1 -- 172.21.15.7:0/3197302663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 msgr2=0x55a8b1e06d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.881+0000 7f782d5e7740 1 --2- 172.21.15.7:0/3197302663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 0x55a8b1e06d20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f781c000f80 tx=0x7f781c0095c0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.883+0000 7f782d5e7740 1 -- 172.21.15.7:0/3197302663 shutdown_connections 2024-04-21T08:46:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.883+0000 7f782d5e7740 1 --2- 172.21.15.7:0/3197302663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1dea0e0 0x55a8b1de9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.883+0000 7f782d5e7740 1 --2- 172.21.15.7:0/3197302663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1de95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.883+0000 7f782d5e7740 1 --2- 172.21.15.7:0/3197302663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1de0b60 0x55a8b1e06d20 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.883+0000 7f782d5e7740 1 -- 172.21.15.7:0/3197302663 >> 172.21.15.7:0/3197302663 conn(0x55a8b1da7c90 msgr2=0x55a8b1de5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:09.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.883+0000 7f782d5e7740 1 -- 172.21.15.7:0/3197302663 shutdown_connections 2024-04-21T08:46:09.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.888+0000 7f782d5e7740 1 -- 172.21.15.7:0/3197302663 wait complete. 2024-04-21T08:46:09.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.888+0000 7f782d5e7740 1 -- start start 2024-04-21T08:46:09.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.888+0000 7f782d5e7740 1 Processor -- start 2024-04-21T08:46:09.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.891+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1de0b60 0x55a8b1deac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.891+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1deb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.891+0000 7f782d5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1dea0e0 0x55a8b1deb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.891+0000 7f782d5e7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f78241b49f0 con 0x55a8b1dea0e0 2024-04-21T08:46:09.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.891+0000 7f782d5e7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78241b5e60 con 0x55a8b1de0b60 2024-04-21T08:46:09.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.891+0000 7f782d5e7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f78241b50c0 con 0x55a8b1de18d0 2024-04-21T08:46:09.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.891+0000 7f782d5e7740 10 client.0 _ll_register_callbacks cb 0x7f781800bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:09.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.891+0000 7f7823fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1de0b60 0x55a8b1deac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.893+0000 7f78237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1deb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.893+0000 7f78237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1deb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:59028/0 (socket says 172.21.15.7:59028) 2024-04-21T08:46:09.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.893+0000 7f78237fe640 1 -- 172.21.15.7:0/1163754657 learned_addr learned my addr 172.21.15.7:0/1163754657 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.894+0000 7f7828eb2640 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1dea0e0 0x55a8b1deb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.894+0000 7f7828eb2640 1 -- 172.21.15.7:0/1163754657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 msgr2=0x55a8b1deb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.894+0000 7f7828eb2640 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1deb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.894+0000 7f7828eb2640 1 -- 172.21.15.7:0/1163754657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1de0b60 msgr2=0x55a8b1deac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.894+0000 7f7828eb2640 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1de0b60 0x55a8b1deac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.894+0000 7f7828eb2640 1 -- 172.21.15.7:0/1163754657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8b1e01300 con 0x55a8b1dea0e0 2024-04-21T08:46:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.898+0000 7f7828eb2640 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1dea0e0 0x55a8b1deb6d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7814009780 tx=0x7f7814005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c959cc22a76e087f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.898+0000 7f78037fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.905+0000 7f78037fe640 1 -- 172.21.15.7:0/1163754657 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781404e020 con 0x55a8b1dea0e0 2024-04-21T08:46:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.905+0000 7f782d5e7740 1 -- 172.21.15.7:0/1163754657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f781c01c030 con 0x55a8b1dea0e0 2024-04-21T08:46:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.905+0000 7f78037fe640 1 -- 172.21.15.7:0/1163754657 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f781402c750 con 0x55a8b1dea0e0 2024-04-21T08:46:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.906+0000 7f78037fe640 1 -- 172.21.15.7:0/1163754657 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7814039040 con 0x55a8b1dea0e0 2024-04-21T08:46:09.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.907+0000 7f782d5e7740 10 client.16074 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:09.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.907+0000 7f782d5e7740 1 -- 172.21.15.7:0/1163754657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a8b1e71920 con 0x55a8b1dea0e0 2024-04-21T08:46:09.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.908+0000 7f78037fe640 1 -- 172.21.15.7:0/1163754657 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f781c01c030 con 0x55a8b1dea0e0 2024-04-21T08:46:09.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.909+0000 7f78037fe640 1 -- 172.21.15.7:0/1163754657 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f781403f440 con 0x55a8b1dea0e0 2024-04-21T08:46:09.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.909+0000 7f782d5e7740 10 client.16074 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:09.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.910+0000 7f782d5e7740 10 client.16074 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:09.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.910+0000 7f782d5e7740 20 client.16074 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:09.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.910+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8b1e72020 0x55a8b1e74410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:09.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.910+0000 7f782d5e7740 4 client.16074 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:46:09.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.910+0000 7f782d5e7740 1 -- 172.21.15.7:0/1163754657 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a8b1ce53d0 con 0x55a8b1e72020 2024-04-21T08:46:09.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.911+0000 7f7823fff640 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8b1e72020 0x55a8b1e74410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:09.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.912+0000 7f78037fe640 10 client.16074 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:09.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:09.912+0000 7f7823fff640 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8b1e72020 0x55a8b1e74410 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39c2a937cedd50bc server_cookie=3744b88ba44db876 in_seq=0 out_seq=0 2024-04-21T08:46:10.104 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:46:09 smithi077 podman[65861]: 2024-04-21 08:46:09.652308948 +0000 UTC m=+0.428396509 container init ea868477292cecddd08fc1fc49a28a8531c0129f41958ba23e60fd9c0d5c430b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-21T08:46:10.104 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:46:09 smithi077 podman[65861]: 2024-04-21 08:46:09.663127656 +0000 UTC m=+0.439215215 container start ea868477292cecddd08fc1fc49a28a8531c0129f41958ba23e60fd9c0d5c430b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-21T08:46:10.104 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:46:09 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i[65876]: starting mds.i at 2024-04-21T08:46:10.105 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:46:09 smithi077 bash[65861]: ea868477292cecddd08fc1fc49a28a8531c0129f41958ba23e60fd9c0d5c430b 2024-04-21T08:46:10.105 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:46:09 smithi077 systemd[1]: Started Ceph mds.i for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:46:10.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:09 smithi077 ceph-mon[31615]: daemon mds.i restarted 2024-04-21T08:46:10.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:09 smithi007 ceph-mon[28477]: daemon mds.i restarted 2024-04-21T08:46:10.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:09 smithi016 ceph-mon[31259]: daemon mds.i restarted 2024-04-21T08:46:11.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:10 smithi007 ceph-mon[28477]: pgmap v47: 129 pgs: 129 active+clean; 177 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.7 MiB/s wr, 593 op/s 2024-04-21T08:46:11.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:10 smithi007 ceph-mon[28477]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:46:11.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:10 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.77:6838/3755986498,v1:172.21.15.77:6839/3755986498] up:boot 2024-04-21T08:46:11.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:10 smithi007 ceph-mon[28477]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:46:11.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:10 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:46:11.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:10 smithi007 ceph-mon[28477]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:46:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:10 smithi016 ceph-mon[31259]: pgmap v47: 129 pgs: 129 active+clean; 177 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.7 MiB/s wr, 593 op/s 2024-04-21T08:46:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:10 smithi016 ceph-mon[31259]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:46:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:10 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.77:6838/3755986498,v1:172.21.15.77:6839/3755986498] up:boot 2024-04-21T08:46:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:10 smithi016 ceph-mon[31259]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:46:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:10 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:46:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:10 smithi016 ceph-mon[31259]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:46:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:10 smithi077 ceph-mon[31615]: pgmap v47: 129 pgs: 129 active+clean; 177 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.7 MiB/s wr, 593 op/s 2024-04-21T08:46:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:10 smithi077 ceph-mon[31615]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:46:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:10 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.77:6838/3755986498,v1:172.21.15.77:6839/3755986498] up:boot 2024-04-21T08:46:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:10 smithi077 ceph-mon[31615]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:46:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:10 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:46:11.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:10 smithi077 ceph-mon[31615]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:46:13.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:12 smithi007 ceph-mon[28477]: pgmap v49: 129 pgs: 129 active+clean; 177 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.5 MiB/s wr, 712 op/s 2024-04-21T08:46:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:12 smithi016 ceph-mon[31259]: pgmap v49: 129 pgs: 129 active+clean; 177 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.5 MiB/s wr, 712 op/s 2024-04-21T08:46:13.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:12 smithi077 ceph-mon[31615]: pgmap v49: 129 pgs: 129 active+clean; 177 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.5 MiB/s wr, 712 op/s 2024-04-21T08:46:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:13.897+0000 7f78217fa640 1 -- 172.21.15.7:0/1624800711 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f78180bc0b0 con 0x7f782410c6a0 2024-04-21T08:46:15.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:14 smithi007 ceph-mon[28477]: pgmap v50: 129 pgs: 129 active+clean; 213 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.6 MiB/s wr, 928 op/s 2024-04-21T08:46:15.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:14 smithi007 ceph-mon[28477]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T08:46:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:14 smithi016 ceph-mon[31259]: pgmap v50: 129 pgs: 129 active+clean; 213 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.6 MiB/s wr, 928 op/s 2024-04-21T08:46:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:14 smithi016 ceph-mon[31259]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T08:46:15.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:14 smithi077 ceph-mon[31615]: pgmap v50: 129 pgs: 129 active+clean; 213 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.6 MiB/s wr, 928 op/s 2024-04-21T08:46:15.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:14 smithi077 ceph-mon[31615]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T08:46:17.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:16 smithi007 ceph-mon[28477]: pgmap v51: 129 pgs: 129 active+clean; 213 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.6 MiB/s wr, 928 op/s 2024-04-21T08:46:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:16 smithi016 ceph-mon[31259]: pgmap v51: 129 pgs: 129 active+clean; 213 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.6 MiB/s wr, 928 op/s 2024-04-21T08:46:17.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:16 smithi077 ceph-mon[31615]: pgmap v51: 129 pgs: 129 active+clean; 213 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 7.6 MiB/s wr, 928 op/s 2024-04-21T08:46:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.918+0000 7f78037fe640 1 -- 172.21.15.7:0/1163754657 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a8b1ce53d0 con 0x55a8b1e72020 2024-04-21T08:46:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.918+0000 7f78037fe640 10 client.16074 handle_command_reply: tid=0 2024-04-21T08:46:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.918+0000 7f782d5e7740 1 client.16074 shutdown 2024-04-21T08:46:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782d5e7740 1 -- 172.21.15.7:0/1163754657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1dea0e0 msgr2=0x55a8b1deb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1dea0e0 0x55a8b1deb6d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7814009780 tx=0x7f7814005f90 comp rx=0 tx=0).stop 2024-04-21T08:46:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782d5e7740 1 -- 172.21.15.7:0/1163754657 shutdown_connections 2024-04-21T08:46:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8b1e72020 0x55a8b1e74410 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8b1dea0e0 0x55a8b1deb6d0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8b1de18d0 0x55a8b1deb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782d5e7740 1 --2- 172.21.15.7:0/1163754657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8b1de0b60 0x55a8b1deac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f78037fe640 0 client.16074 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782d5e7740 1 -- 172.21.15.7:0/1163754657 >> 172.21.15.7:0/1163754657 conn(0x55a8b1da7c90 msgr2=0x55a8b1ddcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782d5e7740 1 -- 172.21.15.7:0/1163754657 shutdown_connections 2024-04-21T08:46:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782d5e7740 1 -- 172.21.15.7:0/1163754657 wait complete. 2024-04-21T08:46:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782d5e7740 20 client.16074 trim_cache size 0 max 16384 2024-04-21T08:46:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:46:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:46:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:46:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:46:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782a915640 1 -- 172.21.15.7:0/1624800711 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f780c0a4100 msgr2=0x7f780c0a65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782a915640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f780c0a4100 0x7f780c0a65c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f78140058c0 tx=0x7f7814005850 comp rx=0 tx=0).stop 2024-04-21T08:46:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782a915640 1 -- 172.21.15.7:0/1624800711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f782410c6a0 msgr2=0x7f78241a7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.919+0000 7f782a915640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f782410c6a0 0x7f78241a7bb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7818009fd0 tx=0x7f781800b870 comp rx=0 tx=0).stop 2024-04-21T08:46:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.920+0000 7f782a915640 1 -- 172.21.15.7:0/1624800711 shutdown_connections 2024-04-21T08:46:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.920+0000 7f782a915640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f780c0a4100 0x7f780c0a65c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.920+0000 7f782a915640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f782411b740 0x7f78241abf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.920+0000 7f782a915640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f782410c6a0 0x7f78241a7bb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.920+0000 7f782a915640 1 --2- 172.21.15.7:0/1624800711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7824109520 0x7f78241a7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.920+0000 7f782a915640 1 -- 172.21.15.7:0/1624800711 >> 172.21.15.7:0/1624800711 conn(0x7f782406d4c0 msgr2=0x7f782410a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.920+0000 7f782a915640 1 -- 172.21.15.7:0/1624800711 shutdown_connections 2024-04-21T08:46:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:18.920+0000 7f782a915640 1 -- 172.21.15.7:0/1624800711 wait complete. 2024-04-21T08:46:19.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:18 smithi007 ceph-mon[28477]: pgmap v52: 129 pgs: 129 active+clean; 201 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 1.50k op/s 2024-04-21T08:46:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:18 smithi016 ceph-mon[31259]: pgmap v52: 129 pgs: 129 active+clean; 201 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 1.50k op/s 2024-04-21T08:46:19.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:18 smithi077 ceph-mon[31615]: pgmap v52: 129 pgs: 129 active+clean; 201 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 1.50k op/s 2024-04-21T08:46:21.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:20 smithi007 ceph-mon[28477]: pgmap v53: 129 pgs: 129 active+clean; 201 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 1.50k op/s 2024-04-21T08:46:21.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:46:21.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:46:21.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.280+0000 7fa7074c1640 1 -- 172.21.15.7:0/2567442478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa700069f10 msgr2=0x7fa700106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.280+0000 7fa7074c1640 1 --2- 172.21.15.7:0/2567442478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa700069f10 0x7fa700106bc0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009900 tx=0x7fa6f002bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:21.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.280+0000 7fa7074c1640 1 -- 172.21.15.7:0/2567442478 shutdown_connections 2024-04-21T08:46:21.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.280+0000 7fa7074c1640 1 --2- 172.21.15.7:0/2567442478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7001072a0 0x7fa70010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.280+0000 7fa7074c1640 1 --2- 172.21.15.7:0/2567442478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa700069f10 0x7fa700106bc0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.280+0000 7fa7074c1640 1 --2- 172.21.15.7:0/2567442478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa700069560 0x7fa700069940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.280+0000 7fa7074c1640 1 -- 172.21.15.7:0/2567442478 >> 172.21.15.7:0/2567442478 conn(0x7fa700077fa0 msgr2=0x7fa7000783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.280+0000 7fa7074c1640 1 -- 172.21.15.7:0/2567442478 shutdown_connections 2024-04-21T08:46:21.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.280+0000 7fa7074c1640 1 -- 172.21.15.7:0/2567442478 wait complete. 2024-04-21T08:46:21.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.280+0000 7fa7074c1640 1 Processor -- start 2024-04-21T08:46:21.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa7074c1640 1 -- start start 2024-04-21T08:46:21.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa7074c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa700069560 0x7fa70019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa7074c1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7001072a0 0x7fa70019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa7074c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700198590 0x7fa700198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa7074c1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001121b0 con 0x7fa7001072a0 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa7074c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa700112030 con 0x7fa700198590 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa7074c1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa700112330 con 0x7fa700069560 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa704a5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7001072a0 0x7fa70019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa704a5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7001072a0 0x7fa70019e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50274/0 (socket says 172.21.15.7:50274) 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa704a5c640 1 -- 172.21.15.7:0/633924000 learned_addr learned my addr 172.21.15.7:0/633924000 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa705a5e640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700198590 0x7fa700198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa70525d640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa700069560 0x7fa70019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa704a5c640 1 -- 172.21.15.7:0/633924000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa700069560 msgr2=0x7fa70019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa704a5c640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa700069560 0x7fa70019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa704a5c640 1 -- 172.21.15.7:0/633924000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700198590 msgr2=0x7fa700198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa704a5c640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700198590 0x7fa700198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa704a5c640 1 -- 172.21.15.7:0/633924000 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7001990b0 con 0x7fa7001072a0 2024-04-21T08:46:21.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa705a5e640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700198590 0x7fa700198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:21.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.281+0000 7fa704a5c640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7001072a0 0x7fa70019e920 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0005ab0 tx=0x7fa6f00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.282+0000 7fa6f67fc640 1 -- 172.21.15.7:0/633924000 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0043020 con 0x7fa7001072a0 2024-04-21T08:46:21.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.282+0000 7fa6f67fc640 1 -- 172.21.15.7:0/633924000 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6f002cb70 con 0x7fa7001072a0 2024-04-21T08:46:21.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.282+0000 7fa7074c1640 1 -- 172.21.15.7:0/633924000 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa700199340 con 0x7fa7001072a0 2024-04-21T08:46:21.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.282+0000 7fa6f67fc640 1 -- 172.21.15.7:0/633924000 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0003ac0 con 0x7fa7001072a0 2024-04-21T08:46:21.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.282+0000 7fa7074c1640 1 -- 172.21.15.7:0/633924000 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7001a5590 con 0x7fa7001072a0 2024-04-21T08:46:21.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.284+0000 7fa6f67fc640 1 -- 172.21.15.7:0/633924000 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa6f0046020 con 0x7fa7001072a0 2024-04-21T08:46:21.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa70a194740 1 Processor -- start 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa70a194740 1 -- start start 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa6f67fc640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6ec0a3d20 0x7fa6ec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa70a194740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677776960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa70a194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776ea0 0x56167777f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa70a194740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 0x561677781e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa70a194740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5616776db1e0 con 0x56167777fa80 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa70a194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561677678390 con 0x561677776ea0 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa70a194740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5616776d71a0 con 0x561677776580 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa6f67fc640 1 -- 172.21.15.7:0/633924000 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa6f00f2b70 con 0x7fa7001072a0 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa704a5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776ea0 0x56167777f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa705a5e640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6ec0a3d20 0x7fa6ec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa704a5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776ea0 0x56167777f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:43858/0 (socket says 172.21.15.7:43858) 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa704a5c640 1 -- 172.21.15.7:0/1058044943 learned_addr learned my addr 172.21.15.7:0/1058044943 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa70525d640 1 --2- 172.21.15.7:0/1058044943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677776960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa705a5e640 1 --2- 172.21.15.7:0/1058044943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 0x561677781e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa704a5c640 1 -- 172.21.15.7:0/1058044943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 msgr2=0x561677776960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa704a5c640 1 --2- 172.21.15.7:0/1058044943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677776960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa704a5c640 1 -- 172.21.15.7:0/1058044943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 msgr2=0x561677781e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa704a5c640 1 --2- 172.21.15.7:0/1058044943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 0x561677781e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.285+0000 7fa704a5c640 1 -- 172.21.15.7:0/1058044943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616774b77d0 con 0x561677776ea0 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.286+0000 7fa705a5e640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6ec0a3d20 0x7fa6ec0a61e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8009fd0 tx=0x7fa6f80116b0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.286+0000 7fa704a5c640 1 --2- 172.21.15.7:0/1058044943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776ea0 0x56167777f540 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa6f00f0040 tx=0x7fa6f0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c4f5b8caacab0fc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.286+0000 7fa6eb7fe640 1 -- 172.21.15.7:0/1058044943 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f00e0040 con 0x561677776ea0 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.286+0000 7fa6eb7fe640 1 -- 172.21.15.7:0/1058044943 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6f00bef30 con 0x561677776ea0 2024-04-21T08:46:21.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.286+0000 7fa6eb7fe640 1 -- 172.21.15.7:0/1058044943 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f00a90e0 con 0x561677776ea0 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.286+0000 7fa70a194740 1 -- 172.21.15.7:0/1058044943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776ea0 msgr2=0x56167777f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 --2- 172.21.15.7:0/1058044943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776ea0 0x56167777f540 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa6f00f0040 tx=0x7fa6f0009260 comp rx=0 tx=0).stop 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 -- 172.21.15.7:0/1058044943 shutdown_connections 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 --2- 172.21.15.7:0/1058044943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 0x561677781e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 --2- 172.21.15.7:0/1058044943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776ea0 0x56167777f540 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 --2- 172.21.15.7:0/1058044943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677776960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 -- 172.21.15.7:0/1058044943 >> 172.21.15.7:0/1058044943 conn(0x561677770cb0 msgr2=0x561677771090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 -- 172.21.15.7:0/1058044943 shutdown_connections 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 -- 172.21.15.7:0/1058044943 wait complete. 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 -- start start 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 Processor -- start 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677789a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56167777fa80 0x561677783e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561677782630 con 0x561677776ea0 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5616776942f0 con 0x56167777fa80 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.287+0000 7fa70a194740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5616777827b0 con 0x561677776580 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa70a194740 10 client.0 _ll_register_callbacks cb 0x5616776a1580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa704a5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa704a5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50294/0 (socket says 172.21.15.7:50294) 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa704a5c640 1 -- 172.21.15.7:0/1892544541 learned_addr learned my addr 172.21.15.7:0/1892544541 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa705a5e640 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56167777fa80 0x561677783e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa70525d640 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677789a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa704a5c640 1 -- 172.21.15.7:0/1892544541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 msgr2=0x561677789a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa704a5c640 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677789a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa704a5c640 1 -- 172.21.15.7:0/1892544541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56167777fa80 msgr2=0x561677783e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa704a5c640 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56167777fa80 0x561677783e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa704a5c640 1 -- 172.21.15.7:0/1892544541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616773ed3f0 con 0x561677776ea0 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa70525d640 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677789a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa704a5c640 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789f70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa6f00bac90 tx=0x7fa6f00f0380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e035280202e5d8e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa6e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/1892544541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f00be830 con 0x561677776ea0 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/1892544541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa6f0003c60 con 0x561677776ea0 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa70a194740 1 -- 172.21.15.7:0/1892544541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6f00bef30 con 0x561677776ea0 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/1892544541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa6f00dc050 con 0x561677776ea0 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa70a194740 10 client.16086 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.288+0000 7fa70a194740 1 -- 172.21.15.7:0/1892544541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5616773ad0f0 con 0x561677776ea0 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.289+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/1892544541 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6f00bef30 con 0x561677776ea0 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.289+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/1892544541 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa6f00bea90 con 0x561677776ea0 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.289+0000 7fa70a194740 10 client.16086 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:21.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.289+0000 7fa70a194740 10 client.16086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:21.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.289+0000 7fa70a194740 20 client.16086 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:21.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.289+0000 7fa70a194740 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561677793ab0 0x561677795ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.289+0000 7fa70a194740 4 client.16086 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:21.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.289+0000 7fa70a194740 1 -- 172.21.15.7:0/1892544541 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616776e8c00 con 0x561677793ab0 2024-04-21T08:46:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.291+0000 7fa70525d640 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561677793ab0 0x561677795ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.291+0000 7fa70525d640 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561677793ab0 0x561677795ea0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c05001d08833a8d server_cookie=55417c2da77a8bb3 in_seq=0 out_seq=0 2024-04-21T08:46:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.291+0000 7fa6e8ff9640 10 client.16086 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.292+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/1892544541 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616776e8c00 con 0x561677793ab0 2024-04-21T08:46:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.292+0000 7fa6e8ff9640 10 client.16086 handle_command_reply: tid=0 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.292+0000 7fa70a194740 1 client.16086 shutdown 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.292+0000 7fa70a194740 1 -- 172.21.15.7:0/1892544541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 msgr2=0x561677789f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.292+0000 7fa70a194740 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789f70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa6f00bac90 tx=0x7fa6f00f0380 comp rx=0 tx=0).stop 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.293+0000 7fa70a194740 1 -- 172.21.15.7:0/1892544541 shutdown_connections 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.293+0000 7fa70a194740 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561677793ab0 0x561677795ea0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.293+0000 7fa70a194740 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56167777fa80 0x561677783e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.293+0000 7fa70a194740 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789f70 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.293+0000 7fa70a194740 1 --2- 172.21.15.7:0/1892544541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677789a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.293+0000 7fa6e8ff9640 0 client.16086 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.293+0000 7fa70a194740 1 -- 172.21.15.7:0/1892544541 >> 172.21.15.7:0/1892544541 conn(0x561677770cb0 msgr2=0x561677772a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.293+0000 7fa70a194740 1 -- 172.21.15.7:0/1892544541 shutdown_connections 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.293+0000 7fa70a194740 1 -- 172.21.15.7:0/1892544541 wait complete. 2024-04-21T08:46:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.293+0000 7fa70a194740 20 client.16086 trim_cache size 0 max 16384 2024-04-21T08:46:21.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:20 smithi016 ceph-mon[31259]: pgmap v53: 129 pgs: 129 active+clean; 201 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 1.50k op/s 2024-04-21T08:46:21.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.311+0000 7fa70a194740 1 Processor -- start 2024-04-21T08:46:21.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.311+0000 7fa70a194740 1 -- start start 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa70a194740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677783d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa70a194740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa70a194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56167777fa80 0x56167779b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa70a194740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001ab250 con 0x561677776ea0 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa70a194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001adad0 con 0x56167777fa80 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa70a194740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001ab8c0 con 0x561677776580 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa704a5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa704a5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50298/0 (socket says 172.21.15.7:50298) 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa704a5c640 1 -- 172.21.15.7:0/342621639 learned_addr learned my addr 172.21.15.7:0/342621639 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa705a5e640 1 --2- 172.21.15.7:0/342621639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56167777fa80 0x56167779b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa70525d640 1 --2- 172.21.15.7:0/342621639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677783d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa704a5c640 1 -- 172.21.15.7:0/342621639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 msgr2=0x561677783d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa704a5c640 1 --2- 172.21.15.7:0/342621639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677783d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa704a5c640 1 -- 172.21.15.7:0/342621639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56167777fa80 msgr2=0x56167779b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa704a5c640 1 --2- 172.21.15.7:0/342621639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56167777fa80 0x56167779b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa704a5c640 1 -- 172.21.15.7:0/342621639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561677776cd0 con 0x561677776ea0 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa70525d640 1 --2- 172.21.15.7:0/342621639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677783d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa704a5c640 1 --2- 172.21.15.7:0/342621639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789c90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa6f00f04d0 tx=0x7fa6f0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e231c322d956542 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa6eaffd640 1 -- 172.21.15.7:0/342621639 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f00e0020 con 0x561677776ea0 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa6eaffd640 1 -- 172.21.15.7:0/342621639 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6f00bef30 con 0x561677776ea0 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.312+0000 7fa6eaffd640 1 -- 172.21.15.7:0/342621639 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f00c74a0 con 0x561677776ea0 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.313+0000 7fa70a194740 1 -- 172.21.15.7:0/342621639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 msgr2=0x561677789c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.313+0000 7fa70a194740 1 --2- 172.21.15.7:0/342621639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789c90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa6f00f04d0 tx=0x7fa6f0000bc0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.313+0000 7fa70a194740 1 -- 172.21.15.7:0/342621639 shutdown_connections 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.313+0000 7fa70a194740 1 --2- 172.21.15.7:0/342621639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56167777fa80 0x56167779b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.313+0000 7fa70a194740 1 --2- 172.21.15.7:0/342621639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561677776ea0 0x561677789c90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.313+0000 7fa70a194740 1 --2- 172.21.15.7:0/342621639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776580 0x561677783d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.313+0000 7fa70a194740 1 -- 172.21.15.7:0/342621639 >> 172.21.15.7:0/342621639 conn(0x561677770cb0 msgr2=0x561677789580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.313+0000 7fa70a194740 1 -- 172.21.15.7:0/342621639 shutdown_connections 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.313+0000 7fa70a194740 1 -- 172.21.15.7:0/342621639 wait complete. 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.313+0000 7fa70a194740 1 -- start start 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70a194740 1 Processor -- start 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70a194740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776580 0x5616777902f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70a194740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776ea0 0x56167778d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70a194740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 0x56167778dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70a194740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001abf90 con 0x56167777fa80 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70a194740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001ad400 con 0x561677776580 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70a194740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001ac660 con 0x561677776ea0 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70a194740 10 client.0 _ll_register_callbacks cb 0x7fa6f00c2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa705a5e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 0x56167778dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70525d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776580 0x5616777902f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70525d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776580 0x5616777902f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:43878/0 (socket says 172.21.15.7:43878) 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70525d640 1 -- 172.21.15.7:0/2897193493 learned_addr learned my addr 172.21.15.7:0/2897193493 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa705a5e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 0x56167778dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50302/0 (socket says 172.21.15.7:50302) 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa704a5c640 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776ea0 0x56167778d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70525d640 1 -- 172.21.15.7:0/2897193493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776ea0 msgr2=0x56167778d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70525d640 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776ea0 0x56167778d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70525d640 1 -- 172.21.15.7:0/2897193493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 msgr2=0x56167778dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70525d640 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 0x56167778dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa70525d640 1 -- 172.21.15.7:0/2897193493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56167778e200 con 0x561677776580 2024-04-21T08:46:21.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa705a5e640 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 0x56167778dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:21.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.314+0000 7fa704a5c640 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776ea0 0x56167778d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:21.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa70525d640 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776580 0x5616777902f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x5616777965d0 tx=0x7fa6fc00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=936f5df61b486de2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa6e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:46:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/2897193493 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc01d5b0 con 0x561677776580 2024-04-21T08:46:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa70a194740 1 -- 172.21.15.7:0/2897193493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6f00bef30 con 0x561677776580 2024-04-21T08:46:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/2897193493 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa6fc01d750 con 0x561677776580 2024-04-21T08:46:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/2897193493 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa6fc02d050 con 0x561677776580 2024-04-21T08:46:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa70a194740 10 client.25667 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa70a194740 1 -- 172.21.15.7:0/2897193493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5616777f9dd0 con 0x561677776580 2024-04-21T08:46:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/2897193493 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6f00bef30 con 0x561677776580 2024-04-21T08:46:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/2897193493 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa6f00a90b0 con 0x561677776580 2024-04-21T08:46:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa70a194740 10 client.25667 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa70a194740 10 client.25667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa70a194740 20 client.25667 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.315+0000 7fa70a194740 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5616777fa500 0x5616777fc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.316+0000 7fa70a194740 4 client.25667 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:46:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.316+0000 7fa70a194740 1 -- 172.21.15.7:0/2897193493 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5616776e2a70 con 0x5616777fa500 2024-04-21T08:46:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.316+0000 7fa704a5c640 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5616777fa500 0x5616777fc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.316+0000 7fa704a5c640 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5616777fa500 0x5616777fc920 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7131918895cea74f server_cookie=da522220a33830f9 in_seq=0 out_seq=0 2024-04-21T08:46:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.316+0000 7fa6e8ff9640 10 client.25667 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa6e8ff9640 1 -- 172.21.15.7:0/2897193493 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5616776e2a70 con 0x5616777fa500 2024-04-21T08:46:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa6e8ff9640 10 client.25667 handle_command_reply: tid=0 2024-04-21T08:46:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 1 client.25667 shutdown 2024-04-21T08:46:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 1 -- 172.21.15.7:0/2897193493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776580 msgr2=0x5616777902f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776580 0x5616777902f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x5616777965d0 tx=0x7fa6fc00bea0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 1 -- 172.21.15.7:0/2897193493 shutdown_connections 2024-04-21T08:46:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5616777fa500 0x5616777fc920 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56167777fa80 0x56167778dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561677776ea0 0x56167778d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa6e8ff9640 0 client.25667 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:21.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 1 --2- 172.21.15.7:0/2897193493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561677776580 0x5616777902f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 1 -- 172.21.15.7:0/2897193493 >> 172.21.15.7:0/2897193493 conn(0x561677770cb0 msgr2=0x56167776f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 1 -- 172.21.15.7:0/2897193493 shutdown_connections 2024-04-21T08:46:21.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 1 -- 172.21.15.7:0/2897193493 wait complete. 2024-04-21T08:46:21.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.320+0000 7fa70a194740 20 client.25667 trim_cache size 0 max 16384 2024-04-21T08:46:21.330 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:46:21.330 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:46:21.331 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:46:21.331 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:46:21.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 -- 172.21.15.7:0/633924000 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6ec0a3d20 msgr2=0x7fa6ec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6ec0a3d20 0x7fa6ec0a61e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8009fd0 tx=0x7fa6f80116b0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 -- 172.21.15.7:0/633924000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7001072a0 msgr2=0x7fa70019e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7001072a0 0x7fa70019e920 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0005ab0 tx=0x7fa6f00045e0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 -- 172.21.15.7:0/633924000 shutdown_connections 2024-04-21T08:46:21.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa700198590 0x7fa700198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7001072a0 0x7fa70019e920 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6ec0a3d20 0x7fa6ec0a61e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 --2- 172.21.15.7:0/633924000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa700069560 0x7fa70019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 -- 172.21.15.7:0/633924000 >> 172.21.15.7:0/633924000 conn(0x7fa700077fa0 msgr2=0x7fa7000ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 -- 172.21.15.7:0/633924000 shutdown_connections 2024-04-21T08:46:21.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.321+0000 7fa7074c1640 1 -- 172.21.15.7:0/633924000 wait complete. 2024-04-21T08:46:21.336 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:46:21.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:46:21.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:20 smithi077 ceph-mon[31615]: pgmap v53: 129 pgs: 129 active+clean; 201 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.1 MiB/s wr, 1.50k op/s 2024-04-21T08:46:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.444+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/3850679327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f681021a0 msgr2=0x7f4f68102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.444+0000 7f4f6e1b2640 1 --2- 172.21.15.7:0/3850679327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f681021a0 0x7f4f68102580 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4f58009900 tx=0x7f4f5802bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.444+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/3850679327 shutdown_connections 2024-04-21T08:46:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.444+0000 7f4f6e1b2640 1 --2- 172.21.15.7:0/3850679327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f68107270 0x7f4f6810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.444+0000 7f4f6e1b2640 1 --2- 172.21.15.7:0/3850679327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f68102b50 0x7f4f68106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.444+0000 7f4f6e1b2640 1 --2- 172.21.15.7:0/3850679327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f681021a0 0x7f4f68102580 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.444+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/3850679327 >> 172.21.15.7:0/3850679327 conn(0x7f4f680793e0 msgr2=0x7f4f680797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.444+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/3850679327 shutdown_connections 2024-04-21T08:46:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.444+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/3850679327 wait complete. 2024-04-21T08:46:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.445+0000 7f4f6e1b2640 1 Processor -- start 2024-04-21T08:46:21.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.445+0000 7f4f6e1b2640 1 -- start start 2024-04-21T08:46:21.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.445+0000 7f4f6e1b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f681021a0 0x7f4f681a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.445+0000 7f4f6e1b2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f68107270 0x7f4f681a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.445+0000 7f4f6e1b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f6819a730 0x7f4f6819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.445+0000 7f4f6e1b2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4f681121d0 con 0x7f4f68107270 2024-04-21T08:46:21.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.445+0000 7f4f6e1b2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4f68112050 con 0x7f4f6819a730 2024-04-21T08:46:21.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.445+0000 7f4f6e1b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4f68112350 con 0x7f4f681021a0 2024-04-21T08:46:21.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.445+0000 7f4f66ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f68107270 0x7f4f681a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.445+0000 7f4f67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f6819a730 0x7f4f6819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.445+0000 7f4f677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f681021a0 0x7f4f681a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f66ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f68107270 0x7f4f681a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50328/0 (socket says 172.21.15.7:50328) 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f66ffd640 1 -- 172.21.15.7:0/506656179 learned_addr learned my addr 172.21.15.7:0/506656179 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f6819a730 0x7f4f6819abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:43890/0 (socket says 172.21.15.7:43890) 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f66ffd640 1 -- 172.21.15.7:0/506656179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f681021a0 msgr2=0x7f4f681a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f66ffd640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f681021a0 0x7f4f681a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f66ffd640 1 -- 172.21.15.7:0/506656179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f6819a730 msgr2=0x7f4f6819abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f66ffd640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f6819a730 0x7f4f6819abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f66ffd640 1 -- 172.21.15.7:0/506656179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6819b330 con 0x7f4f68107270 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f67fff640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f6819a730 0x7f4f6819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f66ffd640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f68107270 0x7f4f681a0bb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00d740 tx=0x7f4f5c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f64ff9640 1 -- 172.21.15.7:0/506656179 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c017070 con 0x7f4f68107270 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f64ff9640 1 -- 172.21.15.7:0/506656179 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f5c00de50 con 0x7f4f68107270 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/506656179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f681a7410 con 0x7f4f68107270 2024-04-21T08:46:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f64ff9640 1 -- 172.21.15.7:0/506656179 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c004ce0 con 0x7f4f68107270 2024-04-21T08:46:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.446+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/506656179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f681a77c0 con 0x7f4f68107270 2024-04-21T08:46:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.447+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/506656179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f68103670 con 0x7f4f68107270 2024-04-21T08:46:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.448+0000 7f4f64ff9640 1 -- 172.21.15.7:0/506656179 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f4f5c004e80 con 0x7f4f68107270 2024-04-21T08:46:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.449+0000 7f4f64ff9640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4f500a3d20 0x7f4f500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.449+0000 7f4f677fe640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4f500a3d20 0x7f4f500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.449+0000 7f4f64ff9640 1 -- 172.21.15.7:0/506656179 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f4f5c0d0590 con 0x7f4f68107270 2024-04-21T08:46:21.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.449+0000 7f4f677fe640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4f500a3d20 0x7f4f500a61e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4f58006010 tx=0x7f4f58005e40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.454+0000 7f4f64ff9640 1 -- 172.21.15.7:0/506656179 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f5c09c190 con 0x7f4f68107270 2024-04-21T08:46:21.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.691+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/506656179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f68069250 con 0x7f4f68107270 2024-04-21T08:46:21.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.692+0000 7f4f64ff9640 1 -- 172.21.15.7:0/506656179 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7f4f5c0a0490 con 0x7f4f68107270 2024-04-21T08:46:21.691 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:46:21.691 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:46:21.692 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:46:21.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/506656179 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4f500a3d20 msgr2=0x7f4f500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4f500a3d20 0x7f4f500a61e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4f58006010 tx=0x7f4f58005e40 comp rx=0 tx=0).stop 2024-04-21T08:46:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/506656179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f68107270 msgr2=0x7f4f681a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f68107270 0x7f4f681a0bb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00d740 tx=0x7f4f5c00dc00 comp rx=0 tx=0).stop 2024-04-21T08:46:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/506656179 shutdown_connections 2024-04-21T08:46:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4f6819a730 0x7f4f6819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f68107270 0x7f4f681a0bb0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f4f500a3d20 0x7f4f500a61e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 --2- 172.21.15.7:0/506656179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f681021a0 0x7f4f681a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/506656179 >> 172.21.15.7:0/506656179 conn(0x7f4f680793e0 msgr2=0x7f4f680fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/506656179 shutdown_connections 2024-04-21T08:46:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.694+0000 7f4f6e1b2640 1 -- 172.21.15.7:0/506656179 wait complete. 2024-04-21T08:46:21.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:46:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.813+0000 7f90131f4640 1 -- 172.21.15.7:0/1833081110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900c102350 msgr2=0x7f900c10de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.813+0000 7f90131f4640 1 --2- 172.21.15.7:0/1833081110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900c102350 0x7f900c10de20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009900 tx=0x7f8ffc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.813+0000 7f90131f4640 1 -- 172.21.15.7:0/1833081110 shutdown_connections 2024-04-21T08:46:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.813+0000 7f90131f4640 1 --2- 172.21.15.7:0/1833081110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900c102350 0x7f900c10de20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.813+0000 7f90131f4640 1 --2- 172.21.15.7:0/1833081110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900c101990 0x7f900c101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.813+0000 7f90131f4640 1 --2- 172.21.15.7:0/1833081110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c102d70 0x7f900c103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.813+0000 7f90131f4640 1 -- 172.21.15.7:0/1833081110 >> 172.21.15.7:0/1833081110 conn(0x7f900c0760d0 msgr2=0x7f900c0764e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.813+0000 7f90131f4640 1 -- 172.21.15.7:0/1833081110 shutdown_connections 2024-04-21T08:46:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.813+0000 7f90131f4640 1 -- 172.21.15.7:0/1833081110 wait complete. 2024-04-21T08:46:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f90131f4640 1 Processor -- start 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f90131f4640 1 -- start start 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f90131f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c101990 0x7f900c19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f90131f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900c102350 0x7f900c19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f90131f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900c102d70 0x7f900c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f90131f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f900c1111c0 con 0x7f900c102350 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f90131f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f900c111040 con 0x7f900c102d70 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f90131f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f900c111340 con 0x7f900c101990 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f900bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900c102350 0x7f900c19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f900bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900c102350 0x7f900c19eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50338/0 (socket says 172.21.15.7:50338) 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f900bfff640 1 -- 172.21.15.7:0/2572174358 learned_addr learned my addr 172.21.15.7:0/2572174358 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f9011791640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900c102d70 0x7f900c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f9010f90640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c101990 0x7f900c19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f900bfff640 1 -- 172.21.15.7:0/2572174358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c101990 msgr2=0x7f900c19e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f900bfff640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c101990 0x7f900c19e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f900bfff640 1 -- 172.21.15.7:0/2572174358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900c102d70 msgr2=0x7f900c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f900bfff640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900c102d70 0x7f900c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f900bfff640 1 -- 172.21.15.7:0/2572174358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f900c198ef0 con 0x7f900c102350 2024-04-21T08:46:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.814+0000 7f9011791640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900c102d70 0x7f900c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.815+0000 7f900bfff640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900c102350 0x7f900c19eab0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f900000e9f0 tx=0x7f900000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.815+0000 7f9009ffb640 1 -- 172.21.15.7:0/2572174358 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900000cd80 con 0x7f900c102350 2024-04-21T08:46:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.815+0000 7f9009ffb640 1 -- 172.21.15.7:0/2572174358 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90000108e0 con 0x7f900c102350 2024-04-21T08:46:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.815+0000 7f90131f4640 1 -- 172.21.15.7:0/2572174358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f900c1991e0 con 0x7f900c102350 2024-04-21T08:46:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.815+0000 7f9009ffb640 1 -- 172.21.15.7:0/2572174358 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9000018980 con 0x7f900c102350 2024-04-21T08:46:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.815+0000 7f90131f4640 1 -- 172.21.15.7:0/2572174358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f900c1a54f0 con 0x7f900c102350 2024-04-21T08:46:21.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.817+0000 7f9009ffb640 1 -- 172.21.15.7:0/2572174358 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f9000010430 con 0x7f900c102350 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.817+0000 7f9015deb740 1 Processor -- start 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.817+0000 7f9015deb740 1 -- start start 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9015deb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ecffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9015deb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 0x55597ed08840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9015deb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed08e10 0x55597ed0b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9015deb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55597ec601f0 con 0x55597ed08e10 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9015deb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55597ebfd4b0 con 0x55597ed00290 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9015deb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55597ec5c1b0 con 0x55597ecff8e0 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f900bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed08e10 0x55597ed0b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9010f90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 0x55597ed08840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9010f90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 0x55597ed08840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:43918/0 (socket says 172.21.15.7:43918) 2024-04-21T08:46:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9010f90640 1 -- 172.21.15.7:0/142231706 learned_addr learned my addr 172.21.15.7:0/142231706 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9011791640 1 --2- 172.21.15.7:0/142231706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ecffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9009ffb640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ff40a3d20 0x7f8ff40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9010f90640 1 -- 172.21.15.7:0/142231706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 msgr2=0x55597ecffcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9010f90640 1 --2- 172.21.15.7:0/142231706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ecffcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9010f90640 1 -- 172.21.15.7:0/142231706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed08e10 msgr2=0x55597ed0b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9010f90640 1 --2- 172.21.15.7:0/142231706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed08e10 0x55597ed0b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9010f90640 1 -- 172.21.15.7:0/142231706 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55597ebef1b0 con 0x55597ed00290 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9011791640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ff40a3d20 0x7f8ff40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9009ffb640 1 -- 172.21.15.7:0/2572174358 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f90000d0eb0 con 0x7f900c102350 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.818+0000 7f9010f90640 1 --2- 172.21.15.7:0/142231706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 0x55597ed08840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9004009fd0 tx=0x7f90040118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cd9b071dc390eda server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f8ff2ffd640 1 -- 172.21.15.7:0/142231706 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9004004b20 con 0x55597ed00290 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f8ff2ffd640 1 -- 172.21.15.7:0/142231706 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9004004cc0 con 0x55597ed00290 2024-04-21T08:46:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f8ff2ffd640 1 -- 172.21.15.7:0/142231706 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9004005020 con 0x55597ed00290 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f9015deb740 1 -- 172.21.15.7:0/142231706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 msgr2=0x55597ed08840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f9011791640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ff40a3d20 0x7f8ff40a61e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc005ab0 tx=0x7f8ffc009260 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f9015deb740 1 --2- 172.21.15.7:0/142231706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 0x55597ed08840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9004009fd0 tx=0x7f90040118d0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f9015deb740 1 -- 172.21.15.7:0/142231706 shutdown_connections 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f9015deb740 1 --2- 172.21.15.7:0/142231706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed08e10 0x55597ed0b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f9015deb740 1 --2- 172.21.15.7:0/142231706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 0x55597ed08840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f9015deb740 1 --2- 172.21.15.7:0/142231706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ecffcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f9015deb740 1 -- 172.21.15.7:0/142231706 >> 172.21.15.7:0/142231706 conn(0x55597ecf9fb0 msgr2=0x55597ecfa390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f9015deb740 1 -- 172.21.15.7:0/142231706 shutdown_connections 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.819+0000 7f9015deb740 1 -- 172.21.15.7:0/142231706 wait complete. 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9015deb740 1 -- start start 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9015deb740 1 Processor -- start 2024-04-21T08:46:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9015deb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ed086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9015deb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 0x55597ed05a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9015deb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 0x55597ed05fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9015deb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55597ed0bd80 con 0x55597ed00290 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9015deb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55597ec19500 con 0x55597ed08e10 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9015deb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55597ed0bf00 con 0x55597ecff8e0 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9015deb740 10 client.0 _ll_register_callbacks cb 0x55597ebe7b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f900bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 0x55597ed05a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9011791640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 0x55597ed05fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9011791640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 0x55597ed05fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:43922/0 (socket says 172.21.15.7:43922) 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9011791640 1 -- 172.21.15.7:0/987852637 learned_addr learned my addr 172.21.15.7:0/987852637 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.820+0000 7f9010f90640 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ed086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f9011791640 1 -- 172.21.15.7:0/987852637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 msgr2=0x55597ed086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f9011791640 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ed086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f9011791640 1 -- 172.21.15.7:0/987852637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 msgr2=0x55597ed05a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f9011791640 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 0x55597ed05a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f9011791640 1 -- 172.21.15.7:0/987852637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55597e9723f0 con 0x55597ed08e10 2024-04-21T08:46:21.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f900bfff640 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 0x55597ed05a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:21.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f9011791640 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 0x55597ed05fd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc005ec0 tx=0x7f8ffc0058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b41acd50c6d29259 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f8febfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:46:21.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f8febfff640 1 -- 172.21.15.7:0/987852637 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc050070 con 0x55597ed08e10 2024-04-21T08:46:21.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f8febfff640 1 -- 172.21.15.7:0/987852637 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f8ffc003b40 con 0x55597ed08e10 2024-04-21T08:46:21.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f9015deb740 1 -- 172.21.15.7:0/987852637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9004004cc0 con 0x55597ed08e10 2024-04-21T08:46:21.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f8febfff640 1 -- 172.21.15.7:0/987852637 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8ffc05c050 con 0x55597ed08e10 2024-04-21T08:46:21.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f9015deb740 10 client.25685 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:21.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f9015deb740 1 -- 172.21.15.7:0/987852637 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55597ea3c7d0 con 0x55597ed08e10 2024-04-21T08:46:21.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.821+0000 7f8febfff640 1 -- 172.21.15.7:0/987852637 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9004004cc0 con 0x55597ed08e10 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.822+0000 7f8febfff640 1 -- 172.21.15.7:0/987852637 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f8ffc034440 con 0x55597ed08e10 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.822+0000 7f9015deb740 10 client.25685 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.822+0000 7f9015deb740 10 client.25685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.822+0000 7f9015deb740 20 client.25685 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.822+0000 7f9015deb740 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55597ed18f90 0x55597ed1b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.822+0000 7f9015deb740 4 client.25685 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.822+0000 7f9015deb740 1 -- 172.21.15.7:0/987852637 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55597ebff930 con 0x55597ed18f90 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.822+0000 7f9010f90640 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55597ed18f90 0x55597ed1b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.822+0000 7f9010f90640 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55597ed18f90 0x55597ed1b410 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac9bcf4aa01fc947 server_cookie=ccb15c0a6d615cfd in_seq=0 out_seq=0 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.822+0000 7f8febfff640 10 client.25685 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.823+0000 7f8febfff640 1 -- 172.21.15.7:0/987852637 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55597ebff930 con 0x55597ed18f90 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.823+0000 7f8febfff640 10 client.25685 handle_command_reply: tid=0 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.823+0000 7f9015deb740 1 client.25685 shutdown 2024-04-21T08:46:21.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f9015deb740 1 -- 172.21.15.7:0/987852637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 msgr2=0x55597ed05fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f9015deb740 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 0x55597ed05fd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc005ec0 tx=0x7f8ffc0058d0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f9015deb740 1 -- 172.21.15.7:0/987852637 shutdown_connections 2024-04-21T08:46:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f9015deb740 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55597ed18f90 0x55597ed1b410 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f9015deb740 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 0x55597ed05fd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f9015deb740 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 0x55597ed05a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f9015deb740 1 --2- 172.21.15.7:0/987852637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ed086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f8febfff640 0 client.25685 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f9015deb740 1 -- 172.21.15.7:0/987852637 >> 172.21.15.7:0/987852637 conn(0x55597ecf9fb0 msgr2=0x55597ecfbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f9015deb740 1 -- 172.21.15.7:0/987852637 shutdown_connections 2024-04-21T08:46:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f9015deb740 1 -- 172.21.15.7:0/987852637 wait complete. 2024-04-21T08:46:21.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.824+0000 7f9015deb740 20 client.25685 trim_cache size 0 max 16384 2024-04-21T08:46:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.837+0000 7f9015deb740 1 Processor -- start 2024-04-21T08:46:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.837+0000 7f9015deb740 1 -- start start 2024-04-21T08:46:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9015deb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ecff8e0 0x55597ed14ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9015deb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 0x55597ed081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9015deb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ed08e10 0x55597ed086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9015deb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f900c1ab280 con 0x55597ecff8e0 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9015deb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f900c1adb00 con 0x55597ed00290 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9015deb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f900c1ab8f0 con 0x55597ed08e10 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9010f90640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ecff8e0 0x55597ed14ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9010f90640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ecff8e0 0x55597ed14ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50370/0 (socket says 172.21.15.7:50370) 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f900bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 0x55597ed081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9010f90640 1 -- 172.21.15.7:0/3329431942 learned_addr learned my addr 172.21.15.7:0/3329431942 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9011791640 1 --2- 172.21.15.7:0/3329431942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ed08e10 0x55597ed086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9010f90640 1 -- 172.21.15.7:0/3329431942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ed08e10 msgr2=0x55597ed086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9010f90640 1 --2- 172.21.15.7:0/3329431942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ed08e10 0x55597ed086e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9010f90640 1 -- 172.21.15.7:0/3329431942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 msgr2=0x55597ed081a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9010f90640 1 --2- 172.21.15.7:0/3329431942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 0x55597ed081a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9010f90640 1 -- 172.21.15.7:0/3329431942 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55597ed0b6c0 con 0x55597ecff8e0 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f900bfff640 1 --2- 172.21.15.7:0/3329431942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 0x55597ed081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f9010f90640 1 --2- 172.21.15.7:0/3329431942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ecff8e0 0x55597ed14ca0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9004014040 tx=0x7f9004004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe521fb78a2fb59 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f8ff27fc640 1 -- 172.21.15.7:0/3329431942 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9004031070 con 0x55597ecff8e0 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f8ff27fc640 1 -- 172.21.15.7:0/3329431942 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90040048d0 con 0x55597ecff8e0 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.838+0000 7f8ff27fc640 1 -- 172.21.15.7:0/3329431942 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9004025670 con 0x55597ecff8e0 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.839+0000 7f9015deb740 1 -- 172.21.15.7:0/3329431942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ecff8e0 msgr2=0x55597ed14ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.839+0000 7f9015deb740 1 --2- 172.21.15.7:0/3329431942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ecff8e0 0x55597ed14ca0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9004014040 tx=0x7f9004004760 comp rx=0 tx=0).stop 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.839+0000 7f9015deb740 1 -- 172.21.15.7:0/3329431942 shutdown_connections 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.839+0000 7f9015deb740 1 --2- 172.21.15.7:0/3329431942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ed08e10 0x55597ed086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.839+0000 7f9015deb740 1 --2- 172.21.15.7:0/3329431942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed00290 0x55597ed081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.839+0000 7f9015deb740 1 --2- 172.21.15.7:0/3329431942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ecff8e0 0x55597ed14ca0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.839+0000 7f9015deb740 1 -- 172.21.15.7:0/3329431942 >> 172.21.15.7:0/3329431942 conn(0x55597ecf9fb0 msgr2=0x55597ecf4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.839+0000 7f9015deb740 1 -- 172.21.15.7:0/3329431942 shutdown_connections 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.839+0000 7f9015deb740 1 -- 172.21.15.7:0/3329431942 wait complete. 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.839+0000 7f9015deb740 1 -- start start 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.839+0000 7f9015deb740 1 Processor -- start 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f9015deb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ed11620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f9015deb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 0x55597ed0c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f9015deb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 0x55597ed0cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f9015deb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f900c1abfc0 con 0x55597ed00290 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f9015deb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f900c1ad430 con 0x55597ed08e10 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f9015deb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f900c1ac690 con 0x55597ecff8e0 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f9015deb740 10 client.0 _ll_register_callbacks cb 0x7f8ffc009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f900bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 0x55597ed0c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f900bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 0x55597ed0c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50372/0 (socket says 172.21.15.7:50372) 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f900bfff640 1 -- 172.21.15.7:0/2516120750 learned_addr learned my addr 172.21.15.7:0/2516120750 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f9011791640 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 0x55597ed0cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f9010f90640 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ed11620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f900bfff640 1 -- 172.21.15.7:0/2516120750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 msgr2=0x55597ed11620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f900bfff640 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ed11620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f900bfff640 1 -- 172.21.15.7:0/2516120750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 msgr2=0x55597ed0cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f900bfff640 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 0x55597ed0cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f900bfff640 1 -- 172.21.15.7:0/2516120750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55597ed0d760 con 0x55597ed00290 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f900bfff640 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 0x55597ed0c990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f90000ce810 tx=0x7f90000d10b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7614c7744414068b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.840+0000 7f8febfff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f8febfff640 1 -- 172.21.15.7:0/2516120750 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90000d4db0 con 0x55597ed00290 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f8febfff640 1 -- 172.21.15.7:0/2516120750 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f900009bf60 con 0x55597ed00290 2024-04-21T08:46:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f9015deb740 1 -- 172.21.15.7:0/2516120750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90040048d0 con 0x55597ed00290 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f8febfff640 1 -- 172.21.15.7:0/2516120750 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9000010720 con 0x55597ed00290 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f9015deb740 10 client.16116 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f9015deb740 1 -- 172.21.15.7:0/2516120750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55597ed7f6d0 con 0x55597ed00290 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f8febfff640 1 -- 172.21.15.7:0/2516120750 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90040048d0 con 0x55597ed00290 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f8febfff640 1 -- 172.21.15.7:0/2516120750 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f9000098860 con 0x55597ed00290 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f9015deb740 10 client.16116 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f9015deb740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f9015deb740 20 client.16116 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f9015deb740 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55597ed7fe70 0x55597ed82290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f9015deb740 4 client.16116 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f9015deb740 1 -- 172.21.15.7:0/2516120750 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55597ec6dc10 con 0x55597ed7fe70 2024-04-21T08:46:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.841+0000 7f9010f90640 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55597ed7fe70 0x55597ed82290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.842+0000 7f8febfff640 10 client.16116 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.842+0000 7f9010f90640 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55597ed7fe70 0x55597ed82290 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=239ac9db000a1220 server_cookie=e5f0b22422bfa867 in_seq=0 out_seq=0 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.842+0000 7f8febfff640 1 -- 172.21.15.7:0/2516120750 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55597ec6dc10 con 0x55597ed7fe70 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.842+0000 7f8febfff640 10 client.16116 handle_command_reply: tid=0 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.842+0000 7f9015deb740 1 client.16116 shutdown 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f9015deb740 1 -- 172.21.15.7:0/2516120750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 msgr2=0x55597ed0c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f9015deb740 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 0x55597ed0c990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f90000ce810 tx=0x7f90000d10b0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f9015deb740 1 -- 172.21.15.7:0/2516120750 shutdown_connections 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f9015deb740 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55597ed7fe70 0x55597ed82290 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f9015deb740 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55597ed08e10 0x55597ed0cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f9015deb740 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55597ed00290 0x55597ed0c990 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f8febfff640 0 client.16116 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f9015deb740 1 --2- 172.21.15.7:0/2516120750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55597ecff8e0 0x55597ed11620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f9015deb740 1 -- 172.21.15.7:0/2516120750 >> 172.21.15.7:0/2516120750 conn(0x55597ecf9fb0 msgr2=0x55597ecf4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f9015deb740 1 -- 172.21.15.7:0/2516120750 shutdown_connections 2024-04-21T08:46:21.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f9015deb740 1 -- 172.21.15.7:0/2516120750 wait complete. 2024-04-21T08:46:21.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f9015deb740 20 client.16116 trim_cache size 0 max 16384 2024-04-21T08:46:21.849 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:46:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f90131f4640 1 -- 172.21.15.7:0/2572174358 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ff40a3d20 msgr2=0x7f8ff40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.843+0000 7f90131f4640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ff40a3d20 0x7f8ff40a61e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc005ab0 tx=0x7f8ffc009260 comp rx=0 tx=0).stop 2024-04-21T08:46:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.844+0000 7f90131f4640 1 -- 172.21.15.7:0/2572174358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900c102350 msgr2=0x7f900c19eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.844+0000 7f90131f4640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900c102350 0x7f900c19eab0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f900000e9f0 tx=0x7f900000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.844+0000 7f90131f4640 1 -- 172.21.15.7:0/2572174358 shutdown_connections 2024-04-21T08:46:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.844+0000 7f90131f4640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f900c102d70 0x7f900c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.844+0000 7f90131f4640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f900c102350 0x7f900c19eab0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.844+0000 7f90131f4640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8ff40a3d20 0x7f8ff40a61e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.844+0000 7f90131f4640 1 --2- 172.21.15.7:0/2572174358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f900c101990 0x7f900c19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.844+0000 7f90131f4640 1 -- 172.21.15.7:0/2572174358 >> 172.21.15.7:0/2572174358 conn(0x7f900c0760d0 msgr2=0x7f900c106660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.844+0000 7f90131f4640 1 -- 172.21.15.7:0/2572174358 shutdown_connections 2024-04-21T08:46:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:21.844+0000 7f90131f4640 1 -- 172.21.15.7:0/2572174358 wait complete. 2024-04-21T08:46:22.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:21 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/506656179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:22.204 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:21 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/506656179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:22.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:21 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/506656179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:22.854 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:46:22.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:46:22.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:46:22.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.964+0000 7f87eeda4640 1 -- 172.21.15.7:0/1685806832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87e8106160 msgr2=0x7f87e810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:22.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.964+0000 7f87eeda4640 1 --2- 172.21.15.7:0/1685806832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87e8106160 0x7f87e810dd30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009ca0 tx=0x7f87e002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.964+0000 7f87eeda4640 1 -- 172.21.15.7:0/1685806832 shutdown_connections 2024-04-21T08:46:22.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.964+0000 7f87eeda4640 1 --2- 172.21.15.7:0/1685806832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87e8106160 0x7f87e810dd30 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.964+0000 7f87eeda4640 1 --2- 172.21.15.7:0/1685806832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87e8101a40 0x7f87e8105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.964+0000 7f87eeda4640 1 --2- 172.21.15.7:0/1685806832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e8101090 0x7f87e8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.964+0000 7f87eeda4640 1 -- 172.21.15.7:0/1685806832 >> 172.21.15.7:0/1685806832 conn(0x7f87e8078000 msgr2=0x7f87e8078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:22.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.964+0000 7f87eeda4640 1 -- 172.21.15.7:0/1685806832 shutdown_connections 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.964+0000 7f87eeda4640 1 -- 172.21.15.7:0/1685806832 wait complete. 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87eeda4640 1 Processor -- start 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87eeda4640 1 -- start start 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87eeda4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87e8101090 0x7f87e810e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87eeda4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87e8106160 0x7f87e810ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87eeda4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e810f140 0x7f87e81a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87eeda4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81121e0 con 0x7f87e8106160 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87eeda4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87e8112060 con 0x7f87e810f140 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87eeda4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f87e8112360 con 0x7f87e8101090 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87e7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87e8106160 0x7f87e810ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87e7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87e8106160 0x7f87e810ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50392/0 (socket says 172.21.15.7:50392) 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87e7fff640 1 -- 172.21.15.7:0/349419212 learned_addr learned my addr 172.21.15.7:0/349419212 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87ed341640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e810f140 0x7f87e81a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87ecb40640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87e8101090 0x7f87e810e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87e7fff640 1 -- 172.21.15.7:0/349419212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87e8101090 msgr2=0x7f87e810e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:22.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87e7fff640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87e8101090 0x7f87e810e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87e7fff640 1 -- 172.21.15.7:0/349419212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e810f140 msgr2=0x7f87e81a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:22.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87e7fff640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e810f140 0x7f87e81a67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87e7fff640 1 -- 172.21.15.7:0/349419212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e81a6ef0 con 0x7f87e8106160 2024-04-21T08:46:22.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87ed341640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e810f140 0x7f87e81a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:22.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.965+0000 7f87e7fff640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87e8106160 0x7f87e810ec00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f87d400c970 tx=0x7f87d400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:22.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.966+0000 7f87e5ffb640 1 -- 172.21.15.7:0/349419212 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4014070 con 0x7f87e8106160 2024-04-21T08:46:22.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.966+0000 7f87e5ffb640 1 -- 172.21.15.7:0/349419212 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d4004440 con 0x7f87e8106160 2024-04-21T08:46:22.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.966+0000 7f87eeda4640 1 -- 172.21.15.7:0/349419212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e81a71e0 con 0x7f87e8106160 2024-04-21T08:46:22.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.966+0000 7f87e5ffb640 1 -- 172.21.15.7:0/349419212 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4003e50 con 0x7f87e8106160 2024-04-21T08:46:22.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.966+0000 7f87eeda4640 1 -- 172.21.15.7:0/349419212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e81a7780 con 0x7f87e8106160 2024-04-21T08:46:22.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.968+0000 7f87e5ffb640 1 -- 172.21.15.7:0/349419212 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f87d4017070 con 0x7f87e8106160 2024-04-21T08:46:22.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.968+0000 7f87f1998740 1 Processor -- start 2024-04-21T08:46:22.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.968+0000 7f87f1998740 1 -- start start 2024-04-21T08:46:22.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87e5ffb640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3f70 0x7f87d80a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87f1998740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c7058e0 0x55852c705cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87f1998740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87f1998740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c70ee10 0x55852c711260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87f1998740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55852c669f00 con 0x55852c706290 2024-04-21T08:46:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87f1998740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55852c607590 con 0x55852c70ee10 2024-04-21T08:46:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87f1998740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55852c665ec0 con 0x55852c7058e0 2024-04-21T08:46:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87ed341640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3f70 0x7f87d80a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87e7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87e7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50408/0 (socket says 172.21.15.7:50408) 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87e5ffb640 1 -- 172.21.15.7:0/349419212 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f87d40d05f0 con 0x7f87e8106160 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87e7fff640 1 -- 172.21.15.7:0/3920539635 learned_addr learned my addr 172.21.15.7:0/3920539635 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87ecb40640 1 --2- 172.21.15.7:0/3920539635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c7058e0 0x55852c705cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87ed341640 1 --2- 172.21.15.7:0/3920539635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c70ee10 0x55852c711260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87ecb40640 1 -- 172.21.15.7:0/3920539635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c70ee10 msgr2=0x55852c711260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87ecb40640 1 --2- 172.21.15.7:0/3920539635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c70ee10 0x55852c711260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87ecb40640 1 -- 172.21.15.7:0/3920539635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 msgr2=0x55852c70e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87ecb40640 1 --2- 172.21.15.7:0/3920539635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87ecb40640 1 -- 172.21.15.7:0/3920539635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55852c4447e0 con 0x55852c7058e0 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87e7fff640 1 --2- 172.21.15.7:0/3920539635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87ed341640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3f70 0x7f87d80a6430 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f87e0004af0 tx=0x7f87e0032580 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.969+0000 7f87ecb40640 1 --2- 172.21.15.7:0/3920539635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c7058e0 0x55852c705cc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009c20 tx=0x7f87dc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91bd9e70a8e5bf0e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87d2ffd640 1 -- 172.21.15.7:0/3920539635 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc01e070 con 0x55852c7058e0 2024-04-21T08:46:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87d2ffd640 1 -- 172.21.15.7:0/3920539635 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87dc009dd0 con 0x55852c7058e0 2024-04-21T08:46:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87d2ffd640 1 -- 172.21.15.7:0/3920539635 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc019410 con 0x55852c7058e0 2024-04-21T08:46:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87f1998740 1 -- 172.21.15.7:0/3920539635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c7058e0 msgr2=0x55852c705cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87f1998740 1 --2- 172.21.15.7:0/3920539635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c7058e0 0x55852c705cc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009c20 tx=0x7f87dc011810 comp rx=0 tx=0).stop 2024-04-21T08:46:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87f1998740 1 -- 172.21.15.7:0/3920539635 shutdown_connections 2024-04-21T08:46:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87f1998740 1 --2- 172.21.15.7:0/3920539635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c70ee10 0x55852c711260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87f1998740 1 --2- 172.21.15.7:0/3920539635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87f1998740 1 --2- 172.21.15.7:0/3920539635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c7058e0 0x55852c705cc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87f1998740 1 -- 172.21.15.7:0/3920539635 >> 172.21.15.7:0/3920539635 conn(0x55852c6fffb0 msgr2=0x55852c700390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87f1998740 1 -- 172.21.15.7:0/3920539635 shutdown_connections 2024-04-21T08:46:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.970+0000 7f87f1998740 1 -- 172.21.15.7:0/3920539635 wait complete. 2024-04-21T08:46:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87f1998740 1 -- start start 2024-04-21T08:46:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87f1998740 1 Processor -- start 2024-04-21T08:46:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87f1998740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 0x55852c70be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87f1998740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87f1998740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c70ee10 0x55852c70c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87f1998740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55852c711d80 con 0x55852c706290 2024-04-21T08:46:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87f1998740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55852c623580 con 0x55852c7058e0 2024-04-21T08:46:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87f1998740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55852c711f00 con 0x55852c70ee10 2024-04-21T08:46:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87f1998740 10 client.0 _ll_register_callbacks cb 0x55852c629650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87e7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87ecb40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 0x55852c70be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87e7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50416/0 (socket says 172.21.15.7:50416) 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87ecb40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 0x55852c70be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:43956/0 (socket says 172.21.15.7:43956) 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87e7fff640 1 -- 172.21.15.7:0/1751824822 learned_addr learned my addr 172.21.15.7:0/1751824822 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.971+0000 7f87ed341640 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c70ee10 0x55852c70c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87ecb40640 1 -- 172.21.15.7:0/1751824822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c70ee10 msgr2=0x55852c70c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87ecb40640 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c70ee10 0x55852c70c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87ecb40640 1 -- 172.21.15.7:0/1751824822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 msgr2=0x55852c70c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87ecb40640 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87ecb40640 1 -- 172.21.15.7:0/1751824822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55852c4a4f70 con 0x55852c7058e0 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87e7fff640 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87ecb40640 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 0x55852c70be50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f87dc000c00 tx=0x7f87dc0052f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bb3db470b331597 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87c7fff640 1 -- 172.21.15.7:0/1751824822 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc01e040 con 0x55852c7058e0 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87c7fff640 1 -- 172.21.15.7:0/1751824822 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f87dc0054e0 con 0x55852c7058e0 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87f1998740 1 -- 172.21.15.7:0/1751824822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87dc009dd0 con 0x55852c7058e0 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87c7fff640 1 -- 172.21.15.7:0/1751824822 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f87dc031050 con 0x55852c7058e0 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87f1998740 10 client.25703 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.972+0000 7f87f1998740 1 -- 172.21.15.7:0/1751824822 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55852c33c170 con 0x55852c7058e0 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.973+0000 7f87c7fff640 1 -- 172.21.15.7:0/1751824822 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87dc009dd0 con 0x55852c7058e0 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.973+0000 7f87c7fff640 1 -- 172.21.15.7:0/1751824822 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f87dc02c030 con 0x55852c7058e0 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.973+0000 7f87f1998740 10 client.25703 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.973+0000 7f87f1998740 10 client.25703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.973+0000 7f87f1998740 20 client.25703 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.973+0000 7f87f1998740 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55852c71f2e0 0x55852c7216d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.973+0000 7f87f1998740 4 client.25703 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.973+0000 7f87f1998740 1 -- 172.21.15.7:0/1751824822 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55852c677920 con 0x55852c71f2e0 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.973+0000 7f87e7fff640 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55852c71f2e0 0x55852c7216d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.973+0000 7f87e7fff640 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55852c71f2e0 0x55852c7216d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4b9620d5ac6ab18 server_cookie=ba23e1682aba6dd5 in_seq=0 out_seq=0 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.973+0000 7f87c7fff640 10 client.25703 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.974+0000 7f87c7fff640 1 -- 172.21.15.7:0/1751824822 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55852c677920 con 0x55852c71f2e0 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.974+0000 7f87c7fff640 10 client.25703 handle_command_reply: tid=0 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.974+0000 7f87f1998740 1 client.25703 shutdown 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87f1998740 1 -- 172.21.15.7:0/1751824822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 msgr2=0x55852c70be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87f1998740 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 0x55852c70be50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f87dc000c00 tx=0x7f87dc0052f0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87f1998740 1 -- 172.21.15.7:0/1751824822 shutdown_connections 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87f1998740 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55852c71f2e0 0x55852c7216d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87f1998740 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c70ee10 0x55852c70c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87f1998740 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c706290 0x55852c70c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87f1998740 1 --2- 172.21.15.7:0/1751824822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 0x55852c70be50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87c7fff640 0 client.25703 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:22.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87f1998740 1 -- 172.21.15.7:0/1751824822 >> 172.21.15.7:0/1751824822 conn(0x55852c6fffb0 msgr2=0x55852c701d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:22.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87f1998740 1 -- 172.21.15.7:0/1751824822 shutdown_connections 2024-04-21T08:46:22.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87f1998740 1 -- 172.21.15.7:0/1751824822 wait complete. 2024-04-21T08:46:22.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.975+0000 7f87f1998740 20 client.25703 trim_cache size 0 max 16384 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87f1998740 1 Processor -- start 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87f1998740 1 -- start start 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87f1998740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 0x55852c70e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87f1998740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c706290 0x55852c71ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87f1998740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c70ee10 0x55852c6fee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87f1998740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81af680 con 0x55852c70ee10 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87f1998740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81b1f00 con 0x55852c7058e0 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87f1998740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81afcf0 con 0x55852c706290 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87ed341640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c70ee10 0x55852c6fee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87ed341640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c70ee10 0x55852c6fee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50420/0 (socket says 172.21.15.7:50420) 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87ecb40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 0x55852c70e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.994+0000 7f87ed341640 1 -- 172.21.15.7:0/1695976869 learned_addr learned my addr 172.21.15.7:0/1695976869 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87e7fff640 1 --2- 172.21.15.7:0/1695976869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c706290 0x55852c71ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87ed341640 1 -- 172.21.15.7:0/1695976869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c706290 msgr2=0x55852c71ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87ed341640 1 --2- 172.21.15.7:0/1695976869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c706290 0x55852c71ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87ed341640 1 -- 172.21.15.7:0/1695976869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 msgr2=0x55852c70e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87ed341640 1 --2- 172.21.15.7:0/1695976869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 0x55852c70e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87ed341640 1 -- 172.21.15.7:0/1695976869 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55852c70ae20 con 0x55852c70ee10 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87ecb40640 1 --2- 172.21.15.7:0/1695976869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 0x55852c70e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87e7fff640 1 --2- 172.21.15.7:0/1695976869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c706290 0x55852c71ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87ed341640 1 --2- 172.21.15.7:0/1695976869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c70ee10 0x55852c6fee90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009ca0 tx=0x7f87e0047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c3e6167f61d335 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87d27fc640 1 -- 172.21.15.7:0/1695976869 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e0053070 con 0x55852c70ee10 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87d27fc640 1 -- 172.21.15.7:0/1695976869 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87e0003b70 con 0x55852c70ee10 2024-04-21T08:46:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.995+0000 7f87d27fc640 1 -- 172.21.15.7:0/1695976869 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e0039040 con 0x55852c70ee10 2024-04-21T08:46:23.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.996+0000 7f87f1998740 1 -- 172.21.15.7:0/1695976869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c70ee10 msgr2=0x55852c6fee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.996+0000 7f87f1998740 1 --2- 172.21.15.7:0/1695976869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c70ee10 0x55852c6fee90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009ca0 tx=0x7f87e0047000 comp rx=0 tx=0).stop 2024-04-21T08:46:23.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.996+0000 7f87f1998740 1 -- 172.21.15.7:0/1695976869 shutdown_connections 2024-04-21T08:46:23.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.996+0000 7f87f1998740 1 --2- 172.21.15.7:0/1695976869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c70ee10 0x55852c6fee90 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.996+0000 7f87f1998740 1 --2- 172.21.15.7:0/1695976869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c706290 0x55852c71ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.996+0000 7f87f1998740 1 --2- 172.21.15.7:0/1695976869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c7058e0 0x55852c70e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.996+0000 7f87f1998740 1 -- 172.21.15.7:0/1695976869 >> 172.21.15.7:0/1695976869 conn(0x55852c6fffb0 msgr2=0x55852c6fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.996+0000 7f87f1998740 1 -- 172.21.15.7:0/1695976869 shutdown_connections 2024-04-21T08:46:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.996+0000 7f87f1998740 1 -- 172.21.15.7:0/1695976869 wait complete. 2024-04-21T08:46:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.996+0000 7f87f1998740 1 -- start start 2024-04-21T08:46:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.996+0000 7f87f1998740 1 Processor -- start 2024-04-21T08:46:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87f1998740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c7058e0 0x55852c708ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87f1998740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c706290 0x55852c709000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87f1998740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c70ee10 0x55852c709540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87f1998740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81b03c0 con 0x55852c7058e0 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87f1998740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81b1830 con 0x55852c706290 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87f1998740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81b0a90 con 0x55852c70ee10 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87f1998740 10 client.0 _ll_register_callbacks cb 0x7f87dc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87ecb40640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c7058e0 0x55852c708ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87e7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c706290 0x55852c709000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87ecb40640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c7058e0 0x55852c708ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50428/0 (socket says 172.21.15.7:50428) 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87ecb40640 1 -- 172.21.15.7:0/3765245571 learned_addr learned my addr 172.21.15.7:0/3765245571 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87ed341640 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c70ee10 0x55852c709540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87e7fff640 1 -- 172.21.15.7:0/3765245571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c70ee10 msgr2=0x55852c709540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87e7fff640 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c70ee10 0x55852c709540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87e7fff640 1 -- 172.21.15.7:0/3765245571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c7058e0 msgr2=0x55852c708ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87e7fff640 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c7058e0 0x55852c708ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87e7fff640 1 -- 172.21.15.7:0/3765245571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55852c784700 con 0x55852c706290 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87ecb40640 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c7058e0 0x55852c708ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87e7fff640 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c706290 0x55852c709000 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f87d4098b10 tx=0x7f87d401e400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=549644b5d89a5bf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.997+0000 7f87c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87c7fff640 1 -- 172.21.15.7:0/3765245571 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d40a4db0 con 0x55852c706290 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87c7fff640 1 -- 172.21.15.7:0/3765245571 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f87d40035f0 con 0x55852c706290 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87f1998740 1 -- 172.21.15.7:0/3765245571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87e0003b70 con 0x55852c706290 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87c7fff640 1 -- 172.21.15.7:0/3765245571 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f87d401e210 con 0x55852c706290 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87f1998740 10 client.25709 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87f1998740 1 -- 172.21.15.7:0/3765245571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55852c785a00 con 0x55852c706290 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87c7fff640 1 -- 172.21.15.7:0/3765245571 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87e0003b70 con 0x55852c706290 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87c7fff640 1 -- 172.21.15.7:0/3765245571 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f87d40a4920 con 0x55852c706290 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87f1998740 10 client.25709 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87f1998740 10 client.25709 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87f1998740 20 client.25709 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87f1998740 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55852c786180 0x55852c7885a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87f1998740 4 client.25709 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87f1998740 1 -- 172.21.15.7:0/3765245571 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55852c671790 con 0x55852c786180 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.998+0000 7f87ecb40640 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55852c786180 0x55852c7885a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.999+0000 7f87c7fff640 10 client.25709 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:22.999+0000 7f87ecb40640 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55852c786180 0x55852c7885a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42047d35615f9aa2 server_cookie=c5147eac74c23e02 in_seq=0 out_seq=0 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87c7fff640 1 -- 172.21.15.7:0/3765245571 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55852c671790 con 0x55852c786180 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87c7fff640 10 client.25709 handle_command_reply: tid=0 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 1 client.25709 shutdown 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 1 -- 172.21.15.7:0/3765245571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c706290 msgr2=0x55852c709000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c706290 0x55852c709000 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f87d4098b10 tx=0x7f87d401e400 comp rx=0 tx=0).stop 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 1 -- 172.21.15.7:0/3765245571 shutdown_connections 2024-04-21T08:46:23.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55852c786180 0x55852c7885a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55852c70ee10 0x55852c709540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55852c706290 0x55852c709000 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87c7fff640 0 client.25709 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 1 --2- 172.21.15.7:0/3765245571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55852c7058e0 0x55852c708ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 1 -- 172.21.15.7:0/3765245571 >> 172.21.15.7:0/3765245571 conn(0x55852c6fffb0 msgr2=0x55852c6ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 1 -- 172.21.15.7:0/3765245571 shutdown_connections 2024-04-21T08:46:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 1 -- 172.21.15.7:0/3765245571 wait complete. 2024-04-21T08:46:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.000+0000 7f87f1998740 20 client.25709 trim_cache size 0 max 16384 2024-04-21T08:46:23.006 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:46:23.007 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:46:23.007 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "459cc2b3-4fe4-4832-a407-e5d53a169cd8", 2024-04-21T08:46:23.007 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:46:23.007 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 -- 172.21.15.7:0/349419212 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3f70 msgr2=0x7f87d80a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3f70 0x7f87d80a6430 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f87e0004af0 tx=0x7f87e0032580 comp rx=0 tx=0).stop 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 -- 172.21.15.7:0/349419212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87e8106160 msgr2=0x7f87e810ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87e8106160 0x7f87e810ec00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f87d400c970 tx=0x7f87d400ce30 comp rx=0 tx=0).stop 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 -- 172.21.15.7:0/349419212 shutdown_connections 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87e810f140 0x7f87e81a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87e8106160 0x7f87e810ec00 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3f70 0x7f87d80a6430 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 --2- 172.21.15.7:0/349419212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87e8101090 0x7f87e810e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 -- 172.21.15.7:0/349419212 >> 172.21.15.7:0/349419212 conn(0x7f87e8078000 msgr2=0x7f87e80feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 -- 172.21.15.7:0/349419212 shutdown_connections 2024-04-21T08:46:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.001+0000 7f87eeda4640 1 -- 172.21.15.7:0/349419212 wait complete. 2024-04-21T08:46:23.012 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:46:23.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:46:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.121+0000 7f672b25b640 1 -- 172.21.15.7:0/2419230705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67241021b0 msgr2=0x7f6724102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.121+0000 7f672b25b640 1 --2- 172.21.15.7:0/2419230705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67241021b0 0x7f6724102590 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6718009900 tx=0x7f671802bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.121+0000 7f672b25b640 1 -- 172.21.15.7:0/2419230705 shutdown_connections 2024-04-21T08:46:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.121+0000 7f672b25b640 1 --2- 172.21.15.7:0/2419230705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6724107280 0x7f672410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.121+0000 7f672b25b640 1 --2- 172.21.15.7:0/2419230705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6724102b60 0x7f6724106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.121+0000 7f672b25b640 1 --2- 172.21.15.7:0/2419230705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67241021b0 0x7f6724102590 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.121+0000 7f672b25b640 1 -- 172.21.15.7:0/2419230705 >> 172.21.15.7:0/2419230705 conn(0x7f67240fd9d0 msgr2=0x7f67240ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.121+0000 7f672b25b640 1 -- 172.21.15.7:0/2419230705 shutdown_connections 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.121+0000 7f672b25b640 1 -- 172.21.15.7:0/2419230705 wait complete. 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f672b25b640 1 Processor -- start 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f672b25b640 1 -- start start 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f672b25b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67241021b0 0x7f67241a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f672b25b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6724107280 0x7f67241a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f672b25b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f672419a770 0x7f672419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f672b25b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6724112210 con 0x7f67241021b0 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f672b25b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6724112090 con 0x7f6724107280 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f672b25b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6724112390 con 0x7f672419a770 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f6728ff7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67241021b0 0x7f67241a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f6723fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6724107280 0x7f67241a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f6728ff7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67241021b0 0x7f67241a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50450/0 (socket says 172.21.15.7:50450) 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f6728ff7640 1 -- 172.21.15.7:0/4255373184 learned_addr learned my addr 172.21.15.7:0/4255373184 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.122+0000 7f67297f8640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f672419a770 0x7f672419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.123+0000 7f6728ff7640 1 -- 172.21.15.7:0/4255373184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f672419a770 msgr2=0x7f672419abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.123+0000 7f6728ff7640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f672419a770 0x7f672419abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.123+0000 7f6728ff7640 1 -- 172.21.15.7:0/4255373184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6724107280 msgr2=0x7f67241a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.123+0000 7f6728ff7640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6724107280 0x7f67241a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.123+0000 7f6728ff7640 1 -- 172.21.15.7:0/4255373184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672419b370 con 0x7f67241021b0 2024-04-21T08:46:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.123+0000 7f6728ff7640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67241021b0 0x7f67241a06b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6718000c00 tx=0x7f671802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.123+0000 7f6721ffb640 1 -- 172.21.15.7:0/4255373184 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718043020 con 0x7f67241021b0 2024-04-21T08:46:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.123+0000 7f6721ffb640 1 -- 172.21.15.7:0/4255373184 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f671802cbb0 con 0x7f67241021b0 2024-04-21T08:46:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.123+0000 7f672b25b640 1 -- 172.21.15.7:0/4255373184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672419b600 con 0x7f67241021b0 2024-04-21T08:46:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.123+0000 7f6721ffb640 1 -- 172.21.15.7:0/4255373184 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718003ac0 con 0x7f67241021b0 2024-04-21T08:46:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.123+0000 7f672b25b640 1 -- 172.21.15.7:0/4255373184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67241a77a0 con 0x7f67241021b0 2024-04-21T08:46:23.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f6721ffb640 1 -- 172.21.15.7:0/4255373184 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f6718046020 con 0x7f67241021b0 2024-04-21T08:46:23.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f672de45740 1 Processor -- start 2024-04-21T08:46:23.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f672de45740 1 -- start start 2024-04-21T08:46:23.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f672de45740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564056554580 0x564056554960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f672de45740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f672de45740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56405655da80 0x56405655fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f672de45740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5640564b91e0 con 0x564056554ea0 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f672de45740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564056456390 con 0x56405655da80 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f672de45740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5640564b51a0 con 0x564056554580 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f6728ff7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f6728ff7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50452/0 (socket says 172.21.15.7:50452) 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f6728ff7640 1 -- 172.21.15.7:0/2057416116 learned_addr learned my addr 172.21.15.7:0/2057416116 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f6723fff640 1 --2- 172.21.15.7:0/2057416116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56405655da80 0x56405655fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f6721ffb640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67140a3d20 0x7f67140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f67297f8640 1 --2- 172.21.15.7:0/2057416116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564056554580 0x564056554960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f67297f8640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67140a3d20 0x7f67140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f6728ff7640 1 -- 172.21.15.7:0/2057416116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564056554580 msgr2=0x564056554960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f6728ff7640 1 --2- 172.21.15.7:0/2057416116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564056554580 0x564056554960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f6728ff7640 1 -- 172.21.15.7:0/2057416116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56405655da80 msgr2=0x56405655fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:23.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.126+0000 7f6728ff7640 1 --2- 172.21.15.7:0/2057416116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56405655da80 0x56405655fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.127+0000 7f6721ffb640 1 -- 172.21.15.7:0/4255373184 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f67180f2b90 con 0x7f67241021b0 2024-04-21T08:46:23.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.127+0000 7f6728ff7640 1 -- 172.21.15.7:0/2057416116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640562957d0 con 0x564056554ea0 2024-04-21T08:46:23.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.127+0000 7f6723fff640 1 --2- 172.21.15.7:0/2057416116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56405655da80 0x56405655fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:23.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.127+0000 7f6728ff7640 1 --2- 172.21.15.7:0/2057416116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655d540 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f67180be550 tx=0x7f6718009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef9bc1bff7501f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:23.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.127+0000 7f67297f8640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67140a3d20 0x7f67140a61e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f671c005fd0 tx=0x7f671c009500 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:23.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.127+0000 7f670effd640 1 -- 172.21.15.7:0/2057416116 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67180be1b0 con 0x564056554ea0 2024-04-21T08:46:23.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.127+0000 7f670effd640 1 -- 172.21.15.7:0/2057416116 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f671807ae50 con 0x564056554ea0 2024-04-21T08:46:23.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.127+0000 7f670effd640 1 -- 172.21.15.7:0/2057416116 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67180c2c00 con 0x564056554ea0 2024-04-21T08:46:23.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.128+0000 7f672de45740 1 -- 172.21.15.7:0/2057416116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 msgr2=0x56405655d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.128+0000 7f672de45740 1 --2- 172.21.15.7:0/2057416116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655d540 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f67180be550 tx=0x7f6718009260 comp rx=0 tx=0).stop 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.128+0000 7f672de45740 1 -- 172.21.15.7:0/2057416116 shutdown_connections 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.128+0000 7f672de45740 1 --2- 172.21.15.7:0/2057416116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56405655da80 0x56405655fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.128+0000 7f672de45740 1 --2- 172.21.15.7:0/2057416116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655d540 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.128+0000 7f672de45740 1 --2- 172.21.15.7:0/2057416116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564056554580 0x564056554960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.128+0000 7f672de45740 1 -- 172.21.15.7:0/2057416116 >> 172.21.15.7:0/2057416116 conn(0x56405654ecb0 msgr2=0x56405654f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.128+0000 7f672de45740 1 -- 172.21.15.7:0/2057416116 shutdown_connections 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.128+0000 7f672de45740 1 -- 172.21.15.7:0/2057416116 wait complete. 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.128+0000 7f672de45740 1 -- start start 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.128+0000 7f672de45740 1 Processor -- start 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f672de45740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554580 0x56405655aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f672de45740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f672de45740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56405655da80 0x56405655b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f672de45740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564056560630 con 0x564056554ea0 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f672de45740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5640564722f0 con 0x564056554580 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f672de45740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5640565607b0 con 0x56405655da80 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f672de45740 10 client.0 _ll_register_callbacks cb 0x56405647f580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6723fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6723fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50468/0 (socket says 172.21.15.7:50468) 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6728ff7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554580 0x56405655aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6723fff640 1 -- 172.21.15.7:0/1568293087 learned_addr learned my addr 172.21.15.7:0/1568293087 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f67297f8640 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56405655da80 0x56405655b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6723fff640 1 -- 172.21.15.7:0/1568293087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56405655da80 msgr2=0x56405655b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6723fff640 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56405655da80 0x56405655b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6723fff640 1 -- 172.21.15.7:0/1568293087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554580 msgr2=0x56405655aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6723fff640 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554580 0x56405655aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6723fff640 1 -- 172.21.15.7:0/1568293087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640561cb3f0 con 0x564056554ea0 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6728ff7640 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554580 0x56405655aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6723fff640 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655afe0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f671000bad0 tx=0x7f671000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0cf0acbee61f376 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6703fff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.129+0000 7f6703fff640 1 -- 172.21.15.7:0/1568293087 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6710003d60 con 0x564056554ea0 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f6703fff640 1 -- 172.21.15.7:0/1568293087 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f6710005020 con 0x564056554ea0 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f672de45740 1 -- 172.21.15.7:0/1568293087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f671807ae50 con 0x564056554ea0 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f6703fff640 1 -- 172.21.15.7:0/1568293087 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6710005220 con 0x564056554ea0 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f672de45740 10 client.16146 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f672de45740 1 -- 172.21.15.7:0/1568293087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56405618b0f0 con 0x564056554ea0 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f6703fff640 1 -- 172.21.15.7:0/1568293087 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f671807ae50 con 0x564056554ea0 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f6703fff640 1 -- 172.21.15.7:0/1568293087 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f671001b470 con 0x564056554ea0 2024-04-21T08:46:23.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f672de45740 10 client.16146 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:23.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f672de45740 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:23.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f672de45740 20 client.16146 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:23.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f672de45740 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56405656d7e0 0x56405656fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f672de45740 4 client.16146 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:23.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f672de45740 1 -- 172.21.15.7:0/1568293087 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640564c6c00 con 0x56405656d7e0 2024-04-21T08:46:23.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.130+0000 7f6728ff7640 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56405656d7e0 0x56405656fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.131+0000 7f6703fff640 10 client.16146 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:23.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.131+0000 7f6728ff7640 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56405656d7e0 0x56405656fc10 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd3d83922e1ecf27 server_cookie=94b52ea3b3400d3 in_seq=0 out_seq=0 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f6703fff640 1 -- 172.21.15.7:0/1568293087 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640564c6c00 con 0x56405656d7e0 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f6703fff640 10 client.16146 handle_command_reply: tid=0 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 1 client.16146 shutdown 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 1 -- 172.21.15.7:0/1568293087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 msgr2=0x56405655afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655afe0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f671000bad0 tx=0x7f671000bed0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 1 -- 172.21.15.7:0/1568293087 shutdown_connections 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56405656d7e0 0x56405656fc10 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56405655da80 0x56405655b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655afe0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f6703fff640 0 client.16146 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 1 --2- 172.21.15.7:0/1568293087 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554580 0x56405655aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 1 -- 172.21.15.7:0/1568293087 >> 172.21.15.7:0/1568293087 conn(0x56405654ecb0 msgr2=0x564056550a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:23.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 1 -- 172.21.15.7:0/1568293087 shutdown_connections 2024-04-21T08:46:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 1 -- 172.21.15.7:0/1568293087 wait complete. 2024-04-21T08:46:23.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.132+0000 7f672de45740 20 client.16146 trim_cache size 0 max 16384 2024-04-21T08:46:23.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f672de45740 1 Processor -- start 2024-04-21T08:46:23.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f672de45740 1 -- start start 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f672de45740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554580 0x564056569410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f672de45740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f672de45740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56405655da80 0x564056575520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f672de45740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f67241af730 con 0x564056554ea0 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f672de45740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f67241b1fb0 con 0x564056554580 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f672de45740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f67241afda0 con 0x56405655da80 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f6723fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f6723fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50474/0 (socket says 172.21.15.7:50474) 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f6723fff640 1 -- 172.21.15.7:0/2216336938 learned_addr learned my addr 172.21.15.7:0/2216336938 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f6728ff7640 1 --2- 172.21.15.7:0/2216336938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554580 0x564056569410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f67297f8640 1 --2- 172.21.15.7:0/2216336938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56405655da80 0x564056575520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f6723fff640 1 -- 172.21.15.7:0/2216336938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56405655da80 msgr2=0x564056575520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f6723fff640 1 --2- 172.21.15.7:0/2216336938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56405655da80 0x564056575520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.151+0000 7f6723fff640 1 -- 172.21.15.7:0/2216336938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554580 msgr2=0x564056569410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.152+0000 7f6723fff640 1 --2- 172.21.15.7:0/2216336938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554580 0x564056569410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.152+0000 7f6723fff640 1 -- 172.21.15.7:0/2216336938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56405655fe70 con 0x564056554ea0 2024-04-21T08:46:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.152+0000 7f6723fff640 1 --2- 172.21.15.7:0/2216336938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655c8d0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6710004340 tx=0x7f671001cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7080c03a67ca028 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:23.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:22 smithi007 ceph-mon[28477]: pgmap v54: 129 pgs: 129 active+clean; 201 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.4 MiB/s wr, 1.31k op/s 2024-04-21T08:46:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.152+0000 7f670e7fc640 1 -- 172.21.15.7:0/2216336938 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671002c070 con 0x564056554ea0 2024-04-21T08:46:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.152+0000 7f670e7fc640 1 -- 172.21.15.7:0/2216336938 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6710003e20 con 0x564056554ea0 2024-04-21T08:46:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.152+0000 7f670e7fc640 1 -- 172.21.15.7:0/2216336938 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6710028900 con 0x564056554ea0 2024-04-21T08:46:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 -- 172.21.15.7:0/2216336938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 msgr2=0x56405655c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 --2- 172.21.15.7:0/2216336938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655c8d0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6710004340 tx=0x7f671001cf40 comp rx=0 tx=0).stop 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 -- 172.21.15.7:0/2216336938 shutdown_connections 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 --2- 172.21.15.7:0/2216336938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56405655da80 0x564056575520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 --2- 172.21.15.7:0/2216336938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564056554ea0 0x56405655c8d0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 --2- 172.21.15.7:0/2216336938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554580 0x564056569410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 -- 172.21.15.7:0/2216336938 >> 172.21.15.7:0/2216336938 conn(0x56405654ecb0 msgr2=0x56405655c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 -- 172.21.15.7:0/2216336938 shutdown_connections 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 -- 172.21.15.7:0/2216336938 wait complete. 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 -- start start 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 Processor -- start 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564056554580 0x564056557b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554ea0 0x5640565580d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56405655da80 0x5640565d2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f67241b0470 con 0x56405655da80 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f67241b18e0 con 0x564056554ea0 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f67241b0b40 con 0x564056554580 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.153+0000 7f672de45740 10 client.0 _ll_register_callbacks cb 0x7f6710004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f67297f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56405655da80 0x5640565d2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f6723fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554ea0 0x5640565580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f67297f8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56405655da80 0x5640565d2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50486/0 (socket says 172.21.15.7:50486) 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f67297f8640 1 -- 172.21.15.7:0/1281213335 learned_addr learned my addr 172.21.15.7:0/1281213335 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f6728ff7640 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564056554580 0x564056557b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f67297f8640 1 -- 172.21.15.7:0/1281213335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564056554580 msgr2=0x564056557b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f67297f8640 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564056554580 0x564056557b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f67297f8640 1 -- 172.21.15.7:0/1281213335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554ea0 msgr2=0x5640565580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f67297f8640 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554ea0 0x5640565580d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f67297f8640 1 -- 172.21.15.7:0/1281213335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640565d32d0 con 0x56405655da80 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f67297f8640 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56405655da80 0x5640565d2d90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f671c010de0 tx=0x7f671c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157d3df84cea6b07 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f6703fff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f6703fff640 1 -- 172.21.15.7:0/1281213335 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c02e060 con 0x56405655da80 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f6703fff640 1 -- 172.21.15.7:0/1281213335 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f671c020bf0 con 0x56405655da80 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f672de45740 1 -- 172.21.15.7:0/1281213335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6710003e20 con 0x56405655da80 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f6703fff640 1 -- 172.21.15.7:0/1281213335 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f671c020df0 con 0x56405655da80 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f672de45740 10 client.16152 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.154+0000 7f672de45740 1 -- 172.21.15.7:0/1281213335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5640565d4500 con 0x56405655da80 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.155+0000 7f6703fff640 1 -- 172.21.15.7:0/1281213335 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6710003e20 con 0x56405655da80 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.155+0000 7f6703fff640 1 -- 172.21.15.7:0/1281213335 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f671c01c410 con 0x56405655da80 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.155+0000 7f672de45740 10 client.16152 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.155+0000 7f672de45740 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.155+0000 7f672de45740 20 client.16152 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.155+0000 7f672de45740 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5640565d4ce0 0x5640565d7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.155+0000 7f672de45740 4 client.16152 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.155+0000 7f672de45740 1 -- 172.21.15.7:0/1281213335 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5640564c0a70 con 0x5640565d4ce0 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.155+0000 7f6728ff7640 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5640565d4ce0 0x5640565d7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.155+0000 7f6703fff640 10 client.16152 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.155+0000 7f6728ff7640 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5640565d4ce0 0x5640565d7100 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af138dae9d2a00b7 server_cookie=39362b3241c41215 in_seq=0 out_seq=0 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.156+0000 7f6703fff640 1 -- 172.21.15.7:0/1281213335 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5640564c0a70 con 0x5640565d4ce0 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.156+0000 7f6703fff640 10 client.16152 handle_command_reply: tid=0 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.156+0000 7f672de45740 1 client.16152 shutdown 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672de45740 1 -- 172.21.15.7:0/1281213335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56405655da80 msgr2=0x5640565d2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672de45740 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56405655da80 0x5640565d2d90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f671c010de0 tx=0x7f671c004a40 comp rx=0 tx=0).stop 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672de45740 1 -- 172.21.15.7:0/1281213335 shutdown_connections 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672de45740 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5640565d4ce0 0x5640565d7100 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672de45740 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56405655da80 0x5640565d2d90 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672de45740 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564056554ea0 0x5640565580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f6703fff640 0 client.16152 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:23.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672de45740 1 --2- 172.21.15.7:0/1281213335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564056554580 0x564056557b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672de45740 1 -- 172.21.15.7:0/1281213335 >> 172.21.15.7:0/1281213335 conn(0x56405654ecb0 msgr2=0x56405655b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:23.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672de45740 1 -- 172.21.15.7:0/1281213335 shutdown_connections 2024-04-21T08:46:23.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672de45740 1 -- 172.21.15.7:0/1281213335 wait complete. 2024-04-21T08:46:23.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672de45740 20 client.16152 trim_cache size 0 max 16384 2024-04-21T08:46:23.162 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:46:23.162 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (101 inodes in the stack)", 2024-04-21T08:46:23.162 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:46:23.162 INFO:teuthology.orchestra.run.smithi007.stdout: "459cc2b3-4fe4-4832-a407-e5d53a169cd8": { 2024-04-21T08:46:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:46:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "459cc2b3-4fe4-4832-a407-e5d53a169cd8", 2024-04-21T08:46:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:46:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:46:23.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:46:23.163 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:46:23.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672b25b640 1 -- 172.21.15.7:0/4255373184 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67140a3d20 msgr2=0x7f67140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672b25b640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67140a3d20 0x7f67140a61e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f671c005fd0 tx=0x7f671c009500 comp rx=0 tx=0).stop 2024-04-21T08:46:23.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672b25b640 1 -- 172.21.15.7:0/4255373184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67241021b0 msgr2=0x7f67241a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:23.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672b25b640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67241021b0 0x7f67241a06b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6718000c00 tx=0x7f671802c6c0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672b25b640 1 -- 172.21.15.7:0/4255373184 shutdown_connections 2024-04-21T08:46:23.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672b25b640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67140a3d20 0x7f67140a61e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672b25b640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f672419a770 0x7f672419abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672b25b640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6724107280 0x7f67241a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.157+0000 7f672b25b640 1 --2- 172.21.15.7:0/4255373184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67241021b0 0x7f67241a06b0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:23.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.158+0000 7f672b25b640 1 -- 172.21.15.7:0/4255373184 >> 172.21.15.7:0/4255373184 conn(0x7f67240fd9d0 msgr2=0x7f67240ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:23.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.158+0000 7f672b25b640 1 -- 172.21.15.7:0/4255373184 shutdown_connections 2024-04-21T08:46:23.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:23.158+0000 7f672b25b640 1 -- 172.21.15.7:0/4255373184 wait complete. 2024-04-21T08:46:23.168 INFO:tasks.cephfs.filesystem:scrub status for tag:459cc2b3-4fe4-4832-a407-e5d53a169cd8 - {'path': '/', 'tag': '459cc2b3-4fe4-4832-a407-e5d53a169cd8', 'options': 'recursive,force'} 2024-04-21T08:46:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:22 smithi016 ceph-mon[31259]: pgmap v54: 129 pgs: 129 active+clean; 201 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.4 MiB/s wr, 1.31k op/s 2024-04-21T08:46:23.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:22 smithi077 ceph-mon[31615]: pgmap v54: 129 pgs: 129 active+clean; 201 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.4 MiB/s wr, 1.31k op/s 2024-04-21T08:46:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:23 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:46:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:23 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:24.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:23 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:46:24.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:23 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:46:24.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:23 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:24.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:23 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:46:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:23 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:46:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:23 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:24.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:23 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:46:25.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:24 smithi016 ceph-mon[31259]: pgmap v55: 129 pgs: 129 active+clean; 208 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.9 MiB/s wr, 1.31k op/s 2024-04-21T08:46:25.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:24 smithi077 ceph-mon[31615]: pgmap v55: 129 pgs: 129 active+clean; 208 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.9 MiB/s wr, 1.31k op/s 2024-04-21T08:46:25.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:24 smithi007 ceph-mon[28477]: pgmap v55: 129 pgs: 129 active+clean; 208 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.9 MiB/s wr, 1.31k op/s 2024-04-21T08:46:27.156 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:26 smithi077 ceph-mon[31615]: pgmap v56: 129 pgs: 129 active+clean; 208 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 1.11k op/s 2024-04-21T08:46:27.156 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:26 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:27.156 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:26 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:46:27.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:26 smithi016 ceph-mon[31259]: pgmap v56: 129 pgs: 129 active+clean; 208 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 1.11k op/s 2024-04-21T08:46:27.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:26 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:27.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:26 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:46:27.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:26 smithi007 ceph-mon[28477]: pgmap v56: 129 pgs: 129 active+clean; 208 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 1.11k op/s 2024-04-21T08:46:27.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:26 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:27.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:26 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:46:28.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:27 smithi016 ceph-mon[31259]: pgmap v57: 129 pgs: 129 active+clean; 212 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 1.95k op/s 2024-04-21T08:46:28.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:27 smithi077 ceph-mon[31615]: pgmap v57: 129 pgs: 129 active+clean; 212 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 1.95k op/s 2024-04-21T08:46:28.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:27 smithi007 ceph-mon[28477]: pgmap v57: 129 pgs: 129 active+clean; 212 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 1.95k op/s 2024-04-21T08:46:30.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:30 smithi077 ceph-mon[31615]: pgmap v58: 129 pgs: 129 active+clean; 212 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 899 op/s 2024-04-21T08:46:30.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:30 smithi007 ceph-mon[28477]: pgmap v58: 129 pgs: 129 active+clean; 212 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 899 op/s 2024-04-21T08:46:30.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:30 smithi016 ceph-mon[31259]: pgmap v58: 129 pgs: 129 active+clean; 212 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 899 op/s 2024-04-21T08:46:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.011+0000 7fd547ee0640 1 -- 172.21.15.7:0/652958391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd54006af70 msgr2=0x7fd54010ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.011+0000 7fd547ee0640 1 --2- 172.21.15.7:0/652958391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd54006af70 0x7fd54010ede0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd538009ca0 tx=0x7fd53802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.011+0000 7fd547ee0640 1 -- 172.21.15.7:0/652958391 shutdown_connections 2024-04-21T08:46:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.011+0000 7fd547ee0640 1 --2- 172.21.15.7:0/652958391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd54006af70 0x7fd54010ede0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.011+0000 7fd547ee0640 1 --2- 172.21.15.7:0/652958391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd54006a5d0 0x7fd54006aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.011+0000 7fd547ee0640 1 --2- 172.21.15.7:0/652958391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd540108040 0x7fd540108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.011+0000 7fd547ee0640 1 -- 172.21.15.7:0/652958391 >> 172.21.15.7:0/652958391 conn(0x7fd540077cd0 msgr2=0x7fd5400780e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.011+0000 7fd547ee0640 1 -- 172.21.15.7:0/652958391 shutdown_connections 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.011+0000 7fd547ee0640 1 -- 172.21.15.7:0/652958391 wait complete. 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd547ee0640 1 Processor -- start 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd547ee0640 1 -- start start 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd547ee0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54006a5d0 0x7fd5401986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd547ee0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd54006af70 0x7fd540198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd547ee0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd540108040 0x7fd54019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd547ee0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401121a0 con 0x7fd54006af70 2024-04-21T08:46:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd547ee0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd540112020 con 0x7fd540108040 2024-04-21T08:46:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd547ee0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd540112320 con 0x7fd54006a5d0 2024-04-21T08:46:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd54547b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd54006af70 0x7fd540198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd54547b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd54006af70 0x7fd540198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38490/0 (socket says 172.21.15.7:38490) 2024-04-21T08:46:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd54547b640 1 -- 172.21.15.7:0/1837069759 learned_addr learned my addr 172.21.15.7:0/1837069759 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd54647d640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd540108040 0x7fd54019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd545c7c640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54006a5d0 0x7fd5401986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd54547b640 1 -- 172.21.15.7:0/1837069759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54006a5d0 msgr2=0x7fd5401986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd54547b640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54006a5d0 0x7fd5401986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd54547b640 1 -- 172.21.15.7:0/1837069759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd540108040 msgr2=0x7fd54019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd54547b640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd540108040 0x7fd54019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd54547b640 1 -- 172.21.15.7:0/1837069759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54019d6d0 con 0x7fd54006af70 2024-04-21T08:46:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.012+0000 7fd54647d640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd540108040 0x7fd54019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.013+0000 7fd54547b640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd54006af70 0x7fd540198c20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd53000e9f0 tx=0x7fd53000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.013+0000 7fd536ffd640 1 -- 172.21.15.7:0/1837069759 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53000cd80 con 0x7fd54006af70 2024-04-21T08:46:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.013+0000 7fd547ee0640 1 -- 172.21.15.7:0/1837069759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd54019d9c0 con 0x7fd54006af70 2024-04-21T08:46:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.013+0000 7fd536ffd640 1 -- 172.21.15.7:0/1837069759 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5300108e0 con 0x7fd54006af70 2024-04-21T08:46:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.013+0000 7fd536ffd640 1 -- 172.21.15.7:0/1837069759 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd530018980 con 0x7fd54006af70 2024-04-21T08:46:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.013+0000 7fd547ee0640 1 -- 172.21.15.7:0/1837069759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5401a5250 con 0x7fd54006af70 2024-04-21T08:46:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd536ffd640 1 -- 172.21.15.7:0/1837069759 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd530010430 con 0x7fd54006af70 2024-04-21T08:46:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd54abad740 1 Processor -- start 2024-04-21T08:46:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd54abad740 1 -- start start 2024-04-21T08:46:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd54abad740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 0x5638c5717620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd54abad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c5717fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd54abad740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57210e0 0x5638c57234d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd54abad740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5638c567c8d0 con 0x5638c57188d0 2024-04-21T08:46:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd54abad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5638c5619e80 con 0x5638c5717b60 2024-04-21T08:46:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd54abad740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5638c5678890 con 0x5638c57210e0 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd54647d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 0x5638c5717620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd536ffd640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd52c0a3d10 0x7fd52c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.016+0000 7fd545c7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c5717fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd545c7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c5717fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:49338/0 (socket says 172.21.15.7:49338) 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd545c7c640 1 -- 172.21.15.7:0/4241164839 learned_addr learned my addr 172.21.15.7:0/4241164839 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd54547b640 1 --2- 172.21.15.7:0/4241164839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57210e0 0x5638c57234d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd54647d640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd52c0a3d10 0x7fd52c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd545c7c640 1 -- 172.21.15.7:0/4241164839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57210e0 msgr2=0x5638c57234d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd545c7c640 1 --2- 172.21.15.7:0/4241164839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57210e0 0x5638c57234d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd545c7c640 1 -- 172.21.15.7:0/4241164839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 msgr2=0x5638c5717620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd545c7c640 1 --2- 172.21.15.7:0/4241164839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 0x5638c5717620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd536ffd640 1 -- 172.21.15.7:0/1837069759 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fd5300d0eb0 con 0x7fd54006af70 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd545c7c640 1 -- 172.21.15.7:0/4241164839 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638c563b320 con 0x5638c5717b60 2024-04-21T08:46:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd545c7c640 1 --2- 172.21.15.7:0/4241164839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c5717fe0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd53c009ca0 tx=0x7fd53c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36f0c7b217ca8b1b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd54647d640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd52c0a3d10 0x7fd52c0a61d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd538004af0 tx=0x7fd53802c0f0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd52bfff640 1 -- 172.21.15.7:0/4241164839 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c01e070 con 0x5638c5717b60 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd52bfff640 1 -- 172.21.15.7:0/4241164839 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd53c019ae0 con 0x5638c5717b60 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.017+0000 7fd52bfff640 1 -- 172.21.15.7:0/4241164839 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c00e9f0 con 0x5638c5717b60 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.018+0000 7fd54abad740 1 -- 172.21.15.7:0/4241164839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 msgr2=0x5638c5717fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.018+0000 7fd54abad740 1 --2- 172.21.15.7:0/4241164839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c5717fe0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd53c009ca0 tx=0x7fd53c011810 comp rx=0 tx=0).stop 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.018+0000 7fd54abad740 1 -- 172.21.15.7:0/4241164839 shutdown_connections 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.018+0000 7fd54abad740 1 --2- 172.21.15.7:0/4241164839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57210e0 0x5638c57234d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.018+0000 7fd54abad740 1 --2- 172.21.15.7:0/4241164839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c5717fe0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.018+0000 7fd54abad740 1 --2- 172.21.15.7:0/4241164839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 0x5638c5717620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.018+0000 7fd54abad740 1 -- 172.21.15.7:0/4241164839 >> 172.21.15.7:0/4241164839 conn(0x5638c56dec90 msgr2=0x5638c5712850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.018+0000 7fd54abad740 1 -- 172.21.15.7:0/4241164839 shutdown_connections 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.018+0000 7fd54abad740 1 -- 172.21.15.7:0/4241164839 wait complete. 2024-04-21T08:46:32.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.018+0000 7fd54abad740 1 -- start start 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54abad740 1 Processor -- start 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54abad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c57209e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54abad740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57188d0 0x5638c571dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54abad740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c571e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54abad740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5638c5723e10 con 0x5638c57210e0 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54abad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5638c5604eb0 con 0x5638c5717b60 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54abad740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5638c5723f90 con 0x5638c57188d0 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54abad740 10 client.0 _ll_register_callbacks cb 0x5638c5631b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54647d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c571e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54647d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c571e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38514/0 (socket says 172.21.15.7:38514) 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54647d640 1 -- 172.21.15.7:0/1213663429 learned_addr learned my addr 172.21.15.7:0/1213663429 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd545c7c640 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c57209e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54547b640 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57188d0 0x5638c571dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54647d640 1 -- 172.21.15.7:0/1213663429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57188d0 msgr2=0x5638c571dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54647d640 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57188d0 0x5638c571dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54647d640 1 -- 172.21.15.7:0/1213663429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 msgr2=0x5638c57209e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54647d640 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c57209e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54647d640 1 -- 172.21.15.7:0/1213663429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638c538f210 con 0x5638c57210e0 2024-04-21T08:46:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.019+0000 7fd54547b640 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57188d0 0x5638c571dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd54647d640 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c571e340 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd538036000 tx=0x7fd5380095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc12f34daffe5eb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd5297fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd5297fa640 1 -- 172.21.15.7:0/1213663429 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd538039040 con 0x5638c57210e0 2024-04-21T08:46:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd5297fa640 1 -- 172.21.15.7:0/1213663429 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fd5380324c0 con 0x5638c57210e0 2024-04-21T08:46:32.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd54abad740 1 -- 172.21.15.7:0/1213663429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd53c019ae0 con 0x5638c57210e0 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd5297fa640 1 -- 172.21.15.7:0/1213663429 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd53805a050 con 0x5638c57210e0 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd54abad740 10 client.16164 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd54abad740 1 -- 172.21.15.7:0/1213663429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5638c545b620 con 0x5638c57210e0 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd5297fa640 1 -- 172.21.15.7:0/1213663429 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd53c019ae0 con 0x5638c57210e0 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd5297fa640 1 -- 172.21.15.7:0/1213663429 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fd538034830 con 0x5638c57210e0 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd54abad740 10 client.16164 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd54abad740 10 client.16164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd54abad740 20 client.16164 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd54abad740 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5638c5731100 0x5638c57334f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd54abad740 4 client.16164 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.020+0000 7fd54abad740 1 -- 172.21.15.7:0/1213663429 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638c5690880 con 0x5638c5731100 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.021+0000 7fd545c7c640 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5638c5731100 0x5638c57334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.021+0000 7fd545c7c640 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5638c5731100 0x5638c57334f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4640eb1f874af64 server_cookie=215d8a6f82e1b841 in_seq=0 out_seq=0 2024-04-21T08:46:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.021+0000 7fd5297fa640 10 client.16164 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd5297fa640 1 -- 172.21.15.7:0/1213663429 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638c5690880 con 0x5638c5731100 2024-04-21T08:46:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd5297fa640 10 client.16164 handle_command_reply: tid=0 2024-04-21T08:46:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 1 client.16164 shutdown 2024-04-21T08:46:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 1 -- 172.21.15.7:0/1213663429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 msgr2=0x5638c571e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c571e340 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd538036000 tx=0x7fd5380095c0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 1 -- 172.21.15.7:0/1213663429 shutdown_connections 2024-04-21T08:46:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5638c5731100 0x5638c57334f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c571e340 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57188d0 0x5638c571dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd5297fa640 0 client.16164 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:32.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 1 --2- 172.21.15.7:0/1213663429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c57209e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 1 -- 172.21.15.7:0/1213663429 >> 172.21.15.7:0/1213663429 conn(0x5638c56dec90 msgr2=0x5638c57145a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 1 -- 172.21.15.7:0/1213663429 shutdown_connections 2024-04-21T08:46:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 1 -- 172.21.15.7:0/1213663429 wait complete. 2024-04-21T08:46:32.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.023+0000 7fd54abad740 20 client.16164 trim_cache size 0 max 16384 2024-04-21T08:46:32.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.038+0000 7fd54abad740 1 Processor -- start 2024-04-21T08:46:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54abad740 1 -- start start 2024-04-21T08:46:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54abad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c572cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54abad740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 0x5638c571d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54abad740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57210e0 0x5638c5738e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54abad740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401ad450 con 0x5638c57188d0 2024-04-21T08:46:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54abad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401afcd0 con 0x5638c5717b60 2024-04-21T08:46:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54abad740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401adac0 con 0x5638c57210e0 2024-04-21T08:46:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54547b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 0x5638c571d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54547b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 0x5638c571d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38522/0 (socket says 172.21.15.7:38522) 2024-04-21T08:46:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd545c7c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c572cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54547b640 1 -- 172.21.15.7:0/3709561812 learned_addr learned my addr 172.21.15.7:0/3709561812 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54647d640 1 --2- 172.21.15.7:0/3709561812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57210e0 0x5638c5738e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54547b640 1 -- 172.21.15.7:0/3709561812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57210e0 msgr2=0x5638c5738e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54547b640 1 --2- 172.21.15.7:0/3709561812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57210e0 0x5638c5738e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54547b640 1 -- 172.21.15.7:0/3709561812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 msgr2=0x5638c572cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54547b640 1 --2- 172.21.15.7:0/3709561812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c572cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54547b640 1 -- 172.21.15.7:0/3709561812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638c5716e30 con 0x5638c57188d0 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd545c7c640 1 --2- 172.21.15.7:0/3709561812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c572cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd54547b640 1 --2- 172.21.15.7:0/3709561812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 0x5638c571d320 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd53009c810 tx=0x7fd53006e610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e569e0957e94e7d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.039+0000 7fd52b7fe640 1 -- 172.21.15.7:0/3709561812 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5300a0e10 con 0x5638c57188d0 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd52b7fe640 1 -- 172.21.15.7:0/3709561812 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5300a56d0 con 0x5638c57188d0 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd52b7fe640 1 -- 172.21.15.7:0/3709561812 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd530018b20 con 0x5638c57188d0 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd54abad740 1 -- 172.21.15.7:0/3709561812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 msgr2=0x5638c571d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd54abad740 1 --2- 172.21.15.7:0/3709561812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 0x5638c571d320 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd53009c810 tx=0x7fd53006e610 comp rx=0 tx=0).stop 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd54abad740 1 -- 172.21.15.7:0/3709561812 shutdown_connections 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd54abad740 1 --2- 172.21.15.7:0/3709561812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c57210e0 0x5638c5738e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd54abad740 1 --2- 172.21.15.7:0/3709561812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57188d0 0x5638c571d320 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd54abad740 1 --2- 172.21.15.7:0/3709561812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c5717b60 0x5638c572cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd54abad740 1 -- 172.21.15.7:0/3709561812 >> 172.21.15.7:0/3709561812 conn(0x5638c56dec90 msgr2=0x5638c5719060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd54abad740 1 -- 172.21.15.7:0/3709561812 shutdown_connections 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd54abad740 1 -- 172.21.15.7:0/3709561812 wait complete. 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.040+0000 7fd54abad740 1 -- start start 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54abad740 1 Processor -- start 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54abad740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c5717b60 0x5638c5721db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54abad740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c57188d0 0x5638c57222f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54abad740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c5722830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54abad740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401ae190 con 0x5638c57210e0 2024-04-21T08:46:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54abad740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401af600 con 0x5638c57188d0 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54abad740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5401ae860 con 0x5638c5717b60 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54abad740 10 client.0 _ll_register_callbacks cb 0x7fd53c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54647d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c5722830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54547b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c57188d0 0x5638c57222f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54647d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c5722830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38538/0 (socket says 172.21.15.7:38538) 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54647d640 1 -- 172.21.15.7:0/1770753703 learned_addr learned my addr 172.21.15.7:0/1770753703 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd545c7c640 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c5717b60 0x5638c5721db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54547b640 1 -- 172.21.15.7:0/1770753703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c5717b60 msgr2=0x5638c5721db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54547b640 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c5717b60 0x5638c5721db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54547b640 1 -- 172.21.15.7:0/1770753703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 msgr2=0x5638c5722830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54547b640 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c5722830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54547b640 1 -- 172.21.15.7:0/1770753703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638c570df30 con 0x5638c57188d0 2024-04-21T08:46:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.041+0000 7fd54647d640 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c5722830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd54547b640 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c57188d0 0x5638c57222f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5638c57184f0 tx=0x7fd5300d1240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5041b21ed160b4a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd5297fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd5297fa640 1 -- 172.21.15.7:0/1770753703 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53009c160 con 0x5638c57188d0 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd54abad740 1 -- 172.21.15.7:0/1770753703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5300a56d0 con 0x5638c57188d0 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd5297fa640 1 -- 172.21.15.7:0/1770753703 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fd530018b20 con 0x5638c57188d0 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd5297fa640 1 -- 172.21.15.7:0/1770753703 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd5300a0e10 con 0x5638c57188d0 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd54abad740 10 client.25745 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd54abad740 1 -- 172.21.15.7:0/1770753703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5638c5723940 con 0x5638c57188d0 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd5297fa640 1 -- 172.21.15.7:0/1770753703 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5300a56d0 con 0x5638c57188d0 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd5297fa640 1 -- 172.21.15.7:0/1770753703 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fd53006e200 con 0x5638c57188d0 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd54abad740 10 client.25745 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd54abad740 10 client.25745 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd54abad740 20 client.25745 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd54abad740 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5638c5797e10 0x5638c579a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd54abad740 4 client.25745 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:46:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.042+0000 7fd54abad740 1 -- 172.21.15.7:0/1770753703 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5638c561c3d0 con 0x5638c5797e10 2024-04-21T08:46:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.043+0000 7fd545c7c640 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5638c5797e10 0x5638c579a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.043+0000 7fd545c7c640 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5638c5797e10 0x5638c579a200 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c00cb3ec03cbe0f server_cookie=a0ef77f60f9e4161 in_seq=0 out_seq=0 2024-04-21T08:46:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:32.043+0000 7fd5297fa640 10 client.25745 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:32.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:32 smithi077 ceph-mon[31615]: pgmap v59: 129 pgs: 129 active+clean; 212 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 899 op/s 2024-04-21T08:46:32.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:32 smithi007 ceph-mon[28477]: pgmap v59: 129 pgs: 129 active+clean; 212 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 899 op/s 2024-04-21T08:46:32.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:32 smithi016 ceph-mon[31259]: pgmap v59: 129 pgs: 129 active+clean; 212 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 899 op/s 2024-04-21T08:46:34.603 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:34 smithi077 ceph-mon[31615]: pgmap v60: 129 pgs: 129 active+clean; 251 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 1.38k op/s 2024-04-21T08:46:34.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:34 smithi007 ceph-mon[28477]: pgmap v60: 129 pgs: 129 active+clean; 251 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 1.38k op/s 2024-04-21T08:46:34.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:34 smithi016 ceph-mon[31259]: pgmap v60: 129 pgs: 129 active+clean; 251 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 1.38k op/s 2024-04-21T08:46:36.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:36 smithi007 ceph-mon[28477]: pgmap v61: 129 pgs: 129 active+clean; 251 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 1.31k op/s 2024-04-21T08:46:36.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:36 smithi016 ceph-mon[31259]: pgmap v61: 129 pgs: 129 active+clean; 251 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 1.31k op/s 2024-04-21T08:46:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:36 smithi077 ceph-mon[31615]: pgmap v61: 129 pgs: 129 active+clean; 251 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 1.31k op/s 2024-04-21T08:46:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd5297fa640 1 -- 172.21.15.7:0/1770753703 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5638c561c3d0 con 0x5638c5797e10 2024-04-21T08:46:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd5297fa640 10 client.25745 handle_command_reply: tid=0 2024-04-21T08:46:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd54abad740 1 client.25745 shutdown 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd54abad740 1 -- 172.21.15.7:0/1770753703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c57188d0 msgr2=0x5638c57222f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd54abad740 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c57188d0 0x5638c57222f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5638c57184f0 tx=0x7fd5300d1240 comp rx=0 tx=0).stop 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd54abad740 1 -- 172.21.15.7:0/1770753703 shutdown_connections 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd54abad740 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5638c5797e10 0x5638c579a200 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd54abad740 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5638c57210e0 0x5638c5722830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd54abad740 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5638c57188d0 0x5638c57222f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd54abad740 1 --2- 172.21.15.7:0/1770753703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5638c5717b60 0x5638c5721db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd5297fa640 0 client.25745 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.964+0000 7fd54abad740 1 -- 172.21.15.7:0/1770753703 >> 172.21.15.7:0/1770753703 conn(0x5638c56dec90 msgr2=0x5638c571c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd54abad740 1 -- 172.21.15.7:0/1770753703 shutdown_connections 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd54abad740 1 -- 172.21.15.7:0/1770753703 wait complete. 2024-04-21T08:46:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd54abad740 20 client.25745 trim_cache size 0 max 16384 2024-04-21T08:46:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:46:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:46:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:46:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:46:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 -- 172.21.15.7:0/1837069759 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd52c0a3d10 msgr2=0x7fd52c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd52c0a3d10 0x7fd52c0a61d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd538004af0 tx=0x7fd53802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 -- 172.21.15.7:0/1837069759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd54006af70 msgr2=0x7fd540198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd54006af70 0x7fd540198c20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd53000e9f0 tx=0x7fd53000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:46:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 -- 172.21.15.7:0/1837069759 shutdown_connections 2024-04-21T08:46:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd540108040 0x7fd54019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd54006af70 0x7fd540198c20 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd52c0a3d10 0x7fd52c0a61d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 --2- 172.21.15.7:0/1837069759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54006a5d0 0x7fd5401986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 -- 172.21.15.7:0/1837069759 >> 172.21.15.7:0/1837069759 conn(0x7fd540077cd0 msgr2=0x7fd5400ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:36.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 -- 172.21.15.7:0/1837069759 shutdown_connections 2024-04-21T08:46:36.968 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:36.965+0000 7fd547ee0640 1 -- 172.21.15.7:0/1837069759 wait complete. 2024-04-21T08:46:38.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:38 smithi007 ceph-mon[28477]: pgmap v62: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 3.00k op/s 2024-04-21T08:46:38.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:38 smithi016 ceph-mon[31259]: pgmap v62: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 3.00k op/s 2024-04-21T08:46:38.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:38 smithi077 ceph-mon[31615]: pgmap v62: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 3.00k op/s 2024-04-21T08:46:40.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:40 smithi007 ceph-mon[28477]: pgmap v63: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 2.17k op/s 2024-04-21T08:46:40.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:40 smithi016 ceph-mon[31259]: pgmap v63: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 2.17k op/s 2024-04-21T08:46:40.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:40 smithi077 ceph-mon[31615]: pgmap v63: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 2.17k op/s 2024-04-21T08:46:42.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:42 smithi007 ceph-mon[28477]: pgmap v64: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 2.17k op/s 2024-04-21T08:46:42.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:42 smithi016 ceph-mon[31259]: pgmap v64: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 2.17k op/s 2024-04-21T08:46:42.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:42 smithi077 ceph-mon[31615]: pgmap v64: 129 pgs: 129 active+clean; 227 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 2.17k op/s 2024-04-21T08:46:44.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:44 smithi007 ceph-mon[28477]: pgmap v65: 129 pgs: 129 active+clean; 228 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 2.17k op/s 2024-04-21T08:46:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:44 smithi016 ceph-mon[31259]: pgmap v65: 129 pgs: 129 active+clean; 228 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 2.17k op/s 2024-04-21T08:46:44.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:44 smithi077 ceph-mon[31615]: pgmap v65: 129 pgs: 129 active+clean; 228 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 2.17k op/s 2024-04-21T08:46:46.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:46 smithi007 ceph-mon[28477]: pgmap v66: 129 pgs: 129 active+clean; 228 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.4 MiB/s wr, 1.69k op/s 2024-04-21T08:46:46.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:46 smithi016 ceph-mon[31259]: pgmap v66: 129 pgs: 129 active+clean; 228 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.4 MiB/s wr, 1.69k op/s 2024-04-21T08:46:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:46 smithi077 ceph-mon[31615]: pgmap v66: 129 pgs: 129 active+clean; 228 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.4 MiB/s wr, 1.69k op/s 2024-04-21T08:46:48.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:48 smithi007 ceph-mon[28477]: pgmap v67: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 1.5 MiB/s wr, 1.70k op/s 2024-04-21T08:46:48.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:48 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:48.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:48 smithi016 ceph-mon[31259]: pgmap v67: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 1.5 MiB/s wr, 1.70k op/s 2024-04-21T08:46:48.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:48 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:48.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:48 smithi077 ceph-mon[31615]: pgmap v67: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 1.5 MiB/s wr, 1.70k op/s 2024-04-21T08:46:48.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:48 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:49.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:49 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:49.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:49 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:49.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:49 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:46:49.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:49 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:49.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:49 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:49.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:49 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:46:49.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:49 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:49.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:49 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:49.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:49 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:46:50.066 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.063+0000 7f49ba396640 1 -- 172.21.15.7:0/3881144112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b406a590 msgr2=0x7f49b406aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.063+0000 7f49ba396640 1 --2- 172.21.15.7:0/3881144112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b406a590 0x7f49b406aa10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009900 tx=0x7f49a802bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.063+0000 7f49ba396640 1 -- 172.21.15.7:0/3881144112 shutdown_connections 2024-04-21T08:46:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.063+0000 7f49ba396640 1 --2- 172.21.15.7:0/3881144112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b406af50 0x7f49b410edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.063+0000 7f49ba396640 1 --2- 172.21.15.7:0/3881144112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b406a590 0x7f49b406aa10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.063+0000 7f49ba396640 1 --2- 172.21.15.7:0/3881144112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b4106f60 0x7f49b4107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.063+0000 7f49ba396640 1 -- 172.21.15.7:0/3881144112 >> 172.21.15.7:0/3881144112 conn(0x7f49b4077d10 msgr2=0x7f49b4078120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.063+0000 7f49ba396640 1 -- 172.21.15.7:0/3881144112 shutdown_connections 2024-04-21T08:46:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.063+0000 7f49ba396640 1 -- 172.21.15.7:0/3881144112 wait complete. 2024-04-21T08:46:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.064+0000 7f49ba396640 1 Processor -- start 2024-04-21T08:46:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.064+0000 7f49ba396640 1 -- start start 2024-04-21T08:46:50.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49ba396640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b406a590 0x7f49b41a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49ba396640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b406af50 0x7f49b41a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49ba396640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b4106f60 0x7f49b419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49ba396640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f49b41121b0 con 0x7f49b406a590 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49ba396640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49b4112030 con 0x7f49b4106f60 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49ba396640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49b4112330 con 0x7f49b406af50 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b406a590 0x7f49b41a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b406a590 0x7f49b41a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53384/0 (socket says 172.21.15.7:53384) 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b3fff640 1 -- 172.21.15.7:0/2599947366 learned_addr learned my addr 172.21.15.7:0/2599947366 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b37fe640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b406af50 0x7f49b41a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b8933640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b4106f60 0x7f49b419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b3fff640 1 -- 172.21.15.7:0/2599947366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b406af50 msgr2=0x7f49b41a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b3fff640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b406af50 0x7f49b41a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b3fff640 1 -- 172.21.15.7:0/2599947366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b4106f60 msgr2=0x7f49b419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:50.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b3fff640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b4106f60 0x7f49b419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b3fff640 1 -- 172.21.15.7:0/2599947366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b419b010 con 0x7f49b406a590 2024-04-21T08:46:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b3fff640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b406a590 0x7f49b41a0690 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f49a400e9c0 tx=0x7f49a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.065+0000 7f49b17fa640 1 -- 172.21.15.7:0/2599947366 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a400cd80 con 0x7f49b406a590 2024-04-21T08:46:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.066+0000 7f49b17fa640 1 -- 172.21.15.7:0/2599947366 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49a40035f0 con 0x7f49b406a590 2024-04-21T08:46:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.066+0000 7f49ba396640 1 -- 172.21.15.7:0/2599947366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b419b300 con 0x7f49b406a590 2024-04-21T08:46:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.066+0000 7f49b17fa640 1 -- 172.21.15.7:0/2599947366 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a4010690 con 0x7f49b406a590 2024-04-21T08:46:50.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.066+0000 7f49ba396640 1 -- 172.21.15.7:0/2599947366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b41a7610 con 0x7f49b406a590 2024-04-21T08:46:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.068+0000 7f49b17fa640 1 -- 172.21.15.7:0/2599947366 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f49a40037a0 con 0x7f49b406a590 2024-04-21T08:46:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.068+0000 7f49bcf85740 1 Processor -- start 2024-04-21T08:46:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.068+0000 7f49bcf85740 1 -- start start 2024-04-21T08:46:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f77453e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453eb60 0x55f77453efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7745480e0 0x55f77454a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49bcf85740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f7744a38d0 con 0x55f77453eb60 2024-04-21T08:46:50.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49bcf85740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f774440e80 con 0x55f7745480e0 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49bcf85740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f77449f890 con 0x55f77453f8d0 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b17fa640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f499c0a3d90 0x7f499c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453eb60 0x55f77453efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453eb60 0x55f77453efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53388/0 (socket says 172.21.15.7:53388) 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b3fff640 1 -- 172.21.15.7:0/1922311778 learned_addr learned my addr 172.21.15.7:0/1922311778 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b8933640 1 --2- 172.21.15.7:0/1922311778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f77453e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b37fe640 1 --2- 172.21.15.7:0/1922311778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7745480e0 0x55f77454a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b8933640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f499c0a3d90 0x7f499c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b17fa640 1 -- 172.21.15.7:0/2599947366 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f49a40d0ed0 con 0x7f49b406a590 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b3fff640 1 -- 172.21.15.7:0/1922311778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 msgr2=0x55f77453e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b3fff640 1 --2- 172.21.15.7:0/1922311778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f77453e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b3fff640 1 -- 172.21.15.7:0/1922311778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7745480e0 msgr2=0x55f77454a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:50.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b3fff640 1 --2- 172.21.15.7:0/1922311778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7745480e0 0x55f77454a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b3fff640 1 -- 172.21.15.7:0/1922311778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f774462320 con 0x55f77453eb60 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.069+0000 7f49b3fff640 1 --2- 172.21.15.7:0/1922311778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453eb60 0x55f77453efe0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f49a400cfd0 tx=0x7f49a409c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f432ddde9e64b14 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f49b8933640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f499c0a3d90 0x7f499c0a6250 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f49ac005c40 tx=0x7f49ac009500 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f499a7fc640 1 -- 172.21.15.7:0/1922311778 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a40a0dd0 con 0x55f77453eb60 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f499a7fc640 1 -- 172.21.15.7:0/1922311778 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49a40bd070 con 0x55f77453eb60 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f499a7fc640 1 -- 172.21.15.7:0/1922311778 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a40ce450 con 0x55f77453eb60 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f49bcf85740 1 -- 172.21.15.7:0/1922311778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453eb60 msgr2=0x55f77453efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f49bcf85740 1 --2- 172.21.15.7:0/1922311778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453eb60 0x55f77453efe0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f49a400cfd0 tx=0x7f49a409c920 comp rx=0 tx=0).stop 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f49bcf85740 1 -- 172.21.15.7:0/1922311778 shutdown_connections 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f49bcf85740 1 --2- 172.21.15.7:0/1922311778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f7745480e0 0x55f77454a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f49bcf85740 1 --2- 172.21.15.7:0/1922311778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453eb60 0x55f77453efe0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f49bcf85740 1 --2- 172.21.15.7:0/1922311778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f77453e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f49bcf85740 1 -- 172.21.15.7:0/1922311778 >> 172.21.15.7:0/1922311778 conn(0x55f774505c90 msgr2=0x55f774539850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.070+0000 7f49bcf85740 1 -- 172.21.15.7:0/1922311778 shutdown_connections 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.071+0000 7f49bcf85740 1 -- 172.21.15.7:0/1922311778 wait complete. 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.071+0000 7f49bcf85740 1 -- start start 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.071+0000 7f49bcf85740 1 Processor -- start 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.071+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774547a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.071+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f774544de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.071+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774545340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.071+0000 7f49bcf85740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f77454ae10 con 0x55f7745480e0 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.071+0000 7f49bcf85740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f77442beb0 con 0x55f77453eb60 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.071+0000 7f49bcf85740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f77454af90 con 0x55f77453f8d0 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.071+0000 7f49bcf85740 10 client.0 _ll_register_callbacks cb 0x55f774458b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:50.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b8933640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774545340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b8933640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774545340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53390/0 (socket says 172.21.15.7:53390) 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b8933640 1 -- 172.21.15.7:0/2283502358 learned_addr learned my addr 172.21.15.7:0/2283502358 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b3fff640 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774547a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b37fe640 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f774544de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b8933640 1 -- 172.21.15.7:0/2283502358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 msgr2=0x55f774544de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b8933640 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f774544de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b8933640 1 -- 172.21.15.7:0/2283502358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 msgr2=0x55f774547a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b8933640 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774547a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b8933640 1 -- 172.21.15.7:0/2283502358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7741b6210 con 0x55f7745480e0 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b37fe640 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f774544de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49b8933640 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774545340 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00ffd0 tx=0x7f49ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52bb23fbadc141f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49937fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:50.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49937fe640 1 -- 172.21.15.7:0/2283502358 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac02d060 con 0x55f7745480e0 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49937fe640 1 -- 172.21.15.7:0/2283502358 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f49ac020460 con 0x55f7745480e0 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49bcf85740 1 -- 172.21.15.7:0/2283502358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49a40bd070 con 0x55f7745480e0 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49937fe640 1 -- 172.21.15.7:0/2283502358 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f49ac004b20 con 0x55f7745480e0 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49bcf85740 10 client.16182 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.072+0000 7f49bcf85740 1 -- 172.21.15.7:0/2283502358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f774282620 con 0x55f7745480e0 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.073+0000 7f49937fe640 1 -- 172.21.15.7:0/2283502358 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49a40bd070 con 0x55f7745480e0 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.073+0000 7f49937fe640 1 -- 172.21.15.7:0/2283502358 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f49ac027030 con 0x55f7745480e0 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.073+0000 7f49bcf85740 10 client.16182 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.073+0000 7f49bcf85740 10 client.16182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.073+0000 7f49bcf85740 20 client.16182 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.073+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f774558110 0x55f77455a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.073+0000 7f49bcf85740 4 client.16182 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.073+0000 7f49bcf85740 1 -- 172.21.15.7:0/2283502358 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7744b7880 con 0x55f774558110 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.073+0000 7f49b3fff640 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f774558110 0x55f77455a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.074+0000 7f49937fe640 10 client.16182 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.074+0000 7f49b3fff640 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f774558110 0x55f77455a500 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1b6188cad36554d server_cookie=ce9349611524441 in_seq=0 out_seq=0 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.074+0000 7f49937fe640 1 -- 172.21.15.7:0/2283502358 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7744b7880 con 0x55f774558110 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.074+0000 7f49937fe640 10 client.16182 handle_command_reply: tid=0 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.074+0000 7f49bcf85740 1 client.16182 shutdown 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49bcf85740 1 -- 172.21.15.7:0/2283502358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 msgr2=0x55f774545340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774545340 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00ffd0 tx=0x7f49ac004a40 comp rx=0 tx=0).stop 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49bcf85740 1 -- 172.21.15.7:0/2283502358 shutdown_connections 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f774558110 0x55f77455a500 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774545340 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f774544de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2283502358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774547a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49937fe640 0 client.16182 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49bcf85740 1 -- 172.21.15.7:0/2283502358 >> 172.21.15.7:0/2283502358 conn(0x55f774505c90 msgr2=0x55f77453b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49bcf85740 1 -- 172.21.15.7:0/2283502358 shutdown_connections 2024-04-21T08:46:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49bcf85740 1 -- 172.21.15.7:0/2283502358 wait complete. 2024-04-21T08:46:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.075+0000 7f49bcf85740 20 client.16182 trim_cache size 0 max 16384 2024-04-21T08:46:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.090+0000 7f49bcf85740 1 Processor -- start 2024-04-21T08:46:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.090+0000 7f49bcf85740 1 -- start start 2024-04-21T08:46:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.090+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774553d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.090+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f7745405d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.090+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774547510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.090+0000 7f49bcf85740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f49b41af770 con 0x55f7745480e0 2024-04-21T08:46:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.090+0000 7f49bcf85740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49b41b1ff0 con 0x55f77453eb60 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.090+0000 7f49bcf85740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49b41afde0 con 0x55f77453f8d0 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.090+0000 7f49b8933640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774547510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.090+0000 7f49b8933640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774547510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53404/0 (socket says 172.21.15.7:53404) 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.090+0000 7f49b8933640 1 -- 172.21.15.7:0/797218090 learned_addr learned my addr 172.21.15.7:0/797218090 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49b3fff640 1 --2- 172.21.15.7:0/797218090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774553d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49b8933640 1 -- 172.21.15.7:0/797218090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 msgr2=0x55f7745405d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49b37fe640 1 --2- 172.21.15.7:0/797218090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f7745405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49b8933640 1 --2- 172.21.15.7:0/797218090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f7745405d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49b8933640 1 -- 172.21.15.7:0/797218090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 msgr2=0x55f774553d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49b8933640 1 --2- 172.21.15.7:0/797218090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774553d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49b8933640 1 -- 172.21.15.7:0/797218090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f774543330 con 0x55f7745480e0 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49b3fff640 1 --2- 172.21.15.7:0/797218090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774553d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49b37fe640 1 --2- 172.21.15.7:0/797218090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f7745405d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49b8933640 1 --2- 172.21.15.7:0/797218090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774547510 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009270 tx=0x7f49ac009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4225618e41bd0f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f4999ffb640 1 -- 172.21.15.7:0/797218090 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac03d070 con 0x55f7745480e0 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f4999ffb640 1 -- 172.21.15.7:0/797218090 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49ac011d20 con 0x55f7745480e0 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f4999ffb640 1 -- 172.21.15.7:0/797218090 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac0206c0 con 0x55f7745480e0 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49bcf85740 1 -- 172.21.15.7:0/797218090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 msgr2=0x55f774547510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.091+0000 7f49bcf85740 1 --2- 172.21.15.7:0/797218090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774547510 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009270 tx=0x7f49ac009610 comp rx=0 tx=0).stop 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 -- 172.21.15.7:0/797218090 shutdown_connections 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 --2- 172.21.15.7:0/797218090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7745480e0 0x55f774547510 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 --2- 172.21.15.7:0/797218090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f77453f8d0 0x55f7745405d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 --2- 172.21.15.7:0/797218090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774553d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 -- 172.21.15.7:0/797218090 >> 172.21.15.7:0/797218090 conn(0x55f774505c90 msgr2=0x55f77453fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 -- 172.21.15.7:0/797218090 shutdown_connections 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 -- 172.21.15.7:0/797218090 wait complete. 2024-04-21T08:46:50.097 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 -- start start 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 Processor -- start 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774550630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453f8d0 0x55f77454b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7745480e0 0x55f77454bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f49b41b04b0 con 0x55f77453f8d0 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f49b41b1920 con 0x55f77453eb60 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.092+0000 7f49bcf85740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f49b41b0b80 con 0x55f7745480e0 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49bcf85740 10 client.0 _ll_register_callbacks cb 0x7f49ac011550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453f8d0 0x55f77454b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453f8d0 0x55f77454b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53412/0 (socket says 172.21.15.7:53412) 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b37fe640 1 -- 172.21.15.7:0/2621061916 learned_addr learned my addr 172.21.15.7:0/2621061916 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b8933640 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7745480e0 0x55f77454bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b3fff640 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774550630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b37fe640 1 -- 172.21.15.7:0/2621061916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7745480e0 msgr2=0x55f77454bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b37fe640 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7745480e0 0x55f77454bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b37fe640 1 -- 172.21.15.7:0/2621061916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 msgr2=0x55f774550630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b37fe640 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774550630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b37fe640 1 -- 172.21.15.7:0/2621061916 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f77454c760 con 0x55f77453f8d0 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b3fff640 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774550630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49b37fe640 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453f8d0 0x55f77454b9f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009900 tx=0x7f49a8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4470746d5416bb4e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49937fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49937fe640 1 -- 172.21.15.7:0/2621061916 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8005030 con 0x55f77453f8d0 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49937fe640 1 -- 172.21.15.7:0/2621061916 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f49a80051d0 con 0x55f77453f8d0 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49bcf85740 1 -- 172.21.15.7:0/2621061916 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f7744433d0 con 0x55f77453f8d0 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49937fe640 1 -- 172.21.15.7:0/2621061916 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f49a80053d0 con 0x55f77453f8d0 2024-04-21T08:46:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49bcf85740 10 client.16188 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.093+0000 7f49bcf85740 1 -- 172.21.15.7:0/2621061916 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f7745be660 con 0x55f77453f8d0 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.094+0000 7f49937fe640 1 -- 172.21.15.7:0/2621061916 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f7744433d0 con 0x55f77453f8d0 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.094+0000 7f49937fe640 1 -- 172.21.15.7:0/2621061916 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f49a80037b0 con 0x55f77453f8d0 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.094+0000 7f49bcf85740 10 client.16188 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.094+0000 7f49bcf85740 10 client.16188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.094+0000 7f49bcf85740 20 client.16188 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.094+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f7745bee60 0x55f7745c1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.094+0000 7f49bcf85740 4 client.16188 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.094+0000 7f49bcf85740 1 -- 172.21.15.7:0/2621061916 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f7744b12f0 con 0x55f7745bee60 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.094+0000 7f49b3fff640 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f7745bee60 0x55f7745c1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.095+0000 7f49937fe640 10 client.16188 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:50.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:50.095+0000 7f49b3fff640 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f7745bee60 0x55f7745c1280 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e3d9fac86873719 server_cookie=b2599fd66ab5bba3 in_seq=0 out_seq=0 2024-04-21T08:46:50.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:50 smithi016 ceph-mon[31259]: pgmap v68: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 241 KiB/s wr, 13 op/s 2024-04-21T08:46:50.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:50 smithi077 ceph-mon[31615]: pgmap v68: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 241 KiB/s wr, 13 op/s 2024-04-21T08:46:50.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:50 smithi007 ceph-mon[28477]: pgmap v68: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 241 KiB/s wr, 13 op/s 2024-04-21T08:46:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.146+0000 7f49937fe640 1 -- 172.21.15.7:0/2621061916 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f7744b12f0 con 0x55f7745bee60 2024-04-21T08:46:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.146+0000 7f49937fe640 10 client.16188 handle_command_reply: tid=0 2024-04-21T08:46:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.146+0000 7f49bcf85740 1 client.16188 shutdown 2024-04-21T08:46:51.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49bcf85740 1 -- 172.21.15.7:0/2621061916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453f8d0 msgr2=0x55f77454b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453f8d0 0x55f77454b9f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009900 tx=0x7f49a8004270 comp rx=0 tx=0).stop 2024-04-21T08:46:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49bcf85740 1 -- 172.21.15.7:0/2621061916 shutdown_connections 2024-04-21T08:46:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f7745bee60 0x55f7745c1280 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f7745480e0 0x55f77454bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f77453f8d0 0x55f77454b9f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49937fe640 0 client.16188 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49bcf85740 1 --2- 172.21.15.7:0/2621061916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f77453eb60 0x55f774550630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:51.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49bcf85740 1 -- 172.21.15.7:0/2621061916 >> 172.21.15.7:0/2621061916 conn(0x55f774505c90 msgr2=0x55f774538040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:51.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49bcf85740 1 -- 172.21.15.7:0/2621061916 shutdown_connections 2024-04-21T08:46:51.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49bcf85740 1 -- 172.21.15.7:0/2621061916 wait complete. 2024-04-21T08:46:51.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.147+0000 7f49bcf85740 20 client.16188 trim_cache size 0 max 16384 2024-04-21T08:46:51.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:46:51.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:46:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:46:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 -- 172.21.15.7:0/2599947366 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f499c0a3d90 msgr2=0x7f499c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f499c0a3d90 0x7f499c0a6250 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f49ac005c40 tx=0x7f49ac009500 comp rx=0 tx=0).stop 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 -- 172.21.15.7:0/2599947366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b406a590 msgr2=0x7f49b41a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b406a590 0x7f49b41a0690 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f49a400e9c0 tx=0x7f49a400ee80 comp rx=0 tx=0).stop 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 -- 172.21.15.7:0/2599947366 shutdown_connections 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b4106f60 0x7f49b419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f499c0a3d90 0x7f499c0a6250 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49b406af50 0x7f49b41a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 --2- 172.21.15.7:0/2599947366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b406a590 0x7f49b41a0690 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 -- 172.21.15.7:0/2599947366 >> 172.21.15.7:0/2599947366 conn(0x7f49b4077d10 msgr2=0x7f49b410d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 -- 172.21.15.7:0/2599947366 shutdown_connections 2024-04-21T08:46:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:46:51.148+0000 7f49ba396640 1 -- 172.21.15.7:0/2599947366 wait complete. 2024-04-21T08:46:52.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:52 smithi016 ceph-mon[31259]: pgmap v69: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 241 KiB/s wr, 13 op/s 2024-04-21T08:46:52.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:52 smithi077 ceph-mon[31615]: pgmap v69: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 241 KiB/s wr, 13 op/s 2024-04-21T08:46:52.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:52 smithi007 ceph-mon[28477]: pgmap v69: 129 pgs: 129 active+clean; 229 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 241 KiB/s wr, 13 op/s 2024-04-21T08:46:53.168 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:46:53.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:46:53.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.367+0000 7fc97ffff640 1 -- 172.21.15.7:0/103114086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9780957e0 msgr2=0x7fc9780a1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.367+0000 7fc97ffff640 1 --2- 172.21.15.7:0/103114086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9780957e0 0x7fc9780a1ab0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc980067dc0 tx=0x7fc980097fb0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.368+0000 7fc97ffff640 1 -- 172.21.15.7:0/103114086 shutdown_connections 2024-04-21T08:46:53.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.368+0000 7fc97ffff640 1 --2- 172.21.15.7:0/103114086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9780957e0 0x7fc9780a1ab0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.368+0000 7fc97ffff640 1 --2- 172.21.15.7:0/103114086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc978094e40 0x7fc9780952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.368+0000 7fc97ffff640 1 --2- 172.21.15.7:0/103114086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978099bb0 0x7fc978099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.368+0000 7fc97ffff640 1 -- 172.21.15.7:0/103114086 >> 172.21.15.7:0/103114086 conn(0x7fc97800a730 msgr2=0x7fc97800ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.373+0000 7fc97ffff640 1 -- 172.21.15.7:0/103114086 shutdown_connections 2024-04-21T08:46:53.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.373+0000 7fc97ffff640 1 -- 172.21.15.7:0/103114086 wait complete. 2024-04-21T08:46:53.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.374+0000 7fc97ffff640 1 Processor -- start 2024-04-21T08:46:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.374+0000 7fc97ffff640 1 -- start start 2024-04-21T08:46:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.374+0000 7fc97ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc978094e40 0x7fc97813afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.374+0000 7fc97ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9780957e0 0x7fc97813b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.374+0000 7fc97ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978099bb0 0x7fc97813f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.375+0000 7fc97ffff640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc9780a3cf0 con 0x7fc9780957e0 2024-04-21T08:46:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.375+0000 7fc97ffff640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc9780a3b70 con 0x7fc978099bb0 2024-04-21T08:46:53.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.375+0000 7fc97ffff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc9780a3e70 con 0x7fc978094e40 2024-04-21T08:46:53.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.375+0000 7fc97e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9780957e0 0x7fc97813b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.375+0000 7fc97e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9780957e0 0x7fc97813b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53430/0 (socket says 172.21.15.7:53430) 2024-04-21T08:46:53.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.375+0000 7fc97e7fc640 1 -- 172.21.15.7:0/2303529800 learned_addr learned my addr 172.21.15.7:0/2303529800 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:53.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.375+0000 7fc97effd640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc978094e40 0x7fc97813afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.376+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978099bb0 0x7fc97813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.376+0000 7fc97e7fc640 1 -- 172.21.15.7:0/2303529800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc978094e40 msgr2=0x7fc97813afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.376+0000 7fc97e7fc640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc978094e40 0x7fc97813afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.376+0000 7fc97e7fc640 1 -- 172.21.15.7:0/2303529800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978099bb0 msgr2=0x7fc97813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.376+0000 7fc97e7fc640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978099bb0 0x7fc97813f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.376+0000 7fc97e7fc640 1 -- 172.21.15.7:0/2303529800 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97813ffb0 con 0x7fc9780957e0 2024-04-21T08:46:53.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.376+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978099bb0 0x7fc97813f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:53.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.377+0000 7fc97e7fc640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9780957e0 0x7fc97813b500 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc97400b590 tx=0x7fc97400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.377+0000 7fc967fff640 1 -- 172.21.15.7:0/2303529800 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974008500 con 0x7fc9780957e0 2024-04-21T08:46:53.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.377+0000 7fc967fff640 1 -- 172.21.15.7:0/2303529800 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc974008b50 con 0x7fc9780957e0 2024-04-21T08:46:53.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.377+0000 7fc967fff640 1 -- 172.21.15.7:0/2303529800 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97401eca0 con 0x7fc9780957e0 2024-04-21T08:46:53.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.377+0000 7fc97ffff640 1 -- 172.21.15.7:0/2303529800 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9781402a0 con 0x7fc9780957e0 2024-04-21T08:46:53.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.377+0000 7fc97ffff640 1 -- 172.21.15.7:0/2303529800 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc978147b30 con 0x7fc9780957e0 2024-04-21T08:46:53.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.382+0000 7fc987feb740 1 Processor -- start 2024-04-21T08:46:53.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.383+0000 7fc987feb740 1 -- start start 2024-04-21T08:46:53.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.383+0000 7fc987feb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20d710 0x564f7f20daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.383+0000 7fc987feb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20e030 0x564f7f2166d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.383+0000 7fc987feb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f216c10 0x564f7f219000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.383+0000 7fc987feb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564f7f1721e0 con 0x564f7f20d710 2024-04-21T08:46:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.383+0000 7fc987feb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f7f10f390 con 0x564f7f20e030 2024-04-21T08:46:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.383+0000 7fc987feb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564f7f16e1a0 con 0x564f7f216c10 2024-04-21T08:46:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.383+0000 7fc967fff640 1 -- 172.21.15.7:0/2303529800 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fc974008d00 con 0x7fc9780957e0 2024-04-21T08:46:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.384+0000 7fc97f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20d710 0x564f7f20daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.384+0000 7fc97f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20d710 0x564f7f20daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53444/0 (socket says 172.21.15.7:53444) 2024-04-21T08:46:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.384+0000 7fc97f7fe640 1 -- 172.21.15.7:0/1832080964 learned_addr learned my addr 172.21.15.7:0/1832080964 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.384+0000 7fc97e7fc640 1 --2- 172.21.15.7:0/1832080964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f216c10 0x564f7f219000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.384+0000 7fc967fff640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc9680a3ea0 0x7fc9680a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.384+0000 7fc967fff640 1 -- 172.21.15.7:0/2303529800 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc97400e0a0 con 0x7fc9780957e0 2024-04-21T08:46:53.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.385+0000 7fc97effd640 1 --2- 172.21.15.7:0/1832080964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20e030 0x564f7f2166d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.385+0000 7fc97f7fe640 1 -- 172.21.15.7:0/1832080964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f216c10 msgr2=0x564f7f219000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.385+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/1832080964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f216c10 0x564f7f219000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.385+0000 7fc97f7fe640 1 -- 172.21.15.7:0/1832080964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20e030 msgr2=0x564f7f2166d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.385+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/1832080964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20e030 0x564f7f2166d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.386+0000 7fc97f7fe640 1 -- 172.21.15.7:0/1832080964 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f7ef4e7d0 con 0x564f7f20d710 2024-04-21T08:46:53.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.386+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc9680a3ea0 0x7fc9680a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.388+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/1832080964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20d710 0x564f7f20daf0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fc98006b100 tx=0x7fc98006a450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168fa32205ffe69f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.389+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc9680a3ea0 0x7fc9680a6360 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc9800679b0 tx=0x7fc9800a8000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.389+0000 7fc9657fa640 1 -- 172.21.15.7:0/1832080964 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9800b0070 con 0x564f7f20d710 2024-04-21T08:46:53.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.389+0000 7fc9657fa640 1 -- 172.21.15.7:0/1832080964 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc98009c040 con 0x564f7f20d710 2024-04-21T08:46:53.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.390+0000 7fc987feb740 1 -- 172.21.15.7:0/1832080964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20d710 msgr2=0x564f7f20daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.390+0000 7fc987feb740 1 --2- 172.21.15.7:0/1832080964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20d710 0x564f7f20daf0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fc98006b100 tx=0x7fc98006a450 comp rx=0 tx=0).stop 2024-04-21T08:46:53.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.390+0000 7fc987feb740 1 -- 172.21.15.7:0/1832080964 shutdown_connections 2024-04-21T08:46:53.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.390+0000 7fc987feb740 1 --2- 172.21.15.7:0/1832080964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f216c10 0x564f7f219000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.390+0000 7fc987feb740 1 --2- 172.21.15.7:0/1832080964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20e030 0x564f7f2166d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.390+0000 7fc987feb740 1 --2- 172.21.15.7:0/1832080964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20d710 0x564f7f20daf0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.390+0000 7fc987feb740 1 -- 172.21.15.7:0/1832080964 >> 172.21.15.7:0/1832080964 conn(0x564f7f207e40 msgr2=0x564f7f208220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.392+0000 7fc987feb740 1 -- 172.21.15.7:0/1832080964 shutdown_connections 2024-04-21T08:46:53.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.393+0000 7fc987feb740 1 -- 172.21.15.7:0/1832080964 wait complete. 2024-04-21T08:46:53.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.394+0000 7fc987feb740 1 -- start start 2024-04-21T08:46:53.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.394+0000 7fc987feb740 1 Processor -- start 2024-04-21T08:46:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.394+0000 7fc987feb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f20d710 0x564f7f213c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.394+0000 7fc987feb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20e030 0x564f7f214160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.394+0000 7fc987feb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f2146a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.394+0000 7fc987feb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564f7f22aa00 con 0x564f7f216c10 2024-04-21T08:46:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.394+0000 7fc987feb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564f7f12b2f0 con 0x564f7f20e030 2024-04-21T08:46:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.394+0000 7fc987feb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564f7f22ab80 con 0x564f7f20d710 2024-04-21T08:46:53.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.394+0000 7fc987feb740 10 client.0 _ll_register_callbacks cb 0x564f7f138580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.395+0000 7fc97effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f20d710 0x564f7f213c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.395+0000 7fc97f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f2146a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.395+0000 7fc97e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20e030 0x564f7f214160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.395+0000 7fc97f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f2146a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53456/0 (socket says 172.21.15.7:53456) 2024-04-21T08:46:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.395+0000 7fc97f7fe640 1 -- 172.21.15.7:0/3295429311 learned_addr learned my addr 172.21.15.7:0/3295429311 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:53.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.395+0000 7fc97effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f20d710 0x564f7f213c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:56504/0 (socket says 172.21.15.7:56504) 2024-04-21T08:46:53.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.396+0000 7fc97f7fe640 1 -- 172.21.15.7:0/3295429311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f20d710 msgr2=0x564f7f213c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.396+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/3295429311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f20d710 0x564f7f213c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.396+0000 7fc97f7fe640 1 -- 172.21.15.7:0/3295429311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20e030 msgr2=0x564f7f214160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.396+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/3295429311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20e030 0x564f7f214160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.396+0000 7fc97f7fe640 1 -- 172.21.15.7:0/3295429311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f7ee843f0 con 0x564f7f216c10 2024-04-21T08:46:53.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.396+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/3295429311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f2146a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc980067c30 tx=0x7fc980097fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee274ecaf4c2960 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.396+0000 7fc962ff5640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:53.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.397+0000 7fc962ff5640 1 -- 172.21.15.7:0/3295429311 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9800b0070 con 0x564f7f216c10 2024-04-21T08:46:53.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.397+0000 7fc987feb740 1 -- 172.21.15.7:0/3295429311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564f7f17fc00 con 0x564f7f216c10 2024-04-21T08:46:53.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.397+0000 7fc962ff5640 1 -- 172.21.15.7:0/3295429311 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc9800ab630 con 0x564f7f216c10 2024-04-21T08:46:53.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.398+0000 7fc962ff5640 1 -- 172.21.15.7:0/3295429311 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc9800c3050 con 0x564f7f216c10 2024-04-21T08:46:53.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.400+0000 7fc987feb740 10 client.16200 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:53.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.400+0000 7fc987feb740 1 -- 172.21.15.7:0/3295429311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564f7ee440f0 con 0x564f7f216c10 2024-04-21T08:46:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.402+0000 7fc962ff5640 1 -- 172.21.15.7:0/3295429311 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564f7f17fc00 con 0x564f7f216c10 2024-04-21T08:46:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.402+0000 7fc962ff5640 1 -- 172.21.15.7:0/3295429311 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fc9800bf4d0 con 0x564f7f216c10 2024-04-21T08:46:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.402+0000 7fc987feb740 10 client.16200 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:53.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.402+0000 7fc987feb740 10 client.16200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:53.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.402+0000 7fc987feb740 20 client.16200 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:53.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.402+0000 7fc987feb740 1 --2- 172.21.15.7:0/3295429311 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564f7f237ce0 0x564f7f23a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.402+0000 7fc987feb740 4 client.16200 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:53.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.402+0000 7fc987feb740 1 -- 172.21.15.7:0/3295429311 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f7f179a70 con 0x564f7f237ce0 2024-04-21T08:46:53.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.403+0000 7fc97effd640 1 --2- 172.21.15.7:0/3295429311 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564f7f237ce0 0x564f7f23a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.407+0000 7fc97effd640 1 --2- 172.21.15.7:0/3295429311 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564f7f237ce0 0x564f7f23a180 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f52fad52c57d60a1 server_cookie=6153f31197a7e3a3 in_seq=0 out_seq=0 2024-04-21T08:46:53.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.409+0000 7fc962ff5640 10 client.16200 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:53.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.410+0000 7fc962ff5640 1 -- 172.21.15.7:0/3295429311 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f7f179a70 con 0x564f7f237ce0 2024-04-21T08:46:53.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.410+0000 7fc962ff5640 10 client.16200 handle_command_reply: tid=0 2024-04-21T08:46:53.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.411+0000 7fc987feb740 1 client.16200 shutdown 2024-04-21T08:46:53.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.412+0000 7fc987feb740 1 -- 172.21.15.7:0/3295429311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 msgr2=0x564f7f2146a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.412+0000 7fc987feb740 1 --2- 172.21.15.7:0/3295429311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f2146a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc980067c30 tx=0x7fc980097fb0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.414+0000 7fc987feb740 1 -- 172.21.15.7:0/3295429311 shutdown_connections 2024-04-21T08:46:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.414+0000 7fc987feb740 1 --2- 172.21.15.7:0/3295429311 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564f7f237ce0 0x564f7f23a180 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.414+0000 7fc987feb740 1 --2- 172.21.15.7:0/3295429311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f2146a0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.414+0000 7fc987feb740 1 --2- 172.21.15.7:0/3295429311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20e030 0x564f7f214160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.414+0000 7fc987feb740 1 --2- 172.21.15.7:0/3295429311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f20d710 0x564f7f213c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.414+0000 7fc987feb740 1 -- 172.21.15.7:0/3295429311 >> 172.21.15.7:0/3295429311 conn(0x564f7f207e40 msgr2=0x564f7f207120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.415+0000 7fc962ff5640 0 client.16200 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.415+0000 7fc987feb740 1 -- 172.21.15.7:0/3295429311 shutdown_connections 2024-04-21T08:46:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.416+0000 7fc987feb740 1 -- 172.21.15.7:0/3295429311 wait complete. 2024-04-21T08:46:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.416+0000 7fc987feb740 20 client.16200 trim_cache size 0 max 16384 2024-04-21T08:46:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.453+0000 7fc987feb740 1 Processor -- start 2024-04-21T08:46:53.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.454+0000 7fc987feb740 1 -- start start 2024-04-21T08:46:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.454+0000 7fc987feb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20d710 0x564f7f216350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.454+0000 7fc987feb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f20e030 0x564f7f233940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.454+0000 7fc987feb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f20e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.454+0000 7fc987feb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc97814fde0 con 0x564f7f216c10 2024-04-21T08:46:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.454+0000 7fc987feb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc978152660 con 0x564f7f20d710 2024-04-21T08:46:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.454+0000 7fc987feb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc978150450 con 0x564f7f20e030 2024-04-21T08:46:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.455+0000 7fc97f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f20e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.455+0000 7fc97f7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f20e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53472/0 (socket says 172.21.15.7:53472) 2024-04-21T08:46:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.455+0000 7fc97f7fe640 1 -- 172.21.15.7:0/4241347901 learned_addr learned my addr 172.21.15.7:0/4241347901 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.455+0000 7fc97f7fe640 1 -- 172.21.15.7:0/4241347901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f20e030 msgr2=0x564f7f233940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:46:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.455+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/4241347901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f20e030 0x564f7f233940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.455+0000 7fc97f7fe640 1 -- 172.21.15.7:0/4241347901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20d710 msgr2=0x564f7f216350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:46:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.455+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/4241347901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20d710 0x564f7f216350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.455+0000 7fc97f7fe640 1 -- 172.21.15.7:0/4241347901 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f7f212c50 con 0x564f7f216c10 2024-04-21T08:46:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.456+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/4241347901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f20e410 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc980067dc0 tx=0x7fc980050260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5b3657a92e23cc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.458+0000 7fc964ff9640 1 -- 172.21.15.7:0/4241347901 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9800b0040 con 0x564f7f216c10 2024-04-21T08:46:53.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.458+0000 7fc964ff9640 1 -- 172.21.15.7:0/4241347901 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc98009c040 con 0x564f7f216c10 2024-04-21T08:46:53.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.458+0000 7fc964ff9640 1 -- 172.21.15.7:0/4241347901 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9800ab430 con 0x564f7f216c10 2024-04-21T08:46:53.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.459+0000 7fc987feb740 1 -- 172.21.15.7:0/4241347901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 msgr2=0x564f7f20e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.459+0000 7fc987feb740 1 --2- 172.21.15.7:0/4241347901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f20e410 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc980067dc0 tx=0x7fc980050260 comp rx=0 tx=0).stop 2024-04-21T08:46:53.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.460+0000 7fc987feb740 1 -- 172.21.15.7:0/4241347901 shutdown_connections 2024-04-21T08:46:53.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.460+0000 7fc987feb740 1 --2- 172.21.15.7:0/4241347901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f216c10 0x564f7f20e410 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.460+0000 7fc987feb740 1 --2- 172.21.15.7:0/4241347901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f20e030 0x564f7f233940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.460+0000 7fc987feb740 1 --2- 172.21.15.7:0/4241347901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20d710 0x564f7f216350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.460+0000 7fc987feb740 1 -- 172.21.15.7:0/4241347901 >> 172.21.15.7:0/4241347901 conn(0x564f7f207e40 msgr2=0x564f7f206890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.461+0000 7fc987feb740 1 -- 172.21.15.7:0/4241347901 shutdown_connections 2024-04-21T08:46:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.462+0000 7fc987feb740 1 -- 172.21.15.7:0/4241347901 wait complete. 2024-04-21T08:46:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.463+0000 7fc987feb740 1 -- start start 2024-04-21T08:46:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.463+0000 7fc987feb740 1 Processor -- start 2024-04-21T08:46:53.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.464+0000 7fc987feb740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20d710 0x564f7f2300f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.464+0000 7fc987feb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20e030 0x564f7f210530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.464+0000 7fc987feb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f216c10 0x564f7f210a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.464+0000 7fc987feb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc978150b20 con 0x564f7f20e030 2024-04-21T08:46:53.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.464+0000 7fc987feb740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc978151f90 con 0x564f7f20d710 2024-04-21T08:46:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.464+0000 7fc987feb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc9781511f0 con 0x564f7f216c10 2024-04-21T08:46:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.464+0000 7fc987feb740 10 client.0 _ll_register_callbacks cb 0x7fc9800ae070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.465+0000 7fc97e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20e030 0x564f7f210530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.465+0000 7fc97e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20e030 0x564f7f210530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53480/0 (socket says 172.21.15.7:53480) 2024-04-21T08:46:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.465+0000 7fc97e7fc640 1 -- 172.21.15.7:0/4270493814 learned_addr learned my addr 172.21.15.7:0/4270493814 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.465+0000 7fc97f7fe640 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f216c10 0x564f7f210a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.465+0000 7fc97effd640 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20d710 0x564f7f2300f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.465+0000 7fc97e7fc640 1 -- 172.21.15.7:0/4270493814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f216c10 msgr2=0x564f7f210a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.465+0000 7fc97e7fc640 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f216c10 0x564f7f210a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.465+0000 7fc97e7fc640 1 -- 172.21.15.7:0/4270493814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20d710 msgr2=0x564f7f2300f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.465+0000 7fc97e7fc640 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20d710 0x564f7f2300f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.465+0000 7fc97e7fc640 1 -- 172.21.15.7:0/4270493814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f7f20de60 con 0x564f7f20e030 2024-04-21T08:46:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.466+0000 7fc97e7fc640 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20e030 0x564f7f210530 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc9740d6300 tx=0x7fc97401e6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b275650328a2801c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.466+0000 7fc962ff5640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.466+0000 7fc962ff5640 1 -- 172.21.15.7:0/4270493814 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974010310 con 0x564f7f20e030 2024-04-21T08:46:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.466+0000 7fc962ff5640 1 -- 172.21.15.7:0/4270493814 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc9740d6520 con 0x564f7f20e030 2024-04-21T08:46:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.467+0000 7fc987feb740 1 -- 172.21.15.7:0/4270493814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc98009c040 con 0x564f7f20e030 2024-04-21T08:46:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.467+0000 7fc962ff5640 1 -- 172.21.15.7:0/4270493814 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc9740d6720 con 0x564f7f20e030 2024-04-21T08:46:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.468+0000 7fc987feb740 10 client.16206 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.468+0000 7fc987feb740 1 -- 172.21.15.7:0/4270493814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564f7f29df80 con 0x564f7f20e030 2024-04-21T08:46:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.468+0000 7fc962ff5640 1 -- 172.21.15.7:0/4270493814 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc98009c040 con 0x564f7f20e030 2024-04-21T08:46:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.468+0000 7fc962ff5640 1 -- 172.21.15.7:0/4270493814 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fc9740a3f40 con 0x564f7f20e030 2024-04-21T08:46:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.468+0000 7fc987feb740 10 client.16206 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.468+0000 7fc987feb740 10 client.16206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.468+0000 7fc987feb740 20 client.16206 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.468+0000 7fc987feb740 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564f7f29e780 0x564f7f2a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.468+0000 7fc987feb740 4 client.16206 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:46:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.468+0000 7fc987feb740 1 -- 172.21.15.7:0/4270493814 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f7f0f5590 con 0x564f7f29e780 2024-04-21T08:46:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.469+0000 7fc97effd640 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564f7f29e780 0x564f7f2a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.472+0000 7fc97effd640 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564f7f29e780 0x564f7f2a0ba0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1687d424051067c server_cookie=b4511bb1a0a09bf5 in_seq=0 out_seq=0 2024-04-21T08:46:53.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.472+0000 7fc962ff5640 10 client.16206 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:53.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.474+0000 7fc962ff5640 1 -- 172.21.15.7:0/4270493814 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564f7f0f5590 con 0x564f7f29e780 2024-04-21T08:46:53.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.474+0000 7fc962ff5640 10 client.16206 handle_command_reply: tid=0 2024-04-21T08:46:53.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.474+0000 7fc987feb740 1 client.16206 shutdown 2024-04-21T08:46:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.476+0000 7fc987feb740 1 -- 172.21.15.7:0/4270493814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20e030 msgr2=0x564f7f210530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.476+0000 7fc987feb740 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20e030 0x564f7f210530 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc9740d6300 tx=0x7fc97401e6e0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.478+0000 7fc987feb740 1 -- 172.21.15.7:0/4270493814 shutdown_connections 2024-04-21T08:46:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.478+0000 7fc987feb740 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564f7f29e780 0x564f7f2a0ba0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.478+0000 7fc987feb740 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564f7f216c10 0x564f7f210a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.478+0000 7fc987feb740 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564f7f20e030 0x564f7f210530 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.478+0000 7fc987feb740 1 --2- 172.21.15.7:0/4270493814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564f7f20d710 0x564f7f2300f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.478+0000 7fc987feb740 1 -- 172.21.15.7:0/4270493814 >> 172.21.15.7:0/4270493814 conn(0x564f7f207e40 msgr2=0x564f7f207120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.478+0000 7fc962ff5640 0 client.16206 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.479+0000 7fc987feb740 1 -- 172.21.15.7:0/4270493814 shutdown_connections 2024-04-21T08:46:53.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.482+0000 7fc987feb740 1 -- 172.21.15.7:0/4270493814 wait complete. 2024-04-21T08:46:53.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.482+0000 7fc987feb740 20 client.16206 trim_cache size 0 max 16384 2024-04-21T08:46:53.483 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:46:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:46:53.483 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:46:53.483 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:46:53.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.484+0000 7fc97ffff640 1 -- 172.21.15.7:0/2303529800 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc9680a3ea0 msgr2=0x7fc9680a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.484+0000 7fc97ffff640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc9680a3ea0 0x7fc9680a6360 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc9800679b0 tx=0x7fc9800a8000 comp rx=0 tx=0).stop 2024-04-21T08:46:53.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.484+0000 7fc97ffff640 1 -- 172.21.15.7:0/2303529800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9780957e0 msgr2=0x7fc97813b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.484+0000 7fc97ffff640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9780957e0 0x7fc97813b500 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc97400b590 tx=0x7fc97400ba50 comp rx=0 tx=0).stop 2024-04-21T08:46:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.484+0000 7fc97ffff640 1 -- 172.21.15.7:0/2303529800 shutdown_connections 2024-04-21T08:46:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.484+0000 7fc97ffff640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc978099bb0 0x7fc97813f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.484+0000 7fc97ffff640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9780957e0 0x7fc97813b500 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.484+0000 7fc97ffff640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc9680a3ea0 0x7fc9680a6360 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.484+0000 7fc97ffff640 1 --2- 172.21.15.7:0/2303529800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc978094e40 0x7fc97813afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.484+0000 7fc97ffff640 1 -- 172.21.15.7:0/2303529800 >> 172.21.15.7:0/2303529800 conn(0x7fc97800a730 msgr2=0x7fc978092870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.485+0000 7fc97ffff640 1 -- 172.21.15.7:0/2303529800 shutdown_connections 2024-04-21T08:46:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.485+0000 7fc97ffff640 1 -- 172.21.15.7:0/2303529800 wait complete. 2024-04-21T08:46:53.505 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:46:53.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:46:53.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.662+0000 7f7dc8869640 1 -- 172.21.15.7:0/1031807490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4101a20 msgr2=0x7f7dc4105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.662+0000 7f7dc8869640 1 --2- 172.21.15.7:0/1031807490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4101a20 0x7f7dc4105a60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7db8009900 tx=0x7f7db802bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:53.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.662+0000 7f7dc8869640 1 -- 172.21.15.7:0/1031807490 shutdown_connections 2024-04-21T08:46:53.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.662+0000 7f7dc8869640 1 --2- 172.21.15.7:0/1031807490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dc4106140 0x7f7dc410dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.662+0000 7f7dc8869640 1 --2- 172.21.15.7:0/1031807490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4101a20 0x7f7dc4105a60 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.662+0000 7f7dc8869640 1 --2- 172.21.15.7:0/1031807490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc4101070 0x7f7dc4101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.662+0000 7f7dc8869640 1 -- 172.21.15.7:0/1031807490 >> 172.21.15.7:0/1031807490 conn(0x7f7dc40771d0 msgr2=0x7f7dc40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.662+0000 7f7dc8869640 1 -- 172.21.15.7:0/1031807490 shutdown_connections 2024-04-21T08:46:53.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.662+0000 7f7dc8869640 1 -- 172.21.15.7:0/1031807490 wait complete. 2024-04-21T08:46:53.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.663+0000 7f7dc8869640 1 Processor -- start 2024-04-21T08:46:53.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.663+0000 7f7dc8869640 1 -- start start 2024-04-21T08:46:53.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc8869640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dc4101070 0x7f7dc410e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc8869640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4106140 0x7f7dc410e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc8869640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc410ef20 0x7f7dc41a6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc8869640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc41121b0 con 0x7f7dc4106140 2024-04-21T08:46:53.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc8869640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4112030 con 0x7f7dc410ef20 2024-04-21T08:46:53.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc8869640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4112330 con 0x7f7dc4101070 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dc4101070 0x7f7dc410e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dc4101070 0x7f7dc410e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:56538/0 (socket says 172.21.15.7:56538) 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc259c640 1 -- 172.21.15.7:0/837910991 learned_addr learned my addr 172.21.15.7:0/837910991 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc1d9b640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4106140 0x7f7dc410e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc2d9d640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc410ef20 0x7f7dc41a6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc259c640 1 -- 172.21.15.7:0/837910991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc410ef20 msgr2=0x7f7dc41a6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc259c640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc410ef20 0x7f7dc41a6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc259c640 1 -- 172.21.15.7:0/837910991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4106140 msgr2=0x7f7dc410e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc259c640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4106140 0x7f7dc410e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc259c640 1 -- 172.21.15.7:0/837910991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc41a7020 con 0x7f7dc4101070 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.664+0000 7f7dc259c640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dc4101070 0x7f7dc410e4a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7db400e9f0 tx=0x7f7db400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.665+0000 7f7db37fe640 1 -- 172.21.15.7:0/837910991 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db400cd80 con 0x7f7dc4101070 2024-04-21T08:46:53.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.665+0000 7f7dc8869640 1 -- 172.21.15.7:0/837910991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dc4198500 con 0x7f7dc4101070 2024-04-21T08:46:53.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.665+0000 7f7db37fe640 1 -- 172.21.15.7:0/837910991 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7db40108e0 con 0x7f7dc4101070 2024-04-21T08:46:53.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.665+0000 7f7db37fe640 1 -- 172.21.15.7:0/837910991 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4018980 con 0x7f7dc4101070 2024-04-21T08:46:53.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.665+0000 7f7dc8869640 1 -- 172.21.15.7:0/837910991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dc4198a20 con 0x7f7dc4101070 2024-04-21T08:46:53.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.666+0000 7f7dc8869640 1 -- 172.21.15.7:0/837910991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dc4068e00 con 0x7f7dc4101070 2024-04-21T08:46:53.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.667+0000 7f7db37fe640 1 -- 172.21.15.7:0/837910991 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f7db4010430 con 0x7f7dc4101070 2024-04-21T08:46:53.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.668+0000 7f7db37fe640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7dac0a3d50 0x7f7dac0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.668+0000 7f7dc1d9b640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7dac0a3d50 0x7f7dac0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.668+0000 7f7db37fe640 1 -- 172.21.15.7:0/837910991 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f7db40d1040 con 0x7f7dc4101070 2024-04-21T08:46:53.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.668+0000 7f7dc1d9b640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7dac0a3d50 0x7f7dac0a6210 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7db8006010 tx=0x7f7db802c690 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.672+0000 7f7db37fe640 1 -- 172.21.15.7:0/837910991 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7db409d2f0 con 0x7f7dc4101070 2024-04-21T08:46:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.904+0000 7f7dc8869640 1 -- 172.21.15.7:0/837910991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7dc41093d0 con 0x7f7dc4101070 2024-04-21T08:46:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.906+0000 7f7db37fe640 1 -- 172.21.15.7:0/837910991 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7f7db409cbf0 con 0x7f7dc4101070 2024-04-21T08:46:53.906 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:46:53.907 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:46:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:46:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 -- 172.21.15.7:0/837910991 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7dac0a3d50 msgr2=0x7f7dac0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7dac0a3d50 0x7f7dac0a6210 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7db8006010 tx=0x7f7db802c690 comp rx=0 tx=0).stop 2024-04-21T08:46:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 -- 172.21.15.7:0/837910991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dc4101070 msgr2=0x7f7dc410e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dc4101070 0x7f7dc410e4a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7db400e9f0 tx=0x7f7db400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 -- 172.21.15.7:0/837910991 shutdown_connections 2024-04-21T08:46:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7dc410ef20 0x7f7dc41a6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4106140 0x7f7dc410e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7dac0a3d50 0x7f7dac0a6210 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 --2- 172.21.15.7:0/837910991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7dc4101070 0x7f7dc410e4a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 -- 172.21.15.7:0/837910991 >> 172.21.15.7:0/837910991 conn(0x7f7dc40771d0 msgr2=0x7f7dc4107310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 -- 172.21.15.7:0/837910991 shutdown_connections 2024-04-21T08:46:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:53.908+0000 7f7dc8869640 1 -- 172.21.15.7:0/837910991 wait complete. 2024-04-21T08:46:53.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:46:54.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.022+0000 7fb02c5ef640 1 -- 172.21.15.7:0/3363703737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb02410c500 msgr2=0x7fb02410e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:54.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.022+0000 7fb02c5ef640 1 --2- 172.21.15.7:0/3363703737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb02410c500 0x7fb02410e9a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb01c009ca0 tx=0x7fb01c02c180 comp rx=0 tx=0).stop 2024-04-21T08:46:54.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 -- 172.21.15.7:0/3363703737 shutdown_connections 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 --2- 172.21.15.7:0/3363703737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb02410c500 0x7fb02410e9a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 --2- 172.21.15.7:0/3363703737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb024103ae0 0x7fb02410bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 --2- 172.21.15.7:0/3363703737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0241031c0 0x7fb0241035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 -- 172.21.15.7:0/3363703737 >> 172.21.15.7:0/3363703737 conn(0x7fb024078000 msgr2=0x7fb024078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 -- 172.21.15.7:0/3363703737 shutdown_connections 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 -- 172.21.15.7:0/3363703737 wait complete. 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 Processor -- start 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 -- start start 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0241031c0 0x7fb024198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb024103ae0 0x7fb024198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb02410c500 0x7fb02419cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb0241121f0 con 0x7fb02410c500 2024-04-21T08:46:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb024112070 con 0x7fb0241031c0 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.023+0000 7fb02c5ef640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb024112370 con 0x7fb024103ae0 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02ab8c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb02410c500 0x7fb02419cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02ab8c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb02410c500 0x7fb02419cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53518/0 (socket says 172.21.15.7:53518) 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb029b8a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb024103ae0 0x7fb024198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02ab8c640 1 -- 172.21.15.7:0/2023146541 learned_addr learned my addr 172.21.15.7:0/2023146541 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02a38b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0241031c0 0x7fb024198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02ab8c640 1 -- 172.21.15.7:0/2023146541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb024103ae0 msgr2=0x7fb024198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02ab8c640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb024103ae0 0x7fb024198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02ab8c640 1 -- 172.21.15.7:0/2023146541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0241031c0 msgr2=0x7fb024198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02ab8c640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0241031c0 0x7fb024198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02ab8c640 1 -- 172.21.15.7:0/2023146541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02419d710 con 0x7fb02410c500 2024-04-21T08:46:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02ab8c640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb02410c500 0x7fb02419cfe0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb01c02c790 tx=0x7fb01c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb01b7fe640 1 -- 172.21.15.7:0/2023146541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c039040 con 0x7fb02410c500 2024-04-21T08:46:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb01b7fe640 1 -- 172.21.15.7:0/2023146541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb01c044070 con 0x7fb02410c500 2024-04-21T08:46:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02c5ef640 1 -- 172.21.15.7:0/2023146541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb02419d9a0 con 0x7fb02410c500 2024-04-21T08:46:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb01b7fe640 1 -- 172.21.15.7:0/2023146541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c0327c0 con 0x7fb02410c500 2024-04-21T08:46:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.024+0000 7fb02c5ef640 1 -- 172.21.15.7:0/2023146541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0241a5290 con 0x7fb02410c500 2024-04-21T08:46:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.026+0000 7fb01b7fe640 1 -- 172.21.15.7:0/2023146541 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fb01c004240 con 0x7fb02410c500 2024-04-21T08:46:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.027+0000 7fb02f1d7740 1 Processor -- start 2024-04-21T08:46:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.027+0000 7fb02f1d7740 1 -- start start 2024-04-21T08:46:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.027+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca10f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.027+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca118430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.027+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca11ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.027+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55eeca0741f0 con 0x55eeca118a00 2024-04-21T08:46:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.027+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55eeca0114b0 con 0x55eeca10fe80 2024-04-21T08:46:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.027+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55eeca0701b0 con 0x55eeca10f4d0 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb01b7fe640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0100a3da0 0x7fb0100a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb02a38b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca118430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb029b8a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca11ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb02a38b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca118430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:48332/0 (socket says 172.21.15.7:48332) 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb02a38b640 1 -- 172.21.15.7:0/2842247733 learned_addr learned my addr 172.21.15.7:0/2842247733 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb029b8a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca11ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53528/0 (socket says 172.21.15.7:53528) 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb029b8a640 1 --2- 172.21.15.7:0/2842247733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca10f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb02ab8c640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0100a3da0 0x7fb0100a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb01b7fe640 1 -- 172.21.15.7:0/2023146541 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb01c0f2ec0 con 0x7fb02410c500 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb02a38b640 1 -- 172.21.15.7:0/2842247733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 msgr2=0x55eeca10f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb02a38b640 1 --2- 172.21.15.7:0/2842247733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca10f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb02a38b640 1 -- 172.21.15.7:0/2842247733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 msgr2=0x55eeca11ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:54.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb02a38b640 1 --2- 172.21.15.7:0/2842247733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca11ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb02a38b640 1 -- 172.21.15.7:0/2842247733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeca0031b0 con 0x55eeca10fe80 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb02a38b640 1 --2- 172.21.15.7:0/2842247733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca118430 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb020009a70 tx=0x7fb020009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da04add26a506678 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb02ab8c640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0100a3da0 0x7fb0100a6260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0f0820 tx=0x7fb01c0df040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb018bfe640 1 -- 172.21.15.7:0/2842247733 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb020017070 con 0x55eeca10fe80 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb018bfe640 1 -- 172.21.15.7:0/2842247733 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb020013470 con 0x55eeca10fe80 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.028+0000 7fb018bfe640 1 -- 172.21.15.7:0/2842247733 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb020012630 con 0x55eeca10fe80 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.029+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2842247733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 msgr2=0x55eeca118430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.029+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2842247733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca118430 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb020009a70 tx=0x7fb020009f30 comp rx=0 tx=0).stop 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.029+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2842247733 shutdown_connections 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.029+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2842247733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca11ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.029+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2842247733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca118430 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.029+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2842247733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.029+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2842247733 >> 172.21.15.7:0/2842247733 conn(0x55eeca109ba0 msgr2=0x55eeca109f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.029+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2842247733 shutdown_connections 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.029+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2842247733 wait complete. 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb02f1d7740 1 -- start start 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb02f1d7740 1 Processor -- start 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10f4d0 0x55eeca118270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10fe80 0x55eeca115600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca115b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55eeca11b970 con 0x55eeca118a00 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55eeca02d500 con 0x55eeca10f4d0 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55eeca11baf0 con 0x55eeca10fe80 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb02f1d7740 10 client.0 _ll_register_callbacks cb 0x55eec9ffbb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb029b8a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca115b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb029b8a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca115b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53544/0 (socket says 172.21.15.7:53544) 2024-04-21T08:46:54.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb029b8a640 1 -- 172.21.15.7:0/1540793622 learned_addr learned my addr 172.21.15.7:0/1540793622 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb029b8a640 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10f4d0 0x55eeca118270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb02a38b640 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10fe80 0x55eeca115600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb029b8a640 1 -- 172.21.15.7:0/1540793622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10fe80 msgr2=0x55eeca115600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb029b8a640 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10fe80 0x55eeca115600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb029b8a640 1 -- 172.21.15.7:0/1540793622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10f4d0 msgr2=0x55eeca118270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb029b8a640 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10f4d0 0x55eeca118270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb029b8a640 1 -- 172.21.15.7:0/1540793622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eec9d863f0 con 0x55eeca118a00 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.030+0000 7fb02a38b640 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10fe80 0x55eeca115600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb029b8a640 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca115b40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fb014012c00 tx=0x7fb014004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3554f8f565f65178 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb00dffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb00dffb640 1 -- 172.21.15.7:0/1540793622 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01400eca0 con 0x55eeca118a00 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb00dffb640 1 -- 172.21.15.7:0/1540793622 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb01400ee40 con 0x55eeca118a00 2024-04-21T08:46:54.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb02f1d7740 1 -- 172.21.15.7:0/1540793622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb020013470 con 0x55eeca118a00 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb00dffb640 1 -- 172.21.15.7:0/1540793622 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb01402c050 con 0x55eeca118a00 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb02f1d7740 10 client.16224 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb02f1d7740 1 -- 172.21.15.7:0/1540793622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55eec9e507d0 con 0x55eeca118a00 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb00dffb640 1 -- 172.21.15.7:0/1540793622 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb020013470 con 0x55eeca118a00 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb00dffb640 1 -- 172.21.15.7:0/1540793622 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb014010430 con 0x55eeca118a00 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb02f1d7740 10 client.16224 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb02f1d7740 10 client.16224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb02f1d7740 20 client.16224 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eeca128ba0 0x55eeca12aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb02f1d7740 4 client.16224 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.031+0000 7fb02f1d7740 1 -- 172.21.15.7:0/1540793622 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eeca013930 con 0x55eeca128ba0 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.032+0000 7fb02a38b640 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eeca128ba0 0x55eeca12aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.032+0000 7fb00dffb640 10 client.16224 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.032+0000 7fb02a38b640 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eeca128ba0 0x55eeca12aff0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=870bfea0acc611f6 server_cookie=efcd70ec722f71fa in_seq=0 out_seq=0 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.036+0000 7fb00dffb640 1 -- 172.21.15.7:0/1540793622 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eeca013930 con 0x55eeca128ba0 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.036+0000 7fb00dffb640 10 client.16224 handle_command_reply: tid=0 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.036+0000 7fb02f1d7740 1 client.16224 shutdown 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb02f1d7740 1 -- 172.21.15.7:0/1540793622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 msgr2=0x55eeca115b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca115b40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fb014012c00 tx=0x7fb014004730 comp rx=0 tx=0).stop 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb02f1d7740 1 -- 172.21.15.7:0/1540793622 shutdown_connections 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eeca128ba0 0x55eeca12aff0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca115b40 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10fe80 0x55eeca115600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb00dffb640 0 client.16224 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/1540793622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10f4d0 0x55eeca118270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb02f1d7740 1 -- 172.21.15.7:0/1540793622 >> 172.21.15.7:0/1540793622 conn(0x55eeca109ba0 msgr2=0x55eeca10b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:54.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb02f1d7740 1 -- 172.21.15.7:0/1540793622 shutdown_connections 2024-04-21T08:46:54.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb02f1d7740 1 -- 172.21.15.7:0/1540793622 wait complete. 2024-04-21T08:46:54.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.037+0000 7fb02f1d7740 20 client.16224 trim_cache size 0 max 16384 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.050+0000 7fb02f1d7740 1 Processor -- start 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.050+0000 7fb02f1d7740 1 -- start start 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.050+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca124860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.050+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca10fe80 0x55eeca114760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.050+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca118a00 0x55eeca108a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.050+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb0241ad4a0 con 0x55eeca10fe80 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.050+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0241afd20 con 0x55eeca118a00 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.050+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb0241adb10 con 0x55eeca10f4d0 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb02a38b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca10fe80 0x55eeca114760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb029b8a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca124860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb02a38b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca10fe80 0x55eeca114760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53552/0 (socket says 172.21.15.7:53552) 2024-04-21T08:46:54.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb029b8a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca124860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:56596/0 (socket says 172.21.15.7:56596) 2024-04-21T08:46:54.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb02a38b640 1 -- 172.21.15.7:0/2581238415 learned_addr learned my addr 172.21.15.7:0/2581238415 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:54.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb029b8a640 1 --2- 172.21.15.7:0/2581238415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca118a00 0x55eeca108a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb02a38b640 1 -- 172.21.15.7:0/2581238415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 msgr2=0x55eeca124860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:54.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb02a38b640 1 --2- 172.21.15.7:0/2581238415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca124860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb02a38b640 1 -- 172.21.15.7:0/2581238415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca118a00 msgr2=0x55eeca108a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:54.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb02a38b640 1 --2- 172.21.15.7:0/2581238415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca118a00 0x55eeca108a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb02a38b640 1 -- 172.21.15.7:0/2581238415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeca105470 con 0x55eeca10fe80 2024-04-21T08:46:54.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb02a38b640 1 --2- 172.21.15.7:0/2581238415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca10fe80 0x55eeca114760 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb020004810 tx=0x7fb020009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63c1fc851a8461c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:54.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb00ffff640 1 -- 172.21.15.7:0/2581238415 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02002b070 con 0x55eeca10fe80 2024-04-21T08:46:54.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb00ffff640 1 -- 172.21.15.7:0/2581238415 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0200258f0 con 0x55eeca10fe80 2024-04-21T08:46:54.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.051+0000 7fb00ffff640 1 -- 172.21.15.7:0/2581238415 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb020017040 con 0x55eeca10fe80 2024-04-21T08:46:54.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2581238415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca10fe80 msgr2=0x55eeca114760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:54.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2581238415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca10fe80 0x55eeca114760 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb020004810 tx=0x7fb020009110 comp rx=0 tx=0).stop 2024-04-21T08:46:54.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2581238415 shutdown_connections 2024-04-21T08:46:54.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2581238415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca118a00 0x55eeca108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2581238415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca10fe80 0x55eeca114760 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2581238415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca124860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2581238415 >> 172.21.15.7:0/2581238415 conn(0x55eeca109ba0 msgr2=0x55eeca108370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:54.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2581238415 shutdown_connections 2024-04-21T08:46:54.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2581238415 wait complete. 2024-04-21T08:46:54.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 -- start start 2024-04-21T08:46:54.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 Processor -- start 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca187430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca187970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca11bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb0241ae1e0 con 0x55eeca118a00 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0241af650 con 0x55eeca10fe80 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.052+0000 7fb02f1d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb0241ae8b0 con 0x55eeca10f4d0 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02f1d7740 10 client.0 _ll_register_callbacks cb 0x7fb014012a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02a38b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca187970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb029b8a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca11bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02a38b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca187970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:48368/0 (socket says 172.21.15.7:48368) 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02a38b640 1 -- 172.21.15.7:0/2589544458 learned_addr learned my addr 172.21.15.7:0/2589544458 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb029b8a640 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca187430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02a38b640 1 -- 172.21.15.7:0/2589544458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 msgr2=0x55eeca187430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02a38b640 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca187430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02a38b640 1 -- 172.21.15.7:0/2589544458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 msgr2=0x55eeca11bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02a38b640 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca11bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02a38b640 1 -- 172.21.15.7:0/2589544458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eeca11c640 con 0x55eeca10fe80 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02a38b640 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca187970 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb020006aa0 tx=0x7fb0200134f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d4852d7394cfaf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb00dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb00dffb640 1 -- 172.21.15.7:0/2589544458 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02002b070 con 0x55eeca10fe80 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2589544458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0200258f0 con 0x55eeca10fe80 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb00dffb640 1 -- 172.21.15.7:0/2589544458 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb020025640 con 0x55eeca10fe80 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb00dffb640 1 -- 172.21.15.7:0/2589544458 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb02000edd0 con 0x55eeca10fe80 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02f1d7740 10 client.25796 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.053+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2589544458 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55eeca18ebd0 con 0x55eeca10fe80 2024-04-21T08:46:54.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.054+0000 7fb00dffb640 1 -- 172.21.15.7:0/2589544458 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0200258f0 con 0x55eeca10fe80 2024-04-21T08:46:54.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.054+0000 7fb00dffb640 1 -- 172.21.15.7:0/2589544458 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb02001c740 con 0x55eeca10fe80 2024-04-21T08:46:54.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.054+0000 7fb02f1d7740 10 client.25796 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:54.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.054+0000 7fb02f1d7740 10 client.25796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:54.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.054+0000 7fb02f1d7740 20 client.25796 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:54.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.054+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eeca18ef60 0x55eeca191380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:54.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.054+0000 7fb02f1d7740 4 client.25796 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:46:54.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.054+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2589544458 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eeca081c10 con 0x55eeca18ef60 2024-04-21T08:46:54.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.054+0000 7fb029b8a640 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eeca18ef60 0x55eeca191380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:54.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.054+0000 7fb00dffb640 10 client.25796 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.054+0000 7fb029b8a640 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eeca18ef60 0x55eeca191380 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9759f59ec8e7d27 server_cookie=4824a1539925ad3c in_seq=0 out_seq=0 2024-04-21T08:46:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.057+0000 7fb00dffb640 1 -- 172.21.15.7:0/2589544458 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eeca081c10 con 0x55eeca18ef60 2024-04-21T08:46:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.057+0000 7fb00dffb640 10 client.25796 handle_command_reply: tid=0 2024-04-21T08:46:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.057+0000 7fb02f1d7740 1 client.25796 shutdown 2024-04-21T08:46:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.057+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2589544458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 msgr2=0x55eeca187970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.057+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca187970 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb020006aa0 tx=0x7fb0200134f0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2589544458 shutdown_connections 2024-04-21T08:46:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eeca18ef60 0x55eeca191380 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eeca118a00 0x55eeca11bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb00dffb640 0 client.25796 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eeca10fe80 0x55eeca187970 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02f1d7740 1 --2- 172.21.15.7:0/2589544458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eeca10f4d0 0x55eeca187430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2589544458 >> 172.21.15.7:0/2589544458 conn(0x55eeca109ba0 msgr2=0x55eeca110260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2589544458 shutdown_connections 2024-04-21T08:46:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02f1d7740 1 -- 172.21.15.7:0/2589544458 wait complete. 2024-04-21T08:46:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02f1d7740 20 client.25796 trim_cache size 0 max 16384 2024-04-21T08:46:54.063 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:46:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 -- 172.21.15.7:0/2023146541 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0100a3da0 msgr2=0x7fb0100a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0100a3da0 0x7fb0100a6260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0f0820 tx=0x7fb01c0df040 comp rx=0 tx=0).stop 2024-04-21T08:46:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 -- 172.21.15.7:0/2023146541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb02410c500 msgr2=0x7fb02419cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb02410c500 0x7fb02419cfe0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb01c02c790 tx=0x7fb01c003c20 comp rx=0 tx=0).stop 2024-04-21T08:46:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 -- 172.21.15.7:0/2023146541 shutdown_connections 2024-04-21T08:46:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb02410c500 0x7fb02419cfe0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0100a3da0 0x7fb0100a6260 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb024103ae0 0x7fb024198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 --2- 172.21.15.7:0/2023146541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0241031c0 0x7fb024198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 -- 172.21.15.7:0/2023146541 >> 172.21.15.7:0/2023146541 conn(0x7fb024078000 msgr2=0x7fb0240fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 -- 172.21.15.7:0/2023146541 shutdown_connections 2024-04-21T08:46:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:54.058+0000 7fb02c5ef640 1 -- 172.21.15.7:0/2023146541 wait complete. 2024-04-21T08:46:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:54 smithi016 ceph-mon[31259]: pgmap v70: 129 pgs: 129 active+clean; 212 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 733 KiB/s wr, 478 op/s 2024-04-21T08:46:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:54 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/837910991' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:54.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:54 smithi077 ceph-mon[31615]: pgmap v70: 129 pgs: 129 active+clean; 212 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 733 KiB/s wr, 478 op/s 2024-04-21T08:46:54.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:54 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/837910991' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:54.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:54 smithi007 ceph-mon[28477]: pgmap v70: 129 pgs: 129 active+clean; 212 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 733 KiB/s wr, 478 op/s 2024-04-21T08:46:54.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:54 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/837910991' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:55.070 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:46:55.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:46:55.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:46:55.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.183+0000 7fb7bd596640 1 -- 172.21.15.7:0/2454296604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b8101090 msgr2=0x7fb7b8101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.183+0000 7fb7bd596640 1 --2- 172.21.15.7:0/2454296604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b8101090 0x7fb7b8101470 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009900 tx=0x7fb7a802bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:55.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.183+0000 7fb7bd596640 1 -- 172.21.15.7:0/2454296604 shutdown_connections 2024-04-21T08:46:55.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.183+0000 7fb7bd596640 1 --2- 172.21.15.7:0/2454296604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b8106160 0x7fb7b810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.183+0000 7fb7bd596640 1 --2- 172.21.15.7:0/2454296604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b8101a40 0x7fb7b8105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.183+0000 7fb7bd596640 1 --2- 172.21.15.7:0/2454296604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b8101090 0x7fb7b8101470 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.183+0000 7fb7bd596640 1 -- 172.21.15.7:0/2454296604 >> 172.21.15.7:0/2454296604 conn(0x7fb7b8078000 msgr2=0x7fb7b8078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.183+0000 7fb7bd596640 1 -- 172.21.15.7:0/2454296604 shutdown_connections 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.183+0000 7fb7bd596640 1 -- 172.21.15.7:0/2454296604 wait complete. 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7bd596640 1 Processor -- start 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7bd596640 1 -- start start 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7bd596640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b8101090 0x7fb7b819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7bd596640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b8106160 0x7fb7b819e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7bd596640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b81985b0 0x7fb7b8198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7bd596640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b81110c0 con 0x7fb7b8106160 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7bd596640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b8110f40 con 0x7fb7b81985b0 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7bd596640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b8111240 con 0x7fb7b8101090 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7affff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b8106160 0x7fb7b819e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7affff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b8106160 0x7fb7b819e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40736/0 (socket says 172.21.15.7:40736) 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7affff640 1 -- 172.21.15.7:0/1538425570 learned_addr learned my addr 172.21.15.7:0/1538425570 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:55.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7b77fe640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b81985b0 0x7fb7b8198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.184+0000 7fb7b6ffd640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b8101090 0x7fb7b819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7affff640 1 -- 172.21.15.7:0/1538425570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b8101090 msgr2=0x7fb7b819e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7affff640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b8101090 0x7fb7b819e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7affff640 1 -- 172.21.15.7:0/1538425570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b81985b0 msgr2=0x7fb7b8198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7affff640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b81985b0 0x7fb7b8198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7affff640 1 -- 172.21.15.7:0/1538425570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b81990d0 con 0x7fb7b8106160 2024-04-21T08:46:55.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7b6ffd640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b8101090 0x7fb7b819e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:55.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7affff640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b8106160 0x7fb7b819e940 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000d740 tx=0x7fb7a000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7b4ff9640 1 -- 172.21.15.7:0/1538425570 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0017070 con 0x7fb7b8106160 2024-04-21T08:46:55.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7b4ff9640 1 -- 172.21.15.7:0/1538425570 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7a000de50 con 0x7fb7b8106160 2024-04-21T08:46:55.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7bd596640 1 -- 172.21.15.7:0/1538425570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b81993c0 con 0x7fb7b8106160 2024-04-21T08:46:55.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7b4ff9640 1 -- 172.21.15.7:0/1538425570 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0004ce0 con 0x7fb7b8106160 2024-04-21T08:46:55.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.185+0000 7fb7bd596640 1 -- 172.21.15.7:0/1538425570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b81a55c0 con 0x7fb7b8106160 2024-04-21T08:46:55.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.187+0000 7fb7b4ff9640 1 -- 172.21.15.7:0/1538425570 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fb7a0004e80 con 0x7fb7b8106160 2024-04-21T08:46:55.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.187+0000 7fb7c0185740 1 Processor -- start 2024-04-21T08:46:55.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.187+0000 7fb7c0185740 1 -- start start 2024-04-21T08:46:55.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 0x55ea2fe17cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe18290 0x55ea2fe20840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 0x55ea2fe23260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7c0185740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ea2fd7bf00 con 0x55ea2fe20e10 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7c0185740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea2fd19590 con 0x55ea2fe178e0 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7c0185740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ea2fd77ec0 con 0x55ea2fe18290 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7b4ff9640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3d20 0x7fb7a40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7b77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 0x55ea2fe17cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7affff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 0x55ea2fe23260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7b77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 0x55ea2fe17cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33326/0 (socket says 172.21.15.7:33326) 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7b77fe640 1 -- 172.21.15.7:0/1379696180 learned_addr learned my addr 172.21.15.7:0/1379696180 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7b77fe640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3d20 0x7fb7a40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7b6ffd640 1 --2- 172.21.15.7:0/1379696180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe18290 0x55ea2fe20840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.188+0000 7fb7b4ff9640 1 -- 172.21.15.7:0/1538425570 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb7a00d0560 con 0x7fb7b8106160 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.189+0000 7fb7b77fe640 1 -- 172.21.15.7:0/1379696180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe18290 msgr2=0x55ea2fe20840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.189+0000 7fb7b77fe640 1 --2- 172.21.15.7:0/1379696180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe18290 0x55ea2fe20840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.189+0000 7fb7b77fe640 1 -- 172.21.15.7:0/1379696180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 msgr2=0x55ea2fe23260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.189+0000 7fb7b77fe640 1 --2- 172.21.15.7:0/1379696180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 0x55ea2fe23260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.189+0000 7fb7b77fe640 1 -- 172.21.15.7:0/1379696180 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea2fb567e0 con 0x55ea2fe178e0 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.189+0000 7fb7b77fe640 1 --2- 172.21.15.7:0/1379696180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 0x55ea2fe17cc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0012cf0 tx=0x7fb7b0010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=464c225247933068 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.189+0000 7fb7b77fe640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3d20 0x7fb7a40a61e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0010c40 tx=0x7fb7b0017000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.189+0000 7fb7ad7fa640 1 -- 172.21.15.7:0/1379696180 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0023070 con 0x55ea2fe178e0 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.189+0000 7fb7ad7fa640 1 -- 172.21.15.7:0/1379696180 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7b0009cf0 con 0x55ea2fe178e0 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.189+0000 7fb7ad7fa640 1 -- 172.21.15.7:0/1379696180 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b000de40 con 0x55ea2fe178e0 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 -- 172.21.15.7:0/1379696180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 msgr2=0x55ea2fe17cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 --2- 172.21.15.7:0/1379696180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 0x55ea2fe17cc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0012cf0 tx=0x7fb7b0010740 comp rx=0 tx=0).stop 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 -- 172.21.15.7:0/1379696180 shutdown_connections 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 --2- 172.21.15.7:0/1379696180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 0x55ea2fe23260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 --2- 172.21.15.7:0/1379696180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe18290 0x55ea2fe20840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 --2- 172.21.15.7:0/1379696180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 0x55ea2fe17cc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 -- 172.21.15.7:0/1379696180 >> 172.21.15.7:0/1379696180 conn(0x55ea2fe11fb0 msgr2=0x55ea2fe12390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 -- 172.21.15.7:0/1379696180 shutdown_connections 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 -- 172.21.15.7:0/1379696180 wait complete. 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 -- start start 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 Processor -- start 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 0x55ea2fe1deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe18290 0x55ea2fe1e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 0x55ea2fe1e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ea2fe23d20 con 0x55ea2fe20e10 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ea2fd35580 con 0x55ea2fe178e0 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.190+0000 7fb7c0185740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ea2fe23ea0 con 0x55ea2fe18290 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7c0185740 10 client.0 _ll_register_callbacks cb 0x55ea2fd3b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7b77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 0x55ea2fe1e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7b6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 0x55ea2fe1deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7b77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 0x55ea2fe1e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40756/0 (socket says 172.21.15.7:40756) 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7b77fe640 1 -- 172.21.15.7:0/1739990919 learned_addr learned my addr 172.21.15.7:0/1739990919 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7affff640 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe18290 0x55ea2fe1e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7b77fe640 1 -- 172.21.15.7:0/1739990919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe18290 msgr2=0x55ea2fe1e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7b77fe640 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe18290 0x55ea2fe1e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7b77fe640 1 -- 172.21.15.7:0/1739990919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 msgr2=0x55ea2fe1deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7b77fe640 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 0x55ea2fe1deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7b77fe640 1 -- 172.21.15.7:0/1739990919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea2fbb6f70 con 0x55ea2fe20e10 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7b77fe640 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 0x55ea2fe1e930 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0012140 tx=0x7fb7b0012800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce12c4d52c294870 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb79e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb79e7fc640 1 -- 172.21.15.7:0/1739990919 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b002b070 con 0x55ea2fe20e10 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb79e7fc640 1 -- 172.21.15.7:0/1739990919 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb7b000ddc0 con 0x55ea2fe20e10 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7c0185740 1 -- 172.21.15.7:0/1739990919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7b0009cf0 con 0x55ea2fe20e10 2024-04-21T08:46:55.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb79e7fc640 1 -- 172.21.15.7:0/1739990919 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb7b0037050 con 0x55ea2fe20e10 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7c0185740 10 client.16242 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.191+0000 7fb7c0185740 1 -- 172.21.15.7:0/1739990919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ea2fa4e170 con 0x55ea2fe20e10 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.192+0000 7fb79e7fc640 1 -- 172.21.15.7:0/1739990919 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7b0009cf0 con 0x55ea2fe20e10 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.192+0000 7fb79e7fc640 1 -- 172.21.15.7:0/1739990919 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb7b0005700 con 0x55ea2fe20e10 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.192+0000 7fb7c0185740 10 client.16242 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.192+0000 7fb7c0185740 10 client.16242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.192+0000 7fb7c0185740 20 client.16242 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.192+0000 7fb7c0185740 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ea2fe31310 0x55ea2fe33700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.192+0000 7fb7c0185740 4 client.16242 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.192+0000 7fb7c0185740 1 -- 172.21.15.7:0/1739990919 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea2fd89920 con 0x55ea2fe31310 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.192+0000 7fb7b6ffd640 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ea2fe31310 0x55ea2fe33700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.192+0000 7fb79e7fc640 10 client.16242 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.192+0000 7fb7b6ffd640 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ea2fe31310 0x55ea2fe33700 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=279a1e48d00c8660 server_cookie=8fbb070fc0af57f4 in_seq=0 out_seq=0 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.193+0000 7fb79e7fc640 1 -- 172.21.15.7:0/1739990919 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea2fd89920 con 0x55ea2fe31310 2024-04-21T08:46:55.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.193+0000 7fb79e7fc640 10 client.16242 handle_command_reply: tid=0 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 1 client.16242 shutdown 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 1 -- 172.21.15.7:0/1739990919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 msgr2=0x55ea2fe1e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 0x55ea2fe1e930 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0012140 tx=0x7fb7b0012800 comp rx=0 tx=0).stop 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 1 -- 172.21.15.7:0/1739990919 shutdown_connections 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ea2fe31310 0x55ea2fe33700 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe20e10 0x55ea2fe1e930 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe18290 0x55ea2fe1e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 1 --2- 172.21.15.7:0/1739990919 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe178e0 0x55ea2fe1deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb79e7fc640 0 client.16242 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 1 -- 172.21.15.7:0/1739990919 >> 172.21.15.7:0/1739990919 conn(0x55ea2fe11fb0 msgr2=0x55ea2fe13d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 1 -- 172.21.15.7:0/1739990919 shutdown_connections 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 1 -- 172.21.15.7:0/1739990919 wait complete. 2024-04-21T08:46:55.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.194+0000 7fb7c0185740 20 client.16242 trim_cache size 0 max 16384 2024-04-21T08:46:55.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7c0185740 1 Processor -- start 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7c0185740 1 -- start start 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe205e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe18290 0x55ea2fe2cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe20e10 0x55ea2fe38d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7c0185740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b81ad4b0 con 0x55ea2fe178e0 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7c0185740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b81afd30 con 0x55ea2fe18290 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7c0185740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b81adb20 con 0x55ea2fe20e10 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7b6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe205e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7b6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe205e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40772/0 (socket says 172.21.15.7:40772) 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7b6ffd640 1 -- 172.21.15.7:0/3555735604 learned_addr learned my addr 172.21.15.7:0/3555735604 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.212+0000 7fb7affff640 1 --2- 172.21.15.7:0/3555735604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe18290 0x55ea2fe2cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.213+0000 7fb7b77fe640 1 --2- 172.21.15.7:0/3555735604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe20e10 0x55ea2fe38d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.213+0000 7fb7b6ffd640 1 -- 172.21.15.7:0/3555735604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe20e10 msgr2=0x55ea2fe38d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.213+0000 7fb7b6ffd640 1 --2- 172.21.15.7:0/3555735604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe20e10 0x55ea2fe38d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.213+0000 7fb7b6ffd640 1 -- 172.21.15.7:0/3555735604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe18290 msgr2=0x55ea2fe2cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.213+0000 7fb7b6ffd640 1 --2- 172.21.15.7:0/3555735604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe18290 0x55ea2fe2cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.213+0000 7fb7b6ffd640 1 -- 172.21.15.7:0/3555735604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea2fe33b60 con 0x55ea2fe178e0 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.213+0000 7fb7affff640 1 --2- 172.21.15.7:0/3555735604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe18290 0x55ea2fe2cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.213+0000 7fb7b6ffd640 1 --2- 172.21.15.7:0/3555735604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe205e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009900 tx=0x7fb7a8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f69715ee97e93184 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.213+0000 7fb7acff9640 1 -- 172.21.15.7:0/3555735604 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8004340 con 0x55ea2fe178e0 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.213+0000 7fb7acff9640 1 -- 172.21.15.7:0/3555735604 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7a802cc90 con 0x55ea2fe178e0 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.213+0000 7fb7acff9640 1 -- 172.21.15.7:0/3555735604 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a80046c0 con 0x55ea2fe178e0 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 -- 172.21.15.7:0/3555735604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 msgr2=0x55ea2fe205e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 --2- 172.21.15.7:0/3555735604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe205e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009900 tx=0x7fb7a8000bc0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 -- 172.21.15.7:0/3555735604 shutdown_connections 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 --2- 172.21.15.7:0/3555735604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe20e10 0x55ea2fe38d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 --2- 172.21.15.7:0/3555735604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe18290 0x55ea2fe2cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 --2- 172.21.15.7:0/3555735604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe205e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 -- 172.21.15.7:0/3555735604 >> 172.21.15.7:0/3555735604 conn(0x55ea2fe11fb0 msgr2=0x55ea2fe13c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 -- 172.21.15.7:0/3555735604 shutdown_connections 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 -- 172.21.15.7:0/3555735604 wait complete. 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 -- start start 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 Processor -- start 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe24ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe18290 0x55ea2fe25400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe20e10 0x55ea2fe9aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b81ae1f0 con 0x55ea2fe178e0 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b81af660 con 0x55ea2fe18290 2024-04-21T08:46:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b81ae8c0 con 0x55ea2fe20e10 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7c0185740 10 client.0 _ll_register_callbacks cb 0x7fb7a0009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7b6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe24ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.214+0000 7fb7b6ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe24ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40780/0 (socket says 172.21.15.7:40780) 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7b6ffd640 1 -- 172.21.15.7:0/931911519 learned_addr learned my addr 172.21.15.7:0/931911519 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7affff640 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe18290 0x55ea2fe25400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7b77fe640 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe20e10 0x55ea2fe9aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7b6ffd640 1 -- 172.21.15.7:0/931911519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe20e10 msgr2=0x55ea2fe9aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7b6ffd640 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe20e10 0x55ea2fe9aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7b6ffd640 1 -- 172.21.15.7:0/931911519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe18290 msgr2=0x55ea2fe25400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7b6ffd640 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe18290 0x55ea2fe25400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7b6ffd640 1 -- 172.21.15.7:0/931911519 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea2fe9af50 con 0x55ea2fe178e0 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7b6ffd640 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe24ec0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80057c0 tx=0x7fb7a802c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38712ffbbc46900a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb79e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb79e7fc640 1 -- 172.21.15.7:0/931911519 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8005250 con 0x55ea2fe178e0 2024-04-21T08:46:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb79e7fc640 1 -- 172.21.15.7:0/931911519 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb7a804c050 con 0x55ea2fe178e0 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7c0185740 1 -- 172.21.15.7:0/931911519 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7a802cc90 con 0x55ea2fe178e0 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb79e7fc640 1 -- 172.21.15.7:0/931911519 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb7a802c7e0 con 0x55ea2fe178e0 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7c0185740 10 client.16248 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.215+0000 7fb7c0185740 1 -- 172.21.15.7:0/931911519 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ea2fe9c0f0 con 0x55ea2fe178e0 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.216+0000 7fb79e7fc640 1 -- 172.21.15.7:0/931911519 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7a802cc90 con 0x55ea2fe178e0 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.216+0000 7fb79e7fc640 1 -- 172.21.15.7:0/931911519 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb7a80035f0 con 0x55ea2fe178e0 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.216+0000 7fb7c0185740 10 client.16248 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.216+0000 7fb7c0185740 10 client.16248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.216+0000 7fb7c0185740 20 client.16248 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.216+0000 7fb7c0185740 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ea2fe9c800 0x55ea2fe9ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.216+0000 7fb7c0185740 4 client.16248 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.216+0000 7fb7c0185740 1 -- 172.21.15.7:0/931911519 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ea2fd83790 con 0x55ea2fe9c800 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.216+0000 7fb7affff640 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ea2fe9c800 0x55ea2fe9ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.216+0000 7fb7affff640 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ea2fe9c800 0x55ea2fe9ec20 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1d729dcc2a1989e server_cookie=997947e06cd9115d in_seq=0 out_seq=0 2024-04-21T08:46:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.216+0000 7fb79e7fc640 10 client.16248 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:55.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb79e7fc640 1 -- 172.21.15.7:0/931911519 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ea2fd83790 con 0x55ea2fe9c800 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb79e7fc640 10 client.16248 handle_command_reply: tid=0 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb7c0185740 1 client.16248 shutdown 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb7c0185740 1 -- 172.21.15.7:0/931911519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 msgr2=0x55ea2fe24ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb7c0185740 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe24ec0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80057c0 tx=0x7fb7a802c3e0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb7c0185740 1 -- 172.21.15.7:0/931911519 shutdown_connections 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb7c0185740 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ea2fe9c800 0x55ea2fe9ec20 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb79e7fc640 0 client.16248 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb7c0185740 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ea2fe20e10 0x55ea2fe9aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb7c0185740 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ea2fe18290 0x55ea2fe25400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb7c0185740 1 --2- 172.21.15.7:0/931911519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ea2fe178e0 0x55ea2fe24ec0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb7c0185740 1 -- 172.21.15.7:0/931911519 >> 172.21.15.7:0/931911519 conn(0x55ea2fe11fb0 msgr2=0x55ea2fe13380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb7c0185740 1 -- 172.21.15.7:0/931911519 shutdown_connections 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.217+0000 7fb7c0185740 1 -- 172.21.15.7:0/931911519 wait complete. 2024-04-21T08:46:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7c0185740 20 client.16248 trim_cache size 0 max 16384 2024-04-21T08:46:55.224 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:46:55.224 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:46:55.224 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "b0adb205-b024-4f89-ba40-5cd98d91d5a6", 2024-04-21T08:46:55.224 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:46:55.224 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:46:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 -- 172.21.15.7:0/1538425570 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3d20 msgr2=0x7fb7a40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3d20 0x7fb7a40a61e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0010c40 tx=0x7fb7b0017000 comp rx=0 tx=0).stop 2024-04-21T08:46:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 -- 172.21.15.7:0/1538425570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b8106160 msgr2=0x7fb7b819e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b8106160 0x7fb7b819e940 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000d740 tx=0x7fb7a000dc00 comp rx=0 tx=0).stop 2024-04-21T08:46:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 -- 172.21.15.7:0/1538425570 shutdown_connections 2024-04-21T08:46:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b81985b0 0x7fb7b8198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7b8106160 0x7fb7b819e940 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3d20 0x7fb7a40a61e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 --2- 172.21.15.7:0/1538425570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7b8101090 0x7fb7b819e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 -- 172.21.15.7:0/1538425570 >> 172.21.15.7:0/1538425570 conn(0x7fb7b8078000 msgr2=0x7fb7b80696f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 -- 172.21.15.7:0/1538425570 shutdown_connections 2024-04-21T08:46:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.218+0000 7fb7bd596640 1 -- 172.21.15.7:0/1538425570 wait complete. 2024-04-21T08:46:55.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:46:55.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:46:55.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.337+0000 7fb7c2626640 1 -- 172.21.15.7:0/4126638185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7bc1031e0 msgr2=0x7fb7bc1035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.337+0000 7fb7c2626640 1 --2- 172.21.15.7:0/4126638185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7bc1031e0 0x7fb7bc1035c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009900 tx=0x7fb7ac02bd60 comp rx=0 tx=0).stop 2024-04-21T08:46:55.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.337+0000 7fb7c2626640 1 -- 172.21.15.7:0/4126638185 shutdown_connections 2024-04-21T08:46:55.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.337+0000 7fb7c2626640 1 --2- 172.21.15.7:0/4126638185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc10c520 0x7fb7bc10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.337+0000 7fb7c2626640 1 --2- 172.21.15.7:0/4126638185 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7bc103b00 0x7fb7bc10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.337+0000 7fb7c2626640 1 --2- 172.21.15.7:0/4126638185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7bc1031e0 0x7fb7bc1035c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.337+0000 7fb7c2626640 1 -- 172.21.15.7:0/4126638185 >> 172.21.15.7:0/4126638185 conn(0x7fb7bc077f70 msgr2=0x7fb7bc078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.337+0000 7fb7c2626640 1 -- 172.21.15.7:0/4126638185 shutdown_connections 2024-04-21T08:46:55.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.337+0000 7fb7c2626640 1 -- 172.21.15.7:0/4126638185 wait complete. 2024-04-21T08:46:55.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.337+0000 7fb7c2626640 1 Processor -- start 2024-04-21T08:46:55.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7c2626640 1 -- start start 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7c2626640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1031e0 0x7fb7bc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7c2626640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7bc103b00 0x7fb7bc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7c2626640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7bc10c520 0x7fb7bc19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7c2626640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc112190 con 0x7fb7bc103b00 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7c2626640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc112010 con 0x7fb7bc10c520 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7c2626640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc112310 con 0x7fb7bc1031e0 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7bc103b00 0x7fb7bc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7bc103b00 0x7fb7bc198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40800/0 (socket says 172.21.15.7:40800) 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/578890678 learned_addr learned my addr 172.21.15.7:0/578890678 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7c0bc3640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7bc10c520 0x7fb7bc19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1031e0 0x7fb7bc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/578890678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1031e0 msgr2=0x7fb7bc198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7bb7fe640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1031e0 0x7fb7bc198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/578890678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7bc10c520 msgr2=0x7fb7bc19cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7bb7fe640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7bc10c520 0x7fb7bc19cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/578890678 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7bc19d700 con 0x7fb7bc103b00 2024-04-21T08:46:55.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7c0bc3640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7bc10c520 0x7fb7bc19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:55.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.338+0000 7fb7bb7fe640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7bc103b00 0x7fb7bc198c50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000e9f0 tx=0x7fb7b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.339+0000 7fb7b97fa640 1 -- 172.21.15.7:0/578890678 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b000cd80 con 0x7fb7bc103b00 2024-04-21T08:46:55.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.339+0000 7fb7b97fa640 1 -- 172.21.15.7:0/578890678 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7b00108e0 con 0x7fb7bc103b00 2024-04-21T08:46:55.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.339+0000 7fb7c2626640 1 -- 172.21.15.7:0/578890678 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7bc19d9f0 con 0x7fb7bc103b00 2024-04-21T08:46:55.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.339+0000 7fb7b97fa640 1 -- 172.21.15.7:0/578890678 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0018980 con 0x7fb7bc103b00 2024-04-21T08:46:55.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.339+0000 7fb7c2626640 1 -- 172.21.15.7:0/578890678 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7bc1a5280 con 0x7fb7bc103b00 2024-04-21T08:46:55.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.341+0000 7fb7b97fa640 1 -- 172.21.15.7:0/578890678 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fb7b0010430 con 0x7fb7bc103b00 2024-04-21T08:46:55.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.341+0000 7fb7c5210740 1 Processor -- start 2024-04-21T08:46:55.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.341+0000 7fb7c5210740 1 -- start start 2024-04-21T08:46:55.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.341+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 0x558a99306690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.341+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99313950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.341+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a99313e90 0x558a99316280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.341+0000 7fb7c5210740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558a9926b1e0 con 0x558a99306bd0 2024-04-21T08:46:55.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.341+0000 7fb7c5210740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558a99208390 con 0x558a993062b0 2024-04-21T08:46:55.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.341+0000 7fb7c5210740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558a992671a0 con 0x558a99313e90 2024-04-21T08:46:55.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7bbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99313950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7c0bc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 0x558a99306690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7c0bc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 0x558a99306690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33382/0 (socket says 172.21.15.7:33382) 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7bbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99313950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40806/0 (socket says 172.21.15.7:40806) 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7c0bc3640 1 -- 172.21.15.7:0/3606726029 learned_addr learned my addr 172.21.15.7:0/3606726029 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7bb7fe640 1 --2- 172.21.15.7:0/3606726029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a99313e90 0x558a99316280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7bbfff640 1 -- 172.21.15.7:0/3606726029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a99313e90 msgr2=0x558a99316280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/3606726029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a99313e90 0x558a99316280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7bbfff640 1 -- 172.21.15.7:0/3606726029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 msgr2=0x558a99306690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/3606726029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 0x558a99306690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7bbfff640 1 -- 172.21.15.7:0/3606726029 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a990477d0 con 0x558a99306bd0 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7b97fa640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3da0 0x7fb7a40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/3606726029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99313950 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac000c00 tx=0x7fb7ac02c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de557165fdbd6e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7c0bc3640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3da0 0x7fb7a40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7b97fa640 1 -- 172.21.15.7:0/578890678 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb7b00d0e80 con 0x7fb7bc103b00 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7a27fc640 1 -- 172.21.15.7:0/3606726029 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac0040c0 con 0x558a99306bd0 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7a27fc640 1 -- 172.21.15.7:0/3606726029 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7ac02cdc0 con 0x558a99306bd0 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.342+0000 7fb7a27fc640 1 -- 172.21.15.7:0/3606726029 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac0046c0 con 0x558a99306bd0 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.343+0000 7fb7c5210740 1 -- 172.21.15.7:0/3606726029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 msgr2=0x558a99313950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.343+0000 7fb7c5210740 1 --2- 172.21.15.7:0/3606726029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99313950 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac000c00 tx=0x7fb7ac02c740 comp rx=0 tx=0).stop 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.343+0000 7fb7c0bc3640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3da0 0x7fb7a40a6260 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4011950 tx=0x7fb7b40096a0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.343+0000 7fb7c5210740 1 -- 172.21.15.7:0/3606726029 shutdown_connections 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.343+0000 7fb7c5210740 1 --2- 172.21.15.7:0/3606726029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a99313e90 0x558a99316280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.343+0000 7fb7c5210740 1 --2- 172.21.15.7:0/3606726029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99313950 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.343+0000 7fb7c5210740 1 --2- 172.21.15.7:0/3606726029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 0x558a99306690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.343+0000 7fb7c5210740 1 -- 172.21.15.7:0/3606726029 >> 172.21.15.7:0/3606726029 conn(0x558a99300cb0 msgr2=0x558a99301090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.343+0000 7fb7c5210740 1 -- 172.21.15.7:0/3606726029 shutdown_connections 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.343+0000 7fb7c5210740 1 -- 172.21.15.7:0/3606726029 wait complete. 2024-04-21T08:46:55.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.343+0000 7fb7c5210740 1 -- start start 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7c5210740 1 Processor -- start 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a993062b0 0x558a99310e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99311390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a99313e90 0x558a993118d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7c5210740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558a99316bf0 con 0x558a99306bd0 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7c5210740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558a992242f0 con 0x558a99313e90 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7c5210740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558a99316d70 con 0x558a993062b0 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7c5210740 10 client.0 _ll_register_callbacks cb 0x558a99231580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99311390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99311390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40820/0 (socket says 172.21.15.7:40820) 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/1884799000 learned_addr learned my addr 172.21.15.7:0/1884799000 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a993062b0 0x558a99310e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7c0bc3640 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a99313e90 0x558a993118d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/1884799000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a993062b0 msgr2=0x558a99310e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7bb7fe640 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a993062b0 0x558a99310e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/1884799000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a99313e90 msgr2=0x558a993118d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7bb7fe640 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a99313e90 0x558a993118d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/1884799000 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a98f7d3f0 con 0x558a99306bd0 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.344+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a993062b0 0x558a99310e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb7bb7fe640 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99311390 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0000c00 tx=0x7fb7b0018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f221e09f53c2e7a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb79b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb79b7fe640 1 -- 172.21.15.7:0/1884799000 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b009c440 con 0x558a99306bd0 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb79b7fe640 1 -- 172.21.15.7:0/1884799000 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb7b00d4b80 con 0x558a99306bd0 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb7c5210740 1 -- 172.21.15.7:0/1884799000 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7ac02cdc0 con 0x558a99306bd0 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb79b7fe640 1 -- 172.21.15.7:0/1884799000 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb7b00bd0b0 con 0x558a99306bd0 2024-04-21T08:46:55.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb7c5210740 10 client.16260 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb7c5210740 1 -- 172.21.15.7:0/1884799000 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558a98f3d0f0 con 0x558a99306bd0 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb79b7fe640 1 -- 172.21.15.7:0/1884799000 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7ac02cdc0 con 0x558a99306bd0 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb79b7fe640 1 -- 172.21.15.7:0/1884799000 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb7b0018b20 con 0x558a99306bd0 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb7c5210740 10 client.16260 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb7c5210740 10 client.16260 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb7c5210740 20 client.16260 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb7c5210740 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558a99323db0 0x558a993261a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb7c5210740 4 client.16260 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.345+0000 7fb7c5210740 1 -- 172.21.15.7:0/1884799000 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a99278c00 con 0x558a99323db0 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.346+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558a99323db0 0x558a993261a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.346+0000 7fb79b7fe640 10 client.16260 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.346+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558a99323db0 0x558a993261a0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4286526b77022069 server_cookie=ac0e50ce8f7fc4de in_seq=0 out_seq=0 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb79b7fe640 1 -- 172.21.15.7:0/1884799000 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a99278c00 con 0x558a99323db0 2024-04-21T08:46:55.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb79b7fe640 10 client.16260 handle_command_reply: tid=0 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 1 client.16260 shutdown 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 1 -- 172.21.15.7:0/1884799000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 msgr2=0x558a99311390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99311390 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0000c00 tx=0x7fb7b0018420 comp rx=0 tx=0).stop 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 1 -- 172.21.15.7:0/1884799000 shutdown_connections 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558a99323db0 0x558a993261a0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a99313e90 0x558a993118d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb79b7fe640 0 client.16260 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a99311390 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 1 --2- 172.21.15.7:0/1884799000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a993062b0 0x558a99310e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 1 -- 172.21.15.7:0/1884799000 >> 172.21.15.7:0/1884799000 conn(0x558a99300cb0 msgr2=0x558a99302a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 1 -- 172.21.15.7:0/1884799000 shutdown_connections 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 1 -- 172.21.15.7:0/1884799000 wait complete. 2024-04-21T08:46:55.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.347+0000 7fb7c5210740 20 client.16260 trim_cache size 0 max 16384 2024-04-21T08:46:55.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7c5210740 1 Processor -- start 2024-04-21T08:46:55.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7c5210740 1 -- start start 2024-04-21T08:46:55.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a993062b0 0x558a9930fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a9931faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a99313e90 0x558a9930c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7c5210740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1ab2a0 con 0x558a99306bd0 2024-04-21T08:46:55.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7c5210740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1adb20 con 0x558a99313e90 2024-04-21T08:46:55.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7c5210740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1ab910 con 0x558a993062b0 2024-04-21T08:46:55.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a9931faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a9931faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40834/0 (socket says 172.21.15.7:40834) 2024-04-21T08:46:55.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/2864306392 learned_addr learned my addr 172.21.15.7:0/2864306392 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7c0bc3640 1 --2- 172.21.15.7:0/2864306392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a99313e90 0x558a9930c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/2864306392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a993062b0 0x558a9930fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/2864306392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a993062b0 msgr2=0x558a9930fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7bb7fe640 1 --2- 172.21.15.7:0/2864306392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a993062b0 0x558a9930fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/2864306392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a99313e90 msgr2=0x558a9930c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7bb7fe640 1 --2- 172.21.15.7:0/2864306392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a99313e90 0x558a9930c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.365+0000 7fb7bb7fe640 1 -- 172.21.15.7:0/2864306392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a99306a00 con 0x558a99306bd0 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7bb7fe640 1 --2- 172.21.15.7:0/2864306392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a9931faf0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0098f40 tx=0x7fb7b0010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414c8a940b0b145c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7a1ffb640 1 -- 172.21.15.7:0/2864306392 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0010720 con 0x558a99306bd0 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7a1ffb640 1 -- 172.21.15.7:0/2864306392 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7b009cae0 con 0x558a99306bd0 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7a1ffb640 1 -- 172.21.15.7:0/2864306392 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0098830 con 0x558a99306bd0 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7c5210740 1 -- 172.21.15.7:0/2864306392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 msgr2=0x558a9931faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7c5210740 1 --2- 172.21.15.7:0/2864306392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a9931faf0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0098f40 tx=0x7fb7b0010ce0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7c5210740 1 -- 172.21.15.7:0/2864306392 shutdown_connections 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7c5210740 1 --2- 172.21.15.7:0/2864306392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a99313e90 0x558a9930c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7c5210740 1 --2- 172.21.15.7:0/2864306392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99306bd0 0x558a9931faf0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7c5210740 1 --2- 172.21.15.7:0/2864306392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a993062b0 0x558a9930fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7c5210740 1 -- 172.21.15.7:0/2864306392 >> 172.21.15.7:0/2864306392 conn(0x558a99300cb0 msgr2=0x558a992ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7c5210740 1 -- 172.21.15.7:0/2864306392 shutdown_connections 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.366+0000 7fb7c5210740 1 -- 172.21.15.7:0/2864306392 wait complete. 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c5210740 1 -- start start 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c5210740 1 Processor -- start 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 0x558a9930d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a99306bd0 0x558a9930df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c5210740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99313e90 0x558a9930e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c5210740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1abfe0 con 0x558a99313e90 2024-04-21T08:46:55.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c5210740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1ad450 con 0x558a993062b0 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c5210740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1ac6b0 con 0x558a99306bd0 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c5210740 10 client.0 _ll_register_callbacks cb 0x7fb7b00d46c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c0bc3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99313e90 0x558a9930e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 0x558a9930d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c0bc3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99313e90 0x558a9930e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40840/0 (socket says 172.21.15.7:40840) 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7c0bc3640 1 -- 172.21.15.7:0/2464474938 learned_addr learned my addr 172.21.15.7:0/2464474938 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.367+0000 7fb7bb7fe640 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a99306bd0 0x558a9930df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb7c0bc3640 1 -- 172.21.15.7:0/2464474938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a99306bd0 msgr2=0x558a9930df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb7c0bc3640 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a99306bd0 0x558a9930df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb7c0bc3640 1 -- 172.21.15.7:0/2464474938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 msgr2=0x558a9930d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb7c0bc3640 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 0x558a9930d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb7c0bc3640 1 -- 172.21.15.7:0/2464474938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a99389290 con 0x558a99313e90 2024-04-21T08:46:55.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 0x558a9930d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb7c0bc3640 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99313e90 0x558a9930e470 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4010d20 tx=0x7fb7b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58daa456914d263e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb79b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb79b7fe640 1 -- 172.21.15.7:0/2464474938 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b402e060 con 0x558a99313e90 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb79b7fe640 1 -- 172.21.15.7:0/2464474938 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb7b401ebb0 con 0x558a99313e90 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb7c5210740 1 -- 172.21.15.7:0/2464474938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7b009cae0 con 0x558a99313e90 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb79b7fe640 1 -- 172.21.15.7:0/2464474938 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb7b4035050 con 0x558a99313e90 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb7c5210740 10 client.16266 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb7c5210740 1 -- 172.21.15.7:0/2464474938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558a9938a4c0 con 0x558a99313e90 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.368+0000 7fb79b7fe640 1 -- 172.21.15.7:0/2464474938 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7b009cae0 con 0x558a99313e90 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.369+0000 7fb79b7fe640 1 -- 172.21.15.7:0/2464474938 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb7b4004470 con 0x558a99313e90 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.369+0000 7fb7c5210740 10 client.16266 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.369+0000 7fb7c5210740 10 client.16266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:55.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.369+0000 7fb7c5210740 20 client.16266 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:55.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.369+0000 7fb7c5210740 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558a9938ac40 0x558a9938d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.369+0000 7fb7c5210740 4 client.16266 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:46:55.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.369+0000 7fb7c5210740 1 -- 172.21.15.7:0/2464474938 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558a99272a70 con 0x558a9938ac40 2024-04-21T08:46:55.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.369+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558a9938ac40 0x558a9938d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.369+0000 7fb7bbfff640 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558a9938ac40 0x558a9938d060 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c66ad014835dff71 server_cookie=2eeb11da5a16ac16 in_seq=0 out_seq=0 2024-04-21T08:46:55.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.369+0000 7fb79b7fe640 10 client.16266 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:55.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb79b7fe640 1 -- 172.21.15.7:0/2464474938 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x558a99272a70 con 0x558a9938ac40 2024-04-21T08:46:55.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb79b7fe640 10 client.16266 handle_command_reply: tid=0 2024-04-21T08:46:55.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb7c5210740 1 client.16266 shutdown 2024-04-21T08:46:55.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb7c5210740 1 -- 172.21.15.7:0/2464474938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99313e90 msgr2=0x558a9930e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb7c5210740 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99313e90 0x558a9930e470 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4010d20 tx=0x7fb7b4004a40 comp rx=0 tx=0).stop 2024-04-21T08:46:55.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb7c5210740 1 -- 172.21.15.7:0/2464474938 shutdown_connections 2024-04-21T08:46:55.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb7c5210740 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558a9938ac40 0x558a9938d060 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb7c5210740 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558a99313e90 0x558a9930e470 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb7c5210740 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a99306bd0 0x558a9930df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb79b7fe640 0 client.16266 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:55.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb7c5210740 1 --2- 172.21.15.7:0/2464474938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558a993062b0 0x558a9930d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.370+0000 7fb7c5210740 1 -- 172.21.15.7:0/2464474938 >> 172.21.15.7:0/2464474938 conn(0x558a99300cb0 msgr2=0x558a99312490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c5210740 1 -- 172.21.15.7:0/2464474938 shutdown_connections 2024-04-21T08:46:55.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c5210740 1 -- 172.21.15.7:0/2464474938 wait complete. 2024-04-21T08:46:55.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c5210740 20 client.16266 trim_cache size 0 max 16384 2024-04-21T08:46:55.377 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:46:55.378 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (211 inodes in the stack)", 2024-04-21T08:46:55.378 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:46:55.378 INFO:teuthology.orchestra.run.smithi007.stdout: "b0adb205-b024-4f89-ba40-5cd98d91d5a6": { 2024-04-21T08:46:55.378 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:46:55.378 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "b0adb205-b024-4f89-ba40-5cd98d91d5a6", 2024-04-21T08:46:55.378 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:46:55.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:46:55.378 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:46:55.378 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:46:55.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 -- 172.21.15.7:0/578890678 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3da0 msgr2=0x7fb7a40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3da0 0x7fb7a40a6260 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4011950 tx=0x7fb7b40096a0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 -- 172.21.15.7:0/578890678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7bc103b00 msgr2=0x7fb7bc198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7bc103b00 0x7fb7bc198c50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000e9f0 tx=0x7fb7b000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 -- 172.21.15.7:0/578890678 shutdown_connections 2024-04-21T08:46:55.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7bc10c520 0x7fb7bc19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7bc103b00 0x7fb7bc198c50 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb7a40a3da0 0x7fb7a40a6260 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 --2- 172.21.15.7:0/578890678 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7bc1031e0 0x7fb7bc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 -- 172.21.15.7:0/578890678 >> 172.21.15.7:0/578890678 conn(0x7fb7bc077f70 msgr2=0x7fb7bc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 -- 172.21.15.7:0/578890678 shutdown_connections 2024-04-21T08:46:55.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.371+0000 7fb7c2626640 1 -- 172.21.15.7:0/578890678 wait complete. 2024-04-21T08:46:55.383 INFO:tasks.cephfs.filesystem:scrub status for tag:b0adb205-b024-4f89-ba40-5cd98d91d5a6 - {'path': '/', 'tag': 'b0adb205-b024-4f89-ba40-5cd98d91d5a6', 'options': 'recursive,force'} 2024-04-21T08:46:55.798 INFO:tasks.quiescer.fs.[cephfs]:Total duration (57.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-21T08:46:55.798 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 28.95 2024-04-21T08:46:55.801 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:46:55.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.905+0000 7f77a9fdb640 1 -- 172.21.15.7:0/1824535343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a4106140 msgr2=0x7f77a410dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.905+0000 7f77a9fdb640 1 --2- 172.21.15.7:0/1824535343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a4106140 0x7f77a410dd10 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f779c009ca0 tx=0x7f779c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.905+0000 7f77a9fdb640 1 -- 172.21.15.7:0/1824535343 shutdown_connections 2024-04-21T08:46:55.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.905+0000 7f77a9fdb640 1 --2- 172.21.15.7:0/1824535343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a4106140 0x7f77a410dd10 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.905+0000 7f77a9fdb640 1 --2- 172.21.15.7:0/1824535343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77a4101a20 0x7f77a4105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.905+0000 7f77a9fdb640 1 --2- 172.21.15.7:0/1824535343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a4101070 0x7f77a4101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.905+0000 7f77a9fdb640 1 -- 172.21.15.7:0/1824535343 >> 172.21.15.7:0/1824535343 conn(0x7f77a40771d0 msgr2=0x7f77a40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.905+0000 7f77a9fdb640 1 -- 172.21.15.7:0/1824535343 shutdown_connections 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.905+0000 7f77a9fdb640 1 -- 172.21.15.7:0/1824535343 wait complete. 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a9fdb640 1 Processor -- start 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a9fdb640 1 -- start start 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a9fdb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a4101070 0x7f77a41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a9fdb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77a4106140 0x7f77a4198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a9fdb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a419cf80 0x7f77a419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a9fdb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f77a41121c0 con 0x7f77a4101070 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a9fdb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77a4112040 con 0x7f77a419cf80 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a9fdb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f77a4112340 con 0x7f77a4106140 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a4101070 0x7f77a41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a4101070 0x7f77a41986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40858/0 (socket says 172.21.15.7:40858) 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a37fe640 1 -- 172.21.15.7:0/269006597 learned_addr learned my addr 172.21.15.7:0/269006597 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a3fff640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a419cf80 0x7f77a419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.906+0000 7f77a2ffd640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77a4106140 0x7f77a4198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a37fe640 1 -- 172.21.15.7:0/269006597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77a4106140 msgr2=0x7f77a4198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a37fe640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77a4106140 0x7f77a4198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a37fe640 1 -- 172.21.15.7:0/269006597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a419cf80 msgr2=0x7f77a419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a37fe640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a419cf80 0x7f77a419d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a37fe640 1 -- 172.21.15.7:0/269006597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a419da70 con 0x7f77a4101070 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a2ffd640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77a4106140 0x7f77a4198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a37fe640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a4101070 0x7f77a41986c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f779400ea60 tx=0x7f779400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a0ff9640 1 -- 172.21.15.7:0/269006597 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779400cde0 con 0x7f77a4101070 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a0ff9640 1 -- 172.21.15.7:0/269006597 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77940035f0 con 0x7f77a4101070 2024-04-21T08:46:55.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a9fdb640 1 -- 172.21.15.7:0/269006597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77a419dd60 con 0x7f77a4101070 2024-04-21T08:46:55.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a0ff9640 1 -- 172.21.15.7:0/269006597 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7794010690 con 0x7f77a4101070 2024-04-21T08:46:55.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.907+0000 7f77a9fdb640 1 -- 172.21.15.7:0/269006597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a41a5550 con 0x7f77a4101070 2024-04-21T08:46:55.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.908+0000 7f77a9fdb640 1 -- 172.21.15.7:0/269006597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77a4068e00 con 0x7f77a4101070 2024-04-21T08:46:55.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.910+0000 7f77a0ff9640 1 -- 172.21.15.7:0/269006597 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f77940037a0 con 0x7f77a4101070 2024-04-21T08:46:55.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.910+0000 7f77a0ff9640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f778c0a3d20 0x7f778c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:55.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.910+0000 7f77a2ffd640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f778c0a3d20 0x7f778c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:55.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.910+0000 7f77a0ff9640 1 -- 172.21.15.7:0/269006597 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f77940d1200 con 0x7f77a4101070 2024-04-21T08:46:55.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.911+0000 7f77a2ffd640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f778c0a3d20 0x7f778c0a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f77a40ff6b0 tx=0x7f7798005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:55.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:55.914+0000 7f77a0ff9640 1 -- 172.21.15.7:0/269006597 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7794053ff0 con 0x7f77a4101070 2024-04-21T08:46:56.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.156+0000 7f77a9fdb640 1 -- 172.21.15.7:0/269006597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77a41093d0 con 0x7f77a4101070 2024-04-21T08:46:56.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.157+0000 7f77a0ff9640 1 -- 172.21.15.7:0/269006597 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7f779409d450 con 0x7f77a4101070 2024-04-21T08:46:56.157 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:46:56.158 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:46:56.158 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:46:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 -- 172.21.15.7:0/269006597 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f778c0a3d20 msgr2=0x7f778c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f778c0a3d20 0x7f778c0a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f77a40ff6b0 tx=0x7f7798005f90 comp rx=0 tx=0).stop 2024-04-21T08:46:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 -- 172.21.15.7:0/269006597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a4101070 msgr2=0x7f77a41986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a4101070 0x7f77a41986c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f779400ea60 tx=0x7f779400ef20 comp rx=0 tx=0).stop 2024-04-21T08:46:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 -- 172.21.15.7:0/269006597 shutdown_connections 2024-04-21T08:46:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a419cf80 0x7f77a419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f778c0a3d20 0x7f778c0a61e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77a4106140 0x7f77a4198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 --2- 172.21.15.7:0/269006597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a4101070 0x7f77a41986c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 -- 172.21.15.7:0/269006597 >> 172.21.15.7:0/269006597 conn(0x7f77a40771d0 msgr2=0x7f77a4100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:56.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 -- 172.21.15.7:0/269006597 shutdown_connections 2024-04-21T08:46:56.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.159+0000 7f77a9fdb640 1 -- 172.21.15.7:0/269006597 wait complete. 2024-04-21T08:46:56.171 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 30 --expiration 88.95 --await' 2024-04-21T08:46:56.171 DEBUG:teuthology.orchestra.run.smithi007:> timeout 45 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24533 quiesce db / --timeout 30 --expiration 88.95 --await 2024-04-21T08:46:56.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.285+0000 7fb3e908c640 1 -- 172.21.15.7:0/1941035599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3e4106ed0 msgr2=0x7fb3e41072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:56.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.285+0000 7fb3e908c640 1 --2- 172.21.15.7:0/1941035599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3e4106ed0 0x7fb3e41072b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009900 tx=0x7fb3d402bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:56.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.285+0000 7fb3e908c640 1 -- 172.21.15.7:0/1941035599 shutdown_connections 2024-04-21T08:46:56.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.285+0000 7fb3e908c640 1 --2- 172.21.15.7:0/1941035599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e4101a40 0x7fb3e410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.285+0000 7fb3e908c640 1 --2- 172.21.15.7:0/1941035599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3e41010a0 0x7fb3e4101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.285+0000 7fb3e908c640 1 --2- 172.21.15.7:0/1941035599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3e4106ed0 0x7fb3e41072b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.285+0000 7fb3e908c640 1 -- 172.21.15.7:0/1941035599 >> 172.21.15.7:0/1941035599 conn(0x7fb3e4077c40 msgr2=0x7fb3e4078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:56.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.285+0000 7fb3e908c640 1 -- 172.21.15.7:0/1941035599 shutdown_connections 2024-04-21T08:46:56.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.285+0000 7fb3e908c640 1 -- 172.21.15.7:0/1941035599 wait complete. 2024-04-21T08:46:56.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3e908c640 1 Processor -- start 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3e908c640 1 -- start start 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3e908c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3e41010a0 0x7fb3e4198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3e908c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3e4101a40 0x7fb3e4198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3e908c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e4106ed0 0x7fb3e419cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3e908c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e41110a0 con 0x7fb3e4101a40 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3e908c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e4110f20 con 0x7fb3e4106ed0 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3e908c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e4111220 con 0x7fb3e41010a0 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3da59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3e4101a40 0x7fb3e4198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3da59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3e4101a40 0x7fb3e4198cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40872/0 (socket says 172.21.15.7:40872) 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3e359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e4106ed0 0x7fb3e419cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3da59c640 1 -- 172.21.15.7:0/1376091167 learned_addr learned my addr 172.21.15.7:0/1376091167 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.286+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/1376091167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3e41010a0 0x7fb3e4198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3da59c640 1 -- 172.21.15.7:0/1376091167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3e41010a0 msgr2=0x7fb3e4198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:56.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3da59c640 1 --2- 172.21.15.7:0/1376091167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3e41010a0 0x7fb3e4198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3da59c640 1 -- 172.21.15.7:0/1376091167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e4106ed0 msgr2=0x7fb3e419cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3da59c640 1 --2- 172.21.15.7:0/1376091167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e4106ed0 0x7fb3e419cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3da59c640 1 -- 172.21.15.7:0/1376091167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e419d710 con 0x7fb3e4101a40 2024-04-21T08:46:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/1376091167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3e41010a0 0x7fb3e4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3e359e640 1 --2- 172.21.15.7:0/1376091167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3e4106ed0 0x7fb3e419cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3da59c640 1 --2- 172.21.15.7:0/1376091167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3e4101a40 0x7fb3e4198cb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00e9f0 tx=0x7fb3cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3e0d99640 1 -- 172.21.15.7:0/1376091167 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc00cd80 con 0x7fb3e4101a40 2024-04-21T08:46:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3e0d99640 1 -- 172.21.15.7:0/1376091167 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3cc0054d0 con 0x7fb3e4101a40 2024-04-21T08:46:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3e908c640 1 -- 172.21.15.7:0/1376091167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e419da00 con 0x7fb3e4101a40 2024-04-21T08:46:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3e0d99640 1 -- 172.21.15.7:0/1376091167 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc010690 con 0x7fb3e4101a40 2024-04-21T08:46:56.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.287+0000 7fb3e908c640 1 -- 172.21.15.7:0/1376091167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e41a5290 con 0x7fb3e4101a40 2024-04-21T08:46:56.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.290+0000 7fb3e0d99640 1 -- 172.21.15.7:0/1376091167 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fb3cc004e90 con 0x7fb3e4101a40 2024-04-21T08:46:56.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.290+0000 7fb3eb922740 1 Processor -- start 2024-04-21T08:46:56.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.290+0000 7fb3eb922740 1 -- start start 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.290+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e47346fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.290+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4734702e0 0x55e473478980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.290+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e47347b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.290+0000 7fb3eb922740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e4733d00b0 con 0x55e47346f9c0 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.290+0000 7fb3eb922740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e473381680 con 0x55e473478ec0 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.290+0000 7fb3eb922740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e47338dc60 con 0x55e4734702e0 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.290+0000 7fb3e359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e47346fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e47346fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40874/0 (socket says 172.21.15.7:40874) 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e359e640 1 -- 172.21.15.7:0/1277125861 learned_addr learned my addr 172.21.15.7:0/1277125861 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3da59c640 1 --2- 172.21.15.7:0/1277125861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e47347b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/1277125861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4734702e0 0x55e473478980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e0d99640 1 --2- 172.21.15.7:0/1376091167 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb3d00a3d20 0x7fb3d00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e2d9d640 1 -- 172.21.15.7:0/1277125861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 msgr2=0x55e47347b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/1277125861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e47347b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e2d9d640 1 -- 172.21.15.7:0/1277125861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 msgr2=0x55e47346fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/1277125861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e47346fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e2d9d640 1 -- 172.21.15.7:0/1277125861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4731b2c90 con 0x55e4734702e0 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e359e640 1 --2- 172.21.15.7:0/1277125861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e47346fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e359e640 1 --2- 172.21.15.7:0/1376091167 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb3d00a3d20 0x7fb3d00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e0d99640 1 -- 172.21.15.7:0/1376091167 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb3cc0d0fe0 con 0x7fb3e4101a40 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/1277125861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4734702e0 0x55e473478980 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4000c00 tx=0x7fb3d40095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41fa5abfaecf8421 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3d959a640 1 -- 172.21.15.7:0/1277125861 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4045070 con 0x55e4734702e0 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3d959a640 1 -- 172.21.15.7:0/1277125861 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3d402c7e0 con 0x55e4734702e0 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3d959a640 1 -- 172.21.15.7:0/1277125861 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4003ab0 con 0x55e4734702e0 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.291+0000 7fb3e359e640 1 --2- 172.21.15.7:0/1376091167 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb3d00a3d20 0x7fb3d00a61e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009fd0 tx=0x7fb3dc004a40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.292+0000 7fb3eb922740 1 -- 172.21.15.7:0/1277125861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4734702e0 msgr2=0x55e473478980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.292+0000 7fb3eb922740 1 --2- 172.21.15.7:0/1277125861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4734702e0 0x55e473478980 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4000c00 tx=0x7fb3d40095f0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.292+0000 7fb3eb922740 1 -- 172.21.15.7:0/1277125861 shutdown_connections 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.292+0000 7fb3eb922740 1 --2- 172.21.15.7:0/1277125861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e47347b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.292+0000 7fb3eb922740 1 --2- 172.21.15.7:0/1277125861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4734702e0 0x55e473478980 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.292+0000 7fb3eb922740 1 --2- 172.21.15.7:0/1277125861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e47346fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.292+0000 7fb3eb922740 1 -- 172.21.15.7:0/1277125861 >> 172.21.15.7:0/1277125861 conn(0x55e47346a0f0 msgr2=0x55e47346a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.292+0000 7fb3eb922740 1 -- 172.21.15.7:0/1277125861 shutdown_connections 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.292+0000 7fb3eb922740 1 -- 172.21.15.7:0/1277125861 wait complete. 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3eb922740 1 -- start start 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3eb922740 1 Processor -- start 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 0x55e473475ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e4734763e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e473476920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3eb922740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e4733d3400 con 0x55e4734702e0 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3eb922740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e47338f840 con 0x55e473478ec0 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3eb922740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e4733d8720 con 0x55e47346f9c0 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3eb922740 10 client.0 _ll_register_callbacks cb 0x55e4733e0ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3da59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e4734763e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3e2d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 0x55e473475ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3da59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e4734763e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40888/0 (socket says 172.21.15.7:40888) 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3da59c640 1 -- 172.21.15.7:0/1946667319 learned_addr learned my addr 172.21.15.7:0/1946667319 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3e359e640 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e473476920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3da59c640 1 -- 172.21.15.7:0/1946667319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 msgr2=0x55e473475ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3da59c640 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 0x55e473475ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3da59c640 1 -- 172.21.15.7:0/1946667319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 msgr2=0x55e473476920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3da59c640 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e473476920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3da59c640 1 -- 172.21.15.7:0/1946667319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e473213420 con 0x55e4734702e0 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 0x55e473475ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.293+0000 7fb3e359e640 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e473476920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3da59c640 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e4734763e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00e9c0 tx=0x7fb3cc0d4820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a533aa8e3138adb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3ca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/1946667319 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc09c4f0 con 0x55e4734702e0 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/1946667319 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb3cc099200 con 0x55e4734702e0 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3eb922740 1 -- 172.21.15.7:0/1946667319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3d402c7e0 con 0x55e4734702e0 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/1946667319 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb3cc005180 con 0x55e4734702e0 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3eb922740 10 client.16284 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3eb922740 1 -- 172.21.15.7:0/1946667319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e4730a6580 con 0x55e4734702e0 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/1946667319 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3d402c7e0 con 0x55e4734702e0 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/1946667319 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb3cc0d4c60 con 0x55e4734702e0 2024-04-21T08:46:56.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3eb922740 10 client.16284 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:56.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3eb922740 10 client.16284 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-21T08:46:56.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3eb922740 20 client.16284 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:56.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3eb922740 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e47348cf10 0x55e47348f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3eb922740 4 client.16284 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:56.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3eb922740 1 -- 172.21.15.7:0/1946667319 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4733e1b10 con 0x55e47348cf10 2024-04-21T08:46:56.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.294+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e47348cf10 0x55e47348f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.295+0000 7fb3ca7fc640 10 client.16284 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:56.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.295+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e47348cf10 0x55e47348f320 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95dafa835f8019c5 server_cookie=6c9dbde01dc43a68 in_seq=0 out_seq=0 2024-04-21T08:46:56.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.296+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/1946667319 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e4733e1b10 con 0x55e47348cf10 2024-04-21T08:46:56.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.296+0000 7fb3ca7fc640 10 client.16284 handle_command_reply: tid=0 2024-04-21T08:46:56.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.296+0000 7fb3eb922740 1 client.16284 shutdown 2024-04-21T08:46:56.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3eb922740 1 -- 172.21.15.7:0/1946667319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 msgr2=0x55e4734763e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3eb922740 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e4734763e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00e9c0 tx=0x7fb3cc0d4820 comp rx=0 tx=0).stop 2024-04-21T08:46:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3eb922740 1 -- 172.21.15.7:0/1946667319 shutdown_connections 2024-04-21T08:46:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3eb922740 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e47348cf10 0x55e47348f320 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3eb922740 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e473476920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3eb922740 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e4734763e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3ca7fc640 0 client.16284 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3eb922740 1 --2- 172.21.15.7:0/1946667319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 0x55e473475ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3eb922740 1 -- 172.21.15.7:0/1946667319 >> 172.21.15.7:0/1946667319 conn(0x55e47346a0f0 msgr2=0x55e47346be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3eb922740 1 -- 172.21.15.7:0/1946667319 shutdown_connections 2024-04-21T08:46:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3eb922740 1 -- 172.21.15.7:0/1946667319 wait complete. 2024-04-21T08:46:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.297+0000 7fb3eb922740 20 client.16284 trim_cache size 0 max 16384 2024-04-21T08:46:56.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3eb922740 1 Processor -- start 2024-04-21T08:46:56.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3eb922740 1 -- start start 2024-04-21T08:46:56.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 0x55e4734785d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e473471240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e473494bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3eb922740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e41ada40 con 0x55e4734702e0 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3eb922740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e41ab830 con 0x55e473478ec0 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3eb922740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e41ad370 con 0x55e47346f9c0 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3da59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e473471240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3da59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e473471240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40898/0 (socket says 172.21.15.7:40898) 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3da59c640 1 -- 172.21.15.7:0/576225448 learned_addr learned my addr 172.21.15.7:0/576225448 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/576225448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 0x55e4734785d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3e359e640 1 --2- 172.21.15.7:0/576225448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e473494bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3da59c640 1 -- 172.21.15.7:0/576225448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 msgr2=0x55e4734785d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3da59c640 1 --2- 172.21.15.7:0/576225448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 0x55e4734785d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3da59c640 1 -- 172.21.15.7:0/576225448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 msgr2=0x55e473494bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:56.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3da59c640 1 --2- 172.21.15.7:0/576225448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e473494bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3da59c640 1 -- 172.21.15.7:0/576225448 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e473476540 con 0x55e4734702e0 2024-04-21T08:46:56.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/576225448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 0x55e4734785d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:56.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3da59c640 1 --2- 172.21.15.7:0/576225448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e473471240 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc098ff0 tx=0x7fb3cc004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b35bc26fcd20e7d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:56.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.315+0000 7fb3d8d99640 1 -- 172.21.15.7:0/576225448 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc0b5050 con 0x55e4734702e0 2024-04-21T08:46:56.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.316+0000 7fb3d8d99640 1 -- 172.21.15.7:0/576225448 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3cc0b60f0 con 0x55e4734702e0 2024-04-21T08:46:56.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.316+0000 7fb3d8d99640 1 -- 172.21.15.7:0/576225448 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc09d290 con 0x55e4734702e0 2024-04-21T08:46:56.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.316+0000 7fb3eb922740 1 -- 172.21.15.7:0/576225448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 msgr2=0x55e473471240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:56.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.316+0000 7fb3eb922740 1 --2- 172.21.15.7:0/576225448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e473471240 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc098ff0 tx=0x7fb3cc004a90 comp rx=0 tx=0).stop 2024-04-21T08:46:56.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.316+0000 7fb3eb922740 1 -- 172.21.15.7:0/576225448 shutdown_connections 2024-04-21T08:46:56.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.316+0000 7fb3eb922740 1 --2- 172.21.15.7:0/576225448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e473478ec0 0x55e473494bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.316+0000 7fb3eb922740 1 --2- 172.21.15.7:0/576225448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e4734702e0 0x55e473471240 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.316+0000 7fb3eb922740 1 --2- 172.21.15.7:0/576225448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e47346f9c0 0x55e4734785d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.316+0000 7fb3eb922740 1 -- 172.21.15.7:0/576225448 >> 172.21.15.7:0/576225448 conn(0x55e47346a0f0 msgr2=0x55e473468940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:56.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.316+0000 7fb3eb922740 1 -- 172.21.15.7:0/576225448 shutdown_connections 2024-04-21T08:46:56.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3eb922740 1 -- 172.21.15.7:0/576225448 wait complete. 2024-04-21T08:46:56.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3eb922740 1 -- start start 2024-04-21T08:46:56.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3eb922740 1 Processor -- start 2024-04-21T08:46:56.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e473473270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4734702e0 0x55e4734737b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3eb922740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e473478ec0 0x55e4734f2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3eb922740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e41ac5d0 con 0x55e47346f9c0 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3eb922740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e41abf00 con 0x55e4734702e0 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3eb922740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e41acca0 con 0x55e473478ec0 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3eb922740 10 client.0 _ll_register_callbacks cb 0x7fb3cc004990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3e2d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e473473270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3e2d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e473473270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40914/0 (socket says 172.21.15.7:40914) 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3e2d9d640 1 -- 172.21.15.7:0/2130336208 learned_addr learned my addr 172.21.15.7:0/2130336208 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3da59c640 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4734702e0 0x55e4734737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3e359e640 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e473478ec0 0x55e4734f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.317+0000 7fb3e2d9d640 1 -- 172.21.15.7:0/2130336208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e473478ec0 msgr2=0x55e4734f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e473478ec0 0x55e4734f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3e2d9d640 1 -- 172.21.15.7:0/2130336208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4734702e0 msgr2=0x55e4734737b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:56.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4734702e0 0x55e4734737b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3e2d9d640 1 -- 172.21.15.7:0/2130336208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4734f2970 con 0x55e47346f9c0 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3e359e640 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e473478ec0 0x55e4734f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3e2d9d640 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e473473270 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4000c00 tx=0x7fb3d402cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d7305837f19c2a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3ca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/2130336208 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4045040 con 0x55e47346f9c0 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/2130336208 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb3d403f8f0 con 0x55e47346f9c0 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3eb922740 1 -- 172.21.15.7:0/2130336208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e4733db980 con 0x55e47346f9c0 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/2130336208 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb3d4061050 con 0x55e47346f9c0 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3eb922740 10 client.16290 fetch_fsmap learned FSMap version 23 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3eb922740 1 -- 172.21.15.7:0/2130336208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e4734f3c80 con 0x55e47346f9c0 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/2130336208 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e4733db980 con 0x55e47346f9c0 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.318+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/2130336208 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb3d404a8e0 con 0x55e47346f9c0 2024-04-21T08:46:56.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.319+0000 7fb3eb922740 10 client.16290 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:46:56.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.319+0000 7fb3eb922740 10 client.16290 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-21T08:46:56.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.319+0000 7fb3eb922740 20 client.16290 populate_metadata read hostname 'smithi007' 2024-04-21T08:46:56.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.319+0000 7fb3eb922740 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e4734f4080 0x55e4734f6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:56.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.319+0000 7fb3eb922740 4 client.16290 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 30.0, "expiration": 88.95, "await": true}] 2024-04-21T08:46:56.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.319+0000 7fb3eb922740 1 -- 172.21.15.7:0/2130336208 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 30.0, "expiration": 88.95, "await": true}) -- 0x55e473357d30 con 0x55e4734f4080 2024-04-21T08:46:56.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.319+0000 7fb3da59c640 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e4734f4080 0x55e4734f6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:56.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.319+0000 7fb3da59c640 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e4734f4080 0x55e4734f6470 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ea8d654a27d8fcd server_cookie=cc45cd9444638acd in_seq=0 out_seq=0 2024-04-21T08:46:56.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:46:56.319+0000 7fb3ca7fc640 10 client.16290 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:46:56.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:56 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:46:56.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:56 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:56.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:56 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:46:56.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:56 smithi016 ceph-mon[31259]: pgmap v71: 129 pgs: 129 active+clean; 212 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 616 KiB/s wr, 473 op/s 2024-04-21T08:46:56.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:56 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/269006597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:56.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:56 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:46:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:56 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:56 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:46:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:56 smithi077 ceph-mon[31615]: pgmap v71: 129 pgs: 129 active+clean; 212 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 616 KiB/s wr, 473 op/s 2024-04-21T08:46:56.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:56 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/269006597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:56.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:56 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:46:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:56 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:56 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:46:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:56 smithi007 ceph-mon[28477]: pgmap v71: 129 pgs: 129 active+clean; 212 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 616 KiB/s wr, 473 op/s 2024-04-21T08:46:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:56 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/269006597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:58.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:46:58 smithi016 ceph-mon[31259]: pgmap v72: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 804 op/s 2024-04-21T08:46:58.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:46:58 smithi077 ceph-mon[31615]: pgmap v72: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 804 op/s 2024-04-21T08:46:58.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:46:58 smithi007 ceph-mon[28477]: pgmap v72: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 804 op/s 2024-04-21T08:47:00.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:00 smithi016 ceph-mon[31259]: pgmap v73: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 2.9 MiB/s wr, 796 op/s 2024-04-21T08:47:00.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:00 smithi077 ceph-mon[31615]: pgmap v73: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 2.9 MiB/s wr, 796 op/s 2024-04-21T08:47:00.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:00 smithi007 ceph-mon[28477]: pgmap v73: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 2.9 MiB/s wr, 796 op/s 2024-04-21T08:47:02.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:02 smithi016 ceph-mon[31259]: pgmap v74: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 2.9 MiB/s wr, 796 op/s 2024-04-21T08:47:02.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:02 smithi077 ceph-mon[31615]: pgmap v74: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 2.9 MiB/s wr, 796 op/s 2024-04-21T08:47:02.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:02 smithi007 ceph-mon[28477]: pgmap v74: 129 pgs: 129 active+clean; 239 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 2.9 MiB/s wr, 796 op/s 2024-04-21T08:47:04.241 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.239+0000 7fde7c13d640 1 -- 172.21.15.7:0/1086686585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde74103ae0 msgr2=0x7fde7410bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.239+0000 7fde7c13d640 1 --2- 172.21.15.7:0/1086686585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde74103ae0 0x7fde7410bfc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fde64009900 tx=0x7fde6402bcd0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.239+0000 7fde7c13d640 1 -- 172.21.15.7:0/1086686585 shutdown_connections 2024-04-21T08:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.239+0000 7fde7c13d640 1 --2- 172.21.15.7:0/1086686585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7410c590 0x7fde7410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.239+0000 7fde7c13d640 1 --2- 172.21.15.7:0/1086686585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde74103ae0 0x7fde7410bfc0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.239+0000 7fde7c13d640 1 --2- 172.21.15.7:0/1086686585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde741031c0 0x7fde741035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.239+0000 7fde7c13d640 1 -- 172.21.15.7:0/1086686585 >> 172.21.15.7:0/1086686585 conn(0x7fde74077f70 msgr2=0x7fde74078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.239+0000 7fde7c13d640 1 -- 172.21.15.7:0/1086686585 shutdown_connections 2024-04-21T08:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.239+0000 7fde7c13d640 1 -- 172.21.15.7:0/1086686585 wait complete. 2024-04-21T08:47:04.242 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde7c13d640 1 Processor -- start 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde7c13d640 1 -- start start 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde7c13d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde741031c0 0x7fde7419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde7c13d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde74103ae0 0x7fde7419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde7c13d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde7410c590 0x7fde74198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde7c13d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fde741121b0 con 0x7fde7410c590 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde7c13d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde74112030 con 0x7fde741031c0 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde7c13d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fde74112330 con 0x7fde74103ae0 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde7a6da640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde7410c590 0x7fde74198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde7a6da640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde7410c590 0x7fde74198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40926/0 (socket says 172.21.15.7:40926) 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde79ed9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde741031c0 0x7fde7419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.240+0000 7fde7a6da640 1 -- 172.21.15.7:0/1355247537 learned_addr learned my addr 172.21.15.7:0/1355247537 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde796d8640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde74103ae0 0x7fde7419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde7a6da640 1 -- 172.21.15.7:0/1355247537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde74103ae0 msgr2=0x7fde7419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde7a6da640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde74103ae0 0x7fde7419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde7a6da640 1 -- 172.21.15.7:0/1355247537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde741031c0 msgr2=0x7fde7419e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde7a6da640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde741031c0 0x7fde7419e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde7a6da640 1 -- 172.21.15.7:0/1355247537 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde74198da0 con 0x7fde7410c590 2024-04-21T08:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde796d8640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde74103ae0 0x7fde7419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde7a6da640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde7410c590 0x7fde74198620 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00eb40 tx=0x7fde6c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde6affd640 1 -- 172.21.15.7:0/1355247537 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c004b20 con 0x7fde7410c590 2024-04-21T08:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde6affd640 1 -- 172.21.15.7:0/1355247537 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde6c004cc0 con 0x7fde7410c590 2024-04-21T08:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde6affd640 1 -- 172.21.15.7:0/1355247537 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c010720 con 0x7fde7410c590 2024-04-21T08:47:04.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde7c13d640 1 -- 172.21.15.7:0/1355247537 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde74199090 con 0x7fde7410c590 2024-04-21T08:47:04.245 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.241+0000 7fde7c13d640 1 -- 172.21.15.7:0/1355247537 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde741a5330 con 0x7fde7410c590 2024-04-21T08:47:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.244+0000 7fde6affd640 1 -- 172.21.15.7:0/1355247537 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fde6c005560 con 0x7fde7410c590 2024-04-21T08:47:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.244+0000 7fde7ee06740 1 Processor -- start 2024-04-21T08:47:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.244+0000 7fde7ee06740 1 -- start start 2024-04-21T08:47:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.244+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cc2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.244+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9cc2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.244+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9ccc0e0 0x55eac9cce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.244+0000 7fde7ee06740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55eac9c278d0 con 0x55eac9cc38d0 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.244+0000 7fde7ee06740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55eac9bc4e80 con 0x55eac9ccc0e0 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.244+0000 7fde7ee06740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55eac9c23890 con 0x55eac9cc2b60 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde796d8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cc2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde796d8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cc2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40928/0 (socket says 172.21.15.7:40928) 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde796d8640 1 -- 172.21.15.7:0/3938866231 learned_addr learned my addr 172.21.15.7:0/3938866231 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde79ed9640 1 --2- 172.21.15.7:0/3938866231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9cc2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde796d8640 1 --2- 172.21.15.7:0/3938866231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9ccc0e0 0x55eac9cce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde6affd640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde600a3d90 0x7fde600a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde796d8640 1 -- 172.21.15.7:0/3938866231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 msgr2=0x55eac9cc2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde796d8640 1 --2- 172.21.15.7:0/3938866231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9cc2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde796d8640 1 -- 172.21.15.7:0/3938866231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9ccc0e0 msgr2=0x55eac9cce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde7a6da640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde600a3d90 0x7fde600a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde796d8640 1 --2- 172.21.15.7:0/3938866231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9ccc0e0 0x55eac9cce4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde796d8640 1 -- 172.21.15.7:0/3938866231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eac9be6320 con 0x55eac9cc38d0 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde6affd640 1 -- 172.21.15.7:0/1355247537 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fde6c0d1310 con 0x7fde7410c590 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde796d8640 1 --2- 172.21.15.7:0/3938866231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cc2620 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fde64005ab0 tx=0x7fde64005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ae8e04a32202b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.245+0000 7fde7a6da640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde600a3d90 0x7fde600a6250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00c7d0 tx=0x7fde6c092350 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.246+0000 7fde5ffff640 1 -- 172.21.15.7:0/3938866231 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde64004140 con 0x55eac9cc38d0 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.246+0000 7fde5ffff640 1 -- 172.21.15.7:0/3938866231 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde640054f0 con 0x55eac9cc38d0 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.246+0000 7fde5ffff640 1 -- 172.21.15.7:0/3938866231 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6402cdc0 con 0x55eac9cc38d0 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.246+0000 7fde7ee06740 1 -- 172.21.15.7:0/3938866231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 msgr2=0x55eac9cc2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.246+0000 7fde7ee06740 1 --2- 172.21.15.7:0/3938866231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cc2620 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fde64005ab0 tx=0x7fde64005b50 comp rx=0 tx=0).stop 2024-04-21T08:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.246+0000 7fde7ee06740 1 -- 172.21.15.7:0/3938866231 shutdown_connections 2024-04-21T08:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.246+0000 7fde7ee06740 1 --2- 172.21.15.7:0/3938866231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9ccc0e0 0x55eac9cce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.246+0000 7fde7ee06740 1 --2- 172.21.15.7:0/3938866231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9cc2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.246+0000 7fde7ee06740 1 --2- 172.21.15.7:0/3938866231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cc2620 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.246+0000 7fde7ee06740 1 -- 172.21.15.7:0/3938866231 >> 172.21.15.7:0/3938866231 conn(0x55eac9c89c90 msgr2=0x55eac9cbd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.247+0000 7fde7ee06740 1 -- 172.21.15.7:0/3938866231 shutdown_connections 2024-04-21T08:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.247+0000 7fde7ee06740 1 -- 172.21.15.7:0/3938866231 wait complete. 2024-04-21T08:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.247+0000 7fde7ee06740 1 -- start start 2024-04-21T08:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.247+0000 7fde7ee06740 1 Processor -- start 2024-04-21T08:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.247+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9ccb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.247+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9cc38d0 0x55eac9cc8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.247+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9cc9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.247+0000 7fde7ee06740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55eac9ccedb0 con 0x55eac9ccc0e0 2024-04-21T08:47:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.247+0000 7fde7ee06740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55eac9bafeb0 con 0x55eac9cc38d0 2024-04-21T08:47:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.247+0000 7fde7ee06740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55eac9ccef30 con 0x55eac9cc2b60 2024-04-21T08:47:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.247+0000 7fde7ee06740 10 client.0 _ll_register_callbacks cb 0x55eac9bdcb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde796d8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9cc9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde79ed9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9cc38d0 0x55eac9cc8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde796d8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9cc9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40940/0 (socket says 172.21.15.7:40940) 2024-04-21T08:47:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde79ed9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9cc38d0 0x55eac9cc8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33516/0 (socket says 172.21.15.7:33516) 2024-04-21T08:47:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde796d8640 1 -- 172.21.15.7:0/932962706 learned_addr learned my addr 172.21.15.7:0/932962706 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde796d8640 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9ccb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde796d8640 1 -- 172.21.15.7:0/932962706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 msgr2=0x55eac9ccb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde796d8640 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9ccb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde796d8640 1 -- 172.21.15.7:0/932962706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9cc38d0 msgr2=0x55eac9cc8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde796d8640 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9cc38d0 0x55eac9cc8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde796d8640 1 -- 172.21.15.7:0/932962706 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eac993a210 con 0x55eac9ccc0e0 2024-04-21T08:47:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde796d8640 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9cc9330 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fde64005a80 tx=0x7fde6402ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39fb6270663514d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:47:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde5d7fa640 1 -- 172.21.15.7:0/932962706 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde64005250 con 0x55eac9ccc0e0 2024-04-21T08:47:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde5d7fa640 1 -- 172.21.15.7:0/932962706 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fde64004140 con 0x55eac9ccc0e0 2024-04-21T08:47:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde7ee06740 1 -- 172.21.15.7:0/932962706 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde640054f0 con 0x55eac9ccc0e0 2024-04-21T08:47:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde5d7fa640 1 -- 172.21.15.7:0/932962706 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fde64004340 con 0x55eac9ccc0e0 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde7ee06740 10 client.16302 fetch_fsmap learned FSMap version 23 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.248+0000 7fde7ee06740 1 -- 172.21.15.7:0/932962706 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55eac9a06620 con 0x55eac9ccc0e0 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.249+0000 7fde5d7fa640 1 -- 172.21.15.7:0/932962706 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde640054f0 con 0x55eac9ccc0e0 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.249+0000 7fde5d7fa640 1 -- 172.21.15.7:0/932962706 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fde6404b040 con 0x55eac9ccc0e0 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.249+0000 7fde7ee06740 10 client.16302 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.249+0000 7fde7ee06740 10 client.16302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.249+0000 7fde7ee06740 20 client.16302 populate_metadata read hostname 'smithi007' 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.249+0000 7fde7ee06740 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eac9cdbf40 0x55eac9cde330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.249+0000 7fde7ee06740 4 client.16302 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.249+0000 7fde7ee06740 1 -- 172.21.15.7:0/932962706 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eac9c3b880 con 0x55eac9cdbf40 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.249+0000 7fde79ed9640 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eac9cdbf40 0x55eac9cde330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.250+0000 7fde5d7fa640 10 client.16302 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.250+0000 7fde79ed9640 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eac9cdbf40 0x55eac9cde330 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7c08e9766a42c17 server_cookie=d07e6969747edb59 in_seq=0 out_seq=0 2024-04-21T08:47:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde5d7fa640 1 -- 172.21.15.7:0/932962706 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eac9c3b880 con 0x55eac9cdbf40 2024-04-21T08:47:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde5d7fa640 10 client.16302 handle_command_reply: tid=0 2024-04-21T08:47:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 1 client.16302 shutdown 2024-04-21T08:47:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 1 -- 172.21.15.7:0/932962706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 msgr2=0x55eac9cc9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9cc9330 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fde64005a80 tx=0x7fde6402ca00 comp rx=0 tx=0).stop 2024-04-21T08:47:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 1 -- 172.21.15.7:0/932962706 shutdown_connections 2024-04-21T08:47:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eac9cdbf40 0x55eac9cde330 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9cc9330 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9cc38d0 0x55eac9cc8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 1 --2- 172.21.15.7:0/932962706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9ccb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde5d7fa640 0 client.16302 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 1 -- 172.21.15.7:0/932962706 >> 172.21.15.7:0/932962706 conn(0x55eac9c89c90 msgr2=0x55eac9cbf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 1 -- 172.21.15.7:0/932962706 shutdown_connections 2024-04-21T08:47:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 1 -- 172.21.15.7:0/932962706 wait complete. 2024-04-21T08:47:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.255+0000 7fde7ee06740 20 client.16302 trim_cache size 0 max 16384 2024-04-21T08:47:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde7ee06740 1 Processor -- start 2024-04-21T08:47:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde7ee06740 1 -- start start 2024-04-21T08:47:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9cc2b60 0x55eac9cc8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc38d0 0x55eac9ce1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9ce1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde7ee06740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fde741ad470 con 0x55eac9ccc0e0 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde7ee06740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde741afcf0 con 0x55eac9cc2b60 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde7ee06740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fde741adae0 con 0x55eac9cc38d0 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde796d8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9ce1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde796d8640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9ce1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40956/0 (socket says 172.21.15.7:40956) 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde796d8640 1 -- 172.21.15.7:0/3236315495 learned_addr learned my addr 172.21.15.7:0/3236315495 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde796d8640 1 --2- 172.21.15.7:0/3236315495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9cc2b60 0x55eac9cc8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde79ed9640 1 --2- 172.21.15.7:0/3236315495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc38d0 0x55eac9ce1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde796d8640 1 -- 172.21.15.7:0/3236315495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc38d0 msgr2=0x55eac9ce1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde796d8640 1 --2- 172.21.15.7:0/3236315495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc38d0 0x55eac9ce1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde796d8640 1 -- 172.21.15.7:0/3236315495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9cc2b60 msgr2=0x55eac9cc8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde796d8640 1 --2- 172.21.15.7:0/3236315495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9cc2b60 0x55eac9cc8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde796d8640 1 -- 172.21.15.7:0/3236315495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eac9cc1e30 con 0x55eac9ccc0e0 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde796d8640 1 --2- 172.21.15.7:0/3236315495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9ce1890 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fde64009850 tx=0x7fde64005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2bca294b1bbc0f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde5f7fe640 1 -- 172.21.15.7:0/3236315495 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6402c350 con 0x55eac9ccc0e0 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde5f7fe640 1 -- 172.21.15.7:0/3236315495 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde64009520 con 0x55eac9ccc0e0 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.271+0000 7fde5f7fe640 1 -- 172.21.15.7:0/3236315495 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde640046c0 con 0x55eac9ccc0e0 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.272+0000 7fde7ee06740 1 -- 172.21.15.7:0/3236315495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 msgr2=0x55eac9ce1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.272+0000 7fde7ee06740 1 --2- 172.21.15.7:0/3236315495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9ce1890 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fde64009850 tx=0x7fde64005b50 comp rx=0 tx=0).stop 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.272+0000 7fde7ee06740 1 -- 172.21.15.7:0/3236315495 shutdown_connections 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.272+0000 7fde7ee06740 1 --2- 172.21.15.7:0/3236315495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9ccc0e0 0x55eac9ce1890 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.272+0000 7fde7ee06740 1 --2- 172.21.15.7:0/3236315495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc38d0 0x55eac9ce1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.272+0000 7fde7ee06740 1 --2- 172.21.15.7:0/3236315495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9cc2b60 0x55eac9cc8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.272+0000 7fde7ee06740 1 -- 172.21.15.7:0/3236315495 >> 172.21.15.7:0/3236315495 conn(0x55eac9c89c90 msgr2=0x55eac9cc76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.272+0000 7fde7ee06740 1 -- 172.21.15.7:0/3236315495 shutdown_connections 2024-04-21T08:47:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.272+0000 7fde7ee06740 1 -- 172.21.15.7:0/3236315495 wait complete. 2024-04-21T08:47:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.272+0000 7fde7ee06740 1 -- start start 2024-04-21T08:47:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.272+0000 7fde7ee06740 1 Processor -- start 2024-04-21T08:47:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9cdef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cdf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde7ee06740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9ccc0e0 0x55eac9cdf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde7ee06740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fde741ae1b0 con 0x55eac9cc38d0 2024-04-21T08:47:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde7ee06740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde741af620 con 0x55eac9ccc0e0 2024-04-21T08:47:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde7ee06740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fde741ae880 con 0x55eac9cc2b60 2024-04-21T08:47:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde7ee06740 10 client.0 _ll_register_callbacks cb 0x7fde7000b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde79ed9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cdf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde79ed9640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cdf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40966/0 (socket says 172.21.15.7:40966) 2024-04-21T08:47:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde79ed9640 1 -- 172.21.15.7:0/1697846770 learned_addr learned my addr 172.21.15.7:0/1697846770 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde796d8640 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9cdef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde796d8640 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9ccc0e0 0x55eac9cdf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde79ed9640 1 -- 172.21.15.7:0/1697846770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 msgr2=0x55eac9cdef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde79ed9640 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9cdef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde79ed9640 1 -- 172.21.15.7:0/1697846770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9ccc0e0 msgr2=0x55eac9cdf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde79ed9640 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9ccc0e0 0x55eac9cdf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde79ed9640 1 -- 172.21.15.7:0/1697846770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eac9d41370 con 0x55eac9cc38d0 2024-04-21T08:47:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde79ed9640 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cdf4b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fde7000be90 tx=0x7fde70020550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b0d57d425e4d46 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.273+0000 7fde5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:47:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde5d7fa640 1 -- 172.21.15.7:0/1697846770 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde70004220 con 0x55eac9cc38d0 2024-04-21T08:47:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde5d7fa640 1 -- 172.21.15.7:0/1697846770 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fde700043c0 con 0x55eac9cc38d0 2024-04-21T08:47:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde7ee06740 1 -- 172.21.15.7:0/1697846770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde64009520 con 0x55eac9cc38d0 2024-04-21T08:47:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde5d7fa640 1 -- 172.21.15.7:0/1697846770 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fde70011830 con 0x55eac9cc38d0 2024-04-21T08:47:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde7ee06740 10 client.16308 fetch_fsmap learned FSMap version 23 2024-04-21T08:47:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde7ee06740 1 -- 172.21.15.7:0/1697846770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55eac9d426c0 con 0x55eac9cc38d0 2024-04-21T08:47:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde5d7fa640 1 -- 172.21.15.7:0/1697846770 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde64009520 con 0x55eac9cc38d0 2024-04-21T08:47:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde5d7fa640 1 -- 172.21.15.7:0/1697846770 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fde70027070 con 0x55eac9cc38d0 2024-04-21T08:47:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde7ee06740 10 client.16308 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:47:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde7ee06740 10 client.16308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde7ee06740 20 client.16308 populate_metadata read hostname 'smithi007' 2024-04-21T08:47:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde7ee06740 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eac9d42e40 0x55eac9d45260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde7ee06740 4 client.16308 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:47:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde7ee06740 1 -- 172.21.15.7:0/1697846770 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eac9bc73d0 con 0x55eac9d42e40 2024-04-21T08:47:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.274+0000 7fde796d8640 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eac9d42e40 0x55eac9d45260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.275+0000 7fde5d7fa640 10 client.16308 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:04.275+0000 7fde796d8640 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eac9d42e40 0x55eac9d45260 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa8aba08c94edc14 server_cookie=55851a01d4c9e8e6 in_seq=0 out_seq=0 2024-04-21T08:47:04.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:04 smithi016 ceph-mon[31259]: pgmap v75: 129 pgs: 129 active+clean; 244 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 1.16k op/s 2024-04-21T08:47:04.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:04 smithi077 ceph-mon[31615]: pgmap v75: 129 pgs: 129 active+clean; 244 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 1.16k op/s 2024-04-21T08:47:04.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:04 smithi007 ceph-mon[28477]: pgmap v75: 129 pgs: 129 active+clean; 244 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 1.16k op/s 2024-04-21T08:47:06.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:06 smithi016 ceph-mon[31259]: pgmap v76: 129 pgs: 129 active+clean; 244 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 689 op/s 2024-04-21T08:47:06.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:06 smithi077 ceph-mon[31615]: pgmap v76: 129 pgs: 129 active+clean; 244 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 689 op/s 2024-04-21T08:47:06.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:06 smithi007 ceph-mon[28477]: pgmap v76: 129 pgs: 129 active+clean; 244 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 689 op/s 2024-04-21T08:47:08.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:08 smithi016 ceph-mon[31259]: pgmap v77: 129 pgs: 129 active+clean; 250 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 1.53k op/s 2024-04-21T08:47:08.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:08 smithi077 ceph-mon[31615]: pgmap v77: 129 pgs: 129 active+clean; 250 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 1.53k op/s 2024-04-21T08:47:08.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:08 smithi007 ceph-mon[28477]: pgmap v77: 129 pgs: 129 active+clean; 250 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.7 MiB/s wr, 1.53k op/s 2024-04-21T08:47:10.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:10 smithi016 ceph-mon[31259]: pgmap v78: 129 pgs: 129 active+clean; 250 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.3 MiB/s wr, 1.20k op/s 2024-04-21T08:47:10.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:10 smithi077 ceph-mon[31615]: pgmap v78: 129 pgs: 129 active+clean; 250 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.3 MiB/s wr, 1.20k op/s 2024-04-21T08:47:10.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:10 smithi007 ceph-mon[28477]: pgmap v78: 129 pgs: 129 active+clean; 250 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.3 MiB/s wr, 1.20k op/s 2024-04-21T08:47:12.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:12 smithi016 ceph-mon[31259]: pgmap v79: 129 pgs: 129 active+clean; 250 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.3 MiB/s wr, 1.20k op/s 2024-04-21T08:47:12.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:12 smithi077 ceph-mon[31615]: pgmap v79: 129 pgs: 129 active+clean; 250 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.3 MiB/s wr, 1.20k op/s 2024-04-21T08:47:12.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:12 smithi007 ceph-mon[28477]: pgmap v79: 129 pgs: 129 active+clean; 250 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.3 MiB/s wr, 1.20k op/s 2024-04-21T08:47:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:14 smithi016 ceph-mon[31259]: pgmap v80: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 1.76k op/s 2024-04-21T08:47:14.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:14 smithi077 ceph-mon[31615]: pgmap v80: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 1.76k op/s 2024-04-21T08:47:14.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:14 smithi007 ceph-mon[28477]: pgmap v80: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 1.76k op/s 2024-04-21T08:47:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:16 smithi016 ceph-mon[31259]: pgmap v81: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 1.40k op/s 2024-04-21T08:47:16.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:16 smithi077 ceph-mon[31615]: pgmap v81: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 1.40k op/s 2024-04-21T08:47:16.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:16 smithi007 ceph-mon[28477]: pgmap v81: 129 pgs: 129 active+clean; 255 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 1.40k op/s 2024-04-21T08:47:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:18 smithi016 ceph-mon[31259]: pgmap v82: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 2.17k op/s 2024-04-21T08:47:18.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:18 smithi077 ceph-mon[31615]: pgmap v82: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 2.17k op/s 2024-04-21T08:47:18.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:18 smithi007 ceph-mon[28477]: pgmap v82: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 2.17k op/s 2024-04-21T08:47:20.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:20 smithi016 ceph-mon[31259]: pgmap v83: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1.33k op/s 2024-04-21T08:47:20.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:20 smithi077 ceph-mon[31615]: pgmap v83: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1.33k op/s 2024-04-21T08:47:20.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:20 smithi007 ceph-mon[28477]: pgmap v83: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1.33k op/s 2024-04-21T08:47:22.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:22 smithi016 ceph-mon[31259]: pgmap v84: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1.33k op/s 2024-04-21T08:47:22.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:22 smithi077 ceph-mon[31615]: pgmap v84: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1.33k op/s 2024-04-21T08:47:22.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:22 smithi007 ceph-mon[28477]: pgmap v84: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1.33k op/s 2024-04-21T08:47:24.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:24 smithi016 ceph-mon[31259]: pgmap v85: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1.37k op/s 2024-04-21T08:47:24.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:24 smithi077 ceph-mon[31615]: pgmap v85: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1.37k op/s 2024-04-21T08:47:24.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:24 smithi007 ceph-mon[28477]: pgmap v85: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 MiB/s wr, 1.37k op/s 2024-04-21T08:47:25.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:47:25.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:47:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.502+0000 7f8609c06640 1 -- 172.21.15.7:0/2048232610 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8604106140 msgr2=0x7f860410dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.502+0000 7f8609c06640 1 --2- 172.21.15.7:0/2048232610 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8604106140 0x7f860410dd10 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009ca0 tx=0x7f85fc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.502+0000 7f8609c06640 1 -- 172.21.15.7:0/2048232610 shutdown_connections 2024-04-21T08:47:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.502+0000 7f8609c06640 1 --2- 172.21.15.7:0/2048232610 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8604106140 0x7f860410dd10 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.502+0000 7f8609c06640 1 --2- 172.21.15.7:0/2048232610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604101a20 0x7f8604105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.502+0000 7f8609c06640 1 --2- 172.21.15.7:0/2048232610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8604101070 0x7f8604101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.502+0000 7f8609c06640 1 -- 172.21.15.7:0/2048232610 >> 172.21.15.7:0/2048232610 conn(0x7f8604077f40 msgr2=0x7f8604078350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.502+0000 7f8609c06640 1 -- 172.21.15.7:0/2048232610 shutdown_connections 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.502+0000 7f8609c06640 1 -- 172.21.15.7:0/2048232610 wait complete. 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.503+0000 7f8609c06640 1 Processor -- start 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.503+0000 7f8609c06640 1 -- start start 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.503+0000 7f8609c06640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604101070 0x7f8604198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.503+0000 7f8609c06640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8604106140 0x7f8604198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.503+0000 7f8609c06640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860419d020 0x7f860419d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.503+0000 7f8609c06640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f86041110b0 con 0x7f860419d020 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.503+0000 7f8609c06640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8604110f30 con 0x7f8604101070 2024-04-21T08:47:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8609c06640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8604111230 con 0x7f8604106140 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8603fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860419d020 0x7f860419d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8603fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860419d020 0x7f860419d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60554/0 (socket says 172.21.15.7:60554) 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8603fff640 1 -- 172.21.15.7:0/3598735715 learned_addr learned my addr 172.21.15.7:0/3598735715 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8602ffd640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8604106140 0x7f8604198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f86037fe640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604101070 0x7f8604198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8603fff640 1 -- 172.21.15.7:0/3598735715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8604106140 msgr2=0x7f8604198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8603fff640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8604106140 0x7f8604198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8603fff640 1 -- 172.21.15.7:0/3598735715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604101070 msgr2=0x7f8604198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8603fff640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604101070 0x7f8604198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8603fff640 1 -- 172.21.15.7:0/3598735715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860419db10 con 0x7f860419d020 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8603fff640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860419d020 0x7f860419d4d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f85fc005e30 tx=0x7f85fc0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8600ff9640 1 -- 172.21.15.7:0/3598735715 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc041020 con 0x7f860419d020 2024-04-21T08:47:25.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8600ff9640 1 -- 172.21.15.7:0/3598735715 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85fc032970 con 0x7f860419d020 2024-04-21T08:47:25.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8600ff9640 1 -- 172.21.15.7:0/3598735715 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc039040 con 0x7f860419d020 2024-04-21T08:47:25.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8609c06640 1 -- 172.21.15.7:0/3598735715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860419dda0 con 0x7f860419d020 2024-04-21T08:47:25.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.504+0000 7f8609c06640 1 -- 172.21.15.7:0/3598735715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86041a55c0 con 0x7f860419d020 2024-04-21T08:47:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f8600ff9640 1 -- 172.21.15.7:0/3598735715 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f85fc044020 con 0x7f860419d020 2024-04-21T08:47:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f860c7fc740 1 Processor -- start 2024-04-21T08:47:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f860c7fc740 1 -- start start 2024-04-21T08:47:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f8600ff9640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f85ec0a3da0 0x7f85ec0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 0x55d0f0294960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f029da80 0x55d0f029fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f860c7fc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d0f01f91e0 con 0x55d0f029da80 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f860c7fc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d0f0196390 con 0x55d0f0294580 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f860c7fc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d0f01f51a0 con 0x55d0f0294ea0 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f8602ffd640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f85ec0a3da0 0x7f85ec0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.507+0000 7f8600ff9640 1 -- 172.21.15.7:0/3598735715 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f85fc0f26f0 con 0x7f860419d020 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f86037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 0x55d0f0294960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f8603fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f029da80 0x55d0f029fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f86037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 0x55d0f0294960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:32964/0 (socket says 172.21.15.7:32964) 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f86037fe640 1 -- 172.21.15.7:0/3999285045 learned_addr learned my addr 172.21.15.7:0/3999285045 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f8602ffd640 1 --2- 172.21.15.7:0/3999285045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f86037fe640 1 -- 172.21.15.7:0/3999285045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 msgr2=0x55d0f029d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f86037fe640 1 --2- 172.21.15.7:0/3999285045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f86037fe640 1 -- 172.21.15.7:0/3999285045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f029da80 msgr2=0x55d0f029fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f86037fe640 1 --2- 172.21.15.7:0/3999285045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f029da80 0x55d0f029fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f86037fe640 1 -- 172.21.15.7:0/3999285045 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0effd57d0 con 0x55d0f0294580 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f8603fff640 1 --2- 172.21.15.7:0/3999285045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f029da80 0x55d0f029fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f8602ffd640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f85ec0a3da0 0x7f85ec0a6260 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8604100c80 tx=0x7f85f8009f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f86037fe640 1 --2- 172.21.15.7:0/3999285045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 0x55d0f0294960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009c10 tx=0x7f85f40117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2138565cee7af19c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f85e9ffb640 1 -- 172.21.15.7:0/3999285045 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4012450 con 0x55d0f0294580 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f85e9ffb640 1 -- 172.21.15.7:0/3999285045 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f4012aa0 con 0x55d0f0294580 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.508+0000 7f85e9ffb640 1 -- 172.21.15.7:0/3999285045 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f400f9f0 con 0x55d0f0294580 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.509+0000 7f860c7fc740 1 -- 172.21.15.7:0/3999285045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 msgr2=0x55d0f0294960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.509+0000 7f860c7fc740 1 --2- 172.21.15.7:0/3999285045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 0x55d0f0294960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009c10 tx=0x7f85f40117b0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.509+0000 7f860c7fc740 1 -- 172.21.15.7:0/3999285045 shutdown_connections 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.509+0000 7f860c7fc740 1 --2- 172.21.15.7:0/3999285045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f029da80 0x55d0f029fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.509+0000 7f860c7fc740 1 --2- 172.21.15.7:0/3999285045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.509+0000 7f860c7fc740 1 --2- 172.21.15.7:0/3999285045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 0x55d0f0294960 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.509+0000 7f860c7fc740 1 -- 172.21.15.7:0/3999285045 >> 172.21.15.7:0/3999285045 conn(0x55d0f028ecb0 msgr2=0x55d0f028f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.509+0000 7f860c7fc740 1 -- 172.21.15.7:0/3999285045 shutdown_connections 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.509+0000 7f860c7fc740 1 -- 172.21.15.7:0/3999285045 wait complete. 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.509+0000 7f860c7fc740 1 -- start start 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f860c7fc740 1 Processor -- start 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f029da80 0x55d0f029b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f860c7fc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d0f02a0870 con 0x55d0f0294580 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f860c7fc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d0f01b22f0 con 0x55d0f029da80 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f860c7fc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d0f02a09f0 con 0x55d0f0294ea0 2024-04-21T08:47:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f860c7fc740 10 client.0 _ll_register_callbacks cb 0x55d0f01bf580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f86037fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f86037fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60564/0 (socket says 172.21.15.7:60564) 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f86037fe640 1 -- 172.21.15.7:0/2049948536 learned_addr learned my addr 172.21.15.7:0/2049948536 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f8602ffd640 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f8603fff640 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f029da80 0x55d0f029b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f86037fe640 1 -- 172.21.15.7:0/2049948536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 msgr2=0x55d0f029af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f86037fe640 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f86037fe640 1 -- 172.21.15.7:0/2049948536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f029da80 msgr2=0x55d0f029b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f86037fe640 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f029da80 0x55d0f029b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.510+0000 7f86037fe640 1 -- 172.21.15.7:0/2049948536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0eff0b3f0 con 0x55d0f0294580 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f86037fe640 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029aa50 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f85f400f710 tx=0x7f85f4012770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75063258ff8fd902 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f85e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f85e2ffd640 1 -- 172.21.15.7:0/2049948536 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4029020 con 0x55d0f0294580 2024-04-21T08:47:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f85e2ffd640 1 -- 172.21.15.7:0/2049948536 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f85f400f9c0 con 0x55d0f0294580 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f860c7fc740 1 -- 172.21.15.7:0/2049948536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85f4012aa0 con 0x55d0f0294580 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f85e2ffd640 1 -- 172.21.15.7:0/2049948536 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f85f4034050 con 0x55d0f0294580 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f860c7fc740 10 client.16320 fetch_fsmap learned FSMap version 23 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f860c7fc740 1 -- 172.21.15.7:0/2049948536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d0efecb0f0 con 0x55d0f0294580 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f85e2ffd640 1 -- 172.21.15.7:0/2049948536 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f4012aa0 con 0x55d0f0294580 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f85e2ffd640 1 -- 172.21.15.7:0/2049948536 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f85f40046c0 con 0x55d0f0294580 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f860c7fc740 10 client.16320 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f860c7fc740 10 client.16320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f860c7fc740 20 client.16320 populate_metadata read hostname 'smithi007' 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f860c7fc740 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0f02ada10 0x55d0f02afe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f860c7fc740 4 client.16320 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.511+0000 7f860c7fc740 1 -- 172.21.15.7:0/2049948536 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0f0206c00 con 0x55d0f02ada10 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.512+0000 7f8602ffd640 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0f02ada10 0x55d0f02afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.512+0000 7f8602ffd640 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0f02ada10 0x55d0f02afe00 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=751cb01908697299 server_cookie=c0068c82dbcebaae in_seq=0 out_seq=0 2024-04-21T08:47:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:25.512+0000 7f85e2ffd640 10 client.16320 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:26.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3ca7fc640 1 -- 172.21.15.7:0/2130336208 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: -110 ) ==== 8+0+590 (crc 0 0 0) 0x55e473357d30 con 0x55e4734f4080 2024-04-21T08:47:26.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3ca7fc640 10 client.16290 handle_command_reply: tid=0 2024-04-21T08:47:26.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3eb922740 1 client.16290 shutdown 2024-04-21T08:47:26.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3eb922740 1 -- 172.21.15.7:0/2130336208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 msgr2=0x55e473473270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:26.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3eb922740 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e473473270 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4000c00 tx=0x7fb3d402cdc0 comp rx=0 tx=0).stop 2024-04-21T08:47:26.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3eb922740 1 -- 172.21.15.7:0/2130336208 shutdown_connections 2024-04-21T08:47:26.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3eb922740 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e4734f4080 0x55e4734f6470 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:26.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3eb922740 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e473478ec0 0x55e4734f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:26.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3eb922740 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e4734702e0 0x55e4734737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:26.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3eb922740 1 --2- 172.21.15.7:0/2130336208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e47346f9c0 0x55e473473270 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:26.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3ca7fc640 0 client.16290 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:26.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3eb922740 1 -- 172.21.15.7:0/2130336208 >> 172.21.15.7:0/2130336208 conn(0x55e47346a0f0 msgr2=0x55e4734691d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:26.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3eb922740 1 -- 172.21.15.7:0/2130336208 shutdown_connections 2024-04-21T08:47:26.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.322+0000 7fb3eb922740 1 -- 172.21.15.7:0/2130336208 wait complete. 2024-04-21T08:47:26.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:26.323+0000 7fb3eb922740 20 client.16290 trim_cache size 0 max 16384 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 23, 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: "leader": 24533, 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: "set_version": 2, 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: "sets": { 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: "6d199cab": { 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: "version": 2, 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: "age_ref": 0.0, 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: "state": { 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "TIMEDOUT", 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 0.0 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:47:26.324 INFO:teuthology.orchestra.run.smithi007.stdout: "timeout": 30.0, 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: "expiration": 89.0, 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: "members": { 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: "file:/": { 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: "excluded": false, 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: "state": { 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "QUIESCING", 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: "age": 30.0 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:47:26.325 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:47:26.326 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-21T08:47:26.326 INFO:teuthology.orchestra.run.smithi007.stderr:Error ETIMEDOUT: 2024-04-21T08:47:26.327 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: -110 (ETIMEDOUT), stdout: { "epoch": 23, "leader": 24533, "set_version": 2, "sets": { "6d199cab": { "version": 2, "age_ref": 0.0, "state": { "name": "TIMEDOUT", "age": 0.0 }, "timeout": 30.0, "expiration": 89.0, "members": { "file:/": { "excluded": false, "state": { "name": "QUIESCING", "age": 30.0 } } } } } } 2024-04-21T08:47:26.327 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 240, in _run self.do_quiesce(d) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 198, in do_quiesce raise RuntimeError(f"Error quiescing root: {rcinfo}") RuntimeError: Error quiescing root: -110 (ETIMEDOUT) 2024-04-21T08:47:26.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:26 smithi007 ceph-mon[28477]: pgmap v86: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 560 KiB/s wr, 809 op/s 2024-04-21T08:47:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:26 smithi016 ceph-mon[31259]: pgmap v86: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 560 KiB/s wr, 809 op/s 2024-04-21T08:47:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:26 smithi077 ceph-mon[31615]: pgmap v86: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 560 KiB/s wr, 809 op/s 2024-04-21T08:47:27.363 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.363+0000 7fde5d7fa640 1 -- 172.21.15.7:0/1697846770 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eac9bc73d0 con 0x55eac9d42e40 2024-04-21T08:47:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.363+0000 7fde5d7fa640 10 client.16308 handle_command_reply: tid=0 2024-04-21T08:47:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.363+0000 7fde7ee06740 1 client.16308 shutdown 2024-04-21T08:47:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde7ee06740 1 -- 172.21.15.7:0/1697846770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 msgr2=0x55eac9cdf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde7ee06740 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cdf4b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fde7000be90 tx=0x7fde70020550 comp rx=0 tx=0).stop 2024-04-21T08:47:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde7ee06740 1 -- 172.21.15.7:0/1697846770 shutdown_connections 2024-04-21T08:47:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde7ee06740 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55eac9d42e40 0x55eac9d45260 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde7ee06740 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55eac9ccc0e0 0x55eac9cdf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde7ee06740 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55eac9cc38d0 0x55eac9cdf4b0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde5d7fa640 0 client.16308 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde7ee06740 1 --2- 172.21.15.7:0/1697846770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55eac9cc2b60 0x55eac9cdef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde7ee06740 1 -- 172.21.15.7:0/1697846770 >> 172.21.15.7:0/1697846770 conn(0x55eac9c89c90 msgr2=0x55eac9cd7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:27.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f85e2ffd640 1 -- 172.21.15.7:0/2049948536 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0f0206c00 con 0x55d0f02ada10 2024-04-21T08:47:27.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f85e2ffd640 10 client.16320 handle_command_reply: tid=0 2024-04-21T08:47:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde7ee06740 1 -- 172.21.15.7:0/1697846770 shutdown_connections 2024-04-21T08:47:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde7ee06740 1 -- 172.21.15.7:0/1697846770 wait complete. 2024-04-21T08:47:27.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f860c7fc740 1 client.16320 shutdown 2024-04-21T08:47:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.364+0000 7fde7ee06740 20 client.16308 trim_cache size 0 max 16384 2024-04-21T08:47:27.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f860c7fc740 1 -- 172.21.15.7:0/2049948536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 msgr2=0x55d0f029aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:27.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f860c7fc740 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029aa50 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f85f400f710 tx=0x7f85f4012770 comp rx=0 tx=0).stop 2024-04-21T08:47:27.367 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:47:27.367 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:47:27.367 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:47:27.367 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:47:27.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f860c7fc740 1 -- 172.21.15.7:0/2049948536 shutdown_connections 2024-04-21T08:47:27.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f860c7fc740 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0f02ada10 0x55d0f02afe00 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f860c7fc740 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f029da80 0x55d0f029b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f860c7fc740 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f860c7fc740 1 --2- 172.21.15.7:0/2049948536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029aa50 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f85e2ffd640 0 client.16320 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:27.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f860c7fc740 1 -- 172.21.15.7:0/2049948536 >> 172.21.15.7:0/2049948536 conn(0x55d0f028ecb0 msgr2=0x55d0f0290a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:27.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f860c7fc740 1 -- 172.21.15.7:0/2049948536 shutdown_connections 2024-04-21T08:47:27.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.364+0000 7f860c7fc740 1 -- 172.21.15.7:0/2049948536 wait complete. 2024-04-21T08:47:27.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.365+0000 7f860c7fc740 20 client.16320 trim_cache size 0 max 16384 2024-04-21T08:47:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 -- 172.21.15.7:0/1355247537 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde600a3d90 msgr2=0x7fde600a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde600a3d90 0x7fde600a6250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00c7d0 tx=0x7fde6c092350 comp rx=0 tx=0).stop 2024-04-21T08:47:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 -- 172.21.15.7:0/1355247537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde7410c590 msgr2=0x7fde74198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:27.370 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde7410c590 0x7fde74198620 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00eb40 tx=0x7fde6c00c2d0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.370 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 -- 172.21.15.7:0/1355247537 shutdown_connections 2024-04-21T08:47:27.370 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde7410c590 0x7fde74198620 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.370 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fde600a3d90 0x7fde600a6250 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.370 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde74103ae0 0x7fde7419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.370 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 --2- 172.21.15.7:0/1355247537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde741031c0 0x7fde7419e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.370 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 -- 172.21.15.7:0/1355247537 >> 172.21.15.7:0/1355247537 conn(0x7fde74077f70 msgr2=0x7fde740ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:27.370 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 -- 172.21.15.7:0/1355247537 shutdown_connections 2024-04-21T08:47:27.370 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:27.365+0000 7fde7c13d640 1 -- 172.21.15.7:0/1355247537 wait complete. 2024-04-21T08:47:27.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.384+0000 7f860c7fc740 1 Processor -- start 2024-04-21T08:47:27.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.384+0000 7f860c7fc740 1 -- start start 2024-04-21T08:47:27.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 0x55d0f02a9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:27.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294ea0 0x55d0f02a6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:27.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f029da80 0x55d0f0299fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f860c7fc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f86041ad470 con 0x55d0f0294ea0 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f860c7fc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f86041afcf0 con 0x55d0f0294580 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f860c7fc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86041adae0 con 0x55d0f029da80 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f8602ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294ea0 0x55d0f02a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f86037fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 0x55d0f02a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f8602ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294ea0 0x55d0f02a6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60570/0 (socket says 172.21.15.7:60570) 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f8602ffd640 1 -- 172.21.15.7:0/3159943703 learned_addr learned my addr 172.21.15.7:0/3159943703 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f8603fff640 1 --2- 172.21.15.7:0/3159943703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f029da80 0x55d0f0299fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f8602ffd640 1 -- 172.21.15.7:0/3159943703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f029da80 msgr2=0x55d0f0299fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f8602ffd640 1 --2- 172.21.15.7:0/3159943703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f029da80 0x55d0f0299fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f8602ffd640 1 -- 172.21.15.7:0/3159943703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 msgr2=0x55d0f02a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:27.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f8602ffd640 1 --2- 172.21.15.7:0/3159943703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 0x55d0f02a9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f8602ffd640 1 -- 172.21.15.7:0/3159943703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0f028f2c0 con 0x55d0f0294ea0 2024-04-21T08:47:27.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f86037fe640 1 --2- 172.21.15.7:0/3159943703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 0x55d0f02a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:27.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f8602ffd640 1 --2- 172.21.15.7:0/3159943703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294ea0 0x55d0f02a6020 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009d70 tx=0x7f85f8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c207911bf6b52b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:27.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.385+0000 7f85e97fa640 1 -- 172.21.15.7:0/3159943703 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8028e50 con 0x55d0f0294ea0 2024-04-21T08:47:27.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.386+0000 7f85e97fa640 1 -- 172.21.15.7:0/3159943703 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f80054d0 con 0x55d0f0294ea0 2024-04-21T08:47:27.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.386+0000 7f85e97fa640 1 -- 172.21.15.7:0/3159943703 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8004a00 con 0x55d0f0294ea0 2024-04-21T08:47:27.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.386+0000 7f860c7fc740 1 -- 172.21.15.7:0/3159943703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294ea0 msgr2=0x55d0f02a6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:27.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.386+0000 7f860c7fc740 1 --2- 172.21.15.7:0/3159943703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294ea0 0x55d0f02a6020 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009d70 tx=0x7f85f8005f90 comp rx=0 tx=0).stop 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.386+0000 7f860c7fc740 1 -- 172.21.15.7:0/3159943703 shutdown_connections 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.386+0000 7f860c7fc740 1 --2- 172.21.15.7:0/3159943703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f029da80 0x55d0f0299fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.386+0000 7f860c7fc740 1 --2- 172.21.15.7:0/3159943703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294ea0 0x55d0f02a6020 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.386+0000 7f860c7fc740 1 --2- 172.21.15.7:0/3159943703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f0294580 0x55d0f02a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.386+0000 7f860c7fc740 1 -- 172.21.15.7:0/3159943703 >> 172.21.15.7:0/3159943703 conn(0x55d0f028ecb0 msgr2=0x55d0f028dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.386+0000 7f860c7fc740 1 -- 172.21.15.7:0/3159943703 shutdown_connections 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.386+0000 7f860c7fc740 1 -- 172.21.15.7:0/3159943703 wait complete. 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f860c7fc740 1 -- start start 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f860c7fc740 1 Processor -- start 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:27.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f860c7fc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f029da80 0x55d0f030f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f860c7fc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f86041ae1b0 con 0x55d0f0294580 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f860c7fc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f86041af620 con 0x55d0f029da80 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f860c7fc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f86041ae880 con 0x55d0f0294ea0 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f860c7fc740 10 client.0 _ll_register_callbacks cb 0x7f85fc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f86037fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f86037fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60580/0 (socket says 172.21.15.7:60580) 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f86037fe640 1 -- 172.21.15.7:0/1498122759 learned_addr learned my addr 172.21.15.7:0/1498122759 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.387+0000 7f8603fff640 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f029da80 0x55d0f030f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f8602ffd640 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f86037fe640 1 -- 172.21.15.7:0/1498122759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 msgr2=0x55d0f029ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f86037fe640 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f86037fe640 1 -- 172.21.15.7:0/1498122759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f029da80 msgr2=0x55d0f030f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f86037fe640 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f029da80 0x55d0f030f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f86037fe640 1 -- 172.21.15.7:0/1498122759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0f030f7e0 con 0x55d0f0294580 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f8602ffd640 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f8603fff640 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f029da80 0x55d0f030f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f86037fe640 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029a8e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f85f4004ae0 tx=0x7f85f40127e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd36f93a9370af75 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f85e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f85e2ffd640 1 -- 172.21.15.7:0/1498122759 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f403d060 con 0x55d0f0294580 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f85e2ffd640 1 -- 172.21.15.7:0/1498122759 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f85f400f440 con 0x55d0f0294580 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f860c7fc740 1 -- 172.21.15.7:0/1498122759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85f80054d0 con 0x55d0f0294580 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f85e2ffd640 1 -- 172.21.15.7:0/1498122759 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f85f4044050 con 0x55d0f0294580 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f860c7fc740 10 client.16326 fetch_fsmap learned FSMap version 23 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.388+0000 7f860c7fc740 1 -- 172.21.15.7:0/1498122759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d0f0317d90 con 0x55d0f0294580 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.389+0000 7f85e2ffd640 1 -- 172.21.15.7:0/1498122759 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f80054d0 con 0x55d0f0294580 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.389+0000 7f85e2ffd640 1 -- 172.21.15.7:0/1498122759 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f85f40035f0 con 0x55d0f0294580 2024-04-21T08:47:27.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.389+0000 7f860c7fc740 10 client.16326 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.389+0000 7f860c7fc740 10 client.16326 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.389+0000 7f860c7fc740 20 client.16326 populate_metadata read hostname 'smithi007' 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.389+0000 7f860c7fc740 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0f0318540 0x55d0f031a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.389+0000 7f860c7fc740 4 client.16326 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.389+0000 7f860c7fc740 1 -- 172.21.15.7:0/1498122759 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d0f0200a70 con 0x55d0f0318540 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.389+0000 7f8602ffd640 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0f0318540 0x55d0f031a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.389+0000 7f8602ffd640 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0f0318540 0x55d0f031a930 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7413a3f59ad249d server_cookie=1ed1eed655e7b22 in_seq=0 out_seq=0 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.389+0000 7f85e2ffd640 10 client.16326 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f85e2ffd640 1 -- 172.21.15.7:0/1498122759 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x55d0f0200a70 con 0x55d0f0318540 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f85e2ffd640 10 client.16326 handle_command_reply: tid=0 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 1 client.16326 shutdown 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 1 -- 172.21.15.7:0/1498122759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 msgr2=0x55d0f029a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029a8e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f85f4004ae0 tx=0x7f85f40127e0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 1 -- 172.21.15.7:0/1498122759 shutdown_connections 2024-04-21T08:47:27.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0f0318540 0x55d0f031a930 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0f029da80 0x55d0f030f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0f0294ea0 0x55d0f029ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f85e2ffd640 0 client.16326 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:27.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 1 --2- 172.21.15.7:0/1498122759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0f0294580 0x55d0f029a8e0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 1 -- 172.21.15.7:0/1498122759 >> 172.21.15.7:0/1498122759 conn(0x55d0f028ecb0 msgr2=0x55d0f028ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:27.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 1 -- 172.21.15.7:0/1498122759 shutdown_connections 2024-04-21T08:47:27.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 1 -- 172.21.15.7:0/1498122759 wait complete. 2024-04-21T08:47:27.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.391+0000 7f860c7fc740 20 client.16326 trim_cache size 0 max 16384 2024-04-21T08:47:27.396 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:47:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (12 inodes in the stack)", 2024-04-21T08:47:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:47:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: "b0adb205-b024-4f89-ba40-5cd98d91d5a6": { 2024-04-21T08:47:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:47:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "b0adb205-b024-4f89-ba40-5cd98d91d5a6", 2024-04-21T08:47:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:47:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:47:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:47:27.397 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:47:27.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 -- 172.21.15.7:0/3598735715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f85ec0a3da0 msgr2=0x7f85ec0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:27.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f85ec0a3da0 0x7f85ec0a6260 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8604100c80 tx=0x7f85f8009f90 comp rx=0 tx=0).stop 2024-04-21T08:47:27.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 -- 172.21.15.7:0/3598735715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860419d020 msgr2=0x7f860419d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:27.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860419d020 0x7f860419d4d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f85fc005e30 tx=0x7f85fc0346d0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 -- 172.21.15.7:0/3598735715 shutdown_connections 2024-04-21T08:47:27.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860419d020 0x7f860419d4d0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f85ec0a3da0 0x7f85ec0a6260 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8604106140 0x7f8604198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 --2- 172.21.15.7:0/3598735715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8604101070 0x7f8604198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:27.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 -- 172.21.15.7:0/3598735715 >> 172.21.15.7:0/3598735715 conn(0x7f8604077f40 msgr2=0x7f860406a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:27.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 -- 172.21.15.7:0/3598735715 shutdown_connections 2024-04-21T08:47:27.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:27.392+0000 7f8609c06640 1 -- 172.21.15.7:0/3598735715 wait complete. 2024-04-21T08:47:27.404 INFO:tasks.cephfs.filesystem:scrub status for tag:b0adb205-b024-4f89-ba40-5cd98d91d5a6 - {'path': '/', 'tag': 'b0adb205-b024-4f89-ba40-5cd98d91d5a6', 'options': 'recursive,force'} 2024-04-21T08:47:28.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:28 smithi016 ceph-mon[31259]: pgmap v87: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 588 KiB/s wr, 817 op/s 2024-04-21T08:47:28.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:28 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:28.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:28 smithi077 ceph-mon[31615]: pgmap v87: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 588 KiB/s wr, 817 op/s 2024-04-21T08:47:28.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:28 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:28.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:28 smithi007 ceph-mon[28477]: pgmap v87: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 588 KiB/s wr, 817 op/s 2024-04-21T08:47:28.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:28 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:30.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:30 smithi016 ceph-mon[31259]: pgmap v88: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 52 KiB/s wr, 45 op/s 2024-04-21T08:47:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:30 smithi077 ceph-mon[31615]: pgmap v88: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 52 KiB/s wr, 45 op/s 2024-04-21T08:47:30.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:30 smithi007 ceph-mon[28477]: pgmap v88: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 52 KiB/s wr, 45 op/s 2024-04-21T08:47:32.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:32 smithi016 ceph-mon[31259]: pgmap v89: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 52 KiB/s wr, 45 op/s 2024-04-21T08:47:32.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:32 smithi077 ceph-mon[31615]: pgmap v89: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 52 KiB/s wr, 45 op/s 2024-04-21T08:47:32.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:32 smithi007 ceph-mon[28477]: pgmap v89: 129 pgs: 129 active+clean; 261 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 52 KiB/s wr, 45 op/s 2024-04-21T08:47:33.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:33 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:47:33.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:33 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:47:33.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:33 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:47:34.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:34 smithi016 ceph-mon[31259]: pgmap v90: 129 pgs: 129 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-21T08:47:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:34 smithi077 ceph-mon[31615]: pgmap v90: 129 pgs: 129 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-21T08:47:34.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:34 smithi007 ceph-mon[28477]: pgmap v90: 129 pgs: 129 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.6 MiB/s wr, 193 op/s 2024-04-21T08:47:36.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:36 smithi016 ceph-mon[31259]: pgmap v91: 129 pgs: 129 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.5 MiB/s wr, 155 op/s 2024-04-21T08:47:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:36 smithi077 ceph-mon[31615]: pgmap v91: 129 pgs: 129 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.5 MiB/s wr, 155 op/s 2024-04-21T08:47:36.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:36 smithi007 ceph-mon[28477]: pgmap v91: 129 pgs: 129 active+clean; 262 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.5 MiB/s wr, 155 op/s 2024-04-21T08:47:38.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:38 smithi016 ceph-mon[31259]: pgmap v92: 129 pgs: 129 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-04-21T08:47:38.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:38 smithi077 ceph-mon[31615]: pgmap v92: 129 pgs: 129 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-04-21T08:47:38.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:38 smithi007 ceph-mon[28477]: pgmap v92: 129 pgs: 129 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-04-21T08:47:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.466+0000 7fc72de93640 1 -- 172.21.15.7:0/3052086739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc728107270 msgr2=0x7fc72810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.466+0000 7fc72de93640 1 --2- 172.21.15.7:0/3052086739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc728107270 0x7fc72810ee40 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc720009ca0 tx=0x7fc72002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.466+0000 7fc72de93640 1 -- 172.21.15.7:0/3052086739 shutdown_connections 2024-04-21T08:47:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.466+0000 7fc72de93640 1 --2- 172.21.15.7:0/3052086739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc728107270 0x7fc72810ee40 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.466 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.466+0000 7fc72de93640 1 --2- 172.21.15.7:0/3052086739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc728102b50 0x7fc728106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.466+0000 7fc72de93640 1 --2- 172.21.15.7:0/3052086739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc7281021a0 0x7fc728102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.466+0000 7fc72de93640 1 -- 172.21.15.7:0/3052086739 >> 172.21.15.7:0/3052086739 conn(0x7fc7280fda40 msgr2=0x7fc7280ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.466+0000 7fc72de93640 1 -- 172.21.15.7:0/3052086739 shutdown_connections 2024-04-21T08:47:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.466+0000 7fc72de93640 1 -- 172.21.15.7:0/3052086739 wait complete. 2024-04-21T08:47:40.467 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.467+0000 7fc72de93640 1 Processor -- start 2024-04-21T08:47:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.467+0000 7fc72de93640 1 -- start start 2024-04-21T08:47:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.467+0000 7fc72de93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7281021a0 0x7fc72819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.467+0000 7fc72de93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc728107270 0x7fc72819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.467+0000 7fc72de93640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7281985a0 0x7fc728198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.467+0000 7fc72de93640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc7281121e0 con 0x7fc7281985a0 2024-04-21T08:47:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.467+0000 7fc72de93640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc728112060 con 0x7fc7281021a0 2024-04-21T08:47:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.467+0000 7fc72de93640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc728112360 con 0x7fc728107270 2024-04-21T08:47:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.467+0000 7fc727fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7281985a0 0x7fc728198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.467+0000 7fc727fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7281985a0 0x7fc728198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35504/0 (socket says 172.21.15.7:35504) 2024-04-21T08:47:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.467+0000 7fc727fff640 1 -- 172.21.15.7:0/1621501679 learned_addr learned my addr 172.21.15.7:0/1621501679 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc7277fe640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7281021a0 0x7fc72819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc726ffd640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc728107270 0x7fc72819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc727fff640 1 -- 172.21.15.7:0/1621501679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc728107270 msgr2=0x7fc72819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc727fff640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc728107270 0x7fc72819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc727fff640 1 -- 172.21.15.7:0/1621501679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7281021a0 msgr2=0x7fc72819e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc727fff640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7281021a0 0x7fc72819e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc727fff640 1 -- 172.21.15.7:0/1621501679 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7281990c0 con 0x7fc7281985a0 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc7277fe640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7281021a0 0x7fc72819e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc726ffd640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc728107270 0x7fc72819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc727fff640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7281985a0 0x7fc728198a50 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc720005e30 tx=0x7fc720032420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc724ff9640 1 -- 172.21.15.7:0/1621501679 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc720041020 con 0x7fc7281985a0 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc72de93640 1 -- 172.21.15.7:0/1621501679 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc728199350 con 0x7fc7281985a0 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc724ff9640 1 -- 172.21.15.7:0/1621501679 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc720032970 con 0x7fc7281985a0 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc724ff9640 1 -- 172.21.15.7:0/1621501679 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc720039040 con 0x7fc7281985a0 2024-04-21T08:47:40.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.468+0000 7fc72de93640 1 -- 172.21.15.7:0/1621501679 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7281a55a0 con 0x7fc7281985a0 2024-04-21T08:47:40.479 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.470+0000 7fc724ff9640 1 -- 172.21.15.7:0/1621501679 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fc720044020 con 0x7fc7281985a0 2024-04-21T08:47:40.479 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc730b66740 1 Processor -- start 2024-04-21T08:47:40.479 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc730b66740 1 -- start start 2024-04-21T08:47:40.479 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc730b66740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.479 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc730b66740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e307b60 0x55e93e307fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc730b66740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e3134d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc724ff9640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc7100a3d90 0x7fc7100a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc730b66740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e93e26c8d0 con 0x55e93e3088d0 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc730b66740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e93e209e80 con 0x55e93e307b60 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc730b66740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e93e268890 con 0x55e93e3110e0 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc726ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc726ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35514/0 (socket says 172.21.15.7:35514) 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc726ffd640 1 -- 172.21.15.7:0/2832775538 learned_addr learned my addr 172.21.15.7:0/2832775538 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc727fff640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc7100a3d90 0x7fc7100a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc724ff9640 1 -- 172.21.15.7:0/1621501679 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc7200f28a0 con 0x7fc7281985a0 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc7277fe640 1 --2- 172.21.15.7:0/2832775538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e307b60 0x55e93e307fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.471+0000 7fc726ffd640 1 --2- 172.21.15.7:0/2832775538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e3134d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.472+0000 7fc726ffd640 1 -- 172.21.15.7:0/2832775538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 msgr2=0x55e93e3134d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.472+0000 7fc726ffd640 1 --2- 172.21.15.7:0/2832775538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e3134d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.472+0000 7fc726ffd640 1 -- 172.21.15.7:0/2832775538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e307b60 msgr2=0x55e93e307fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.472+0000 7fc726ffd640 1 --2- 172.21.15.7:0/2832775538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e307b60 0x55e93e307fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.472+0000 7fc726ffd640 1 -- 172.21.15.7:0/2832775538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e93e22b320 con 0x55e93e3088d0 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.472+0000 7fc7277fe640 1 --2- 172.21.15.7:0/2832775538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e307b60 0x55e93e307fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.472+0000 7fc726ffd640 1 --2- 172.21.15.7:0/2832775538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307620 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00bfc0 tx=0x7fc71c00b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df23c143cc552cd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.472+0000 7fc727fff640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc7100a3d90 0x7fc7100a6250 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc7200efdc0 tx=0x7fc7200b39f0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.472+0000 7fc70dffb640 1 -- 172.21.15.7:0/2832775538 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c019030 con 0x55e93e3088d0 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.472+0000 7fc70dffb640 1 -- 172.21.15.7:0/2832775538 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71c0035f0 con 0x55e93e3088d0 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.472+0000 7fc70dffb640 1 -- 172.21.15.7:0/2832775538 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c011700 con 0x55e93e3088d0 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.473+0000 7fc730b66740 1 -- 172.21.15.7:0/2832775538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 msgr2=0x55e93e307620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.473+0000 7fc730b66740 1 --2- 172.21.15.7:0/2832775538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307620 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00bfc0 tx=0x7fc71c00b670 comp rx=0 tx=0).stop 2024-04-21T08:47:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.473+0000 7fc730b66740 1 -- 172.21.15.7:0/2832775538 shutdown_connections 2024-04-21T08:47:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.473+0000 7fc730b66740 1 --2- 172.21.15.7:0/2832775538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e3134d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.473+0000 7fc730b66740 1 --2- 172.21.15.7:0/2832775538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e307b60 0x55e93e307fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.473+0000 7fc730b66740 1 --2- 172.21.15.7:0/2832775538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307620 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.473+0000 7fc730b66740 1 -- 172.21.15.7:0/2832775538 >> 172.21.15.7:0/2832775538 conn(0x55e93e2cec90 msgr2=0x55e93e302850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.473+0000 7fc730b66740 1 -- 172.21.15.7:0/2832775538 shutdown_connections 2024-04-21T08:47:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.473+0000 7fc730b66740 1 -- 172.21.15.7:0/2832775538 wait complete. 2024-04-21T08:47:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.473+0000 7fc730b66740 1 -- start start 2024-04-21T08:47:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc730b66740 1 Processor -- start 2024-04-21T08:47:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc730b66740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e307b60 0x55e93e310a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc730b66740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e3088d0 0x55e93e30ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc730b66740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e30e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc730b66740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e93e313e10 con 0x55e93e307b60 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc730b66740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e93e1f4eb0 con 0x55e93e3088d0 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc730b66740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e93e313f90 con 0x55e93e3110e0 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc730b66740 10 client.0 _ll_register_callbacks cb 0x55e93e221b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc726ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e307b60 0x55e93e310a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc726ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e307b60 0x55e93e310a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35522/0 (socket says 172.21.15.7:35522) 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc726ffd640 1 -- 172.21.15.7:0/3833344322 learned_addr learned my addr 172.21.15.7:0/3833344322 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc7277fe640 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e3088d0 0x55e93e30ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.474+0000 7fc726ffd640 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e30e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc726ffd640 1 -- 172.21.15.7:0/3833344322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 msgr2=0x55e93e30e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc726ffd640 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e30e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc726ffd640 1 -- 172.21.15.7:0/3833344322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e3088d0 msgr2=0x55e93e30ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc726ffd640 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e3088d0 0x55e93e30ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc726ffd640 1 -- 172.21.15.7:0/3833344322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e93df7f210 con 0x55e93e307b60 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc726ffd640 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e307b60 0x55e93e310a40 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fc71c0057a0 tx=0x7fc71c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e303faf92897b1f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc706ffd640 1 -- 172.21.15.7:0/3833344322 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c022020 con 0x55e93e307b60 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc706ffd640 1 -- 172.21.15.7:0/3833344322 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc71c0114c0 con 0x55e93e307b60 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc730b66740 1 -- 172.21.15.7:0/3833344322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc71c0035f0 con 0x55e93e307b60 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc706ffd640 1 -- 172.21.15.7:0/3833344322 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc71c032050 con 0x55e93e307b60 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc730b66740 10 client.16338 fetch_fsmap learned FSMap version 23 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc730b66740 1 -- 172.21.15.7:0/3833344322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e93e04b620 con 0x55e93e307b60 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.475+0000 7fc706ffd640 1 -- 172.21.15.7:0/3833344322 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71c0035f0 con 0x55e93e307b60 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.476+0000 7fc706ffd640 1 -- 172.21.15.7:0/3833344322 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fc71c01e580 con 0x55e93e307b60 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.476+0000 7fc730b66740 10 client.16338 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:47:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.476+0000 7fc730b66740 10 client.16338 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.476+0000 7fc730b66740 20 client.16338 populate_metadata read hostname 'smithi007' 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.476+0000 7fc730b66740 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e93e321020 0x55e93e323410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.476+0000 7fc730b66740 4 client.16338 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.476+0000 7fc730b66740 1 -- 172.21.15.7:0/3833344322 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e93e280880 con 0x55e93e321020 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.476+0000 7fc7277fe640 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e93e321020 0x55e93e323410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.476+0000 7fc7277fe640 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e93e321020 0x55e93e323410 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f25f469550acb277 server_cookie=be01b30c4b1ff734 in_seq=0 out_seq=0 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.476+0000 7fc706ffd640 10 client.16338 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc706ffd640 1 -- 172.21.15.7:0/3833344322 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e93e280880 con 0x55e93e321020 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc706ffd640 10 client.16338 handle_command_reply: tid=0 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc730b66740 1 client.16338 shutdown 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc730b66740 1 -- 172.21.15.7:0/3833344322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e307b60 msgr2=0x55e93e310a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc730b66740 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e307b60 0x55e93e310a40 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fc71c0057a0 tx=0x7fc71c0056d0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc730b66740 1 -- 172.21.15.7:0/3833344322 shutdown_connections 2024-04-21T08:47:40.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc730b66740 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e93e321020 0x55e93e323410 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc730b66740 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e30e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc706ffd640 0 client.16338 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc730b66740 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e3088d0 0x55e93e30ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc730b66740 1 --2- 172.21.15.7:0/3833344322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e307b60 0x55e93e310a40 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc730b66740 1 -- 172.21.15.7:0/3833344322 >> 172.21.15.7:0/3833344322 conn(0x55e93e2cec90 msgr2=0x55e93e3045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc730b66740 1 -- 172.21.15.7:0/3833344322 shutdown_connections 2024-04-21T08:47:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.477+0000 7fc730b66740 1 -- 172.21.15.7:0/3833344322 wait complete. 2024-04-21T08:47:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.478+0000 7fc730b66740 20 client.16338 trim_cache size 0 max 16384 2024-04-21T08:47:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc730b66740 1 Processor -- start 2024-04-21T08:47:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc730b66740 1 -- start start 2024-04-21T08:47:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc730b66740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e307b60 0x55e93e31cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc730b66740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc730b66740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e3110e0 0x55e93e328ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc730b66740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc7281ad490 con 0x55e93e3088d0 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc730b66740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7281afd10 con 0x55e93e3110e0 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc730b66740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc7281adb00 con 0x55e93e307b60 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc7277fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc7277fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35538/0 (socket says 172.21.15.7:35538) 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc7277fe640 1 -- 172.21.15.7:0/3096773075 learned_addr learned my addr 172.21.15.7:0/3096773075 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc726ffd640 1 --2- 172.21.15.7:0/3096773075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e307b60 0x55e93e31cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc7277fe640 1 -- 172.21.15.7:0/3096773075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e307b60 msgr2=0x55e93e31cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc7277fe640 1 --2- 172.21.15.7:0/3096773075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e307b60 0x55e93e31cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc7277fe640 1 -- 172.21.15.7:0/3096773075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e3110e0 msgr2=0x55e93e328ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc726ffd640 1 --2- 172.21.15.7:0/3096773075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e3110e0 0x55e93e328ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc7277fe640 1 --2- 172.21.15.7:0/3096773075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e3110e0 0x55e93e328ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc7277fe640 1 -- 172.21.15.7:0/3096773075 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e93e306e30 con 0x55e93e3088d0 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.493+0000 7fc7277fe640 1 --2- 172.21.15.7:0/3096773075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307080 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fc718005d00 tx=0x7fc7180138f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d42f9421aeb2f85 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc70d7fa640 1 -- 172.21.15.7:0/3096773075 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc718005220 con 0x55e93e3088d0 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc70d7fa640 1 -- 172.21.15.7:0/3096773075 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc718004c50 con 0x55e93e3088d0 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc70d7fa640 1 -- 172.21.15.7:0/3096773075 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71801e590 con 0x55e93e3088d0 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc730b66740 1 -- 172.21.15.7:0/3096773075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 msgr2=0x55e93e307080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc730b66740 1 --2- 172.21.15.7:0/3096773075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307080 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fc718005d00 tx=0x7fc7180138f0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc730b66740 1 -- 172.21.15.7:0/3096773075 shutdown_connections 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc730b66740 1 --2- 172.21.15.7:0/3096773075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e3110e0 0x55e93e328ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc730b66740 1 --2- 172.21.15.7:0/3096773075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e307080 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc730b66740 1 --2- 172.21.15.7:0/3096773075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e307b60 0x55e93e31cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc730b66740 1 -- 172.21.15.7:0/3096773075 >> 172.21.15.7:0/3096773075 conn(0x55e93e2cec90 msgr2=0x55e93e30c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc730b66740 1 -- 172.21.15.7:0/3096773075 shutdown_connections 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.494+0000 7fc730b66740 1 -- 172.21.15.7:0/3096773075 wait complete. 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc730b66740 1 -- start start 2024-04-21T08:47:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc730b66740 1 Processor -- start 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc730b66740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e307b60 0x55e93e314cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc730b66740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e315230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc730b66740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e315770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc730b66740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc7281ae1d0 con 0x55e93e3088d0 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc730b66740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7281af640 con 0x55e93e307b60 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc730b66740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc7281ae8a0 con 0x55e93e3110e0 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc730b66740 10 client.0 _ll_register_callbacks cb 0x7fc71c013400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc7277fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e315230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc7277fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e315230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35548/0 (socket says 172.21.15.7:35548) 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc7277fe640 1 -- 172.21.15.7:0/70899036 learned_addr learned my addr 172.21.15.7:0/70899036 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc726ffd640 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e315770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.495+0000 7fc726ffd640 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e307b60 0x55e93e314cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc7277fe640 1 -- 172.21.15.7:0/70899036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 msgr2=0x55e93e315770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc7277fe640 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e315770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc7277fe640 1 -- 172.21.15.7:0/70899036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e307b60 msgr2=0x55e93e314cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc7277fe640 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e307b60 0x55e93e314cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc7277fe640 1 -- 172.21.15.7:0/70899036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e93e304b50 con 0x55e93e3088d0 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc7277fe640 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e315230 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fc718009d30 tx=0x7fc7180048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74808c3d7519dd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc706ffd640 1 -- 172.21.15.7:0/70899036 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71802b070 con 0x55e93e3088d0 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc706ffd640 1 -- 172.21.15.7:0/70899036 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc718005000 con 0x55e93e3088d0 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc730b66740 1 -- 172.21.15.7:0/70899036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc718004c50 con 0x55e93e3088d0 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc706ffd640 1 -- 172.21.15.7:0/70899036 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc718037050 con 0x55e93e3088d0 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc730b66740 10 client.16344 fetch_fsmap learned FSMap version 23 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc730b66740 1 -- 172.21.15.7:0/70899036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e93e3875b0 con 0x55e93e3088d0 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.496+0000 7fc706ffd640 1 -- 172.21.15.7:0/70899036 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc718004c50 con 0x55e93e3088d0 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.497+0000 7fc706ffd640 1 -- 172.21.15.7:0/70899036 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fc718011500 con 0x55e93e3088d0 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.497+0000 7fc730b66740 10 client.16344 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:47:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.497+0000 7fc730b66740 10 client.16344 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.497+0000 7fc730b66740 20 client.16344 populate_metadata read hostname 'smithi007' 2024-04-21T08:47:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.497+0000 7fc730b66740 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e93e387ca0 0x55e93e38a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.497+0000 7fc730b66740 4 client.16344 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:47:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.497+0000 7fc730b66740 1 -- 172.21.15.7:0/70899036 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e93e307850 con 0x55e93e387ca0 2024-04-21T08:47:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.497+0000 7fc726ffd640 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e93e387ca0 0x55e93e38a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.497+0000 7fc726ffd640 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e93e387ca0 0x55e93e38a0c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97beb87e758c7743 server_cookie=741d4c490e832773 in_seq=0 out_seq=0 2024-04-21T08:47:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:40.497+0000 7fc706ffd640 10 client.16344 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:40.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:40 smithi016 ceph-mon[31259]: pgmap v93: 129 pgs: 129 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 165 op/s 2024-04-21T08:47:40.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:40 smithi077 ceph-mon[31615]: pgmap v93: 129 pgs: 129 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 165 op/s 2024-04-21T08:47:40.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:40 smithi007 ceph-mon[28477]: pgmap v93: 129 pgs: 129 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 165 op/s 2024-04-21T08:47:42.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:42 smithi016 ceph-mon[31259]: pgmap v94: 129 pgs: 129 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 165 op/s 2024-04-21T08:47:42.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:42 smithi077 ceph-mon[31615]: pgmap v94: 129 pgs: 129 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 165 op/s 2024-04-21T08:47:42.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:42 smithi007 ceph-mon[28477]: pgmap v94: 129 pgs: 129 active+clean; 267 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 165 op/s 2024-04-21T08:47:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.604+0000 7fc706ffd640 1 -- 172.21.15.7:0/70899036 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e93e307850 con 0x55e93e387ca0 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.604+0000 7fc706ffd640 10 client.16344 handle_command_reply: tid=0 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.604+0000 7fc730b66740 1 client.16344 shutdown 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.604+0000 7fc730b66740 1 -- 172.21.15.7:0/70899036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 msgr2=0x55e93e315230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.604+0000 7fc730b66740 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e315230 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fc718009d30 tx=0x7fc7180048b0 comp rx=0 tx=0).stop 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.604+0000 7fc730b66740 1 -- 172.21.15.7:0/70899036 shutdown_connections 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.604+0000 7fc730b66740 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e93e387ca0 0x55e93e38a0c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.605+0000 7fc730b66740 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e93e3110e0 0x55e93e315770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.605+0000 7fc706ffd640 0 client.16344 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.605+0000 7fc730b66740 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93e3088d0 0x55e93e315230 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.605+0000 7fc730b66740 1 --2- 172.21.15.7:0/70899036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e93e307b60 0x55e93e314cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.605+0000 7fc730b66740 1 -- 172.21.15.7:0/70899036 >> 172.21.15.7:0/70899036 conn(0x55e93e2cec90 msgr2=0x55e93e3018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.605+0000 7fc730b66740 1 -- 172.21.15.7:0/70899036 shutdown_connections 2024-04-21T08:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.605+0000 7fc730b66740 1 -- 172.21.15.7:0/70899036 wait complete. 2024-04-21T08:47:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.605+0000 7fc730b66740 20 client.16344 trim_cache size 0 max 16384 2024-04-21T08:47:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:47:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:47:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:47:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.606+0000 7fc72de93640 1 -- 172.21.15.7:0/1621501679 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc7100a3d90 msgr2=0x7fc7100a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.606+0000 7fc72de93640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc7100a3d90 0x7fc7100a6250 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc7200efdc0 tx=0x7fc7200b39f0 comp rx=0 tx=0).stop 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.606+0000 7fc72de93640 1 -- 172.21.15.7:0/1621501679 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7281985a0 msgr2=0x7fc728198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.606+0000 7fc72de93640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7281985a0 0x7fc728198a50 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc720005e30 tx=0x7fc720032420 comp rx=0 tx=0).stop 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.606+0000 7fc72de93640 1 -- 172.21.15.7:0/1621501679 shutdown_connections 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.606+0000 7fc72de93640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7281985a0 0x7fc728198a50 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.606+0000 7fc72de93640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc7100a3d90 0x7fc7100a6250 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.606+0000 7fc72de93640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc728107270 0x7fc72819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.606+0000 7fc72de93640 1 --2- 172.21.15.7:0/1621501679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7281021a0 0x7fc72819e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.606+0000 7fc72de93640 1 -- 172.21.15.7:0/1621501679 >> 172.21.15.7:0/1621501679 conn(0x7fc7280fda40 msgr2=0x7fc7280feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.607+0000 7fc72de93640 1 -- 172.21.15.7:0/1621501679 shutdown_connections 2024-04-21T08:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:43.607+0000 7fc72de93640 1 -- 172.21.15.7:0/1621501679 wait complete. 2024-04-21T08:47:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:44 smithi016 ceph-mon[31259]: pgmap v95: 129 pgs: 129 active+clean; 269 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 1.26k op/s 2024-04-21T08:47:44.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:44 smithi077 ceph-mon[31615]: pgmap v95: 129 pgs: 129 active+clean; 269 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 1.26k op/s 2024-04-21T08:47:44.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:44 smithi007 ceph-mon[28477]: pgmap v95: 129 pgs: 129 active+clean; 269 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 1.26k op/s 2024-04-21T08:47:46.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:46 smithi016 ceph-mon[31259]: pgmap v96: 129 pgs: 129 active+clean; 269 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 1.1 MiB/s wr, 1.11k op/s 2024-04-21T08:47:46.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:46 smithi077 ceph-mon[31615]: pgmap v96: 129 pgs: 129 active+clean; 269 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 1.1 MiB/s wr, 1.11k op/s 2024-04-21T08:47:46.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:46 smithi007 ceph-mon[28477]: pgmap v96: 129 pgs: 129 active+clean; 269 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 1.1 MiB/s wr, 1.11k op/s 2024-04-21T08:47:48.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:48 smithi016 ceph-mon[31259]: pgmap v97: 129 pgs: 129 active+clean; 255 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 1.6 MiB/s wr, 1.26k op/s 2024-04-21T08:47:48.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:48 smithi077 ceph-mon[31615]: pgmap v97: 129 pgs: 129 active+clean; 255 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 1.6 MiB/s wr, 1.26k op/s 2024-04-21T08:47:48.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:48 smithi007 ceph-mon[28477]: pgmap v97: 129 pgs: 129 active+clean; 255 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 1.6 MiB/s wr, 1.26k op/s 2024-04-21T08:47:49.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:49 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:49.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:49 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:49.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:49 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:49.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:49 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:47:49.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:49 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:49.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:49 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:49.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:49 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:49.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:49 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:47:49.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:49 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:49.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:49 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:49.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:49 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:49.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:49 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:47:50.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:50 smithi016 ceph-mon[31259]: pgmap v98: 129 pgs: 129 active+clean; 277 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 3.9 MiB/s wr, 2.09k op/s 2024-04-21T08:47:50.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:50 smithi077 ceph-mon[31615]: pgmap v98: 129 pgs: 129 active+clean; 277 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 3.9 MiB/s wr, 2.09k op/s 2024-04-21T08:47:50.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:50 smithi007 ceph-mon[28477]: pgmap v98: 129 pgs: 129 active+clean; 277 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 3.9 MiB/s wr, 2.09k op/s 2024-04-21T08:47:52.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:52 smithi016 ceph-mon[31259]: pgmap v99: 129 pgs: 129 active+clean; 277 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 3.9 MiB/s wr, 2.09k op/s 2024-04-21T08:47:52.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:52 smithi077 ceph-mon[31615]: pgmap v99: 129 pgs: 129 active+clean; 277 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 3.9 MiB/s wr, 2.09k op/s 2024-04-21T08:47:52.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:52 smithi007 ceph-mon[28477]: pgmap v99: 129 pgs: 129 active+clean; 277 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 3.9 MiB/s wr, 2.09k op/s 2024-04-21T08:47:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:54 smithi016 ceph-mon[31259]: pgmap v100: 129 pgs: 129 active+clean; 285 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 2.11k op/s 2024-04-21T08:47:54.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:54 smithi077 ceph-mon[31615]: pgmap v100: 129 pgs: 129 active+clean; 285 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 2.11k op/s 2024-04-21T08:47:54.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:54 smithi007 ceph-mon[28477]: pgmap v100: 129 pgs: 129 active+clean; 285 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 2.11k op/s 2024-04-21T08:47:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.696+0000 7f0e5ec50640 1 -- 172.21.15.7:0/512985511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e58100d40 msgr2=0x7f0e58101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.696+0000 7f0e5ec50640 1 --2- 172.21.15.7:0/512985511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e58100d40 0x7f0e58101120 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009ca0 tx=0x7f0e5002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.696+0000 7f0e5ec50640 1 -- 172.21.15.7:0/512985511 shutdown_connections 2024-04-21T08:47:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.696+0000 7f0e5ec50640 1 --2- 172.21.15.7:0/512985511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e5810e8f0 0x7f0e58110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.696+0000 7f0e5ec50640 1 --2- 172.21.15.7:0/512985511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58101660 0x7f0e5810e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.696+0000 7f0e5ec50640 1 --2- 172.21.15.7:0/512985511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e58100d40 0x7f0e58101120 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.696+0000 7f0e5ec50640 1 -- 172.21.15.7:0/512985511 >> 172.21.15.7:0/512985511 conn(0x7f0e58077f70 msgr2=0x7f0e58078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.696+0000 7f0e5ec50640 1 -- 172.21.15.7:0/512985511 shutdown_connections 2024-04-21T08:47:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.696+0000 7f0e5ec50640 1 -- 172.21.15.7:0/512985511 wait complete. 2024-04-21T08:47:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.696+0000 7f0e5ec50640 1 Processor -- start 2024-04-21T08:47:56.698 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5ec50640 1 -- start start 2024-04-21T08:47:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5ec50640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58100d40 0x7f0e581a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5ec50640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e58101660 0x7f0e581a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5ec50640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e5810e8f0 0x7f0e5819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5ec50640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581143f0 con 0x7f0e5810e8f0 2024-04-21T08:47:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5ec50640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58114270 con 0x7f0e58100d40 2024-04-21T08:47:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5ec50640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58114570 con 0x7f0e58101660 2024-04-21T08:47:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5c9ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58100d40 0x7f0e581a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5d1ed640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e5810e8f0 0x7f0e5819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5c9ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58100d40 0x7f0e581a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39552/0 (socket says 172.21.15.7:39552) 2024-04-21T08:47:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5c9ec640 1 -- 172.21.15.7:0/1738018899 learned_addr learned my addr 172.21.15.7:0/1738018899 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e57fff640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e58101660 0x7f0e581a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5d1ed640 1 -- 172.21.15.7:0/1738018899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e58101660 msgr2=0x7f0e581a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5d1ed640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e58101660 0x7f0e581a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5d1ed640 1 -- 172.21.15.7:0/1738018899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58100d40 msgr2=0x7f0e581a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5d1ed640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58100d40 0x7f0e581a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5d1ed640 1 -- 172.21.15.7:0/1738018899 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5819afe0 con 0x7f0e5810e8f0 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e57fff640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e58101660 0x7f0e581a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5c9ec640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58100d40 0x7f0e581a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.697+0000 7f0e5d1ed640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e5810e8f0 0x7f0e5819a810 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0e58102f20 tx=0x7f0e4000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.698+0000 7f0e55ffb640 1 -- 172.21.15.7:0/1738018899 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e40017070 con 0x7f0e5810e8f0 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.698+0000 7f0e55ffb640 1 -- 172.21.15.7:0/1738018899 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e400047a0 con 0x7f0e5810e8f0 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.698+0000 7f0e5ec50640 1 -- 172.21.15.7:0/1738018899 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e5819b2d0 con 0x7f0e5810e8f0 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.698+0000 7f0e55ffb640 1 -- 172.21.15.7:0/1738018899 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e40009c00 con 0x7f0e5810e8f0 2024-04-21T08:47:56.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.698+0000 7f0e5ec50640 1 -- 172.21.15.7:0/1738018899 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e581a74f0 con 0x7f0e5810e8f0 2024-04-21T08:47:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.700+0000 7f0e55ffb640 1 -- 172.21.15.7:0/1738018899 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0e40004950 con 0x7f0e5810e8f0 2024-04-21T08:47:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.700+0000 7f0e6183d740 1 Processor -- start 2024-04-21T08:47:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.700+0000 7f0e6183d740 1 -- start start 2024-04-21T08:47:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.700+0000 7f0e55ffb640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0e440a3d10 0x7f0e440a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 0x55b5466a5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e57fff640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0e440a3d10 0x7f0e440a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e6183d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b54660a8d0 con 0x55b5466a68d0 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e6183d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b5465a7e80 con 0x55b5466a5b60 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e6183d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b546606890 con 0x55b5466af0e0 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e55ffb640 1 -- 172.21.15.7:0/1738018899 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f0e400d03f0 con 0x7f0e5810e8f0 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e5c9ec640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 0x55b5466a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466a5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39562/0 (socket says 172.21.15.7:39562) 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e57fff640 1 -- 172.21.15.7:0/538058380 learned_addr learned my addr 172.21.15.7:0/538058380 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e5d1ed640 1 --2- 172.21.15.7:0/538058380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e5c9ec640 1 -- 172.21.15.7:0/538058380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 msgr2=0x55b5466b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e5c9ec640 1 --2- 172.21.15.7:0/538058380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e5c9ec640 1 -- 172.21.15.7:0/538058380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 msgr2=0x55b5466a5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e5c9ec640 1 --2- 172.21.15.7:0/538058380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466a5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e5c9ec640 1 -- 172.21.15.7:0/538058380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5465c9320 con 0x55b5466a68d0 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e57fff640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0e440a3d10 0x7f0e440a61d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0e480096a0 tx=0x7f0e48009bb0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e5c9ec640 1 --2- 172.21.15.7:0/538058380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 0x55b5466a5620 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0e50005e30 tx=0x7f0e50005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf6262a74b07fcc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e4effd640 1 -- 172.21.15.7:0/538058380 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50043070 con 0x55b5466a68d0 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.701+0000 7f0e4effd640 1 -- 172.21.15.7:0/538058380 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e50032970 con 0x55b5466a68d0 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.702+0000 7f0e4effd640 1 -- 172.21.15.7:0/538058380 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50039040 con 0x55b5466a68d0 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.702+0000 7f0e6183d740 1 -- 172.21.15.7:0/538058380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 msgr2=0x55b5466a5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.702+0000 7f0e6183d740 1 --2- 172.21.15.7:0/538058380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 0x55b5466a5620 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0e50005e30 tx=0x7f0e50005f70 comp rx=0 tx=0).stop 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.702+0000 7f0e6183d740 1 -- 172.21.15.7:0/538058380 shutdown_connections 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.702+0000 7f0e6183d740 1 --2- 172.21.15.7:0/538058380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.702+0000 7f0e6183d740 1 --2- 172.21.15.7:0/538058380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466a5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.702+0000 7f0e6183d740 1 --2- 172.21.15.7:0/538058380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 0x55b5466a5620 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.702+0000 7f0e6183d740 1 -- 172.21.15.7:0/538058380 >> 172.21.15.7:0/538058380 conn(0x55b54666cc90 msgr2=0x55b5466a0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.702+0000 7f0e6183d740 1 -- 172.21.15.7:0/538058380 shutdown_connections 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.702+0000 7f0e6183d740 1 -- 172.21.15.7:0/538058380 wait complete. 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e6183d740 1 -- start start 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e6183d740 1 Processor -- start 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466aea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466a68d0 0x55b5466abe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466af0e0 0x55b5466ac3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e6183d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b5466b1db0 con 0x55b5466af0e0 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e6183d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b546592eb0 con 0x55b5466a5b60 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e6183d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b5466b1f30 con 0x55b5466a68d0 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e6183d740 10 client.0 _ll_register_callbacks cb 0x55b5465bfb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e5d1ed640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466af0e0 0x55b5466ac3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e5d1ed640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466af0e0 0x55b5466ac3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44432/0 (socket says 172.21.15.7:44432) 2024-04-21T08:47:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e5c9ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e5d1ed640 1 -- 172.21.15.7:0/473146142 learned_addr learned my addr 172.21.15.7:0/473146142 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e57fff640 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466a68d0 0x55b5466abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e5d1ed640 1 -- 172.21.15.7:0/473146142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466a68d0 msgr2=0x55b5466abe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e5d1ed640 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466a68d0 0x55b5466abe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e5d1ed640 1 -- 172.21.15.7:0/473146142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 msgr2=0x55b5466aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e5d1ed640 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466aea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.703+0000 7f0e5d1ed640 1 -- 172.21.15.7:0/473146142 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b54631d210 con 0x55b5466af0e0 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.704+0000 7f0e5c9ec640 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.704+0000 7f0e5d1ed640 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466af0e0 0x55b5466ac3b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0e4009bf30 tx=0x7f0e4004a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470b6366dd297e99 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.704+0000 7f0e37fff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.704+0000 7f0e37fff640 1 -- 172.21.15.7:0/473146142 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e40020040 con 0x55b5466af0e0 2024-04-21T08:47:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.704+0000 7f0e37fff640 1 -- 172.21.15.7:0/473146142 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f0e40004c70 con 0x55b5466af0e0 2024-04-21T08:47:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.704+0000 7f0e6183d740 1 -- 172.21.15.7:0/473146142 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e50032970 con 0x55b5466af0e0 2024-04-21T08:47:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.704+0000 7f0e37fff640 1 -- 172.21.15.7:0/473146142 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0e400a0350 con 0x55b5466af0e0 2024-04-21T08:47:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.704+0000 7f0e6183d740 10 client.16356 fetch_fsmap learned FSMap version 23 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.704+0000 7f0e6183d740 1 -- 172.21.15.7:0/473146142 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b5463e9620 con 0x55b5466af0e0 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.704+0000 7f0e37fff640 1 -- 172.21.15.7:0/473146142 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e50032970 con 0x55b5466af0e0 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.705+0000 7f0e37fff640 1 -- 172.21.15.7:0/473146142 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f0e4009b3f0 con 0x55b5466af0e0 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.705+0000 7f0e6183d740 10 client.16356 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.705+0000 7f0e6183d740 10 client.16356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.705+0000 7f0e6183d740 20 client.16356 populate_metadata read hostname 'smithi007' 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.705+0000 7f0e6183d740 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b5466bf140 0x55b5466c1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.705+0000 7f0e6183d740 4 client.16356 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.705+0000 7f0e6183d740 1 -- 172.21.15.7:0/473146142 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b54661e880 con 0x55b5466bf140 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.705+0000 7f0e5c9ec640 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b5466bf140 0x55b5466c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.705+0000 7f0e5c9ec640 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b5466bf140 0x55b5466c1570 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25b9d347fed594cf server_cookie=dcd7de7a4d42b3a in_seq=0 out_seq=0 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.705+0000 7f0e37fff640 10 client.16356 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e37fff640 1 -- 172.21.15.7:0/473146142 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b54661e880 con 0x55b5466bf140 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e37fff640 10 client.16356 handle_command_reply: tid=0 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 1 client.16356 shutdown 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 1 -- 172.21.15.7:0/473146142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466af0e0 msgr2=0x55b5466ac3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466af0e0 0x55b5466ac3b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0e4009bf30 tx=0x7f0e4004a480 comp rx=0 tx=0).stop 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 1 -- 172.21.15.7:0/473146142 shutdown_connections 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b5466bf140 0x55b5466c1570 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466af0e0 0x55b5466ac3b0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466a68d0 0x55b5466abe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e37fff640 0 client.16356 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 1 --2- 172.21.15.7:0/473146142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 1 -- 172.21.15.7:0/473146142 >> 172.21.15.7:0/473146142 conn(0x55b54666cc90 msgr2=0x55b5466a25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 1 -- 172.21.15.7:0/473146142 shutdown_connections 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 1 -- 172.21.15.7:0/473146142 wait complete. 2024-04-21T08:47:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.706+0000 7f0e6183d740 20 client.16356 trim_cache size 0 max 16384 2024-04-21T08:47:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e6183d740 1 Processor -- start 2024-04-21T08:47:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e6183d740 1 -- start start 2024-04-21T08:47:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466bacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 0x55b5466ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466aea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e6183d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581af740 con 0x55b5466a68d0 2024-04-21T08:47:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e6183d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581b1fc0 con 0x55b5466a5b60 2024-04-21T08:47:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e6183d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581afdb0 con 0x55b5466af0e0 2024-04-21T08:47:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e57fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 0x55b5466ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e5c9ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466bacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e57fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 0x55b5466ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44448/0 (socket says 172.21.15.7:44448) 2024-04-21T08:47:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.722+0000 7f0e57fff640 1 -- 172.21.15.7:0/629685878 learned_addr learned my addr 172.21.15.7:0/629685878 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.723+0000 7f0e5d1ed640 1 --2- 172.21.15.7:0/629685878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.723+0000 7f0e57fff640 1 -- 172.21.15.7:0/629685878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 msgr2=0x55b5466aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.723+0000 7f0e57fff640 1 --2- 172.21.15.7:0/629685878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466aea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.723+0000 7f0e57fff640 1 -- 172.21.15.7:0/629685878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 msgr2=0x55b5466bacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.723+0000 7f0e57fff640 1 --2- 172.21.15.7:0/629685878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466bacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.723+0000 7f0e57fff640 1 -- 172.21.15.7:0/629685878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5466a4e30 con 0x55b5466a68d0 2024-04-21T08:47:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.723+0000 7f0e5c9ec640 1 --2- 172.21.15.7:0/629685878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466bacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.723+0000 7f0e57fff640 1 --2- 172.21.15.7:0/629685878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 0x55b5466ae540 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0e480096f0 tx=0x7f0e48005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b730af50e0f8d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.723+0000 7f0e4e7fc640 1 -- 172.21.15.7:0/629685878 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e48009720 con 0x55b5466a68d0 2024-04-21T08:47:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.723+0000 7f0e4e7fc640 1 -- 172.21.15.7:0/629685878 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e480218c0 con 0x55b5466a68d0 2024-04-21T08:47:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.723+0000 7f0e4e7fc640 1 -- 172.21.15.7:0/629685878 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e48012750 con 0x55b5466a68d0 2024-04-21T08:47:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.724+0000 7f0e6183d740 1 -- 172.21.15.7:0/629685878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 msgr2=0x55b5466ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.724+0000 7f0e6183d740 1 --2- 172.21.15.7:0/629685878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 0x55b5466ae540 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0e480096f0 tx=0x7f0e48005f90 comp rx=0 tx=0).stop 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.724+0000 7f0e6183d740 1 -- 172.21.15.7:0/629685878 shutdown_connections 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.724+0000 7f0e6183d740 1 --2- 172.21.15.7:0/629685878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.724+0000 7f0e6183d740 1 --2- 172.21.15.7:0/629685878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a68d0 0x55b5466ae540 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.724+0000 7f0e6183d740 1 --2- 172.21.15.7:0/629685878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a5b60 0x55b5466bacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.724+0000 7f0e6183d740 1 -- 172.21.15.7:0/629685878 >> 172.21.15.7:0/629685878 conn(0x55b54666cc90 msgr2=0x55b5466a7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.724+0000 7f0e6183d740 1 -- 172.21.15.7:0/629685878 shutdown_connections 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.724+0000 7f0e6183d740 1 -- 172.21.15.7:0/629685878 wait complete. 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.724+0000 7f0e6183d740 1 -- start start 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.724+0000 7f0e6183d740 1 Processor -- start 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a5b60 0x55b5466b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a68d0 0x55b5466a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e6183d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466a8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e6183d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581b0480 con 0x55b5466a5b60 2024-04-21T08:47:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e6183d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581b18f0 con 0x55b5466a68d0 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e6183d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581b0b50 con 0x55b5466af0e0 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e6183d740 10 client.0 _ll_register_callbacks cb 0x7f0e4009be00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e5c9ec640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a5b60 0x55b5466b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e5c9ec640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a5b60 0x55b5466b73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44452/0 (socket says 172.21.15.7:44452) 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e5c9ec640 1 -- 172.21.15.7:0/2516419555 learned_addr learned my addr 172.21.15.7:0/2516419555 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e57fff640 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a68d0 0x55b5466a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e5d1ed640 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466a8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e5c9ec640 1 -- 172.21.15.7:0/2516419555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 msgr2=0x55b5466a8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e5c9ec640 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466a8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e5c9ec640 1 -- 172.21.15.7:0/2516419555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a68d0 msgr2=0x55b5466a86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e5c9ec640 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a68d0 0x55b5466a86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e5c9ec640 1 -- 172.21.15.7:0/2516419555 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5466a91c0 con 0x55b5466a5b60 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e5c9ec640 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a5b60 0x55b5466b73b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0e50003dd0 tx=0x7f0e50005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5839c22d10ae6e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e37fff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:47:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e37fff640 1 -- 172.21.15.7:0/2516419555 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50057060 con 0x55b5466a5b60 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e37fff640 1 -- 172.21.15.7:0/2516419555 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f0e500326c0 con 0x55b5466a5b60 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.725+0000 7f0e6183d740 1 -- 172.21.15.7:0/2516419555 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e480218c0 con 0x55b5466a5b60 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e37fff640 1 -- 172.21.15.7:0/2516419555 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0e5005e050 con 0x55b5466a5b60 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e6183d740 10 client.16362 fetch_fsmap learned FSMap version 23 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e6183d740 1 -- 172.21.15.7:0/2516419555 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b54672dd60 con 0x55b5466a5b60 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e37fff640 1 -- 172.21.15.7:0/2516419555 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e480218c0 con 0x55b5466a5b60 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e37fff640 1 -- 172.21.15.7:0/2516419555 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f0e50003f90 con 0x55b5466a5b60 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e6183d740 10 client.16362 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e6183d740 10 client.16362 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e6183d740 20 client.16362 populate_metadata read hostname 'smithi007' 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e6183d740 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b54672e0f0 0x55b5467304e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:56.732 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e6183d740 4 client.16362 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:47:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e6183d740 1 -- 172.21.15.7:0/2516419555 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b5465aa3d0 con 0x55b54672e0f0 2024-04-21T08:47:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.726+0000 7f0e57fff640 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b54672e0f0 0x55b5467304e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.727+0000 7f0e57fff640 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b54672e0f0 0x55b5467304e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5a9e25ed1f481ef server_cookie=4f0d8680c5be3bfa in_seq=0 out_seq=0 2024-04-21T08:47:56.733 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:47:56.727+0000 7f0e37fff640 10 client.16362 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:56.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:56 smithi016 ceph-mon[31259]: pgmap v101: 129 pgs: 129 active+clean; 285 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 1.02k op/s 2024-04-21T08:47:56.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:56 smithi077 ceph-mon[31615]: pgmap v101: 129 pgs: 129 active+clean; 285 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 1.02k op/s 2024-04-21T08:47:56.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:56 smithi007 ceph-mon[28477]: pgmap v101: 129 pgs: 129 active+clean; 285 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 1.02k op/s 2024-04-21T08:47:57.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:47:57.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:47:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.518+0000 7fdf0cb46640 1 -- 172.21.15.7:0/1001050009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf08102b40 msgr2=0x7fdf08106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.518+0000 7fdf0cb46640 1 --2- 172.21.15.7:0/1001050009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf08102b40 0x7fdf08106b80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009860 tx=0x7fdefc02bb70 comp rx=0 tx=0).stop 2024-04-21T08:47:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.518+0000 7fdf0cb46640 1 -- 172.21.15.7:0/1001050009 shutdown_connections 2024-04-21T08:47:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.518+0000 7fdf0cb46640 1 --2- 172.21.15.7:0/1001050009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf08107260 0x7fdf0810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.518+0000 7fdf0cb46640 1 --2- 172.21.15.7:0/1001050009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf08102b40 0x7fdf08106b80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.518+0000 7fdf0cb46640 1 --2- 172.21.15.7:0/1001050009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08102190 0x7fdf08102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.518+0000 7fdf0cb46640 1 -- 172.21.15.7:0/1001050009 >> 172.21.15.7:0/1001050009 conn(0x7fdf080fda30 msgr2=0x7fdf080ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:57.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.519+0000 7fdf0cb46640 1 -- 172.21.15.7:0/1001050009 shutdown_connections 2024-04-21T08:47:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.519+0000 7fdf0cb46640 1 -- 172.21.15.7:0/1001050009 wait complete. 2024-04-21T08:47:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.520+0000 7fdf0cb46640 1 Processor -- start 2024-04-21T08:47:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.520+0000 7fdf0cb46640 1 -- start start 2024-04-21T08:47:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.520+0000 7fdf0cb46640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf08102190 0x7fdf08074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.520+0000 7fdf0cb46640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf08107260 0x7fdf08074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.520+0000 7fdf0cb46640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08074cd0 0x7fdf081a6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.520+0000 7fdf0cb46640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdf08112130 con 0x7fdf08074cd0 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.520+0000 7fdf0cb46640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf08111fb0 con 0x7fdf08102190 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.520+0000 7fdf0cb46640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081122b0 con 0x7fdf08107260 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdf0659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf08102190 0x7fdf08074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdf06d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08074cd0 0x7fdf081a6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdf0659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf08102190 0x7fdf08074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39604/0 (socket says 172.21.15.7:39604) 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdf0659c640 1 -- 172.21.15.7:0/107704855 learned_addr learned my addr 172.21.15.7:0/107704855 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdf05d9b640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf08107260 0x7fdf08074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdf05d9b640 1 -- 172.21.15.7:0/107704855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf08102190 msgr2=0x7fdf08074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdf05d9b640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf08102190 0x7fdf08074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdf05d9b640 1 -- 172.21.15.7:0/107704855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08074cd0 msgr2=0x7fdf081a6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdf05d9b640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08074cd0 0x7fdf081a6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdf05d9b640 1 -- 172.21.15.7:0/107704855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf081a7020 con 0x7fdf08107260 2024-04-21T08:47:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdf05d9b640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf08107260 0x7fdf08074790 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdefc02c0b0 tx=0x7fdefc0359a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.521+0000 7fdef77fe640 1 -- 172.21.15.7:0/107704855 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc041020 con 0x7fdf08107260 2024-04-21T08:47:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.522+0000 7fdf0cb46640 1 -- 172.21.15.7:0/107704855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf081a72b0 con 0x7fdf08107260 2024-04-21T08:47:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.522+0000 7fdef77fe640 1 -- 172.21.15.7:0/107704855 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdefc02c9c0 con 0x7fdf08107260 2024-04-21T08:47:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.522+0000 7fdef77fe640 1 -- 172.21.15.7:0/107704855 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc039040 con 0x7fdf08107260 2024-04-21T08:47:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.522+0000 7fdf0cb46640 1 -- 172.21.15.7:0/107704855 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf081a7770 con 0x7fdf08107260 2024-04-21T08:47:57.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.524+0000 7fdef77fe640 1 -- 172.21.15.7:0/107704855 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fdefc044020 con 0x7fdf08107260 2024-04-21T08:47:57.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.525+0000 7fdf0f80b740 1 Processor -- start 2024-04-21T08:47:57.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.525+0000 7fdf0f80b740 1 -- start start 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.525+0000 7fdef77fe640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdef00a3d50 0x7fdef00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.525+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d6090e580 0x559d6090e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.525+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090eea0 0x559d60917540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.525+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d60917a80 0x559d60919e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf06d9d640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdef00a3d50 0x7fdef00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x559d608731e0 con 0x559d6090e580 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559d60810390 con 0x559d6090eea0 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559d6086f1a0 con 0x559d60917a80 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdef77fe640 1 -- 172.21.15.7:0/107704855 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fdefc0f2110 con 0x7fdf08107260 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d6090e580 0x559d6090e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d6090e580 0x559d6090e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44474/0 (socket says 172.21.15.7:44474) 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0659c640 1 -- 172.21.15.7:0/2851779674 learned_addr learned my addr 172.21.15.7:0/2851779674 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:57.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf05d9b640 1 --2- 172.21.15.7:0/2851779674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090eea0 0x559d60917540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf06d9d640 1 --2- 172.21.15.7:0/2851779674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d60917a80 0x559d60919e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0659c640 1 -- 172.21.15.7:0/2851779674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d60917a80 msgr2=0x559d60919e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0659c640 1 --2- 172.21.15.7:0/2851779674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d60917a80 0x559d60919e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0659c640 1 -- 172.21.15.7:0/2851779674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090eea0 msgr2=0x559d60917540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0659c640 1 --2- 172.21.15.7:0/2851779674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090eea0 0x559d60917540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0659c640 1 -- 172.21.15.7:0/2851779674 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d6064f7d0 con 0x559d6090e580 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf05d9b640 1 --2- 172.21.15.7:0/2851779674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090eea0 0x559d60917540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf06d9d640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdef00a3d50 0x7fdef00a6210 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdf00005fd0 tx=0x7fdf00013b00 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdf0659c640 1 --2- 172.21.15.7:0/2851779674 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d6090e580 0x559d6090e960 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009d80 tx=0x7fdef8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf6188463b24581 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdef4ff9640 1 -- 172.21.15.7:0/2851779674 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef801e070 con 0x559d6090e580 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdef4ff9640 1 -- 172.21.15.7:0/2851779674 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdef80035f0 con 0x559d6090e580 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.526+0000 7fdef4ff9640 1 -- 172.21.15.7:0/2851779674 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef80196b0 con 0x559d6090e580 2024-04-21T08:47:57.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.527+0000 7fdf0f80b740 1 -- 172.21.15.7:0/2851779674 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d6090e580 msgr2=0x559d6090e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.527+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/2851779674 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d6090e580 0x559d6090e960 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009d80 tx=0x7fdef8011810 comp rx=0 tx=0).stop 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.527+0000 7fdf0f80b740 1 -- 172.21.15.7:0/2851779674 shutdown_connections 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.527+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/2851779674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d60917a80 0x559d60919e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.527+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/2851779674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090eea0 0x559d60917540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.527+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/2851779674 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d6090e580 0x559d6090e960 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.527+0000 7fdf0f80b740 1 -- 172.21.15.7:0/2851779674 >> 172.21.15.7:0/2851779674 conn(0x559d60908cb0 msgr2=0x559d60909090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.527+0000 7fdf0f80b740 1 -- 172.21.15.7:0/2851779674 shutdown_connections 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.527+0000 7fdf0f80b740 1 -- 172.21.15.7:0/2851779674 wait complete. 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0f80b740 1 -- start start 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0f80b740 1 Processor -- start 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 0x559d60914ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 0x559d60915020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d60915560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x559d6091a870 con 0x559d60917a80 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559d6082c2f0 con 0x559d6090e580 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559d6091a9f0 con 0x559d6090eea0 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0f80b740 10 client.0 _ll_register_callbacks cb 0x559d60839580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 0x559d60914ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf06d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d60915560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 0x559d60914ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39626/0 (socket says 172.21.15.7:39626) 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0659c640 1 -- 172.21.15.7:0/718876507 learned_addr learned my addr 172.21.15.7:0/718876507 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf05d9b640 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 0x559d60915020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf06d9d640 1 -- 172.21.15.7:0/718876507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 msgr2=0x559d60915020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf06d9d640 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 0x559d60915020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf06d9d640 1 -- 172.21.15.7:0/718876507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 msgr2=0x559d60914ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf06d9d640 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 0x559d60914ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf06d9d640 1 -- 172.21.15.7:0/718876507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d605853f0 con 0x559d60917a80 2024-04-21T08:47:57.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf05d9b640 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 0x559d60915020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:57.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.528+0000 7fdf0659c640 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 0x559d60914ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:57.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdf06d9d640 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d60915560 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fdf0000f3c0 tx=0x7fdf00004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7917a9519c7dac29 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdeedffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:47:57.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdeedffb640 1 -- 172.21.15.7:0/718876507 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf00028400 con 0x559d60917a80 2024-04-21T08:47:57.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdeedffb640 1 -- 172.21.15.7:0/718876507 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fdf000285a0 con 0x559d60917a80 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdf0f80b740 1 -- 172.21.15.7:0/718876507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdef80035f0 con 0x559d60917a80 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdeedffb640 1 -- 172.21.15.7:0/718876507 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdf000287a0 con 0x559d60917a80 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdf0f80b740 10 client.16374 fetch_fsmap learned FSMap version 23 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdf0f80b740 1 -- 172.21.15.7:0/718876507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559d605450f0 con 0x559d60917a80 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdeedffb640 1 -- 172.21.15.7:0/718876507 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdef80035f0 con 0x559d60917a80 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdeedffb640 1 -- 172.21.15.7:0/718876507 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fdf0001f070 con 0x559d60917a80 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdf0f80b740 10 client.16374 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdf0f80b740 10 client.16374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdf0f80b740 20 client.16374 populate_metadata read hostname 'smithi007' 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.529+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559d60927ee0 0x559d6092a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.530+0000 7fdf0f80b740 4 client.16374 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.530+0000 7fdf0f80b740 1 -- 172.21.15.7:0/718876507 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d60880c00 con 0x559d60927ee0 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.530+0000 7fdf0659c640 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559d60927ee0 0x559d6092a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.530+0000 7fdf0659c640 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559d60927ee0 0x559d6092a360 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=737efc9addfa1485 server_cookie=1ad61f3a53e05397 in_seq=0 out_seq=0 2024-04-21T08:47:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:47:57.530+0000 7fdeedffb640 10 client.16374 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:47:58.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:47:58 smithi016 ceph-mon[31259]: pgmap v102: 129 pgs: 129 active+clean; 286 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 1.03k op/s 2024-04-21T08:47:58.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:47:58 smithi077 ceph-mon[31615]: pgmap v102: 129 pgs: 129 active+clean; 286 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 1.03k op/s 2024-04-21T08:47:58.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:47:58 smithi007 ceph-mon[28477]: pgmap v102: 129 pgs: 129 active+clean; 286 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 1.03k op/s 2024-04-21T08:48:00.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:00 smithi077 ceph-mon[31615]: pgmap v103: 129 pgs: 129 active+clean; 289 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 886 op/s 2024-04-21T08:48:00.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:00 smithi007 ceph-mon[28477]: pgmap v103: 129 pgs: 129 active+clean; 289 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 886 op/s 2024-04-21T08:48:01.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:00 smithi016 ceph-mon[31259]: pgmap v103: 129 pgs: 129 active+clean; 289 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 886 op/s 2024-04-21T08:48:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.605+0000 7f0e37fff640 1 -- 172.21.15.7:0/2516419555 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b5465aa3d0 con 0x55b54672e0f0 2024-04-21T08:48:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.605+0000 7f0e37fff640 10 client.16362 handle_command_reply: tid=0 2024-04-21T08:48:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 1 client.16362 shutdown 2024-04-21T08:48:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 1 -- 172.21.15.7:0/2516419555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a5b60 msgr2=0x55b5466b73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a5b60 0x55b5466b73b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0e50003dd0 tx=0x7f0e50005bd0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 1 -- 172.21.15.7:0/2516419555 shutdown_connections 2024-04-21T08:48:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b54672e0f0 0x55b5467304e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b5466af0e0 0x55b5466a8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b5466a68d0 0x55b5466a86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 1 --2- 172.21.15.7:0/2516419555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b5466a5b60 0x55b5466b73b0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 1 -- 172.21.15.7:0/2516419555 >> 172.21.15.7:0/2516419555 conn(0x55b54666cc90 msgr2=0x55b5466aa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e37fff640 0 client.16362 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 1 -- 172.21.15.7:0/2516419555 shutdown_connections 2024-04-21T08:48:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 1 -- 172.21.15.7:0/2516419555 wait complete. 2024-04-21T08:48:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.606+0000 7f0e6183d740 20 client.16362 trim_cache size 0 max 16384 2024-04-21T08:48:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:48:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:48:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:48:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:48:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 -- 172.21.15.7:0/1738018899 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0e440a3d10 msgr2=0x7f0e440a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0e440a3d10 0x7f0e440a61d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0e480096a0 tx=0x7f0e48009bb0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 -- 172.21.15.7:0/1738018899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e5810e8f0 msgr2=0x7f0e5819a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e5810e8f0 0x7f0e5819a810 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0e58102f20 tx=0x7f0e4000dae0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 -- 172.21.15.7:0/1738018899 shutdown_connections 2024-04-21T08:48:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e5810e8f0 0x7f0e5819a810 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0e440a3d10 0x7f0e440a61d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e58101660 0x7f0e581a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 --2- 172.21.15.7:0/1738018899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e58100d40 0x7f0e581a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 -- 172.21.15.7:0/1738018899 >> 172.21.15.7:0/1738018899 conn(0x7f0e58077f70 msgr2=0x7f0e5810ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 -- 172.21.15.7:0/1738018899 shutdown_connections 2024-04-21T08:48:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:02.607+0000 7f0e5ec50640 1 -- 172.21.15.7:0/1738018899 wait complete. 2024-04-21T08:48:02.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdeedffb640 1 -- 172.21.15.7:0/718876507 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559d60880c00 con 0x559d60927ee0 2024-04-21T08:48:02.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdeedffb640 10 client.16374 handle_command_reply: tid=0 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 1 client.16374 shutdown 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 1 -- 172.21.15.7:0/718876507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 msgr2=0x559d60915560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d60915560 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fdf0000f3c0 tx=0x7fdf00004ab0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 1 -- 172.21.15.7:0/718876507 shutdown_connections 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559d60927ee0 0x559d6092a360 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d60915560 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 0x559d60915020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdeedffb640 0 client.16374 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/718876507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 0x559d60914ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 1 -- 172.21.15.7:0/718876507 >> 172.21.15.7:0/718876507 conn(0x559d60908cb0 msgr2=0x559d6090aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 1 -- 172.21.15.7:0/718876507 shutdown_connections 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 1 -- 172.21.15.7:0/718876507 wait complete. 2024-04-21T08:48:02.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.608+0000 7fdf0f80b740 20 client.16374 trim_cache size 0 max 16384 2024-04-21T08:48:02.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.632+0000 7fdf0f80b740 1 Processor -- start 2024-04-21T08:48:02.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.632+0000 7fdf0f80b740 1 -- start start 2024-04-21T08:48:02.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.632+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090e580 0x559d60917210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:02.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.632+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090eea0 0x559d60923730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:02.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.632+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d60907bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:02.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.632+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081af620 con 0x559d60917a80 2024-04-21T08:48:02.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.632+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081b1ea0 con 0x559d6090eea0 2024-04-21T08:48:02.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.632+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081afc90 con 0x559d6090e580 2024-04-21T08:48:02.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.632+0000 7fdf06d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d60907bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:02.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.632+0000 7fdf0659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090e580 0x559d60917210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.632+0000 7fdf06d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d60907bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44502/0 (socket says 172.21.15.7:44502) 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.633+0000 7fdf06d9d640 1 -- 172.21.15.7:0/812858032 learned_addr learned my addr 172.21.15.7:0/812858032 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.633+0000 7fdf06d9d640 1 -- 172.21.15.7:0/812858032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090e580 msgr2=0x559d60917210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.633+0000 7fdf06d9d640 1 --2- 172.21.15.7:0/812858032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090e580 0x559d60917210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.633+0000 7fdf06d9d640 1 -- 172.21.15.7:0/812858032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090eea0 msgr2=0x559d60923730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.633+0000 7fdf06d9d640 1 --2- 172.21.15.7:0/812858032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090eea0 0x559d60923730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.633+0000 7fdf06d9d640 1 -- 172.21.15.7:0/812858032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d60919e70 con 0x559d60917a80 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.633+0000 7fdf0659c640 1 --2- 172.21.15.7:0/812858032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090e580 0x559d60917210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.633+0000 7fdf06d9d640 1 --2- 172.21.15.7:0/812858032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d60907bf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fdf000098c0 tx=0x7fdf00005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112ce82371c59808 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.633+0000 7fdeeffff640 1 -- 172.21.15.7:0/812858032 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf00028400 con 0x559d60917a80 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.633+0000 7fdeeffff640 1 -- 172.21.15.7:0/812858032 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf000285a0 con 0x559d60917a80 2024-04-21T08:48:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.633+0000 7fdeeffff640 1 -- 172.21.15.7:0/812858032 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf00003b70 con 0x559d60917a80 2024-04-21T08:48:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.634+0000 7fdf0f80b740 1 -- 172.21.15.7:0/812858032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 msgr2=0x559d60907bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.634+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/812858032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d60907bf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fdf000098c0 tx=0x7fdf00005f10 comp rx=0 tx=0).stop 2024-04-21T08:48:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.634+0000 7fdf0f80b740 1 -- 172.21.15.7:0/812858032 shutdown_connections 2024-04-21T08:48:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.634+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/812858032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d60907bf0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.634+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/812858032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090eea0 0x559d60923730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.634+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/812858032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090e580 0x559d60917210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.634+0000 7fdf0f80b740 1 -- 172.21.15.7:0/812858032 >> 172.21.15.7:0/812858032 conn(0x559d60908cb0 msgr2=0x559d609074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.634+0000 7fdf0f80b740 1 -- 172.21.15.7:0/812858032 shutdown_connections 2024-04-21T08:48:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.634+0000 7fdf0f80b740 1 -- 172.21.15.7:0/812858032 wait complete. 2024-04-21T08:48:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0f80b740 1 -- start start 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0f80b740 1 Processor -- start 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 0x559d6091b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 0x559d6091bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0f80b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d6091c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081b0360 con 0x559d60917a80 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081b17d0 con 0x559d6090e580 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0f80b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdf081b0a30 con 0x559d6090eea0 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0f80b740 10 client.0 _ll_register_callbacks cb 0x7fdf00012070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf06d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d6091c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 0x559d6091b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf05d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 0x559d6091bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 0x559d6091b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39648/0 (socket says 172.21.15.7:39648) 2024-04-21T08:48:02.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf06d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d6091c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44508/0 (socket says 172.21.15.7:44508) 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.635+0000 7fdf0659c640 1 -- 172.21.15.7:0/3510166606 learned_addr learned my addr 172.21.15.7:0/3510166606 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdf05d9b640 1 -- 172.21.15.7:0/3510166606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 msgr2=0x559d6091b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdf05d9b640 1 --2- 172.21.15.7:0/3510166606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 0x559d6091b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdf05d9b640 1 -- 172.21.15.7:0/3510166606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 msgr2=0x559d6091c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdf05d9b640 1 --2- 172.21.15.7:0/3510166606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d6091c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdf05d9b640 1 -- 172.21.15.7:0/3510166606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d6098d5e0 con 0x559d6090eea0 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdf05d9b640 1 --2- 172.21.15.7:0/3510166606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 0x559d6091bcf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0ef550 tx=0x7fdefc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f8597bb79453ddf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdeedffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdeedffb640 1 -- 172.21.15.7:0/3510166606 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdefc0bd730 con 0x559d6090eea0 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdeedffb640 1 -- 172.21.15.7:0/3510166606 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fdefc02cb70 con 0x559d6090eea0 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdf0f80b740 1 -- 172.21.15.7:0/3510166606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf000285a0 con 0x559d6090eea0 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdeedffb640 1 -- 172.21.15.7:0/3510166606 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdefc0be4d0 con 0x559d6090eea0 2024-04-21T08:48:02.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdf0f80b740 10 client.25951 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.636+0000 7fdf0f80b740 1 -- 172.21.15.7:0/3510166606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559d6098e950 con 0x559d6090eea0 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.637+0000 7fdeedffb640 1 -- 172.21.15.7:0/3510166606 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf000285a0 con 0x559d6090eea0 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.637+0000 7fdeedffb640 1 -- 172.21.15.7:0/3510166606 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fdefc03c030 con 0x559d6090eea0 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.637+0000 7fdf0f80b740 10 client.25951 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.637+0000 7fdf0f80b740 10 client.25951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.637+0000 7fdf0f80b740 20 client.25951 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.637+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/3510166606 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559d6098f070 0x559d60991460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.637+0000 7fdf0f80b740 4 client.25951 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.637+0000 7fdf0f80b740 1 -- 172.21.15.7:0/3510166606 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559d6087aa70 con 0x559d6098f070 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.637+0000 7fdf0659c640 1 --2- 172.21.15.7:0/3510166606 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559d6098f070 0x559d60991460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.639+0000 7fdeedffb640 10 client.25951 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.639+0000 7fdf0659c640 1 --2- 172.21.15.7:0/3510166606 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559d6098f070 0x559d60991460 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0b781b59c6eb522 server_cookie=dc379dfd1defd628 in_seq=0 out_seq=0 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdeedffb640 1 -- 172.21.15.7:0/3510166606 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559d6087aa70 con 0x559d6098f070 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdeedffb640 10 client.25951 handle_command_reply: tid=0 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 1 client.25951 shutdown 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 1 -- 172.21.15.7:0/3510166606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 msgr2=0x559d6091bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/3510166606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 0x559d6091bcf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0ef550 tx=0x7fdefc005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 1 -- 172.21.15.7:0/3510166606 shutdown_connections 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/3510166606 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559d6098f070 0x559d60991460 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/3510166606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559d60917a80 0x559d6091c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/3510166606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559d6090eea0 0x559d6091bcf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdeedffb640 0 client.25951 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 1 --2- 172.21.15.7:0/3510166606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559d6090e580 0x559d6091b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 1 -- 172.21.15.7:0/3510166606 >> 172.21.15.7:0/3510166606 conn(0x559d60908cb0 msgr2=0x559d6090a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:02.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 1 -- 172.21.15.7:0/3510166606 shutdown_connections 2024-04-21T08:48:02.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 1 -- 172.21.15.7:0/3510166606 wait complete. 2024-04-21T08:48:02.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.641+0000 7fdf0f80b740 20 client.25951 trim_cache size 0 max 16384 2024-04-21T08:48:02.645 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:48:02.645 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:48:02.645 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:48:02.645 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:48:02.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 -- 172.21.15.7:0/107704855 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdef00a3d50 msgr2=0x7fdef00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:02.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdef00a3d50 0x7fdef00a6210 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdf00005fd0 tx=0x7fdf00013b00 comp rx=0 tx=0).stop 2024-04-21T08:48:02.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 -- 172.21.15.7:0/107704855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf08107260 msgr2=0x7fdf08074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:02.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf08107260 0x7fdf08074790 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdefc02c0b0 tx=0x7fdefc0359a0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 -- 172.21.15.7:0/107704855 shutdown_connections 2024-04-21T08:48:02.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08074cd0 0x7fdf081a6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdef00a3d50 0x7fdef00a6210 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf08107260 0x7fdf08074790 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 --2- 172.21.15.7:0/107704855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf08102190 0x7fdf08074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 -- 172.21.15.7:0/107704855 >> 172.21.15.7:0/107704855 conn(0x7fdf080fda30 msgr2=0x7fdf080fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:02.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 -- 172.21.15.7:0/107704855 shutdown_connections 2024-04-21T08:48:02.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.642+0000 7fdf0cb46640 1 -- 172.21.15.7:0/107704855 wait complete. 2024-04-21T08:48:02.656 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:48:02.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.758+0000 7f906d550640 1 -- 172.21.15.7:0/2121163555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90680692f0 msgr2=0x7f90680696d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.758+0000 7f906d550640 1 --2- 172.21.15.7:0/2121163555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90680692f0 0x7f90680696d0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f9058009900 tx=0x7f905802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.759+0000 7f906d550640 1 -- 172.21.15.7:0/2121163555 shutdown_connections 2024-04-21T08:48:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.759+0000 7f906d550640 1 --2- 172.21.15.7:0/2121163555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f906810e910 0x7f9068110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.759+0000 7f906d550640 1 --2- 172.21.15.7:0/2121163555 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9068069c10 0x7f906810e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.759+0000 7f906d550640 1 --2- 172.21.15.7:0/2121163555 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90680692f0 0x7f90680696d0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.759+0000 7f906d550640 1 -- 172.21.15.7:0/2121163555 >> 172.21.15.7:0/2121163555 conn(0x7f90680771d0 msgr2=0x7f90680775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.759+0000 7f906d550640 1 -- 172.21.15.7:0/2121163555 shutdown_connections 2024-04-21T08:48:02.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.759+0000 7f906d550640 1 -- 172.21.15.7:0/2121163555 wait complete. 2024-04-21T08:48:02.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.759+0000 7f906d550640 1 Processor -- start 2024-04-21T08:48:02.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.759+0000 7f906d550640 1 -- start start 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f906d550640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90680692f0 0x7f90681a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f906d550640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9068069c10 0x7f90681a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f906d550640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f906810e910 0x7f906819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f906d550640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9068114350 con 0x7f9068069c10 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f906d550640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90681141d0 con 0x7f90680692f0 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f906d550640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f90681144d0 con 0x7f906810e910 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f90667fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9068069c10 0x7f90681a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f9066ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90680692f0 0x7f90681a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f90667fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9068069c10 0x7f90681a0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44524/0 (socket says 172.21.15.7:44524) 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f90667fc640 1 -- 172.21.15.7:0/1773096879 learned_addr learned my addr 172.21.15.7:0/1773096879 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f90677fe640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f906810e910 0x7f906819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:02.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f90667fc640 1 -- 172.21.15.7:0/1773096879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f906810e910 msgr2=0x7f906819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f90667fc640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f906810e910 0x7f906819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f90667fc640 1 -- 172.21.15.7:0/1773096879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90680692f0 msgr2=0x7f90681a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f90667fc640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90680692f0 0x7f90681a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f90667fc640 1 -- 172.21.15.7:0/1773096879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906819af50 con 0x7f9068069c10 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f90677fe640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f906810e910 0x7f906819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f9066ffd640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90680692f0 0x7f90681a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f90667fc640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9068069c10 0x7f90681a0b10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f905c00e9f0 tx=0x7f905c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f904ffff640 1 -- 172.21.15.7:0/1773096879 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c00cd80 con 0x7f9068069c10 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f904ffff640 1 -- 172.21.15.7:0/1773096879 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f905c0108e0 con 0x7f9068069c10 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f906d550640 1 -- 172.21.15.7:0/1773096879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906819b240 con 0x7f9068069c10 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f904ffff640 1 -- 172.21.15.7:0/1773096879 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c018980 con 0x7f9068069c10 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.760+0000 7f906d550640 1 -- 172.21.15.7:0/1773096879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90681a7550 con 0x7f9068069c10 2024-04-21T08:48:02.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.761+0000 7f906d550640 1 -- 172.21.15.7:0/1773096879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9068101440 con 0x7f9068069c10 2024-04-21T08:48:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.764+0000 7f904ffff640 1 -- 172.21.15.7:0/1773096879 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f905c010430 con 0x7f9068069c10 2024-04-21T08:48:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.764+0000 7f904ffff640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90500a3d20 0x7f90500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.764+0000 7f9066ffd640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90500a3d20 0x7f90500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.764+0000 7f904ffff640 1 -- 172.21.15.7:0/1773096879 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f905c0d1490 con 0x7f9068069c10 2024-04-21T08:48:02.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.765+0000 7f9066ffd640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90500a3d20 0x7f90500a61e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9058006010 tx=0x7f9058005e40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:02.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.768+0000 7f904ffff640 1 -- 172.21.15.7:0/1773096879 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f905c09d6e0 con 0x7f9068069c10 2024-04-21T08:48:02.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:02 smithi077 ceph-mon[31615]: pgmap v104: 129 pgs: 129 active+clean; 289 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 31 op/s 2024-04-21T08:48:02.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:02 smithi007 ceph-mon[28477]: pgmap v104: 129 pgs: 129 active+clean; 289 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 31 op/s 2024-04-21T08:48:02.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:02.999+0000 7f906d550640 1 -- 172.21.15.7:0/1773096879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9068109aa0 con 0x7f9068069c10 2024-04-21T08:48:03.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.000+0000 7f904ffff640 1 -- 172.21.15.7:0/1773096879 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7f905c09cfe0 con 0x7f9068069c10 2024-04-21T08:48:03.000 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:48:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:48:03.001 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:48:03.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 -- 172.21.15.7:0/1773096879 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90500a3d20 msgr2=0x7f90500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90500a3d20 0x7f90500a61e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9058006010 tx=0x7f9058005e40 comp rx=0 tx=0).stop 2024-04-21T08:48:03.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 -- 172.21.15.7:0/1773096879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9068069c10 msgr2=0x7f90681a0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9068069c10 0x7f90681a0b10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f905c00e9f0 tx=0x7f905c00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 -- 172.21.15.7:0/1773096879 shutdown_connections 2024-04-21T08:48:03.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f90500a3d20 0x7f90500a61e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f906810e910 0x7f906819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9068069c10 0x7f90681a0b10 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 --2- 172.21.15.7:0/1773096879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90680692f0 0x7f90681a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 -- 172.21.15.7:0/1773096879 >> 172.21.15.7:0/1773096879 conn(0x7f90680771d0 msgr2=0x7f906810fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:03.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 -- 172.21.15.7:0/1773096879 shutdown_connections 2024-04-21T08:48:03.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.002+0000 7f906d550640 1 -- 172.21.15.7:0/1773096879 wait complete. 2024-04-21T08:48:03.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:48:03.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:02 smithi016 ceph-mon[31259]: pgmap v104: 129 pgs: 129 active+clean; 289 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 31 op/s 2024-04-21T08:48:03.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.119+0000 7f723e66a640 1 -- 172.21.15.7:0/758593212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7238103b70 msgr2=0x7f723810bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.119+0000 7f723e66a640 1 --2- 172.21.15.7:0/758593212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7238103b70 0x7f723810bfc0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f722c009900 tx=0x7f722c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:03.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.119+0000 7f723e66a640 1 -- 172.21.15.7:0/758593212 shutdown_connections 2024-04-21T08:48:03.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.119+0000 7f723e66a640 1 --2- 172.21.15.7:0/758593212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f723810c6d0 0x7f723810eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.119+0000 7f723e66a640 1 --2- 172.21.15.7:0/758593212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7238103b70 0x7f723810bfc0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.119+0000 7f723e66a640 1 --2- 172.21.15.7:0/758593212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72381031c0 0x7f72381035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.119+0000 7f723e66a640 1 -- 172.21.15.7:0/758593212 >> 172.21.15.7:0/758593212 conn(0x7f7238077f70 msgr2=0x7f7238078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:03.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.119+0000 7f723e66a640 1 -- 172.21.15.7:0/758593212 shutdown_connections 2024-04-21T08:48:03.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.119+0000 7f723e66a640 1 -- 172.21.15.7:0/758593212 wait complete. 2024-04-21T08:48:03.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723e66a640 1 Processor -- start 2024-04-21T08:48:03.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723e66a640 1 -- start start 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723e66a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72381031c0 0x7f723819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723e66a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f723810c6d0 0x7f723819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723e66a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7238198710 0x7f7238198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723e66a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7238112350 con 0x7f7238198710 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723e66a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f72381121d0 con 0x7f72381031c0 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723e66a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72381124d0 con 0x7f723810c6d0 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723cc07640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7238198710 0x7f7238198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f7237fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72381031c0 0x7f723819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723cc07640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7238198710 0x7f7238198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44550/0 (socket says 172.21.15.7:44550) 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723cc07640 1 -- 172.21.15.7:0/3889800583 learned_addr learned my addr 172.21.15.7:0/3889800583 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f72377fe640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f723810c6d0 0x7f723819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723cc07640 1 -- 172.21.15.7:0/3889800583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f723810c6d0 msgr2=0x7f723819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723cc07640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f723810c6d0 0x7f723819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723cc07640 1 -- 172.21.15.7:0/3889800583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72381031c0 msgr2=0x7f723819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.120+0000 7f723cc07640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72381031c0 0x7f723819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.121+0000 7f723cc07640 1 -- 172.21.15.7:0/3889800583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7238199310 con 0x7f7238198710 2024-04-21T08:48:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.121+0000 7f723cc07640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7238198710 0x7f7238198b90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f723000eb10 tx=0x7f723000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.121+0000 7f72357fa640 1 -- 172.21.15.7:0/3889800583 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230004b20 con 0x7f7238198710 2024-04-21T08:48:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.121+0000 7f723e66a640 1 -- 172.21.15.7:0/3889800583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72381a53f0 con 0x7f7238198710 2024-04-21T08:48:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.121+0000 7f72357fa640 1 -- 172.21.15.7:0/3889800583 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7230004cc0 con 0x7f7238198710 2024-04-21T08:48:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.121+0000 7f72357fa640 1 -- 172.21.15.7:0/3889800583 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230010690 con 0x7f7238198710 2024-04-21T08:48:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.121+0000 7f723e66a640 1 -- 172.21.15.7:0/3889800583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72381a57f0 con 0x7f7238198710 2024-04-21T08:48:03.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f72357fa640 1 -- 172.21.15.7:0/3889800583 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f7230005020 con 0x7f7238198710 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f724125e740 1 Processor -- start 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f724125e740 1 -- start start 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f724125e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb044d0 0x55953eb048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f72357fa640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f72200a3d20 0x7f72200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f724125e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f724125e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb0da00 0x55953eb0fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f724125e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55953ea691f0 con 0x55953eb04e80 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f724125e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55953ea064b0 con 0x55953eb0da00 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f724125e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55953ea651b0 con 0x55953eb044d0 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f723cc07640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb0da00 0x55953eb0fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f72377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f7237fff640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f72200a3d20 0x7f72200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f72377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44552/0 (socket says 172.21.15.7:44552) 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f72377fe640 1 -- 172.21.15.7:0/2216898034 learned_addr learned my addr 172.21.15.7:0/2216898034 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.124+0000 7f72377fe640 1 --2- 172.21.15.7:0/2216898034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb044d0 0x55953eb048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f72357fa640 1 -- 172.21.15.7:0/3889800583 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f72300d1130 con 0x7f7238198710 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f723cc07640 1 -- 172.21.15.7:0/2216898034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb044d0 msgr2=0x55953eb048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f723cc07640 1 --2- 172.21.15.7:0/2216898034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb044d0 0x55953eb048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f723cc07640 1 -- 172.21.15.7:0/2216898034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 msgr2=0x55953eb0d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f723cc07640 1 --2- 172.21.15.7:0/2216898034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f723cc07640 1 -- 172.21.15.7:0/2216898034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55953e9f81b0 con 0x55953eb0da00 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f72377fe640 1 --2- 172.21.15.7:0/2216898034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f7237fff640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f72200a3d20 0x7f72200a61e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7228009bd0 tx=0x7f7228005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f723cc07640 1 --2- 172.21.15.7:0/2216898034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb0da00 0x55953eb0fe50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f72300ce430 tx=0x7f7230000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29826d973dc8a086 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:03.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f721e7fc640 1 -- 172.21.15.7:0/2216898034 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723009c5e0 con 0x55953eb0da00 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f721e7fc640 1 -- 172.21.15.7:0/2216898034 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72300a67b0 con 0x55953eb0da00 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.125+0000 7f721e7fc640 1 -- 172.21.15.7:0/2216898034 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72300bd5c0 con 0x55953eb0da00 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.126+0000 7f724125e740 1 -- 172.21.15.7:0/2216898034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb0da00 msgr2=0x55953eb0fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.126+0000 7f724125e740 1 --2- 172.21.15.7:0/2216898034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb0da00 0x55953eb0fe50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f72300ce430 tx=0x7f7230000f40 comp rx=0 tx=0).stop 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.126+0000 7f724125e740 1 -- 172.21.15.7:0/2216898034 shutdown_connections 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.126+0000 7f724125e740 1 --2- 172.21.15.7:0/2216898034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb0da00 0x55953eb0fe50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.126+0000 7f724125e740 1 --2- 172.21.15.7:0/2216898034 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.126+0000 7f724125e740 1 --2- 172.21.15.7:0/2216898034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb044d0 0x55953eb048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.126+0000 7f724125e740 1 -- 172.21.15.7:0/2216898034 >> 172.21.15.7:0/2216898034 conn(0x55953eafeba0 msgr2=0x55953eafef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.126+0000 7f724125e740 1 -- 172.21.15.7:0/2216898034 shutdown_connections 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.126+0000 7f724125e740 1 -- 172.21.15.7:0/2216898034 wait complete. 2024-04-21T08:48:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.126+0000 7f724125e740 1 -- start start 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.126+0000 7f724125e740 1 Processor -- start 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f724125e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 0x55953eb0d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f724125e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f724125e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb0da00 0x55953eb0ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f724125e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55953eb10910 con 0x55953eb04e80 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f724125e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55953ea22500 con 0x55953eb044d0 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f724125e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55953eb10a90 con 0x55953eb0da00 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f724125e740 10 client.0 _ll_register_callbacks cb 0x55953e9f0b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f72377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f7237fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 0x55953eb0d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f72377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44554/0 (socket says 172.21.15.7:44554) 2024-04-21T08:48:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f72377fe640 1 -- 172.21.15.7:0/2023889914 learned_addr learned my addr 172.21.15.7:0/2023889914 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f7237fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 0x55953eb0d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39696/0 (socket says 172.21.15.7:39696) 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f723cc07640 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb0da00 0x55953eb0ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f72377fe640 1 -- 172.21.15.7:0/2023889914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb0da00 msgr2=0x55953eb0ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f72377fe640 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb0da00 0x55953eb0ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f72377fe640 1 -- 172.21.15.7:0/2023889914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 msgr2=0x55953eb0d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f72377fe640 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 0x55953eb0d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f72377fe640 1 -- 172.21.15.7:0/2023889914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55953e77b3f0 con 0x55953eb04e80 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f7237fff640 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 0x55953eb0d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f72377fe640 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0a690 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f722c036040 tx=0x7f722c0058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5322bc32ce8ca21 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.127+0000 7f72177fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f72177fe640 1 -- 172.21.15.7:0/2023889914 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c005250 con 0x55953eb04e80 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f72177fe640 1 -- 172.21.15.7:0/2023889914 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f722c0053f0 con 0x55953eb04e80 2024-04-21T08:48:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f724125e740 1 -- 172.21.15.7:0/2023889914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72300a67b0 con 0x55953eb04e80 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f72177fe640 1 -- 172.21.15.7:0/2023889914 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f722c0055f0 con 0x55953eb04e80 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f724125e740 10 client.16398 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f724125e740 1 -- 172.21.15.7:0/2023889914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55953e8457d0 con 0x55953eb04e80 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f72177fe640 1 -- 172.21.15.7:0/2023889914 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72300a67b0 con 0x55953eb04e80 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f72177fe640 1 -- 172.21.15.7:0/2023889914 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f722c04b060 con 0x55953eb04e80 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f724125e740 10 client.16398 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f724125e740 10 client.16398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f724125e740 20 client.16398 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f724125e740 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55953eb1dbe0 0x55953eb20010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f724125e740 4 client.16398 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f724125e740 1 -- 172.21.15.7:0/2023889914 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55953ea08930 con 0x55953eb1dbe0 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.128+0000 7f7237fff640 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55953eb1dbe0 0x55953eb20010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.129+0000 7f72177fe640 10 client.16398 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.129+0000 7f7237fff640 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55953eb1dbe0 0x55953eb20010 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17950454924ad7c4 server_cookie=137dd58506a31d17 in_seq=0 out_seq=0 2024-04-21T08:48:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f72177fe640 1 -- 172.21.15.7:0/2023889914 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55953ea08930 con 0x55953eb1dbe0 2024-04-21T08:48:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f72177fe640 10 client.16398 handle_command_reply: tid=0 2024-04-21T08:48:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 1 client.16398 shutdown 2024-04-21T08:48:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 1 -- 172.21.15.7:0/2023889914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 msgr2=0x55953eb0a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0a690 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f722c036040 tx=0x7f722c0058f0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 1 -- 172.21.15.7:0/2023889914 shutdown_connections 2024-04-21T08:48:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55953eb1dbe0 0x55953eb20010 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb0da00 0x55953eb0ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb0a690 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f72177fe640 0 client.16398 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 1 --2- 172.21.15.7:0/2023889914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 0x55953eb0d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 1 -- 172.21.15.7:0/2023889914 >> 172.21.15.7:0/2023889914 conn(0x55953eafeba0 msgr2=0x55953eb008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 1 -- 172.21.15.7:0/2023889914 shutdown_connections 2024-04-21T08:48:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 1 -- 172.21.15.7:0/2023889914 wait complete. 2024-04-21T08:48:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.133+0000 7f724125e740 20 client.16398 trim_cache size 0 max 16384 2024-04-21T08:48:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f724125e740 1 Processor -- start 2024-04-21T08:48:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f724125e740 1 -- start start 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f724125e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb044d0 0x55953eb197d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f724125e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb006e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f724125e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb0da00 0x55953eb0cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f724125e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f72381ad570 con 0x55953eb04e80 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f724125e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f72381afdf0 con 0x55953eb0da00 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f724125e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72381adbe0 con 0x55953eb044d0 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f72377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f72377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44568/0 (socket says 172.21.15.7:44568) 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f72377fe640 1 -- 172.21.15.7:0/2766886917 learned_addr learned my addr 172.21.15.7:0/2766886917 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f723cc07640 1 --2- 172.21.15.7:0/2766886917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb0da00 0x55953eb0cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f7237fff640 1 --2- 172.21.15.7:0/2766886917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb044d0 0x55953eb197d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f72377fe640 1 -- 172.21.15.7:0/2766886917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb044d0 msgr2=0x55953eb197d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f72377fe640 1 --2- 172.21.15.7:0/2766886917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb044d0 0x55953eb197d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f72377fe640 1 -- 172.21.15.7:0/2766886917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb0da00 msgr2=0x55953eb0cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f72377fe640 1 --2- 172.21.15.7:0/2766886917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb0da00 0x55953eb0cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f72377fe640 1 -- 172.21.15.7:0/2766886917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55953eb04c20 con 0x55953eb04e80 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.146+0000 7f72377fe640 1 --2- 172.21.15.7:0/2766886917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb006e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f722c02c170 tx=0x7f722c02c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b180902018eed9c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f721dffb640 1 -- 172.21.15.7:0/2766886917 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c0055f0 con 0x55953eb04e80 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f721dffb640 1 -- 172.21.15.7:0/2766886917 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f722c02cc70 con 0x55953eb04e80 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f721dffb640 1 -- 172.21.15.7:0/2766886917 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c004b70 con 0x55953eb04e80 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f724125e740 1 -- 172.21.15.7:0/2766886917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 msgr2=0x55953eb006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f724125e740 1 --2- 172.21.15.7:0/2766886917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb006e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f722c02c170 tx=0x7f722c02c690 comp rx=0 tx=0).stop 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f724125e740 1 -- 172.21.15.7:0/2766886917 shutdown_connections 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f724125e740 1 --2- 172.21.15.7:0/2766886917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb0da00 0x55953eb0cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f724125e740 1 --2- 172.21.15.7:0/2766886917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb006e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f724125e740 1 --2- 172.21.15.7:0/2766886917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb044d0 0x55953eb197d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f724125e740 1 -- 172.21.15.7:0/2766886917 >> 172.21.15.7:0/2766886917 conn(0x55953eafeba0 msgr2=0x55953eafffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f724125e740 1 -- 172.21.15.7:0/2766886917 shutdown_connections 2024-04-21T08:48:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f724125e740 1 -- 172.21.15.7:0/2766886917 wait complete. 2024-04-21T08:48:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.147+0000 7f724125e740 1 -- start start 2024-04-21T08:48:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f724125e740 1 Processor -- start 2024-04-21T08:48:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f724125e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 0x55953eb15ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f724125e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb112a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f724125e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb0da00 0x55953eb117e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f724125e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f72381ae2b0 con 0x55953eb04e80 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f724125e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f72381af720 con 0x55953eb044d0 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f724125e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72381ae980 con 0x55953eb0da00 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f724125e740 10 client.0 _ll_register_callbacks cb 0x7f722c02caf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f72377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb112a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f72377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb112a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44570/0 (socket says 172.21.15.7:44570) 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f72377fe640 1 -- 172.21.15.7:0/2427067110 learned_addr learned my addr 172.21.15.7:0/2427067110 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f7237fff640 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 0x55953eb15ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f723cc07640 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb0da00 0x55953eb117e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f72377fe640 1 -- 172.21.15.7:0/2427067110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb0da00 msgr2=0x55953eb117e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f72377fe640 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb0da00 0x55953eb117e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f72377fe640 1 -- 172.21.15.7:0/2427067110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 msgr2=0x55953eb15ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f72377fe640 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 0x55953eb15ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.148+0000 7f72377fe640 1 -- 172.21.15.7:0/2427067110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55953eb11e20 con 0x55953eb04e80 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f72377fe640 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb112a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f722c040900 tx=0x7f722c0406b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ee86e059fafa51 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f72177fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f72177fe640 1 -- 172.21.15.7:0/2427067110 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c063060 con 0x55953eb04e80 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f72177fe640 1 -- 172.21.15.7:0/2427067110 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f722c040de0 con 0x55953eb04e80 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f724125e740 1 -- 172.21.15.7:0/2427067110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f722c02cc70 con 0x55953eb04e80 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f72177fe640 1 -- 172.21.15.7:0/2427067110 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f722c06a050 con 0x55953eb04e80 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f724125e740 10 client.16404 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f724125e740 1 -- 172.21.15.7:0/2427067110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55953eb88390 con 0x55953eb04e80 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f72177fe640 1 -- 172.21.15.7:0/2427067110 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f722c02cc70 con 0x55953eb04e80 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f72177fe640 1 -- 172.21.15.7:0/2427067110 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f722c04b040 con 0x55953eb04e80 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f724125e740 10 client.16404 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f724125e740 10 client.16404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f724125e740 20 client.16404 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f724125e740 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55953eb88940 0x55953eb8ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f724125e740 4 client.16404 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:48:03.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f724125e740 1 -- 172.21.15.7:0/2427067110 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55953ea76c10 con 0x55953eb88940 2024-04-21T08:48:03.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.149+0000 7f7237fff640 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55953eb88940 0x55953eb8ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.150+0000 7f7237fff640 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55953eb88940 0x55953eb8ad60 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28663fd9003b2c77 server_cookie=7878c7d3a12d031a in_seq=0 out_seq=0 2024-04-21T08:48:03.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.150+0000 7f72177fe640 10 client.16404 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:03.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.152+0000 7f72177fe640 1 -- 172.21.15.7:0/2427067110 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55953ea76c10 con 0x55953eb88940 2024-04-21T08:48:03.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.152+0000 7f72177fe640 10 client.16404 handle_command_reply: tid=0 2024-04-21T08:48:03.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 1 client.16404 shutdown 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 1 -- 172.21.15.7:0/2427067110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 msgr2=0x55953eb112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb112a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f722c040900 tx=0x7f722c0406b0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 1 -- 172.21.15.7:0/2427067110 shutdown_connections 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55953eb88940 0x55953eb8ad60 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55953eb0da00 0x55953eb117e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55953eb04e80 0x55953eb112a0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f72177fe640 0 client.16404 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 1 --2- 172.21.15.7:0/2427067110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55953eb044d0 0x55953eb15ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 1 -- 172.21.15.7:0/2427067110 >> 172.21.15.7:0/2427067110 conn(0x55953eafeba0 msgr2=0x55953eb00860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 1 -- 172.21.15.7:0/2427067110 shutdown_connections 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 1 -- 172.21.15.7:0/2427067110 wait complete. 2024-04-21T08:48:03.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f724125e740 20 client.16404 trim_cache size 0 max 16384 2024-04-21T08:48:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:48:03.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f723e66a640 1 -- 172.21.15.7:0/3889800583 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f72200a3d20 msgr2=0x7f72200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f723e66a640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f72200a3d20 0x7f72200a61e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7228009bd0 tx=0x7f7228005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:03.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f723e66a640 1 -- 172.21.15.7:0/3889800583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7238198710 msgr2=0x7f7238198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.153+0000 7f723e66a640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7238198710 0x7f7238198b90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f723000eb10 tx=0x7f723000c2d0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.154+0000 7f723e66a640 1 -- 172.21.15.7:0/3889800583 shutdown_connections 2024-04-21T08:48:03.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.154+0000 7f723e66a640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7238198710 0x7f7238198b90 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.154+0000 7f723e66a640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f72200a3d20 0x7f72200a61e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.154+0000 7f723e66a640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f723810c6d0 0x7f723819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.154+0000 7f723e66a640 1 --2- 172.21.15.7:0/3889800583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72381031c0 0x7f723819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.154+0000 7f723e66a640 1 -- 172.21.15.7:0/3889800583 >> 172.21.15.7:0/3889800583 conn(0x7f7238077f70 msgr2=0x7f72380fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:03.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.154+0000 7f723e66a640 1 -- 172.21.15.7:0/3889800583 shutdown_connections 2024-04-21T08:48:03.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:03.154+0000 7f723e66a640 1 -- 172.21.15.7:0/3889800583 wait complete. 2024-04-21T08:48:03.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:03 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1773096879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:03.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:03 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1773096879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:04.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:03 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1773096879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:04.166 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:48:04.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:04.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:48:04.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.284+0000 7ff1b5031640 1 -- 172.21.15.7:0/317891788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1b01031c0 msgr2=0x7ff1b01035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.284+0000 7ff1b5031640 1 --2- 172.21.15.7:0/317891788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1b01031c0 0x7ff1b01035a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009ca0 tx=0x7ff1a002c180 comp rx=0 tx=0).stop 2024-04-21T08:48:04.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.284+0000 7ff1b5031640 1 -- 172.21.15.7:0/317891788 shutdown_connections 2024-04-21T08:48:04.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.284+0000 7ff1b5031640 1 --2- 172.21.15.7:0/317891788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b010c500 0x7ff1b010e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.284+0000 7ff1b5031640 1 --2- 172.21.15.7:0/317891788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b0103ae0 0x7ff1b010bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.284+0000 7ff1b5031640 1 --2- 172.21.15.7:0/317891788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1b01031c0 0x7ff1b01035a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.284+0000 7ff1b5031640 1 -- 172.21.15.7:0/317891788 >> 172.21.15.7:0/317891788 conn(0x7ff1b0078000 msgr2=0x7ff1b0078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.284+0000 7ff1b5031640 1 -- 172.21.15.7:0/317891788 shutdown_connections 2024-04-21T08:48:04.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.284+0000 7ff1b5031640 1 -- 172.21.15.7:0/317891788 wait complete. 2024-04-21T08:48:04.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1b5031640 1 Processor -- start 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1b5031640 1 -- start start 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1b5031640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b01031c0 0x7ff1b0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1b5031640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b0103ae0 0x7ff1b0198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1b5031640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1b010c500 0x7ff1b019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1b5031640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b0112190 con 0x7ff1b010c500 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1b5031640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b0112010 con 0x7ff1b01031c0 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1b5031640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b0112310 con 0x7ff1b0103ae0 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1af59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1b010c500 0x7ff1b019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1af59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1b010c500 0x7ff1b019cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44588/0 (socket says 172.21.15.7:44588) 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1aed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b01031c0 0x7ff1b0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1af59e640 1 -- 172.21.15.7:0/679508251 learned_addr learned my addr 172.21.15.7:0/679508251 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:04.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b0103ae0 0x7ff1b0198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1af59e640 1 -- 172.21.15.7:0/679508251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b0103ae0 msgr2=0x7ff1b0198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1af59e640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b0103ae0 0x7ff1b0198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1af59e640 1 -- 172.21.15.7:0/679508251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b01031c0 msgr2=0x7ff1b0198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1af59e640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b01031c0 0x7ff1b0198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1af59e640 1 -- 172.21.15.7:0/679508251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b019d700 con 0x7ff1b010c500 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.285+0000 7ff1aed9d640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b01031c0 0x7ff1b0198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.286+0000 7ff1af59e640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1b010c500 0x7ff1b019cfd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800e970 tx=0x7ff1a800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.286+0000 7ff197fff640 1 -- 172.21.15.7:0/679508251 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a800ccc0 con 0x7ff1b010c500 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.286+0000 7ff197fff640 1 -- 172.21.15.7:0/679508251 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1a800ce60 con 0x7ff1b010c500 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.286+0000 7ff1b5031640 1 -- 172.21.15.7:0/679508251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b019d9f0 con 0x7ff1b010c500 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.286+0000 7ff197fff640 1 -- 172.21.15.7:0/679508251 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8010430 con 0x7ff1b010c500 2024-04-21T08:48:04.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.286+0000 7ff1b5031640 1 -- 172.21.15.7:0/679508251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b01a5280 con 0x7ff1b010c500 2024-04-21T08:48:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.288+0000 7ff197fff640 1 -- 172.21.15.7:0/679508251 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7ff1a8004f00 con 0x7ff1b010c500 2024-04-21T08:48:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.288+0000 7ff1b7c29740 1 Processor -- start 2024-04-21T08:48:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.288+0000 7ff1b7c29740 1 -- start start 2024-04-21T08:48:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.288+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb28e0 0x559b27fb2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb3290 0x559b27fbb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fbe260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x559b27f16f00 con 0x559b27fb3290 2024-04-21T08:48:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559b27eb4590 con 0x559b27fbbe10 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559b27f12ec0 con 0x559b27fb28e0 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1aed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb3290 0x559b27fbb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1ae59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb28e0 0x559b27fb2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1aed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb3290 0x559b27fbb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44596/0 (socket says 172.21.15.7:44596) 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1aed9d640 1 -- 172.21.15.7:0/4124633335 learned_addr learned my addr 172.21.15.7:0/4124633335 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/4124633335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fbe260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff197fff640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff1980a3d20 0x7ff1980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1ae59c640 1 -- 172.21.15.7:0/4124633335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 msgr2=0x559b27fbe260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/4124633335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fbe260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1ae59c640 1 -- 172.21.15.7:0/4124633335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb3290 msgr2=0x559b27fbb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/4124633335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb3290 0x559b27fbb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1af59e640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff1980a3d20 0x7ff1980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1ae59c640 1 -- 172.21.15.7:0/4124633335 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b27cf17e0 con 0x559b27fb28e0 2024-04-21T08:48:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff197fff640 1 -- 172.21.15.7:0/679508251 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7ff1a80d0e30 con 0x7ff1b010c500 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/4124633335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb28e0 0x559b27fb2cc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff1b01008f0 tx=0x7ff1a4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6ea1cea0c7ad20b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1957fa640 1 -- 172.21.15.7:0/4124633335 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4004050 con 0x559b27fb28e0 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1af59e640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff1980a3d20 0x7ff1980a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8098e40 tx=0x7ff1a80bd040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1957fa640 1 -- 172.21.15.7:0/4124633335 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1a400fc70 con 0x559b27fb28e0 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.289+0000 7ff1957fa640 1 -- 172.21.15.7:0/4124633335 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a40038e0 con 0x559b27fb28e0 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.290+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4124633335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb28e0 msgr2=0x559b27fb2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.290+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/4124633335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb28e0 0x559b27fb2cc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff1b01008f0 tx=0x7ff1a4004730 comp rx=0 tx=0).stop 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.290+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4124633335 shutdown_connections 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.290+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/4124633335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fbe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.290+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/4124633335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb3290 0x559b27fbb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.290+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/4124633335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb28e0 0x559b27fb2cc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.290+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4124633335 >> 172.21.15.7:0/4124633335 conn(0x559b27facfb0 msgr2=0x559b27fad390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.290+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4124633335 shutdown_connections 2024-04-21T08:48:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.290+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4124633335 wait complete. 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1b7c29740 1 -- start start 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1b7c29740 1 Processor -- start 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fb8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb3290 0x559b27fb93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fb98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x559b27fbed80 con 0x559b27fb28e0 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559b27ed0580 con 0x559b27fbbe10 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559b27fbef00 con 0x559b27fb3290 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1b7c29740 10 client.0 _ll_register_callbacks cb 0x559b27ed6650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fb8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fb8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44602/0 (socket says 172.21.15.7:44602) 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1aed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb3290 0x559b27fb93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1ae59c640 1 -- 172.21.15.7:0/4261051762 learned_addr learned my addr 172.21.15.7:0/4261051762 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fb98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1ae59c640 1 -- 172.21.15.7:0/4261051762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb3290 msgr2=0x559b27fb93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb3290 0x559b27fb93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1ae59c640 1 -- 172.21.15.7:0/4261051762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 msgr2=0x559b27fb98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fb98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.291+0000 7ff1ae59c640 1 -- 172.21.15.7:0/4261051762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b27d51f70 con 0x559b27fb28e0 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fb8e60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x559b27fac910 tx=0x7ff1a4004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a593976665e86c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff192ff5640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff192ff5640 1 -- 172.21.15.7:0/4261051762 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4021190 con 0x559b27fb28e0 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff192ff5640 1 -- 172.21.15.7:0/4261051762 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7ff1a4021960 con 0x559b27fb28e0 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4261051762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1a400fc70 con 0x559b27fb28e0 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff192ff5640 1 -- 172.21.15.7:0/4261051762 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff1a4021b60 con 0x559b27fb28e0 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff1b7c29740 10 client.16416 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:04.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4261051762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559b27be9170 con 0x559b27fb28e0 2024-04-21T08:48:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff192ff5640 1 -- 172.21.15.7:0/4261051762 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1a400fc70 con 0x559b27fb28e0 2024-04-21T08:48:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff192ff5640 1 -- 172.21.15.7:0/4261051762 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7ff1a40047a0 con 0x559b27fb28e0 2024-04-21T08:48:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff1b7c29740 10 client.16416 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff1b7c29740 10 client.16416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff1b7c29740 20 client.16416 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559b27fcc260 0x559b27fce650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff1b7c29740 4 client.16416 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.292+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4261051762 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b27f24920 con 0x559b27fcc260 2024-04-21T08:48:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.293+0000 7ff1aed9d640 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559b27fcc260 0x559b27fce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.293+0000 7ff1aed9d640 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559b27fcc260 0x559b27fce650 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da97e7746b5cf5e3 server_cookie=2a02780ee8270fba in_seq=0 out_seq=0 2024-04-21T08:48:04.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.293+0000 7ff192ff5640 10 client.16416 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:04.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff192ff5640 1 -- 172.21.15.7:0/4261051762 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559b27f24920 con 0x559b27fcc260 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff192ff5640 10 client.16416 handle_command_reply: tid=0 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 1 client.16416 shutdown 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4261051762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 msgr2=0x559b27fb8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fb8e60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x559b27fac910 tx=0x7ff1a4004550 comp rx=0 tx=0).stop 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4261051762 shutdown_connections 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559b27fcc260 0x559b27fce650 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fb98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb3290 0x559b27fb93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/4261051762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fb8e60 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4261051762 >> 172.21.15.7:0/4261051762 conn(0x559b27facfb0 msgr2=0x559b27faed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff192ff5640 0 client.16416 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4261051762 shutdown_connections 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 1 -- 172.21.15.7:0/4261051762 wait complete. 2024-04-21T08:48:04.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.295+0000 7ff1b7c29740 20 client.16416 trim_cache size 0 max 16384 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.313+0000 7ff1b7c29740 1 Processor -- start 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.313+0000 7ff1b7c29740 1 -- start start 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.313+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.313+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb3290 0x559b27fb8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.313+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fb9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.313+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b01ad4c0 con 0x559b27fb28e0 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.313+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b01afd40 con 0x559b27fbbe10 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.313+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b01adb30 con 0x559b27fb3290 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff1ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff1ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44610/0 (socket says 172.21.15.7:44610) 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff1ae59c640 1 -- 172.21.15.7:0/390077966 learned_addr learned my addr 172.21.15.7:0/390077966 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff1aed9d640 1 --2- 172.21.15.7:0/390077966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb3290 0x559b27fb8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/390077966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fb9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff1ae59c640 1 -- 172.21.15.7:0/390077966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb3290 msgr2=0x559b27fb8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/390077966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb3290 0x559b27fb8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff1ae59c640 1 -- 172.21.15.7:0/390077966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 msgr2=0x559b27fb9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/390077966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fb9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff1ae59c640 1 -- 172.21.15.7:0/390077966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b27faf280 con 0x559b27fb28e0 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/390077966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc7c40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009710 tx=0x7ff1a4009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2fe15ad02b11ac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff194ff9640 1 -- 172.21.15.7:0/390077966 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4009e90 con 0x559b27fb28e0 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff194ff9640 1 -- 172.21.15.7:0/390077966 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1a4025020 con 0x559b27fb28e0 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.314+0000 7ff194ff9640 1 -- 172.21.15.7:0/390077966 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a400fbc0 con 0x559b27fb28e0 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 -- 172.21.15.7:0/390077966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 msgr2=0x559b27fc7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/390077966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc7c40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009710 tx=0x7ff1a4009660 comp rx=0 tx=0).stop 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 -- 172.21.15.7:0/390077966 shutdown_connections 2024-04-21T08:48:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/390077966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fbbe10 0x559b27fb9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/390077966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fb3290 0x559b27fb8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/390077966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc7c40 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 -- 172.21.15.7:0/390077966 >> 172.21.15.7:0/390077966 conn(0x559b27facfb0 msgr2=0x559b27fb8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 -- 172.21.15.7:0/390077966 shutdown_connections 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 -- 172.21.15.7:0/390077966 wait complete. 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 -- start start 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 Processor -- start 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fb3290 0x559b27fbf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fbbe10 0x559b27fbfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b01ae200 con 0x559b27fb28e0 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b01af670 con 0x559b27fb3290 2024-04-21T08:48:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b01ae8d0 con 0x559b27fbbe10 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.315+0000 7ff1b7c29740 10 client.0 _ll_register_callbacks cb 0x7ff1a8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44624/0 (socket says 172.21.15.7:44624) 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1ae59c640 1 -- 172.21.15.7:0/3060181166 learned_addr learned my addr 172.21.15.7:0/3060181166 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1aed9d640 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fb3290 0x559b27fbf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fbbe10 0x559b27fbfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1ae59c640 1 -- 172.21.15.7:0/3060181166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fbbe10 msgr2=0x559b27fbfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fbbe10 0x559b27fbfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1ae59c640 1 -- 172.21.15.7:0/3060181166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fb3290 msgr2=0x559b27fbf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fb3290 0x559b27fbf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1ae59c640 1 -- 172.21.15.7:0/3060181166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b27fc0250 con 0x559b27fb28e0 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1aed9d640 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fb3290 0x559b27fbf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1ae59c640 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc4360 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff1a40059f0 tx=0x7ff1a4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d022cec9f9eb354 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff192ff5640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff192ff5640 1 -- 172.21.15.7:0/3060181166 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4021190 con 0x559b27fb28e0 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff192ff5640 1 -- 172.21.15.7:0/3060181166 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7ff1a40217e0 con 0x559b27fb28e0 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1b7c29740 1 -- 172.21.15.7:0/3060181166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1a4025020 con 0x559b27fb28e0 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff192ff5640 1 -- 172.21.15.7:0/3060181166 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff1a4011040 con 0x559b27fb28e0 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1b7c29740 10 client.16422 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.316+0000 7ff1b7c29740 1 -- 172.21.15.7:0/3060181166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559b28032490 con 0x559b27fb28e0 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.317+0000 7ff192ff5640 1 -- 172.21.15.7:0/3060181166 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1a4025020 con 0x559b27fb28e0 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.317+0000 7ff192ff5640 1 -- 172.21.15.7:0/3060181166 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7ff1a4039070 con 0x559b27fb28e0 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.317+0000 7ff1b7c29740 10 client.16422 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.317+0000 7ff1b7c29740 10 client.16422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.317+0000 7ff1b7c29740 20 client.16422 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.317+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559b28032af0 0x559b28034f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.317+0000 7ff1b7c29740 4 client.16422 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:48:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.317+0000 7ff1b7c29740 1 -- 172.21.15.7:0/3060181166 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559b27f1e790 con 0x559b28032af0 2024-04-21T08:48:04.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.317+0000 7ff1aed9d640 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559b28032af0 0x559b28034f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.317+0000 7ff192ff5640 10 client.16422 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:04.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.317+0000 7ff1aed9d640 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559b28032af0 0x559b28034f10 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f712bfe383a0b1d5 server_cookie=6a86091161644912 in_seq=0 out_seq=0 2024-04-21T08:48:04.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.602+0000 7ff192ff5640 1 -- 172.21.15.7:0/3060181166 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559b27f1e790 con 0x559b28032af0 2024-04-21T08:48:04.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.602+0000 7ff192ff5640 10 client.16422 handle_command_reply: tid=0 2024-04-21T08:48:04.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.602+0000 7ff1b7c29740 1 client.16422 shutdown 2024-04-21T08:48:04.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.602+0000 7ff1b7c29740 1 -- 172.21.15.7:0/3060181166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 msgr2=0x559b27fc4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.602+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc4360 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff1a40059f0 tx=0x7ff1a4005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:04.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.602+0000 7ff1b7c29740 1 -- 172.21.15.7:0/3060181166 shutdown_connections 2024-04-21T08:48:04.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.602+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559b28032af0 0x559b28034f10 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b27fbbe10 0x559b27fbfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff192ff5640 0 client.16422 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:04.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559b27fb3290 0x559b27fbf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b7c29740 1 --2- 172.21.15.7:0/3060181166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b27fb28e0 0x559b27fc4360 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b7c29740 1 -- 172.21.15.7:0/3060181166 >> 172.21.15.7:0/3060181166 conn(0x559b27facfb0 msgr2=0x559b27fac010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b7c29740 1 -- 172.21.15.7:0/3060181166 shutdown_connections 2024-04-21T08:48:04.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b7c29740 1 -- 172.21.15.7:0/3060181166 wait complete. 2024-04-21T08:48:04.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b7c29740 20 client.16422 trim_cache size 0 max 16384 2024-04-21T08:48:04.604 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:48:04.604 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:48:04.605 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "a8ceae46-67db-44e4-95b3-be9792f1a693", 2024-04-21T08:48:04.605 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:48:04.605 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:48:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b5031640 1 -- 172.21.15.7:0/679508251 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff1980a3d20 msgr2=0x7ff1980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b5031640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff1980a3d20 0x7ff1980a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8098e40 tx=0x7ff1a80bd040 comp rx=0 tx=0).stop 2024-04-21T08:48:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b5031640 1 -- 172.21.15.7:0/679508251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1b010c500 msgr2=0x7ff1b019cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b5031640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1b010c500 0x7ff1b019cfd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800e970 tx=0x7ff1a800ee30 comp rx=0 tx=0).stop 2024-04-21T08:48:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b5031640 1 -- 172.21.15.7:0/679508251 shutdown_connections 2024-04-21T08:48:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b5031640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1b010c500 0x7ff1b019cfd0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b5031640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff1980a3d20 0x7ff1980a61e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b5031640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1b0103ae0 0x7ff1b0198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b5031640 1 --2- 172.21.15.7:0/679508251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1b01031c0 0x7ff1b0198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.603+0000 7ff1b5031640 1 -- 172.21.15.7:0/679508251 >> 172.21.15.7:0/679508251 conn(0x7ff1b0078000 msgr2=0x7ff1b00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.604+0000 7ff1b5031640 1 -- 172.21.15.7:0/679508251 shutdown_connections 2024-04-21T08:48:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.604+0000 7ff1b5031640 1 -- 172.21.15.7:0/679508251 wait complete. 2024-04-21T08:48:04.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:48:04.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:48:04.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.722+0000 7fd868f85640 1 -- 172.21.15.7:0/291925889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd864105d80 msgr2=0x7fd8641061e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.722+0000 7fd868f85640 1 --2- 172.21.15.7:0/291925889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd864105d80 0x7fd8641061e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009930 tx=0x7fd84c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.722+0000 7fd868f85640 1 -- 172.21.15.7:0/291925889 shutdown_connections 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.722+0000 7fd868f85640 1 --2- 172.21.15.7:0/291925889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd864106720 0x7fd864111120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.722+0000 7fd868f85640 1 --2- 172.21.15.7:0/291925889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd864105d80 0x7fd8641061e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.722+0000 7fd868f85640 1 --2- 172.21.15.7:0/291925889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8641044a0 0x7fd864104880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.722+0000 7fd868f85640 1 -- 172.21.15.7:0/291925889 >> 172.21.15.7:0/291925889 conn(0x7fd8640ffc40 msgr2=0x7fd864102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.722+0000 7fd868f85640 1 -- 172.21.15.7:0/291925889 shutdown_connections 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.722+0000 7fd868f85640 1 -- 172.21.15.7:0/291925889 wait complete. 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd868f85640 1 Processor -- start 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd868f85640 1 -- start start 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd868f85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8641044a0 0x7fd8641a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd868f85640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd864105d80 0x7fd8641a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd868f85640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd864106720 0x7fd86419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd868f85640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641144c0 con 0x7fd864105d80 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd868f85640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd864114340 con 0x7fd864106720 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd868f85640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd864114640 con 0x7fd8641044a0 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd85bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd864105d80 0x7fd8641a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd85bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd864105d80 0x7fd8641a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35132/0 (socket says 172.21.15.7:35132) 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd85bfff640 1 -- 172.21.15.7:0/1940693575 learned_addr learned my addr 172.21.15.7:0/1940693575 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.723+0000 7fd862d9d640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd864106720 0x7fd86419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd86259c640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8641044a0 0x7fd8641a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd85bfff640 1 -- 172.21.15.7:0/1940693575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8641044a0 msgr2=0x7fd8641a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd85bfff640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8641044a0 0x7fd8641a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd85bfff640 1 -- 172.21.15.7:0/1940693575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd864106720 msgr2=0x7fd86419a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd85bfff640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd864106720 0x7fd86419a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd85bfff640 1 -- 172.21.15.7:0/1940693575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86419afd0 con 0x7fd864105d80 2024-04-21T08:48:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd86259c640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8641044a0 0x7fd8641a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd85bfff640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd864105d80 0x7fd8641a0bd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009230 tx=0x7fd84c02c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd85b7fe640 1 -- 172.21.15.7:0/1940693575 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84c043020 con 0x7fd864105d80 2024-04-21T08:48:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd868f85640 1 -- 172.21.15.7:0/1940693575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86419b260 con 0x7fd864105d80 2024-04-21T08:48:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd85b7fe640 1 -- 172.21.15.7:0/1940693575 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd84c02caf0 con 0x7fd864105d80 2024-04-21T08:48:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd85b7fe640 1 -- 172.21.15.7:0/1940693575 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84c003af0 con 0x7fd864105d80 2024-04-21T08:48:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.724+0000 7fd868f85640 1 -- 172.21.15.7:0/1940693575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8641a74e0 con 0x7fd864105d80 2024-04-21T08:48:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.726+0000 7fd85b7fe640 1 -- 172.21.15.7:0/1940693575 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd84c046020 con 0x7fd864105d80 2024-04-21T08:48:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd86bb71740 1 Processor -- start 2024-04-21T08:48:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd86bb71740 1 -- start start 2024-04-21T08:48:04.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b1b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b24540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 0x55fbe8b26e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd86bb71740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55fbe8a801e0 con 0x55fbe8b24a80 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd86bb71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fbe8a1d390 con 0x55fbe8b1b580 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd86bb71740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fbe8a7c1a0 con 0x55fbe8b1bea0 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd85bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 0x55fbe8b26e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd862d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b1b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd862d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b1b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:40614/0 (socket says 172.21.15.7:40614) 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd862d9d640 1 -- 172.21.15.7:0/2926680776 learned_addr learned my addr 172.21.15.7:0/2926680776 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd86259c640 1 --2- 172.21.15.7:0/2926680776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b24540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.727+0000 7fd85b7fe640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd8500a3d20 0x7fd8500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd85bfff640 1 -- 172.21.15.7:0/2926680776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b1bea0 msgr2=0x55fbe8b24540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd85bfff640 1 --2- 172.21.15.7:0/2926680776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b24540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd85bfff640 1 -- 172.21.15.7:0/2926680776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 msgr2=0x55fbe8b1b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd85bfff640 1 --2- 172.21.15.7:0/2926680776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b1b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd85bfff640 1 -- 172.21.15.7:0/2926680776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbe885c7d0 con 0x55fbe8b24a80 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd862d9d640 1 --2- 172.21.15.7:0/2926680776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b1b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd86259c640 1 --2- 172.21.15.7:0/2926680776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b24540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd862d9d640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd8500a3d20 0x7fd8500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd85b7fe640 1 -- 172.21.15.7:0/1940693575 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fd84c0f2c20 con 0x7fd864105d80 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd85bfff640 1 --2- 172.21.15.7:0/2926680776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 0x55fbe8b26e70 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0f0090 tx=0x7fd84c07e380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7497d5a06636a1bc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd858ff9640 1 -- 172.21.15.7:0/2926680776 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84c0be6b0 con 0x55fbe8b24a80 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd858ff9640 1 -- 172.21.15.7:0/2926680776 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd84c0befe0 con 0x55fbe8b24a80 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd858ff9640 1 -- 172.21.15.7:0/2926680776 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84c02cca0 con 0x55fbe8b24a80 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.728+0000 7fd862d9d640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd8500a3d20 0x7fd8500a61e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009fd0 tx=0x7fd85c004a40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 -- 172.21.15.7:0/2926680776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 msgr2=0x55fbe8b26e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 --2- 172.21.15.7:0/2926680776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 0x55fbe8b26e70 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0f0090 tx=0x7fd84c07e380 comp rx=0 tx=0).stop 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 -- 172.21.15.7:0/2926680776 shutdown_connections 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 --2- 172.21.15.7:0/2926680776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 0x55fbe8b26e70 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 --2- 172.21.15.7:0/2926680776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b24540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 --2- 172.21.15.7:0/2926680776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b1b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 -- 172.21.15.7:0/2926680776 >> 172.21.15.7:0/2926680776 conn(0x55fbe8b15cb0 msgr2=0x55fbe8b16090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 -- 172.21.15.7:0/2926680776 shutdown_connections 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 -- 172.21.15.7:0/2926680776 wait complete. 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 -- start start 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 Processor -- start 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b1b580 0x55fbe8b21ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b22020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 0x55fbe8b22560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55fbe8a801e0 con 0x55fbe8b24a80 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fbe8a392f0 con 0x55fbe8b1bea0 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.729+0000 7fd86bb71740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fbe8b27630 con 0x55fbe8b1b580 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd86bb71740 10 client.0 _ll_register_callbacks cb 0x55fbe8a46580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd862d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 0x55fbe8b22560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd862d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 0x55fbe8b22560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35144/0 (socket says 172.21.15.7:35144) 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd862d9d640 1 -- 172.21.15.7:0/3054264244 learned_addr learned my addr 172.21.15.7:0/3054264244 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd85bfff640 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b22020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd86259c640 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b1b580 0x55fbe8b21ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd862d9d640 1 -- 172.21.15.7:0/3054264244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b1b580 msgr2=0x55fbe8b21ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd862d9d640 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b1b580 0x55fbe8b21ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd862d9d640 1 -- 172.21.15.7:0/3054264244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1bea0 msgr2=0x55fbe8b22020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd862d9d640 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b22020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd862d9d640 1 -- 172.21.15.7:0/3054264244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbe87923f0 con 0x55fbe8b24a80 2024-04-21T08:48:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd85bfff640 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b22020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd862d9d640 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 0x55fbe8b22560 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd85c013af0 tx=0x7fd85c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce49b01ad66e1ce5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd849ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd849ffb640 1 -- 172.21.15.7:0/3054264244 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c028440 con 0x55fbe8b24a80 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd849ffb640 1 -- 172.21.15.7:0/3054264244 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fd85c01c410 con 0x55fbe8b24a80 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd86bb71740 1 -- 172.21.15.7:0/3054264244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd84c0befe0 con 0x55fbe8b24a80 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd849ffb640 1 -- 172.21.15.7:0/3054264244 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd85c01c610 con 0x55fbe8b24a80 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd86bb71740 10 client.16434 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.730+0000 7fd86bb71740 1 -- 172.21.15.7:0/3054264244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fbe87520f0 con 0x55fbe8b24a80 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.731+0000 7fd849ffb640 1 -- 172.21.15.7:0/3054264244 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd84c0befe0 con 0x55fbe8b24a80 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.731+0000 7fd849ffb640 1 -- 172.21.15.7:0/3054264244 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fd85c026070 con 0x55fbe8b24a80 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.731+0000 7fd86bb71740 10 client.16434 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.731+0000 7fd86bb71740 10 client.16434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.731+0000 7fd86bb71740 20 client.16434 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.731+0000 7fd86bb71740 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fbe8b347e0 0x55fbe8b36bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.731+0000 7fd86bb71740 4 client.16434 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.731+0000 7fd86bb71740 1 -- 172.21.15.7:0/3054264244 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbe8a8dc00 con 0x55fbe8b347e0 2024-04-21T08:48:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.731+0000 7fd86259c640 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fbe8b347e0 0x55fbe8b36bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.731+0000 7fd849ffb640 10 client.16434 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.731+0000 7fd86259c640 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fbe8b347e0 0x55fbe8b36bd0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86557be3931d8f99 server_cookie=2780843c2ed19929 in_seq=0 out_seq=0 2024-04-21T08:48:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.735+0000 7fd849ffb640 1 -- 172.21.15.7:0/3054264244 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fbe8a8dc00 con 0x55fbe8b347e0 2024-04-21T08:48:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.735+0000 7fd849ffb640 10 client.16434 handle_command_reply: tid=0 2024-04-21T08:48:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.735+0000 7fd86bb71740 1 client.16434 shutdown 2024-04-21T08:48:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd86bb71740 1 -- 172.21.15.7:0/3054264244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 msgr2=0x55fbe8b22560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd86bb71740 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 0x55fbe8b22560 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd85c013af0 tx=0x7fd85c004ab0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd86bb71740 1 -- 172.21.15.7:0/3054264244 shutdown_connections 2024-04-21T08:48:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd86bb71740 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fbe8b347e0 0x55fbe8b36bd0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd86bb71740 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b24a80 0x55fbe8b22560 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd86bb71740 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b22020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd849ffb640 0 client.16434 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd86bb71740 1 --2- 172.21.15.7:0/3054264244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b1b580 0x55fbe8b21ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd86bb71740 1 -- 172.21.15.7:0/3054264244 >> 172.21.15.7:0/3054264244 conn(0x55fbe8b15cb0 msgr2=0x55fbe8b17a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd86bb71740 1 -- 172.21.15.7:0/3054264244 shutdown_connections 2024-04-21T08:48:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd86bb71740 1 -- 172.21.15.7:0/3054264244 wait complete. 2024-04-21T08:48:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.736+0000 7fd86bb71740 20 client.16434 trim_cache size 0 max 16384 2024-04-21T08:48:04.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.753+0000 7fd86bb71740 1 Processor -- start 2024-04-21T08:48:04.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd86bb71740 1 -- start start 2024-04-21T08:48:04.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b24210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b30370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b24a80 0x55fbe8b20c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd86bb71740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641af6f0 con 0x55fbe8b1bea0 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd86bb71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641b1f70 con 0x55fbe8b1b580 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd86bb71740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641afd60 con 0x55fbe8b24a80 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd85bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b30370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd86259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b24210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd85bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b30370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35158/0 (socket says 172.21.15.7:35158) 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd85bfff640 1 -- 172.21.15.7:0/1312410978 learned_addr learned my addr 172.21.15.7:0/1312410978 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd862d9d640 1 --2- 172.21.15.7:0/1312410978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b24a80 0x55fbe8b20c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd85bfff640 1 -- 172.21.15.7:0/1312410978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b24a80 msgr2=0x55fbe8b20c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd85bfff640 1 --2- 172.21.15.7:0/1312410978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b24a80 0x55fbe8b20c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd85bfff640 1 -- 172.21.15.7:0/1312410978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 msgr2=0x55fbe8b24210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd85bfff640 1 --2- 172.21.15.7:0/1312410978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b24210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd85bfff640 1 -- 172.21.15.7:0/1312410978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbe8b1bc90 con 0x55fbe8b1bea0 2024-04-21T08:48:04.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd85bfff640 1 --2- 172.21.15.7:0/1312410978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b30370 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0bad40 tx=0x7fd84c0b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b27cf445892a6de server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.754+0000 7fd84bfff640 1 -- 172.21.15.7:0/1312410978 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84c0f5070 con 0x55fbe8b1bea0 2024-04-21T08:48:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd84bfff640 1 -- 172.21.15.7:0/1312410978 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd84c0befe0 con 0x55fbe8b1bea0 2024-04-21T08:48:04.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd84bfff640 1 -- 172.21.15.7:0/1312410978 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84c03bb80 con 0x55fbe8b1bea0 2024-04-21T08:48:04.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd86bb71740 1 -- 172.21.15.7:0/1312410978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 msgr2=0x55fbe8b30370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd86bb71740 1 --2- 172.21.15.7:0/1312410978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b30370 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0bad40 tx=0x7fd84c0b9040 comp rx=0 tx=0).stop 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd86bb71740 1 -- 172.21.15.7:0/1312410978 shutdown_connections 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd86bb71740 1 --2- 172.21.15.7:0/1312410978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b24a80 0x55fbe8b20c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd86bb71740 1 --2- 172.21.15.7:0/1312410978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b30370 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd86bb71740 1 --2- 172.21.15.7:0/1312410978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b24210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd86bb71740 1 -- 172.21.15.7:0/1312410978 >> 172.21.15.7:0/1312410978 conn(0x55fbe8b15cb0 msgr2=0x55fbe8b144e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd86bb71740 1 -- 172.21.15.7:0/1312410978 shutdown_connections 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd86bb71740 1 -- 172.21.15.7:0/1312410978 wait complete. 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.755+0000 7fd86bb71740 1 -- start start 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd86bb71740 1 Processor -- start 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b2caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b27e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd86bb71740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b24a80 0x55fbe8b28380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd86bb71740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641b0430 con 0x55fbe8b1bea0 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd86bb71740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641b18a0 con 0x55fbe8b1b580 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd86bb71740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641b0b00 con 0x55fbe8b24a80 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd86bb71740 10 client.0 _ll_register_callbacks cb 0x7fd85c0137a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd85bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b27e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd85bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b27e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35166/0 (socket says 172.21.15.7:35166) 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd85bfff640 1 -- 172.21.15.7:0/2278736737 learned_addr learned my addr 172.21.15.7:0/2278736737 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd86259c640 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b2caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd862d9d640 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b24a80 0x55fbe8b28380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd85bfff640 1 -- 172.21.15.7:0/2278736737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b24a80 msgr2=0x55fbe8b28380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd85bfff640 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b24a80 0x55fbe8b28380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd85bfff640 1 -- 172.21.15.7:0/2278736737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 msgr2=0x55fbe8b2caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd85bfff640 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b2caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd85bfff640 1 -- 172.21.15.7:0/2278736737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbe8b28970 con 0x55fbe8b1bea0 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd85bfff640 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b27e40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0df0d0 tx=0x7fd84c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5caf22416928d11a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd849ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.756+0000 7fd849ffb640 1 -- 172.21.15.7:0/2278736737 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84c0f5070 con 0x55fbe8b1bea0 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd849ffb640 1 -- 172.21.15.7:0/2278736737 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fd84c0d2fc0 con 0x55fbe8b1bea0 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd86bb71740 1 -- 172.21.15.7:0/2278736737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd84c0befe0 con 0x55fbe8b1bea0 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd849ffb640 1 -- 172.21.15.7:0/2278736737 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd84c03aa70 con 0x55fbe8b1bea0 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd86bb71740 10 client.16440 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd86bb71740 1 -- 172.21.15.7:0/2278736737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fbe8b9ab40 con 0x55fbe8b1bea0 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd849ffb640 1 -- 172.21.15.7:0/2278736737 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd84c0befe0 con 0x55fbe8b1bea0 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd849ffb640 1 -- 172.21.15.7:0/2278736737 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fd84c0f8070 con 0x55fbe8b1bea0 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd86bb71740 10 client.16440 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd86bb71740 10 client.16440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd86bb71740 20 client.16440 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd86bb71740 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fbe8b9ae80 0x55fbe8b9d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd86bb71740 4 client.16440 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd86bb71740 1 -- 172.21.15.7:0/2278736737 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fbe8a87a70 con 0x55fbe8b9ae80 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.757+0000 7fd86259c640 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fbe8b9ae80 0x55fbe8b9d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.758+0000 7fd849ffb640 10 client.16440 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.758+0000 7fd86259c640 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fbe8b9ae80 0x55fbe8b9d270 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ba47d94c9d10115 server_cookie=da3d25f0e01abb03 in_seq=0 out_seq=0 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd849ffb640 1 -- 172.21.15.7:0/2278736737 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55fbe8a87a70 con 0x55fbe8b9ae80 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd849ffb640 10 client.16440 handle_command_reply: tid=0 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 1 client.16440 shutdown 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 1 -- 172.21.15.7:0/2278736737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 msgr2=0x55fbe8b27e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b27e40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0df0d0 tx=0x7fd84c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 1 -- 172.21.15.7:0/2278736737 shutdown_connections 2024-04-21T08:48:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fbe8b9ae80 0x55fbe8b9d270 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fbe8b24a80 0x55fbe8b28380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fbe8b1bea0 0x55fbe8b27e40 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd849ffb640 0 client.16440 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 1 --2- 172.21.15.7:0/2278736737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fbe8b1b580 0x55fbe8b2caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 1 -- 172.21.15.7:0/2278736737 >> 172.21.15.7:0/2278736737 conn(0x55fbe8b15cb0 msgr2=0x55fbe8b21aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 1 -- 172.21.15.7:0/2278736737 shutdown_connections 2024-04-21T08:48:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 1 -- 172.21.15.7:0/2278736737 wait complete. 2024-04-21T08:48:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd86bb71740 20 client.16440 trim_cache size 0 max 16384 2024-04-21T08:48:04.765 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:48:04.766 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (4 inodes in the stack)", 2024-04-21T08:48:04.766 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:48:04.766 INFO:teuthology.orchestra.run.smithi007.stdout: "a8ceae46-67db-44e4-95b3-be9792f1a693": { 2024-04-21T08:48:04.766 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:48:04.766 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "a8ceae46-67db-44e4-95b3-be9792f1a693", 2024-04-21T08:48:04.766 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:48:04.766 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:48:04.766 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:48:04.766 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:48:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd868f85640 1 -- 172.21.15.7:0/1940693575 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd8500a3d20 msgr2=0x7fd8500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd868f85640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd8500a3d20 0x7fd8500a61e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009fd0 tx=0x7fd85c004a40 comp rx=0 tx=0).stop 2024-04-21T08:48:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd868f85640 1 -- 172.21.15.7:0/1940693575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd864105d80 msgr2=0x7fd8641a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.760+0000 7fd868f85640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd864105d80 0x7fd8641a0bd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009230 tx=0x7fd84c02c710 comp rx=0 tx=0).stop 2024-04-21T08:48:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.761+0000 7fd868f85640 1 -- 172.21.15.7:0/1940693575 shutdown_connections 2024-04-21T08:48:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.761+0000 7fd868f85640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd864106720 0x7fd86419a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.761+0000 7fd868f85640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd864105d80 0x7fd8641a0bd0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.761+0000 7fd868f85640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd8500a3d20 0x7fd8500a61e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.761+0000 7fd868f85640 1 --2- 172.21.15.7:0/1940693575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8641044a0 0x7fd8641a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.761+0000 7fd868f85640 1 -- 172.21.15.7:0/1940693575 >> 172.21.15.7:0/1940693575 conn(0x7fd8640ffc40 msgr2=0x7fd864100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.761+0000 7fd868f85640 1 -- 172.21.15.7:0/1940693575 shutdown_connections 2024-04-21T08:48:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:04.761+0000 7fd868f85640 1 -- 172.21.15.7:0/1940693575 wait complete. 2024-04-21T08:48:04.771 INFO:tasks.cephfs.filesystem:scrub status for tag:a8ceae46-67db-44e4-95b3-be9792f1a693 - {'path': '/', 'tag': 'a8ceae46-67db-44e4-95b3-be9792f1a693', 'options': 'recursive,force'} 2024-04-21T08:48:04.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:04 smithi077 ceph-mon[31615]: pgmap v105: 129 pgs: 129 active+clean; 329 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 1.72k op/s 2024-04-21T08:48:04.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:04 smithi007 ceph-mon[28477]: pgmap v105: 129 pgs: 129 active+clean; 329 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 1.72k op/s 2024-04-21T08:48:05.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:04 smithi016 ceph-mon[31259]: pgmap v105: 129 pgs: 129 active+clean; 329 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 1.72k op/s 2024-04-21T08:48:05.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:05 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:48:05.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:05 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:05.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:05 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:48:05.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:05 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:48:05.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:05 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:05.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:05 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:48:06.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:05 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:48:06.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:05 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:06.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:05 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:48:06.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:06 smithi077 ceph-mon[31615]: pgmap v106: 129 pgs: 129 active+clean; 329 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 6.1 MiB/s wr, 1.71k op/s 2024-04-21T08:48:06.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:06 smithi007 ceph-mon[28477]: pgmap v106: 129 pgs: 129 active+clean; 329 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 6.1 MiB/s wr, 1.71k op/s 2024-04-21T08:48:07.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:06 smithi016 ceph-mon[31259]: pgmap v106: 129 pgs: 129 active+clean; 329 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 6.1 MiB/s wr, 1.71k op/s 2024-04-21T08:48:08.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:08 smithi077 ceph-mon[31615]: pgmap v107: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 2.07k op/s 2024-04-21T08:48:08.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:08 smithi007 ceph-mon[28477]: pgmap v107: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 2.07k op/s 2024-04-21T08:48:09.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:08 smithi016 ceph-mon[31259]: pgmap v107: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 2.07k op/s 2024-04-21T08:48:10.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:10 smithi077 ceph-mon[31615]: pgmap v108: 129 pgs: 129 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 2.90k op/s 2024-04-21T08:48:10.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:10 smithi007 ceph-mon[28477]: pgmap v108: 129 pgs: 129 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 2.90k op/s 2024-04-21T08:48:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:10 smithi016 ceph-mon[31259]: pgmap v108: 129 pgs: 129 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 2.90k op/s 2024-04-21T08:48:11.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:11 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:11.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:11 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:11 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:12.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:12 smithi077 ceph-mon[31615]: pgmap v109: 129 pgs: 129 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.6 MiB/s wr, 2.90k op/s 2024-04-21T08:48:12.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:12 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:48:12.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:12 smithi007 ceph-mon[28477]: pgmap v109: 129 pgs: 129 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.6 MiB/s wr, 2.90k op/s 2024-04-21T08:48:12.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:12 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:48:13.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:12 smithi016 ceph-mon[31259]: pgmap v109: 129 pgs: 129 active+clean; 341 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.6 MiB/s wr, 2.90k op/s 2024-04-21T08:48:13.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:12 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:48:14.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:14 smithi077 ceph-mon[31615]: pgmap v110: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 7.1 MiB/s wr, 4.24k op/s 2024-04-21T08:48:14.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:14 smithi007 ceph-mon[28477]: pgmap v110: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 7.1 MiB/s wr, 4.24k op/s 2024-04-21T08:48:15.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:14 smithi016 ceph-mon[31259]: pgmap v110: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 7.1 MiB/s wr, 4.24k op/s 2024-04-21T08:48:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.709+0000 7f8c57806640 1 -- 172.21.15.7:0/2332386703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c501031c0 msgr2=0x7f8c501035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.709+0000 7f8c57806640 1 --2- 172.21.15.7:0/2332386703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c501031c0 0x7f8c501035a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009900 tx=0x7f8c4c02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.709+0000 7f8c57806640 1 -- 172.21.15.7:0/2332386703 shutdown_connections 2024-04-21T08:48:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.709+0000 7f8c57806640 1 --2- 172.21.15.7:0/2332386703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c5010c590 0x7f8c5010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.709+0000 7f8c57806640 1 --2- 172.21.15.7:0/2332386703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c50103ae0 0x7f8c5010bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.709+0000 7f8c57806640 1 --2- 172.21.15.7:0/2332386703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c501031c0 0x7f8c501035a0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.709+0000 7f8c57806640 1 -- 172.21.15.7:0/2332386703 >> 172.21.15.7:0/2332386703 conn(0x7f8c50077f70 msgr2=0x7f8c50078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.709+0000 7f8c57806640 1 -- 172.21.15.7:0/2332386703 shutdown_connections 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.709+0000 7f8c57806640 1 -- 172.21.15.7:0/2332386703 wait complete. 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c57806640 1 Processor -- start 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c57806640 1 -- start start 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c57806640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c501031c0 0x7f8c50100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c57806640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c50103ae0 0x7f8c50101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c57806640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5010c590 0x7f8c50101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c57806640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501121e0 con 0x7f8c50103ae0 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c57806640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c50112060 con 0x7f8c5010c590 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c57806640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8c50112360 con 0x7f8c501031c0 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c54da1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c50103ae0 0x7f8c50101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c54da1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c50103ae0 0x7f8c50101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48364/0 (socket says 172.21.15.7:48364) 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c54da1640 1 -- 172.21.15.7:0/1963850215 learned_addr learned my addr 172.21.15.7:0/1963850215 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c55da3640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5010c590 0x7f8c50101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c555a2640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c501031c0 0x7f8c50100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c54da1640 1 -- 172.21.15.7:0/1963850215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c501031c0 msgr2=0x7f8c50100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c54da1640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c501031c0 0x7f8c50100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c54da1640 1 -- 172.21.15.7:0/1963850215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5010c590 msgr2=0x7f8c50101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.710+0000 7f8c54da1640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5010c590 0x7f8c50101890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.711+0000 7f8c54da1640 1 -- 172.21.15.7:0/1963850215 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c501a6cc0 con 0x7f8c50103ae0 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.711+0000 7f8c55da3640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5010c590 0x7f8c50101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.711+0000 7f8c54da1640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c50103ae0 0x7f8c50101350 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000ea60 tx=0x7f8c4000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.711+0000 7f8c467fc640 1 -- 172.21.15.7:0/1963850215 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4000cde0 con 0x7f8c50103ae0 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.711+0000 7f8c467fc640 1 -- 172.21.15.7:0/1963850215 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c400035f0 con 0x7f8c50103ae0 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.711+0000 7f8c57806640 1 -- 172.21.15.7:0/1963850215 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c501a6fb0 con 0x7f8c50103ae0 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.711+0000 7f8c467fc640 1 -- 172.21.15.7:0/1963850215 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c40010690 con 0x7f8c50103ae0 2024-04-21T08:48:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.711+0000 7f8c57806640 1 -- 172.21.15.7:0/1963850215 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c501a7470 con 0x7f8c50103ae0 2024-04-21T08:48:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.714+0000 7f8c467fc640 1 -- 172.21.15.7:0/1963850215 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f8c400037a0 con 0x7f8c50103ae0 2024-04-21T08:48:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c5a3f4740 1 Processor -- start 2024-04-21T08:48:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c5a3f4740 1 -- start start 2024-04-21T08:48:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b45948d0 0x55b8b4593620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b4593b60 0x55b8b4593fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b459d0e0 0x55b8b459f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b8b44f88d0 con 0x55b8b4593b60 2024-04-21T08:48:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b8b4495e80 con 0x55b8b459d0e0 2024-04-21T08:48:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b8b44f4890 con 0x55b8b45948d0 2024-04-21T08:48:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c467fc640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8c3c0a3d10 0x7f8c3c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c54da1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b459d0e0 0x55b8b459f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c555a2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b4593b60 0x55b8b4593fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c555a2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b4593b60 0x55b8b4593fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48378/0 (socket says 172.21.15.7:48378) 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c54da1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b459d0e0 0x55b8b459f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:54494/0 (socket says 172.21.15.7:54494) 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c555a2640 1 -- 172.21.15.7:0/1452938901 learned_addr learned my addr 172.21.15.7:0/1452938901 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c55da3640 1 --2- 172.21.15.7:0/1452938901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b45948d0 0x55b8b4593620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c467fc640 1 -- 172.21.15.7:0/1963850215 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f8c400d0ee0 con 0x7f8c50103ae0 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c55da3640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8c3c0a3d10 0x7f8c3c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c555a2640 1 -- 172.21.15.7:0/1452938901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b45948d0 msgr2=0x55b8b4593620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c555a2640 1 --2- 172.21.15.7:0/1452938901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b45948d0 0x55b8b4593620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c555a2640 1 -- 172.21.15.7:0/1452938901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b459d0e0 msgr2=0x55b8b459f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c555a2640 1 --2- 172.21.15.7:0/1452938901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b459d0e0 0x55b8b459f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.715+0000 7f8c555a2640 1 -- 172.21.15.7:0/1452938901 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8b44b7320 con 0x55b8b4593b60 2024-04-21T08:48:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.716+0000 7f8c555a2640 1 --2- 172.21.15.7:0/1452938901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b4593b60 0x55b8b4593fe0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009900 tx=0x7f8c4c0095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a112c9132c6677c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.716+0000 7f8c55da3640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8c3c0a3d10 0x7f8c3c0a61d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c48005c40 tx=0x7f8c48009500 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.716+0000 7f8c3b7fe640 1 -- 172.21.15.7:0/1452938901 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c004040 con 0x55b8b4593b60 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.716+0000 7f8c3b7fe640 1 -- 172.21.15.7:0/1452938901 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c4c0054f0 con 0x55b8b4593b60 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.716+0000 7f8c3b7fe640 1 -- 172.21.15.7:0/1452938901 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c02ce50 con 0x55b8b4593b60 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.717+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/1452938901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b4593b60 msgr2=0x55b8b4593fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.717+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/1452938901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b4593b60 0x55b8b4593fe0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009900 tx=0x7f8c4c0095f0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.717+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/1452938901 shutdown_connections 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.717+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/1452938901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b459d0e0 0x55b8b459f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.717+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/1452938901 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b4593b60 0x55b8b4593fe0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.717+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/1452938901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b45948d0 0x55b8b4593620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.717+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/1452938901 >> 172.21.15.7:0/1452938901 conn(0x55b8b455ac90 msgr2=0x55b8b458e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.717+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/1452938901 shutdown_connections 2024-04-21T08:48:15.722 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.717+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/1452938901 wait complete. 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.717+0000 7f8c5a3f4740 1 -- start start 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.717+0000 7f8c5a3f4740 1 Processor -- start 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 0x55b8b459ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b4599d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b459d0e0 0x55b8b459a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b8b459fdb0 con 0x55b8b45948d0 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b8b4480eb0 con 0x55b8b4593b60 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b8b459ff30 con 0x55b8b459d0e0 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c5a3f4740 10 client.0 _ll_register_callbacks cb 0x55b8b44adb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c54da1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b4599d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c54da1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b4599d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48380/0 (socket says 172.21.15.7:48380) 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c54da1640 1 -- 172.21.15.7:0/3525686845 learned_addr learned my addr 172.21.15.7:0/3525686845 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c555a2640 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 0x55b8b459ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c55da3640 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b459d0e0 0x55b8b459a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c54da1640 1 -- 172.21.15.7:0/3525686845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b459d0e0 msgr2=0x55b8b459a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c54da1640 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b459d0e0 0x55b8b459a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c54da1640 1 -- 172.21.15.7:0/3525686845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 msgr2=0x55b8b459ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c54da1640 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 0x55b8b459ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c54da1640 1 -- 172.21.15.7:0/3525686845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8b420b210 con 0x55b8b45948d0 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c555a2640 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 0x55b8b459ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c54da1640 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b4599d90 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000d010 tx=0x7f8c4009d1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f269d3da3acce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c38ff9640 1 -- 172.21.15.7:0/3525686845 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c400a56a0 con 0x55b8b45948d0 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.718+0000 7f8c38ff9640 1 -- 172.21.15.7:0/3525686845 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f8c400b10c0 con 0x55b8b45948d0 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3525686845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c4c0054f0 con 0x55b8b45948d0 2024-04-21T08:48:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c38ff9640 1 -- 172.21.15.7:0/3525686845 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8c400ce460 con 0x55b8b45948d0 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c5a3f4740 10 client.16452 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3525686845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b8b42d7620 con 0x55b8b45948d0 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c38ff9640 1 -- 172.21.15.7:0/3525686845 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c4c0054f0 con 0x55b8b45948d0 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c38ff9640 1 -- 172.21.15.7:0/3525686845 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f8c4009bf10 con 0x55b8b45948d0 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c5a3f4740 10 client.16452 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c5a3f4740 10 client.16452 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c5a3f4740 20 client.16452 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b8b45ad000 0x55b8b45af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c5a3f4740 4 client.16452 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3525686845 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8b450c880 con 0x55b8b45ad000 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.719+0000 7f8c555a2640 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b8b45ad000 0x55b8b45af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.720+0000 7f8c38ff9640 10 client.16452 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.720+0000 7f8c555a2640 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b8b45ad000 0x55b8b45af3f0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad09b9f9fefd6ef3 server_cookie=1b177c1f594e5d0c in_seq=0 out_seq=0 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.721+0000 7f8c38ff9640 1 -- 172.21.15.7:0/3525686845 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b8b450c880 con 0x55b8b45ad000 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.721+0000 7f8c38ff9640 10 client.16452 handle_command_reply: tid=0 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.721+0000 7f8c5a3f4740 1 client.16452 shutdown 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3525686845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 msgr2=0x55b8b4599d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b4599d90 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000d010 tx=0x7f8c4009d1f0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3525686845 shutdown_connections 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b8b45ad000 0x55b8b45af3f0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b459d0e0 0x55b8b459a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b4599d90 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c38ff9640 0 client.16452 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/3525686845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 0x55b8b459ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3525686845 >> 172.21.15.7:0/3525686845 conn(0x55b8b455ac90 msgr2=0x55b8b45905a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3525686845 shutdown_connections 2024-04-21T08:48:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3525686845 wait complete. 2024-04-21T08:48:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.722+0000 7f8c5a3f4740 20 client.16452 trim_cache size 0 max 16384 2024-04-21T08:48:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c5a3f4740 1 Processor -- start 2024-04-21T08:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c5a3f4740 1 -- start start 2024-04-21T08:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 0x55b8b45a8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b459a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b459d0e0 0x55b8b459c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501af650 con 0x55b8b45948d0 2024-04-21T08:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501b1ed0 con 0x55b8b4593b60 2024-04-21T08:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501afcc0 con 0x55b8b459d0e0 2024-04-21T08:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c54da1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b459a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c555a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 0x55b8b45a8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c54da1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b459a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48392/0 (socket says 172.21.15.7:48392) 2024-04-21T08:48:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.738+0000 7f8c54da1640 1 -- 172.21.15.7:0/3119421615 learned_addr learned my addr 172.21.15.7:0/3119421615 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c55da3640 1 --2- 172.21.15.7:0/3119421615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b459d0e0 0x55b8b459c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c555a2640 1 -- 172.21.15.7:0/3119421615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b459d0e0 msgr2=0x55b8b459c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c555a2640 1 --2- 172.21.15.7:0/3119421615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b459d0e0 0x55b8b459c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c555a2640 1 -- 172.21.15.7:0/3119421615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 msgr2=0x55b8b459a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c555a2640 1 --2- 172.21.15.7:0/3119421615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b459a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c555a2640 1 -- 172.21.15.7:0/3119421615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8b4592e30 con 0x55b8b4593b60 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c55da3640 1 --2- 172.21.15.7:0/3119421615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b459d0e0 0x55b8b459c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c54da1640 1 --2- 172.21.15.7:0/3119421615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b459a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c555a2640 1 --2- 172.21.15.7:0/3119421615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 0x55b8b45a8cb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c02c210 tx=0x7f8c4c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d2ff56ffa797ac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c3affd640 1 -- 172.21.15.7:0/3119421615 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c0042c0 con 0x55b8b4593b60 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c3affd640 1 -- 172.21.15.7:0/3119421615 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c4c004460 con 0x55b8b4593b60 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.739+0000 7f8c3affd640 1 -- 172.21.15.7:0/3119421615 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c004ae0 con 0x55b8b4593b60 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.740+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3119421615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 msgr2=0x55b8b45a8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.740+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/3119421615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 0x55b8b45a8cb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c02c210 tx=0x7f8c4c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.740+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3119421615 shutdown_connections 2024-04-21T08:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.740+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/3119421615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b459d0e0 0x55b8b459c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.740+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/3119421615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b45948d0 0x55b8b459a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.740+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/3119421615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b4593b60 0x55b8b45a8cb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.740+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3119421615 >> 172.21.15.7:0/3119421615 conn(0x55b8b455ac90 msgr2=0x55b8b4598680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.740+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3119421615 shutdown_connections 2024-04-21T08:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.740+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/3119421615 wait complete. 2024-04-21T08:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.740+0000 7f8c5a3f4740 1 -- start start 2024-04-21T08:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.740+0000 7f8c5a3f4740 1 Processor -- start 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b4593b60 0x55b8b45a5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b45948d0 0x55b8b45966a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c5a3f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b459d0e0 0x55b8b4596be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501b0390 con 0x55b8b459d0e0 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501b1800 con 0x55b8b45948d0 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c5a3f4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8c501b0a60 con 0x55b8b4593b60 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c5a3f4740 10 client.0 _ll_register_callbacks cb 0x7f8c40004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c54da1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b45948d0 0x55b8b45966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c555a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b4593b60 0x55b8b45a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c55da3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b459d0e0 0x55b8b4596be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c54da1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b45948d0 0x55b8b45966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:54508/0 (socket says 172.21.15.7:54508) 2024-04-21T08:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c555a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b4593b60 0x55b8b45a5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:44242/0 (socket says 172.21.15.7:44242) 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c54da1640 1 -- 172.21.15.7:0/2046584680 learned_addr learned my addr 172.21.15.7:0/2046584680 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c555a2640 1 -- 172.21.15.7:0/2046584680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b45948d0 msgr2=0x55b8b45966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c555a2640 1 --2- 172.21.15.7:0/2046584680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b45948d0 0x55b8b45966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c555a2640 1 -- 172.21.15.7:0/2046584680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b459d0e0 msgr2=0x55b8b4596be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c555a2640 1 --2- 172.21.15.7:0/2046584680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b459d0e0 0x55b8b4596be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c555a2640 1 -- 172.21.15.7:0/2046584680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8b45972a0 con 0x55b8b4593b60 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c555a2640 1 --2- 172.21.15.7:0/2046584680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b4593b60 0x55b8b45a5490 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009900 tx=0x7f8c4c02cb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8bb403417fde54c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.741+0000 7f8c38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c38ff9640 1 -- 172.21.15.7:0/2046584680 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c005450 con 0x55b8b4593b60 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c38ff9640 1 -- 172.21.15.7:0/2046584680 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f8c4c0055f0 con 0x55b8b4593b60 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/2046584680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c4c004460 con 0x55b8b4593b60 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c38ff9640 1 -- 172.21.15.7:0/2046584680 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8c4c004760 con 0x55b8b4593b60 2024-04-21T08:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c5a3f4740 10 client.26023 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/2046584680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b8b4613360 con 0x55b8b4593b60 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c38ff9640 1 -- 172.21.15.7:0/2046584680 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c4c004460 con 0x55b8b4593b60 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c38ff9640 1 -- 172.21.15.7:0/2046584680 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f8c4c003af0 con 0x55b8b4593b60 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c5a3f4740 10 client.26023 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c5a3f4740 10 client.26023 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c5a3f4740 20 client.26023 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/2046584680 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b8b4613690 0x55b8b4615a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c5a3f4740 4 client.26023 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/2046584680 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b8b44983d0 con 0x55b8b4613690 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.742+0000 7f8c54da1640 1 --2- 172.21.15.7:0/2046584680 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b8b4613690 0x55b8b4615a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.743+0000 7f8c54da1640 1 --2- 172.21.15.7:0/2046584680 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b8b4613690 0x55b8b4615a80 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4064495deebad46 server_cookie=4da6345e7ed8d510 in_seq=0 out_seq=0 2024-04-21T08:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:15.743+0000 7f8c38ff9640 10 client.26023 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:16 smithi077 ceph-mon[31615]: pgmap v111: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 2.55k op/s 2024-04-21T08:48:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:16 smithi007 ceph-mon[28477]: pgmap v111: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 2.55k op/s 2024-04-21T08:48:17.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:16 smithi016 ceph-mon[31259]: pgmap v111: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 2.55k op/s 2024-04-21T08:48:18.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:18 smithi077 ceph-mon[31615]: pgmap v112: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 2.55k op/s 2024-04-21T08:48:18.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:18 smithi007 ceph-mon[28477]: pgmap v112: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 2.55k op/s 2024-04-21T08:48:19.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:18 smithi016 ceph-mon[31259]: pgmap v112: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 2.55k op/s 2024-04-21T08:48:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.331+0000 7f8c38ff9640 1 -- 172.21.15.7:0/2046584680 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b8b44983d0 con 0x55b8b4613690 2024-04-21T08:48:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.331+0000 7f8c38ff9640 10 client.26023 handle_command_reply: tid=0 2024-04-21T08:48:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.331+0000 7f8c5a3f4740 1 client.26023 shutdown 2024-04-21T08:48:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/2046584680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b4593b60 msgr2=0x55b8b45a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/2046584680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b4593b60 0x55b8b45a5490 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009900 tx=0x7f8c4c02cb60 comp rx=0 tx=0).stop 2024-04-21T08:48:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/2046584680 shutdown_connections 2024-04-21T08:48:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/2046584680 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b8b4613690 0x55b8b4615a80 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/2046584680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b8b459d0e0 0x55b8b4596be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/2046584680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b8b45948d0 0x55b8b45966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c5a3f4740 1 --2- 172.21.15.7:0/2046584680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b8b4593b60 0x55b8b45a5490 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/2046584680 >> 172.21.15.7:0/2046584680 conn(0x55b8b455ac90 msgr2=0x55b8b458fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c38ff9640 0 client.26023 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/2046584680 shutdown_connections 2024-04-21T08:48:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c5a3f4740 1 -- 172.21.15.7:0/2046584680 wait complete. 2024-04-21T08:48:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.332+0000 7f8c5a3f4740 20 client.26023 trim_cache size 0 max 16384 2024-04-21T08:48:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:48:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:48:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:48:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.333+0000 7f8c57806640 1 -- 172.21.15.7:0/1963850215 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8c3c0a3d10 msgr2=0x7f8c3c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.333+0000 7f8c57806640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8c3c0a3d10 0x7f8c3c0a61d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c48005c40 tx=0x7f8c48009500 comp rx=0 tx=0).stop 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.333+0000 7f8c57806640 1 -- 172.21.15.7:0/1963850215 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c50103ae0 msgr2=0x7f8c50101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.333+0000 7f8c57806640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c50103ae0 0x7f8c50101350 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000ea60 tx=0x7f8c4000ef20 comp rx=0 tx=0).stop 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.334+0000 7f8c57806640 1 -- 172.21.15.7:0/1963850215 shutdown_connections 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.334+0000 7f8c57806640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5010c590 0x7f8c50101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.334+0000 7f8c57806640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c50103ae0 0x7f8c50101350 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.334+0000 7f8c57806640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8c3c0a3d10 0x7f8c3c0a61d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.334+0000 7f8c57806640 1 --2- 172.21.15.7:0/1963850215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c501031c0 0x7f8c50100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.334+0000 7f8c57806640 1 -- 172.21.15.7:0/1963850215 >> 172.21.15.7:0/1963850215 conn(0x7f8c50077f70 msgr2=0x7f8c500ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.334+0000 7f8c57806640 1 -- 172.21.15.7:0/1963850215 shutdown_connections 2024-04-21T08:48:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:19.334+0000 7f8c57806640 1 -- 172.21.15.7:0/1963850215 wait complete. 2024-04-21T08:48:20.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:20 smithi077 ceph-mon[31615]: pgmap v113: 129 pgs: 129 active+clean; 392 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.1 MiB/s wr, 2.79k op/s 2024-04-21T08:48:20.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:20 smithi007 ceph-mon[28477]: pgmap v113: 129 pgs: 129 active+clean; 392 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.1 MiB/s wr, 2.79k op/s 2024-04-21T08:48:21.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:20 smithi016 ceph-mon[31259]: pgmap v113: 129 pgs: 129 active+clean; 392 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.1 MiB/s wr, 2.79k op/s 2024-04-21T08:48:22.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:22 smithi077 ceph-mon[31615]: pgmap v114: 129 pgs: 129 active+clean; 392 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.8 MiB/s wr, 1.95k op/s 2024-04-21T08:48:22.855 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:22 smithi016 ceph-mon[31259]: pgmap v114: 129 pgs: 129 active+clean; 392 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.8 MiB/s wr, 1.95k op/s 2024-04-21T08:48:22.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:22 smithi007 ceph-mon[28477]: pgmap v114: 129 pgs: 129 active+clean; 392 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.8 MiB/s wr, 1.95k op/s 2024-04-21T08:48:24.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:24 smithi077 ceph-mon[31615]: pgmap v115: 129 pgs: 129 active+clean; 365 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.3 MiB/s wr, 2.63k op/s 2024-04-21T08:48:24.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:24 smithi007 ceph-mon[28477]: pgmap v115: 129 pgs: 129 active+clean; 365 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.3 MiB/s wr, 2.63k op/s 2024-04-21T08:48:25.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:24 smithi016 ceph-mon[31259]: pgmap v115: 129 pgs: 129 active+clean; 365 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.3 MiB/s wr, 2.63k op/s 2024-04-21T08:48:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:26 smithi077 ceph-mon[31615]: pgmap v116: 129 pgs: 129 active+clean; 365 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.9 MiB/s wr, 1.28k op/s 2024-04-21T08:48:26.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:26 smithi007 ceph-mon[28477]: pgmap v116: 129 pgs: 129 active+clean; 365 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.9 MiB/s wr, 1.28k op/s 2024-04-21T08:48:27.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:26 smithi016 ceph-mon[31259]: pgmap v116: 129 pgs: 129 active+clean; 365 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.9 MiB/s wr, 1.28k op/s 2024-04-21T08:48:28.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:28 smithi007 ceph-mon[28477]: pgmap v117: 129 pgs: 129 active+clean; 365 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.9 MiB/s wr, 1.28k op/s 2024-04-21T08:48:29.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:28 smithi016 ceph-mon[31259]: pgmap v117: 129 pgs: 129 active+clean; 365 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.9 MiB/s wr, 1.28k op/s 2024-04-21T08:48:29.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:28 smithi077 ceph-mon[31615]: pgmap v117: 129 pgs: 129 active+clean; 365 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.9 MiB/s wr, 1.28k op/s 2024-04-21T08:48:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:30 smithi007 ceph-mon[28477]: pgmap v118: 129 pgs: 129 active+clean; 556 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 21 MiB/s wr, 1.41k op/s 2024-04-21T08:48:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:30 smithi016 ceph-mon[31259]: pgmap v118: 129 pgs: 129 active+clean; 556 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 21 MiB/s wr, 1.41k op/s 2024-04-21T08:48:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:30 smithi077 ceph-mon[31615]: pgmap v118: 129 pgs: 129 active+clean; 556 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 21 MiB/s wr, 1.41k op/s 2024-04-21T08:48:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.434+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3711761824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a0102b80 msgr2=0x7fd5a0106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.434+0000 7fd5a7d52640 1 --2- 172.21.15.7:0/3711761824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a0102b80 0x7fd5a0106bc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd590009860 tx=0x7fd59002bb70 comp rx=0 tx=0).stop 2024-04-21T08:48:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.435+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3711761824 shutdown_connections 2024-04-21T08:48:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.435+0000 7fd5a7d52640 1 --2- 172.21.15.7:0/3711761824 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5a01072a0 0x7fd5a010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.435+0000 7fd5a7d52640 1 --2- 172.21.15.7:0/3711761824 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a0102b80 0x7fd5a0106bc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.435+0000 7fd5a7d52640 1 --2- 172.21.15.7:0/3711761824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5a01021d0 0x7fd5a01025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.435+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3711761824 >> 172.21.15.7:0/3711761824 conn(0x7fd5a00fda10 msgr2=0x7fd5a00ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.435+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3711761824 shutdown_connections 2024-04-21T08:48:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.435+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3711761824 wait complete. 2024-04-21T08:48:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.435+0000 7fd5a7d52640 1 Processor -- start 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.435+0000 7fd5a7d52640 1 -- start start 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a7d52640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5a01021d0 0x7fd5a0074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a7d52640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5a01072a0 0x7fd5a0074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a7d52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a0074cd0 0x7fd5a01a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a7d52640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a0112190 con 0x7fd5a01072a0 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a7d52640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a0112010 con 0x7fd5a0074cd0 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a7d52640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a0112310 con 0x7fd5a01021d0 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a52ed640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5a01072a0 0x7fd5a0074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a52ed640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5a01072a0 0x7fd5a0074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49830/0 (socket says 172.21.15.7:49830) 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a52ed640 1 -- 172.21.15.7:0/3485549938 learned_addr learned my addr 172.21.15.7:0/3485549938 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a62ef640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a0074cd0 0x7fd5a01a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5a01021d0 0x7fd5a0074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a52ed640 1 -- 172.21.15.7:0/3485549938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5a01021d0 msgr2=0x7fd5a0074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a52ed640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5a01021d0 0x7fd5a0074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a52ed640 1 -- 172.21.15.7:0/3485549938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a0074cd0 msgr2=0x7fd5a01a6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a52ed640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a0074cd0 0x7fd5a01a6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a52ed640 1 -- 172.21.15.7:0/3485549938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a01a7030 con 0x7fd5a01072a0 2024-04-21T08:48:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a52ed640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5a01072a0 0x7fd5a0074790 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd590000c00 tx=0x7fd5900045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd596ffd640 1 -- 172.21.15.7:0/3485549938 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590041020 con 0x7fd5a01072a0 2024-04-21T08:48:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3485549938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a01a72c0 con 0x7fd5a01072a0 2024-04-21T08:48:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd596ffd640 1 -- 172.21.15.7:0/3485549938 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd59002c9c0 con 0x7fd5a01072a0 2024-04-21T08:48:32.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.436+0000 7fd596ffd640 1 -- 172.21.15.7:0/3485549938 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590039040 con 0x7fd5a01072a0 2024-04-21T08:48:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.437+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3485549938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a01a7800 con 0x7fd5a01072a0 2024-04-21T08:48:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.439+0000 7fd596ffd640 1 -- 172.21.15.7:0/3485549938 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd590044020 con 0x7fd5a01072a0 2024-04-21T08:48:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5aaa28740 1 Processor -- start 2024-04-21T08:48:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5aaa28740 1 -- start start 2024-04-21T08:48:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 0x55bb010e3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010e3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010ed0e0 0x55bb010ef4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bb010488d0 con 0x55bb010e48d0 2024-04-21T08:48:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bb00fe5e80 con 0x55bb010e3b60 2024-04-21T08:48:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bb01044890 con 0x55bb010ed0e0 2024-04-21T08:48:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd596ffd640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd58c0a3d10 0x7fd58c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5a5aee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010e3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5a62ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 0x55bb010e3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5a5aee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010e3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:47714/0 (socket says 172.21.15.7:47714) 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5a5aee640 1 -- 172.21.15.7:0/3483929845 learned_addr learned my addr 172.21.15.7:0/3483929845 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.440+0000 7fd5a62ef640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd58c0a3d10 0x7fd58c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd596ffd640 1 -- 172.21.15.7:0/3485549938 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fd5900f2310 con 0x7fd5a01072a0 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd5a52ed640 1 --2- 172.21.15.7:0/3483929845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010ed0e0 0x55bb010ef4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd5a5aee640 1 -- 172.21.15.7:0/3483929845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010ed0e0 msgr2=0x55bb010ef4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3483929845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010ed0e0 0x55bb010ef4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd5a5aee640 1 -- 172.21.15.7:0/3483929845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 msgr2=0x55bb010e3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3483929845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 0x55bb010e3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd5a5aee640 1 -- 172.21.15.7:0/3483929845 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb01007320 con 0x55bb010e3b60 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd5a62ef640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd58c0a3d10 0x7fd58c0a61d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd598009a60 tx=0x7fd5980095c0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:32.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3483929845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010e3fe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009c90 tx=0x7fd59c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=852c72be3d92b911 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd58bfff640 1 -- 172.21.15.7:0/3483929845 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c012450 con 0x55bb010e3b60 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd58bfff640 1 -- 172.21.15.7:0/3483929845 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd59c012aa0 con 0x55bb010e3b60 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.441+0000 7fd58bfff640 1 -- 172.21.15.7:0/3483929845 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c00fa70 con 0x55bb010e3b60 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.442+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3483929845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 msgr2=0x55bb010e3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.442+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3483929845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010e3fe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009c90 tx=0x7fd59c0117b0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.442+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3483929845 shutdown_connections 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.442+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3483929845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010ed0e0 0x55bb010ef4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.442+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3483929845 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010e3fe0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.442+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3483929845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 0x55bb010e3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.442+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3483929845 >> 172.21.15.7:0/3483929845 conn(0x55bb010aac90 msgr2=0x55bb010de850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.442+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3483929845 shutdown_connections 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.442+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3483929845 wait complete. 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.442+0000 7fd5aaa28740 1 -- start start 2024-04-21T08:48:32.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.442+0000 7fd5aaa28740 1 Processor -- start 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010ecab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010e48d0 0x55bb010e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010ed0e0 0x55bb010ea450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bb010efe10 con 0x55bb010ed0e0 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bb00fd0eb0 con 0x55bb010e3b60 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bb010eff90 con 0x55bb010e48d0 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5aaa28740 10 client.0 _ll_register_callbacks cb 0x55bb00ffdb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a62ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010ed0e0 0x55bb010ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a5aee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010ecab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a62ef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010ed0e0 0x55bb010ea450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49850/0 (socket says 172.21.15.7:49850) 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a5aee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010ecab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:47730/0 (socket says 172.21.15.7:47730) 2024-04-21T08:48:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a62ef640 1 -- 172.21.15.7:0/3226562480 learned_addr learned my addr 172.21.15.7:0/3226562480 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a52ed640 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010e48d0 0x55bb010e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a62ef640 1 -- 172.21.15.7:0/3226562480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010e48d0 msgr2=0x55bb010e9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a62ef640 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010e48d0 0x55bb010e9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a62ef640 1 -- 172.21.15.7:0/3226562480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 msgr2=0x55bb010ecab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a62ef640 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010ecab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a62ef640 1 -- 172.21.15.7:0/3226562480 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb00d5b210 con 0x55bb010ed0e0 2024-04-21T08:48:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.443+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010ecab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5a62ef640 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010ed0e0 0x55bb010ea450 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd598003ef0 tx=0x7fd598004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5294cce805b100 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5897fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5897fa640 1 -- 172.21.15.7:0/3226562480 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59802c060 con 0x55bb010ed0e0 2024-04-21T08:48:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5897fa640 1 -- 172.21.15.7:0/3226562480 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fd598017040 con 0x55bb010ed0e0 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3226562480 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd59c012aa0 con 0x55bb010ed0e0 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5897fa640 1 -- 172.21.15.7:0/3226562480 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd598033050 con 0x55bb010ed0e0 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5aaa28740 10 client.16470 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3226562480 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bb00e27620 con 0x55bb010ed0e0 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5897fa640 1 -- 172.21.15.7:0/3226562480 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd59c012aa0 con 0x55bb010ed0e0 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5897fa640 1 -- 172.21.15.7:0/3226562480 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fd59800b040 con 0x55bb010ed0e0 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5aaa28740 10 client.16470 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5aaa28740 10 client.16470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5aaa28740 20 client.16470 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bb010fd110 0x55bb010ff500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5aaa28740 4 client.16470 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.444+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3226562480 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb0105c880 con 0x55bb010fd110 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.445+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bb010fd110 0x55bb010ff500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.445+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bb010fd110 0x55bb010ff500 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c2448fab3d5e813 server_cookie=98bacf8a3e3645be in_seq=0 out_seq=0 2024-04-21T08:48:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.445+0000 7fd5897fa640 10 client.16470 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5897fa640 1 -- 172.21.15.7:0/3226562480 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb0105c880 con 0x55bb010fd110 2024-04-21T08:48:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5897fa640 10 client.16470 handle_command_reply: tid=0 2024-04-21T08:48:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 1 client.16470 shutdown 2024-04-21T08:48:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3226562480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010ed0e0 msgr2=0x55bb010ea450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010ed0e0 0x55bb010ea450 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd598003ef0 tx=0x7fd598004a40 comp rx=0 tx=0).stop 2024-04-21T08:48:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3226562480 shutdown_connections 2024-04-21T08:48:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bb010fd110 0x55bb010ff500 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010ed0e0 0x55bb010ea450 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010e48d0 0x55bb010e9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5897fa640 0 client.16470 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:32.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3226562480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010ecab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3226562480 >> 172.21.15.7:0/3226562480 conn(0x55bb010aac90 msgr2=0x55bb010e05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:32.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3226562480 shutdown_connections 2024-04-21T08:48:32.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3226562480 wait complete. 2024-04-21T08:48:32.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.447+0000 7fd5aaa28740 20 client.16470 trim_cache size 0 max 16384 2024-04-21T08:48:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.463+0000 7fd5aaa28740 1 Processor -- start 2024-04-21T08:48:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.463+0000 7fd5aaa28740 1 -- start start 2024-04-21T08:48:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.463+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e3b60 0x55bb010f8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.463+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010e48d0 0x55bb010ec570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.463+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010ed0e0 0x55bb010ecab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.463+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a01af640 con 0x55bb010e3b60 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.463+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a01b1ec0 con 0x55bb010ed0e0 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.463+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a01afcb0 con 0x55bb010e48d0 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.463+0000 7fd5a5aee640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e3b60 0x55bb010f8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.463+0000 7fd5a5aee640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e3b60 0x55bb010f8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49860/0 (socket says 172.21.15.7:49860) 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd5a5aee640 1 -- 172.21.15.7:0/3804470893 learned_addr learned my addr 172.21.15.7:0/3804470893 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd5a62ef640 1 --2- 172.21.15.7:0/3804470893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010ed0e0 0x55bb010ecab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd5a52ed640 1 --2- 172.21.15.7:0/3804470893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010e48d0 0x55bb010ec570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd5a5aee640 1 -- 172.21.15.7:0/3804470893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010e48d0 msgr2=0x55bb010ec570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3804470893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010e48d0 0x55bb010ec570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd5a5aee640 1 -- 172.21.15.7:0/3804470893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010ed0e0 msgr2=0x55bb010ecab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3804470893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010ed0e0 0x55bb010ecab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd5a5aee640 1 -- 172.21.15.7:0/3804470893 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb010eaa50 con 0x55bb010e3b60 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd5a52ed640 1 --2- 172.21.15.7:0/3804470893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010e48d0 0x55bb010ec570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3804470893 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e3b60 0x55bb010f8d10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x55bb010e5030 tx=0x7fd59c00f560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba57d288ce54896 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd58b7fe640 1 -- 172.21.15.7:0/3804470893 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c033070 con 0x55bb010e3b60 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd58b7fe640 1 -- 172.21.15.7:0/3804470893 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd59c02b070 con 0x55bb010e3b60 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.464+0000 7fd58b7fe640 1 -- 172.21.15.7:0/3804470893 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c005530 con 0x55bb010e3b60 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.465+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3804470893 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e3b60 msgr2=0x55bb010f8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.465+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3804470893 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e3b60 0x55bb010f8d10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x55bb010e5030 tx=0x7fd59c00f560 comp rx=0 tx=0).stop 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.465+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3804470893 shutdown_connections 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.465+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3804470893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010ed0e0 0x55bb010ecab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.465+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3804470893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010e48d0 0x55bb010ec570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.465+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3804470893 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e3b60 0x55bb010f8d10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.465+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3804470893 >> 172.21.15.7:0/3804470893 conn(0x55bb010aac90 msgr2=0x55bb010e9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.465+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3804470893 shutdown_connections 2024-04-21T08:48:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.465+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3804470893 wait complete. 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.465+0000 7fd5aaa28740 1 -- start start 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.465+0000 7fd5aaa28740 1 Processor -- start 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010f5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 0x55bb010ed8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5aaa28740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010ed0e0 0x55bb010eddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a01b0380 con 0x55bb010e48d0 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a01b17f0 con 0x55bb010e3b60 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5aaa28740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a01b0a50 con 0x55bb010ed0e0 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5aaa28740 10 client.0 _ll_register_callbacks cb 0x7fd59800e520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a52ed640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 0x55bb010ed8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a52ed640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 0x55bb010ed8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49872/0 (socket says 172.21.15.7:49872) 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a5aee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a52ed640 1 -- 172.21.15.7:0/3907619060 learned_addr learned my addr 172.21.15.7:0/3907619060 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a62ef640 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010ed0e0 0x55bb010eddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a52ed640 1 -- 172.21.15.7:0/3907619060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010ed0e0 msgr2=0x55bb010eddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a52ed640 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010ed0e0 0x55bb010eddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a52ed640 1 -- 172.21.15.7:0/3907619060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 msgr2=0x55bb010f5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a52ed640 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010f5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a52ed640 1 -- 172.21.15.7:0/3907619060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb010d9f30 con 0x55bb010e48d0 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010f5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5a52ed640 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 0x55bb010ed8b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd5900bdf60 tx=0x7fd59002bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b1173d2416af99 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5897fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5897fa640 1 -- 172.21.15.7:0/3907619060 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5900dc060 con 0x55bb010e48d0 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5897fa640 1 -- 172.21.15.7:0/3907619060 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fd5900035f0 con 0x55bb010e48d0 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.466+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3907619060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd59c02b070 con 0x55bb010e48d0 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5897fa640 1 -- 172.21.15.7:0/3907619060 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd5900d7070 con 0x55bb010e48d0 2024-04-21T08:48:32.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5aaa28740 10 client.16476 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3907619060 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bb010e2e30 con 0x55bb010e48d0 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5897fa640 1 -- 172.21.15.7:0/3907619060 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd59c02b070 con 0x55bb010e48d0 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5897fa640 1 -- 172.21.15.7:0/3907619060 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fd59003c030 con 0x55bb010e48d0 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5aaa28740 10 client.16476 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5aaa28740 10 client.16476 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5aaa28740 20 client.16476 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bb01163860 0x55bb01165c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5aaa28740 4 client.16476 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3907619060 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bb00fe83d0 con 0x55bb01163860 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.467+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bb01163860 0x55bb01165c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.468+0000 7fd5897fa640 10 client.16476 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:32.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:32.468+0000 7fd5a5aee640 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bb01163860 0x55bb01165c80 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db8aa5bd68e752f6 server_cookie=4bb69c5954cc52a2 in_seq=0 out_seq=0 2024-04-21T08:48:32.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:32 smithi007 ceph-mon[28477]: pgmap v119: 129 pgs: 129 active+clean; 556 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 17 MiB/s wr, 804 op/s 2024-04-21T08:48:33.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:32 smithi016 ceph-mon[31259]: pgmap v119: 129 pgs: 129 active+clean; 556 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 17 MiB/s wr, 804 op/s 2024-04-21T08:48:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:32 smithi077 ceph-mon[31615]: pgmap v119: 129 pgs: 129 active+clean; 556 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 17 MiB/s wr, 804 op/s 2024-04-21T08:48:34.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:48:34.772 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:48:34.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:34 smithi007 ceph-mon[28477]: pgmap v120: 129 pgs: 129 active+clean; 559 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 17 MiB/s wr, 835 op/s 2024-04-21T08:48:34.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.980+0000 7fa70f63a640 1 -- 172.21.15.7:0/1065113817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708108f50 msgr2=0x7fa7081093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:34.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.980+0000 7fa70f63a640 1 --2- 172.21.15.7:0/1065113817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708108f50 0x7fa7081093b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8005fd0 tx=0x7fa6f802cd00 comp rx=0 tx=0).stop 2024-04-21T08:48:34.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.986+0000 7fa70f63a640 1 -- 172.21.15.7:0/1065113817 shutdown_connections 2024-04-21T08:48:34.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.986+0000 7fa70f63a640 1 --2- 172.21.15.7:0/1065113817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7081098f0 0x7fa7081157d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:34.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.986+0000 7fa70f63a640 1 --2- 172.21.15.7:0/1065113817 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708108f50 0x7fa7081093b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:34.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.986+0000 7fa70f63a640 1 --2- 172.21.15.7:0/1065113817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70810e980 0x7fa70810ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:34.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.986+0000 7fa70f63a640 1 -- 172.21.15.7:0/1065113817 >> 172.21.15.7:0/1065113817 conn(0x7fa70806e440 msgr2=0x7fa70806e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:34.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.988+0000 7fa70f63a640 1 -- 172.21.15.7:0/1065113817 shutdown_connections 2024-04-21T08:48:34.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.991+0000 7fa70f63a640 1 -- 172.21.15.7:0/1065113817 wait complete. 2024-04-21T08:48:34.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.992+0000 7fa70f63a640 1 Processor -- start 2024-04-21T08:48:34.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.992+0000 7fa70f63a640 1 -- start start 2024-04-21T08:48:34.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.992+0000 7fa70f63a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa708108f50 0x7fa708122d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:34.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.992+0000 7fa70f63a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7081098f0 0x7fa708123290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:34.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.992+0000 7fa70f63a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70810e980 0x7fa7081148a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:34.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.992+0000 7fa70f63a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa708117cf0 con 0x7fa708108f50 2024-04-21T08:48:34.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.992+0000 7fa70f63a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa708117b70 con 0x7fa7081098f0 2024-04-21T08:48:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.992+0000 7fa70f63a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa708117e70 con 0x7fa70810e980 2024-04-21T08:48:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.993+0000 7fa70dbd7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70810e980 0x7fa7081148a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.993+0000 7fa70dbd7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70810e980 0x7fa7081148a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:37366/0 (socket says 172.21.15.7:37366) 2024-04-21T08:48:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.993+0000 7fa70dbd7640 1 -- 172.21.15.7:0/1196165820 learned_addr learned my addr 172.21.15.7:0/1196165820 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.994+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa708108f50 0x7fa708122d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.995+0000 7fa70dbd7640 1 -- 172.21.15.7:0/1196165820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7081098f0 msgr2=0x7fa708123290 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:48:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.995+0000 7fa70dbd7640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7081098f0 0x7fa708123290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.995+0000 7fa70dbd7640 1 -- 172.21.15.7:0/1196165820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa708108f50 msgr2=0x7fa708122d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.995+0000 7fa70dbd7640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa708108f50 0x7fa708122d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:34.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.995+0000 7fa70dbd7640 1 -- 172.21.15.7:0/1196165820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa708115020 con 0x7fa70810e980 2024-04-21T08:48:35.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.997+0000 7fa70dbd7640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70810e980 0x7fa7081148a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc02d530 tx=0x7fa6fc02d9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:35.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.999+0000 7fa7067fc640 1 -- 172.21.15.7:0/1196165820 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc031800 con 0x7fa70810e980 2024-04-21T08:48:35.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.999+0000 7fa7067fc640 1 -- 172.21.15.7:0/1196165820 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6fc031e50 con 0x7fa70810e980 2024-04-21T08:48:35.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:34.999+0000 7fa7067fc640 1 -- 172.21.15.7:0/1196165820 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc02f390 con 0x7fa70810e980 2024-04-21T08:48:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.001+0000 7fa70f63a640 1 -- 172.21.15.7:0/1196165820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa708115310 con 0x7fa70810e980 2024-04-21T08:48:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.001+0000 7fa70f63a640 1 -- 172.21.15.7:0/1196165820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa70811e880 con 0x7fa70810e980 2024-04-21T08:48:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.005+0000 7fa712220740 1 Processor -- start 2024-04-21T08:48:35.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.007+0000 7fa712220740 1 -- start start 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.007+0000 7fa712220740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5df960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.007+0000 7fa712220740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5dfea0 0x5631ac5e8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.007+0000 7fa712220740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5e8a80 0x5631ac5eae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.007+0000 7fa712220740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5631ac5441e0 con 0x5631ac5df580 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.007+0000 7fa712220740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5631ac4e1390 con 0x5631ac5e8a80 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.007+0000 7fa712220740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5631ac5401a0 con 0x5631ac5dfea0 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.009+0000 7fa7067fc640 1 -- 172.21.15.7:0/1196165820 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa6fc0319a0 con 0x7fa70810e980 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.009+0000 7fa70cbd5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5df960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.009+0000 7fa70cbd5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5df960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55560/0 (socket says 172.21.15.7:55560) 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.009+0000 7fa70cbd5640 1 -- 172.21.15.7:0/2444109109 learned_addr learned my addr 172.21.15.7:0/2444109109 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.009+0000 7fa70cbd5640 1 --2- 172.21.15.7:0/2444109109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5e8a80 0x5631ac5eae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.010+0000 7fa70cbd5640 1 -- 172.21.15.7:0/2444109109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5dfea0 msgr2=0x5631ac5e8540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.010+0000 7fa70cbd5640 1 --2- 172.21.15.7:0/2444109109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5dfea0 0x5631ac5e8540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.010+0000 7fa70cbd5640 1 -- 172.21.15.7:0/2444109109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5e8a80 msgr2=0x5631ac5eae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:35.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.010+0000 7fa70cbd5640 1 --2- 172.21.15.7:0/2444109109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5e8a80 0x5631ac5eae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.010+0000 7fa70cbd5640 1 -- 172.21.15.7:0/2444109109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631ac3207d0 con 0x5631ac5df580 2024-04-21T08:48:35.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.012+0000 7fa7067fc640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6f00a3f30 0x7fa6f00a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:35.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.012+0000 7fa7067fc640 1 -- 172.21.15.7:0/1196165820 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa6fc0f28b0 con 0x7fa70810e980 2024-04-21T08:48:35.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.013+0000 7fa70cbd5640 1 --2- 172.21.15.7:0/2444109109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5df960 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8035560 tx=0x7fa6f800d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4365df9a1cdfbc3f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:35.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.013+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6f00a3f30 0x7fa6f00a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:35.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.020+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6f00a3f30 0x7fa6f00a63f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa700005d90 tx=0x7fa700005d20 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:35.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.024+0000 7fa6ef7fe640 1 -- 172.21.15.7:0/2444109109 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f806a070 con 0x5631ac5df580 2024-04-21T08:48:35.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.024+0000 7fa6ef7fe640 1 -- 172.21.15.7:0/2444109109 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6f800a730 con 0x5631ac5df580 2024-04-21T08:48:35.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.024+0000 7fa6ef7fe640 1 -- 172.21.15.7:0/2444109109 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8003970 con 0x5631ac5df580 2024-04-21T08:48:35.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.025+0000 7fa712220740 1 -- 172.21.15.7:0/2444109109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 msgr2=0x5631ac5df960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:35.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.025+0000 7fa712220740 1 --2- 172.21.15.7:0/2444109109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5df960 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8035560 tx=0x7fa6f800d000 comp rx=0 tx=0).stop 2024-04-21T08:48:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.033+0000 7fa712220740 1 -- 172.21.15.7:0/2444109109 shutdown_connections 2024-04-21T08:48:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.033+0000 7fa712220740 1 --2- 172.21.15.7:0/2444109109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5e8a80 0x5631ac5eae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.033+0000 7fa712220740 1 --2- 172.21.15.7:0/2444109109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5dfea0 0x5631ac5e8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.033+0000 7fa712220740 1 --2- 172.21.15.7:0/2444109109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5df960 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.033+0000 7fa712220740 1 -- 172.21.15.7:0/2444109109 >> 172.21.15.7:0/2444109109 conn(0x5631ac5d9cb0 msgr2=0x5631ac5da090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.034+0000 7fa712220740 1 -- 172.21.15.7:0/2444109109 shutdown_connections 2024-04-21T08:48:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.035+0000 7fa712220740 1 -- 172.21.15.7:0/2444109109 wait complete. 2024-04-21T08:48:35.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.036+0000 7fa712220740 1 -- start start 2024-04-21T08:48:35.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.036+0000 7fa712220740 1 Processor -- start 2024-04-21T08:48:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.037+0000 7fa712220740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5df580 0x5631ac5e5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.037+0000 7fa712220740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5dfea0 0x5631ac5e60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.038+0000 7fa712220740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5e8a80 0x5631ac5e6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.038+0000 7fa712220740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5631ac5eb810 con 0x5631ac5dfea0 2024-04-21T08:48:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.038+0000 7fa712220740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5631ac4fd2f0 con 0x5631ac5df580 2024-04-21T08:48:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.038+0000 7fa712220740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5631ac5eb990 con 0x5631ac5e8a80 2024-04-21T08:48:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.038+0000 7fa712220740 10 client.0 _ll_register_callbacks cb 0x5631ac50a580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.039+0000 7fa70d3d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5df580 0x5631ac5e5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.039+0000 7fa70dbd7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5e8a80 0x5631ac5e6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.039+0000 7fa70d3d6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5df580 0x5631ac5e5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:36964/0 (socket says 172.21.15.7:36964) 2024-04-21T08:48:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.039+0000 7fa70d3d6640 1 -- 172.21.15.7:0/1114425088 learned_addr learned my addr 172.21.15.7:0/1114425088 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.042+0000 7fa70d3d6640 1 -- 172.21.15.7:0/1114425088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5e8a80 msgr2=0x5631ac5e6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.042+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/1114425088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5e8a80 0x5631ac5e6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.042+0000 7fa70d3d6640 1 -- 172.21.15.7:0/1114425088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5dfea0 msgr2=0x5631ac5e60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.042+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/1114425088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5dfea0 0x5631ac5e60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.042+0000 7fa70d3d6640 1 -- 172.21.15.7:0/1114425088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631ac2563f0 con 0x5631ac5df580 2024-04-21T08:48:35.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.043+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/1114425088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5df580 0x5631ac5e5ba0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa70000c3a0 tx=0x7fa7000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c0c5ba80e7a270b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:35.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.047+0000 7fa6ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:48:35.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.049+0000 7fa6ecff9640 1 -- 172.21.15.7:0/1114425088 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa700021070 con 0x5631ac5df580 2024-04-21T08:48:35.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.049+0000 7fa6ecff9640 1 -- 172.21.15.7:0/1114425088 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa70000fac0 con 0x5631ac5df580 2024-04-21T08:48:35.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.050+0000 7fa712220740 1 -- 172.21.15.7:0/1114425088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6f800a730 con 0x5631ac5df580 2024-04-21T08:48:35.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:34 smithi016 ceph-mon[31259]: pgmap v120: 129 pgs: 129 active+clean; 559 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 17 MiB/s wr, 835 op/s 2024-04-21T08:48:35.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.055+0000 7fa6ecff9640 1 -- 172.21.15.7:0/1114425088 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa70002b050 con 0x5631ac5df580 2024-04-21T08:48:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.055+0000 7fa712220740 10 client.26042 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.055+0000 7fa712220740 1 -- 172.21.15.7:0/1114425088 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5631ac2160f0 con 0x5631ac5df580 2024-04-21T08:48:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.064+0000 7fa6ecff9640 1 -- 172.21.15.7:0/1114425088 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6f800a730 con 0x5631ac5df580 2024-04-21T08:48:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.064+0000 7fa6ecff9640 1 -- 172.21.15.7:0/1114425088 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa700026440 con 0x5631ac5df580 2024-04-21T08:48:35.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.068+0000 7fa712220740 10 client.26042 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:35.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.068+0000 7fa712220740 10 client.26042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.068+0000 7fa712220740 20 client.26042 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.068+0000 7fa712220740 1 --2- 172.21.15.7:0/1114425088 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5631ac5f8b60 0x5631ac5fafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.068+0000 7fa712220740 4 client.26042 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.068+0000 7fa712220740 1 -- 172.21.15.7:0/1114425088 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631ac551c00 con 0x5631ac5f8b60 2024-04-21T08:48:35.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.069+0000 7fa70cbd5640 1 --2- 172.21.15.7:0/1114425088 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5631ac5f8b60 0x5631ac5fafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.084+0000 7fa70cbd5640 1 --2- 172.21.15.7:0/1114425088 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5631ac5f8b60 0x5631ac5fafd0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad585d9d53756707 server_cookie=63f036ca4f7ae325 in_seq=0 out_seq=0 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:35.084+0000 7fa6ecff9640 10 client.26042 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:35.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:34 smithi077 ceph-mon[31615]: pgmap v120: 129 pgs: 129 active+clean; 559 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 17 MiB/s wr, 835 op/s 2024-04-21T08:48:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.309+0000 7fd5897fa640 1 -- 172.21.15.7:0/3907619060 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bb00fe83d0 con 0x55bb01163860 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.309+0000 7fd5897fa640 10 client.16476 handle_command_reply: tid=0 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 1 client.16476 shutdown 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3907619060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 msgr2=0x55bb010ed8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 0x55bb010ed8b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd5900bdf60 tx=0x7fd59002bb70 comp rx=0 tx=0).stop 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3907619060 shutdown_connections 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bb01163860 0x55bb01165c80 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb010ed0e0 0x55bb010eddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bb010e48d0 0x55bb010ed8b0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 1 --2- 172.21.15.7:0/3907619060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bb010e3b60 0x55bb010f5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5897fa640 0 client.16476 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3907619060 >> 172.21.15.7:0/3907619060 conn(0x55bb010aac90 msgr2=0x55bb010e8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3907619060 shutdown_connections 2024-04-21T08:48:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 1 -- 172.21.15.7:0/3907619060 wait complete. 2024-04-21T08:48:36.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa6ecff9640 1 -- 172.21.15.7:0/1114425088 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631ac551c00 con 0x5631ac5f8b60 2024-04-21T08:48:36.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa6ecff9640 10 client.26042 handle_command_reply: tid=0 2024-04-21T08:48:36.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa712220740 1 client.26042 shutdown 2024-04-21T08:48:36.312 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fd5aaa28740 20 client.16476 trim_cache size 0 max 16384 2024-04-21T08:48:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa712220740 1 -- 172.21.15.7:0/1114425088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5df580 msgr2=0x5631ac5e5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa712220740 1 --2- 172.21.15.7:0/1114425088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5df580 0x5631ac5e5ba0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa70000c3a0 tx=0x7fa7000046c0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:48:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:48:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:48:36.313 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:48:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa712220740 1 -- 172.21.15.7:0/1114425088 shutdown_connections 2024-04-21T08:48:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa712220740 1 --2- 172.21.15.7:0/1114425088 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5631ac5f8b60 0x5631ac5fafd0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa712220740 1 --2- 172.21.15.7:0/1114425088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5e8a80 0x5631ac5e6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa6ecff9640 0 client.26042 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:36.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa712220740 1 --2- 172.21.15.7:0/1114425088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5dfea0 0x5631ac5e60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa712220740 1 --2- 172.21.15.7:0/1114425088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5df580 0x5631ac5e5ba0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.310+0000 7fa712220740 1 -- 172.21.15.7:0/1114425088 >> 172.21.15.7:0/1114425088 conn(0x5631ac5d9cb0 msgr2=0x5631ac5dba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fa712220740 1 -- 172.21.15.7:0/1114425088 shutdown_connections 2024-04-21T08:48:36.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fa712220740 1 -- 172.21.15.7:0/1114425088 wait complete. 2024-04-21T08:48:36.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fa712220740 20 client.26042 trim_cache size 0 max 16384 2024-04-21T08:48:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3485549938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd58c0a3d10 msgr2=0x7fd58c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd58c0a3d10 0x7fd58c0a61d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd598009a60 tx=0x7fd5980095c0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3485549938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5a01072a0 msgr2=0x7fd5a0074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5a01072a0 0x7fd5a0074790 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd590000c00 tx=0x7fd5900045c0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3485549938 shutdown_connections 2024-04-21T08:48:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5a0074cd0 0x7fd5a01a6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5a01072a0 0x7fd5a0074790 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd58c0a3d10 0x7fd58c0a61d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 --2- 172.21.15.7:0/3485549938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5a01021d0 0x7fd5a0074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3485549938 >> 172.21.15.7:0/3485549938 conn(0x7fd5a00fda10 msgr2=0x7fd5a00fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3485549938 shutdown_connections 2024-04-21T08:48:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:36.311+0000 7fd5a7d52640 1 -- 172.21.15.7:0/3485549938 wait complete. 2024-04-21T08:48:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.329+0000 7fa712220740 1 Processor -- start 2024-04-21T08:48:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.329+0000 7fa712220740 1 -- start start 2024-04-21T08:48:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.329+0000 7fa712220740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5e82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.329+0000 7fa712220740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5dfea0 0x5631ac5f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa712220740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5e8a80 0x5631ac5e4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa712220740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa70807b4f0 con 0x5631ac5df580 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa712220740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa70807dd70 con 0x5631ac5dfea0 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa712220740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa70807bb60 con 0x5631ac5e8a80 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70d3d6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5e82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70d3d6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5e82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55590/0 (socket says 172.21.15.7:55590) 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70d3d6640 1 -- 172.21.15.7:0/315328614 learned_addr learned my addr 172.21.15.7:0/315328614 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70dbd7640 1 --2- 172.21.15.7:0/315328614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5e8a80 0x5631ac5e4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70cbd5640 1 --2- 172.21.15.7:0/315328614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5dfea0 0x5631ac5f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70d3d6640 1 -- 172.21.15.7:0/315328614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5e8a80 msgr2=0x5631ac5e4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/315328614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5e8a80 0x5631ac5e4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70d3d6640 1 -- 172.21.15.7:0/315328614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5dfea0 msgr2=0x5631ac5f46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/315328614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5dfea0 0x5631ac5f46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70d3d6640 1 -- 172.21.15.7:0/315328614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631ac5ded20 con 0x5631ac5df580 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70cbd5640 1 --2- 172.21.15.7:0/315328614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5dfea0 0x5631ac5f46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70dbd7640 1 --2- 172.21.15.7:0/315328614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5e8a80 0x5631ac5e4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/315328614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5e82d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa7000058c0 tx=0x7fa700005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19624558a01fabf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa6eeffd640 1 -- 172.21.15.7:0/315328614 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa700021070 con 0x5631ac5df580 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa6eeffd640 1 -- 172.21.15.7:0/315328614 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa700026e50 con 0x5631ac5df580 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.330+0000 7fa6eeffd640 1 -- 172.21.15.7:0/315328614 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70001d6b0 con 0x5631ac5df580 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.331+0000 7fa712220740 1 -- 172.21.15.7:0/315328614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 msgr2=0x5631ac5e82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.331+0000 7fa712220740 1 --2- 172.21.15.7:0/315328614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5e82d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa7000058c0 tx=0x7fa700005740 comp rx=0 tx=0).stop 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.331+0000 7fa712220740 1 -- 172.21.15.7:0/315328614 shutdown_connections 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.331+0000 7fa712220740 1 --2- 172.21.15.7:0/315328614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5e8a80 0x5631ac5e4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.331+0000 7fa712220740 1 --2- 172.21.15.7:0/315328614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5dfea0 0x5631ac5f46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.331+0000 7fa712220740 1 --2- 172.21.15.7:0/315328614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5e82d0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.331+0000 7fa712220740 1 -- 172.21.15.7:0/315328614 >> 172.21.15.7:0/315328614 conn(0x5631ac5d9cb0 msgr2=0x5631ac5d8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.331+0000 7fa712220740 1 -- 172.21.15.7:0/315328614 shutdown_connections 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.331+0000 7fa712220740 1 -- 172.21.15.7:0/315328614 wait complete. 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa712220740 1 -- start start 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa712220740 1 Processor -- start 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa712220740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5f0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa712220740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5dfea0 0x5631ac5f1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa712220740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5e8a80 0x5631ac5edd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa712220740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa70807c230 con 0x5631ac5df580 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa712220740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa70807d6a0 con 0x5631ac5e8a80 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa712220740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa70807c900 con 0x5631ac5dfea0 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa712220740 10 client.0 _ll_register_callbacks cb 0x7fa700014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa70d3d6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5f0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa70d3d6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5f0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55604/0 (socket says 172.21.15.7:55604) 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa70d3d6640 1 -- 172.21.15.7:0/3785762705 learned_addr learned my addr 172.21.15.7:0/3785762705 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa70dbd7640 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5e8a80 0x5631ac5edd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa70cbd5640 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5dfea0 0x5631ac5f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa70d3d6640 1 -- 172.21.15.7:0/3785762705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5dfea0 msgr2=0x5631ac5f1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5dfea0 0x5631ac5f1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa70d3d6640 1 -- 172.21.15.7:0/3785762705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5e8a80 msgr2=0x5631ac5edd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5e8a80 0x5631ac5edd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.332+0000 7fa70d3d6640 1 -- 172.21.15.7:0/3785762705 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631ac5ee590 con 0x5631ac5df580 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa70dbd7640 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5e8a80 0x5631ac5edd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa70d3d6640 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5f0be0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa7000106e0 tx=0x7fa700005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acacdf0e6022763 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa6ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa6ecff9640 1 -- 172.21.15.7:0/3785762705 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa700021050 con 0x5631ac5df580 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa6ecff9640 1 -- 172.21.15.7:0/3785762705 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa70000f9a0 con 0x5631ac5df580 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa712220740 1 -- 172.21.15.7:0/3785762705 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa700026e50 con 0x5631ac5df580 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa6ecff9640 1 -- 172.21.15.7:0/3785762705 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa70004b050 con 0x5631ac5df580 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa712220740 10 client.16488 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa712220740 1 -- 172.21.15.7:0/3785762705 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5631ac65fd10 con 0x5631ac5df580 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa6ecff9640 1 -- 172.21.15.7:0/3785762705 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa700026e50 con 0x5631ac5df580 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa6ecff9640 1 -- 172.21.15.7:0/3785762705 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa700037020 con 0x5631ac5df580 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa712220740 10 client.16488 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa712220740 10 client.16488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa712220740 20 client.16488 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa712220740 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5631ac660310 0x5631ac662700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa712220740 4 client.16488 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.333+0000 7fa712220740 1 -- 172.21.15.7:0/3785762705 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5631ac54ba70 con 0x5631ac660310 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa70cbd5640 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5631ac660310 0x5631ac662700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa70cbd5640 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5631ac660310 0x5631ac662700 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d03d64839c31be9a server_cookie=ad761e2095921603 in_seq=0 out_seq=0 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa6ecff9640 10 client.16488 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa6ecff9640 1 -- 172.21.15.7:0/3785762705 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5631ac54ba70 con 0x5631ac660310 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa6ecff9640 10 client.16488 handle_command_reply: tid=0 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa712220740 1 client.16488 shutdown 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa712220740 1 -- 172.21.15.7:0/3785762705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 msgr2=0x5631ac5f0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa712220740 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5f0be0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa7000106e0 tx=0x7fa700005f80 comp rx=0 tx=0).stop 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa712220740 1 -- 172.21.15.7:0/3785762705 shutdown_connections 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa712220740 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5631ac660310 0x5631ac662700 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa712220740 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5631ac5e8a80 0x5631ac5edd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa712220740 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631ac5dfea0 0x5631ac5f1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa6ecff9640 0 client.16488 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa712220740 1 --2- 172.21.15.7:0/3785762705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5631ac5df580 0x5631ac5f0be0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.334+0000 7fa712220740 1 -- 172.21.15.7:0/3785762705 >> 172.21.15.7:0/3785762705 conn(0x5631ac5d9cb0 msgr2=0x5631ac5daf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa712220740 1 -- 172.21.15.7:0/3785762705 shutdown_connections 2024-04-21T08:48:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa712220740 1 -- 172.21.15.7:0/3785762705 wait complete. 2024-04-21T08:48:36.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa712220740 20 client.16488 trim_cache size 0 max 16384 2024-04-21T08:48:36.342 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:48:36.342 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:48:36.343 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:48:36.343 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:48:36.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 -- 172.21.15.7:0/1196165820 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6f00a3f30 msgr2=0x7fa6f00a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6f00a3f30 0x7fa6f00a63f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa700005d90 tx=0x7fa700005d20 comp rx=0 tx=0).stop 2024-04-21T08:48:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 -- 172.21.15.7:0/1196165820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70810e980 msgr2=0x7fa7081148a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70810e980 0x7fa7081148a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc02d530 tx=0x7fa6fc02d9f0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 -- 172.21.15.7:0/1196165820 shutdown_connections 2024-04-21T08:48:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa6f00a3f30 0x7fa6f00a63f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70810e980 0x7fa7081148a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7081098f0 0x7fa708123290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 --2- 172.21.15.7:0/1196165820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa708108f50 0x7fa708122d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 -- 172.21.15.7:0/1196165820 >> 172.21.15.7:0/1196165820 conn(0x7fa70806e440 msgr2=0x7fa7080709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 -- 172.21.15.7:0/1196165820 shutdown_connections 2024-04-21T08:48:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.335+0000 7fa70f63a640 1 -- 172.21.15.7:0/1196165820 wait complete. 2024-04-21T08:48:36.345 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:48:36.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.448+0000 7fd9a5798640 1 -- 172.21.15.7:0/3129336052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a006ab80 msgr2=0x7fd9a006b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.448+0000 7fd9a5798640 1 --2- 172.21.15.7:0/3129336052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a006ab80 0x7fd9a006b000 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd988009900 tx=0x7fd98802bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.448+0000 7fd9a5798640 1 -- 172.21.15.7:0/3129336052 shutdown_connections 2024-04-21T08:48:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.448+0000 7fd9a5798640 1 --2- 172.21.15.7:0/3129336052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a0104930 0x7fd9a0104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.448+0000 7fd9a5798640 1 --2- 172.21.15.7:0/3129336052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a006ab80 0x7fd9a006b000 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.448+0000 7fd9a5798640 1 --2- 172.21.15.7:0/3129336052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a0106f40 0x7fd9a006a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.448+0000 7fd9a5798640 1 -- 172.21.15.7:0/3129336052 >> 172.21.15.7:0/3129336052 conn(0x7fd9a0077c40 msgr2=0x7fd9a0078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.448+0000 7fd9a5798640 1 -- 172.21.15.7:0/3129336052 shutdown_connections 2024-04-21T08:48:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.448+0000 7fd9a5798640 1 -- 172.21.15.7:0/3129336052 wait complete. 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd9a5798640 1 Processor -- start 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd9a5798640 1 -- start start 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd9a5798640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a006ab80 0x7fd9a01a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd9a5798640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a0104930 0x7fd9a01a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd9a5798640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a0106f40 0x7fd9a019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd9a5798640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a0118740 con 0x7fd9a006ab80 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd9a5798640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01185c0 con 0x7fd9a0104930 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd9a5798640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01188c0 con 0x7fd9a0106f40 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd99effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a006ab80 0x7fd9a01a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd99effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a006ab80 0x7fd9a01a47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55620/0 (socket says 172.21.15.7:55620) 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd99effd640 1 -- 172.21.15.7:0/3498194319 learned_addr learned my addr 172.21.15.7:0/3498194319 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd997fff640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a0104930 0x7fd9a01a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd99f7fe640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a0106f40 0x7fd9a019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd99effd640 1 -- 172.21.15.7:0/3498194319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a0106f40 msgr2=0x7fd9a019e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd99effd640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a0106f40 0x7fd9a019e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd99effd640 1 -- 172.21.15.7:0/3498194319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a0104930 msgr2=0x7fd9a01a4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.449+0000 7fd99effd640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a0104930 0x7fd9a01a4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.450+0000 7fd99effd640 1 -- 172.21.15.7:0/3498194319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a019f120 con 0x7fd9a006ab80 2024-04-21T08:48:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.450+0000 7fd99effd640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a006ab80 0x7fd9a01a47f0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd99000d950 tx=0x7fd99000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.450+0000 7fd99cff9640 1 -- 172.21.15.7:0/3498194319 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd990017070 con 0x7fd9a006ab80 2024-04-21T08:48:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.450+0000 7fd99cff9640 1 -- 172.21.15.7:0/3498194319 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd990009c30 con 0x7fd9a006ab80 2024-04-21T08:48:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.450+0000 7fd9a5798640 1 -- 172.21.15.7:0/3498194319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a019f410 con 0x7fd9a006ab80 2024-04-21T08:48:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.450+0000 7fd99cff9640 1 -- 172.21.15.7:0/3498194319 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99000f620 con 0x7fd9a006ab80 2024-04-21T08:48:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.450+0000 7fd9a5798640 1 -- 172.21.15.7:0/3498194319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a01ab6b0 con 0x7fd9a006ab80 2024-04-21T08:48:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.451+0000 7fd9a5798640 1 -- 172.21.15.7:0/3498194319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a0101470 con 0x7fd9a006ab80 2024-04-21T08:48:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.452+0000 7fd99cff9640 1 -- 172.21.15.7:0/3498194319 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd990010460 con 0x7fd9a006ab80 2024-04-21T08:48:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.453+0000 7fd99cff9640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd98c0a3d20 0x7fd98c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.453+0000 7fd997fff640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd98c0a3d20 0x7fd98c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.453+0000 7fd99cff9640 1 -- 172.21.15.7:0/3498194319 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fd9900d0870 con 0x7fd9a006ab80 2024-04-21T08:48:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.453+0000 7fd997fff640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd98c0a3d20 0x7fd98c0a61e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd988005ab0 tx=0x7fd988005a40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.457+0000 7fd99cff9640 1 -- 172.21.15.7:0/3498194319 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd99009cac0 con 0x7fd9a006ab80 2024-04-21T08:48:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.686+0000 7fd9a5798640 1 -- 172.21.15.7:0/3498194319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9a0069250 con 0x7fd9a006ab80 2024-04-21T08:48:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.686+0000 7fd99cff9640 1 -- 172.21.15.7:0/3498194319 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7fd990009de0 con 0x7fd9a006ab80 2024-04-21T08:48:36.686 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:48:36.686 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:48:36.690 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:48:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 -- 172.21.15.7:0/3498194319 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd98c0a3d20 msgr2=0x7fd98c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd98c0a3d20 0x7fd98c0a61e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd988005ab0 tx=0x7fd988005a40 comp rx=0 tx=0).stop 2024-04-21T08:48:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 -- 172.21.15.7:0/3498194319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a006ab80 msgr2=0x7fd9a01a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a006ab80 0x7fd9a01a47f0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd99000d950 tx=0x7fd99000de10 comp rx=0 tx=0).stop 2024-04-21T08:48:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 -- 172.21.15.7:0/3498194319 shutdown_connections 2024-04-21T08:48:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd98c0a3d20 0x7fd98c0a61e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9a0106f40 0x7fd9a019e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a0104930 0x7fd9a01a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 --2- 172.21.15.7:0/3498194319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a006ab80 0x7fd9a01a47f0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 -- 172.21.15.7:0/3498194319 >> 172.21.15.7:0/3498194319 conn(0x7fd9a0077c40 msgr2=0x7fd9a0103910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 -- 172.21.15.7:0/3498194319 shutdown_connections 2024-04-21T08:48:36.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.688+0000 7fd9a5798640 1 -- 172.21.15.7:0/3498194319 wait complete. 2024-04-21T08:48:36.699 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:48:36.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.804+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/3502842357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f8107250 msgr2=0x7fb0f810ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.804+0000 7fb0ff5b2640 1 --2- 172.21.15.7:0/3502842357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f8107250 0x7fb0f810ee20 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009ca0 tx=0x7fb0f002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.804+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/3502842357 shutdown_connections 2024-04-21T08:48:36.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.804+0000 7fb0ff5b2640 1 --2- 172.21.15.7:0/3502842357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f8107250 0x7fb0f810ee20 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.804+0000 7fb0ff5b2640 1 --2- 172.21.15.7:0/3502842357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0f8102b30 0x7fb0f8106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.804+0000 7fb0ff5b2640 1 --2- 172.21.15.7:0/3502842357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0f8102180 0x7fb0f8102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.804+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/3502842357 >> 172.21.15.7:0/3502842357 conn(0x7fb0f80fda00 msgr2=0x7fb0f80ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.804+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/3502842357 shutdown_connections 2024-04-21T08:48:36.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.804+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/3502842357 wait complete. 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.804+0000 7fb0ff5b2640 1 Processor -- start 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.804+0000 7fb0ff5b2640 1 -- start start 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0ff5b2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0f8102180 0x7fb0f8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0ff5b2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f8107250 0x7fb0f8198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0ff5b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0f819cf70 0x7fb0f819d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0ff5b2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f81121c0 con 0x7fb0f8107250 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0ff5b2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f8112040 con 0x7fb0f8102180 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0ff5b2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f8112340 con 0x7fb0f819cf70 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0fcb4d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f8107250 0x7fb0f8198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0fcb4d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f8107250 0x7fb0f8198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55632/0 (socket says 172.21.15.7:55632) 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0fcb4d640 1 -- 172.21.15.7:0/913356520 learned_addr learned my addr 172.21.15.7:0/913356520 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0fdb4f640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0f819cf70 0x7fb0f819d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0f8102180 0x7fb0f8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0fcb4d640 1 -- 172.21.15.7:0/913356520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0f819cf70 msgr2=0x7fb0f819d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0f819cf70 0x7fb0f819d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0fcb4d640 1 -- 172.21.15.7:0/913356520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0f8102180 msgr2=0x7fb0f8198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0f8102180 0x7fb0f8198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0fcb4d640 1 -- 172.21.15.7:0/913356520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f819da60 con 0x7fb0f8107250 2024-04-21T08:48:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f8107250 0x7fb0f8198c40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800c970 tx=0x7fb0e800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0ee7fc640 1 -- 172.21.15.7:0/913356520 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8014070 con 0x7fb0f8107250 2024-04-21T08:48:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/913356520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0f819dd50 con 0x7fb0f8107250 2024-04-21T08:48:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0ee7fc640 1 -- 172.21.15.7:0/913356520 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0e8004440 con 0x7fb0f8107250 2024-04-21T08:48:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0ee7fc640 1 -- 172.21.15.7:0/913356520 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8003e50 con 0x7fb0f8107250 2024-04-21T08:48:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.805+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/913356520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0f81a5590 con 0x7fb0f8107250 2024-04-21T08:48:36.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.808+0000 7fb0ee7fc640 1 -- 172.21.15.7:0/913356520 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fb0e8017070 con 0x7fb0f8107250 2024-04-21T08:48:36.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.808+0000 7fb1021a7740 1 Processor -- start 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.808+0000 7fb1021a7740 1 -- start start 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.808+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79ead3b0 0x563f79ead790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.808+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eb6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.808+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79eb6e60 0x563f79eb72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.808+0000 7fb1021a7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x563f79e121f0 con 0x563f79eadd60 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.808+0000 7fb1021a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563f79daf4b0 con 0x563f79eb6e60 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.808+0000 7fb1021a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563f79e0e1b0 con 0x563f79ead3b0 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0ee7fc640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e40a3ec0 0x7fb0e40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fd34e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eb6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fd34e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eb6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55638/0 (socket says 172.21.15.7:55638) 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fd34e640 1 -- 172.21.15.7:0/927169656 learned_addr learned my addr 172.21.15.7:0/927169656 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/927169656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79eb6e60 0x563f79eb72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fdb4f640 1 --2- 172.21.15.7:0/927169656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79ead3b0 0x563f79ead790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fdb4f640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e40a3ec0 0x7fb0e40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0ee7fc640 1 -- 172.21.15.7:0/913356520 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb0e80d0920 con 0x7fb0f8107250 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fd34e640 1 -- 172.21.15.7:0/927169656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79ead3b0 msgr2=0x563f79ead790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/927169656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79ead3b0 0x563f79ead790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fd34e640 1 -- 172.21.15.7:0/927169656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79eb6e60 msgr2=0x563f79eb72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/927169656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79eb6e60 0x563f79eb72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fd34e640 1 -- 172.21.15.7:0/927169656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f79da11b0 con 0x563f79eadd60 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/927169656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79eb6e60 0x563f79eb72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/927169656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eb6890 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400b760 tx=0x7fb0f400bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf8bed1c9902244 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0fdb4f640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e40a3ec0 0x7fb0e40a6380 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0004af0 tx=0x7fb0f0005c50 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0e37fe640 1 -- 172.21.15.7:0/927169656 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4003dd0 con 0x563f79eadd60 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0e37fe640 1 -- 172.21.15.7:0/927169656 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0f4015070 con 0x563f79eadd60 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.809+0000 7fb0e37fe640 1 -- 172.21.15.7:0/927169656 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4004da0 con 0x563f79eadd60 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.810+0000 7fb1021a7740 1 -- 172.21.15.7:0/927169656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 msgr2=0x563f79eb6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.810+0000 7fb1021a7740 1 --2- 172.21.15.7:0/927169656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eb6890 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400b760 tx=0x7fb0f400bc20 comp rx=0 tx=0).stop 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.810+0000 7fb1021a7740 1 -- 172.21.15.7:0/927169656 shutdown_connections 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.810+0000 7fb1021a7740 1 --2- 172.21.15.7:0/927169656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79eb6e60 0x563f79eb72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.810+0000 7fb1021a7740 1 --2- 172.21.15.7:0/927169656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eb6890 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.810+0000 7fb1021a7740 1 --2- 172.21.15.7:0/927169656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79ead3b0 0x563f79ead790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.810+0000 7fb1021a7740 1 -- 172.21.15.7:0/927169656 >> 172.21.15.7:0/927169656 conn(0x563f79ea7ba0 msgr2=0x563f79ea7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.810+0000 7fb1021a7740 1 -- 172.21.15.7:0/927169656 shutdown_connections 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.810+0000 7fb1021a7740 1 -- 172.21.15.7:0/927169656 wait complete. 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.810+0000 7fb1021a7740 1 -- start start 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb1021a7740 1 Processor -- start 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79ead3b0 0x563f79ebd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eba660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79eb6e60 0x563f79ebaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb1021a7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x563f79ebdda0 con 0x563f79eadd60 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb1021a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563f79dcb500 con 0x563f79ead3b0 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb1021a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563f79ebdf20 con 0x563f79eb6e60 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb1021a7740 10 client.0 _ll_register_callbacks cb 0x563f79d99b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb0fcb4d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eba660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb0fcb4d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eba660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55640/0 (socket says 172.21.15.7:55640) 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb0fcb4d640 1 -- 172.21.15.7:0/2239627538 learned_addr learned my addr 172.21.15.7:0/2239627538 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb0fdb4f640 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79eb6e60 0x563f79ebaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79ead3b0 0x563f79ebd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb0fcb4d640 1 -- 172.21.15.7:0/2239627538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79eb6e60 msgr2=0x563f79ebaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79eb6e60 0x563f79ebaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb0fcb4d640 1 -- 172.21.15.7:0/2239627538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79ead3b0 msgr2=0x563f79ebd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79ead3b0 0x563f79ebd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.811+0000 7fb0fcb4d640 1 -- 172.21.15.7:0/2239627538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f79b243f0 con 0x563f79eadd60 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eba660 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8001e90 tx=0x7fb0e80164b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277a4100789ab770 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb0e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/2239627538 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e80b4070 con 0x563f79eadd60 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb1021a7740 1 -- 172.21.15.7:0/2239627538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0f4015070 con 0x563f79eadd60 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/2239627538 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb0e809c730 con 0x563f79eadd60 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/2239627538 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb0e809cbd0 con 0x563f79eadd60 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb1021a7740 10 client.16506 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb1021a7740 1 -- 172.21.15.7:0/2239627538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563f79bee7d0 con 0x563f79eadd60 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/2239627538 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0f4015070 con 0x563f79eadd60 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/2239627538 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb0e80a4cb0 con 0x563f79eadd60 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb1021a7740 10 client.16506 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb1021a7740 10 client.16506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb1021a7740 20 client.16506 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb1021a7740 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x563f79ecb020 0x563f79ecd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb1021a7740 4 client.16506 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.812+0000 7fb1021a7740 1 -- 172.21.15.7:0/2239627538 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f79db1930 con 0x563f79ecb020 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.813+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x563f79ecb020 0x563f79ecd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.813+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x563f79ecb020 0x563f79ecd450 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dd1d300bda14400 server_cookie=a48912a3b42bc3b9 in_seq=0 out_seq=0 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.813+0000 7fb0e0ff9640 10 client.16506 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/2239627538 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f79db1930 con 0x563f79ecb020 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb0e0ff9640 10 client.16506 handle_command_reply: tid=0 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 1 client.16506 shutdown 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 1 -- 172.21.15.7:0/2239627538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 msgr2=0x563f79eba660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eba660 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8001e90 tx=0x7fb0e80164b0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 1 -- 172.21.15.7:0/2239627538 shutdown_connections 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x563f79ecb020 0x563f79ecd450 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79eb6e60 0x563f79ebaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79eadd60 0x563f79eba660 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 1 --2- 172.21.15.7:0/2239627538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79ead3b0 0x563f79ebd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 1 -- 172.21.15.7:0/2239627538 >> 172.21.15.7:0/2239627538 conn(0x563f79ea7ba0 msgr2=0x563f79ea98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb0e0ff9640 0 client.16506 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 1 -- 172.21.15.7:0/2239627538 shutdown_connections 2024-04-21T08:48:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 1 -- 172.21.15.7:0/2239627538 wait complete. 2024-04-21T08:48:36.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.814+0000 7fb1021a7740 20 client.16506 trim_cache size 0 max 16384 2024-04-21T08:48:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.827+0000 7fb1021a7740 1 Processor -- start 2024-04-21T08:48:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.827+0000 7fb1021a7740 1 -- start start 2024-04-21T08:48:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79ead3b0 0x563f79eb64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79eadd60 0x563f79eb8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb1021a7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f81ab230 con 0x563f79ea7ba0 2024-04-21T08:48:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb1021a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f81adab0 con 0x563f79eadd60 2024-04-21T08:48:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb1021a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f81ab8a0 con 0x563f79ead3b0 2024-04-21T08:48:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0fd34e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0fd34e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55644/0 (socket says 172.21.15.7:55644) 2024-04-21T08:48:36.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0fd34e640 1 -- 172.21.15.7:0/2548941916 learned_addr learned my addr 172.21.15.7:0/2548941916 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0fdb4f640 1 --2- 172.21.15.7:0/2548941916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79eadd60 0x563f79eb8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/2548941916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79ead3b0 0x563f79eb64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0fd34e640 1 -- 172.21.15.7:0/2548941916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79ead3b0 msgr2=0x563f79eb64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/2548941916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79ead3b0 0x563f79eb64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0fd34e640 1 -- 172.21.15.7:0/2548941916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79eadd60 msgr2=0x563f79eb8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/2548941916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79eadd60 0x563f79eb8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0fd34e640 1 -- 172.21.15.7:0/2548941916 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f79eac560 con 0x563f79ea7ba0 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/2548941916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec6ca0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4004990 tx=0x7fb0f40049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f131c3312f4848b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0e2ffd640 1 -- 172.21.15.7:0/2548941916 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4003dd0 con 0x563f79ea7ba0 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0e2ffd640 1 -- 172.21.15.7:0/2548941916 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0f4015050 con 0x563f79ea7ba0 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.828+0000 7fb0e2ffd640 1 -- 172.21.15.7:0/2548941916 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4011440 con 0x563f79ea7ba0 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.829+0000 7fb1021a7740 1 -- 172.21.15.7:0/2548941916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 msgr2=0x563f79ec6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.829+0000 7fb1021a7740 1 --2- 172.21.15.7:0/2548941916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec6ca0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4004990 tx=0x7fb0f40049c0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.829+0000 7fb1021a7740 1 -- 172.21.15.7:0/2548941916 shutdown_connections 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.829+0000 7fb1021a7740 1 --2- 172.21.15.7:0/2548941916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79eadd60 0x563f79eb8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.829+0000 7fb1021a7740 1 --2- 172.21.15.7:0/2548941916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79ead3b0 0x563f79eb64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.829+0000 7fb1021a7740 1 --2- 172.21.15.7:0/2548941916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec6ca0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.829+0000 7fb1021a7740 1 -- 172.21.15.7:0/2548941916 >> 172.21.15.7:0/2548941916 conn(0x563f79ebad00 msgr2=0x563f79eb5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.829+0000 7fb1021a7740 1 -- 172.21.15.7:0/2548941916 shutdown_connections 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.829+0000 7fb1021a7740 1 -- 172.21.15.7:0/2548941916 wait complete. 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.829+0000 7fb1021a7740 1 -- start start 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.829+0000 7fb1021a7740 1 Processor -- start 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79ead3b0 0x563f79ec36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb1021a7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79eadd60 0x563f79eb7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb1021a7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f81abf70 con 0x563f79ea7ba0 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb1021a7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f81ad3e0 con 0x563f79ead3b0 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb1021a7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f81ac640 con 0x563f79eadd60 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb1021a7740 10 client.0 _ll_register_callbacks cb 0x7fb0e809c2a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0fd34e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0fd34e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55648/0 (socket says 172.21.15.7:55648) 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0fd34e640 1 -- 172.21.15.7:0/251536510 learned_addr learned my addr 172.21.15.7:0/251536510 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0fdb4f640 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79eadd60 0x563f79eb7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79ead3b0 0x563f79ec36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0fd34e640 1 -- 172.21.15.7:0/251536510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79eadd60 msgr2=0x563f79eb7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79eadd60 0x563f79eb7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0fd34e640 1 -- 172.21.15.7:0/251536510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79ead3b0 msgr2=0x563f79ec36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79ead3b0 0x563f79ec36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0fd34e640 1 -- 172.21.15.7:0/251536510 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f79eb7980 con 0x563f79ea7ba0 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0fd34e640 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec31b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb0f402a040 tx=0x7fb0f4004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d33b8dac88d9b0a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.830+0000 7fb0e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/251536510 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4027120 con 0x563f79ea7ba0 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/251536510 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb0f40272c0 con 0x563f79ea7ba0 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb1021a7740 1 -- 172.21.15.7:0/251536510 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0f4015050 con 0x563f79ea7ba0 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/251536510 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb0f403d050 con 0x563f79ea7ba0 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb1021a7740 10 client.16512 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb1021a7740 1 -- 172.21.15.7:0/251536510 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563f79f30f50 con 0x563f79ea7ba0 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/251536510 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0f4015050 con 0x563f79ea7ba0 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/251536510 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb0f4011710 con 0x563f79ea7ba0 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb1021a7740 10 client.16512 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb1021a7740 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb1021a7740 20 client.16512 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb1021a7740 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x563f79eb6e60 0x563f79f33430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb1021a7740 4 client.16512 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb1021a7740 1 -- 172.21.15.7:0/251536510 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563f79e1fc10 con 0x563f79eb6e60 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.831+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x563f79eb6e60 0x563f79f33430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb0e0ff9640 10 client.16512 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb0fcb4d640 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x563f79eb6e60 0x563f79f33430 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35f441d8964d8cb9 server_cookie=72c8c21e695f764d in_seq=0 out_seq=0 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb0e0ff9640 1 -- 172.21.15.7:0/251536510 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563f79e1fc10 con 0x563f79eb6e60 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb0e0ff9640 10 client.16512 handle_command_reply: tid=0 2024-04-21T08:48:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 1 client.16512 shutdown 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 1 -- 172.21.15.7:0/251536510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 msgr2=0x563f79ec31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec31b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb0f402a040 tx=0x7fb0f4004640 comp rx=0 tx=0).stop 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 1 -- 172.21.15.7:0/251536510 shutdown_connections 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x563f79eb6e60 0x563f79f33430 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563f79eadd60 0x563f79eb7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb0e0ff9640 0 client.16512 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x563f79ead3b0 0x563f79ec36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 1 --2- 172.21.15.7:0/251536510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563f79ea7ba0 0x563f79ec31b0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 1 -- 172.21.15.7:0/251536510 >> 172.21.15.7:0/251536510 conn(0x563f79ebad00 msgr2=0x563f79ea9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 1 -- 172.21.15.7:0/251536510 shutdown_connections 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 1 -- 172.21.15.7:0/251536510 wait complete. 2024-04-21T08:48:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.832+0000 7fb1021a7740 20 client.16512 trim_cache size 0 max 16384 2024-04-21T08:48:36.843 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:48:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/913356520 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e40a3ec0 msgr2=0x7fb0e40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e40a3ec0 0x7fb0e40a6380 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0004af0 tx=0x7fb0f0005c50 comp rx=0 tx=0).stop 2024-04-21T08:48:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/913356520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f8107250 msgr2=0x7fb0f8198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f8107250 0x7fb0f8198c40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800c970 tx=0x7fb0e800ce30 comp rx=0 tx=0).stop 2024-04-21T08:48:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/913356520 shutdown_connections 2024-04-21T08:48:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e40a3ec0 0x7fb0e40a6380 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0f819cf70 0x7fb0f819d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f8107250 0x7fb0f8198c40 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 --2- 172.21.15.7:0/913356520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0f8102180 0x7fb0f8198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/913356520 >> 172.21.15.7:0/913356520 conn(0x7fb0f80fda00 msgr2=0x7fb0f80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/913356520 shutdown_connections 2024-04-21T08:48:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:36.833+0000 7fb0ff5b2640 1 -- 172.21.15.7:0/913356520 wait complete. 2024-04-21T08:48:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:36 smithi007 ceph-mon[28477]: pgmap v121: 129 pgs: 129 active+clean; 559 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 16 MiB/s wr, 157 op/s 2024-04-21T08:48:37.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:36 smithi016 ceph-mon[31259]: pgmap v121: 129 pgs: 129 active+clean; 559 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 16 MiB/s wr, 157 op/s 2024-04-21T08:48:37.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:36 smithi077 ceph-mon[31615]: pgmap v121: 129 pgs: 129 active+clean; 559 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 16 MiB/s wr, 157 op/s 2024-04-21T08:48:37.846 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:48:37.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:37.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:48:37.873 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:37 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3498194319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:37.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.966+0000 7f0f13d59640 1 -- 172.21.15.7:0/2150774054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f0c104480 msgr2=0x7f0f0c104860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.966+0000 7f0f13d59640 1 --2- 172.21.15.7:0/2150774054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f0c104480 0x7f0f0c104860 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009930 tx=0x7f0f0802bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.966+0000 7f0f13d59640 1 -- 172.21.15.7:0/2150774054 shutdown_connections 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.966+0000 7f0f13d59640 1 --2- 172.21.15.7:0/2150774054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c106700 0x7f0f0c111100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.966+0000 7f0f13d59640 1 --2- 172.21.15.7:0/2150774054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f0c105d60 0x7f0f0c1061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.966+0000 7f0f13d59640 1 --2- 172.21.15.7:0/2150774054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f0c104480 0x7f0f0c104860 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.966+0000 7f0f13d59640 1 -- 172.21.15.7:0/2150774054 >> 172.21.15.7:0/2150774054 conn(0x7f0f0c0ffbe0 msgr2=0x7f0f0c102000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.966+0000 7f0f13d59640 1 -- 172.21.15.7:0/2150774054 shutdown_connections 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.966+0000 7f0f13d59640 1 -- 172.21.15.7:0/2150774054 wait complete. 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.967+0000 7f0f13d59640 1 Processor -- start 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.967+0000 7f0f13d59640 1 -- start start 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.967+0000 7f0f13d59640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c104480 0x7f0f0c1a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.967+0000 7f0f13d59640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f0c105d60 0x7f0f0c1a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.967+0000 7f0f13d59640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f0c106700 0x7f0f0c19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.967+0000 7f0f13d59640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c114500 con 0x7f0f0c106700 2024-04-21T08:48:37.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.967+0000 7f0f13d59640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c114380 con 0x7f0f0c104480 2024-04-21T08:48:37.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.967+0000 7f0f13d59640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c114680 con 0x7f0f0c105d60 2024-04-21T08:48:37.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.967+0000 7f0f122f6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f0c106700 0x7f0f0c19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.967+0000 7f0f11af5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c104480 0x7f0f0c1a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.967+0000 7f0f122f6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f0c106700 0x7f0f0c19a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55674/0 (socket says 172.21.15.7:55674) 2024-04-21T08:48:37.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f122f6640 1 -- 172.21.15.7:0/1567081638 learned_addr learned my addr 172.21.15.7:0/1567081638 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:37.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f112f4640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f0c105d60 0x7f0f0c1a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f122f6640 1 -- 172.21.15.7:0/1567081638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f0c105d60 msgr2=0x7f0f0c1a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:37.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f122f6640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f0c105d60 0x7f0f0c1a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f122f6640 1 -- 172.21.15.7:0/1567081638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c104480 msgr2=0x7f0f0c1a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f122f6640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c104480 0x7f0f0c1a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f122f6640 1 -- 172.21.15.7:0/1567081638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0c19b000 con 0x7f0f0c106700 2024-04-21T08:48:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f112f4640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f0c105d60 0x7f0f0c1a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f122f6640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f0c106700 0x7f0f0c19a880 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400eb10 tx=0x7f0f0400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f02ffd640 1 -- 172.21.15.7:0/1567081638 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f04004b20 con 0x7f0f0c106700 2024-04-21T08:48:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f13d59640 1 -- 172.21.15.7:0/1567081638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0c19b2f0 con 0x7f0f0c106700 2024-04-21T08:48:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f02ffd640 1 -- 172.21.15.7:0/1567081638 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f04004cc0 con 0x7f0f0c106700 2024-04-21T08:48:37.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f02ffd640 1 -- 172.21.15.7:0/1567081638 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f04010670 con 0x7f0f0c106700 2024-04-21T08:48:37.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.968+0000 7f0f13d59640 1 -- 172.21.15.7:0/1567081638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0c1a7590 con 0x7f0f0c106700 2024-04-21T08:48:37.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.971+0000 7f0f02ffd640 1 -- 172.21.15.7:0/1567081638 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0f04005020 con 0x7f0f0c106700 2024-04-21T08:48:37.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.971+0000 7f0f16a1f740 1 Processor -- start 2024-04-21T08:48:37.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.971+0000 7f0f16a1f740 1 -- start start 2024-04-21T08:48:37.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 0x556502997cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 0x5565029a0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029a0e10 0x5565029a3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5565028fbf00 con 0x5565029978e0 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556502899590 con 0x556502998290 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5565028f7ec0 con 0x5565029a0e10 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f02ffd640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0ef80a3d20 0x7f0ef80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f112f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 0x556502997cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f11af5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 0x5565029a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f112f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 0x556502997cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55676/0 (socket says 172.21.15.7:55676) 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f11af5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 0x5565029a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:37072/0 (socket says 172.21.15.7:37072) 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f112f4640 1 -- 172.21.15.7:0/2262358693 learned_addr learned my addr 172.21.15.7:0/2262358693 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f122f6640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0ef80a3d20 0x7f0ef80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f112f4640 1 --2- 172.21.15.7:0/2262358693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029a0e10 0x5565029a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f02ffd640 1 -- 172.21.15.7:0/1567081638 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f0f040d1210 con 0x7f0f0c106700 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f11af5640 1 -- 172.21.15.7:0/2262358693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029a0e10 msgr2=0x5565029a3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:37.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f11af5640 1 --2- 172.21.15.7:0/2262358693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029a0e10 0x5565029a3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f11af5640 1 -- 172.21.15.7:0/2262358693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 msgr2=0x556502997cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f11af5640 1 --2- 172.21.15.7:0/2262358693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 0x556502997cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f11af5640 1 -- 172.21.15.7:0/2262358693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565026d67e0 con 0x556502998290 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f122f6640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0ef80a3d20 0x7f0ef80a61e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0f0409cd50 tx=0x7f0f0400efd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.972+0000 7f0f11af5640 1 --2- 172.21.15.7:0/2262358693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 0x5565029a0840 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009230 tx=0x7f0f0802c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=202bba18acdef8e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.973+0000 7f0ef7fff640 1 -- 172.21.15.7:0/2262358693 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f08045020 con 0x556502998290 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.973+0000 7f0ef7fff640 1 -- 172.21.15.7:0/2262358693 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f0802c7e0 con 0x556502998290 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.973+0000 7f0ef7fff640 1 -- 172.21.15.7:0/2262358693 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f08005570 con 0x556502998290 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.973+0000 7f0f16a1f740 1 -- 172.21.15.7:0/2262358693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 msgr2=0x5565029a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.973+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/2262358693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 0x5565029a0840 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009230 tx=0x7f0f0802c5d0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.974+0000 7f0f16a1f740 1 -- 172.21.15.7:0/2262358693 shutdown_connections 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.974+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/2262358693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029a0e10 0x5565029a3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.974+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/2262358693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 0x5565029a0840 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.974+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/2262358693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 0x556502997cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.974+0000 7f0f16a1f740 1 -- 172.21.15.7:0/2262358693 >> 172.21.15.7:0/2262358693 conn(0x556502991fb0 msgr2=0x556502992390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.974+0000 7f0f16a1f740 1 -- 172.21.15.7:0/2262358693 shutdown_connections 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.974+0000 7f0f16a1f740 1 -- 172.21.15.7:0/2262358693 wait complete. 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.974+0000 7f0f16a1f740 1 -- start start 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.974+0000 7f0f16a1f740 1 Processor -- start 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029978e0 0x55650299de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502998290 0x55650299e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565029a0e10 0x55650299e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5565029a3d20 con 0x556502998290 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5565028b5580 con 0x5565029a0e10 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5565029a3ea0 con 0x5565029978e0 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f16a1f740 10 client.0 _ll_register_callbacks cb 0x5565028bb650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:37.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f11af5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502998290 0x55650299e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f11af5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502998290 0x55650299e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55692/0 (socket says 172.21.15.7:55692) 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f11af5640 1 -- 172.21.15.7:0/1885884270 learned_addr learned my addr 172.21.15.7:0/1885884270 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f112f4640 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029978e0 0x55650299de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f112f4640 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565029a0e10 0x55650299e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f11af5640 1 -- 172.21.15.7:0/1885884270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029978e0 msgr2=0x55650299de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f11af5640 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029978e0 0x55650299de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f11af5640 1 -- 172.21.15.7:0/1885884270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565029a0e10 msgr2=0x55650299e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f11af5640 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565029a0e10 0x55650299e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f11af5640 1 -- 172.21.15.7:0/1885884270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556502736f70 con 0x556502998290 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f11af5640 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502998290 0x55650299e350 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f0f0802c170 tx=0x7f0f08004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4764a89526a70c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0ef57fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1885884270 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f08045040 con 0x556502998290 2024-04-21T08:48:37.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1885884270 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f0f080035f0 con 0x556502998290 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.975+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1885884270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f0802c7e0 con 0x556502998290 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1885884270 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0f0805a050 con 0x556502998290 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0f16a1f740 10 client.16524 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1885884270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5565025ce170 con 0x556502998290 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1885884270 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f0802c7e0 con 0x556502998290 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1885884270 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f0f080048a0 con 0x556502998290 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0f16a1f740 10 client.16524 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0f16a1f740 10 client.16524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0f16a1f740 20 client.16524 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5565029b11a0 0x5565029b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0f16a1f740 4 client.16524 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1885884270 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556502909920 con 0x5565029b11a0 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.976+0000 7f0f112f4640 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5565029b11a0 0x5565029b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.977+0000 7f0f112f4640 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5565029b11a0 0x5565029b3590 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b4daeb3a4daf9b1 server_cookie=eaed019130536a06 in_seq=0 out_seq=0 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.977+0000 7f0ef57fa640 10 client.16524 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.978+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1885884270 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556502909920 con 0x5565029b11a0 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.978+0000 7f0ef57fa640 10 client.16524 handle_command_reply: tid=0 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.978+0000 7f0f16a1f740 1 client.16524 shutdown 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1885884270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502998290 msgr2=0x55650299e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502998290 0x55650299e350 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f0f0802c170 tx=0x7f0f08004410 comp rx=0 tx=0).stop 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1885884270 shutdown_connections 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5565029b11a0 0x5565029b3590 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5565029a0e10 0x55650299e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502998290 0x55650299e350 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0ef57fa640 0 client.16524 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1885884270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029978e0 0x55650299de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1885884270 >> 172.21.15.7:0/1885884270 conn(0x556502991fb0 msgr2=0x556502993d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:37.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1885884270 shutdown_connections 2024-04-21T08:48:37.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1885884270 wait complete. 2024-04-21T08:48:37.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.979+0000 7f0f16a1f740 20 client.16524 trim_cache size 0 max 16384 2024-04-21T08:48:38.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f16a1f740 1 Processor -- start 2024-04-21T08:48:38.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f16a1f740 1 -- start start 2024-04-21T08:48:38.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502991fb0 0x5565029a0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029978e0 0x5565029acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 0x5565029b8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c1af5d0 con 0x556502991fb0 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c1b1e50 con 0x556502998290 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c1afc40 con 0x5565029978e0 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f112f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502991fb0 0x5565029a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f112f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502991fb0 0x5565029a0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55700/0 (socket says 172.21.15.7:55700) 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f112f4640 1 -- 172.21.15.7:0/2028694291 learned_addr learned my addr 172.21.15.7:0/2028694291 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f112f4640 1 --2- 172.21.15.7:0/2028694291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 0x5565029b8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f11af5640 1 --2- 172.21.15.7:0/2028694291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029978e0 0x5565029acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f112f4640 1 -- 172.21.15.7:0/2028694291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029978e0 msgr2=0x5565029acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f112f4640 1 --2- 172.21.15.7:0/2028694291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029978e0 0x5565029acbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f112f4640 1 -- 172.21.15.7:0/2028694291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 msgr2=0x5565029b8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f112f4640 1 --2- 172.21.15.7:0/2028694291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 0x5565029b8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f112f4640 1 -- 172.21.15.7:0/2028694291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565029999b0 con 0x556502991fb0 2024-04-21T08:48:38.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0f112f4640 1 --2- 172.21.15.7:0/2028694291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502991fb0 0x5565029a0540 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x5565029b3b00 tx=0x7f0efc0073e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57aba687add71bd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.997+0000 7f0ef77fe640 1 -- 172.21.15.7:0/2028694291 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc01e020 con 0x556502991fb0 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0ef77fe640 1 -- 172.21.15.7:0/2028694291 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0efc010040 con 0x556502991fb0 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0ef77fe640 1 -- 172.21.15.7:0/2028694291 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc019cf0 con 0x556502991fb0 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0f16a1f740 1 -- 172.21.15.7:0/2028694291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502991fb0 msgr2=0x5565029a0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/2028694291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502991fb0 0x5565029a0540 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x5565029b3b00 tx=0x7f0efc0073e0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0f16a1f740 1 -- 172.21.15.7:0/2028694291 shutdown_connections 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/2028694291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502998290 0x5565029b8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/2028694291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5565029978e0 0x5565029acbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/2028694291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556502991fb0 0x5565029a0540 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0f16a1f740 1 -- 172.21.15.7:0/2028694291 >> 172.21.15.7:0/2028694291 conn(0x5565029989e0 msgr2=0x556502993c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0f16a1f740 1 -- 172.21.15.7:0/2028694291 shutdown_connections 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0f16a1f740 1 -- 172.21.15.7:0/2028694291 wait complete. 2024-04-21T08:48:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.998+0000 7f0f16a1f740 1 -- start start 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f16a1f740 1 Processor -- start 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502991fb0 0x5565029a9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 0x5565029a5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f16a1f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556502998290 0x5565029a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c1b0310 con 0x5565029978e0 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c1b1780 con 0x556502991fb0 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f16a1f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c1b09e0 con 0x556502998290 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f16a1f740 10 client.0 _ll_register_callbacks cb 0x7f0efc007240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f11af5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 0x5565029a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f112f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502991fb0 0x5565029a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f112f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502991fb0 0x5565029a9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:37112/0 (socket says 172.21.15.7:37112) 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f112f4640 1 -- 172.21.15.7:0/1782897351 learned_addr learned my addr 172.21.15.7:0/1782897351 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f11af5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 0x5565029a5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55710/0 (socket says 172.21.15.7:55710) 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f112f4640 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556502998290 0x5565029a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f11af5640 1 -- 172.21.15.7:0/1782897351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556502998290 msgr2=0x5565029a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f11af5640 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556502998290 0x5565029a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:37.999+0000 7f0f11af5640 1 -- 172.21.15.7:0/1782897351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502991fb0 msgr2=0x5565029a9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f11af5640 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502991fb0 0x5565029a9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f11af5640 1 -- 172.21.15.7:0/1782897351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565029a6140 con 0x5565029978e0 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f112f4640 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502991fb0 0x5565029a9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f11af5640 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 0x5565029a5640 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0f08000c00 tx=0x7f0f08005160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8acd4f4d2f2bf09 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0ef57fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1782897351 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f08045040 con 0x5565029978e0 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1782897351 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f0f080046c0 con 0x5565029978e0 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1782897351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0efc010040 con 0x5565029978e0 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1782897351 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0f0802cca0 con 0x5565029978e0 2024-04-21T08:48:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f16a1f740 10 client.16530 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1782897351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556502a181b0 con 0x5565029978e0 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1782897351 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0efc010040 con 0x5565029978e0 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1782897351 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f0f08041930 con 0x5565029978e0 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f16a1f740 10 client.16530 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f16a1f740 10 client.16530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f16a1f740 20 client.16530 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5565029a0e10 0x556502a1a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f16a1f740 4 client.16530 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.000+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1782897351 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556502903790 con 0x5565029a0e10 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.001+0000 7f0f112f4640 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5565029a0e10 0x556502a1a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.001+0000 7f0f112f4640 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5565029a0e10 0x556502a1a930 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=620bd03d62fb1572 server_cookie=3d750c4c292175f4 in_seq=0 out_seq=0 2024-04-21T08:48:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.001+0000 7f0ef57fa640 10 client.16530 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:38.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0ef57fa640 1 -- 172.21.15.7:0/1782897351 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556502903790 con 0x5565029a0e10 2024-04-21T08:48:38.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0ef57fa640 10 client.16530 handle_command_reply: tid=0 2024-04-21T08:48:38.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 1 client.16530 shutdown 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1782897351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 msgr2=0x5565029a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 0x5565029a5640 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0f08000c00 tx=0x7f0f08005160 comp rx=0 tx=0).stop 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1782897351 shutdown_connections 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5565029a0e10 0x556502a1a930 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556502998290 0x5565029a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5565029978e0 0x5565029a5640 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0ef57fa640 0 client.16530 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 1 --2- 172.21.15.7:0/1782897351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556502991fb0 0x5565029a9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1782897351 >> 172.21.15.7:0/1782897351 conn(0x5565029989e0 msgr2=0x556502993250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1782897351 shutdown_connections 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 1 -- 172.21.15.7:0/1782897351 wait complete. 2024-04-21T08:48:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.044+0000 7f0f16a1f740 20 client.16530 trim_cache size 0 max 16384 2024-04-21T08:48:38.046 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:48:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:48:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "3de3c1e6-eeba-4558-b224-ce91f2eb28ea", 2024-04-21T08:48:38.046 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:48:38.046 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:48:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.045+0000 7f0f13d59640 1 -- 172.21.15.7:0/1567081638 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0ef80a3d20 msgr2=0x7f0ef80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.045+0000 7f0f13d59640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0ef80a3d20 0x7f0ef80a61e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0f0409cd50 tx=0x7f0f0400efd0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.045+0000 7f0f13d59640 1 -- 172.21.15.7:0/1567081638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f0c106700 msgr2=0x7f0f0c19a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.045+0000 7f0f13d59640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f0c106700 0x7f0f0c19a880 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400eb10 tx=0x7f0f0400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.045+0000 7f0f13d59640 1 -- 172.21.15.7:0/1567081638 shutdown_connections 2024-04-21T08:48:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.045+0000 7f0f13d59640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f0c106700 0x7f0f0c19a880 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.045+0000 7f0f13d59640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0ef80a3d20 0x7f0ef80a61e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.045+0000 7f0f13d59640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f0c105d60 0x7f0f0c1a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.045+0000 7f0f13d59640 1 --2- 172.21.15.7:0/1567081638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f0c104480 0x7f0f0c1a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.045+0000 7f0f13d59640 1 -- 172.21.15.7:0/1567081638 >> 172.21.15.7:0/1567081638 conn(0x7f0f0c0ffbe0 msgr2=0x7f0f0c100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.045+0000 7f0f13d59640 1 -- 172.21.15.7:0/1567081638 shutdown_connections 2024-04-21T08:48:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.046+0000 7f0f13d59640 1 -- 172.21.15.7:0/1567081638 wait complete. 2024-04-21T08:48:38.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:37 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3498194319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:38.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:48:38.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:48:38.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:37 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3498194319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.203+0000 7fdbd90f2640 1 -- 172.21.15.7:0/1865200976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd4101050 msgr2=0x7fdbd4101430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.203+0000 7fdbd90f2640 1 --2- 172.21.15.7:0/1865200976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd4101050 0x7fdbd4101430 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009900 tx=0x7fdbc402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.203+0000 7fdbd90f2640 1 -- 172.21.15.7:0/1865200976 shutdown_connections 2024-04-21T08:48:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.203+0000 7fdbd90f2640 1 --2- 172.21.15.7:0/1865200976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd4106120 0x7fdbd410dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.203+0000 7fdbd90f2640 1 --2- 172.21.15.7:0/1865200976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd4101a00 0x7fdbd4105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.203+0000 7fdbd90f2640 1 --2- 172.21.15.7:0/1865200976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd4101050 0x7fdbd4101430 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.203+0000 7fdbd90f2640 1 -- 172.21.15.7:0/1865200976 >> 172.21.15.7:0/1865200976 conn(0x7fdbd4077fa0 msgr2=0x7fdbd40783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.203+0000 7fdbd90f2640 1 -- 172.21.15.7:0/1865200976 shutdown_connections 2024-04-21T08:48:38.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.203+0000 7fdbd90f2640 1 -- 172.21.15.7:0/1865200976 wait complete. 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbd90f2640 1 Processor -- start 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbd90f2640 1 -- start start 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbd90f2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd4101050 0x7fdbd410e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbd90f2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd4106120 0x7fdbd410ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbd90f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd410f100 0x7fdbd41a6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbd90f2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41121a0 con 0x7fdbd4106120 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbd90f2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd4112020 con 0x7fdbd410f100 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbd90f2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd4112320 con 0x7fdbd4101050 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbcbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd4106120 0x7fdbd410ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbcbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd4106120 0x7fdbd410ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55730/0 (socket says 172.21.15.7:55730) 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbcbfff640 1 -- 172.21.15.7:0/251925375 learned_addr learned my addr 172.21.15.7:0/251925375 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbd359e640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd410f100 0x7fdbd41a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbd2d9d640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd4101050 0x7fdbd410e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbcbfff640 1 -- 172.21.15.7:0/251925375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd4101050 msgr2=0x7fdbd410e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbcbfff640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd4101050 0x7fdbd410e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbcbfff640 1 -- 172.21.15.7:0/251925375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd410f100 msgr2=0x7fdbd41a6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbcbfff640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd410f100 0x7fdbd41a6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbcbfff640 1 -- 172.21.15.7:0/251925375 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd41a7030 con 0x7fdbd4106120 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.204+0000 7fdbd2d9d640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd4101050 0x7fdbd410e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.205+0000 7fdbcbfff640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd4106120 0x7fdbd410ebc0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00e9c0 tx=0x7fdbbc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.205+0000 7fdbd0d99640 1 -- 172.21.15.7:0/251925375 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc00cd80 con 0x7fdbd4106120 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.205+0000 7fdbd90f2640 1 -- 172.21.15.7:0/251925375 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd41a7320 con 0x7fdbd4106120 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.205+0000 7fdbd0d99640 1 -- 172.21.15.7:0/251925375 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbbc0054d0 con 0x7fdbd4106120 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.205+0000 7fdbd0d99640 1 -- 172.21.15.7:0/251925375 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc010430 con 0x7fdbd4106120 2024-04-21T08:48:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.205+0000 7fdbd90f2640 1 -- 172.21.15.7:0/251925375 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd41a78c0 con 0x7fdbd4106120 2024-04-21T08:48:38.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.207+0000 7fdbd0d99640 1 -- 172.21.15.7:0/251925375 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fdbbc004f90 con 0x7fdbd4106120 2024-04-21T08:48:38.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.207+0000 7fdbdbdc1740 1 Processor -- start 2024-04-21T08:48:38.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.207+0000 7fdbdbdc1740 1 -- start start 2024-04-21T08:48:38.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee04960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04ea0 0x55700ee0d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee0da80 0x55700ee0fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55700ed691e0 con 0x55700ee04580 2024-04-21T08:48:38.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55700ed06390 con 0x55700ee04ea0 2024-04-21T08:48:38.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55700ed651a0 con 0x55700ee0da80 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee04960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd0d99640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdbc00a3dd0 0x7fdbc00a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee04960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55740/0 (socket says 172.21.15.7:55740) 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd359e640 1 -- 172.21.15.7:0/3432760761 learned_addr learned my addr 172.21.15.7:0/3432760761 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04ea0 0x55700ee0d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd359e640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdbc00a3dd0 0x7fdbc00a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd0d99640 1 -- 172.21.15.7:0/251925375 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fdbbc0d0fb0 con 0x7fdbd4106120 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbcbfff640 1 --2- 172.21.15.7:0/3432760761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee0da80 0x55700ee0fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd359e640 1 -- 172.21.15.7:0/3432760761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee0da80 msgr2=0x55700ee0fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd359e640 1 --2- 172.21.15.7:0/3432760761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee0da80 0x55700ee0fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd359e640 1 -- 172.21.15.7:0/3432760761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04ea0 msgr2=0x55700ee0d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd359e640 1 --2- 172.21.15.7:0/3432760761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04ea0 0x55700ee0d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd359e640 1 -- 172.21.15.7:0/3432760761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55700eb457d0 con 0x55700ee04580 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbcbfff640 1 --2- 172.21.15.7:0/3432760761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee0da80 0x55700ee0fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.208+0000 7fdbd2d9d640 1 --2- 172.21.15.7:0/3432760761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04ea0 0x55700ee0d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.209+0000 7fdbd359e640 1 --2- 172.21.15.7:0/3432760761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee04960 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc014a60 tx=0x7fdbcc014f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172243693866d261 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.209+0000 7fdbd359e640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdbc00a3dd0 0x7fdbc00a6290 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc01c000 tx=0x7fdbcc01e000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.209+0000 7fdbc97fa640 1 -- 172.21.15.7:0/3432760761 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc012e10 con 0x55700ee04580 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.209+0000 7fdbc97fa640 1 -- 172.21.15.7:0/3432760761 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbcc00ebc0 con 0x55700ee04580 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.209+0000 7fdbc97fa640 1 -- 172.21.15.7:0/3432760761 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc0056d0 con 0x55700ee04580 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/3432760761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 msgr2=0x55700ee04960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/3432760761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee04960 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc014a60 tx=0x7fdbcc014f20 comp rx=0 tx=0).stop 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/3432760761 shutdown_connections 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/3432760761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee0da80 0x55700ee0fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/3432760761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04ea0 0x55700ee0d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/3432760761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee04960 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/3432760761 >> 172.21.15.7:0/3432760761 conn(0x55700edfecb0 msgr2=0x55700edff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/3432760761 shutdown_connections 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/3432760761 wait complete. 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 -- start start 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 Processor -- start 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04580 0x55700ee0ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04ea0 0x55700ee0b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee0b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55700ee10810 con 0x55700ee0da80 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55700ed222f0 con 0x55700ee04ea0 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55700ee10990 con 0x55700ee04580 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbdbdc1740 10 client.0 _ll_register_callbacks cb 0x55700ed2f580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:38.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.210+0000 7fdbd359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee0b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbcbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04ea0 0x55700ee0b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbd359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee0b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55754/0 (socket says 172.21.15.7:55754) 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbd359e640 1 -- 172.21.15.7:0/1121445831 learned_addr learned my addr 172.21.15.7:0/1121445831 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbd2d9d640 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04580 0x55700ee0ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbd359e640 1 -- 172.21.15.7:0/1121445831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04580 msgr2=0x55700ee0ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbd359e640 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04580 0x55700ee0ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbd359e640 1 -- 172.21.15.7:0/1121445831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04ea0 msgr2=0x55700ee0b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbd359e640 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04ea0 0x55700ee0b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbd359e640 1 -- 172.21.15.7:0/1121445831 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55700ea7b3f0 con 0x55700ee0da80 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbd2d9d640 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04580 0x55700ee0ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbd359e640 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee0b5a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0052b0 tx=0x7fdbcc00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b9aab16205e8b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1121445831 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc035060 con 0x55700ee0da80 2024-04-21T08:48:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1121445831 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fdbcc012e10 con 0x55700ee0da80 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1121445831 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbcc00ebc0 con 0x55700ee0da80 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1121445831 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdbcc0049a0 con 0x55700ee0da80 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbdbdc1740 10 client.16542 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1121445831 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55700ea3b0f0 con 0x55700ee0da80 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.211+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1121445831 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbcc00ebc0 con 0x55700ee0da80 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.212+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1121445831 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fdbcc02f030 con 0x55700ee0da80 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.212+0000 7fdbdbdc1740 10 client.16542 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.212+0000 7fdbdbdc1740 10 client.16542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.212+0000 7fdbdbdc1740 20 client.16542 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.212+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55700ee1d980 0x55700ee1fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.212+0000 7fdbdbdc1740 4 client.16542 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.212+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1121445831 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55700ed76c00 con 0x55700ee1d980 2024-04-21T08:48:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.212+0000 7fdbd2d9d640 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55700ee1d980 0x55700ee1fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.212+0000 7fdbba7fc640 10 client.16542 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:38.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.212+0000 7fdbd2d9d640 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55700ee1d980 0x55700ee1fd70 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce1582861405b09d server_cookie=651f940c63e5647 in_seq=0 out_seq=0 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.214+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1121445831 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55700ed76c00 con 0x55700ee1d980 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.214+0000 7fdbba7fc640 10 client.16542 handle_command_reply: tid=0 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.214+0000 7fdbdbdc1740 1 client.16542 shutdown 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1121445831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 msgr2=0x55700ee0b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee0b5a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0052b0 tx=0x7fdbcc00ee00 comp rx=0 tx=0).stop 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1121445831 shutdown_connections 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55700ee1d980 0x55700ee1fd70 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee0b5a0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04ea0 0x55700ee0b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1121445831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04580 0x55700ee0ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbba7fc640 0 client.16542 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1121445831 >> 172.21.15.7:0/1121445831 conn(0x55700edfecb0 msgr2=0x55700ee00a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:38.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1121445831 shutdown_connections 2024-04-21T08:48:38.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1121445831 wait complete. 2024-04-21T08:48:38.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.215+0000 7fdbdbdc1740 20 client.16542 trim_cache size 0 max 16384 2024-04-21T08:48:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.232+0000 7fdbdbdc1740 1 Processor -- start 2024-04-21T08:48:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.232+0000 7fdbdbdc1740 1 -- start start 2024-04-21T08:48:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04580 0x55700ee0d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04ea0 0x55700ee196d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee254e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41af6c0 con 0x55700ee0da80 2024-04-21T08:48:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41b1f40 con 0x55700ee04580 2024-04-21T08:48:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41afd30 con 0x55700ee04ea0 2024-04-21T08:48:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbd359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee254e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbd359e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee254e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55770/0 (socket says 172.21.15.7:55770) 2024-04-21T08:48:38.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbd359e640 1 -- 172.21.15.7:0/1343096868 learned_addr learned my addr 172.21.15.7:0/1343096868 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbd2d9d640 1 --2- 172.21.15.7:0/1343096868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04580 0x55700ee0d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbcbfff640 1 --2- 172.21.15.7:0/1343096868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04ea0 0x55700ee196d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbd359e640 1 -- 172.21.15.7:0/1343096868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04ea0 msgr2=0x55700ee196d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbd359e640 1 --2- 172.21.15.7:0/1343096868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04ea0 0x55700ee196d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbd359e640 1 -- 172.21.15.7:0/1343096868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04580 msgr2=0x55700ee0d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbd359e640 1 --2- 172.21.15.7:0/1343096868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04580 0x55700ee0d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbd359e640 1 -- 172.21.15.7:0/1343096868 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55700ee04cd0 con 0x55700ee0da80 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbcbfff640 1 --2- 172.21.15.7:0/1343096868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04ea0 0x55700ee196d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.233+0000 7fdbd359e640 1 --2- 172.21.15.7:0/1343096868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee254e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0265b0 tx=0x7fdbcc026460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8959059245a3f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbc8ff9640 1 -- 172.21.15.7:0/1343096868 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc035040 con 0x55700ee0da80 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbc8ff9640 1 -- 172.21.15.7:0/1343096868 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbcc0050a0 con 0x55700ee0da80 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbc8ff9640 1 -- 172.21.15.7:0/1343096868 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc012e70 con 0x55700ee0da80 2024-04-21T08:48:38.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1343096868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 msgr2=0x55700ee254e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1343096868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee254e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0265b0 tx=0x7fdbcc026460 comp rx=0 tx=0).stop 2024-04-21T08:48:38.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1343096868 shutdown_connections 2024-04-21T08:48:38.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1343096868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee0da80 0x55700ee254e0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1343096868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04ea0 0x55700ee196d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1343096868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee04580 0x55700ee0d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1343096868 >> 172.21.15.7:0/1343096868 conn(0x55700edfecb0 msgr2=0x55700ee0b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:38.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1343096868 shutdown_connections 2024-04-21T08:48:38.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.234+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1343096868 wait complete. 2024-04-21T08:48:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbdbdc1740 1 -- start start 2024-04-21T08:48:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbdbdc1740 1 Processor -- start 2024-04-21T08:48:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee11d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04ea0 0x55700ee12240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbdbdc1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee0da80 0x55700ee82d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41b0400 con 0x55700ee04580 2024-04-21T08:48:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41b1870 con 0x55700ee0da80 2024-04-21T08:48:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbdbdc1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41b0ad0 con 0x55700ee04ea0 2024-04-21T08:48:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbdbdc1740 10 client.0 _ll_register_callbacks cb 0x7fdbbc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbd2d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee11d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbd2d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee11d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55784/0 (socket says 172.21.15.7:55784) 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbd2d9d640 1 -- 172.21.15.7:0/1599217079 learned_addr learned my addr 172.21.15.7:0/1599217079 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbd359e640 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee0da80 0x55700ee82d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbcbfff640 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04ea0 0x55700ee12240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbd2d9d640 1 -- 172.21.15.7:0/1599217079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04ea0 msgr2=0x55700ee12240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbd2d9d640 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04ea0 0x55700ee12240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbd2d9d640 1 -- 172.21.15.7:0/1599217079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee0da80 msgr2=0x55700ee82d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbd2d9d640 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee0da80 0x55700ee82d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.235+0000 7fdbd2d9d640 1 -- 172.21.15.7:0/1599217079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55700ee83290 con 0x55700ee04580 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbd2d9d640 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee11d00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009230 tx=0x7fdbc40045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4442a7e0572d735d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1599217079 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc404f020 con 0x55700ee04580 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1599217079 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fdbc402cab0 con 0x55700ee04580 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1599217079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbcc0050a0 con 0x55700ee04580 2024-04-21T08:48:38.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1599217079 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdbc4057050 con 0x55700ee04580 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbdbdc1740 10 client.16548 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1599217079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55700ee844a0 con 0x55700ee04580 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1599217079 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbcc0050a0 con 0x55700ee04580 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1599217079 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fdbc4004b00 con 0x55700ee04580 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbdbdc1740 10 client.16548 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbdbdc1740 10 client.16548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbdbdc1740 20 client.16548 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55700ee84bf0 0x55700ee87010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbdbdc1740 4 client.16548 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.236+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1599217079 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55700ed70a70 con 0x55700ee84bf0 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.237+0000 7fdbcbfff640 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55700ee84bf0 0x55700ee87010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.237+0000 7fdbcbfff640 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55700ee84bf0 0x55700ee87010 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33735b73672541f6 server_cookie=c13180bead340939 in_seq=0 out_seq=0 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.237+0000 7fdbba7fc640 10 client.16548 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:38.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbba7fc640 1 -- 172.21.15.7:0/1599217079 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x55700ed70a70 con 0x55700ee84bf0 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbba7fc640 10 client.16548 handle_command_reply: tid=0 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 1 client.16548 shutdown 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1599217079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 msgr2=0x55700ee11d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee11d00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009230 tx=0x7fdbc40045a0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1599217079 shutdown_connections 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55700ee84bf0 0x55700ee87010 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55700ee0da80 0x55700ee82d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55700ee04ea0 0x55700ee12240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbba7fc640 0 client.16548 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 1 --2- 172.21.15.7:0/1599217079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55700ee04580 0x55700ee11d00 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1599217079 >> 172.21.15.7:0/1599217079 conn(0x55700edfecb0 msgr2=0x55700edfffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1599217079 shutdown_connections 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 1 -- 172.21.15.7:0/1599217079 wait complete. 2024-04-21T08:48:38.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.239+0000 7fdbdbdc1740 20 client.16548 trim_cache size 0 max 16384 2024-04-21T08:48:38.244 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:48:38.244 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (18 inodes in the stack)", 2024-04-21T08:48:38.244 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:48:38.244 INFO:teuthology.orchestra.run.smithi007.stdout: "3de3c1e6-eeba-4558-b224-ce91f2eb28ea": { 2024-04-21T08:48:38.244 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:48:38.244 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "3de3c1e6-eeba-4558-b224-ce91f2eb28ea", 2024-04-21T08:48:38.244 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:48:38.244 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:48:38.244 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:48:38.244 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:48:38.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 -- 172.21.15.7:0/251925375 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdbc00a3dd0 msgr2=0x7fdbc00a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdbc00a3dd0 0x7fdbc00a6290 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc01c000 tx=0x7fdbcc01e000 comp rx=0 tx=0).stop 2024-04-21T08:48:38.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 -- 172.21.15.7:0/251925375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd4106120 msgr2=0x7fdbd410ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:38.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd4106120 0x7fdbd410ebc0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00e9c0 tx=0x7fdbbc00ee80 comp rx=0 tx=0).stop 2024-04-21T08:48:38.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 -- 172.21.15.7:0/251925375 shutdown_connections 2024-04-21T08:48:38.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd410f100 0x7fdbd41a6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd4106120 0x7fdbd410ebc0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdbc00a3dd0 0x7fdbc00a6290 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 --2- 172.21.15.7:0/251925375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd4101050 0x7fdbd410e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:38.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 -- 172.21.15.7:0/251925375 >> 172.21.15.7:0/251925375 conn(0x7fdbd4077fa0 msgr2=0x7fdbd40ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:38.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 -- 172.21.15.7:0/251925375 shutdown_connections 2024-04-21T08:48:38.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:48:38.240+0000 7fdbd90f2640 1 -- 172.21.15.7:0/251925375 wait complete. 2024-04-21T08:48:38.250 INFO:tasks.cephfs.filesystem:scrub status for tag:3de3c1e6-eeba-4558-b224-ce91f2eb28ea - {'path': '/', 'tag': '3de3c1e6-eeba-4558-b224-ce91f2eb28ea', 'options': 'recursive,force'} 2024-04-21T08:48:38.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:38 smithi007 ceph-mon[28477]: pgmap v122: 129 pgs: 129 active+clean; 559 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 16 MiB/s wr, 157 op/s 2024-04-21T08:48:38.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:38 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:48:38.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:38 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:38.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:38 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:48:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:38 smithi016 ceph-mon[31259]: pgmap v122: 129 pgs: 129 active+clean; 559 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 16 MiB/s wr, 157 op/s 2024-04-21T08:48:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:38 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:48:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:38 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:38 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:48:39.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:38 smithi077 ceph-mon[31615]: pgmap v122: 129 pgs: 129 active+clean; 559 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 16 MiB/s wr, 157 op/s 2024-04-21T08:48:39.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:38 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:48:39.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:38 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:39.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:38 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:48:40.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:40 smithi007 ceph-mon[28477]: pgmap v123: 129 pgs: 129 active+clean; 722 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 32 MiB/s wr, 1.51k op/s 2024-04-21T08:48:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:40 smithi016 ceph-mon[31259]: pgmap v123: 129 pgs: 129 active+clean; 722 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 32 MiB/s wr, 1.51k op/s 2024-04-21T08:48:41.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:40 smithi077 ceph-mon[31615]: pgmap v123: 129 pgs: 129 active+clean; 722 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 32 MiB/s wr, 1.51k op/s 2024-04-21T08:48:42.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:42 smithi007 ceph-mon[28477]: pgmap v124: 129 pgs: 129 active+clean; 722 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 16 MiB/s wr, 1.38k op/s 2024-04-21T08:48:43.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:42 smithi016 ceph-mon[31259]: pgmap v124: 129 pgs: 129 active+clean; 722 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 16 MiB/s wr, 1.38k op/s 2024-04-21T08:48:43.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:42 smithi077 ceph-mon[31615]: pgmap v124: 129 pgs: 129 active+clean; 722 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 16 MiB/s wr, 1.38k op/s 2024-04-21T08:48:44.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:44 smithi007 ceph-mon[28477]: pgmap v125: 129 pgs: 129 active+clean; 732 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 17 MiB/s wr, 3.61k op/s 2024-04-21T08:48:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:44 smithi016 ceph-mon[31259]: pgmap v125: 129 pgs: 129 active+clean; 732 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 17 MiB/s wr, 3.61k op/s 2024-04-21T08:48:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:44 smithi077 ceph-mon[31615]: pgmap v125: 129 pgs: 129 active+clean; 732 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 17 MiB/s wr, 3.61k op/s 2024-04-21T08:48:46.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:46 smithi007 ceph-mon[28477]: pgmap v126: 129 pgs: 129 active+clean; 732 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 16 MiB/s wr, 3.58k op/s 2024-04-21T08:48:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:46 smithi016 ceph-mon[31259]: pgmap v126: 129 pgs: 129 active+clean; 732 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 16 MiB/s wr, 3.58k op/s 2024-04-21T08:48:47.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:46 smithi077 ceph-mon[31615]: pgmap v126: 129 pgs: 129 active+clean; 732 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 16 MiB/s wr, 3.58k op/s 2024-04-21T08:48:48.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:48 smithi007 ceph-mon[28477]: pgmap v127: 129 pgs: 129 active+clean; 732 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 16 MiB/s wr, 3.58k op/s 2024-04-21T08:48:49.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:48 smithi016 ceph-mon[31259]: pgmap v127: 129 pgs: 129 active+clean; 732 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 16 MiB/s wr, 3.58k op/s 2024-04-21T08:48:49.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:48 smithi077 ceph-mon[31615]: pgmap v127: 129 pgs: 129 active+clean; 732 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 16 MiB/s wr, 3.58k op/s 2024-04-21T08:48:49.407 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.407+0000 7f34ecc11640 1 -- 172.21.15.7:0/1705585161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34e8101120 msgr2=0x7f34e8101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.407+0000 7f34ecc11640 1 --2- 172.21.15.7:0/1705585161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34e8101120 0x7f34e8101500 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009900 tx=0x7f34e002bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.408+0000 7f34ecc11640 1 -- 172.21.15.7:0/1705585161 shutdown_connections 2024-04-21T08:48:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.408+0000 7f34ecc11640 1 --2- 172.21.15.7:0/1705585161 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e81061a0 0x7f34e810dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.408+0000 7f34ecc11640 1 --2- 172.21.15.7:0/1705585161 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e8101ad0 0x7f34e8105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.408+0000 7f34ecc11640 1 --2- 172.21.15.7:0/1705585161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34e8101120 0x7f34e8101500 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.408+0000 7f34ecc11640 1 -- 172.21.15.7:0/1705585161 >> 172.21.15.7:0/1705585161 conn(0x7f34e8077fa0 msgr2=0x7f34e80783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.408+0000 7f34ecc11640 1 -- 172.21.15.7:0/1705585161 shutdown_connections 2024-04-21T08:48:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.408+0000 7f34ecc11640 1 -- 172.21.15.7:0/1705585161 wait complete. 2024-04-21T08:48:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.408+0000 7f34ecc11640 1 Processor -- start 2024-04-21T08:48:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34ecc11640 1 -- start start 2024-04-21T08:48:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34ecc11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e8101120 0x7f34e8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34ecc11640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34e81061a0 0x7f34e8198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34ecc11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e819d000 0x7f34e819d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34ecc11640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f34e8112220 con 0x7f34e81061a0 2024-04-21T08:48:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34ecc11640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34e81120a0 con 0x7f34e8101120 2024-04-21T08:48:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34ecc11640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f34e81123a0 con 0x7f34e819d000 2024-04-21T08:48:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e5d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34e81061a0 0x7f34e8198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e5d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34e81061a0 0x7f34e8198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46922/0 (socket says 172.21.15.7:46922) 2024-04-21T08:48:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e5d9b640 1 -- 172.21.15.7:0/1662995853 learned_addr learned my addr 172.21.15.7:0/1662995853 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e659c640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e8101120 0x7f34e8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e6d9d640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e819d000 0x7f34e819d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e5d9b640 1 -- 172.21.15.7:0/1662995853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e819d000 msgr2=0x7f34e819d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e5d9b640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e819d000 0x7f34e819d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e5d9b640 1 -- 172.21.15.7:0/1662995853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e8101120 msgr2=0x7f34e8198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e5d9b640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e8101120 0x7f34e8198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e5d9b640 1 -- 172.21.15.7:0/1662995853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e819daf0 con 0x7f34e81061a0 2024-04-21T08:48:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e6d9d640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e819d000 0x7f34e819d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.409+0000 7f34e5d9b640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34e81061a0 0x7f34e8198c80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f34d000d770 tx=0x7f34d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.410+0000 7f34df7fe640 1 -- 172.21.15.7:0/1662995853 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0017070 con 0x7f34e81061a0 2024-04-21T08:48:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.410+0000 7f34ecc11640 1 -- 172.21.15.7:0/1662995853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e819dde0 con 0x7f34e81061a0 2024-04-21T08:48:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.410+0000 7f34df7fe640 1 -- 172.21.15.7:0/1662995853 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34d0009c30 con 0x7f34e81061a0 2024-04-21T08:48:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.410+0000 7f34df7fe640 1 -- 172.21.15.7:0/1662995853 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d000f620 con 0x7f34e81061a0 2024-04-21T08:48:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.410+0000 7f34ecc11640 1 -- 172.21.15.7:0/1662995853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e81a55d0 con 0x7f34e81061a0 2024-04-21T08:48:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.412+0000 7f34df7fe640 1 -- 172.21.15.7:0/1662995853 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f34d001a070 con 0x7f34e81061a0 2024-04-21T08:48:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.412+0000 7f34ef801740 1 Processor -- start 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.412+0000 7f34ef801740 1 -- start start 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34ef801740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee38a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34ef801740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee38ab60 0x55b3ee38afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34ef801740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 0x55b3ee3964d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34ef801740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b3ee2ef8d0 con 0x55b3ee3940e0 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34ef801740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b3ee28ce80 con 0x55b3ee38ab60 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34ef801740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b3ee2eb890 con 0x55b3ee38b8d0 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34e5d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 0x55b3ee3964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34df7fe640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34d40a3d10 0x7f34d40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34e5d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 0x55b3ee3964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46926/0 (socket says 172.21.15.7:46926) 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34e659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee38ab60 0x55b3ee38afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34e5d9b640 1 -- 172.21.15.7:0/3797957655 learned_addr learned my addr 172.21.15.7:0/3797957655 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34e6d9d640 1 --2- 172.21.15.7:0/3797957655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee38a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34e6d9d640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34d40a3d10 0x7f34d40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34e659c640 1 -- 172.21.15.7:0/3797957655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 msgr2=0x55b3ee38a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34e659c640 1 --2- 172.21.15.7:0/3797957655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee38a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34e659c640 1 -- 172.21.15.7:0/3797957655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 msgr2=0x55b3ee3964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34e659c640 1 --2- 172.21.15.7:0/3797957655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 0x55b3ee3964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34df7fe640 1 -- 172.21.15.7:0/1662995853 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f34d00d07c0 con 0x7f34e81061a0 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.413+0000 7f34e659c640 1 -- 172.21.15.7:0/3797957655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ee2ae320 con 0x55b3ee38ab60 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.414+0000 7f34e6d9d640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34d40a3d10 0x7f34d40a61d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f34d8005c40 tx=0x7f34d8009200 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.414+0000 7f34e659c640 1 --2- 172.21.15.7:0/3797957655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee38ab60 0x55b3ee38afe0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f34e0000c00 tx=0x7f34e002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82c4c0b3fb7e21b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.414+0000 7f34dcff9640 1 -- 172.21.15.7:0/3797957655 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0045020 con 0x55b3ee38ab60 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.414+0000 7f34dcff9640 1 -- 172.21.15.7:0/3797957655 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e002c7e0 con 0x55b3ee38ab60 2024-04-21T08:48:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.414+0000 7f34dcff9640 1 -- 172.21.15.7:0/3797957655 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e00046c0 con 0x55b3ee38ab60 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 -- 172.21.15.7:0/3797957655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee38ab60 msgr2=0x55b3ee38afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 --2- 172.21.15.7:0/3797957655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee38ab60 0x55b3ee38afe0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f34e0000c00 tx=0x7f34e002c6c0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 -- 172.21.15.7:0/3797957655 shutdown_connections 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 --2- 172.21.15.7:0/3797957655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 0x55b3ee3964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 --2- 172.21.15.7:0/3797957655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee38ab60 0x55b3ee38afe0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 --2- 172.21.15.7:0/3797957655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee38a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 -- 172.21.15.7:0/3797957655 >> 172.21.15.7:0/3797957655 conn(0x55b3ee351c90 msgr2=0x55b3ee385850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 -- 172.21.15.7:0/3797957655 shutdown_connections 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 -- 172.21.15.7:0/3797957655 wait complete. 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 -- start start 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 Processor -- start 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38ab60 0x55b3ee393a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee390e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee3940e0 0x55b3ee3913c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.415+0000 7f34ef801740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b3ee396db0 con 0x55b3ee38ab60 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34ef801740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b3ee277eb0 con 0x55b3ee3940e0 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34ef801740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b3ee396f30 con 0x55b3ee38b8d0 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34ef801740 10 client.0 _ll_register_callbacks cb 0x55b3ee2a4b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34e659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38ab60 0x55b3ee393a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34e659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38ab60 0x55b3ee393a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46940/0 (socket says 172.21.15.7:46940) 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34e659c640 1 -- 172.21.15.7:0/2267175616 learned_addr learned my addr 172.21.15.7:0/2267175616 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34e6d9d640 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee3940e0 0x55b3ee3913c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34e5d9b640 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee390e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34e659c640 1 -- 172.21.15.7:0/2267175616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 msgr2=0x55b3ee390e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34e659c640 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee390e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34e659c640 1 -- 172.21.15.7:0/2267175616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee3940e0 msgr2=0x55b3ee3913c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34e659c640 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee3940e0 0x55b3ee3913c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34e659c640 1 -- 172.21.15.7:0/2267175616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3ee002210 con 0x55b3ee38ab60 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34e659c640 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38ab60 0x55b3ee393a90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f34e002c170 tx=0x7f34e002cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be959405a21229c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34cdffb640 1 -- 172.21.15.7:0/2267175616 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e00035f0 con 0x55b3ee38ab60 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34cdffb640 1 -- 172.21.15.7:0/2267175616 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f34e0003790 con 0x55b3ee38ab60 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.416+0000 7f34ef801740 1 -- 172.21.15.7:0/2267175616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34e002c7e0 con 0x55b3ee38ab60 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34cdffb640 1 -- 172.21.15.7:0/2267175616 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f34e005b050 con 0x55b3ee38ab60 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34ef801740 10 client.16560 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34ef801740 1 -- 172.21.15.7:0/2267175616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b3ee0ce620 con 0x55b3ee38ab60 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34cdffb640 1 -- 172.21.15.7:0/2267175616 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34e002c7e0 con 0x55b3ee38ab60 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34cdffb640 1 -- 172.21.15.7:0/2267175616 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f34e0041bd0 con 0x55b3ee38ab60 2024-04-21T08:48:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34ef801740 10 client.16560 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34ef801740 10 client.16560 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34ef801740 20 client.16560 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34ef801740 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b3ee3a3f20 0x55b3ee3a6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34ef801740 4 client.16560 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34ef801740 1 -- 172.21.15.7:0/2267175616 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3ee303880 con 0x55b3ee3a3f20 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.417+0000 7f34e5d9b640 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b3ee3a3f20 0x55b3ee3a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.418+0000 7f34e5d9b640 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b3ee3a3f20 0x55b3ee3a6310 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2749db15a6760050 server_cookie=333459f217298dc6 in_seq=0 out_seq=0 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.418+0000 7f34cdffb640 10 client.16560 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34cdffb640 1 -- 172.21.15.7:0/2267175616 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3ee303880 con 0x55b3ee3a3f20 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34cdffb640 10 client.16560 handle_command_reply: tid=0 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 1 client.16560 shutdown 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 1 -- 172.21.15.7:0/2267175616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38ab60 msgr2=0x55b3ee393a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38ab60 0x55b3ee393a90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f34e002c170 tx=0x7f34e002cc20 comp rx=0 tx=0).stop 2024-04-21T08:48:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 1 -- 172.21.15.7:0/2267175616 shutdown_connections 2024-04-21T08:48:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b3ee3a3f20 0x55b3ee3a6310 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee3940e0 0x55b3ee3913c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee390e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34cdffb640 0 client.16560 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 1 --2- 172.21.15.7:0/2267175616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38ab60 0x55b3ee393a90 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 1 -- 172.21.15.7:0/2267175616 >> 172.21.15.7:0/2267175616 conn(0x55b3ee351c90 msgr2=0x55b3ee3875a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 1 -- 172.21.15.7:0/2267175616 shutdown_connections 2024-04-21T08:48:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 1 -- 172.21.15.7:0/2267175616 wait complete. 2024-04-21T08:48:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.419+0000 7f34ef801740 20 client.16560 trim_cache size 0 max 16384 2024-04-21T08:48:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34ef801740 1 Processor -- start 2024-04-21T08:48:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34ef801740 1 -- start start 2024-04-21T08:48:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34ef801740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38ab60 0x55b3ee390700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34ef801740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38b8d0 0x55b3ee393550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34ef801740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee3940e0 0x55b3ee393a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34ef801740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f34e81ad420 con 0x55b3ee38b8d0 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34ef801740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34e81afca0 con 0x55b3ee3940e0 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34ef801740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f34e81ada90 con 0x55b3ee38ab60 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34e5d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38b8d0 0x55b3ee393550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34e5d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38b8d0 0x55b3ee393550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46944/0 (socket says 172.21.15.7:46944) 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34e5d9b640 1 -- 172.21.15.7:0/1145869281 learned_addr learned my addr 172.21.15.7:0/1145869281 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.434+0000 7f34e659c640 1 --2- 172.21.15.7:0/1145869281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38ab60 0x55b3ee390700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34e6d9d640 1 --2- 172.21.15.7:0/1145869281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee3940e0 0x55b3ee393a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34e5d9b640 1 -- 172.21.15.7:0/1145869281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38ab60 msgr2=0x55b3ee390700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34e5d9b640 1 --2- 172.21.15.7:0/1145869281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38ab60 0x55b3ee390700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34e5d9b640 1 -- 172.21.15.7:0/1145869281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee3940e0 msgr2=0x55b3ee393a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34e5d9b640 1 --2- 172.21.15.7:0/1145869281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee3940e0 0x55b3ee393a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34e5d9b640 1 -- 172.21.15.7:0/1145869281 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ee385a60 con 0x55b3ee38b8d0 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34e659c640 1 --2- 172.21.15.7:0/1145869281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38ab60 0x55b3ee390700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34e5d9b640 1 --2- 172.21.15.7:0/1145869281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38b8d0 0x55b3ee393550 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009610 tx=0x7f34d000df10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c797320c141bc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34cffff640 1 -- 172.21.15.7:0/1145869281 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0021040 con 0x55b3ee38b8d0 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34cffff640 1 -- 172.21.15.7:0/1145869281 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34d00a4f80 con 0x55b3ee38b8d0 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34cffff640 1 -- 172.21.15.7:0/1145869281 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d00d0330 con 0x55b3ee38b8d0 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34ef801740 1 -- 172.21.15.7:0/1145869281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38b8d0 msgr2=0x55b3ee393550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.435+0000 7f34ef801740 1 --2- 172.21.15.7:0/1145869281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38b8d0 0x55b3ee393550 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009610 tx=0x7f34d000df10 comp rx=0 tx=0).stop 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 -- 172.21.15.7:0/1145869281 shutdown_connections 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 --2- 172.21.15.7:0/1145869281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee3940e0 0x55b3ee393a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 --2- 172.21.15.7:0/1145869281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee38b8d0 0x55b3ee393550 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 --2- 172.21.15.7:0/1145869281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38ab60 0x55b3ee390700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 -- 172.21.15.7:0/1145869281 >> 172.21.15.7:0/1145869281 conn(0x55b3ee351c90 msgr2=0x55b3ee384040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 -- 172.21.15.7:0/1145869281 shutdown_connections 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 -- 172.21.15.7:0/1145869281 wait complete. 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 -- start start 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 Processor -- start 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee38ab60 0x55b3ee394a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee394f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 0x55b3ee39b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f34e81ae160 con 0x55b3ee3940e0 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34e81af5d0 con 0x55b3ee38ab60 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f34e81ae830 con 0x55b3ee38b8d0 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34ef801740 10 client.0 _ll_register_callbacks cb 0x7f34d009c0e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34e6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 0x55b3ee39b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34e6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 0x55b3ee39b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46946/0 (socket says 172.21.15.7:46946) 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34e6d9d640 1 -- 172.21.15.7:0/3189806629 learned_addr learned my addr 172.21.15.7:0/3189806629 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.436+0000 7f34e659c640 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee38ab60 0x55b3ee394a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34e5d9b640 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee394f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34e6d9d640 1 -- 172.21.15.7:0/3189806629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 msgr2=0x55b3ee394f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34e6d9d640 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee394f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34e6d9d640 1 -- 172.21.15.7:0/3189806629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee38ab60 msgr2=0x55b3ee394a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34e6d9d640 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee38ab60 0x55b3ee394a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34e6d9d640 1 -- 172.21.15.7:0/3189806629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3ee39b700 con 0x55b3ee3940e0 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34e6d9d640 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 0x55b3ee39b1c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f34d8014310 tx=0x7f34d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c84d1d255c515e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:48:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34cdffb640 1 -- 172.21.15.7:0/3189806629 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8013900 con 0x55b3ee3940e0 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34cdffb640 1 -- 172.21.15.7:0/3189806629 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f34d8013aa0 con 0x55b3ee3940e0 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34ef801740 1 -- 172.21.15.7:0/3189806629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34d00a4f80 con 0x55b3ee3940e0 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34cdffb640 1 -- 172.21.15.7:0/3189806629 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f34d8013ca0 con 0x55b3ee3940e0 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34ef801740 10 client.16566 fetch_fsmap learned FSMap version 23 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34ef801740 1 -- 172.21.15.7:0/3189806629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b3ee39ca50 con 0x55b3ee3940e0 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.437+0000 7f34cdffb640 1 -- 172.21.15.7:0/3189806629 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34d00a4f80 con 0x55b3ee3940e0 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.438+0000 7f34cdffb640 1 -- 172.21.15.7:0/3189806629 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f34d8023420 con 0x55b3ee3940e0 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.438+0000 7f34ef801740 10 client.16566 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.438+0000 7f34ef801740 10 client.16566 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.438+0000 7f34ef801740 20 client.16566 populate_metadata read hostname 'smithi007' 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.438+0000 7f34ef801740 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b3ee40a6e0 0x55b3ee40cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.438+0000 7f34ef801740 4 client.16566 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:48:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.438+0000 7f34ef801740 1 -- 172.21.15.7:0/3189806629 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b3ee28f3d0 con 0x55b3ee40a6e0 2024-04-21T08:48:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.438+0000 7f34e659c640 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b3ee40a6e0 0x55b3ee40cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.438+0000 7f34e659c640 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b3ee40a6e0 0x55b3ee40cb00 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=327e7929508ff997 server_cookie=4bad159ff21a7479 in_seq=0 out_seq=0 2024-04-21T08:48:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:49.438+0000 7f34cdffb640 10 client.16566 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:49.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:49 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:49.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:49 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:50.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:49 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:50.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:49 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:50.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:49 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:48:50.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:49 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:50.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:50 smithi007 ceph-mon[28477]: pgmap v128: 129 pgs: 129 active+clean; 739 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 17 MiB/s wr, 4.26k op/s 2024-04-21T08:48:50.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:50 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:50.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:50 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:50.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:50 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:48:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:50 smithi016 ceph-mon[31259]: pgmap v128: 129 pgs: 129 active+clean; 739 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 17 MiB/s wr, 4.26k op/s 2024-04-21T08:48:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:50 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:50 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:50 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:48:51.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:50 smithi077 ceph-mon[31615]: pgmap v128: 129 pgs: 129 active+clean; 739 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 17 MiB/s wr, 4.26k op/s 2024-04-21T08:48:51.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:50 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:51.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:50 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:51.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:50 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:48:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34cdffb640 1 -- 172.21.15.7:0/3189806629 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b3ee28f3d0 con 0x55b3ee40a6e0 2024-04-21T08:48:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34cdffb640 10 client.16566 handle_command_reply: tid=0 2024-04-21T08:48:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 1 client.16566 shutdown 2024-04-21T08:48:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 1 -- 172.21.15.7:0/3189806629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 msgr2=0x55b3ee39b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 0x55b3ee39b1c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f34d8014310 tx=0x7f34d8004a40 comp rx=0 tx=0).stop 2024-04-21T08:48:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 1 -- 172.21.15.7:0/3189806629 shutdown_connections 2024-04-21T08:48:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b3ee40a6e0 0x55b3ee40cb00 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b3ee3940e0 0x55b3ee39b1c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ee38b8d0 0x55b3ee394f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 1 --2- 172.21.15.7:0/3189806629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b3ee38ab60 0x55b3ee394a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 1 -- 172.21.15.7:0/3189806629 >> 172.21.15.7:0/3189806629 conn(0x55b3ee351c90 msgr2=0x55b3ee3848d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34cdffb640 0 client.16566 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:48:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 1 -- 172.21.15.7:0/3189806629 shutdown_connections 2024-04-21T08:48:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 1 -- 172.21.15.7:0/3189806629 wait complete. 2024-04-21T08:48:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.742+0000 7f34ef801740 20 client.16566 trim_cache size 0 max 16384 2024-04-21T08:48:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:48:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:48:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:48:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:48:52.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 -- 172.21.15.7:0/1662995853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34d40a3d10 msgr2=0x7f34d40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34d40a3d10 0x7f34d40a61d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f34d8005c40 tx=0x7f34d8009200 comp rx=0 tx=0).stop 2024-04-21T08:48:52.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 -- 172.21.15.7:0/1662995853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34e81061a0 msgr2=0x7f34e8198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34e81061a0 0x7f34e8198c80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f34d000d770 tx=0x7f34d000dc30 comp rx=0 tx=0).stop 2024-04-21T08:48:52.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 -- 172.21.15.7:0/1662995853 shutdown_connections 2024-04-21T08:48:52.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34d40a3d10 0x7f34d40a61d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e819d000 0x7f34e819d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34e81061a0 0x7f34e8198c80 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 --2- 172.21.15.7:0/1662995853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34e8101120 0x7f34e8198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 -- 172.21.15.7:0/1662995853 >> 172.21.15.7:0/1662995853 conn(0x7f34e8077fa0 msgr2=0x7f34e80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:52.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 -- 172.21.15.7:0/1662995853 shutdown_connections 2024-04-21T08:48:52.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:48:52.743+0000 7f34ecc11640 1 -- 172.21.15.7:0/1662995853 wait complete. 2024-04-21T08:48:52.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:52 smithi007 ceph-mon[28477]: pgmap v129: 129 pgs: 129 active+clean; 739 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 2.91k op/s 2024-04-21T08:48:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:52 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:48:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:52 smithi016 ceph-mon[31259]: pgmap v129: 129 pgs: 129 active+clean; 739 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 2.91k op/s 2024-04-21T08:48:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:52 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:48:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:52 smithi077 ceph-mon[31615]: pgmap v129: 129 pgs: 129 active+clean; 739 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 2.91k op/s 2024-04-21T08:48:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:52 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:48:54.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:54 smithi007 ceph-mon[28477]: pgmap v130: 129 pgs: 129 active+clean; 740 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 4.53k op/s 2024-04-21T08:48:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:54 smithi016 ceph-mon[31259]: pgmap v130: 129 pgs: 129 active+clean; 740 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 4.53k op/s 2024-04-21T08:48:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:54 smithi077 ceph-mon[31615]: pgmap v130: 129 pgs: 129 active+clean; 740 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.9 MiB/s wr, 4.53k op/s 2024-04-21T08:48:57.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:56 smithi016 ceph-mon[31259]: pgmap v131: 129 pgs: 129 active+clean; 740 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 2.30k op/s 2024-04-21T08:48:57.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:56 smithi077 ceph-mon[31615]: pgmap v131: 129 pgs: 129 active+clean; 740 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 2.30k op/s 2024-04-21T08:48:57.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:56 smithi007 ceph-mon[28477]: pgmap v131: 129 pgs: 129 active+clean; 740 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 2.30k op/s 2024-04-21T08:48:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:48:58 smithi016 ceph-mon[31259]: pgmap v132: 129 pgs: 129 active+clean; 740 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 2.30k op/s 2024-04-21T08:48:59.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:48:58 smithi077 ceph-mon[31615]: pgmap v132: 129 pgs: 129 active+clean; 740 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 2.30k op/s 2024-04-21T08:48:59.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:48:58 smithi007 ceph-mon[28477]: pgmap v132: 129 pgs: 129 active+clean; 740 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 2.30k op/s 2024-04-21T08:49:01.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:00 smithi016 ceph-mon[31259]: pgmap v133: 129 pgs: 129 active+clean; 823 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 9.1 MiB/s wr, 2.69k op/s 2024-04-21T08:49:01.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:00 smithi077 ceph-mon[31615]: pgmap v133: 129 pgs: 129 active+clean; 823 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 9.1 MiB/s wr, 2.69k op/s 2024-04-21T08:49:01.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:00 smithi007 ceph-mon[28477]: pgmap v133: 129 pgs: 129 active+clean; 823 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 9.1 MiB/s wr, 2.69k op/s 2024-04-21T08:49:03.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:02 smithi016 ceph-mon[31259]: pgmap v134: 129 pgs: 129 active+clean; 823 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 8.6 MiB/s wr, 2.01k op/s 2024-04-21T08:49:03.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:02 smithi077 ceph-mon[31615]: pgmap v134: 129 pgs: 129 active+clean; 823 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 8.6 MiB/s wr, 2.01k op/s 2024-04-21T08:49:03.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:02 smithi007 ceph-mon[28477]: pgmap v134: 129 pgs: 129 active+clean; 823 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 8.6 MiB/s wr, 2.01k op/s 2024-04-21T08:49:05.057 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:04 smithi016 ceph-mon[31259]: pgmap v135: 129 pgs: 129 active+clean; 881 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 14 MiB/s wr, 2.34k op/s 2024-04-21T08:49:05.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:04 smithi077 ceph-mon[31615]: pgmap v135: 129 pgs: 129 active+clean; 881 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 14 MiB/s wr, 2.34k op/s 2024-04-21T08:49:05.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:04 smithi007 ceph-mon[28477]: pgmap v135: 129 pgs: 129 active+clean; 881 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 14 MiB/s wr, 2.34k op/s 2024-04-21T08:49:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.839+0000 7f59fb481640 1 -- 172.21.15.7:0/2469090374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59f4101a50 msgr2=0x7f59f4105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.839+0000 7f59fb481640 1 --2- 172.21.15.7:0/2469090374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59f4101a50 0x7f59f4105a90 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009900 tx=0x7f59e402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.839+0000 7f59fb481640 1 -- 172.21.15.7:0/2469090374 shutdown_connections 2024-04-21T08:49:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.839+0000 7f59fb481640 1 --2- 172.21.15.7:0/2469090374 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f4106170 0x7f59f410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.839+0000 7f59fb481640 1 --2- 172.21.15.7:0/2469090374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59f4101a50 0x7f59f4105a90 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.839+0000 7f59fb481640 1 --2- 172.21.15.7:0/2469090374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f41010a0 0x7f59f4101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.839+0000 7f59fb481640 1 -- 172.21.15.7:0/2469090374 >> 172.21.15.7:0/2469090374 conn(0x7f59f4077f70 msgr2=0x7f59f4078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.839+0000 7f59fb481640 1 -- 172.21.15.7:0/2469090374 shutdown_connections 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.839+0000 7f59fb481640 1 -- 172.21.15.7:0/2469090374 wait complete. 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59fb481640 1 Processor -- start 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59fb481640 1 -- start start 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59fb481640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f41010a0 0x7f59f4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59fb481640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59f4106170 0x7f59f4198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59fb481640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f419d000 0x7f59f419d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59fb481640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f59f4111090 con 0x7f59f4106170 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59fb481640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f59f4110f10 con 0x7f59f419d000 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59fb481640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59f4111210 con 0x7f59f41010a0 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f8a1c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59f4106170 0x7f59f4198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f8a1c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59f4106170 0x7f59f4198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43450/0 (socket says 172.21.15.7:43450) 2024-04-21T08:49:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f9a1e640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f419d000 0x7f59f419d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f8a1c640 1 -- 172.21.15.7:0/2236160036 learned_addr learned my addr 172.21.15.7:0/2236160036 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f921d640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f41010a0 0x7f59f4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f9a1e640 1 -- 172.21.15.7:0/2236160036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f41010a0 msgr2=0x7f59f4198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f9a1e640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f41010a0 0x7f59f4198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f9a1e640 1 -- 172.21.15.7:0/2236160036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59f4106170 msgr2=0x7f59f4198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f9a1e640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59f4106170 0x7f59f4198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f9a1e640 1 -- 172.21.15.7:0/2236160036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f419daf0 con 0x7f59f419d000 2024-04-21T08:49:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f8a1c640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59f4106170 0x7f59f4198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.840+0000 7f59f921d640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f41010a0 0x7f59f4198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.841+0000 7f59f9a1e640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f419d000 0x7f59f419d4b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009fd0 tx=0x7f59ec00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.841+0000 7f59ea7fc640 1 -- 172.21.15.7:0/2236160036 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec00ce20 con 0x7f59f419d000 2024-04-21T08:49:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.841+0000 7f59fb481640 1 -- 172.21.15.7:0/2236160036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f419dde0 con 0x7f59f419d000 2024-04-21T08:49:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.841+0000 7f59ea7fc640 1 -- 172.21.15.7:0/2236160036 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59ec005020 con 0x7f59f419d000 2024-04-21T08:49:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.841+0000 7f59ea7fc640 1 -- 172.21.15.7:0/2236160036 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec010690 con 0x7f59f419d000 2024-04-21T08:49:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.841+0000 7f59fb481640 1 -- 172.21.15.7:0/2236160036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f41a55d0 con 0x7f59f419d000 2024-04-21T08:49:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.843+0000 7f59ea7fc640 1 -- 172.21.15.7:0/2236160036 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f59ec0054d0 con 0x7f59f419d000 2024-04-21T08:49:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59fe07c740 1 Processor -- start 2024-04-21T08:49:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59fe07c740 1 -- start start 2024-04-21T08:49:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59ea7fc640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59e00a3d10 0x7f59e00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1768918d0 0x55d176890620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d176890fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 0x55d1768a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59fe07c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d1767f58d0 con 0x55d17689e4f0 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59fe07c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d176792e80 con 0x55d176890b60 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59fe07c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d1767f1890 con 0x55d1768918d0 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59f8a1c640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59e00a3d10 0x7f59e00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59ea7fc640 1 -- 172.21.15.7:0/2236160036 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f59ec0d1320 con 0x7f59f419d000 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59f9a1e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 0x55d1768a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59f9a1e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 0x55d1768a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43462/0 (socket says 172.21.15.7:43462) 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59f9a1e640 1 -- 172.21.15.7:0/2721493281 learned_addr learned my addr 172.21.15.7:0/2721493281 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59f8a1c640 1 --2- 172.21.15.7:0/2721493281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d176890fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59f921d640 1 --2- 172.21.15.7:0/2721493281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1768918d0 0x55d176890620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59f9a1e640 1 -- 172.21.15.7:0/2721493281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1768918d0 msgr2=0x55d176890620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59f9a1e640 1 --2- 172.21.15.7:0/2721493281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1768918d0 0x55d176890620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59f9a1e640 1 -- 172.21.15.7:0/2721493281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 msgr2=0x55d176890fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59f9a1e640 1 --2- 172.21.15.7:0/2721493281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d176890fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.844+0000 7f59f9a1e640 1 -- 172.21.15.7:0/2721493281 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1767b4320 con 0x55d17689e4f0 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.845+0000 7f59f9a1e640 1 --2- 172.21.15.7:0/2721493281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 0x55d1768a08e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f59ec099330 tx=0x7f59ec09d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3862623fa4b7e3a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.845+0000 7f59f8a1c640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59e00a3d10 0x7f59e00a61d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f59e4005c40 tx=0x7f59e4004040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.845+0000 7f59df7fe640 1 -- 172.21.15.7:0/2721493281 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0a5b40 con 0x55d17689e4f0 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.845+0000 7f59df7fe640 1 -- 172.21.15.7:0/2721493281 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59ec09af70 con 0x55d17689e4f0 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.845+0000 7f59df7fe640 1 -- 172.21.15.7:0/2721493281 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0a56b0 con 0x55d17689e4f0 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.845+0000 7f59fe07c740 1 -- 172.21.15.7:0/2721493281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 msgr2=0x55d1768a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.845+0000 7f59fe07c740 1 --2- 172.21.15.7:0/2721493281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 0x55d1768a08e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f59ec099330 tx=0x7f59ec09d720 comp rx=0 tx=0).stop 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 -- 172.21.15.7:0/2721493281 shutdown_connections 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 --2- 172.21.15.7:0/2721493281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 0x55d1768a08e0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 --2- 172.21.15.7:0/2721493281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d176890fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 --2- 172.21.15.7:0/2721493281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1768918d0 0x55d176890620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 -- 172.21.15.7:0/2721493281 >> 172.21.15.7:0/2721493281 conn(0x55d176857c90 msgr2=0x55d17688b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 -- 172.21.15.7:0/2721493281 shutdown_connections 2024-04-21T08:49:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 -- 172.21.15.7:0/2721493281 wait complete. 2024-04-21T08:49:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 -- start start 2024-04-21T08:49:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 Processor -- start 2024-04-21T08:49:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d176890b60 0x55d17689de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1768918d0 0x55d17689b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d17689e4f0 0x55d17689b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d1768a1220 con 0x55d176890b60 2024-04-21T08:49:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d17677deb0 con 0x55d1768918d0 2024-04-21T08:49:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d1768a13a0 con 0x55d17689e4f0 2024-04-21T08:49:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.846+0000 7f59fe07c740 10 client.0 _ll_register_callbacks cb 0x55d1767aab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f921d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d176890b60 0x55d17689de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f8a1c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1768918d0 0x55d17689b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f921d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d176890b60 0x55d17689de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43470/0 (socket says 172.21.15.7:43470) 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f921d640 1 -- 172.21.15.7:0/1152585363 learned_addr learned my addr 172.21.15.7:0/1152585363 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f8a1c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1768918d0 0x55d17689b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38614/0 (socket says 172.21.15.7:38614) 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f9a1e640 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d17689e4f0 0x55d17689b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f921d640 1 -- 172.21.15.7:0/1152585363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d17689e4f0 msgr2=0x55d17689b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f921d640 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d17689e4f0 0x55d17689b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f921d640 1 -- 172.21.15.7:0/1152585363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1768918d0 msgr2=0x55d17689b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f921d640 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1768918d0 0x55d17689b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f921d640 1 -- 172.21.15.7:0/1152585363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d176508210 con 0x55d176890b60 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59f921d640 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d176890b60 0x55d17689de80 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f59f000bb40 tx=0x7f59f00044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d8def174c75594 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59dcff9640 1 -- 172.21.15.7:0/1152585363 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f001b640 con 0x55d176890b60 2024-04-21T08:49:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59dcff9640 1 -- 172.21.15.7:0/1152585363 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f59f0011440 con 0x55d176890b60 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59fe07c740 1 -- 172.21.15.7:0/1152585363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59ec09af70 con 0x55d176890b60 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59dcff9640 1 -- 172.21.15.7:0/1152585363 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f59f001b7e0 con 0x55d176890b60 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59fe07c740 10 client.16578 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.847+0000 7f59fe07c740 1 -- 172.21.15.7:0/1152585363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d1765d4620 con 0x55d176890b60 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.848+0000 7f59dcff9640 1 -- 172.21.15.7:0/1152585363 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59ec09af70 con 0x55d176890b60 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.848+0000 7f59dcff9640 1 -- 172.21.15.7:0/1152585363 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f59f0019020 con 0x55d176890b60 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.848+0000 7f59fe07c740 10 client.16578 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.848+0000 7f59fe07c740 10 client.16578 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.848+0000 7f59fe07c740 20 client.16578 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.848+0000 7f59fe07c740 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d1768ae4f0 0x55d1768b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.848+0000 7f59fe07c740 4 client.16578 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.848+0000 7f59fe07c740 1 -- 172.21.15.7:0/1152585363 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d176809880 con 0x55d1768ae4f0 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.848+0000 7f59f8a1c640 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d1768ae4f0 0x55d1768b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.848+0000 7f59dcff9640 10 client.16578 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.848+0000 7f59f8a1c640 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d1768ae4f0 0x55d1768b08e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9273ed3ba7eaead server_cookie=fcaf227b2948ff87 in_seq=0 out_seq=0 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59dcff9640 1 -- 172.21.15.7:0/1152585363 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d176809880 con 0x55d1768ae4f0 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59dcff9640 10 client.16578 handle_command_reply: tid=0 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59fe07c740 1 client.16578 shutdown 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59fe07c740 1 -- 172.21.15.7:0/1152585363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d176890b60 msgr2=0x55d17689de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59fe07c740 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d176890b60 0x55d17689de80 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f59f000bb40 tx=0x7f59f00044f0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59fe07c740 1 -- 172.21.15.7:0/1152585363 shutdown_connections 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59fe07c740 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d1768ae4f0 0x55d1768b08e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59fe07c740 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d17689e4f0 0x55d17689b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59fe07c740 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d1768918d0 0x55d17689b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59dcff9640 0 client.16578 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59fe07c740 1 --2- 172.21.15.7:0/1152585363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d176890b60 0x55d17689de80 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59fe07c740 1 -- 172.21.15.7:0/1152585363 >> 172.21.15.7:0/1152585363 conn(0x55d176857c90 msgr2=0x55d17688d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59fe07c740 1 -- 172.21.15.7:0/1152585363 shutdown_connections 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.850+0000 7f59fe07c740 1 -- 172.21.15.7:0/1152585363 wait complete. 2024-04-21T08:49:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.851+0000 7f59fe07c740 20 client.16578 trim_cache size 0 max 16384 2024-04-21T08:49:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.865+0000 7f59fe07c740 1 Processor -- start 2024-04-21T08:49:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.865+0000 7f59fe07c740 1 -- start start 2024-04-21T08:49:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d1768aa120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d1768918d0 0x55d1768b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d17689e4f0 0x55d1768b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59fe07c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f59f41ab290 con 0x55d1768918d0 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59fe07c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f59f41adb10 con 0x55d176890b60 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59fe07c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59f41ab900 con 0x55d17689e4f0 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59f8a1c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d1768918d0 0x55d1768b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59f8a1c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d1768918d0 0x55d1768b61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43478/0 (socket says 172.21.15.7:43478) 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59f8a1c640 1 -- 172.21.15.7:0/2122615398 learned_addr learned my addr 172.21.15.7:0/2122615398 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59f921d640 1 --2- 172.21.15.7:0/2122615398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d1768aa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59f8a1c640 1 -- 172.21.15.7:0/2122615398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d17689e4f0 msgr2=0x55d1768b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59f8a1c640 1 --2- 172.21.15.7:0/2122615398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d17689e4f0 0x55d1768b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59f8a1c640 1 -- 172.21.15.7:0/2122615398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 msgr2=0x55d1768aa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59f8a1c640 1 --2- 172.21.15.7:0/2122615398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d1768aa120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59f8a1c640 1 -- 172.21.15.7:0/2122615398 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d17689bb00 con 0x55d1768918d0 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59f8a1c640 1 --2- 172.21.15.7:0/2122615398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d1768918d0 0x55d1768b61d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f59e4004650 tx=0x7f59e404b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d22b0d6ec86ebc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59deffd640 1 -- 172.21.15.7:0/2122615398 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e405d070 con 0x55d1768918d0 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59deffd640 1 -- 172.21.15.7:0/2122615398 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59e4034440 con 0x55d1768918d0 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.866+0000 7f59deffd640 1 -- 172.21.15.7:0/2122615398 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4050450 con 0x55d1768918d0 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.867+0000 7f59fe07c740 1 -- 172.21.15.7:0/2122615398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d1768918d0 msgr2=0x55d1768b61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.867+0000 7f59fe07c740 1 --2- 172.21.15.7:0/2122615398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d1768918d0 0x55d1768b61d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f59e4004650 tx=0x7f59e404b000 comp rx=0 tx=0).stop 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.867+0000 7f59fe07c740 1 -- 172.21.15.7:0/2122615398 shutdown_connections 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.867+0000 7f59fe07c740 1 --2- 172.21.15.7:0/2122615398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d17689e4f0 0x55d1768b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.867+0000 7f59fe07c740 1 --2- 172.21.15.7:0/2122615398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d1768918d0 0x55d1768b61d0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.867+0000 7f59fe07c740 1 --2- 172.21.15.7:0/2122615398 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d1768aa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.867+0000 7f59fe07c740 1 -- 172.21.15.7:0/2122615398 >> 172.21.15.7:0/2122615398 conn(0x55d176857c90 msgr2=0x55d17688b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.867+0000 7f59fe07c740 1 -- 172.21.15.7:0/2122615398 shutdown_connections 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.867+0000 7f59fe07c740 1 -- 172.21.15.7:0/2122615398 wait complete. 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.867+0000 7f59fe07c740 1 -- start start 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.867+0000 7f59fe07c740 1 Processor -- start 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d1768a6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1768918d0 0x55d176897e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59fe07c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 0x55d176898550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59fe07c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f59f41abfd0 con 0x55d17689e4f0 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59fe07c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f59f41ad440 con 0x55d176890b60 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59fe07c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59f41ac6a0 con 0x55d1768918d0 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59fe07c740 10 client.0 _ll_register_callbacks cb 0x7f59f00043b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59f921d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d1768a6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59f9a1e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 0x55d176898550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59f921d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d1768a6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38630/0 (socket says 172.21.15.7:38630) 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59f921d640 1 -- 172.21.15.7:0/409563884 learned_addr learned my addr 172.21.15.7:0/409563884 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59f8a1c640 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1768918d0 0x55d176897e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59f9a1e640 1 -- 172.21.15.7:0/409563884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1768918d0 msgr2=0x55d176897e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59f9a1e640 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1768918d0 0x55d176897e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59f9a1e640 1 -- 172.21.15.7:0/409563884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 msgr2=0x55d1768a6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59f9a1e640 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d1768a6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59f9a1e640 1 -- 172.21.15.7:0/409563884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d176913a40 con 0x55d17689e4f0 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.868+0000 7f59f921d640 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d1768a6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59f9a1e640 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 0x55d176898550 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0cf000 tx=0x7f59ec005400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c9df9e71c91337 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59dcff9640 1 -- 172.21.15.7:0/409563884 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0bb060 con 0x55d17689e4f0 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59dcff9640 1 -- 172.21.15.7:0/409563884 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f59ec0a56b0 con 0x55d17689e4f0 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59fe07c740 1 -- 172.21.15.7:0/409563884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59e4034440 con 0x55d17689e4f0 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59dcff9640 1 -- 172.21.15.7:0/409563884 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f59ec0ceb10 con 0x55d17689e4f0 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59fe07c740 10 client.16584 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59fe07c740 1 -- 172.21.15.7:0/409563884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d176914c10 con 0x55d17689e4f0 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59dcff9640 1 -- 172.21.15.7:0/409563884 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59ec061e50 con 0x55d17689e4f0 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59dcff9640 1 -- 172.21.15.7:0/409563884 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f59ec087b90 con 0x55d17689e4f0 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59fe07c740 10 client.16584 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59fe07c740 10 client.16584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59fe07c740 20 client.16584 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59fe07c740 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d176914f70 0x55d176917360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59fe07c740 4 client.16584 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:49:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.869+0000 7f59fe07c740 1 -- 172.21.15.7:0/409563884 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d1767953d0 con 0x55d176914f70 2024-04-21T08:49:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.870+0000 7f59f921d640 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d176914f70 0x55d176917360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.870+0000 7f59f921d640 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d176914f70 0x55d176917360 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcd8cc14871d50eb server_cookie=64c7dcd367c77d9c in_seq=0 out_seq=0 2024-04-21T08:49:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:05.870+0000 7f59dcff9640 10 client.16584 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:07.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:06 smithi016 ceph-mon[31259]: pgmap v136: 129 pgs: 129 active+clean; 881 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 MiB/s wr, 712 op/s 2024-04-21T08:49:07.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:06 smithi077 ceph-mon[31615]: pgmap v136: 129 pgs: 129 active+clean; 881 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 MiB/s wr, 712 op/s 2024-04-21T08:49:07.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:06 smithi007 ceph-mon[28477]: pgmap v136: 129 pgs: 129 active+clean; 881 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 MiB/s wr, 712 op/s 2024-04-21T08:49:08.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:49:08.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:49:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.368+0000 7f36c1d24640 1 -- 172.21.15.7:0/694112572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36bc101a50 msgr2=0x7f36bc105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.368+0000 7f36c1d24640 1 --2- 172.21.15.7:0/694112572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36bc101a50 0x7f36bc105a90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009900 tx=0x7f36b002bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.368+0000 7f36c1d24640 1 -- 172.21.15.7:0/694112572 shutdown_connections 2024-04-21T08:49:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.368+0000 7f36c1d24640 1 --2- 172.21.15.7:0/694112572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc106170 0x7f36bc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.369+0000 7f36c1d24640 1 --2- 172.21.15.7:0/694112572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36bc101a50 0x7f36bc105a90 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.369+0000 7f36c1d24640 1 --2- 172.21.15.7:0/694112572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc1010a0 0x7f36bc101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.369+0000 7f36c1d24640 1 -- 172.21.15.7:0/694112572 >> 172.21.15.7:0/694112572 conn(0x7f36bc077f40 msgr2=0x7f36bc078350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.369+0000 7f36c1d24640 1 -- 172.21.15.7:0/694112572 shutdown_connections 2024-04-21T08:49:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.369+0000 7f36c1d24640 1 -- 172.21.15.7:0/694112572 wait complete. 2024-04-21T08:49:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.369+0000 7f36c1d24640 1 Processor -- start 2024-04-21T08:49:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.369+0000 7f36c1d24640 1 -- start start 2024-04-21T08:49:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36c1d24640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36bc1010a0 0x7f36bc1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36c1d24640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc106170 0x7f36bc1a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36c1d24640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc19a6e0 0x7f36bc19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36c1d24640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc1110b0 con 0x7f36bc1010a0 2024-04-21T08:49:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36c1d24640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc110f30 con 0x7f36bc19a6e0 2024-04-21T08:49:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36c1d24640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc111230 con 0x7f36bc106170 2024-04-21T08:49:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36bc1010a0 0x7f36bc1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36bc1010a0 0x7f36bc1a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43508/0 (socket says 172.21.15.7:43508) 2024-04-21T08:49:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36bb7fe640 1 -- 172.21.15.7:0/631413144 learned_addr learned my addr 172.21.15.7:0/631413144 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36baffd640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc106170 0x7f36bc1a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36bbfff640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc19a6e0 0x7f36bc19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36bb7fe640 1 -- 172.21.15.7:0/631413144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc106170 msgr2=0x7f36bc1a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc106170 0x7f36bc1a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36bb7fe640 1 -- 172.21.15.7:0/631413144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc19a6e0 msgr2=0x7f36bc19ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc19a6e0 0x7f36bc19ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36bb7fe640 1 -- 172.21.15.7:0/631413144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36bc19b2e0 con 0x7f36bc1010a0 2024-04-21T08:49:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36bc1010a0 0x7f36bc1a0620 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00e9f0 tx=0x7f36ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36b8ff9640 1 -- 172.21.15.7:0/631413144 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac00cd80 con 0x7f36bc1010a0 2024-04-21T08:49:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36b8ff9640 1 -- 172.21.15.7:0/631413144 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36ac0108e0 con 0x7f36bc1010a0 2024-04-21T08:49:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36c1d24640 1 -- 172.21.15.7:0/631413144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36bc1a73c0 con 0x7f36bc1010a0 2024-04-21T08:49:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.370+0000 7f36b8ff9640 1 -- 172.21.15.7:0/631413144 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac018980 con 0x7f36bc1010a0 2024-04-21T08:49:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.371+0000 7f36c1d24640 1 -- 172.21.15.7:0/631413144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36bc1a7770 con 0x7f36bc1010a0 2024-04-21T08:49:08.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.373+0000 7f36b8ff9640 1 -- 172.21.15.7:0/631413144 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f36ac010430 con 0x7f36bc1010a0 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36c49f2740 1 Processor -- start 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36c49f2740 1 -- start start 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 0x5612e9bc5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bd0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36c49f2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5612e9b2a1e0 con 0x5612e9bc5ea0 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36c49f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5612e9ac7390 con 0x5612e9bc5580 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36c49f2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5612e9b261a0 con 0x5612e9bcea80 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 0x5612e9bc5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bce540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43518/0 (socket says 172.21.15.7:43518) 2024-04-21T08:49:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36baffd640 1 --2- 172.21.15.7:0/1931686765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bd0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36bb7fe640 1 -- 172.21.15.7:0/1931686765 learned_addr learned my addr 172.21.15.7:0/1931686765 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36b8ff9640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36a40a3d20 0x7f36a40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36baffd640 1 -- 172.21.15.7:0/1931686765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 msgr2=0x5612e9bc5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36baffd640 1 --2- 172.21.15.7:0/1931686765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 0x5612e9bc5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36baffd640 1 -- 172.21.15.7:0/1931686765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 msgr2=0x5612e9bce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36baffd640 1 --2- 172.21.15.7:0/1931686765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36baffd640 1 -- 172.21.15.7:0/1931686765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612e99067d0 con 0x5612e9bcea80 2024-04-21T08:49:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36bbfff640 1 --2- 172.21.15.7:0/1931686765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 0x5612e9bc5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/1931686765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.374+0000 7f36bbfff640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36a40a3d20 0x7f36a40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.375+0000 7f36baffd640 1 --2- 172.21.15.7:0/1931686765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bd0e70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f36b0036040 tx=0x7f36b00095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61b51c6f6be8b18d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.375+0000 7f36b8ff9640 1 -- 172.21.15.7:0/631413144 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f36ac0d0ed0 con 0x7f36bc1010a0 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.375+0000 7f36a1ffb640 1 -- 172.21.15.7:0/1931686765 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b0045020 con 0x5612e9bcea80 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.375+0000 7f36bbfff640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36a40a3d20 0x7f36a40a61e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f36b4009fd0 tx=0x7f36b4004a40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.375+0000 7f36a1ffb640 1 -- 172.21.15.7:0/1931686765 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36b002c7e0 con 0x5612e9bcea80 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.375+0000 7f36a1ffb640 1 -- 172.21.15.7:0/1931686765 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b0005440 con 0x5612e9bcea80 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.376+0000 7f36c49f2740 1 -- 172.21.15.7:0/1931686765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 msgr2=0x5612e9bd0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.376+0000 7f36c49f2740 1 --2- 172.21.15.7:0/1931686765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bd0e70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f36b0036040 tx=0x7f36b00095f0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.376+0000 7f36c49f2740 1 -- 172.21.15.7:0/1931686765 shutdown_connections 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.376+0000 7f36c49f2740 1 --2- 172.21.15.7:0/1931686765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bd0e70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.376+0000 7f36c49f2740 1 --2- 172.21.15.7:0/1931686765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.376+0000 7f36c49f2740 1 --2- 172.21.15.7:0/1931686765 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 0x5612e9bc5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.376+0000 7f36c49f2740 1 -- 172.21.15.7:0/1931686765 >> 172.21.15.7:0/1931686765 conn(0x5612e9bbfcb0 msgr2=0x5612e9bc0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.376+0000 7f36c49f2740 1 -- 172.21.15.7:0/1931686765 shutdown_connections 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.376+0000 7f36c49f2740 1 -- 172.21.15.7:0/1931686765 wait complete. 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.376+0000 7f36c49f2740 1 -- start start 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.376+0000 7f36c49f2740 1 Processor -- start 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 0x5612e9bcbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bcc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bcc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36c49f2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5612e9bd1810 con 0x5612e9bc5ea0 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36c49f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5612e9ae32f0 con 0x5612e9bc5580 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36c49f2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5612e9bd1990 con 0x5612e9bcea80 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36c49f2740 10 client.0 _ll_register_callbacks cb 0x5612e9af0580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 0x5612e9bcbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36baffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bcc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 0x5612e9bcbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38642/0 (socket says 172.21.15.7:38642) 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36bb7fe640 1 -- 172.21.15.7:0/736998294 learned_addr learned my addr 172.21.15.7:0/736998294 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36bbfff640 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bcc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36bb7fe640 1 -- 172.21.15.7:0/736998294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 msgr2=0x5612e9bcc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bcc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36bb7fe640 1 -- 172.21.15.7:0/736998294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 msgr2=0x5612e9bcc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bcc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36bb7fe640 1 -- 172.21.15.7:0/736998294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612e983c3f0 con 0x5612e9bc5580 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36baffd640 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bcc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 0x5612e9bcbac0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f36ac09cac0 tx=0x7f36ac00cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af21ee987ce9c26a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f369affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.377+0000 7f369affd640 1 -- 172.21.15.7:0/736998294 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac09c490 con 0x5612e9bc5580 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f36c49f2740 1 -- 172.21.15.7:0/736998294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36b002c7e0 con 0x5612e9bc5580 2024-04-21T08:49:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f369affd640 1 -- 172.21.15.7:0/736998294 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f36ac0d4cb0 con 0x5612e9bc5580 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f369affd640 1 -- 172.21.15.7:0/736998294 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f36ac0d4eb0 con 0x5612e9bc5580 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f36c49f2740 10 client.26150 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f36c49f2740 1 -- 172.21.15.7:0/736998294 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5612e97fc0f0 con 0x5612e9bc5580 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f369affd640 1 -- 172.21.15.7:0/736998294 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36b002c7e0 con 0x5612e9bc5580 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f369affd640 1 -- 172.21.15.7:0/736998294 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f36ac018b20 con 0x5612e9bc5580 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f36c49f2740 10 client.26150 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f36c49f2740 10 client.26150 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f36c49f2740 20 client.26150 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f36c49f2740 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5612e9bdea40 0x5612e9be0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f36c49f2740 4 client.26150 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f36c49f2740 1 -- 172.21.15.7:0/736998294 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612e9b37c00 con 0x5612e9bdea40 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.378+0000 7f36baffd640 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5612e9bdea40 0x5612e9be0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.379+0000 7f36baffd640 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5612e9bdea40 0x5612e9be0ed0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9c4b95dddbb7aea server_cookie=8322ead0c642e3d3 in_seq=0 out_seq=0 2024-04-21T08:49:08.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:08.379+0000 7f369affd640 10 client.26150 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:09.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:08 smithi016 ceph-mon[31259]: pgmap v137: 129 pgs: 129 active+clean; 881 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 MiB/s wr, 712 op/s 2024-04-21T08:49:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:08 smithi077 ceph-mon[31615]: pgmap v137: 129 pgs: 129 active+clean; 881 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 MiB/s wr, 712 op/s 2024-04-21T08:49:09.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:08 smithi007 ceph-mon[28477]: pgmap v137: 129 pgs: 129 active+clean; 881 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 13 MiB/s wr, 712 op/s 2024-04-21T08:49:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:10 smithi016 ceph-mon[31259]: pgmap v138: 129 pgs: 129 active+clean; 942 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 18 MiB/s wr, 962 op/s 2024-04-21T08:49:11.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:10 smithi077 ceph-mon[31615]: pgmap v138: 129 pgs: 129 active+clean; 942 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 18 MiB/s wr, 962 op/s 2024-04-21T08:49:11.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:10 smithi007 ceph-mon[28477]: pgmap v138: 129 pgs: 129 active+clean; 942 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 18 MiB/s wr, 962 op/s 2024-04-21T08:49:13.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:12 smithi016 ceph-mon[31259]: pgmap v139: 129 pgs: 129 active+clean; 942 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 10 MiB/s wr, 576 op/s 2024-04-21T08:49:13.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:12 smithi077 ceph-mon[31615]: pgmap v139: 129 pgs: 129 active+clean; 942 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 10 MiB/s wr, 576 op/s 2024-04-21T08:49:13.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:12 smithi007 ceph-mon[28477]: pgmap v139: 129 pgs: 129 active+clean; 942 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 10 MiB/s wr, 576 op/s 2024-04-21T08:49:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59dcff9640 1 -- 172.21.15.7:0/409563884 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d1767953d0 con 0x55d176914f70 2024-04-21T08:49:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59dcff9640 10 client.16584 handle_command_reply: tid=0 2024-04-21T08:49:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59fe07c740 1 client.16584 shutdown 2024-04-21T08:49:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59fe07c740 1 -- 172.21.15.7:0/409563884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 msgr2=0x55d176898550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59fe07c740 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 0x55d176898550 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0cf000 tx=0x7f59ec005400 comp rx=0 tx=0).stop 2024-04-21T08:49:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f369affd640 1 -- 172.21.15.7:0/736998294 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5612e9b37c00 con 0x5612e9bdea40 2024-04-21T08:49:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f369affd640 10 client.26150 handle_command_reply: tid=0 2024-04-21T08:49:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f36c49f2740 1 client.26150 shutdown 2024-04-21T08:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59fe07c740 1 -- 172.21.15.7:0/409563884 shutdown_connections 2024-04-21T08:49:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59fe07c740 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d176914f70 0x55d176917360 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59fe07c740 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d17689e4f0 0x55d176898550 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59dcff9640 0 client.16584 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59fe07c740 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d1768918d0 0x55d176897e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59fe07c740 1 --2- 172.21.15.7:0/409563884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d176890b60 0x55d1768a6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59fe07c740 1 -- 172.21.15.7:0/409563884 >> 172.21.15.7:0/409563884 conn(0x55d176857c90 msgr2=0x55d17688a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.629+0000 7f59fe07c740 1 -- 172.21.15.7:0/409563884 shutdown_connections 2024-04-21T08:49:14.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f36c49f2740 1 -- 172.21.15.7:0/736998294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 msgr2=0x5612e9bcbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f36c49f2740 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 0x5612e9bcbac0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f36ac09cac0 tx=0x7f36ac00cf20 comp rx=0 tx=0).stop 2024-04-21T08:49:14.632 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fe07c740 1 -- 172.21.15.7:0/409563884 wait complete. 2024-04-21T08:49:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f36c49f2740 1 -- 172.21.15.7:0/736998294 shutdown_connections 2024-04-21T08:49:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f36c49f2740 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5612e9bdea40 0x5612e9be0ed0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f36c49f2740 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bcc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f36c49f2740 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bcc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f369affd640 0 client.26150 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f36c49f2740 1 --2- 172.21.15.7:0/736998294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5580 0x5612e9bcbac0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f36c49f2740 1 -- 172.21.15.7:0/736998294 >> 172.21.15.7:0/736998294 conn(0x5612e9bbfcb0 msgr2=0x5612e9bc1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fe07c740 20 client.16584 trim_cache size 0 max 16384 2024-04-21T08:49:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f36c49f2740 1 -- 172.21.15.7:0/736998294 shutdown_connections 2024-04-21T08:49:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f36c49f2740 1 -- 172.21.15.7:0/736998294 wait complete. 2024-04-21T08:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:49:14.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f36c49f2740 20 client.26150 trim_cache size 0 max 16384 2024-04-21T08:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 -- 172.21.15.7:0/2236160036 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59e00a3d10 msgr2=0x7f59e00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59e00a3d10 0x7f59e00a61d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f59e4005c40 tx=0x7f59e4004040 comp rx=0 tx=0).stop 2024-04-21T08:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 -- 172.21.15.7:0/2236160036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f419d000 msgr2=0x7f59f419d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f419d000 0x7f59f419d4b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009fd0 tx=0x7f59ec00ef10 comp rx=0 tx=0).stop 2024-04-21T08:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 -- 172.21.15.7:0/2236160036 shutdown_connections 2024-04-21T08:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59f419d000 0x7f59f419d4b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59f4106170 0x7f59f4198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59e00a3d10 0x7f59e00a61d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 --2- 172.21.15.7:0/2236160036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f41010a0 0x7f59f4198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 -- 172.21.15.7:0/2236160036 >> 172.21.15.7:0/2236160036 conn(0x7f59f4077f70 msgr2=0x7f59f4069ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 -- 172.21.15.7:0/2236160036 shutdown_connections 2024-04-21T08:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:14.630+0000 7f59fb481640 1 -- 172.21.15.7:0/2236160036 wait complete. 2024-04-21T08:49:14.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36c49f2740 1 Processor -- start 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36c49f2740 1 -- start start 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5580 0x5612e9bce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5ea0 0x5612e9bda6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bc0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36c49f2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc1af5b0 con 0x5612e9bc5580 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36c49f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc1b1e30 con 0x5612e9bc5ea0 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36c49f2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc1afc20 con 0x5612e9bcea80 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5580 0x5612e9bce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36bb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5580 0x5612e9bce1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34728/0 (socket says 172.21.15.7:34728) 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36baffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5ea0 0x5612e9bda6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36bb7fe640 1 -- 172.21.15.7:0/876228098 learned_addr learned my addr 172.21.15.7:0/876228098 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:14.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.650+0000 7f36bbfff640 1 --2- 172.21.15.7:0/876228098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bc0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.651+0000 7f36bb7fe640 1 -- 172.21.15.7:0/876228098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 msgr2=0x5612e9bc0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.651+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/876228098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bc0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.651+0000 7f36bb7fe640 1 -- 172.21.15.7:0/876228098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5ea0 msgr2=0x5612e9bda6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.651+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/876228098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5ea0 0x5612e9bda6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.651+0000 7f36bb7fe640 1 -- 172.21.15.7:0/876228098 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612e9bd0e70 con 0x5612e9bc5580 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.651+0000 7f36bbfff640 1 --2- 172.21.15.7:0/876228098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bc0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.651+0000 7f36baffd640 1 --2- 172.21.15.7:0/876228098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5ea0 0x5612e9bda6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.651+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/876228098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5580 0x5612e9bce1f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f36ac09ade0 tx=0x7f36ac0cf000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b77c6497b1b86d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.651+0000 7f36a17fa640 1 -- 172.21.15.7:0/876228098 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac09c490 con 0x5612e9bc5580 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.651+0000 7f36a17fa640 1 -- 172.21.15.7:0/876228098 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36ac010720 con 0x5612e9bc5580 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.651+0000 7f36a17fa640 1 -- 172.21.15.7:0/876228098 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac018b20 con 0x5612e9bc5580 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.652+0000 7f36c49f2740 1 -- 172.21.15.7:0/876228098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5580 msgr2=0x5612e9bce1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.652+0000 7f36c49f2740 1 --2- 172.21.15.7:0/876228098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5580 0x5612e9bce1f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f36ac09ade0 tx=0x7f36ac0cf000 comp rx=0 tx=0).stop 2024-04-21T08:49:14.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.652+0000 7f36c49f2740 1 -- 172.21.15.7:0/876228098 shutdown_connections 2024-04-21T08:49:14.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.652+0000 7f36c49f2740 1 --2- 172.21.15.7:0/876228098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bcea80 0x5612e9bc0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.652+0000 7f36c49f2740 1 --2- 172.21.15.7:0/876228098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bc5ea0 0x5612e9bda6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.652+0000 7f36c49f2740 1 --2- 172.21.15.7:0/876228098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5580 0x5612e9bce1f0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.652+0000 7f36c49f2740 1 -- 172.21.15.7:0/876228098 >> 172.21.15.7:0/876228098 conn(0x5612e9bbfcb0 msgr2=0x5612e9bbe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.652+0000 7f36c49f2740 1 -- 172.21.15.7:0/876228098 shutdown_connections 2024-04-21T08:49:14.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.652+0000 7f36c49f2740 1 -- 172.21.15.7:0/876228098 wait complete. 2024-04-21T08:49:14.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.652+0000 7f36c49f2740 1 -- start start 2024-04-21T08:49:14.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.652+0000 7f36c49f2740 1 Processor -- start 2024-04-21T08:49:14.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bc5580 0x5612e9bd6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bd2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36c49f2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bcea80 0x5612e9bd27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36c49f2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc1b02f0 con 0x5612e9bc5ea0 2024-04-21T08:49:14.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36c49f2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc1b1760 con 0x5612e9bcea80 2024-04-21T08:49:14.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36c49f2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc1b09c0 con 0x5612e9bc5580 2024-04-21T08:49:14.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36c49f2740 10 client.0 _ll_register_callbacks cb 0x7f36ac009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:14.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36baffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bd2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bcea80 0x5612e9bd27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36baffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bd2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34734/0 (socket says 172.21.15.7:34734) 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36baffd640 1 -- 172.21.15.7:0/104893859 learned_addr learned my addr 172.21.15.7:0/104893859 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bc5580 0x5612e9bd6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36bbfff640 1 -- 172.21.15.7:0/104893859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bc5580 msgr2=0x5612e9bd6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36bbfff640 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bc5580 0x5612e9bd6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36bbfff640 1 -- 172.21.15.7:0/104893859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 msgr2=0x5612e9bd2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36bbfff640 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bd2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36bbfff640 1 -- 172.21.15.7:0/104893859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612e9bd2fa0 con 0x5612e9bcea80 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36baffd640 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bd2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36bbfff640 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bcea80 0x5612e9bd27a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x5612e9be1520 tx=0x7f36b4004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd1a0dc7fe9822f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f369affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f369affd640 1 -- 172.21.15.7:0/104893859 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b4004b20 con 0x5612e9bcea80 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.653+0000 7f36c49f2740 1 -- 172.21.15.7:0/104893859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5612e9b31a70 con 0x5612e9bcea80 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f369affd640 1 -- 172.21.15.7:0/104893859 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f36b4004cc0 con 0x5612e9bcea80 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f369affd640 1 -- 172.21.15.7:0/104893859 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f36b4036050 con 0x5612e9bcea80 2024-04-21T08:49:14.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f36c49f2740 10 client.26156 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f36c49f2740 1 -- 172.21.15.7:0/104893859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5612e9c44f10 con 0x5612e9bcea80 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f369affd640 1 -- 172.21.15.7:0/104893859 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5612e9b31a70 con 0x5612e9bcea80 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f369affd640 1 -- 172.21.15.7:0/104893859 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f36b4029480 con 0x5612e9bcea80 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f36c49f2740 10 client.26156 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f36c49f2740 10 client.26156 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f36c49f2740 20 client.26156 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f36c49f2740 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5612e9bdea40 0x5612e9c476d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f36c49f2740 4 client.26156 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f36c49f2740 1 -- 172.21.15.7:0/104893859 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5612e9aad590 con 0x5612e9bdea40 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.654+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5612e9bdea40 0x5612e9c476d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.655+0000 7f369affd640 10 client.26156 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.655+0000 7f36bb7fe640 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5612e9bdea40 0x5612e9c476d0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1f66266659556ce server_cookie=ee8ea667f46718e6 in_seq=0 out_seq=0 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f369affd640 1 -- 172.21.15.7:0/104893859 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5612e9aad590 con 0x5612e9bdea40 2024-04-21T08:49:14.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f369affd640 10 client.26156 handle_command_reply: tid=0 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 1 client.26156 shutdown 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 1 -- 172.21.15.7:0/104893859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bcea80 msgr2=0x5612e9bd27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bcea80 0x5612e9bd27a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x5612e9be1520 tx=0x7f36b4004ab0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 1 -- 172.21.15.7:0/104893859 shutdown_connections 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5612e9bdea40 0x5612e9c476d0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5612e9bcea80 0x5612e9bd27a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f369affd640 0 client.26156 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5612e9bc5ea0 0x5612e9bd2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 1 --2- 172.21.15.7:0/104893859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5612e9bc5580 0x5612e9bd6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 1 -- 172.21.15.7:0/104893859 >> 172.21.15.7:0/104893859 conn(0x5612e9bbfcb0 msgr2=0x5612e9bbed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 1 -- 172.21.15.7:0/104893859 shutdown_connections 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 1 -- 172.21.15.7:0/104893859 wait complete. 2024-04-21T08:49:14.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c49f2740 20 client.26156 trim_cache size 0 max 16384 2024-04-21T08:49:14.662 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:49:14.662 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:49:14.662 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:49:14.662 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c1d24640 1 -- 172.21.15.7:0/631413144 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36a40a3d20 msgr2=0x7f36a40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c1d24640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36a40a3d20 0x7f36a40a61e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f36b4009fd0 tx=0x7f36b4004a40 comp rx=0 tx=0).stop 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c1d24640 1 -- 172.21.15.7:0/631413144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36bc1010a0 msgr2=0x7f36bc1a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.656+0000 7f36c1d24640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36bc1010a0 0x7f36bc1a0620 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00e9f0 tx=0x7f36ac00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.657+0000 7f36c1d24640 1 -- 172.21.15.7:0/631413144 shutdown_connections 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.657+0000 7f36c1d24640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36bc19a6e0 0x7f36bc19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.657+0000 7f36c1d24640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f36a40a3d20 0x7f36a40a61e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.657+0000 7f36c1d24640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36bc106170 0x7f36bc1a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.657+0000 7f36c1d24640 1 --2- 172.21.15.7:0/631413144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36bc1010a0 0x7f36bc1a0620 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.657+0000 7f36c1d24640 1 -- 172.21.15.7:0/631413144 >> 172.21.15.7:0/631413144 conn(0x7f36bc077f40 msgr2=0x7f36bc1000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.657+0000 7f36c1d24640 1 -- 172.21.15.7:0/631413144 shutdown_connections 2024-04-21T08:49:14.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.657+0000 7f36c1d24640 1 -- 172.21.15.7:0/631413144 wait complete. 2024-04-21T08:49:14.667 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:49:14.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:49:14.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.774+0000 7fad62255640 1 -- 172.21.15.7:0/4212767099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c106160 msgr2=0x7fad5c10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.774+0000 7fad62255640 1 --2- 172.21.15.7:0/4212767099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c106160 0x7fad5c10dd30 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fad54009900 tx=0x7fad5402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.774+0000 7fad62255640 1 -- 172.21.15.7:0/4212767099 shutdown_connections 2024-04-21T08:49:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.774+0000 7fad62255640 1 --2- 172.21.15.7:0/4212767099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c106160 0x7fad5c10dd30 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.774+0000 7fad62255640 1 --2- 172.21.15.7:0/4212767099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad5c101a40 0x7fad5c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.774+0000 7fad62255640 1 --2- 172.21.15.7:0/4212767099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad5c101090 0x7fad5c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.774+0000 7fad62255640 1 -- 172.21.15.7:0/4212767099 >> 172.21.15.7:0/4212767099 conn(0x7fad5c0771d0 msgr2=0x7fad5c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.775+0000 7fad62255640 1 -- 172.21.15.7:0/4212767099 shutdown_connections 2024-04-21T08:49:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.775+0000 7fad62255640 1 -- 172.21.15.7:0/4212767099 wait complete. 2024-04-21T08:49:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.775+0000 7fad62255640 1 Processor -- start 2024-04-21T08:49:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.775+0000 7fad62255640 1 -- start start 2024-04-21T08:49:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.775+0000 7fad62255640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad5c101090 0x7fad5c10e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.775+0000 7fad62255640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c106160 0x7fad5c10ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.775+0000 7fad62255640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad5c10ef40 0x7fad5c1a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad62255640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1121d0 con 0x7fad5c106160 2024-04-21T08:49:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad62255640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c112050 con 0x7fad5c101090 2024-04-21T08:49:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad62255640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c112350 con 0x7fad5c10ef40 2024-04-21T08:49:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c106160 0x7fad5c10ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c106160 0x7fad5c10ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34754/0 (socket says 172.21.15.7:34754) 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad5c101090 0x7fad5c10e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5affd640 1 -- 172.21.15.7:0/3948042627 learned_addr learned my addr 172.21.15.7:0/3948042627 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5bfff640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad5c10ef40 0x7fad5c1a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5affd640 1 -- 172.21.15.7:0/3948042627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad5c10ef40 msgr2=0x7fad5c1a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5affd640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad5c10ef40 0x7fad5c1a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5affd640 1 -- 172.21.15.7:0/3948042627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad5c101090 msgr2=0x7fad5c10e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5affd640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad5c101090 0x7fad5c10e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5affd640 1 -- 172.21.15.7:0/3948042627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5c1a6f80 con 0x7fad5c106160 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5b7fe640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad5c101090 0x7fad5c10e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad5affd640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c106160 0x7fad5c10ea00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fad5000c970 tx=0x7fad5000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad58ff9640 1 -- 172.21.15.7:0/3948042627 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad50007bb0 con 0x7fad5c106160 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad58ff9640 1 -- 172.21.15.7:0/3948042627 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad50007d50 con 0x7fad5c106160 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad58ff9640 1 -- 172.21.15.7:0/3948042627 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5000f450 con 0x7fad5c106160 2024-04-21T08:49:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad62255640 1 -- 172.21.15.7:0/3948042627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad5c1a7270 con 0x7fad5c106160 2024-04-21T08:49:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.776+0000 7fad62255640 1 -- 172.21.15.7:0/3948042627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad5c1a77c0 con 0x7fad5c106160 2024-04-21T08:49:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.777+0000 7fad62255640 1 -- 172.21.15.7:0/3948042627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad5c102560 con 0x7fad5c106160 2024-04-21T08:49:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.779+0000 7fad58ff9640 1 -- 172.21.15.7:0/3948042627 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fad50004f20 con 0x7fad5c106160 2024-04-21T08:49:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.779+0000 7fad58ff9640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fad440a3d50 0x7fad440a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.779+0000 7fad5b7fe640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fad440a3d50 0x7fad440a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.779+0000 7fad58ff9640 1 -- 172.21.15.7:0/3948042627 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fad500d0750 con 0x7fad5c106160 2024-04-21T08:49:14.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.780+0000 7fad5b7fe640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fad440a3d50 0x7fad440a6210 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fad4c005fd0 tx=0x7fad4c009500 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:14.784+0000 7fad58ff9640 1 -- 172.21.15.7:0/3948042627 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad50014030 con 0x7fad5c106160 2024-04-21T08:49:14.953 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:14 smithi007 ceph-mon[28477]: pgmap v140: 129 pgs: 129 active+clean; 957 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 1.33k op/s 2024-04-21T08:49:15.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.029+0000 7fad62255640 1 -- 172.21.15.7:0/3948042627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad5c069250 con 0x7fad5c106160 2024-04-21T08:49:15.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.030+0000 7fad58ff9640 1 -- 172.21.15.7:0/3948042627 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7fad5009c3b0 con 0x7fad5c106160 2024-04-21T08:49:15.029 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:49:15.030 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:49:15.030 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:49:15.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 -- 172.21.15.7:0/3948042627 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fad440a3d50 msgr2=0x7fad440a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fad440a3d50 0x7fad440a6210 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fad4c005fd0 tx=0x7fad4c009500 comp rx=0 tx=0).stop 2024-04-21T08:49:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 -- 172.21.15.7:0/3948042627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c106160 msgr2=0x7fad5c10ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c106160 0x7fad5c10ea00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fad5000c970 tx=0x7fad5000ce30 comp rx=0 tx=0).stop 2024-04-21T08:49:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 -- 172.21.15.7:0/3948042627 shutdown_connections 2024-04-21T08:49:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fad440a3d50 0x7fad440a6210 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad5c10ef40 0x7fad5c1a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c106160 0x7fad5c10ea00 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 --2- 172.21.15.7:0/3948042627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad5c101090 0x7fad5c10e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 -- 172.21.15.7:0/3948042627 >> 172.21.15.7:0/3948042627 conn(0x7fad5c0771d0 msgr2=0x7fad5c0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 -- 172.21.15.7:0/3948042627 shutdown_connections 2024-04-21T08:49:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.032+0000 7fad62255640 1 -- 172.21.15.7:0/3948042627 wait complete. 2024-04-21T08:49:15.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:49:15.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:14 smithi016 ceph-mon[31259]: pgmap v140: 129 pgs: 129 active+clean; 957 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 1.33k op/s 2024-04-21T08:49:15.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:14 smithi077 ceph-mon[31615]: pgmap v140: 129 pgs: 129 active+clean; 957 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 1.33k op/s 2024-04-21T08:49:15.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.155+0000 7f5a7c932640 1 -- 172.21.15.7:0/3494410943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a78106180 msgr2=0x7f5a7810dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.155+0000 7f5a7c932640 1 --2- 172.21.15.7:0/3494410943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a78106180 0x7f5a7810dd50 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009ca0 tx=0x7f5a7002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.155+0000 7f5a7c932640 1 -- 172.21.15.7:0/3494410943 shutdown_connections 2024-04-21T08:49:15.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.155+0000 7f5a7c932640 1 --2- 172.21.15.7:0/3494410943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a78106180 0x7f5a7810dd50 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.155+0000 7f5a7c932640 1 --2- 172.21.15.7:0/3494410943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a78101a60 0x7f5a78105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.155+0000 7f5a7c932640 1 --2- 172.21.15.7:0/3494410943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a781010b0 0x7f5a78101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.155+0000 7f5a7c932640 1 -- 172.21.15.7:0/3494410943 >> 172.21.15.7:0/3494410943 conn(0x7f5a78078000 msgr2=0x7f5a78078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.155+0000 7f5a7c932640 1 -- 172.21.15.7:0/3494410943 shutdown_connections 2024-04-21T08:49:15.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.155+0000 7f5a7c932640 1 -- 172.21.15.7:0/3494410943 wait complete. 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.156+0000 7f5a7c932640 1 Processor -- start 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.156+0000 7f5a7c932640 1 -- start start 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7c932640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a781010b0 0x7f5a7810e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7c932640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a78106180 0x7f5a7810ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7c932640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a7810f160 0x7f5a781a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7c932640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5a78112200 con 0x7f5a781010b0 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7c932640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5a78112080 con 0x7f5a7810f160 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7c932640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5a78112380 con 0x7f5a78106180 2024-04-21T08:49:15.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a781010b0 0x7f5a7810e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a781010b0 0x7f5a7810e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34776/0 (socket says 172.21.15.7:34776) 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7659c640 1 -- 172.21.15.7:0/286227488 learned_addr learned my addr 172.21.15.7:0/286227488 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a6dd9b640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a78106180 0x7f5a7810ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a76d9d640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a7810f160 0x7f5a781a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7659c640 1 -- 172.21.15.7:0/286227488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a78106180 msgr2=0x7f5a7810ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7659c640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a78106180 0x7f5a7810ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7659c640 1 -- 172.21.15.7:0/286227488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a7810f160 msgr2=0x7f5a781a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7659c640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a7810f160 0x7f5a781a68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7659c640 1 -- 172.21.15.7:0/286227488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a781a6ff0 con 0x7f5a781010b0 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a76d9d640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a7810f160 0x7f5a781a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.157+0000 7f5a7659c640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a781010b0 0x7f5a7810e6e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800e9f0 tx=0x7f5a6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.158+0000 7f5a6ffff640 1 -- 172.21.15.7:0/286227488 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6800cd80 con 0x7f5a781010b0 2024-04-21T08:49:15.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.158+0000 7f5a6ffff640 1 -- 172.21.15.7:0/286227488 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a680108e0 con 0x7f5a781010b0 2024-04-21T08:49:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.158+0000 7f5a7c932640 1 -- 172.21.15.7:0/286227488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a781a72e0 con 0x7f5a781010b0 2024-04-21T08:49:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.158+0000 7f5a6ffff640 1 -- 172.21.15.7:0/286227488 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68018980 con 0x7f5a781010b0 2024-04-21T08:49:15.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.158+0000 7f5a7c932640 1 -- 172.21.15.7:0/286227488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a781a7880 con 0x7f5a781010b0 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.160+0000 7f5a6ffff640 1 -- 172.21.15.7:0/286227488 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f5a68010430 con 0x7f5a781010b0 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.160+0000 7f5a7f5fe740 1 Processor -- start 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7f5fe740 1 -- start start 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c71764d0 0x55d5c71768b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 0x55d5c717f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c717fa00 0x55d5c7181e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d5c70db1f0 con 0x55d5c71764d0 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d5c70784b0 con 0x55d5c7176e80 2024-04-21T08:49:15.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d5c70d71b0 con 0x55d5c717fa00 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 0x55d5c717f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a76d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c71764d0 0x55d5c71768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 0x55d5c717f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33230/0 (socket says 172.21.15.7:33230) 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a76d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c71764d0 0x55d5c71768b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34784/0 (socket says 172.21.15.7:34784) 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7659c640 1 -- 172.21.15.7:0/3307270363 learned_addr learned my addr 172.21.15.7:0/3307270363 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a6ffff640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a640a3dd0 0x7f5a640a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a6dd9b640 1 --2- 172.21.15.7:0/3307270363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c717fa00 0x55d5c7181e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a76d9d640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a640a3dd0 0x7f5a640a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7659c640 1 -- 172.21.15.7:0/3307270363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c717fa00 msgr2=0x55d5c7181e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7659c640 1 --2- 172.21.15.7:0/3307270363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c717fa00 0x55d5c7181e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7659c640 1 -- 172.21.15.7:0/3307270363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c71764d0 msgr2=0x55d5c71768b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7659c640 1 --2- 172.21.15.7:0/3307270363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c71764d0 0x55d5c71768b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a6ffff640 1 -- 172.21.15.7:0/286227488 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f5a680d0fe0 con 0x7f5a781010b0 2024-04-21T08:49:15.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a7659c640 1 -- 172.21.15.7:0/3307270363 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5c706a1b0 con 0x55d5c7176e80 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.161+0000 7f5a6dd9b640 1 --2- 172.21.15.7:0/3307270363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c717fa00 0x55d5c7181e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.162+0000 7f5a7659c640 1 --2- 172.21.15.7:0/3307270363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 0x55d5c717f430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5a680ce250 tx=0x7f5a6800cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68bffe9806c913a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.162+0000 7f5a76d9d640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a640a3dd0 0x7f5a640a6290 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5a7002c630 tx=0x7f5a7002c0f0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.162+0000 7f5a6cd99640 1 -- 172.21.15.7:0/3307270363 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68010720 con 0x55d5c7176e80 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.162+0000 7f5a6cd99640 1 -- 172.21.15.7:0/3307270363 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a6809d230 con 0x55d5c7176e80 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.162+0000 7f5a6cd99640 1 -- 172.21.15.7:0/3307270363 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6809aec0 con 0x55d5c7176e80 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/3307270363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 msgr2=0x55d5c717f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/3307270363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 0x55d5c717f430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5a680ce250 tx=0x7f5a6800cf20 comp rx=0 tx=0).stop 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/3307270363 shutdown_connections 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/3307270363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c717fa00 0x55d5c7181e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/3307270363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 0x55d5c717f430 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/3307270363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c71764d0 0x55d5c71768b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/3307270363 >> 172.21.15.7:0/3307270363 conn(0x55d5c7170ba0 msgr2=0x55d5c7170f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/3307270363 shutdown_connections 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/3307270363 wait complete. 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 -- start start 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 Processor -- start 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c717f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c7176e80 0x55d5c717c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c717fa00 0x55d5c717cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d5c7182910 con 0x55d5c7176e80 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d5c7094500 con 0x55d5c71764d0 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.163+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d5c7182a90 con 0x55d5c717fa00 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7f5fe740 10 client.0 _ll_register_callbacks cb 0x55d5c7062b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a6dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c7176e80 0x55d5c717c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c717f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c717f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33246/0 (socket says 172.21.15.7:33246) 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7659c640 1 -- 172.21.15.7:0/2461824325 learned_addr learned my addr 172.21.15.7:0/2461824325 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a76d9d640 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c717fa00 0x55d5c717cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7659c640 1 -- 172.21.15.7:0/2461824325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c717fa00 msgr2=0x55d5c717cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7659c640 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c717fa00 0x55d5c717cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7659c640 1 -- 172.21.15.7:0/2461824325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c7176e80 msgr2=0x55d5c717c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7659c640 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c7176e80 0x55d5c717c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7659c640 1 -- 172.21.15.7:0/2461824325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5c6ded3f0 con 0x55d5c71764d0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a6dd9b640 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c7176e80 0x55d5c717c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7659c640 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c717f270 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5a680a5270 tx=0x7f5a680107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c95d156a56e0cdad server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2461824325 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6809de60 con 0x55d5c71764d0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2461824325 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f5a680991a0 con 0x55d5c71764d0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2461824325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a6809d230 con 0x55d5c71764d0 2024-04-21T08:49:15.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2461824325 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5a680b6080 con 0x55d5c71764d0 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7f5fe740 10 client.26171 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.164+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2461824325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d5c6eb77d0 con 0x55d5c71764d0 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.165+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2461824325 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a6809d230 con 0x55d5c71764d0 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.165+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2461824325 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f5a68022ce0 con 0x55d5c71764d0 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.165+0000 7f5a7f5fe740 10 client.26171 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.165+0000 7f5a7f5fe740 10 client.26171 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.165+0000 7f5a7f5fe740 20 client.26171 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.165+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5c718fa00 0x55d5c7191df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.165+0000 7f5a7f5fe740 4 client.26171 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.165+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2461824325 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5c707a930 con 0x55d5c718fa00 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.165+0000 7f5a6dd9b640 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5c718fa00 0x55d5c7191df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.165+0000 7f5a5dffb640 10 client.26171 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:15.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.165+0000 7f5a6dd9b640 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5c718fa00 0x55d5c7191df0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f648eb9181a35b52 server_cookie=c509b4f5ec6b8bb6 in_seq=0 out_seq=0 2024-04-21T08:49:15.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2461824325 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d5c707a930 con 0x55d5c718fa00 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a5dffb640 10 client.26171 handle_command_reply: tid=0 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 1 client.26171 shutdown 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2461824325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 msgr2=0x55d5c717f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c717f270 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5a680a5270 tx=0x7f5a680107a0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2461824325 shutdown_connections 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5c718fa00 0x55d5c7191df0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c717fa00 0x55d5c717cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a5dffb640 0 client.26171 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c7176e80 0x55d5c717c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2461824325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c717f270 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2461824325 >> 172.21.15.7:0/2461824325 conn(0x55d5c7170ba0 msgr2=0x55d5c71728f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2461824325 shutdown_connections 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2461824325 wait complete. 2024-04-21T08:49:15.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.167+0000 7f5a7f5fe740 20 client.26171 trim_cache size 0 max 16384 2024-04-21T08:49:15.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.181+0000 7f5a7f5fe740 1 Processor -- start 2024-04-21T08:49:15.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.181+0000 7f5a7f5fe740 1 -- start start 2024-04-21T08:49:15.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c718b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c7176e80 0x55d5c716fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 0x55d5c717eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5a781af740 con 0x55d5c717fa00 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5a781b1fc0 con 0x55d5c71764d0 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5a781afdb0 con 0x55d5c7176e80 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c718b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a76d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 0x55d5c717eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c718b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33248/0 (socket says 172.21.15.7:33248) 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7659c640 1 -- 172.21.15.7:0/2784547278 learned_addr learned my addr 172.21.15.7:0/2784547278 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a6dd9b640 1 --2- 172.21.15.7:0/2784547278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c7176e80 0x55d5c716fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7659c640 1 -- 172.21.15.7:0/2784547278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c7176e80 msgr2=0x55d5c716fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7659c640 1 --2- 172.21.15.7:0/2784547278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c7176e80 0x55d5c716fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7659c640 1 -- 172.21.15.7:0/2784547278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 msgr2=0x55d5c717eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7659c640 1 --2- 172.21.15.7:0/2784547278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 0x55d5c717eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7659c640 1 -- 172.21.15.7:0/2784547278 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5c7176c20 con 0x55d5c71764d0 2024-04-21T08:49:15.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a76d9d640 1 --2- 172.21.15.7:0/2784547278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 0x55d5c717eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:15.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a7659c640 1 --2- 172.21.15.7:0/2784547278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c718b7d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5a680ce340 tx=0x7f5a6800cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65aa69ef9779de42 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a5ffff640 1 -- 172.21.15.7:0/2784547278 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6809de60 con 0x55d5c71764d0 2024-04-21T08:49:15.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a5ffff640 1 -- 172.21.15.7:0/2784547278 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a6809d230 con 0x55d5c71764d0 2024-04-21T08:49:15.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.182+0000 7f5a5ffff640 1 -- 172.21.15.7:0/2784547278 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68098880 con 0x55d5c71764d0 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.183+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2784547278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 msgr2=0x55d5c718b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.183+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2784547278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c718b7d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5a680ce340 tx=0x7f5a6800cf20 comp rx=0 tx=0).stop 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.183+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2784547278 shutdown_connections 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.183+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2784547278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 0x55d5c717eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.183+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2784547278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c7176e80 0x55d5c716fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.183+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2784547278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c71764d0 0x55d5c718b7d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.183+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2784547278 >> 172.21.15.7:0/2784547278 conn(0x55d5c7170ba0 msgr2=0x55d5c716f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.183+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2784547278 shutdown_connections 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.183+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2784547278 wait complete. 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.183+0000 7f5a7f5fe740 1 -- start start 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a7f5fe740 1 Processor -- start 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c71764d0 0x55d5c7187fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 0x55d5c7178f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a7f5fe740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 0x55d5c7179480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5a781b0480 con 0x55d5c717fa00 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5a781b18f0 con 0x55d5c7176e80 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a7f5fe740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5a781b0b50 con 0x55d5c71764d0 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a7f5fe740 10 client.0 _ll_register_callbacks cb 0x7f5a6809c9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a76d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 0x55d5c7179480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a7659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c71764d0 0x55d5c7187fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a6dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 0x55d5c7178f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a76d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 0x55d5c7179480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34814/0 (socket says 172.21.15.7:34814) 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a76d9d640 1 -- 172.21.15.7:0/2759289378 learned_addr learned my addr 172.21.15.7:0/2759289378 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a76d9d640 1 -- 172.21.15.7:0/2759289378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c71764d0 msgr2=0x55d5c7187fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a76d9d640 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c71764d0 0x55d5c7187fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a76d9d640 1 -- 172.21.15.7:0/2759289378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 msgr2=0x55d5c7178f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a76d9d640 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 0x55d5c7178f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a76d9d640 1 -- 172.21.15.7:0/2759289378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5c7179bc0 con 0x55d5c717fa00 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a6dd9b640 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 0x55d5c7178f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a7659c640 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c71764d0 0x55d5c7187fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a76d9d640 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 0x55d5c7179480 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5a70036000 tx=0x7f5a70009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cad5f8061feb87d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.184+0000 7f5a5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:15.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2759289378 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a70054070 con 0x55d5c717fa00 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2759289378 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f5a70004310 con 0x55d5c717fa00 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2759289378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a6809d230 con 0x55d5c717fa00 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2759289378 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5a70060050 con 0x55d5c717fa00 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a7f5fe740 10 client.16626 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2759289378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d5c71f5b30 con 0x55d5c717fa00 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2759289378 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a6809d230 con 0x55d5c717fa00 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2759289378 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f5a70034c80 con 0x55d5c717fa00 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a7f5fe740 10 client.16626 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a7f5fe740 10 client.16626 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a7f5fe740 20 client.16626 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5c71f62b0 0x55d5c71f86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a7f5fe740 4 client.16626 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2759289378 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d5c70e8c10 con 0x55d5c71f62b0 2024-04-21T08:49:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.185+0000 7f5a7659c640 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5c71f62b0 0x55d5c71f86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.186+0000 7f5a5dffb640 10 client.16626 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.186+0000 7f5a7659c640 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5c71f62b0 0x55d5c71f86d0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=734d0a18c07174be server_cookie=1f2212736f4ad179 in_seq=0 out_seq=0 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a5dffb640 1 -- 172.21.15.7:0/2759289378 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d5c70e8c10 con 0x55d5c71f62b0 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a5dffb640 10 client.16626 handle_command_reply: tid=0 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 1 client.16626 shutdown 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2759289378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 msgr2=0x55d5c7179480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 0x55d5c7179480 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5a70036000 tx=0x7f5a70009360 comp rx=0 tx=0).stop 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2759289378 shutdown_connections 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5c71f62b0 0x55d5c71f86d0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5c717fa00 0x55d5c7179480 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5c7176e80 0x55d5c7178f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a5dffb640 0 client.16626 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 1 --2- 172.21.15.7:0/2759289378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5c71764d0 0x55d5c7187fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2759289378 >> 172.21.15.7:0/2759289378 conn(0x55d5c7170ba0 msgr2=0x55d5c716fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2759289378 shutdown_connections 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 1 -- 172.21.15.7:0/2759289378 wait complete. 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7f5fe740 20 client.16626 trim_cache size 0 max 16384 2024-04-21T08:49:15.193 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.187+0000 7f5a7c932640 1 -- 172.21.15.7:0/286227488 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a640a3dd0 msgr2=0x7f5a640a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.188+0000 7f5a7c932640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a640a3dd0 0x7f5a640a6290 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5a7002c630 tx=0x7f5a7002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.188+0000 7f5a7c932640 1 -- 172.21.15.7:0/286227488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a781010b0 msgr2=0x7f5a7810e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.188+0000 7f5a7c932640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a781010b0 0x7f5a7810e6e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800e9f0 tx=0x7f5a6800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.188+0000 7f5a7c932640 1 -- 172.21.15.7:0/286227488 shutdown_connections 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.188+0000 7f5a7c932640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a7810f160 0x7f5a781a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.188+0000 7f5a7c932640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a640a3dd0 0x7f5a640a6290 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.188+0000 7f5a7c932640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a78106180 0x7f5a7810ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.188+0000 7f5a7c932640 1 --2- 172.21.15.7:0/286227488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a781010b0 0x7f5a7810e6e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.188+0000 7f5a7c932640 1 -- 172.21.15.7:0/286227488 >> 172.21.15.7:0/286227488 conn(0x7f5a78078000 msgr2=0x7f5a780ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.188+0000 7f5a7c932640 1 -- 172.21.15.7:0/286227488 shutdown_connections 2024-04-21T08:49:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:15.188+0000 7f5a7c932640 1 -- 172.21.15.7:0/286227488 wait complete. 2024-04-21T08:49:16.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:15 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3948042627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:16.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:15 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3948042627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:16.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:15 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3948042627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:16.199 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:49:16.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:49:16.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:49:16.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.313+0000 7f3bef26c640 1 -- 172.21.15.7:0/807889949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be8100eb0 msgr2=0x7f3be8101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.313+0000 7f3bef26c640 1 --2- 172.21.15.7:0/807889949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be8100eb0 0x7f3be8101290 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f3be400d910 tx=0x7f3be402dbb0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.313+0000 7f3bef26c640 1 -- 172.21.15.7:0/807889949 shutdown_connections 2024-04-21T08:49:16.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.313+0000 7f3bef26c640 1 --2- 172.21.15.7:0/807889949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be8105f80 0x7f3be810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.313+0000 7f3bef26c640 1 --2- 172.21.15.7:0/807889949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be8101860 0x7f3be81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.313+0000 7f3bef26c640 1 --2- 172.21.15.7:0/807889949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be8100eb0 0x7f3be8101290 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.313+0000 7f3bef26c640 1 -- 172.21.15.7:0/807889949 >> 172.21.15.7:0/807889949 conn(0x7f3be80fc790 msgr2=0x7f3be80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.313+0000 7f3bef26c640 1 -- 172.21.15.7:0/807889949 shutdown_connections 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.313+0000 7f3bef26c640 1 -- 172.21.15.7:0/807889949 wait complete. 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3bef26c640 1 Processor -- start 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3bef26c640 1 -- start start 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3bef26c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be8100eb0 0x7f3be81964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3bef26c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be8105f80 0x7f3be8196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3bef26c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be819ad80 0x7f3be819b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3bef26c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3be810fe40 con 0x7f3be8105f80 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3bef26c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3be810fcc0 con 0x7f3be819ad80 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3bef26c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3be810ffc0 con 0x7f3be8100eb0 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3beda69640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be8105f80 0x7f3be8196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3beda69640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be8105f80 0x7f3be8196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34828/0 (socket says 172.21.15.7:34828) 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3beda69640 1 -- 172.21.15.7:0/4257742297 learned_addr learned my addr 172.21.15.7:0/4257742297 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3beea6b640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be819ad80 0x7f3be819b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3bee26a640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be8100eb0 0x7f3be81964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3beda69640 1 -- 172.21.15.7:0/4257742297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be8100eb0 msgr2=0x7f3be81964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3beda69640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be8100eb0 0x7f3be81964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3beda69640 1 -- 172.21.15.7:0/4257742297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be819ad80 msgr2=0x7f3be819b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3beda69640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be819ad80 0x7f3be819b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3beda69640 1 -- 172.21.15.7:0/4257742297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be819b870 con 0x7f3be8105f80 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.314+0000 7f3bee26a640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be8100eb0 0x7f3be81964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.315+0000 7f3beda69640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be8105f80 0x7f3be8196a00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800b450 tx=0x7f3bd800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.315+0000 7f3bdf7fe640 1 -- 172.21.15.7:0/4257742297 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8015020 con 0x7f3be8105f80 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.315+0000 7f3bef26c640 1 -- 172.21.15.7:0/4257742297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be819bb60 con 0x7f3be8105f80 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.315+0000 7f3bdf7fe640 1 -- 172.21.15.7:0/4257742297 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bd8011d40 con 0x7f3be8105f80 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.315+0000 7f3bdf7fe640 1 -- 172.21.15.7:0/4257742297 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd800f340 con 0x7f3be8105f80 2024-04-21T08:49:16.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.315+0000 7f3bef26c640 1 -- 172.21.15.7:0/4257742297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be81a33a0 con 0x7f3be8105f80 2024-04-21T08:49:16.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.318+0000 7f3bdf7fe640 1 -- 172.21.15.7:0/4257742297 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f3bd8018020 con 0x7f3be8105f80 2024-04-21T08:49:16.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.318+0000 7f3bf3195740 1 Processor -- start 2024-04-21T08:49:16.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.318+0000 7f3bf3195740 1 -- start start 2024-04-21T08:49:16.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.318+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64721960 0x555a64721d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.318+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64722310 0x555a6472ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.318+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.318+0000 7f3bf3195740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x555a64685f00 con 0x555a64721960 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.318+0000 7f3bf3195740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555a64623590 con 0x555a6472b410 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.318+0000 7f3bf3195740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555a64681ec0 con 0x555a64722310 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beea6b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64721960 0x555a64721d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beda69640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beea6b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64721960 0x555a64721d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34830/0 (socket says 172.21.15.7:34830) 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beea6b640 1 -- 172.21.15.7:0/4184457876 learned_addr learned my addr 172.21.15.7:0/4184457876 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3bee26a640 1 --2- 172.21.15.7:0/4184457876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64722310 0x555a6472ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3bdf7fe640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3bd40a3d20 0x7f3bd40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beda69640 1 -- 172.21.15.7:0/4184457876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64722310 msgr2=0x555a6472ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beda69640 1 --2- 172.21.15.7:0/4184457876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64722310 0x555a6472ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beda69640 1 -- 172.21.15.7:0/4184457876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64721960 msgr2=0x555a64721d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beda69640 1 --2- 172.21.15.7:0/4184457876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64721960 0x555a64721d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beda69640 1 -- 172.21.15.7:0/4184457876 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a644607e0 con 0x555a6472b410 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beea6b640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3bd40a3d20 0x7f3bd40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3bdf7fe640 1 -- 172.21.15.7:0/4257742297 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f3bd80d04f0 con 0x7f3be8105f80 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beda69640 1 --2- 172.21.15.7:0/4184457876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472b870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80cda00 tx=0x7f3bd8098620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b54b3d4de83b0e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3bdcbfe640 1 -- 172.21.15.7:0/4184457876 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd809c020 con 0x555a6472b410 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.319+0000 7f3beea6b640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3bd40a3d20 0x7f3bd40a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3be0009fd0 tx=0x7f3be0004a40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.320+0000 7f3bdcbfe640 1 -- 172.21.15.7:0/4184457876 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bd809c950 con 0x555a6472b410 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.320+0000 7f3bdcbfe640 1 -- 172.21.15.7:0/4184457876 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd809bbb0 con 0x555a6472b410 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.320+0000 7f3bf3195740 1 -- 172.21.15.7:0/4184457876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 msgr2=0x555a6472b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.320+0000 7f3bf3195740 1 --2- 172.21.15.7:0/4184457876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472b870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80cda00 tx=0x7f3bd8098620 comp rx=0 tx=0).stop 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.320+0000 7f3bf3195740 1 -- 172.21.15.7:0/4184457876 shutdown_connections 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.320+0000 7f3bf3195740 1 --2- 172.21.15.7:0/4184457876 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472b870 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.320+0000 7f3bf3195740 1 --2- 172.21.15.7:0/4184457876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64722310 0x555a6472ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.320+0000 7f3bf3195740 1 --2- 172.21.15.7:0/4184457876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64721960 0x555a64721d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.320+0000 7f3bf3195740 1 -- 172.21.15.7:0/4184457876 >> 172.21.15.7:0/4184457876 conn(0x555a6471c150 msgr2=0x555a6471c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.320+0000 7f3bf3195740 1 -- 172.21.15.7:0/4184457876 shutdown_connections 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.320+0000 7f3bf3195740 1 -- 172.21.15.7:0/4184457876 wait complete. 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3bf3195740 1 -- start start 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3bf3195740 1 Processor -- start 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64721960 0x555a6472ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3bf3195740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x555a64732350 con 0x555a64722310 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3bf3195740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555a6463f580 con 0x555a6472b410 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3bf3195740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555a647324d0 con 0x555a64721960 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3bf3195740 10 client.0 _ll_register_callbacks cb 0x555a64645650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3beda69640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3beda69640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34842/0 (socket says 172.21.15.7:34842) 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3beda69640 1 -- 172.21.15.7:0/2828852594 learned_addr learned my addr 172.21.15.7:0/2828852594 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3beea6b640 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3bee26a640 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64721960 0x555a6472ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3beda69640 1 -- 172.21.15.7:0/2828852594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64721960 msgr2=0x555a6472ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3beda69640 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64721960 0x555a6472ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3beda69640 1 -- 172.21.15.7:0/2828852594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 msgr2=0x555a6472f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3beda69640 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.321+0000 7f3beda69640 1 -- 172.21.15.7:0/2828852594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a644c0f70 con 0x555a64722310 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3beda69640 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472f440 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80d6150 tx=0x7f3bd80d08c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbdf4a18d179a840 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/2828852594 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd809bd90 con 0x555a64722310 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/2828852594 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f3bd80988c0 con 0x555a64722310 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bf3195740 1 -- 172.21.15.7:0/2828852594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bd809c950 con 0x555a64722310 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/2828852594 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3bd809bbb0 con 0x555a64722310 2024-04-21T08:49:16.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bf3195740 10 client.16638 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bf3195740 1 -- 172.21.15.7:0/2828852594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555a64358170 con 0x555a64722310 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/2828852594 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bd809c950 con 0x555a64722310 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/2828852594 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f3bd8010500 con 0x555a64722310 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bf3195740 10 client.16638 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bf3195740 10 client.16638 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bf3195740 20 client.16638 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bf3195740 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a6473f790 0x555a64741b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bf3195740 4 client.16638 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.322+0000 7f3bf3195740 1 -- 172.21.15.7:0/2828852594 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a64693920 con 0x555a6473f790 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.323+0000 7f3bee26a640 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a6473f790 0x555a64741b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.323+0000 7f3bd1ffb640 10 client.16638 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.323+0000 7f3bee26a640 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a6473f790 0x555a64741b80 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e289f40913c4e01c server_cookie=53487c60aa9ab16b in_seq=0 out_seq=0 2024-04-21T08:49:16.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/2828852594 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a64693920 con 0x555a6473f790 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bd1ffb640 10 client.16638 handle_command_reply: tid=0 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 1 client.16638 shutdown 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 1 -- 172.21.15.7:0/2828852594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 msgr2=0x555a6472f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472f440 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80d6150 tx=0x7f3bd80d08c0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 1 -- 172.21.15.7:0/2828852594 shutdown_connections 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a6473f790 0x555a64741b80 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472f440 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 1 --2- 172.21.15.7:0/2828852594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64721960 0x555a6472ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bd1ffb640 0 client.16638 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 1 -- 172.21.15.7:0/2828852594 >> 172.21.15.7:0/2828852594 conn(0x555a6471c150 msgr2=0x555a6471dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 1 -- 172.21.15.7:0/2828852594 shutdown_connections 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 1 -- 172.21.15.7:0/2828852594 wait complete. 2024-04-21T08:49:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.324+0000 7f3bf3195740 20 client.16638 trim_cache size 0 max 16384 2024-04-21T08:49:16.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.342+0000 7f3bf3195740 1 Processor -- start 2024-04-21T08:49:16.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.342+0000 7f3bf3195740 1 -- start start 2024-04-21T08:49:16.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.342+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64721960 0x555a64731630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.342+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.342+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.342+0000 7f3bf3195740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81ab1e0 con 0x555a64722310 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.342+0000 7f3bf3195740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81ada60 con 0x555a6472b410 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.342+0000 7f3bf3195740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81ab850 con 0x555a64721960 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3beda69640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3beda69640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34844/0 (socket says 172.21.15.7:34844) 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3beda69640 1 -- 172.21.15.7:0/473337349 learned_addr learned my addr 172.21.15.7:0/473337349 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3bee26a640 1 --2- 172.21.15.7:0/473337349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64721960 0x555a64731630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3beea6b640 1 --2- 172.21.15.7:0/473337349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3beda69640 1 -- 172.21.15.7:0/473337349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64721960 msgr2=0x555a64731630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3beda69640 1 --2- 172.21.15.7:0/473337349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64721960 0x555a64731630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3beda69640 1 -- 172.21.15.7:0/473337349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 msgr2=0x555a6472f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3beda69640 1 --2- 172.21.15.7:0/473337349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3beda69640 1 -- 172.21.15.7:0/473337349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a64717a10 con 0x555a64722310 2024-04-21T08:49:16.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3beea6b640 1 --2- 172.21.15.7:0/473337349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:16.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3beda69640 1 --2- 172.21.15.7:0/473337349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472c820 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800be10 tx=0x7f3bd80108a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06811e91f421a14 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3bd3fff640 1 -- 172.21.15.7:0/473337349 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd809c020 con 0x555a64722310 2024-04-21T08:49:16.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3bd3fff640 1 -- 172.21.15.7:0/473337349 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bd80a0600 con 0x555a64722310 2024-04-21T08:49:16.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.343+0000 7f3bd3fff640 1 -- 172.21.15.7:0/473337349 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8010500 con 0x555a64722310 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.344+0000 7f3bf3195740 1 -- 172.21.15.7:0/473337349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 msgr2=0x555a6472c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.344+0000 7f3bf3195740 1 --2- 172.21.15.7:0/473337349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472c820 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800be10 tx=0x7f3bd80108a0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.344+0000 7f3bf3195740 1 -- 172.21.15.7:0/473337349 shutdown_connections 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.344+0000 7f3bf3195740 1 --2- 172.21.15.7:0/473337349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6472b410 0x555a6472f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.344+0000 7f3bf3195740 1 --2- 172.21.15.7:0/473337349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a64722310 0x555a6472c820 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.344+0000 7f3bf3195740 1 --2- 172.21.15.7:0/473337349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64721960 0x555a64731630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.344+0000 7f3bf3195740 1 -- 172.21.15.7:0/473337349 >> 172.21.15.7:0/473337349 conn(0x555a6471c150 msgr2=0x555a6472d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.344+0000 7f3bf3195740 1 -- 172.21.15.7:0/473337349 shutdown_connections 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.344+0000 7f3bf3195740 1 -- 172.21.15.7:0/473337349 wait complete. 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3bf3195740 1 -- start start 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3bf3195740 1 Processor -- start 2024-04-21T08:49:16.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a64721960 0x555a64727a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64722310 0x555a64727fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3bf3195740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a6472b410 0x555a647a4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3bf3195740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81abf20 con 0x555a6472b410 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3bf3195740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81ad390 con 0x555a64721960 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3bf3195740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81ac5f0 con 0x555a64722310 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3bf3195740 10 client.0 _ll_register_callbacks cb 0x7f3be0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3beea6b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a6472b410 0x555a647a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3beea6b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a6472b410 0x555a647a4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34846/0 (socket says 172.21.15.7:34846) 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3beea6b640 1 -- 172.21.15.7:0/1688844168 learned_addr learned my addr 172.21.15.7:0/1688844168 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.345+0000 7f3bee26a640 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a64721960 0x555a64727a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3beda69640 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64722310 0x555a64727fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3beea6b640 1 -- 172.21.15.7:0/1688844168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64722310 msgr2=0x555a64727fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3beea6b640 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64722310 0x555a64727fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3beea6b640 1 -- 172.21.15.7:0/1688844168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a64721960 msgr2=0x555a64727a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3beea6b640 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a64721960 0x555a64727a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3beea6b640 1 -- 172.21.15.7:0/1688844168 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a647a4ea0 con 0x555a6472b410 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3beda69640 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64722310 0x555a64727fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3bee26a640 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a64721960 0x555a64727a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3beea6b640 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a6472b410 0x555a647a4960 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3be0021990 tx=0x7f3be0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e76818dd2bad15 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3bd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/1688844168 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be001e930 con 0x555a6472b410 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/1688844168 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f3be001ead0 con 0x555a6472b410 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.346+0000 7f3bf3195740 1 -- 172.21.15.7:0/1688844168 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555a6468d790 con 0x555a6472b410 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/1688844168 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3be001ecd0 con 0x555a6472b410 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bf3195740 10 client.16644 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bf3195740 1 -- 172.21.15.7:0/1688844168 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555a647a6070 con 0x555a6472b410 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/1688844168 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555a6468d790 con 0x555a6472b410 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/1688844168 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f3be002c420 con 0x555a6472b410 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bf3195740 10 client.16644 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:16.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bf3195740 10 client.16644 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bf3195740 20 client.16644 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bf3195740 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a647a6850 0x555a647a8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bf3195740 4 client.16644 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bf3195740 1 -- 172.21.15.7:0/1688844168 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555a64609c50 con 0x555a647a6850 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.347+0000 7f3bee26a640 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a647a6850 0x555a647a8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.348+0000 7f3bee26a640 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a647a6850 0x555a647a8c70 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f58c535c493c3c3f server_cookie=255afce05bb4625d in_seq=0 out_seq=0 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.348+0000 7f3bd1ffb640 10 client.16644 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bd1ffb640 1 -- 172.21.15.7:0/1688844168 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555a64609c50 con 0x555a647a6850 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bd1ffb640 10 client.16644 handle_command_reply: tid=0 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bf3195740 1 client.16644 shutdown 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bf3195740 1 -- 172.21.15.7:0/1688844168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a6472b410 msgr2=0x555a647a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bf3195740 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a6472b410 0x555a647a4960 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3be0021990 tx=0x7f3be0004ab0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bf3195740 1 -- 172.21.15.7:0/1688844168 shutdown_connections 2024-04-21T08:49:16.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bf3195740 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a647a6850 0x555a647a8c70 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bf3195740 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a6472b410 0x555a647a4960 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bf3195740 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a64722310 0x555a64727fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bf3195740 1 --2- 172.21.15.7:0/1688844168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a64721960 0x555a64727a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bf3195740 1 -- 172.21.15.7:0/1688844168 >> 172.21.15.7:0/1688844168 conn(0x555a6471c150 msgr2=0x555a6472df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bd1ffb640 0 client.16644 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:16.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bf3195740 1 -- 172.21.15.7:0/1688844168 shutdown_connections 2024-04-21T08:49:16.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.349+0000 7f3bf3195740 1 -- 172.21.15.7:0/1688844168 wait complete. 2024-04-21T08:49:16.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bf3195740 20 client.16644 trim_cache size 0 max 16384 2024-04-21T08:49:16.354 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:49:16.354 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:49:16.355 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "2f4d0e81-05a6-4410-bd5d-10ca0b65358b", 2024-04-21T08:49:16.355 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:49:16.355 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:49:16.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 -- 172.21.15.7:0/4257742297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3bd40a3d20 msgr2=0x7f3bd40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3bd40a3d20 0x7f3bd40a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3be0009fd0 tx=0x7f3be0004a40 comp rx=0 tx=0).stop 2024-04-21T08:49:16.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 -- 172.21.15.7:0/4257742297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be8105f80 msgr2=0x7f3be8196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be8105f80 0x7f3be8196a00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800b450 tx=0x7f3bd800b910 comp rx=0 tx=0).stop 2024-04-21T08:49:16.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 -- 172.21.15.7:0/4257742297 shutdown_connections 2024-04-21T08:49:16.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3be819ad80 0x7f3be819b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be8105f80 0x7f3be8196a00 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3bd40a3d20 0x7f3bd40a61e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 --2- 172.21.15.7:0/4257742297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be8100eb0 0x7f3be81964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 -- 172.21.15.7:0/4257742297 >> 172.21.15.7:0/4257742297 conn(0x7f3be80fc790 msgr2=0x7f3be80fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 -- 172.21.15.7:0/4257742297 shutdown_connections 2024-04-21T08:49:16.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.350+0000 7f3bef26c640 1 -- 172.21.15.7:0/4257742297 wait complete. 2024-04-21T08:49:16.360 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:49:16.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:49:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.472+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2043909994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e58106120 msgr2=0x7f2e5810dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.472+0000 7f2e5eb11640 1 --2- 172.21.15.7:0/2043909994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e58106120 0x7f2e5810dcf0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009900 tx=0x7f2e4802bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.472+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2043909994 shutdown_connections 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.472+0000 7f2e5eb11640 1 --2- 172.21.15.7:0/2043909994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e58106120 0x7f2e5810dcf0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.472+0000 7f2e5eb11640 1 --2- 172.21.15.7:0/2043909994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e58101a00 0x7f2e58105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.472+0000 7f2e5eb11640 1 --2- 172.21.15.7:0/2043909994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e58101050 0x7f2e58101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.472+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2043909994 >> 172.21.15.7:0/2043909994 conn(0x7f2e58077fa0 msgr2=0x7f2e580783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.472+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2043909994 shutdown_connections 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.472+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2043909994 wait complete. 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.472+0000 7f2e5eb11640 1 Processor -- start 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.472+0000 7f2e5eb11640 1 -- start start 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5eb11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e58101050 0x7f2e581986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5eb11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e58106120 0x7f2e58198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5eb11640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e5819cf80 0x7f2e5819d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5eb11640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2e581121a0 con 0x7f2e5819cf80 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5eb11640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e58112020 con 0x7f2e58101050 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5eb11640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2e58112320 con 0x7f2e58106120 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5d0ae640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e5819cf80 0x7f2e5819d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5d0ae640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e5819cf80 0x7f2e5819d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34872/0 (socket says 172.21.15.7:34872) 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5d0ae640 1 -- 172.21.15.7:0/2191263330 learned_addr learned my addr 172.21.15.7:0/2191263330 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e58101050 0x7f2e581986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e57fff640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e58106120 0x7f2e58198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5d0ae640 1 -- 172.21.15.7:0/2191263330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e58106120 msgr2=0x7f2e58198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5d0ae640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e58106120 0x7f2e58198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5d0ae640 1 -- 172.21.15.7:0/2191263330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e58101050 msgr2=0x7f2e581986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5d0ae640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e58101050 0x7f2e581986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5d0ae640 1 -- 172.21.15.7:0/2191263330 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5819da70 con 0x7f2e5819cf80 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e5d0ae640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e5819cf80 0x7f2e5819d430 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009230 tx=0x7f2e4802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.473+0000 7f2e55ffb640 1 -- 172.21.15.7:0/2191263330 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48043020 con 0x7f2e5819cf80 2024-04-21T08:49:16.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.474+0000 7f2e55ffb640 1 -- 172.21.15.7:0/2191263330 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e4802ca70 con 0x7f2e5819cf80 2024-04-21T08:49:16.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.474+0000 7f2e55ffb640 1 -- 172.21.15.7:0/2191263330 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48003960 con 0x7f2e5819cf80 2024-04-21T08:49:16.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.474+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2191263330 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5819dd00 con 0x7f2e5819cf80 2024-04-21T08:49:16.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.474+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2191263330 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e581a5520 con 0x7f2e5819cf80 2024-04-21T08:49:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.476+0000 7f2e55ffb640 1 -- 172.21.15.7:0/2191263330 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f2e48046020 con 0x7f2e5819cf80 2024-04-21T08:49:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.476+0000 7f2e617d7740 1 Processor -- start 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.476+0000 7f2e617d7740 1 -- start start 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796e1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1ea0 0x5620796ea540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 0x5620796ece70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e617d7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5620796461e0 con 0x5620796eaa80 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e617d7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5620795e3390 con 0x5620796e1580 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e617d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5620796421a0 con 0x5620796e1ea0 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796e1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e57fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796e1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33356/0 (socket says 172.21.15.7:33356) 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e57fff640 1 -- 172.21.15.7:0/940034499 learned_addr learned my addr 172.21.15.7:0/940034499 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/940034499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1ea0 0x5620796ea540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e55ffb640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f2e440a3d20 0x7f2e440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e57fff640 1 --2- 172.21.15.7:0/940034499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 0x5620796ece70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e5d0ae640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f2e440a3d20 0x7f2e440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e5c8ad640 1 -- 172.21.15.7:0/940034499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 msgr2=0x5620796e1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/940034499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796e1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e5c8ad640 1 -- 172.21.15.7:0/940034499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 msgr2=0x5620796ece70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/940034499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 0x5620796ece70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e5c8ad640 1 -- 172.21.15.7:0/940034499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620794227d0 con 0x5620796e1ea0 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e57fff640 1 --2- 172.21.15.7:0/940034499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796e1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e55ffb640 1 -- 172.21.15.7:0/2191263330 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f2e480f2bc0 con 0x7f2e5819cf80 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.477+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/940034499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1ea0 0x5620796ea540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2e50009fa0 tx=0x7f2e500118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe362af833a96c3a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e5d0ae640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f2e440a3d20 0x7f2e440a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2e480f00e0 tx=0x7f2e4803abc0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e4effd640 1 -- 172.21.15.7:0/940034499 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50003c30 con 0x5620796e1ea0 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e4effd640 1 -- 172.21.15.7:0/940034499 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e50003dd0 con 0x5620796e1ea0 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e4effd640 1 -- 172.21.15.7:0/940034499 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5000f9f0 con 0x5620796e1ea0 2024-04-21T08:49:16.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e617d7740 1 -- 172.21.15.7:0/940034499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1ea0 msgr2=0x5620796ea540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e617d7740 1 --2- 172.21.15.7:0/940034499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1ea0 0x5620796ea540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2e50009fa0 tx=0x7f2e500118d0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e617d7740 1 -- 172.21.15.7:0/940034499 shutdown_connections 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e617d7740 1 --2- 172.21.15.7:0/940034499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 0x5620796ece70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e617d7740 1 --2- 172.21.15.7:0/940034499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1ea0 0x5620796ea540 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e617d7740 1 --2- 172.21.15.7:0/940034499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796e1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e617d7740 1 -- 172.21.15.7:0/940034499 >> 172.21.15.7:0/940034499 conn(0x5620796dbcb0 msgr2=0x5620796dc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e617d7740 1 -- 172.21.15.7:0/940034499 shutdown_connections 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.478+0000 7f2e617d7740 1 -- 172.21.15.7:0/940034499 wait complete. 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e617d7740 1 -- start start 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e617d7740 1 Processor -- start 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1580 0x5620796e7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796e8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796eaa80 0x5620796e8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e617d7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5620796ed870 con 0x5620796e1ea0 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e617d7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5620795ff2f0 con 0x5620796eaa80 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e617d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5620796ed9f0 con 0x5620796e1580 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e617d7740 10 client.0 _ll_register_callbacks cb 0x56207960c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e5c8ad640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796e8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e5c8ad640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796e8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34892/0 (socket says 172.21.15.7:34892) 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e5c8ad640 1 -- 172.21.15.7:0/3727465362 learned_addr learned my addr 172.21.15.7:0/3727465362 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e57fff640 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796eaa80 0x5620796e8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.479+0000 7f2e57fff640 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1580 0x5620796e7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e5c8ad640 1 -- 172.21.15.7:0/3727465362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1580 msgr2=0x5620796e7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1580 0x5620796e7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e5c8ad640 1 -- 172.21.15.7:0/3727465362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796eaa80 msgr2=0x5620796e8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796eaa80 0x5620796e8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e5c8ad640 1 -- 172.21.15.7:0/3727465362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620793583f0 con 0x5620796e1ea0 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e57fff640 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796eaa80 0x5620796e8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:16.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796e8030 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f2e50004cf0 tx=0x7f2e50004d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8b58e1cec418bb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e37fff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e37fff640 1 -- 172.21.15.7:0/3727465362 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50028020 con 0x5620796e1ea0 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e617d7740 1 -- 172.21.15.7:0/3727465362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e50003dd0 con 0x5620796e1ea0 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e37fff640 1 -- 172.21.15.7:0/3727465362 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f2e5000f940 con 0x5620796e1ea0 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e37fff640 1 -- 172.21.15.7:0/3727465362 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2e50033050 con 0x5620796e1ea0 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e617d7740 10 client.16656 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e617d7740 1 -- 172.21.15.7:0/3727465362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5620793180f0 con 0x5620796e1ea0 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.480+0000 7f2e37fff640 1 -- 172.21.15.7:0/3727465362 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e50003dd0 con 0x5620796e1ea0 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.481+0000 7f2e37fff640 1 -- 172.21.15.7:0/3727465362 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f2e50005040 con 0x5620796e1ea0 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.481+0000 7f2e617d7740 10 client.16656 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.481+0000 7f2e617d7740 10 client.16656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.481+0000 7f2e617d7740 20 client.16656 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.481+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5620796fab70 0x5620796fcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.481+0000 7f2e617d7740 4 client.16656 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:16.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.481+0000 7f2e617d7740 1 -- 172.21.15.7:0/3727465362 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562079653c00 con 0x5620796fab70 2024-04-21T08:49:16.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.481+0000 7f2e57fff640 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5620796fab70 0x5620796fcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.481+0000 7f2e37fff640 10 client.16656 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:16.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.481+0000 7f2e57fff640 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5620796fab70 0x5620796fcf60 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5995b2d87fcbde92 server_cookie=d185ee8c19cb7ca in_seq=0 out_seq=0 2024-04-21T08:49:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e37fff640 1 -- 172.21.15.7:0/3727465362 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562079653c00 con 0x5620796fab70 2024-04-21T08:49:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e37fff640 10 client.16656 handle_command_reply: tid=0 2024-04-21T08:49:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 1 client.16656 shutdown 2024-04-21T08:49:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 1 -- 172.21.15.7:0/3727465362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 msgr2=0x5620796e8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796e8030 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f2e50004cf0 tx=0x7f2e50004d20 comp rx=0 tx=0).stop 2024-04-21T08:49:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 1 -- 172.21.15.7:0/3727465362 shutdown_connections 2024-04-21T08:49:16.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5620796fab70 0x5620796fcf60 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e37fff640 0 client.16656 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796eaa80 0x5620796e8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796e8030 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3727465362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1580 0x5620796e7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 1 -- 172.21.15.7:0/3727465362 >> 172.21.15.7:0/3727465362 conn(0x5620796dbcb0 msgr2=0x5620796dda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 1 -- 172.21.15.7:0/3727465362 shutdown_connections 2024-04-21T08:49:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 1 -- 172.21.15.7:0/3727465362 wait complete. 2024-04-21T08:49:16.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.483+0000 7f2e617d7740 20 client.16656 trim_cache size 0 max 16384 2024-04-21T08:49:16.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e617d7740 1 Processor -- start 2024-04-21T08:49:16.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e617d7740 1 -- start start 2024-04-21T08:49:16.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796ea220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796f6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796eaa80 0x5620796e63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e617d7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2e581ad380 con 0x5620796e1ea0 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e617d7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e581afc00 con 0x5620796e1580 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e617d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2e581ad9f0 con 0x5620796eaa80 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e5c8ad640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e5c8ad640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796f6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34894/0 (socket says 172.21.15.7:34894) 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e5c8ad640 1 -- 172.21.15.7:0/1723172831 learned_addr learned my addr 172.21.15.7:0/1723172831 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e57fff640 1 --2- 172.21.15.7:0/1723172831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796ea220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e5c8ad640 1 -- 172.21.15.7:0/1723172831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796eaa80 msgr2=0x5620796e63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/1723172831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796eaa80 0x5620796e63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e5c8ad640 1 -- 172.21.15.7:0/1723172831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 msgr2=0x5620796ea220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/1723172831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796ea220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.502+0000 7f2e5c8ad640 1 -- 172.21.15.7:0/1723172831 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620796e6040 con 0x5620796e1ea0 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/1723172831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796f6790 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2e50006010 tx=0x7f2e500096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2278ccafc84235 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e4e7fc640 1 -- 172.21.15.7:0/1723172831 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50028040 con 0x5620796e1ea0 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e4e7fc640 1 -- 172.21.15.7:0/1723172831 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e50003d40 con 0x5620796e1ea0 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e4e7fc640 1 -- 172.21.15.7:0/1723172831 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e500051c0 con 0x5620796e1ea0 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e617d7740 1 -- 172.21.15.7:0/1723172831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 msgr2=0x5620796f6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e617d7740 1 --2- 172.21.15.7:0/1723172831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796f6790 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2e50006010 tx=0x7f2e500096c0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e617d7740 1 -- 172.21.15.7:0/1723172831 shutdown_connections 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e617d7740 1 --2- 172.21.15.7:0/1723172831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796eaa80 0x5620796e63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e617d7740 1 --2- 172.21.15.7:0/1723172831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796e1ea0 0x5620796f6790 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e617d7740 1 --2- 172.21.15.7:0/1723172831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796ea220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e617d7740 1 -- 172.21.15.7:0/1723172831 >> 172.21.15.7:0/1723172831 conn(0x5620796dbcb0 msgr2=0x5620796da4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e617d7740 1 -- 172.21.15.7:0/1723172831 shutdown_connections 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.503+0000 7f2e617d7740 1 -- 172.21.15.7:0/1723172831 wait complete. 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e617d7740 1 -- start start 2024-04-21T08:49:16.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e617d7740 1 Processor -- start 2024-04-21T08:49:16.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1ea0 0x5620796f3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e617d7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 0x5620796ee320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e617d7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2e581ae0c0 con 0x5620796eaa80 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e617d7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e581af530 con 0x5620796e1580 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e617d7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2e581ae790 con 0x5620796e1ea0 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e617d7740 10 client.0 _ll_register_callbacks cb 0x7f2e40009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e57fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 0x5620796ee320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e57fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 0x5620796ee320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34910/0 (socket says 172.21.15.7:34910) 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e57fff640 1 -- 172.21.15.7:0/3981056732 learned_addr learned my addr 172.21.15.7:0/3981056732 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.504+0000 7f2e57fff640 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1ea0 0x5620796f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e57fff640 1 -- 172.21.15.7:0/3981056732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1ea0 msgr2=0x5620796f3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e57fff640 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1ea0 0x5620796f3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e57fff640 1 -- 172.21.15.7:0/3981056732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 msgr2=0x5620796f2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e57fff640 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796f2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e57fff640 1 -- 172.21.15.7:0/3981056732 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620796eeb50 con 0x5620796eaa80 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e57fff640 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 0x5620796ee320 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009fd0 tx=0x7f2e40009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76fae2ecc6a6e38 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e37fff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e37fff640 1 -- 172.21.15.7:0/3981056732 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40011550 con 0x5620796eaa80 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e37fff640 1 -- 172.21.15.7:0/3981056732 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f2e400116f0 con 0x5620796eaa80 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e617d7740 1 -- 172.21.15.7:0/3981056732 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e50003d40 con 0x5620796eaa80 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e37fff640 1 -- 172.21.15.7:0/3981056732 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2e400118f0 con 0x5620796eaa80 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e617d7740 10 client.16662 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e617d7740 1 -- 172.21.15.7:0/3981056732 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562079760c80 con 0x5620796eaa80 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.505+0000 7f2e37fff640 1 -- 172.21.15.7:0/3981056732 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e50003d40 con 0x5620796eaa80 2024-04-21T08:49:16.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.506+0000 7f2e37fff640 1 -- 172.21.15.7:0/3981056732 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f2e40022840 con 0x5620796eaa80 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.506+0000 7f2e617d7740 10 client.16662 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.506+0000 7f2e617d7740 10 client.16662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.506+0000 7f2e617d7740 20 client.16662 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.506+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5620796fab70 0x562079763440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.506+0000 7f2e617d7740 4 client.16662 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.506+0000 7f2e617d7740 1 -- 172.21.15.7:0/3981056732 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56207964da70 con 0x5620796fab70 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.506+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5620796fab70 0x562079763440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.506+0000 7f2e5c8ad640 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5620796fab70 0x562079763440 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f86396ffc31276ec server_cookie=3be11134faddd90a in_seq=0 out_seq=0 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.506+0000 7f2e37fff640 10 client.16662 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.507+0000 7f2e37fff640 1 -- 172.21.15.7:0/3981056732 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x56207964da70 con 0x5620796fab70 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.507+0000 7f2e37fff640 10 client.16662 handle_command_reply: tid=0 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.507+0000 7f2e617d7740 1 client.16662 shutdown 2024-04-21T08:49:16.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e617d7740 1 -- 172.21.15.7:0/3981056732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 msgr2=0x5620796ee320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 0x5620796ee320 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009fd0 tx=0x7f2e40009ac0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e617d7740 1 -- 172.21.15.7:0/3981056732 shutdown_connections 2024-04-21T08:49:16.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5620796fab70 0x562079763440 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5620796eaa80 0x5620796ee320 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5620796e1ea0 0x5620796f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e37fff640 0 client.16662 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:16.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e617d7740 1 --2- 172.21.15.7:0/3981056732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5620796e1580 0x5620796f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e617d7740 1 -- 172.21.15.7:0/3981056732 >> 172.21.15.7:0/3981056732 conn(0x5620796dbcb0 msgr2=0x5620796dad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e617d7740 1 -- 172.21.15.7:0/3981056732 shutdown_connections 2024-04-21T08:49:16.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e617d7740 1 -- 172.21.15.7:0/3981056732 wait complete. 2024-04-21T08:49:16.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e617d7740 20 client.16662 trim_cache size 0 max 16384 2024-04-21T08:49:16.514 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:49:16.514 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (3 inodes in the stack)", 2024-04-21T08:49:16.514 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:49:16.514 INFO:teuthology.orchestra.run.smithi007.stdout: "2f4d0e81-05a6-4410-bd5d-10ca0b65358b": { 2024-04-21T08:49:16.514 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:49:16.514 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "2f4d0e81-05a6-4410-bd5d-10ca0b65358b", 2024-04-21T08:49:16.514 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:49:16.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:49:16.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:49:16.514 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:49:16.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2191263330 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f2e440a3d20 msgr2=0x7f2e440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e5eb11640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f2e440a3d20 0x7f2e440a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2e480f00e0 tx=0x7f2e4803abc0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2191263330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e5819cf80 msgr2=0x7f2e5819d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e5eb11640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e5819cf80 0x7f2e5819d430 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009230 tx=0x7f2e4802c5d0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2191263330 shutdown_connections 2024-04-21T08:49:16.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e5eb11640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e5819cf80 0x7f2e5819d430 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e5eb11640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f2e440a3d20 0x7f2e440a61e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e5eb11640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2e58106120 0x7f2e58198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e5eb11640 1 --2- 172.21.15.7:0/2191263330 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e58101050 0x7f2e581986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.508+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2191263330 >> 172.21.15.7:0/2191263330 conn(0x7f2e58077fa0 msgr2=0x7f2e580fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.509+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2191263330 shutdown_connections 2024-04-21T08:49:16.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:16.509+0000 7f2e5eb11640 1 -- 172.21.15.7:0/2191263330 wait complete. 2024-04-21T08:49:16.519 INFO:tasks.cephfs.filesystem:scrub status for tag:2f4d0e81-05a6-4410-bd5d-10ca0b65358b - {'path': '/', 'tag': '2f4d0e81-05a6-4410-bd5d-10ca0b65358b', 'options': 'recursive,force'} 2024-04-21T08:49:16.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:16 smithi007 ceph-mon[28477]: pgmap v141: 129 pgs: 129 active+clean; 957 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 1.01k op/s 2024-04-21T08:49:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:16 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:49:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:16 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:16 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:49:17.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:16 smithi016 ceph-mon[31259]: pgmap v141: 129 pgs: 129 active+clean; 957 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 1.01k op/s 2024-04-21T08:49:17.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:16 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:49:17.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:16 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:17.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:16 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:49:17.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:16 smithi077 ceph-mon[31615]: pgmap v141: 129 pgs: 129 active+clean; 957 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 1.01k op/s 2024-04-21T08:49:17.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:16 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:49:17.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:16 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:17.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:16 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:49:19.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:18 smithi016 ceph-mon[31259]: pgmap v142: 129 pgs: 129 active+clean; 957 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 1.01k op/s 2024-04-21T08:49:19.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:18 smithi077 ceph-mon[31615]: pgmap v142: 129 pgs: 129 active+clean; 957 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 1.01k op/s 2024-04-21T08:49:19.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:18 smithi007 ceph-mon[28477]: pgmap v142: 129 pgs: 129 active+clean; 957 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 7.1 MiB/s wr, 1.01k op/s 2024-04-21T08:49:21.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:20 smithi016 ceph-mon[31259]: pgmap v143: 129 pgs: 129 active+clean; 927 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.9 MiB/s wr, 1.27k op/s 2024-04-21T08:49:21.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:20 smithi077 ceph-mon[31615]: pgmap v143: 129 pgs: 129 active+clean; 927 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.9 MiB/s wr, 1.27k op/s 2024-04-21T08:49:21.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:20 smithi007 ceph-mon[28477]: pgmap v143: 129 pgs: 129 active+clean; 927 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.9 MiB/s wr, 1.27k op/s 2024-04-21T08:49:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:22 smithi016 ceph-mon[31259]: pgmap v144: 129 pgs: 129 active+clean; 927 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 1.02k op/s 2024-04-21T08:49:23.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:22 smithi077 ceph-mon[31615]: pgmap v144: 129 pgs: 129 active+clean; 927 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 1.02k op/s 2024-04-21T08:49:23.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:22 smithi007 ceph-mon[28477]: pgmap v144: 129 pgs: 129 active+clean; 927 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 1.02k op/s 2024-04-21T08:49:25.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:24 smithi016 ceph-mon[31259]: pgmap v145: 129 pgs: 129 active+clean; 944 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 1.29k op/s 2024-04-21T08:49:25.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:24 smithi077 ceph-mon[31615]: pgmap v145: 129 pgs: 129 active+clean; 944 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 1.29k op/s 2024-04-21T08:49:25.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:24 smithi007 ceph-mon[28477]: pgmap v145: 129 pgs: 129 active+clean; 944 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.2 MiB/s wr, 1.29k op/s 2024-04-21T08:49:27.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:26 smithi016 ceph-mon[31259]: pgmap v146: 129 pgs: 129 active+clean; 951 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 611 op/s 2024-04-21T08:49:27.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:26 smithi077 ceph-mon[31615]: pgmap v146: 129 pgs: 129 active+clean; 951 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 611 op/s 2024-04-21T08:49:27.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:26 smithi007 ceph-mon[28477]: pgmap v146: 129 pgs: 129 active+clean; 951 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 611 op/s 2024-04-21T08:49:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.731+0000 7fb100bef640 1 -- 172.21.15.7:0/1994263022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc10c5b0 msgr2=0x7fb0fc10e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.731+0000 7fb100bef640 1 --2- 172.21.15.7:0/1994263022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc10c5b0 0x7fb0fc10e9e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009ca0 tx=0x7fb0f402c180 comp rx=0 tx=0).stop 2024-04-21T08:49:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.731+0000 7fb100bef640 1 -- 172.21.15.7:0/1994263022 shutdown_connections 2024-04-21T08:49:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.731+0000 7fb100bef640 1 --2- 172.21.15.7:0/1994263022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc10c5b0 0x7fb0fc10e9e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.731+0000 7fb100bef640 1 --2- 172.21.15.7:0/1994263022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0fc103b00 0x7fb0fc10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.731+0000 7fb100bef640 1 --2- 172.21.15.7:0/1994263022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0fc1031e0 0x7fb0fc1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.731+0000 7fb100bef640 1 -- 172.21.15.7:0/1994263022 >> 172.21.15.7:0/1994263022 conn(0x7fb0fc077fa0 msgr2=0x7fb0fc0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.731+0000 7fb100bef640 1 -- 172.21.15.7:0/1994263022 shutdown_connections 2024-04-21T08:49:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.731+0000 7fb100bef640 1 -- 172.21.15.7:0/1994263022 wait complete. 2024-04-21T08:49:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.731+0000 7fb100bef640 1 Processor -- start 2024-04-21T08:49:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb100bef640 1 -- start start 2024-04-21T08:49:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb100bef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc1031e0 0x7fb0fc100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb100bef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0fc103b00 0x7fb0fc101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb100bef640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0fc10c5b0 0x7fb0fc1018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb100bef640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1121d0 con 0x7fb0fc10c5b0 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb100bef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc112050 con 0x7fb0fc1031e0 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb100bef640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc112350 con 0x7fb0fc103b00 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0fc10c5b0 0x7fb0fc1018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc1031e0 0x7fb0fc100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0fc10c5b0 0x7fb0fc1018b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48590/0 (socket says 172.21.15.7:48590) 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc1031e0 0x7fb0fc100e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:58790/0 (socket says 172.21.15.7:58790) 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fad9d640 1 -- 172.21.15.7:0/1602707218 learned_addr learned my addr 172.21.15.7:0/1602707218 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0f3fff640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0fc103b00 0x7fb0fc101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fa59c640 1 -- 172.21.15.7:0/1602707218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0fc103b00 msgr2=0x7fb0fc101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0fc103b00 0x7fb0fc101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fa59c640 1 -- 172.21.15.7:0/1602707218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0fc10c5b0 msgr2=0x7fb0fc1018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0fc10c5b0 0x7fb0fc1018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fa59c640 1 -- 172.21.15.7:0/1602707218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0fc1a6c70 con 0x7fb0fc1031e0 2024-04-21T08:49:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fad9d640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0fc10c5b0 0x7fb0fc1018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.732+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc1031e0 0x7fb0fc100e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00e9f0 tx=0x7fb0ec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.733+0000 7fb0f37fe640 1 -- 172.21.15.7:0/1602707218 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec00ccc0 con 0x7fb0fc1031e0 2024-04-21T08:49:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.733+0000 7fb0f37fe640 1 -- 172.21.15.7:0/1602707218 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0ec005020 con 0x7fb0fc1031e0 2024-04-21T08:49:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.733+0000 7fb100bef640 1 -- 172.21.15.7:0/1602707218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0fc1a6f60 con 0x7fb0fc1031e0 2024-04-21T08:49:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.733+0000 7fb0f37fe640 1 -- 172.21.15.7:0/1602707218 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec004a00 con 0x7fb0fc1031e0 2024-04-21T08:49:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.733+0000 7fb100bef640 1 -- 172.21.15.7:0/1602707218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0fc1a7420 con 0x7fb0fc1031e0 2024-04-21T08:49:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.735+0000 7fb0f37fe640 1 -- 172.21.15.7:0/1602707218 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fb0ec004ba0 con 0x7fb0fc1031e0 2024-04-21T08:49:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.735+0000 7fb1037db740 1 Processor -- start 2024-04-21T08:49:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.735+0000 7fb0f37fe640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e80a3d90 0x7fb0e80a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.735+0000 7fb0fad9d640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e80a3d90 0x7fb0e80a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.735+0000 7fb1037db740 1 -- start start 2024-04-21T08:49:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.735+0000 7fb0f37fe640 1 -- 172.21.15.7:0/1602707218 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb0ec0d0ac0 con 0x7fb0fc1031e0 2024-04-21T08:49:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb1037db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 0x558b39bb1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb1037db740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb1b60 0x558b39bb1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb1037db740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 0x558b39bbd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb1037db740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558b39b168d0 con 0x558b39bbb0e0 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb1037db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558b39ab3e80 con 0x558b39bb28d0 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb1037db740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558b39b12890 con 0x558b39bb1b60 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 0x558b39bb1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 0x558b39bb1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:58800/0 (socket says 172.21.15.7:58800) 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fa59c640 1 -- 172.21.15.7:0/3793634044 learned_addr learned my addr 172.21.15.7:0/3793634044 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fad9d640 1 --2- 172.21.15.7:0/3793634044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 0x558b39bbd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0f3fff640 1 --2- 172.21.15.7:0/3793634044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb1b60 0x558b39bb1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fad9d640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e80a3d90 0x7fb0e80a6250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009a60 tx=0x7fb0f4032380 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fa59c640 1 -- 172.21.15.7:0/3793634044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb1b60 msgr2=0x558b39bb1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/3793634044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb1b60 0x558b39bb1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fa59c640 1 -- 172.21.15.7:0/3793634044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 msgr2=0x558b39bbd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/3793634044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 0x558b39bbd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fa59c640 1 -- 172.21.15.7:0/3793634044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b39ad5320 con 0x558b39bb28d0 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fad9d640 1 --2- 172.21.15.7:0/3793634044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 0x558b39bbd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.736+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/3793634044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 0x558b39bb1620 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec098ad0 tx=0x7fb0ec09cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f539435ab008d6fb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb0f0ff9640 1 -- 172.21.15.7:0/3793634044 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec01c040 con 0x558b39bb28d0 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb0f0ff9640 1 -- 172.21.15.7:0/3793634044 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0ec0a0a20 con 0x558b39bb28d0 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb0f0ff9640 1 -- 172.21.15.7:0/3793634044 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec0ce0a0 con 0x558b39bb28d0 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb1037db740 1 -- 172.21.15.7:0/3793634044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 msgr2=0x558b39bb1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb1037db740 1 --2- 172.21.15.7:0/3793634044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 0x558b39bb1620 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec098ad0 tx=0x7fb0ec09cd70 comp rx=0 tx=0).stop 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb1037db740 1 -- 172.21.15.7:0/3793634044 shutdown_connections 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb1037db740 1 --2- 172.21.15.7:0/3793634044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 0x558b39bbd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb1037db740 1 --2- 172.21.15.7:0/3793634044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb1b60 0x558b39bb1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb1037db740 1 --2- 172.21.15.7:0/3793634044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 0x558b39bb1620 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb1037db740 1 -- 172.21.15.7:0/3793634044 >> 172.21.15.7:0/3793634044 conn(0x558b39b78c90 msgr2=0x558b39bac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb1037db740 1 -- 172.21.15.7:0/3793634044 shutdown_connections 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.737+0000 7fb1037db740 1 -- 172.21.15.7:0/3793634044 wait complete. 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb1037db740 1 -- start start 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb1037db740 1 Processor -- start 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb1037db740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb1b60 0x558b39bbaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb1037db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 0x558b39bb7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb1037db740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 0x558b39bb8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb1037db740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558b39bbde10 con 0x558b39bbb0e0 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb1037db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558b39a9eeb0 con 0x558b39bb28d0 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb1037db740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558b39bbdf90 con 0x558b39bb1b60 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb1037db740 10 client.0 _ll_register_callbacks cb 0x558b39acbb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb0fad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 0x558b39bb8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb0fad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 0x558b39bb8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48610/0 (socket says 172.21.15.7:48610) 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb0fad9d640 1 -- 172.21.15.7:0/132766638 learned_addr learned my addr 172.21.15.7:0/132766638 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb0f3fff640 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 0x558b39bb7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.738+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb1b60 0x558b39bbaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb0fad9d640 1 -- 172.21.15.7:0/132766638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb1b60 msgr2=0x558b39bbaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb0fad9d640 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb1b60 0x558b39bbaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb0fad9d640 1 -- 172.21.15.7:0/132766638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 msgr2=0x558b39bb7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb0fad9d640 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 0x558b39bb7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb0fad9d640 1 -- 172.21.15.7:0/132766638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b39829210 con 0x558b39bbb0e0 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb0fad9d640 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 0x558b39bb8380 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4037040 tx=0x7fb0f402c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659760895429b30d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb0e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/132766638 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4039040 con 0x558b39bbb0e0 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/132766638 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb0f404b410 con 0x558b39bbb0e0 2024-04-21T08:49:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb1037db740 1 -- 172.21.15.7:0/132766638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0ec0a0a20 con 0x558b39bbb0e0 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/132766638 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb0f405a050 con 0x558b39bbb0e0 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb1037db740 10 client.16674 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb1037db740 1 -- 172.21.15.7:0/132766638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558b398f5620 con 0x558b39bbb0e0 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.739+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/132766638 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0ec0a0a20 con 0x558b39bbb0e0 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.740+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/132766638 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb0f403f040 con 0x558b39bbb0e0 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.740+0000 7fb1037db740 10 client.16674 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.740+0000 7fb1037db740 10 client.16674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.740+0000 7fb1037db740 20 client.16674 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.740+0000 7fb1037db740 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558b39bcb1e0 0x558b39bcd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.740+0000 7fb1037db740 4 client.16674 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.740+0000 7fb1037db740 1 -- 172.21.15.7:0/132766638 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b39b2a880 con 0x558b39bcb1e0 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.740+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558b39bcb1e0 0x558b39bcd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.740+0000 7fb0e1ffb640 10 client.16674 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.740+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558b39bcb1e0 0x558b39bcd5d0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=301178f3d5f28723 server_cookie=f4309849f3fdb001 in_seq=0 out_seq=0 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/132766638 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b39b2a880 con 0x558b39bcb1e0 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb0e1ffb640 10 client.16674 handle_command_reply: tid=0 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 1 client.16674 shutdown 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 1 -- 172.21.15.7:0/132766638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 msgr2=0x558b39bb8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 0x558b39bb8380 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4037040 tx=0x7fb0f402c180 comp rx=0 tx=0).stop 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 1 -- 172.21.15.7:0/132766638 shutdown_connections 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558b39bcb1e0 0x558b39bcd5d0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bbb0e0 0x558b39bb8380 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bb28d0 0x558b39bb7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 1 --2- 172.21.15.7:0/132766638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb1b60 0x558b39bbaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb0e1ffb640 0 client.16674 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 1 -- 172.21.15.7:0/132766638 >> 172.21.15.7:0/132766638 conn(0x558b39b78c90 msgr2=0x558b39bae5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 1 -- 172.21.15.7:0/132766638 shutdown_connections 2024-04-21T08:49:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 1 -- 172.21.15.7:0/132766638 wait complete. 2024-04-21T08:49:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.742+0000 7fb1037db740 20 client.16674 trim_cache size 0 max 16384 2024-04-21T08:49:27.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.757+0000 7fb1037db740 1 Processor -- start 2024-04-21T08:49:27.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.757+0000 7fb1037db740 1 -- start start 2024-04-21T08:49:27.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb1037db740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.761 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb1037db740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb28d0 0x558b39bba530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb1037db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bbb0e0 0x558b39bbaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb1037db740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1af610 con 0x558b39bb1b60 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb1037db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1b1e90 con 0x558b39bbb0e0 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb1037db740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1afc80 con 0x558b39bb28d0 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0fa59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0fa59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48616/0 (socket says 172.21.15.7:48616) 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0fa59c640 1 -- 172.21.15.7:0/1608700212 learned_addr learned my addr 172.21.15.7:0/1608700212 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0fad9d640 1 --2- 172.21.15.7:0/1608700212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bbb0e0 0x558b39bbaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0f3fff640 1 --2- 172.21.15.7:0/1608700212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb28d0 0x558b39bba530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0fa59c640 1 -- 172.21.15.7:0/1608700212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb28d0 msgr2=0x558b39bba530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/1608700212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb28d0 0x558b39bba530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0fa59c640 1 -- 172.21.15.7:0/1608700212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bbb0e0 msgr2=0x558b39bbaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/1608700212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bbb0e0 0x558b39bbaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0fa59c640 1 -- 172.21.15.7:0/1608700212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b39bb1990 con 0x558b39bb1b60 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/1608700212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc6d10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00e9c0 tx=0x7fb0ec005460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06160585075e532 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0e3fff640 1 -- 172.21.15.7:0/1608700212 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec01c070 con 0x558b39bb1b60 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0e3fff640 1 -- 172.21.15.7:0/1608700212 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0ec0051d0 con 0x558b39bb1b60 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.758+0000 7fb0e3fff640 1 -- 172.21.15.7:0/1608700212 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec098ce0 con 0x558b39bb1b60 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.759+0000 7fb1037db740 1 -- 172.21.15.7:0/1608700212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 msgr2=0x558b39bc6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.759+0000 7fb1037db740 1 --2- 172.21.15.7:0/1608700212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc6d10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00e9c0 tx=0x7fb0ec005460 comp rx=0 tx=0).stop 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.759+0000 7fb1037db740 1 -- 172.21.15.7:0/1608700212 shutdown_connections 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.759+0000 7fb1037db740 1 --2- 172.21.15.7:0/1608700212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bbb0e0 0x558b39bbaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.759+0000 7fb1037db740 1 --2- 172.21.15.7:0/1608700212 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb28d0 0x558b39bba530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.759+0000 7fb1037db740 1 --2- 172.21.15.7:0/1608700212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc6d10 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.759+0000 7fb1037db740 1 -- 172.21.15.7:0/1608700212 >> 172.21.15.7:0/1608700212 conn(0x558b39b78c90 msgr2=0x558b39bac380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.759+0000 7fb1037db740 1 -- 172.21.15.7:0/1608700212 shutdown_connections 2024-04-21T08:49:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.759+0000 7fb1037db740 1 -- 172.21.15.7:0/1608700212 wait complete. 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.759+0000 7fb1037db740 1 -- start start 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb1037db740 1 Processor -- start 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb1037db740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb1037db740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb28d0 0x558b39bb4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb1037db740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bbb0e0 0x558b39bb4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb1037db740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1b0350 con 0x558b39bb1b60 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb1037db740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1b17c0 con 0x558b39bbb0e0 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb1037db740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1b0a20 con 0x558b39bb28d0 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb1037db740 10 client.0 _ll_register_callbacks cb 0x7fb0e4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0fa59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0fa59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48632/0 (socket says 172.21.15.7:48632) 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0fa59c640 1 -- 172.21.15.7:0/1533820879 learned_addr learned my addr 172.21.15.7:0/1533820879 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0fad9d640 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bbb0e0 0x558b39bb4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0f3fff640 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb28d0 0x558b39bb4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0fa59c640 1 -- 172.21.15.7:0/1533820879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb28d0 msgr2=0x558b39bb4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb28d0 0x558b39bb4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0fa59c640 1 -- 172.21.15.7:0/1533820879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bbb0e0 msgr2=0x558b39bb4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bbb0e0 0x558b39bb4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0fa59c640 1 -- 172.21.15.7:0/1533820879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b39bb5190 con 0x558b39bb1b60 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0fa59c640 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc3410 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0ce420 tx=0x7fb0ec0a0ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eebcc5ed9bf7ae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.760+0000 7fb0e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/1533820879 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec01c040 con 0x558b39bb1b60 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/1533820879 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fb0ec09a970 con 0x558b39bb1b60 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb1037db740 1 -- 172.21.15.7:0/1533820879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0ec0051d0 con 0x558b39bb1b60 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/1533820879 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb0ec0ce110 con 0x558b39bb1b60 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb1037db740 10 client.16680 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb1037db740 1 -- 172.21.15.7:0/1533820879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558b39c31530 con 0x558b39bb1b60 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/1533820879 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0ec0051d0 con 0x558b39bb1b60 2024-04-21T08:49:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/1533820879 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb0ec0a4f50 con 0x558b39bb1b60 2024-04-21T08:49:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb1037db740 10 client.16680 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb1037db740 10 client.16680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb1037db740 20 client.16680 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb1037db740 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558b39c31750 0x558b39c33b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb1037db740 4 client.16680 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:49:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb1037db740 1 -- 172.21.15.7:0/1533820879 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558b39ab63d0 con 0x558b39c31750 2024-04-21T08:49:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.761+0000 7fb0f3fff640 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558b39c31750 0x558b39c33b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.762+0000 7fb0f3fff640 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558b39c31750 0x558b39c33b40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=270417045ef65e79 server_cookie=4cddded7412bc497 in_seq=0 out_seq=0 2024-04-21T08:49:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:27.762+0000 7fb0e1ffb640 10 client.16680 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:29.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:28 smithi016 ceph-mon[31259]: pgmap v147: 129 pgs: 129 active+clean; 951 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 611 op/s 2024-04-21T08:49:29.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:28 smithi077 ceph-mon[31615]: pgmap v147: 129 pgs: 129 active+clean; 951 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 611 op/s 2024-04-21T08:49:29.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:28 smithi007 ceph-mon[28477]: pgmap v147: 129 pgs: 129 active+clean; 951 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 611 op/s 2024-04-21T08:49:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:30 smithi016 ceph-mon[31259]: pgmap v148: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 13 MiB/s wr, 1.06k op/s 2024-04-21T08:49:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:30 smithi077 ceph-mon[31615]: pgmap v148: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 13 MiB/s wr, 1.06k op/s 2024-04-21T08:49:31.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:30 smithi007 ceph-mon[28477]: pgmap v148: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 13 MiB/s wr, 1.06k op/s 2024-04-21T08:49:33.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:32 smithi016 ceph-mon[31259]: pgmap v149: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 13 MiB/s wr, 804 op/s 2024-04-21T08:49:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:32 smithi077 ceph-mon[31615]: pgmap v149: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 13 MiB/s wr, 804 op/s 2024-04-21T08:49:33.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:32 smithi007 ceph-mon[28477]: pgmap v149: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 13 MiB/s wr, 804 op/s 2024-04-21T08:49:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.152+0000 7fb0e1ffb640 1 -- 172.21.15.7:0/1533820879 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558b39ab63d0 con 0x558b39c31750 2024-04-21T08:49:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.152+0000 7fb0e1ffb640 10 client.16680 handle_command_reply: tid=0 2024-04-21T08:49:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.152+0000 7fb1037db740 1 client.16680 shutdown 2024-04-21T08:49:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb1037db740 1 -- 172.21.15.7:0/1533820879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 msgr2=0x558b39bc3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb1037db740 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc3410 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0ce420 tx=0x7fb0ec0a0ae0 comp rx=0 tx=0).stop 2024-04-21T08:49:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb1037db740 1 -- 172.21.15.7:0/1533820879 shutdown_connections 2024-04-21T08:49:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb1037db740 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558b39c31750 0x558b39c33b40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb1037db740 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b39bbb0e0 0x558b39bb4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb1037db740 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558b39bb28d0 0x558b39bb4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb1037db740 1 --2- 172.21.15.7:0/1533820879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558b39bb1b60 0x558b39bc3410 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb1037db740 1 -- 172.21.15.7:0/1533820879 >> 172.21.15.7:0/1533820879 conn(0x558b39b78c90 msgr2=0x558b39bb6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb0e1ffb640 0 client.16680 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb1037db740 1 -- 172.21.15.7:0/1533820879 shutdown_connections 2024-04-21T08:49:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb1037db740 1 -- 172.21.15.7:0/1533820879 wait complete. 2024-04-21T08:49:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.153+0000 7fb1037db740 20 client.16680 trim_cache size 0 max 16384 2024-04-21T08:49:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:49:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.154+0000 7fb100bef640 1 -- 172.21.15.7:0/1602707218 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e80a3d90 msgr2=0x7fb0e80a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.154+0000 7fb100bef640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e80a3d90 0x7fb0e80a6250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009a60 tx=0x7fb0f4032380 comp rx=0 tx=0).stop 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.154+0000 7fb100bef640 1 -- 172.21.15.7:0/1602707218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc1031e0 msgr2=0x7fb0fc100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.154+0000 7fb100bef640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc1031e0 0x7fb0fc100e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00e9f0 tx=0x7fb0ec00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.154+0000 7fb100bef640 1 -- 172.21.15.7:0/1602707218 shutdown_connections 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.154+0000 7fb100bef640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0fc10c5b0 0x7fb0fc1018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.154+0000 7fb100bef640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb0e80a3d90 0x7fb0e80a6250 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.154+0000 7fb100bef640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0fc103b00 0x7fb0fc101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.154+0000 7fb100bef640 1 --2- 172.21.15.7:0/1602707218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0fc1031e0 0x7fb0fc100e30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.154+0000 7fb100bef640 1 -- 172.21.15.7:0/1602707218 >> 172.21.15.7:0/1602707218 conn(0x7fb0fc077fa0 msgr2=0x7fb0fc0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.155+0000 7fb100bef640 1 -- 172.21.15.7:0/1602707218 shutdown_connections 2024-04-21T08:49:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:34.155+0000 7fb100bef640 1 -- 172.21.15.7:0/1602707218 wait complete. 2024-04-21T08:49:35.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:34 smithi016 ceph-mon[31259]: pgmap v150: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 13 MiB/s wr, 2.04k op/s 2024-04-21T08:49:35.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:34 smithi077 ceph-mon[31615]: pgmap v150: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 13 MiB/s wr, 2.04k op/s 2024-04-21T08:49:35.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:34 smithi007 ceph-mon[28477]: pgmap v150: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 13 MiB/s wr, 2.04k op/s 2024-04-21T08:49:37.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:36 smithi016 ceph-mon[31259]: pgmap v151: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 12 MiB/s wr, 2.08k op/s 2024-04-21T08:49:37.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:36 smithi077 ceph-mon[31615]: pgmap v151: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 12 MiB/s wr, 2.08k op/s 2024-04-21T08:49:37.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:36 smithi007 ceph-mon[28477]: pgmap v151: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 12 MiB/s wr, 2.08k op/s 2024-04-21T08:49:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:38 smithi016 ceph-mon[31259]: pgmap v152: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 11 MiB/s wr, 2.00k op/s 2024-04-21T08:49:39.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:38 smithi077 ceph-mon[31615]: pgmap v152: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 11 MiB/s wr, 2.00k op/s 2024-04-21T08:49:39.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:38 smithi007 ceph-mon[28477]: pgmap v152: 129 pgs: 129 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 11 MiB/s wr, 2.00k op/s 2024-04-21T08:49:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:40 smithi016 ceph-mon[31259]: pgmap v153: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 13 MiB/s wr, 2.69k op/s 2024-04-21T08:49:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:40 smithi077 ceph-mon[31615]: pgmap v153: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 13 MiB/s wr, 2.69k op/s 2024-04-21T08:49:41.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:40 smithi007 ceph-mon[28477]: pgmap v153: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 13 MiB/s wr, 2.69k op/s 2024-04-21T08:49:43.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:42 smithi016 ceph-mon[31259]: pgmap v154: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 2.24k op/s 2024-04-21T08:49:43.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:42 smithi077 ceph-mon[31615]: pgmap v154: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 2.24k op/s 2024-04-21T08:49:43.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:42 smithi007 ceph-mon[28477]: pgmap v154: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 2.24k op/s 2024-04-21T08:49:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:44 smithi016 ceph-mon[31259]: pgmap v155: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 2.63k op/s 2024-04-21T08:49:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:44 smithi077 ceph-mon[31615]: pgmap v155: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 2.63k op/s 2024-04-21T08:49:45.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:44 smithi007 ceph-mon[28477]: pgmap v155: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 2.63k op/s 2024-04-21T08:49:46.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:49:46.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:49:46.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.637+0000 7f87f3677640 1 -- 172.21.15.7:0/3766064873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87ec069570 msgr2=0x7f87ec069950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.637+0000 7f87f3677640 1 --2- 172.21.15.7:0/3766064873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87ec069570 0x7f87ec069950 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f87e8009900 tx=0x7f87e802bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:46.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.637+0000 7f87f3677640 1 -- 172.21.15.7:0/3766064873 shutdown_connections 2024-04-21T08:49:46.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.637+0000 7f87f3677640 1 --2- 172.21.15.7:0/3766064873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87ec107230 0x7f87ec10ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.637+0000 7f87f3677640 1 --2- 172.21.15.7:0/3766064873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec069f20 0x7f87ec106b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.637+0000 7f87f3677640 1 --2- 172.21.15.7:0/3766064873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87ec069570 0x7f87ec069950 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.637+0000 7f87f3677640 1 -- 172.21.15.7:0/3766064873 >> 172.21.15.7:0/3766064873 conn(0x7f87ec077fa0 msgr2=0x7f87ec0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.637+0000 7f87f3677640 1 -- 172.21.15.7:0/3766064873 shutdown_connections 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.637+0000 7f87f3677640 1 -- 172.21.15.7:0/3766064873 wait complete. 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f3677640 1 Processor -- start 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f3677640 1 -- start start 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f3677640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87ec069570 0x7f87ec100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f3677640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec107230 0x7f87ec101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f3677640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87ec101890 0x7f87ec1a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f3677640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1121a0 con 0x7f87ec101890 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f3677640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec112020 con 0x7f87ec107230 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f3677640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec112320 con 0x7f87ec069570 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f1c14640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87ec101890 0x7f87ec1a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f1413640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87ec069570 0x7f87ec100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f0c12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec107230 0x7f87ec101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f1413640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87ec069570 0x7f87ec100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:33860/0 (socket says 172.21.15.7:33860) 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f1c14640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87ec101890 0x7f87ec1a68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46690/0 (socket says 172.21.15.7:46690) 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f0c12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec107230 0x7f87ec101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:43174/0 (socket says 172.21.15.7:43174) 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.638+0000 7f87f1c14640 1 -- 172.21.15.7:0/369695699 learned_addr learned my addr 172.21.15.7:0/369695699 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.639+0000 7f87f1c14640 1 -- 172.21.15.7:0/369695699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87ec069570 msgr2=0x7f87ec100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.639+0000 7f87f1c14640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87ec069570 0x7f87ec100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.639+0000 7f87f1c14640 1 -- 172.21.15.7:0/369695699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec107230 msgr2=0x7f87ec101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.639+0000 7f87f1c14640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec107230 0x7f87ec101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.639+0000 7f87f1c14640 1 -- 172.21.15.7:0/369695699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ec1a6fb0 con 0x7f87ec101890 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.639+0000 7f87f1c14640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87ec101890 0x7f87ec1a68b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f87e400eb10 tx=0x7f87e400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.639+0000 7f87e27fc640 1 -- 172.21.15.7:0/369695699 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4004b20 con 0x7f87ec101890 2024-04-21T08:49:46.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.639+0000 7f87e27fc640 1 -- 172.21.15.7:0/369695699 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87e4004cc0 con 0x7f87ec101890 2024-04-21T08:49:46.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.639+0000 7f87f3677640 1 -- 172.21.15.7:0/369695699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87ec1a72a0 con 0x7f87ec101890 2024-04-21T08:49:46.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.639+0000 7f87e27fc640 1 -- 172.21.15.7:0/369695699 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4010670 con 0x7f87ec101890 2024-04-21T08:49:46.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.639+0000 7f87f3677640 1 -- 172.21.15.7:0/369695699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87ec1a7840 con 0x7f87ec101890 2024-04-21T08:49:46.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.641+0000 7f87e27fc640 1 -- 172.21.15.7:0/369695699 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f87e4005020 con 0x7f87ec101890 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.642+0000 7f87f625f740 1 Processor -- start 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.642+0000 7f87f625f740 1 -- start start 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.642+0000 7f87e27fc640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3dd0 0x7f87d80a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.642+0000 7f87f625f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1580 0x55a81ded1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.642+0000 7f87f625f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81ded1ea0 0x55a81deda540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.642+0000 7f87f625f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81dedaa80 0x55a81dedce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.642+0000 7f87f625f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a81de361e0 con 0x55a81ded1580 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.642+0000 7f87f625f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a81ddd3390 con 0x55a81ded1ea0 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.642+0000 7f87f625f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a81de321a0 con 0x55a81dedaa80 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.642+0000 7f87f0c12640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3dd0 0x7f87d80a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.642+0000 7f87f1413640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1580 0x55a81ded1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87e27fc640 1 -- 172.21.15.7:0/369695699 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f87e40d1000 con 0x7f87ec101890 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87f1413640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1580 0x55a81ded1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46702/0 (socket says 172.21.15.7:46702) 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87f1c14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81dedaa80 0x55a81dedce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87f0c12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81ded1ea0 0x55a81deda540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87f1413640 1 -- 172.21.15.7:0/1542674289 learned_addr learned my addr 172.21.15.7:0/1542674289 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87f1413640 1 -- 172.21.15.7:0/1542674289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81dedaa80 msgr2=0x55a81dedce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87f1413640 1 --2- 172.21.15.7:0/1542674289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81dedaa80 0x55a81dedce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87f1413640 1 -- 172.21.15.7:0/1542674289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81ded1ea0 msgr2=0x55a81deda540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87f1413640 1 --2- 172.21.15.7:0/1542674289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81ded1ea0 0x55a81deda540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87f1413640 1 -- 172.21.15.7:0/1542674289 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a81dc127d0 con 0x55a81ded1580 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87f0c12640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3dd0 0x7f87d80a6290 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f87ec1028a0 tx=0x7f87dc009b60 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87f1413640 1 --2- 172.21.15.7:0/1542674289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1580 0x55a81ded1960 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f87e8009900 tx=0x7f87e80095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe38c6838a02c510 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87d77fe640 1 -- 172.21.15.7:0/1542674289 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e8045020 con 0x55a81ded1580 2024-04-21T08:49:46.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87d77fe640 1 -- 172.21.15.7:0/1542674289 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87e802c7e0 con 0x55a81ded1580 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.643+0000 7f87d77fe640 1 -- 172.21.15.7:0/1542674289 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e8005440 con 0x55a81ded1580 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.644+0000 7f87f625f740 1 -- 172.21.15.7:0/1542674289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1580 msgr2=0x55a81ded1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.644+0000 7f87f625f740 1 --2- 172.21.15.7:0/1542674289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1580 0x55a81ded1960 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f87e8009900 tx=0x7f87e80095f0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.644+0000 7f87f625f740 1 -- 172.21.15.7:0/1542674289 shutdown_connections 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.644+0000 7f87f625f740 1 --2- 172.21.15.7:0/1542674289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81dedaa80 0x55a81dedce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.644+0000 7f87f625f740 1 --2- 172.21.15.7:0/1542674289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81ded1ea0 0x55a81deda540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.644+0000 7f87f625f740 1 --2- 172.21.15.7:0/1542674289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1580 0x55a81ded1960 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.644+0000 7f87f625f740 1 -- 172.21.15.7:0/1542674289 >> 172.21.15.7:0/1542674289 conn(0x55a81decbcb0 msgr2=0x55a81decc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.644+0000 7f87f625f740 1 -- 172.21.15.7:0/1542674289 shutdown_connections 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.644+0000 7f87f625f740 1 -- 172.21.15.7:0/1542674289 wait complete. 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.644+0000 7f87f625f740 1 -- start start 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.644+0000 7f87f625f740 1 Processor -- start 2024-04-21T08:49:46.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f625f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 0x55a81ded7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f625f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81ded7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f625f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81dedaa80 0x55a81ded8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f625f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a81dedd810 con 0x55a81ded1ea0 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f625f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a81ddef2f0 con 0x55a81dedaa80 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f625f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a81dedd990 con 0x55a81ded1580 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f625f740 10 client.0 _ll_register_callbacks cb 0x55a81ddfc580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f0c12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81ded7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f1413640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 0x55a81ded7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f0c12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81ded7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46718/0 (socket says 172.21.15.7:46718) 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f0c12640 1 -- 172.21.15.7:0/1295714965 learned_addr learned my addr 172.21.15.7:0/1295714965 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f1c14640 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81dedaa80 0x55a81ded8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f1413640 1 -- 172.21.15.7:0/1295714965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81dedaa80 msgr2=0x55a81ded8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f1413640 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81dedaa80 0x55a81ded8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f1413640 1 -- 172.21.15.7:0/1295714965 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 msgr2=0x55a81ded7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f1413640 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81ded7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f1413640 1 -- 172.21.15.7:0/1295714965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a81db483f0 con 0x55a81ded1580 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.645+0000 7f87f0c12640 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81ded7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87f1413640 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 0x55a81ded7a80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87e8004650 tx=0x7f87e802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1e45cb8c7783c24 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87d4ff9640 1 -- 172.21.15.7:0/1295714965 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e804e070 con 0x55a81ded1580 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87d4ff9640 1 -- 172.21.15.7:0/1295714965 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f87e80048c0 con 0x55a81ded1580 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87f625f740 1 -- 172.21.15.7:0/1295714965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87e802c7e0 con 0x55a81ded1580 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87d4ff9640 1 -- 172.21.15.7:0/1295714965 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f87e8004ac0 con 0x55a81ded1580 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87f625f740 10 client.26227 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87f625f740 1 -- 172.21.15.7:0/1295714965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a81db080f0 con 0x55a81ded1580 2024-04-21T08:49:46.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87d4ff9640 1 -- 172.21.15.7:0/1295714965 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87e802c7e0 con 0x55a81ded1580 2024-04-21T08:49:46.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87d4ff9640 1 -- 172.21.15.7:0/1295714965 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f87e8041c00 con 0x55a81ded1580 2024-04-21T08:49:46.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87f625f740 10 client.26227 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:46.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87f625f740 10 client.26227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:46.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.646+0000 7f87f625f740 20 client.26227 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:46.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.647+0000 7f87f625f740 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a81deea860 0x55a81deecc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.647+0000 7f87f625f740 4 client.26227 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:46.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.647+0000 7f87f625f740 1 -- 172.21.15.7:0/1295714965 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a81de43c00 con 0x55a81deea860 2024-04-21T08:49:46.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.647+0000 7f87f0c12640 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a81deea860 0x55a81deecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.647+0000 7f87f0c12640 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a81deea860 0x55a81deecc50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1be01827bb5f0ea server_cookie=20ff66143b1062ed in_seq=0 out_seq=0 2024-04-21T08:49:46.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.647+0000 7f87d4ff9640 10 client.26227 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:46.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.648+0000 7f87d4ff9640 1 -- 172.21.15.7:0/1295714965 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a81de43c00 con 0x55a81deea860 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.648+0000 7f87d4ff9640 10 client.26227 handle_command_reply: tid=0 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 1 client.26227 shutdown 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 1 -- 172.21.15.7:0/1295714965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 msgr2=0x55a81ded7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 0x55a81ded7a80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87e8004650 tx=0x7f87e802bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 1 -- 172.21.15.7:0/1295714965 shutdown_connections 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a81deea860 0x55a81deecc50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81dedaa80 0x55a81ded8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87d4ff9640 0 client.26227 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81ded7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 1 --2- 172.21.15.7:0/1295714965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 0x55a81ded7a80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 1 -- 172.21.15.7:0/1295714965 >> 172.21.15.7:0/1295714965 conn(0x55a81decbcb0 msgr2=0x55a81decda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 1 -- 172.21.15.7:0/1295714965 shutdown_connections 2024-04-21T08:49:46.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 1 -- 172.21.15.7:0/1295714965 wait complete. 2024-04-21T08:49:46.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.649+0000 7f87f625f740 20 client.26227 trim_cache size 0 max 16384 2024-04-21T08:49:46.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f625f740 1 Processor -- start 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f625f740 1 -- start start 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f625f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81ded1580 0x55a81deda1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f625f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f625f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81dedaa80 0x55a81def2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f625f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1af5c0 con 0x55a81ded1ea0 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f625f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1b1e40 con 0x55a81ded1580 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f625f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1afc30 con 0x55a81dedaa80 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f0c12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f1413640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81ded1580 0x55a81deda1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f1c14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81dedaa80 0x55a81def2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f0c12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46730/0 (socket says 172.21.15.7:46730) 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f0c12640 1 -- 172.21.15.7:0/3421299430 learned_addr learned my addr 172.21.15.7:0/3421299430 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f0c12640 1 -- 172.21.15.7:0/3421299430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81dedaa80 msgr2=0x55a81def2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f0c12640 1 --2- 172.21.15.7:0/3421299430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81dedaa80 0x55a81def2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f0c12640 1 -- 172.21.15.7:0/3421299430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81ded1580 msgr2=0x55a81deda1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.670+0000 7f87f0c12640 1 --2- 172.21.15.7:0/3421299430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81ded1580 0x55a81deda1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.671+0000 7f87f0c12640 1 -- 172.21.15.7:0/3421299430 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a81dedd930 con 0x55a81ded1ea0 2024-04-21T08:49:46.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.671+0000 7f87f0c12640 1 --2- 172.21.15.7:0/3421299430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee66d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009d10 tx=0x7f87dc021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf937565d5b5f32f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.671+0000 7f87d6ffd640 1 -- 172.21.15.7:0/3421299430 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0097a0 con 0x55a81ded1ea0 2024-04-21T08:49:46.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.671+0000 7f87d6ffd640 1 -- 172.21.15.7:0/3421299430 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87dc02c900 con 0x55a81ded1ea0 2024-04-21T08:49:46.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.671+0000 7f87d6ffd640 1 -- 172.21.15.7:0/3421299430 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0047a0 con 0x55a81ded1ea0 2024-04-21T08:49:46.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.672+0000 7f87f625f740 1 -- 172.21.15.7:0/3421299430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 msgr2=0x55a81dee66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.672+0000 7f87f625f740 1 --2- 172.21.15.7:0/3421299430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee66d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009d10 tx=0x7f87dc021000 comp rx=0 tx=0).stop 2024-04-21T08:49:46.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.672+0000 7f87f625f740 1 -- 172.21.15.7:0/3421299430 shutdown_connections 2024-04-21T08:49:46.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.672+0000 7f87f625f740 1 --2- 172.21.15.7:0/3421299430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81dedaa80 0x55a81def2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.672+0000 7f87f625f740 1 --2- 172.21.15.7:0/3421299430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee66d0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.672+0000 7f87f625f740 1 --2- 172.21.15.7:0/3421299430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81ded1580 0x55a81deda1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.672+0000 7f87f625f740 1 -- 172.21.15.7:0/3421299430 >> 172.21.15.7:0/3421299430 conn(0x55a81decbcb0 msgr2=0x55a81decc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.672+0000 7f87f625f740 1 -- 172.21.15.7:0/3421299430 shutdown_connections 2024-04-21T08:49:46.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.672+0000 7f87f625f740 1 -- 172.21.15.7:0/3421299430 wait complete. 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.672+0000 7f87f625f740 1 -- start start 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.672+0000 7f87f625f740 1 Processor -- start 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f625f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 0x55a81dee2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f625f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f625f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81dedaa80 0x55a81dede080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f625f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1b0300 con 0x55a81ded1ea0 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f625f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1b1770 con 0x55a81dedaa80 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f625f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1b09d0 con 0x55a81ded1580 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f625f740 10 client.0 _ll_register_callbacks cb 0x7f87dc0114c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f0c12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f0c12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46746/0 (socket says 172.21.15.7:46746) 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f0c12640 1 -- 172.21.15.7:0/4002947241 learned_addr learned my addr 172.21.15.7:0/4002947241 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f1c14640 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81dedaa80 0x55a81dede080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f1413640 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 0x55a81dee2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f0c12640 1 -- 172.21.15.7:0/4002947241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 msgr2=0x55a81dee2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f0c12640 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 0x55a81dee2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f0c12640 1 -- 172.21.15.7:0/4002947241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81dedaa80 msgr2=0x55a81dede080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f0c12640 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81dedaa80 0x55a81dede080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f0c12640 1 -- 172.21.15.7:0/4002947241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a81dede8b0 con 0x55a81ded1ea0 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f1413640 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 0x55a81dee2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87f0c12640 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee30c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00ffe0 tx=0x7f87dc009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e288b69770fa995 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.673+0000 7f87d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87d4ff9640 1 -- 172.21.15.7:0/4002947241 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0055a0 con 0x55a81ded1ea0 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87d4ff9640 1 -- 172.21.15.7:0/4002947241 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f87dc01a050 con 0x55a81ded1ea0 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87f625f740 1 -- 172.21.15.7:0/4002947241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87dc02c900 con 0x55a81ded1ea0 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87d4ff9640 1 -- 172.21.15.7:0/4002947241 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f87dc045050 con 0x55a81ded1ea0 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87f625f740 10 client.16698 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87f625f740 1 -- 172.21.15.7:0/4002947241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a81df50780 con 0x55a81ded1ea0 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87d4ff9640 1 -- 172.21.15.7:0/4002947241 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87dc02c900 con 0x55a81ded1ea0 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87d4ff9640 1 -- 172.21.15.7:0/4002947241 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f87dc035410 con 0x55a81ded1ea0 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87f625f740 10 client.16698 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87f625f740 10 client.16698 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87f625f740 20 client.16698 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87f625f740 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a81deea860 0x55a81df52e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87f625f740 4 client.16698 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.674+0000 7f87f625f740 1 -- 172.21.15.7:0/4002947241 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a81de3da70 con 0x55a81deea860 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.675+0000 7f87f1413640 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a81deea860 0x55a81df52e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.675+0000 7f87f1413640 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a81deea860 0x55a81df52e20 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0c2e0ad515183c0 server_cookie=43058a0db5401bbf in_seq=0 out_seq=0 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.675+0000 7f87d4ff9640 10 client.16698 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.675+0000 7f87d4ff9640 1 -- 172.21.15.7:0/4002947241 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x55a81de3da70 con 0x55a81deea860 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.675+0000 7f87d4ff9640 10 client.16698 handle_command_reply: tid=0 2024-04-21T08:49:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.675+0000 7f87f625f740 1 client.16698 shutdown 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f625f740 1 -- 172.21.15.7:0/4002947241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 msgr2=0x55a81dee30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f625f740 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee30c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00ffe0 tx=0x7f87dc009830 comp rx=0 tx=0).stop 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f625f740 1 -- 172.21.15.7:0/4002947241 shutdown_connections 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f625f740 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a81deea860 0x55a81df52e20 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f625f740 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a81dedaa80 0x55a81dede080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f625f740 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a81ded1ea0 0x55a81dee30c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f625f740 1 --2- 172.21.15.7:0/4002947241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a81ded1580 0x55a81dee2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87d4ff9640 0 client.16698 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f625f740 1 -- 172.21.15.7:0/4002947241 >> 172.21.15.7:0/4002947241 conn(0x55a81decbcb0 msgr2=0x55a81deca4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f625f740 1 -- 172.21.15.7:0/4002947241 shutdown_connections 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f625f740 1 -- 172.21.15.7:0/4002947241 wait complete. 2024-04-21T08:49:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f625f740 20 client.16698 trim_cache size 0 max 16384 2024-04-21T08:49:46.682 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:49:46.682 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (11870 inodes in the stack)", 2024-04-21T08:49:46.682 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:49:46.682 INFO:teuthology.orchestra.run.smithi007.stdout: "2f4d0e81-05a6-4410-bd5d-10ca0b65358b": { 2024-04-21T08:49:46.682 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:49:46.682 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "2f4d0e81-05a6-4410-bd5d-10ca0b65358b", 2024-04-21T08:49:46.682 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:49:46.682 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:49:46.682 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:49:46.682 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:49:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f3677640 1 -- 172.21.15.7:0/369695699 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3dd0 msgr2=0x7f87d80a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f3677640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3dd0 0x7f87d80a6290 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f87ec1028a0 tx=0x7f87dc009b60 comp rx=0 tx=0).stop 2024-04-21T08:49:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f3677640 1 -- 172.21.15.7:0/369695699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87ec101890 msgr2=0x7f87ec1a68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f3677640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87ec101890 0x7f87ec1a68b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f87e400eb10 tx=0x7f87e400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f3677640 1 -- 172.21.15.7:0/369695699 shutdown_connections 2024-04-21T08:49:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f3677640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87ec101890 0x7f87ec1a68b0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f3677640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ec107230 0x7f87ec101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f3677640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f87d80a3dd0 0x7f87d80a6290 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f3677640 1 --2- 172.21.15.7:0/369695699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87ec069570 0x7f87ec100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f3677640 1 -- 172.21.15.7:0/369695699 >> 172.21.15.7:0/369695699 conn(0x7f87ec077fa0 msgr2=0x7f87ec0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.676+0000 7f87f3677640 1 -- 172.21.15.7:0/369695699 shutdown_connections 2024-04-21T08:49:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:49:46.677+0000 7f87f3677640 1 -- 172.21.15.7:0/369695699 wait complete. 2024-04-21T08:49:46.687 INFO:tasks.cephfs.filesystem:scrub status for tag:2f4d0e81-05a6-4410-bd5d-10ca0b65358b - {'path': '/', 'tag': '2f4d0e81-05a6-4410-bd5d-10ca0b65358b', 'options': 'recursive,force'} 2024-04-21T08:49:46.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:46 smithi007 ceph-mon[28477]: pgmap v156: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.4 MiB/s wr, 1.84k op/s 2024-04-21T08:49:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:46 smithi016 ceph-mon[31259]: pgmap v156: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.4 MiB/s wr, 1.84k op/s 2024-04-21T08:49:47.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:46 smithi077 ceph-mon[31615]: pgmap v156: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.4 MiB/s wr, 1.84k op/s 2024-04-21T08:49:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.253+0000 7f207ddf4640 1 -- 172.21.15.7:0/162386718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2078107290 msgr2=0x7f207810ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.253+0000 7f207ddf4640 1 --2- 172.21.15.7:0/162386718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2078107290 0x7f207810ee60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2070009900 tx=0x7f207002bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.253+0000 7f207ddf4640 1 -- 172.21.15.7:0/162386718 shutdown_connections 2024-04-21T08:49:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.253+0000 7f207ddf4640 1 --2- 172.21.15.7:0/162386718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2078107290 0x7f207810ee60 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.253+0000 7f207ddf4640 1 --2- 172.21.15.7:0/162386718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2078102b70 0x7f2078106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.253+0000 7f207ddf4640 1 --2- 172.21.15.7:0/162386718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20781021c0 0x7f20781025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.253+0000 7f207ddf4640 1 -- 172.21.15.7:0/162386718 >> 172.21.15.7:0/162386718 conn(0x7f20780fda40 msgr2=0x7f20780ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.253+0000 7f207ddf4640 1 -- 172.21.15.7:0/162386718 shutdown_connections 2024-04-21T08:49:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.253+0000 7f207ddf4640 1 -- 172.21.15.7:0/162386718 wait complete. 2024-04-21T08:49:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.253+0000 7f207ddf4640 1 Processor -- start 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f207ddf4640 1 -- start start 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f207ddf4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20781021c0 0x7f2078074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f207ddf4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2078107290 0x7f2078074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f207ddf4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2078074cd0 0x7f20781a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f207ddf4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2078112200 con 0x7f2078107290 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f207ddf4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2078112080 con 0x7f20781021c0 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f207ddf4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2078112380 con 0x7f2078074cd0 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f2076ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2078107290 0x7f2078074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f2076ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2078107290 0x7f2078074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46766/0 (socket says 172.21.15.7:46766) 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f2076ffd640 1 -- 172.21.15.7:0/3857115768 learned_addr learned my addr 172.21.15.7:0/3857115768 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f20777fe640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20781021c0 0x7f2078074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f2077fff640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2078074cd0 0x7f20781a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f2076ffd640 1 -- 172.21.15.7:0/3857115768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2078074cd0 msgr2=0x7f20781a6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f2076ffd640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2078074cd0 0x7f20781a6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f2076ffd640 1 -- 172.21.15.7:0/3857115768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20781021c0 msgr2=0x7f2078074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f2076ffd640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20781021c0 0x7f2078074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f2076ffd640 1 -- 172.21.15.7:0/3857115768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20781a7010 con 0x7f2078107290 2024-04-21T08:49:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.254+0000 7f2076ffd640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2078107290 0x7f2078074790 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f206c00e9f0 tx=0x7f206c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.255+0000 7f2074ff9640 1 -- 172.21.15.7:0/3857115768 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c00ccc0 con 0x7f2078107290 2024-04-21T08:49:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.255+0000 7f207ddf4640 1 -- 172.21.15.7:0/3857115768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20781a7300 con 0x7f2078107290 2024-04-21T08:49:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.255+0000 7f2074ff9640 1 -- 172.21.15.7:0/3857115768 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f206c0054d0 con 0x7f2078107290 2024-04-21T08:49:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.255+0000 7f2074ff9640 1 -- 172.21.15.7:0/3857115768 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c010430 con 0x7f2078107290 2024-04-21T08:49:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.255+0000 7f207ddf4640 1 -- 172.21.15.7:0/3857115768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20781a7800 con 0x7f2078107290 2024-04-21T08:49:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f2074ff9640 1 -- 172.21.15.7:0/3857115768 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f206c0046c0 con 0x7f2078107290 2024-04-21T08:49:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f20809e8740 1 Processor -- start 2024-04-21T08:49:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f20809e8740 1 -- start start 2024-04-21T08:49:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f20809e8740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf3429620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f20809e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf3429fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f20809e8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf34354d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f20809e8740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558bf338e8d0 con 0x558bf342a8d0 2024-04-21T08:49:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f20809e8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558bf332be80 con 0x558bf34330e0 2024-04-21T08:49:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f20809e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558bf338a890 con 0x558bf3429b60 2024-04-21T08:49:47.264 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f2077fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf3429620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f2077fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf3429620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46782/0 (socket says 172.21.15.7:46782) 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f2077fff640 1 -- 172.21.15.7:0/2611812303 learned_addr learned my addr 172.21.15.7:0/2611812303 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f20777fe640 1 --2- 172.21.15.7:0/2611812303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf3429fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f2076ffd640 1 --2- 172.21.15.7:0/2611812303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf34354d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.257+0000 7f2074ff9640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f20600a3d10 0x7f20600a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f2077fff640 1 -- 172.21.15.7:0/2611812303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 msgr2=0x558bf3429fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f2077fff640 1 --2- 172.21.15.7:0/2611812303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf3429fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f2077fff640 1 -- 172.21.15.7:0/2611812303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 msgr2=0x558bf34354d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f2077fff640 1 --2- 172.21.15.7:0/2611812303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf34354d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f2077fff640 1 -- 172.21.15.7:0/2611812303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bf334d320 con 0x558bf342a8d0 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f20777fe640 1 --2- 172.21.15.7:0/2611812303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf3429fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f2074ff9640 1 -- 172.21.15.7:0/3857115768 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f206c0d0f20 con 0x7f2078107290 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f2077fff640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f20600a3d10 0x7f20600a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.265 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f2077fff640 1 --2- 172.21.15.7:0/2611812303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf3429620 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2070036040 tx=0x7f20700095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66cd2cecb54db38e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f205dffb640 1 -- 172.21.15.7:0/2611812303 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070046070 con 0x558bf342a8d0 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f205dffb640 1 -- 172.21.15.7:0/2611812303 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20700044a0 con 0x558bf342a8d0 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f205dffb640 1 -- 172.21.15.7:0/2611812303 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070004cf0 con 0x558bf342a8d0 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.258+0000 7f2077fff640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f20600a3d10 0x7f20600a61d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f207004c000 tx=0x7f2070049000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.259+0000 7f20809e8740 1 -- 172.21.15.7:0/2611812303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 msgr2=0x558bf3429620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.259+0000 7f20809e8740 1 --2- 172.21.15.7:0/2611812303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf3429620 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2070036040 tx=0x7f20700095e0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.259+0000 7f20809e8740 1 -- 172.21.15.7:0/2611812303 shutdown_connections 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.259+0000 7f20809e8740 1 --2- 172.21.15.7:0/2611812303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf34354d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.259+0000 7f20809e8740 1 --2- 172.21.15.7:0/2611812303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf3429fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.259+0000 7f20809e8740 1 --2- 172.21.15.7:0/2611812303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf3429620 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.259+0000 7f20809e8740 1 -- 172.21.15.7:0/2611812303 >> 172.21.15.7:0/2611812303 conn(0x558bf33f0c90 msgr2=0x558bf3424850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.259+0000 7f20809e8740 1 -- 172.21.15.7:0/2611812303 shutdown_connections 2024-04-21T08:49:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.259+0000 7f20809e8740 1 -- 172.21.15.7:0/2611812303 wait complete. 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.259+0000 7f20809e8740 1 -- start start 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.259+0000 7f20809e8740 1 Processor -- start 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f20809e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf3432a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f20809e8740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf342fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f20809e8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf34303b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f20809e8740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558bf3435e10 con 0x558bf342a8d0 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f20809e8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558bf3316eb0 con 0x558bf34330e0 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f20809e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558bf3435f90 con 0x558bf3429b60 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f20809e8740 10 client.0 _ll_register_callbacks cb 0x558bf3343b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2076ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf342fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2076ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf342fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46794/0 (socket says 172.21.15.7:46794) 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2076ffd640 1 -- 172.21.15.7:0/402942707 learned_addr learned my addr 172.21.15.7:0/402942707 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f20777fe640 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf3432a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2077fff640 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf34303b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2076ffd640 1 -- 172.21.15.7:0/402942707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 msgr2=0x558bf3432a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2076ffd640 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf3432a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2076ffd640 1 -- 172.21.15.7:0/402942707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 msgr2=0x558bf34303b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2076ffd640 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf34303b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2076ffd640 1 -- 172.21.15.7:0/402942707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bf30a1210 con 0x558bf342a8d0 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2076ffd640 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf342fe10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f206c0ce2e0 tx=0x7f206c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aacfe6da433d056a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2056ffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2056ffd640 1 -- 172.21.15.7:0/402942707 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c09d1d0 con 0x558bf342a8d0 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.260+0000 7f2056ffd640 1 -- 172.21.15.7:0/402942707 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f206c0a52b0 con 0x558bf342a8d0 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f20809e8740 1 -- 172.21.15.7:0/402942707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20700044a0 con 0x558bf342a8d0 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f2056ffd640 1 -- 172.21.15.7:0/402942707 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f206c09cad0 con 0x558bf342a8d0 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f20809e8740 10 client.16710 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f20809e8740 1 -- 172.21.15.7:0/402942707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558bf316d620 con 0x558bf342a8d0 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f2056ffd640 1 -- 172.21.15.7:0/402942707 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20700044a0 con 0x558bf342a8d0 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f2056ffd640 1 -- 172.21.15.7:0/402942707 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f206c09c0b0 con 0x558bf342a8d0 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f20809e8740 10 client.16710 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f20809e8740 10 client.16710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f20809e8740 20 client.16710 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f20809e8740 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558bf3443150 0x558bf3445540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f20809e8740 4 client.16710 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f20809e8740 1 -- 172.21.15.7:0/402942707 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bf33a2880 con 0x558bf3443150 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.261+0000 7f20777fe640 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558bf3443150 0x558bf3445540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.262+0000 7f2056ffd640 10 client.16710 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.262+0000 7f20777fe640 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558bf3443150 0x558bf3445540 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccbe91c2b3b00c3a server_cookie=3c2d451a0f734a30 in_seq=0 out_seq=0 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.263+0000 7f2056ffd640 1 -- 172.21.15.7:0/402942707 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558bf33a2880 con 0x558bf3443150 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.263+0000 7f2056ffd640 10 client.16710 handle_command_reply: tid=0 2024-04-21T08:49:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 1 client.16710 shutdown 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 1 -- 172.21.15.7:0/402942707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 msgr2=0x558bf342fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf342fe10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f206c0ce2e0 tx=0x7f206c009310 comp rx=0 tx=0).stop 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 1 -- 172.21.15.7:0/402942707 shutdown_connections 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558bf3443150 0x558bf3445540 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf34303b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf342fe10 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 1 --2- 172.21.15.7:0/402942707 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf3432a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f2056ffd640 0 client.16710 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 1 -- 172.21.15.7:0/402942707 >> 172.21.15.7:0/402942707 conn(0x558bf33f0c90 msgr2=0x558bf34265a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 1 -- 172.21.15.7:0/402942707 shutdown_connections 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 1 -- 172.21.15.7:0/402942707 wait complete. 2024-04-21T08:49:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.264+0000 7f20809e8740 20 client.16710 trim_cache size 0 max 16384 2024-04-21T08:49:47.284 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f20809e8740 1 Processor -- start 2024-04-21T08:49:47.284 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f20809e8740 1 -- start start 2024-04-21T08:49:47.284 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f20809e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf343ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.284 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f20809e8740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf344ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.284 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f20809e8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf344b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.284 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f20809e8740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f20781af5d0 con 0x558bf342a8d0 2024-04-21T08:49:47.284 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f20809e8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f20781b1e50 con 0x558bf34330e0 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f20809e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f20781afc40 con 0x558bf3429b60 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f2076ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf344ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f20777fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf343ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f2076ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf344ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46808/0 (socket says 172.21.15.7:46808) 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f2076ffd640 1 -- 172.21.15.7:0/2991059574 learned_addr learned my addr 172.21.15.7:0/2991059574 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f2077fff640 1 --2- 172.21.15.7:0/2991059574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf344b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f2076ffd640 1 -- 172.21.15.7:0/2991059574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 msgr2=0x558bf343ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f2076ffd640 1 --2- 172.21.15.7:0/2991059574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf343ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f2076ffd640 1 -- 172.21.15.7:0/2991059574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 msgr2=0x558bf344b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f2076ffd640 1 --2- 172.21.15.7:0/2991059574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf344b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f2076ffd640 1 -- 172.21.15.7:0/2991059574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bf342e360 con 0x558bf342a8d0 2024-04-21T08:49:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.280+0000 7f20777fe640 1 --2- 172.21.15.7:0/2991059574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf343ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f2076ffd640 1 --2- 172.21.15.7:0/2991059574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf344ae10 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f206c09c7d0 tx=0x7f206c0b6120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a009a011320b38 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f205d7fa640 1 -- 172.21.15.7:0/2991059574 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c09d1d0 con 0x558bf342a8d0 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f205d7fa640 1 -- 172.21.15.7:0/2991059574 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f206c0a0e80 con 0x558bf342a8d0 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f205d7fa640 1 -- 172.21.15.7:0/2991059574 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c005000 con 0x558bf342a8d0 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f20809e8740 1 -- 172.21.15.7:0/2991059574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 msgr2=0x558bf344ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f20809e8740 1 --2- 172.21.15.7:0/2991059574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf344ae10 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f206c09c7d0 tx=0x7f206c0b6120 comp rx=0 tx=0).stop 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f20809e8740 1 -- 172.21.15.7:0/2991059574 shutdown_connections 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f20809e8740 1 --2- 172.21.15.7:0/2991059574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf34330e0 0x558bf344b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f20809e8740 1 --2- 172.21.15.7:0/2991059574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf342a8d0 0x558bf344ae10 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f20809e8740 1 --2- 172.21.15.7:0/2991059574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf3429b60 0x558bf343ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f20809e8740 1 -- 172.21.15.7:0/2991059574 >> 172.21.15.7:0/2991059574 conn(0x558bf33f0c90 msgr2=0x558bf3424380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f20809e8740 1 -- 172.21.15.7:0/2991059574 shutdown_connections 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.281+0000 7f20809e8740 1 -- 172.21.15.7:0/2991059574 wait complete. 2024-04-21T08:49:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f20809e8740 1 -- start start 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f20809e8740 1 Processor -- start 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f20809e8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf3429b60 0x558bf343b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f20809e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf342a8d0 0x558bf342c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f20809e8740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf34330e0 0x558bf342cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f20809e8740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f20781b0310 con 0x558bf34330e0 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f20809e8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f20781b1780 con 0x558bf3429b60 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f20809e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f20781b09e0 con 0x558bf342a8d0 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f20809e8740 10 client.0 _ll_register_callbacks cb 0x7f206c099380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f2077fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf34330e0 0x558bf342cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f20777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf3429b60 0x558bf343b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f2076ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf342a8d0 0x558bf342c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f2077fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf34330e0 0x558bf342cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46820/0 (socket says 172.21.15.7:46820) 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f20777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf3429b60 0x558bf343b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:43242/0 (socket says 172.21.15.7:43242) 2024-04-21T08:49:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.282+0000 7f2077fff640 1 -- 172.21.15.7:0/2302030003 learned_addr learned my addr 172.21.15.7:0/2302030003 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2077fff640 1 -- 172.21.15.7:0/2302030003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf342a8d0 msgr2=0x558bf342c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2077fff640 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf342a8d0 0x558bf342c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2077fff640 1 -- 172.21.15.7:0/2302030003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf3429b60 msgr2=0x558bf343b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2077fff640 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf3429b60 0x558bf343b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2077fff640 1 -- 172.21.15.7:0/2302030003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bf342d3e0 con 0x558bf34330e0 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f20777fe640 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf3429b60 0x558bf343b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2077fff640 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf34330e0 0x558bf342cca0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2070006010 tx=0x7f207002c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af3f71a61f8bcee server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2056ffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2056ffd640 1 -- 172.21.15.7:0/2302030003 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070046040 con 0x558bf34330e0 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2056ffd640 1 -- 172.21.15.7:0/2302030003 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f2070004fd0 con 0x558bf34330e0 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f20809e8740 1 -- 172.21.15.7:0/2302030003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f206c0a0e80 con 0x558bf34330e0 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2056ffd640 1 -- 172.21.15.7:0/2302030003 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2070064050 con 0x558bf34330e0 2024-04-21T08:49:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f20809e8740 10 client.16716 fetch_fsmap learned FSMap version 23 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f20809e8740 1 -- 172.21.15.7:0/2302030003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558bf34a95e0 con 0x558bf34330e0 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2056ffd640 1 -- 172.21.15.7:0/2302030003 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f206c0a0e80 con 0x558bf34330e0 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f2056ffd640 1 -- 172.21.15.7:0/2302030003 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f2070003990 con 0x558bf34330e0 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f20809e8740 10 client.16716 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.283+0000 7f20809e8740 10 client.16716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.284+0000 7f20809e8740 20 client.16716 populate_metadata read hostname 'smithi007' 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.284+0000 7f20809e8740 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558bf34a98b0 0x558bf34abca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.284+0000 7f20809e8740 4 client.16716 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.284+0000 7f20809e8740 1 -- 172.21.15.7:0/2302030003 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558bf332e3d0 con 0x558bf34a98b0 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.284+0000 7f20777fe640 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558bf34a98b0 0x558bf34abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.284+0000 7f20777fe640 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558bf34a98b0 0x558bf34abca0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43174f09522fbf04 server_cookie=d0eedc199fb012ac in_seq=0 out_seq=0 2024-04-21T08:49:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:47.284+0000 7f2056ffd640 10 client.16716 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:49.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:48 smithi016 ceph-mon[31259]: pgmap v157: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 1.53k op/s 2024-04-21T08:49:49.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:48 smithi077 ceph-mon[31615]: pgmap v157: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 1.53k op/s 2024-04-21T08:49:49.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:48 smithi007 ceph-mon[28477]: pgmap v157: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 1.53k op/s 2024-04-21T08:49:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:50 smithi016 ceph-mon[31259]: pgmap v158: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.7 MiB/s wr, 3.36k op/s 2024-04-21T08:49:51.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:50 smithi077 ceph-mon[31615]: pgmap v158: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.7 MiB/s wr, 3.36k op/s 2024-04-21T08:49:51.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:50 smithi007 ceph-mon[28477]: pgmap v158: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.7 MiB/s wr, 3.36k op/s 2024-04-21T08:49:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.825+0000 7f2056ffd640 1 -- 172.21.15.7:0/2302030003 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558bf332e3d0 con 0x558bf34a98b0 2024-04-21T08:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.825+0000 7f2056ffd640 10 client.16716 handle_command_reply: tid=0 2024-04-21T08:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.825+0000 7f20809e8740 1 client.16716 shutdown 2024-04-21T08:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.825+0000 7f20809e8740 1 -- 172.21.15.7:0/2302030003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf34330e0 msgr2=0x558bf342cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.825+0000 7f20809e8740 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf34330e0 0x558bf342cca0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2070006010 tx=0x7f207002c6d0 comp rx=0 tx=0).stop 2024-04-21T08:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.825+0000 7f20809e8740 1 -- 172.21.15.7:0/2302030003 shutdown_connections 2024-04-21T08:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.825+0000 7f20809e8740 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558bf34a98b0 0x558bf34abca0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f20809e8740 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558bf34330e0 0x558bf342cca0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f20809e8740 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558bf342a8d0 0x558bf342c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f20809e8740 1 --2- 172.21.15.7:0/2302030003 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558bf3429b60 0x558bf343b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f20809e8740 1 -- 172.21.15.7:0/2302030003 >> 172.21.15.7:0/2302030003 conn(0x558bf33f0c90 msgr2=0x558bf342e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f2056ffd640 0 client.16716 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f20809e8740 1 -- 172.21.15.7:0/2302030003 shutdown_connections 2024-04-21T08:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f20809e8740 1 -- 172.21.15.7:0/2302030003 wait complete. 2024-04-21T08:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f20809e8740 20 client.16716 trim_cache size 0 max 16384 2024-04-21T08:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:49:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f207ddf4640 1 -- 172.21.15.7:0/3857115768 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f20600a3d10 msgr2=0x7f20600a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f207ddf4640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f20600a3d10 0x7f20600a61d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f207004c000 tx=0x7f2070049000 comp rx=0 tx=0).stop 2024-04-21T08:49:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f207ddf4640 1 -- 172.21.15.7:0/3857115768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2078107290 msgr2=0x7f2078074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.826+0000 7f207ddf4640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2078107290 0x7f2078074790 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f206c00e9f0 tx=0x7f206c00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:49:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.827+0000 7f207ddf4640 1 -- 172.21.15.7:0/3857115768 shutdown_connections 2024-04-21T08:49:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.827+0000 7f207ddf4640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f20600a3d10 0x7f20600a61d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.827+0000 7f207ddf4640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2078074cd0 0x7f20781a6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.827+0000 7f207ddf4640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2078107290 0x7f2078074790 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.827+0000 7f207ddf4640 1 --2- 172.21.15.7:0/3857115768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20781021c0 0x7f2078074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.827+0000 7f207ddf4640 1 -- 172.21.15.7:0/3857115768 >> 172.21.15.7:0/3857115768 conn(0x7f20780fda40 msgr2=0x7f20780fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.827+0000 7f207ddf4640 1 -- 172.21.15.7:0/3857115768 shutdown_connections 2024-04-21T08:49:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:49:51.827+0000 7f207ddf4640 1 -- 172.21.15.7:0/3857115768 wait complete. 2024-04-21T08:49:52.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:51 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:52.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:51 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:52.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:51 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:52 smithi016 ceph-mon[31259]: pgmap v159: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.4 MiB/s wr, 2.67k op/s 2024-04-21T08:49:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:52 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:49:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:52 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:49:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:52 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:49:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:52 smithi077 ceph-mon[31615]: pgmap v159: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.4 MiB/s wr, 2.67k op/s 2024-04-21T08:49:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:52 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:49:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:52 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:49:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:52 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:49:53.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:52 smithi007 ceph-mon[28477]: pgmap v159: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.4 MiB/s wr, 2.67k op/s 2024-04-21T08:49:53.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:52 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:49:53.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:52 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:49:53.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:52 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:49:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:54 smithi016 ceph-mon[31259]: pgmap v160: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.6 MiB/s wr, 3.45k op/s 2024-04-21T08:49:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:54 smithi077 ceph-mon[31615]: pgmap v160: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.6 MiB/s wr, 3.45k op/s 2024-04-21T08:49:55.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:54 smithi007 ceph-mon[28477]: pgmap v160: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 5.6 MiB/s wr, 3.45k op/s 2024-04-21T08:49:57.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:56 smithi016 ceph-mon[31259]: pgmap v161: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 3.79k op/s 2024-04-21T08:49:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:56 smithi077 ceph-mon[31615]: pgmap v161: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 3.79k op/s 2024-04-21T08:49:57.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:56 smithi007 ceph-mon[28477]: pgmap v161: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 3.79k op/s 2024-04-21T08:49:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:49:58 smithi016 ceph-mon[31259]: pgmap v162: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 870 KiB/s wr, 3.34k op/s 2024-04-21T08:49:59.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:49:58 smithi077 ceph-mon[31615]: pgmap v162: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 870 KiB/s wr, 3.34k op/s 2024-04-21T08:49:59.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:49:58 smithi007 ceph-mon[28477]: pgmap v162: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 870 KiB/s wr, 3.34k op/s 2024-04-21T08:50:01.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:00 smithi016 ceph-mon[31259]: pgmap v163: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 3.99k op/s 2024-04-21T08:50:01.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:00 smithi016 ceph-mon[31259]: overall HEALTH_OK 2024-04-21T08:50:01.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:00 smithi077 ceph-mon[31615]: pgmap v163: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 3.99k op/s 2024-04-21T08:50:01.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:00 smithi077 ceph-mon[31615]: overall HEALTH_OK 2024-04-21T08:50:01.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:00 smithi007 ceph-mon[28477]: pgmap v163: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 3.99k op/s 2024-04-21T08:50:01.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:00 smithi007 ceph-mon[28477]: overall HEALTH_OK 2024-04-21T08:50:03.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:02 smithi016 ceph-mon[31259]: pgmap v164: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 2.16k op/s 2024-04-21T08:50:03.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:02 smithi077 ceph-mon[31615]: pgmap v164: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 2.16k op/s 2024-04-21T08:50:03.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:02 smithi007 ceph-mon[28477]: pgmap v164: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.8 MiB/s wr, 2.16k op/s 2024-04-21T08:50:04.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.916+0000 7fa92112b640 1 -- 172.21.15.7:0/1964718874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa91c100810 msgr2=0x7fa91c100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:04.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.916+0000 7fa92112b640 1 --2- 172.21.15.7:0/1964718874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa91c100810 0x7fa91c100c90 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa904009930 tx=0x7fa90402bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:04.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.916+0000 7fa92112b640 1 -- 172.21.15.7:0/1964718874 shutdown_connections 2024-04-21T08:50:04.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.916+0000 7fa92112b640 1 --2- 172.21.15.7:0/1964718874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c1011d0 0x7fa91c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.916+0000 7fa92112b640 1 --2- 172.21.15.7:0/1964718874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa91c100810 0x7fa91c100c90 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.919 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.916+0000 7fa92112b640 1 --2- 172.21.15.7:0/1964718874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa91c106fb0 0x7fa91c107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.916+0000 7fa92112b640 1 -- 172.21.15.7:0/1964718874 >> 172.21.15.7:0/1964718874 conn(0x7fa91c076190 msgr2=0x7fa91c0765a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.916+0000 7fa92112b640 1 -- 172.21.15.7:0/1964718874 shutdown_connections 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.916+0000 7fa92112b640 1 -- 172.21.15.7:0/1964718874 wait complete. 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.917+0000 7fa92112b640 1 Processor -- start 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.917+0000 7fa92112b640 1 -- start start 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.917+0000 7fa92112b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c100810 0x7fa91c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.917+0000 7fa92112b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa91c1011d0 0x7fa91c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.917+0000 7fa92112b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa91c106fb0 0x7fa91c19d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.917+0000 7fa92112b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c112260 con 0x7fa91c1011d0 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.917+0000 7fa92112b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1120e0 con 0x7fa91c106fb0 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.917+0000 7fa92112b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1123e0 con 0x7fa91c100810 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.917+0000 7fa91259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa91c1011d0 0x7fa91c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.917+0000 7fa91259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa91c1011d0 0x7fa91c198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48694/0 (socket says 172.21.15.7:48694) 2024-04-21T08:50:04.920 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.917+0000 7fa91259c640 1 -- 172.21.15.7:0/1025107887 learned_addr learned my addr 172.21.15.7:0/1025107887 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c100810 0x7fa91c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa91b59e640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa91c106fb0 0x7fa91c19d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa91259c640 1 -- 172.21.15.7:0/1025107887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c100810 msgr2=0x7fa91c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa91259c640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c100810 0x7fa91c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa91259c640 1 -- 172.21.15.7:0/1025107887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa91c106fb0 msgr2=0x7fa91c19d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa91259c640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa91c106fb0 0x7fa91c19d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa91259c640 1 -- 172.21.15.7:0/1025107887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91c19d730 con 0x7fa91c1011d0 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c100810 0x7fa91c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa91259c640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa91c1011d0 0x7fa91c198c80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa904000c00 tx=0x7fa9040045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa918d99640 1 -- 172.21.15.7:0/1025107887 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa904043020 con 0x7fa91c1011d0 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa92112b640 1 -- 172.21.15.7:0/1025107887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91c19d9c0 con 0x7fa91c1011d0 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa918d99640 1 -- 172.21.15.7:0/1025107887 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa90402cb70 con 0x7fa91c1011d0 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa92112b640 1 -- 172.21.15.7:0/1025107887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91c1a52b0 con 0x7fa91c1011d0 2024-04-21T08:50:04.921 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.918+0000 7fa918d99640 1 -- 172.21.15.7:0/1025107887 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa904003b20 con 0x7fa91c1011d0 2024-04-21T08:50:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.920+0000 7fa918d99640 1 -- 172.21.15.7:0/1025107887 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa904046020 con 0x7fa91c1011d0 2024-04-21T08:50:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa923df7740 1 Processor -- start 2024-04-21T08:50:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa923df7740 1 -- start start 2024-04-21T08:50:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa923df7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38db58d0 0x557d38db4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa923df7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38db4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa923df7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38dbe0e0 0x557d38dc04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa923df7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557d38d198d0 con 0x557d38db4b60 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa918d99640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa9080a3d10 0x7fa9080a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa923df7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557d38cb6e80 con 0x557d38dbe0e0 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa923df7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557d38d15890 con 0x557d38db58d0 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa91ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38db4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa91b59e640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa9080a3d10 0x7fa9080a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa91ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38db4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48696/0 (socket says 172.21.15.7:48696) 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa91ad9d640 1 -- 172.21.15.7:0/4144394956 learned_addr learned my addr 172.21.15.7:0/4144394956 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa918d99640 1 -- 172.21.15.7:0/1025107887 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa9040f2c80 con 0x7fa91c1011d0 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.921+0000 7fa91b59e640 1 --2- 172.21.15.7:0/4144394956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38db58d0 0x557d38db4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.922+0000 7fa91259c640 1 --2- 172.21.15.7:0/4144394956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38dbe0e0 0x557d38dc04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.922+0000 7fa91ad9d640 1 -- 172.21.15.7:0/4144394956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38db58d0 msgr2=0x557d38db4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.922+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/4144394956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38db58d0 0x557d38db4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.922+0000 7fa91ad9d640 1 -- 172.21.15.7:0/4144394956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38dbe0e0 msgr2=0x557d38dc04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.922+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/4144394956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38dbe0e0 0x557d38dc04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.922+0000 7fa91ad9d640 1 -- 172.21.15.7:0/4144394956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d38cd8320 con 0x557d38db4b60 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.922+0000 7fa91b59e640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa9080a3d10 0x7fa9080a61d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa914009270 tx=0x7fa914009e30 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.922+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/4144394956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38db4fe0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00bd40 tx=0x7fa90c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f101f88e2cc64c65 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.922+0000 7fa91159a640 1 -- 172.21.15.7:0/4144394956 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c015070 con 0x557d38db4b60 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.922+0000 7fa91159a640 1 -- 172.21.15.7:0/4144394956 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa90c00be30 con 0x557d38db4b60 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.922+0000 7fa91159a640 1 -- 172.21.15.7:0/4144394956 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c011900 con 0x557d38db4b60 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.923+0000 7fa923df7740 1 -- 172.21.15.7:0/4144394956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 msgr2=0x557d38db4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:04.934 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.923+0000 7fa923df7740 1 --2- 172.21.15.7:0/4144394956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38db4fe0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00bd40 tx=0x7fa90c00f730 comp rx=0 tx=0).stop 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.923+0000 7fa923df7740 1 -- 172.21.15.7:0/4144394956 shutdown_connections 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.923+0000 7fa923df7740 1 --2- 172.21.15.7:0/4144394956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38dbe0e0 0x557d38dc04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.923+0000 7fa923df7740 1 --2- 172.21.15.7:0/4144394956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38db4fe0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.923+0000 7fa923df7740 1 --2- 172.21.15.7:0/4144394956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38db58d0 0x557d38db4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.923+0000 7fa923df7740 1 -- 172.21.15.7:0/4144394956 >> 172.21.15.7:0/4144394956 conn(0x557d38d7bc90 msgr2=0x557d38daf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.923+0000 7fa923df7740 1 -- 172.21.15.7:0/4144394956 shutdown_connections 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.923+0000 7fa923df7740 1 -- 172.21.15.7:0/4144394956 wait complete. 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.923+0000 7fa923df7740 1 -- start start 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.923+0000 7fa923df7740 1 Processor -- start 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa923df7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dbdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa923df7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38db58d0 0x557d38dbae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa923df7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38dbe0e0 0x557d38dbb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa923df7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557d38dc0e10 con 0x557d38db4b60 2024-04-21T08:50:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa923df7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x557d38ca1eb0 con 0x557d38db58d0 2024-04-21T08:50:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa923df7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557d38dc0f90 con 0x557d38dbe0e0 2024-04-21T08:50:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa923df7740 10 client.0 _ll_register_callbacks cb 0x557d38cceb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa91ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dbdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa91ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dbdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48702/0 (socket says 172.21.15.7:48702) 2024-04-21T08:50:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa91ad9d640 1 -- 172.21.15.7:0/1642899273 learned_addr learned my addr 172.21.15.7:0/1642899273 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa91b59e640 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38dbe0e0 0x557d38dbb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa91259c640 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38db58d0 0x557d38dbae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa91ad9d640 1 -- 172.21.15.7:0/1642899273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38dbe0e0 msgr2=0x557d38dbb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38dbe0e0 0x557d38dbb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa91ad9d640 1 -- 172.21.15.7:0/1642899273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38db58d0 msgr2=0x557d38dbae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38db58d0 0x557d38dbae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa91ad9d640 1 -- 172.21.15.7:0/1642899273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d38a2c210 con 0x557d38db4b60 2024-04-21T08:50:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dbdab0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa90c019000 tx=0x7fa90c00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bcba149bc80870 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa9027fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa9027fc640 1 -- 172.21.15.7:0/1642899273 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c015070 con 0x557d38db4b60 2024-04-21T08:50:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa9027fc640 1 -- 172.21.15.7:0/1642899273 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa90c005350 con 0x557d38db4b60 2024-04-21T08:50:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.924+0000 7fa923df7740 1 -- 172.21.15.7:0/1642899273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa90c00be30 con 0x557d38db4b60 2024-04-21T08:50:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa9027fc640 1 -- 172.21.15.7:0/1642899273 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa90c005550 con 0x557d38db4b60 2024-04-21T08:50:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa923df7740 10 client.16728 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa923df7740 1 -- 172.21.15.7:0/1642899273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557d38af8620 con 0x557d38db4b60 2024-04-21T08:50:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa9027fc640 1 -- 172.21.15.7:0/1642899273 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa90c00be30 con 0x557d38db4b60 2024-04-21T08:50:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa9027fc640 1 -- 172.21.15.7:0/1642899273 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa90c0049e0 con 0x557d38db4b60 2024-04-21T08:50:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa923df7740 10 client.16728 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa923df7740 10 client.16728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa923df7740 20 client.16728 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa923df7740 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x557d38dce050 0x557d38dd0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa923df7740 4 client.16728 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa923df7740 1 -- 172.21.15.7:0/1642899273 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d38d2d880 con 0x557d38dce050 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa91259c640 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x557d38dce050 0x557d38dd0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa91259c640 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x557d38dce050 0x557d38dd0440 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56da5cedd8ae067c server_cookie=7e098890cfa57671 in_seq=0 out_seq=0 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.925+0000 7fa9027fc640 10 client.16728 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa9027fc640 1 -- 172.21.15.7:0/1642899273 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d38d2d880 con 0x557d38dce050 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa9027fc640 10 client.16728 handle_command_reply: tid=0 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 1 client.16728 shutdown 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 1 -- 172.21.15.7:0/1642899273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 msgr2=0x557d38dbdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dbdab0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa90c019000 tx=0x7fa90c00bab0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 1 -- 172.21.15.7:0/1642899273 shutdown_connections 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x557d38dce050 0x557d38dd0440 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38dbe0e0 0x557d38dbb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38db58d0 0x557d38dbae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa9027fc640 0 client.16728 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:04.940 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 1 --2- 172.21.15.7:0/1642899273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dbdab0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.940 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 1 -- 172.21.15.7:0/1642899273 >> 172.21.15.7:0/1642899273 conn(0x557d38d7bc90 msgr2=0x557d38db15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:04.940 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 1 -- 172.21.15.7:0/1642899273 shutdown_connections 2024-04-21T08:50:04.940 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 1 -- 172.21.15.7:0/1642899273 wait complete. 2024-04-21T08:50:04.940 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.926+0000 7fa923df7740 20 client.16728 trim_cache size 0 max 16384 2024-04-21T08:50:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.941+0000 7fa923df7740 1 Processor -- start 2024-04-21T08:50:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.941+0000 7fa923df7740 1 -- start start 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.941+0000 7fa923df7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dc9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.941+0000 7fa923df7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38db58d0 0x557d38dbd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.941+0000 7fa923df7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38dbe0e0 0x557d38dbdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.941+0000 7fa923df7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1ad440 con 0x557d38db4b60 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.941+0000 7fa923df7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1afcc0 con 0x557d38db58d0 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.941+0000 7fa923df7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1adab0 con 0x557d38dbe0e0 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa91ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dc9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa91ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dc9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48708/0 (socket says 172.21.15.7:48708) 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa91ad9d640 1 -- 172.21.15.7:0/3717561013 learned_addr learned my addr 172.21.15.7:0/3717561013 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa91b59e640 1 --2- 172.21.15.7:0/3717561013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38dbe0e0 0x557d38dbdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa91259c640 1 --2- 172.21.15.7:0/3717561013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38db58d0 0x557d38dbd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa91ad9d640 1 -- 172.21.15.7:0/3717561013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38dbe0e0 msgr2=0x557d38dbdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/3717561013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38dbe0e0 0x557d38dbdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa91ad9d640 1 -- 172.21.15.7:0/3717561013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38db58d0 msgr2=0x557d38dbd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/3717561013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38db58d0 0x557d38dbd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa91ad9d640 1 -- 172.21.15.7:0/3717561013 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d38db3e30 con 0x557d38db4b60 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/3717561013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dc9d10 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa90c01a500 tx=0x7fa90c00b660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3a8ebc3ffb3aca server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa910d99640 1 -- 172.21.15.7:0/3717561013 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c015050 con 0x557d38db4b60 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa910d99640 1 -- 172.21.15.7:0/3717561013 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa90c005350 con 0x557d38db4b60 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.942+0000 7fa910d99640 1 -- 172.21.15.7:0/3717561013 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c01ab00 con 0x557d38db4b60 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 -- 172.21.15.7:0/3717561013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 msgr2=0x557d38dc9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 --2- 172.21.15.7:0/3717561013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dc9d10 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa90c01a500 tx=0x7fa90c00b660 comp rx=0 tx=0).stop 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 -- 172.21.15.7:0/3717561013 shutdown_connections 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 --2- 172.21.15.7:0/3717561013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38dbe0e0 0x557d38dbdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 --2- 172.21.15.7:0/3717561013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38db58d0 0x557d38dbd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 --2- 172.21.15.7:0/3717561013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db4b60 0x557d38dc9d10 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 -- 172.21.15.7:0/3717561013 >> 172.21.15.7:0/3717561013 conn(0x557d38d7bc90 msgr2=0x557d38daf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 -- 172.21.15.7:0/3717561013 shutdown_connections 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 -- 172.21.15.7:0/3717561013 wait complete. 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 -- start start 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 Processor -- start 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38db4b60 0x557d38dc6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db58d0 0x557d38dc6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38dbe0e0 0x557d38db8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1ae180 con 0x557d38db58d0 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1af5f0 con 0x557d38dbe0e0 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1ae850 con 0x557d38db4b60 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.943+0000 7fa923df7740 10 client.0 _ll_register_callbacks cb 0x7fa914012750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa91259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db58d0 0x557d38dc6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa91259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db58d0 0x557d38dc6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48724/0 (socket says 172.21.15.7:48724) 2024-04-21T08:50:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa91259c640 1 -- 172.21.15.7:0/4166752309 learned_addr learned my addr 172.21.15.7:0/4166752309 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa91b59e640 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38dbe0e0 0x557d38db8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38db4b60 0x557d38dc6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa91259c640 1 -- 172.21.15.7:0/4166752309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38db4b60 msgr2=0x557d38dc6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa91259c640 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38db4b60 0x557d38dc6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa91259c640 1 -- 172.21.15.7:0/4166752309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38dbe0e0 msgr2=0x557d38db8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa91259c640 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38dbe0e0 0x557d38db8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa91259c640 1 -- 172.21.15.7:0/4166752309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d38dd09a0 con 0x557d38db58d0 2024-04-21T08:50:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa91259c640 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db58d0 0x557d38dc6760 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa904006010 tx=0x7fa9040ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41001443d493131 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa9027fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa9027fc640 1 -- 172.21.15.7:0/4166752309 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9040bea60 con 0x557d38db58d0 2024-04-21T08:50:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa9027fc640 1 -- 172.21.15.7:0/4166752309 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa904005490 con 0x557d38db58d0 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa923df7740 1 -- 172.21.15.7:0/4166752309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa90c005350 con 0x557d38db58d0 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa9027fc640 1 -- 172.21.15.7:0/4166752309 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa9040c2e10 con 0x557d38db58d0 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa923df7740 10 client.16734 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa923df7740 1 -- 172.21.15.7:0/4166752309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557d38dbf130 con 0x557d38db58d0 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.944+0000 7fa9027fc640 1 -- 172.21.15.7:0/4166752309 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa90c005350 con 0x557d38db58d0 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.945+0000 7fa9027fc640 1 -- 172.21.15.7:0/4166752309 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa9040bb0d0 con 0x557d38db58d0 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.945+0000 7fa923df7740 10 client.16734 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.945+0000 7fa923df7740 10 client.16734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.945+0000 7fa923df7740 20 client.16734 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.945+0000 7fa923df7740 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x557d38e34740 0x557d38e36b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.945+0000 7fa923df7740 4 client.16734 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.945+0000 7fa923df7740 1 -- 172.21.15.7:0/4166752309 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557d38cb93d0 con 0x557d38e34740 2024-04-21T08:50:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.945+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x557d38e34740 0x557d38e36b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:04.952 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.945+0000 7fa9027fc640 10 client.16734 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:04.952 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:04.945+0000 7fa91ad9d640 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x557d38e34740 0x557d38e36b60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c593fc96570b2083 server_cookie=e846149f00d4def in_seq=0 out_seq=0 2024-04-21T08:50:05.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:04 smithi016 ceph-mon[31259]: pgmap v165: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.4 MiB/s wr, 2.30k op/s 2024-04-21T08:50:05.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:04 smithi077 ceph-mon[31615]: pgmap v165: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.4 MiB/s wr, 2.30k op/s 2024-04-21T08:50:05.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:04 smithi007 ceph-mon[28477]: pgmap v165: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.4 MiB/s wr, 2.30k op/s 2024-04-21T08:50:06.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:05 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:06.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:05 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:06.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:05 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:07.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:06 smithi077 ceph-mon[31615]: pgmap v166: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.8 MiB/s wr, 1.74k op/s 2024-04-21T08:50:07.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:06 smithi007 ceph-mon[28477]: pgmap v166: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.8 MiB/s wr, 1.74k op/s 2024-04-21T08:50:07.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:06 smithi016 ceph-mon[31259]: pgmap v166: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.8 MiB/s wr, 1.74k op/s 2024-04-21T08:50:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:07 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:50:08.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:07 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:50:08.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:07 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:50:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa9027fc640 1 -- 172.21.15.7:0/4166752309 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557d38cb93d0 con 0x557d38e34740 2024-04-21T08:50:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa9027fc640 10 client.16734 handle_command_reply: tid=0 2024-04-21T08:50:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 1 client.16734 shutdown 2024-04-21T08:50:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 1 -- 172.21.15.7:0/4166752309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db58d0 msgr2=0x557d38dc6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db58d0 0x557d38dc6760 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa904006010 tx=0x7fa9040ce040 comp rx=0 tx=0).stop 2024-04-21T08:50:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 1 -- 172.21.15.7:0/4166752309 shutdown_connections 2024-04-21T08:50:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x557d38e34740 0x557d38e36b60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x557d38dbe0e0 0x557d38db8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557d38db58d0 0x557d38dc6760 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa9027fc640 0 client.16734 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 1 --2- 172.21.15.7:0/4166752309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557d38db4b60 0x557d38dc6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 1 -- 172.21.15.7:0/4166752309 >> 172.21.15.7:0/4166752309 conn(0x557d38d7bc90 msgr2=0x557d38db95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 1 -- 172.21.15.7:0/4166752309 shutdown_connections 2024-04-21T08:50:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 1 -- 172.21.15.7:0/4166752309 wait complete. 2024-04-21T08:50:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.774+0000 7fa923df7740 20 client.16734 trim_cache size 0 max 16384 2024-04-21T08:50:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:50:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:50:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:50:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:50:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 -- 172.21.15.7:0/1025107887 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa9080a3d10 msgr2=0x7fa9080a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa9080a3d10 0x7fa9080a61d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa914009270 tx=0x7fa914009e30 comp rx=0 tx=0).stop 2024-04-21T08:50:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 -- 172.21.15.7:0/1025107887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa91c1011d0 msgr2=0x7fa91c198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa91c1011d0 0x7fa91c198c80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa904000c00 tx=0x7fa9040045e0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 -- 172.21.15.7:0/1025107887 shutdown_connections 2024-04-21T08:50:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa91c106fb0 0x7fa91c19d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa91c1011d0 0x7fa91c198c80 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa9080a3d10 0x7fa9080a61d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 --2- 172.21.15.7:0/1025107887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91c100810 0x7fa91c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 -- 172.21.15.7:0/1025107887 >> 172.21.15.7:0/1025107887 conn(0x7fa91c076190 msgr2=0x7fa91c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 -- 172.21.15.7:0/1025107887 shutdown_connections 2024-04-21T08:50:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:08.775+0000 7fa92112b640 1 -- 172.21.15.7:0/1025107887 wait complete. 2024-04-21T08:50:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:08 smithi077 ceph-mon[31615]: pgmap v167: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.5 MiB/s wr, 1.00k op/s 2024-04-21T08:50:09.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:08 smithi007 ceph-mon[28477]: pgmap v167: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.5 MiB/s wr, 1.00k op/s 2024-04-21T08:50:09.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:08 smithi016 ceph-mon[31259]: pgmap v167: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 8.5 MiB/s wr, 1.00k op/s 2024-04-21T08:50:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:10 smithi077 ceph-mon[31615]: pgmap v168: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 9.8 MiB/s wr, 2.43k op/s 2024-04-21T08:50:11.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:10 smithi007 ceph-mon[28477]: pgmap v168: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 9.8 MiB/s wr, 2.43k op/s 2024-04-21T08:50:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:10 smithi016 ceph-mon[31259]: pgmap v168: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 9.8 MiB/s wr, 2.43k op/s 2024-04-21T08:50:13.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:12 smithi077 ceph-mon[31615]: pgmap v169: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.4 MiB/s wr, 1.78k op/s 2024-04-21T08:50:13.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:12 smithi007 ceph-mon[28477]: pgmap v169: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.4 MiB/s wr, 1.78k op/s 2024-04-21T08:50:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:12 smithi016 ceph-mon[31259]: pgmap v169: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.4 MiB/s wr, 1.78k op/s 2024-04-21T08:50:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:14 smithi077 ceph-mon[31615]: pgmap v170: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.5 MiB/s wr, 1.83k op/s 2024-04-21T08:50:15.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:14 smithi007 ceph-mon[28477]: pgmap v170: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.5 MiB/s wr, 1.83k op/s 2024-04-21T08:50:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:14 smithi016 ceph-mon[31259]: pgmap v170: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.5 MiB/s wr, 1.83k op/s 2024-04-21T08:50:16.689 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:50:16.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:50:16.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.800+0000 7f7eda385640 1 -- 172.21.15.7:0/2457350636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ed4069570 msgr2=0x7f7ed4069950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.800+0000 7f7eda385640 1 --2- 172.21.15.7:0/2457350636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ed4069570 0x7f7ed4069950 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009900 tx=0x7f7ec402bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.800+0000 7f7eda385640 1 -- 172.21.15.7:0/2457350636 shutdown_connections 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.800+0000 7f7eda385640 1 --2- 172.21.15.7:0/2457350636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed4107230 0x7f7ed410ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.800+0000 7f7eda385640 1 --2- 172.21.15.7:0/2457350636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ed4069f20 0x7f7ed4106b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.800+0000 7f7eda385640 1 --2- 172.21.15.7:0/2457350636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ed4069570 0x7f7ed4069950 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.800+0000 7f7eda385640 1 -- 172.21.15.7:0/2457350636 >> 172.21.15.7:0/2457350636 conn(0x7f7ed4077fa0 msgr2=0x7f7ed40783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.800+0000 7f7eda385640 1 -- 172.21.15.7:0/2457350636 shutdown_connections 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.800+0000 7f7eda385640 1 -- 172.21.15.7:0/2457350636 wait complete. 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7eda385640 1 Processor -- start 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7eda385640 1 -- start start 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7eda385640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ed4069570 0x7f7ed41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7eda385640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ed4107230 0x7f7ed4198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7eda385640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed419cfa0 0x7f7ed419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7eda385640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41121a0 con 0x7f7ed4069570 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7eda385640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed4112020 con 0x7f7ed4107230 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7eda385640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed4112320 con 0x7f7ed419cfa0 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7ed3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ed4069570 0x7f7ed41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7ed3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ed4069570 0x7f7ed41986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41724/0 (socket says 172.21.15.7:41724) 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7ed3fff640 1 -- 172.21.15.7:0/3498728802 learned_addr learned my addr 172.21.15.7:0/3498728802 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7ed37fe640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ed4107230 0x7f7ed4198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7ed8922640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed419cfa0 0x7f7ed419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7ed3fff640 1 -- 172.21.15.7:0/3498728802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed419cfa0 msgr2=0x7f7ed419d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed419cfa0 0x7f7ed419d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7ed3fff640 1 -- 172.21.15.7:0/3498728802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ed4107230 msgr2=0x7f7ed4198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ed4107230 0x7f7ed4198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.801+0000 7f7ed3fff640 1 -- 172.21.15.7:0/3498728802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed419da90 con 0x7f7ed4069570 2024-04-21T08:50:16.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.802+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ed4069570 0x7f7ed41986e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4005940 tx=0x7f7ec40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:16.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.802+0000 7f7ed17fa640 1 -- 172.21.15.7:0/3498728802 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4043020 con 0x7f7ed4069570 2024-04-21T08:50:16.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.802+0000 7f7ed17fa640 1 -- 172.21.15.7:0/3498728802 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ec402caf0 con 0x7f7ed4069570 2024-04-21T08:50:16.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.802+0000 7f7ed17fa640 1 -- 172.21.15.7:0/3498728802 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4003af0 con 0x7f7ed4069570 2024-04-21T08:50:16.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.802+0000 7f7eda385640 1 -- 172.21.15.7:0/3498728802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed419dd20 con 0x7f7ed4069570 2024-04-21T08:50:16.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.802+0000 7f7eda385640 1 -- 172.21.15.7:0/3498728802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed41a5540 con 0x7f7ed4069570 2024-04-21T08:50:16.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.805+0000 7f7ed17fa640 1 -- 172.21.15.7:0/3498728802 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f7ec4046020 con 0x7f7ed4069570 2024-04-21T08:50:16.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.805+0000 7f7edcf73740 1 Processor -- start 2024-04-21T08:50:16.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.805+0000 7f7edcf73740 1 -- start start 2024-04-21T08:50:16.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed17fa640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ebc0a3d20 0x7f7ebc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e4c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e59950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e5c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7edcf73740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562f27db11e0 con 0x562f27e4cbd0 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7edcf73740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562f27d4e390 con 0x562f27e4c2b0 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7edcf73740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562f27dad1a0 con 0x562f27e59e90 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed8922640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ebc0a3d20 0x7f7ebc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e59950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e4c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e59950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41728/0 (socket says 172.21.15.7:41728) 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e4c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59156/0 (socket says 172.21.15.7:59156) 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed37fe640 1 -- 172.21.15.7:0/3357429938 learned_addr learned my addr 172.21.15.7:0/3357429938 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed17fa640 1 -- 172.21.15.7:0/3498728802 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f7ec40f2cd0 con 0x7f7ed4069570 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed8922640 1 --2- 172.21.15.7:0/3357429938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e5c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed3fff640 1 -- 172.21.15.7:0/3357429938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 msgr2=0x562f27e5c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/3357429938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e5c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed3fff640 1 -- 172.21.15.7:0/3357429938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 msgr2=0x562f27e59950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/3357429938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e59950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed3fff640 1 -- 172.21.15.7:0/3357429938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f27b8d7d0 con 0x562f27e4c2b0 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed37fe640 1 --2- 172.21.15.7:0/3357429938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e59950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed8922640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ebc0a3d20 0x7f7ebc0a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc009fd0 tx=0x7f7ecc0098b0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/3357429938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e4c690 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40f0140 tx=0x7f7ec40be760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c1a08fc52212326 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.806+0000 7f7eba7fc640 1 -- 172.21.15.7:0/3357429938 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec402cca0 con 0x562f27e4c2b0 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.807+0000 7f7eba7fc640 1 -- 172.21.15.7:0/3357429938 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ec40bf090 con 0x562f27e4c2b0 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.807+0000 7f7eba7fc640 1 -- 172.21.15.7:0/3357429938 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40f02b0 con 0x562f27e4c2b0 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.807+0000 7f7edcf73740 1 -- 172.21.15.7:0/3357429938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 msgr2=0x562f27e4c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.807+0000 7f7edcf73740 1 --2- 172.21.15.7:0/3357429938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e4c690 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40f0140 tx=0x7f7ec40be760 comp rx=0 tx=0).stop 2024-04-21T08:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.807+0000 7f7edcf73740 1 -- 172.21.15.7:0/3357429938 shutdown_connections 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.807+0000 7f7edcf73740 1 --2- 172.21.15.7:0/3357429938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e5c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.807+0000 7f7edcf73740 1 --2- 172.21.15.7:0/3357429938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e59950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.807+0000 7f7edcf73740 1 --2- 172.21.15.7:0/3357429938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e4c690 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.807+0000 7f7edcf73740 1 -- 172.21.15.7:0/3357429938 >> 172.21.15.7:0/3357429938 conn(0x562f27e46cb0 msgr2=0x562f27e47090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.807+0000 7f7edcf73740 1 -- 172.21.15.7:0/3357429938 shutdown_connections 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.807+0000 7f7edcf73740 1 -- 172.21.15.7:0/3357429938 wait complete. 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7edcf73740 1 -- start start 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7edcf73740 1 Processor -- start 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e56e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e57380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e578c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7edcf73740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562f27e5ca10 con 0x562f27e4cbd0 2024-04-21T08:50:16.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7edcf73740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562f27d6a2f0 con 0x562f27e4c2b0 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7edcf73740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562f27e5cb90 con 0x562f27e59e90 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7edcf73740 10 client.0 _ll_register_callbacks cb 0x562f27d77580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7ed37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e57380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7ed3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e56e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7ed37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e57380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41734/0 (socket says 172.21.15.7:41734) 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7ed3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e56e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59160/0 (socket says 172.21.15.7:59160) 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7ed37fe640 1 -- 172.21.15.7:0/27339566 learned_addr learned my addr 172.21.15.7:0/27339566 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.808+0000 7f7ed8922640 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7ed3fff640 1 -- 172.21.15.7:0/27339566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 msgr2=0x562f27e578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7ed3fff640 1 -- 172.21.15.7:0/27339566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 msgr2=0x562f27e57380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e57380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7ed3fff640 1 -- 172.21.15.7:0/27339566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f27ac33f0 con 0x562f27e4c2b0 2024-04-21T08:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7ed37fe640 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e57380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e56e40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40beb50 tx=0x7f7ec40c2e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a78099033992914 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7eb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7eb37fe640 1 -- 172.21.15.7:0/27339566 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40be990 con 0x562f27e4c2b0 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7eb37fe640 1 -- 172.21.15.7:0/27339566 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f7ec40053b0 con 0x562f27e4c2b0 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7edcf73740 1 -- 172.21.15.7:0/27339566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ec40bf090 con 0x562f27e4c2b0 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7eb37fe640 1 -- 172.21.15.7:0/27339566 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7ec40c7600 con 0x562f27e4c2b0 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7edcf73740 10 client.26294 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7edcf73740 1 -- 172.21.15.7:0/27339566 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562f27a830f0 con 0x562f27e4c2b0 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.809+0000 7f7eb37fe640 1 -- 172.21.15.7:0/27339566 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ec40bf090 con 0x562f27e4c2b0 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.810+0000 7f7eb37fe640 1 -- 172.21.15.7:0/27339566 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f7ec40c7170 con 0x562f27e4c2b0 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.810+0000 7f7edcf73740 10 client.26294 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.810+0000 7f7edcf73740 10 client.26294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.810+0000 7f7edcf73740 20 client.26294 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.810+0000 7f7edcf73740 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562f27e699e0 0x562f27e6bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.810+0000 7f7edcf73740 4 client.26294 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.810+0000 7f7edcf73740 1 -- 172.21.15.7:0/27339566 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f27dbec00 con 0x562f27e699e0 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.810+0000 7f7ed37fe640 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562f27e699e0 0x562f27e6bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.810+0000 7f7ed37fe640 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562f27e699e0 0x562f27e6bdd0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9af9856d7b6a04a9 server_cookie=c6ce23f5740061e1 in_seq=0 out_seq=0 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.810+0000 7f7eb37fe640 10 client.26294 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7eb37fe640 1 -- 172.21.15.7:0/27339566 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f27dbec00 con 0x562f27e699e0 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7eb37fe640 10 client.26294 handle_command_reply: tid=0 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 1 client.26294 shutdown 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 1 -- 172.21.15.7:0/27339566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 msgr2=0x562f27e56e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e56e40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40beb50 tx=0x7f7ec40c2e40 comp rx=0 tx=0).stop 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 1 -- 172.21.15.7:0/27339566 shutdown_connections 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562f27e699e0 0x562f27e6bdd0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7eb37fe640 0 client.26294 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4cbd0 0x562f27e57380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 1 --2- 172.21.15.7:0/27339566 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4c2b0 0x562f27e56e40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 1 -- 172.21.15.7:0/27339566 >> 172.21.15.7:0/27339566 conn(0x562f27e46cb0 msgr2=0x562f27e48a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 1 -- 172.21.15.7:0/27339566 shutdown_connections 2024-04-21T08:50:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 1 -- 172.21.15.7:0/27339566 wait complete. 2024-04-21T08:50:16.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.812+0000 7f7edcf73740 20 client.26294 trim_cache size 0 max 16384 2024-04-21T08:50:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.830+0000 7f7edcf73740 1 Processor -- start 2024-04-21T08:50:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.830+0000 7f7edcf73740 1 -- start start 2024-04-21T08:50:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4c2b0 0x562f27e59570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4cbd0 0x562f27e658d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e55d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7edcf73740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41ab0c0 con 0x562f27e4c2b0 2024-04-21T08:50:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7edcf73740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41ad940 con 0x562f27e4cbd0 2024-04-21T08:50:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7edcf73740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41ab730 con 0x562f27e59e90 2024-04-21T08:50:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4c2b0 0x562f27e59570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed3fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4c2b0 0x562f27e59570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41742/0 (socket says 172.21.15.7:41742) 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed3fff640 1 -- 172.21.15.7:0/2479737033 learned_addr learned my addr 172.21.15.7:0/2479737033 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed37fe640 1 --2- 172.21.15.7:0/2479737033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4cbd0 0x562f27e658d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed8922640 1 --2- 172.21.15.7:0/2479737033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e55d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed3fff640 1 -- 172.21.15.7:0/2479737033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 msgr2=0x562f27e55d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/2479737033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e55d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed3fff640 1 -- 172.21.15.7:0/2479737033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4cbd0 msgr2=0x562f27e658d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/2479737033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4cbd0 0x562f27e658d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed3fff640 1 -- 172.21.15.7:0/2479737033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f27e51c60 con 0x562f27e4c2b0 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed37fe640 1 --2- 172.21.15.7:0/2479737033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4cbd0 0x562f27e658d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/2479737033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4c2b0 0x562f27e59570 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4000c00 tx=0x7f7ec40c2d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cad3123aa37688a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7eb9ffb640 1 -- 172.21.15.7:0/2479737033 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40be4d0 con 0x562f27e4c2b0 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7eb9ffb640 1 -- 172.21.15.7:0/2479737033 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ec40be990 con 0x562f27e4c2b0 2024-04-21T08:50:16.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.831+0000 7f7eb9ffb640 1 -- 172.21.15.7:0/2479737033 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40f0030 con 0x562f27e4c2b0 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.832+0000 7f7edcf73740 1 -- 172.21.15.7:0/2479737033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4c2b0 msgr2=0x562f27e59570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.832+0000 7f7edcf73740 1 --2- 172.21.15.7:0/2479737033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4c2b0 0x562f27e59570 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4000c00 tx=0x7f7ec40c2d40 comp rx=0 tx=0).stop 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.832+0000 7f7edcf73740 1 -- 172.21.15.7:0/2479737033 shutdown_connections 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.832+0000 7f7edcf73740 1 --2- 172.21.15.7:0/2479737033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e59e90 0x562f27e55d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.832+0000 7f7edcf73740 1 --2- 172.21.15.7:0/2479737033 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4cbd0 0x562f27e658d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.832+0000 7f7edcf73740 1 --2- 172.21.15.7:0/2479737033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e4c2b0 0x562f27e59570 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.832+0000 7f7edcf73740 1 -- 172.21.15.7:0/2479737033 >> 172.21.15.7:0/2479737033 conn(0x562f27e46cb0 msgr2=0x562f27e45d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.832+0000 7f7edcf73740 1 -- 172.21.15.7:0/2479737033 shutdown_connections 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.832+0000 7f7edcf73740 1 -- 172.21.15.7:0/2479737033 wait complete. 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.832+0000 7f7edcf73740 1 -- start start 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7edcf73740 1 Processor -- start 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e4c2b0 0x562f27e539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4cbd0 0x562f27e53ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7edcf73740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e59e90 0x562f27e54430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7edcf73740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41abe00 con 0x562f27e59e90 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7edcf73740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41ad270 con 0x562f27e4cbd0 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7edcf73740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41ac4d0 con 0x562f27e4c2b0 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7edcf73740 10 client.0 _ll_register_callbacks cb 0x7f7ec402cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7ed8922640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e59e90 0x562f27e54430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7ed37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4cbd0 0x562f27e53ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7ed8922640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e59e90 0x562f27e54430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41758/0 (socket says 172.21.15.7:41758) 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7ed8922640 1 -- 172.21.15.7:0/955288580 learned_addr learned my addr 172.21.15.7:0/955288580 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e4c2b0 0x562f27e539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7ed8922640 1 -- 172.21.15.7:0/955288580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e4c2b0 msgr2=0x562f27e539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7ed8922640 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e4c2b0 0x562f27e539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7ed8922640 1 -- 172.21.15.7:0/955288580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4cbd0 msgr2=0x562f27e53ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:16.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7ed8922640 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4cbd0 0x562f27e53ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.833+0000 7f7ed8922640 1 -- 172.21.15.7:0/955288580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f27eceec0 con 0x562f27e59e90 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7ed8922640 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e59e90 0x562f27e54430 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc011850 tx=0x7f7ecc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d744dad075e35f12 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7eb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7eb37fe640 1 -- 172.21.15.7:0/955288580 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc022da0 con 0x562f27e59e90 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7eb37fe640 1 -- 172.21.15.7:0/955288580 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f7ecc004b20 con 0x562f27e59e90 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7edcf73740 1 -- 172.21.15.7:0/955288580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562f27db8a70 con 0x562f27e59e90 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7eb37fe640 1 -- 172.21.15.7:0/955288580 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7ecc035050 con 0x562f27e59e90 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7edcf73740 10 client.16752 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7edcf73740 1 -- 172.21.15.7:0/955288580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562f27ed00f0 con 0x562f27e59e90 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7eb37fe640 1 -- 172.21.15.7:0/955288580 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562f27db8a70 con 0x562f27e59e90 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7eb37fe640 1 -- 172.21.15.7:0/955288580 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f7ecc029030 con 0x562f27e59e90 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7edcf73740 10 client.16752 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:16.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7edcf73740 10 client.16752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:16.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7edcf73740 20 client.16752 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:16.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7edcf73740 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562f27ed08f0 0x562f27ed2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7edcf73740 4 client.16752 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:50:16.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.834+0000 7f7edcf73740 1 -- 172.21.15.7:0/955288580 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562f27d34590 con 0x562f27ed08f0 2024-04-21T08:50:16.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.835+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562f27ed08f0 0x562f27ed2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.835+0000 7f7ed3fff640 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562f27ed08f0 0x562f27ed2d10 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b5cbf0bde92bb30 server_cookie=c3322f7a9ad0c19f in_seq=0 out_seq=0 2024-04-21T08:50:16.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.835+0000 7f7eb37fe640 10 client.16752 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:16.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7eb37fe640 1 -- 172.21.15.7:0/955288580 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562f27d34590 con 0x562f27ed08f0 2024-04-21T08:50:16.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7eb37fe640 10 client.16752 handle_command_reply: tid=0 2024-04-21T08:50:16.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 1 client.16752 shutdown 2024-04-21T08:50:16.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 1 -- 172.21.15.7:0/955288580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e59e90 msgr2=0x562f27e54430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e59e90 0x562f27e54430 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc011850 tx=0x7f7ecc004ab0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 1 -- 172.21.15.7:0/955288580 shutdown_connections 2024-04-21T08:50:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562f27ed08f0 0x562f27ed2d10 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562f27e59e90 0x562f27e54430 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562f27e4cbd0 0x562f27e53ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 1 --2- 172.21.15.7:0/955288580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562f27e4c2b0 0x562f27e539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7eb37fe640 0 client.16752 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 1 -- 172.21.15.7:0/955288580 >> 172.21.15.7:0/955288580 conn(0x562f27e46cb0 msgr2=0x562f27e48090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 1 -- 172.21.15.7:0/955288580 shutdown_connections 2024-04-21T08:50:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 1 -- 172.21.15.7:0/955288580 wait complete. 2024-04-21T08:50:16.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.836+0000 7f7edcf73740 20 client.16752 trim_cache size 0 max 16384 2024-04-21T08:50:16.844 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:50:16.844 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:50:16.844 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:50:16.844 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:50:16.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 -- 172.21.15.7:0/3498728802 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ebc0a3d20 msgr2=0x7f7ebc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ebc0a3d20 0x7f7ebc0a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc009fd0 tx=0x7f7ecc0098b0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 -- 172.21.15.7:0/3498728802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ed4069570 msgr2=0x7f7ed41986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ed4069570 0x7f7ed41986e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4005940 tx=0x7f7ec40045e0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 -- 172.21.15.7:0/3498728802 shutdown_connections 2024-04-21T08:50:16.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ebc0a3d20 0x7f7ebc0a61e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ed419cfa0 0x7f7ed419d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ed4107230 0x7f7ed4198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 --2- 172.21.15.7:0/3498728802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ed4069570 0x7f7ed41986e0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 -- 172.21.15.7:0/3498728802 >> 172.21.15.7:0/3498728802 conn(0x7f7ed4077fa0 msgr2=0x7f7ed40ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:16.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 -- 172.21.15.7:0/3498728802 shutdown_connections 2024-04-21T08:50:16.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.837+0000 7f7eda385640 1 -- 172.21.15.7:0/3498728802 wait complete. 2024-04-21T08:50:16.848 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:50:16.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:50:16.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.953+0000 7f99899ce640 1 -- 172.21.15.7:0/500227713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9984100d40 msgr2=0x7f9984101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.953+0000 7f99899ce640 1 --2- 172.21.15.7:0/500227713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9984100d40 0x7f9984101120 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9974009900 tx=0x7f997402bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:16.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.953+0000 7f99899ce640 1 -- 172.21.15.7:0/500227713 shutdown_connections 2024-04-21T08:50:16.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.953+0000 7f99899ce640 1 --2- 172.21.15.7:0/500227713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f998410e8f0 0x7f9984110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.953+0000 7f99899ce640 1 --2- 172.21.15.7:0/500227713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9984101660 0x7f998410e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.953+0000 7f99899ce640 1 --2- 172.21.15.7:0/500227713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9984100d40 0x7f9984101120 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.953+0000 7f99899ce640 1 -- 172.21.15.7:0/500227713 >> 172.21.15.7:0/500227713 conn(0x7f9984077f70 msgr2=0x7f9984078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.953+0000 7f99899ce640 1 -- 172.21.15.7:0/500227713 shutdown_connections 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.953+0000 7f99899ce640 1 -- 172.21.15.7:0/500227713 wait complete. 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f99899ce640 1 Processor -- start 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f99899ce640 1 -- start start 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f99899ce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9984100d40 0x7f99841a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f99899ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9984101660 0x7f99841a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f99899ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998410e8f0 0x7f998419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f99899ce640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9984114410 con 0x7f9984101660 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f99899ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9984114290 con 0x7f998410e8f0 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f99899ce640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9984114590 con 0x7f9984100d40 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f997bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9984101660 0x7f99841a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f997bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9984101660 0x7f99841a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41784/0 (socket says 172.21.15.7:41784) 2024-04-21T08:50:16.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f997bfff640 1 -- 172.21.15.7:0/970236790 learned_addr learned my addr 172.21.15.7:0/970236790 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f99837fe640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998410e8f0 0x7f998419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f9982ffd640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9984100d40 0x7f99841a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f997bfff640 1 -- 172.21.15.7:0/970236790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9984100d40 msgr2=0x7f99841a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f997bfff640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9984100d40 0x7f99841a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f997bfff640 1 -- 172.21.15.7:0/970236790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998410e8f0 msgr2=0x7f998419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f997bfff640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998410e8f0 0x7f998419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f997bfff640 1 -- 172.21.15.7:0/970236790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f998419b030 con 0x7f9984101660 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.954+0000 7f99837fe640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998410e8f0 0x7f998419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.955+0000 7f997bfff640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9984101660 0x7f99841a0bf0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f996c00e9f0 tx=0x7f996c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.955+0000 7f99889cc640 1 -- 172.21.15.7:0/970236790 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c00cd80 con 0x7f9984101660 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.955+0000 7f99889cc640 1 -- 172.21.15.7:0/970236790 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f996c0108e0 con 0x7f9984101660 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.955+0000 7f99899ce640 1 -- 172.21.15.7:0/970236790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f998419b320 con 0x7f9984101660 2024-04-21T08:50:16.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.955+0000 7f99889cc640 1 -- 172.21.15.7:0/970236790 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c018980 con 0x7f9984101660 2024-04-21T08:50:16.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.955+0000 7f99899ce640 1 -- 172.21.15.7:0/970236790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99841a7630 con 0x7f9984101660 2024-04-21T08:50:16.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.956+0000 7f99899ce640 1 -- 172.21.15.7:0/970236790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f998406ae60 con 0x7f9984101660 2024-04-21T08:50:16.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.958+0000 7f99889cc640 1 -- 172.21.15.7:0/970236790 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f996c010430 con 0x7f9984101660 2024-04-21T08:50:16.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.958+0000 7f99889cc640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f99700a3da0 0x7f99700a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:16.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.958+0000 7f99889cc640 1 -- 172.21.15.7:0/970236790 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f996c0d0f10 con 0x7f9984101660 2024-04-21T08:50:16.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.958+0000 7f9982ffd640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f99700a3da0 0x7f99700a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:16.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.958+0000 7f9982ffd640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f99700a3da0 0x7f99700a6260 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9974000c00 tx=0x7f9974004040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:16.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:16.963+0000 7f99889cc640 1 -- 172.21.15.7:0/970236790 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f996c09d1c0 con 0x7f9984101660 2024-04-21T08:50:17.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:16 smithi077 ceph-mon[31615]: pgmap v171: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 1.70k op/s 2024-04-21T08:50:17.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:16 smithi007 ceph-mon[28477]: pgmap v171: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 1.70k op/s 2024-04-21T08:50:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.201+0000 7f99899ce640 1 -- 172.21.15.7:0/970236790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f998419bcd0 con 0x7f9984101660 2024-04-21T08:50:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.202+0000 7f99889cc640 1 -- 172.21.15.7:0/970236790 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7f998419bcd0 con 0x7f9984101660 2024-04-21T08:50:17.202 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:50:17.202 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:50:17.203 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:50:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.203+0000 7f99899ce640 1 -- 172.21.15.7:0/970236790 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f99700a3da0 msgr2=0x7f99700a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.203+0000 7f99899ce640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f99700a3da0 0x7f99700a6260 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9974000c00 tx=0x7f9974004040 comp rx=0 tx=0).stop 2024-04-21T08:50:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.203+0000 7f99899ce640 1 -- 172.21.15.7:0/970236790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9984101660 msgr2=0x7f99841a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.203+0000 7f99899ce640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9984101660 0x7f99841a0bf0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f996c00e9f0 tx=0x7f996c00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.204+0000 7f99899ce640 1 -- 172.21.15.7:0/970236790 shutdown_connections 2024-04-21T08:50:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.204+0000 7f99899ce640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998410e8f0 0x7f998419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.204+0000 7f99899ce640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9984101660 0x7f99841a0bf0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.204+0000 7f99899ce640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f99700a3da0 0x7f99700a6260 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.204+0000 7f99899ce640 1 --2- 172.21.15.7:0/970236790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9984100d40 0x7f99841a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.204+0000 7f99899ce640 1 -- 172.21.15.7:0/970236790 >> 172.21.15.7:0/970236790 conn(0x7f9984077f70 msgr2=0x7f998410ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.204+0000 7f99899ce640 1 -- 172.21.15.7:0/970236790 shutdown_connections 2024-04-21T08:50:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.204+0000 7f99899ce640 1 -- 172.21.15.7:0/970236790 wait complete. 2024-04-21T08:50:17.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:50:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:16 smithi016 ceph-mon[31259]: pgmap v171: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 1.70k op/s 2024-04-21T08:50:17.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.323+0000 7f412ecac640 1 -- 172.21.15.7:0/882245464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412810c6d0 msgr2=0x7f412810eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.323+0000 7f412ecac640 1 --2- 172.21.15.7:0/882245464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412810c6d0 0x7f412810eb60 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4120009ca0 tx=0x7f412002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.323+0000 7f412ecac640 1 -- 172.21.15.7:0/882245464 shutdown_connections 2024-04-21T08:50:17.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.323+0000 7f412ecac640 1 --2- 172.21.15.7:0/882245464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412810c6d0 0x7f412810eb60 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.323+0000 7f412ecac640 1 --2- 172.21.15.7:0/882245464 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4128103b70 0x7f412810bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.323+0000 7f412ecac640 1 --2- 172.21.15.7:0/882245464 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281031c0 0x7f41281035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.323+0000 7f412ecac640 1 -- 172.21.15.7:0/882245464 >> 172.21.15.7:0/882245464 conn(0x7f4128077f70 msgr2=0x7f4128078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.323+0000 7f412ecac640 1 -- 172.21.15.7:0/882245464 shutdown_connections 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.323+0000 7f412ecac640 1 -- 172.21.15.7:0/882245464 wait complete. 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f412ecac640 1 Processor -- start 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f412ecac640 1 -- start start 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f412ecac640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281031c0 0x7f412819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f412ecac640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412810c6d0 0x7f412819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f412ecac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4128198720 0x7f4128198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f412ecac640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f41281123a0 con 0x7f412810c6d0 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f412ecac640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4128112220 con 0x7f41281031c0 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f412ecac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4128112520 con 0x7f4128198720 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f4127fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412810c6d0 0x7f412819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f4127fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412810c6d0 0x7f412819eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41806/0 (socket says 172.21.15.7:41806) 2024-04-21T08:50:17.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f4127fff640 1 -- 172.21.15.7:0/1223240938 learned_addr learned my addr 172.21.15.7:0/1223240938 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f412ca48640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281031c0 0x7f412819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f412d249640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4128198720 0x7f4128198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f4127fff640 1 -- 172.21.15.7:0/1223240938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4128198720 msgr2=0x7f4128198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f4127fff640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4128198720 0x7f4128198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f4127fff640 1 -- 172.21.15.7:0/1223240938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281031c0 msgr2=0x7f412819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f4127fff640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281031c0 0x7f412819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.324+0000 7f4127fff640 1 -- 172.21.15.7:0/1223240938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4128199320 con 0x7f412810c6d0 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.325+0000 7f4127fff640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412810c6d0 0x7f412819eba0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f411c00d750 tx=0x7f411c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.325+0000 7f4125ffb640 1 -- 172.21.15.7:0/1223240938 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c017070 con 0x7f412810c6d0 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.325+0000 7f412ecac640 1 -- 172.21.15.7:0/1223240938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41281a5400 con 0x7f412810c6d0 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.325+0000 7f4125ffb640 1 -- 172.21.15.7:0/1223240938 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f411c0054d0 con 0x7f412810c6d0 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.325+0000 7f4125ffb640 1 -- 172.21.15.7:0/1223240938 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c004c30 con 0x7f412810c6d0 2024-04-21T08:50:17.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.325+0000 7f412ecac640 1 -- 172.21.15.7:0/1223240938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41281a57b0 con 0x7f412810c6d0 2024-04-21T08:50:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.327+0000 7f4125ffb640 1 -- 172.21.15.7:0/1223240938 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f411c004dd0 con 0x7f412810c6d0 2024-04-21T08:50:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f4125ffb640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41100a3d20 0x7f41100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f4131978740 1 Processor -- start 2024-04-21T08:50:17.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f4131978740 1 -- start start 2024-04-21T08:50:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f4125ffb640 1 -- 172.21.15.7:0/1223240938 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f411c0d04e0 con 0x7f412810c6d0 2024-04-21T08:50:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f412ca48640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41100a3d20 0x7f41100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f4131978740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b024d0 0x561c99b028b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f4131978740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b02e80 0x561c99b0b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f4131978740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b0ba00 0x561c99b0de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f4131978740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561c99a671f0 con 0x561c99b024d0 2024-04-21T08:50:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f4131978740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561c99a044b0 con 0x561c99b0ba00 2024-04-21T08:50:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f4131978740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561c99a631b0 con 0x561c99b02e80 2024-04-21T08:50:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f412ca48640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b024d0 0x561c99b028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f412ca48640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b024d0 0x561c99b028b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41816/0 (socket says 172.21.15.7:41816) 2024-04-21T08:50:17.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f412ca48640 1 -- 172.21.15.7:0/2285277435 learned_addr learned my addr 172.21.15.7:0/2285277435 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f412d249640 1 --2- 172.21.15.7:0/2285277435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b0ba00 0x561c99b0de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f4127fff640 1 --2- 172.21.15.7:0/2285277435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b02e80 0x561c99b0b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f412ca48640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41100a3d20 0x7f41100a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4118011630 tx=0x7f4118011a30 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f412ca48640 1 -- 172.21.15.7:0/2285277435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b02e80 msgr2=0x561c99b0b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.328+0000 7f412ca48640 1 --2- 172.21.15.7:0/2285277435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b02e80 0x561c99b0b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.329+0000 7f412ca48640 1 -- 172.21.15.7:0/2285277435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b0ba00 msgr2=0x561c99b0de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.329+0000 7f412ca48640 1 --2- 172.21.15.7:0/2285277435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b0ba00 0x561c99b0de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.329+0000 7f412ca48640 1 -- 172.21.15.7:0/2285277435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c999f61b0 con 0x561c99b024d0 2024-04-21T08:50:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.329+0000 7f412d249640 1 --2- 172.21.15.7:0/2285277435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b0ba00 0x561c99b0de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.329+0000 7f412ca48640 1 --2- 172.21.15.7:0/2285277435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b024d0 0x561c99b028b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f4118011f30 tx=0x7f411801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e00f3ba49128ff server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.329+0000 7f410effd640 1 -- 172.21.15.7:0/2285277435 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411801a040 con 0x561c99b024d0 2024-04-21T08:50:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.329+0000 7f410effd640 1 -- 172.21.15.7:0/2285277435 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4118003aa0 con 0x561c99b024d0 2024-04-21T08:50:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.329+0000 7f410effd640 1 -- 172.21.15.7:0/2285277435 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4118020950 con 0x561c99b024d0 2024-04-21T08:50:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 -- 172.21.15.7:0/2285277435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b024d0 msgr2=0x561c99b028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 --2- 172.21.15.7:0/2285277435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b024d0 0x561c99b028b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f4118011f30 tx=0x7f411801d000 comp rx=0 tx=0).stop 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 -- 172.21.15.7:0/2285277435 shutdown_connections 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 --2- 172.21.15.7:0/2285277435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b0ba00 0x561c99b0de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 --2- 172.21.15.7:0/2285277435 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b02e80 0x561c99b0b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 --2- 172.21.15.7:0/2285277435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b024d0 0x561c99b028b0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 -- 172.21.15.7:0/2285277435 >> 172.21.15.7:0/2285277435 conn(0x561c99afcba0 msgr2=0x561c99afcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 -- 172.21.15.7:0/2285277435 shutdown_connections 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 -- 172.21.15.7:0/2285277435 wait complete. 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 -- start start 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 Processor -- start 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b024d0 0x561c99b0b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.330+0000 7f4131978740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b02e80 0x561c99b08620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4131978740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 0x561c99b08b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4131978740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561c99b0e910 con 0x561c99b0ba00 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4131978740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561c99a20500 con 0x561c99b024d0 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4131978740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561c99b0ea90 con 0x561c99b02e80 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4131978740 10 client.0 _ll_register_callbacks cb 0x561c999eeb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f412d249640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 0x561c99b08b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f412ca48640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b024d0 0x561c99b0b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f412d249640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 0x561c99b08b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41828/0 (socket says 172.21.15.7:41828) 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f412d249640 1 -- 172.21.15.7:0/2671758007 learned_addr learned my addr 172.21.15.7:0/2671758007 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4127fff640 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b02e80 0x561c99b08620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f412d249640 1 -- 172.21.15.7:0/2671758007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b02e80 msgr2=0x561c99b08620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f412d249640 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b02e80 0x561c99b08620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f412d249640 1 -- 172.21.15.7:0/2671758007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b024d0 msgr2=0x561c99b0b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f412d249640 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b024d0 0x561c99b0b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f412d249640 1 -- 172.21.15.7:0/2671758007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c997793f0 con 0x561c99b0ba00 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f412d249640 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 0x561c99b08b60 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f4120009ca0 tx=0x7f4120005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1292d0527fd454 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4107fff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4107fff640 1 -- 172.21.15.7:0/2671758007 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120045020 con 0x561c99b0ba00 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4107fff640 1 -- 172.21.15.7:0/2671758007 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f4120003970 con 0x561c99b0ba00 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4131978740 1 -- 172.21.15.7:0/2671758007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4118003aa0 con 0x561c99b0ba00 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4107fff640 1 -- 172.21.15.7:0/2671758007 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4120050050 con 0x561c99b0ba00 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4131978740 10 client.16770 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.331+0000 7f4131978740 1 -- 172.21.15.7:0/2671758007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561c998437d0 con 0x561c99b0ba00 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.332+0000 7f4107fff640 1 -- 172.21.15.7:0/2671758007 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4118003aa0 con 0x561c99b0ba00 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.332+0000 7f4107fff640 1 -- 172.21.15.7:0/2671758007 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f4120040480 con 0x561c99b0ba00 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.332+0000 7f4131978740 10 client.16770 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:17.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.332+0000 7f4131978740 10 client.16770 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.332+0000 7f4131978740 20 client.16770 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.332+0000 7f4131978740 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561c99b1bb70 0x561c99b1dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.332+0000 7f4131978740 4 client.16770 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.332+0000 7f4131978740 1 -- 172.21.15.7:0/2671758007 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c99a06930 con 0x561c99b1bb70 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.332+0000 7f412ca48640 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561c99b1bb70 0x561c99b1dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.332+0000 7f412ca48640 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561c99b1bb70 0x561c99b1dfa0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50fc1603e823c2d3 server_cookie=c5162a044feb8bce in_seq=0 out_seq=0 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.332+0000 7f4107fff640 10 client.16770 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.335+0000 7f4107fff640 1 -- 172.21.15.7:0/2671758007 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561c99a06930 con 0x561c99b1bb70 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.335+0000 7f4107fff640 10 client.16770 handle_command_reply: tid=0 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 1 client.16770 shutdown 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 1 -- 172.21.15.7:0/2671758007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 msgr2=0x561c99b08b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 0x561c99b08b60 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f4120009ca0 tx=0x7f4120005ef0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 1 -- 172.21.15.7:0/2671758007 shutdown_connections 2024-04-21T08:50:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561c99b1bb70 0x561c99b1dfa0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 0x561c99b08b60 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4107fff640 0 client.16770 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b02e80 0x561c99b08620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 1 --2- 172.21.15.7:0/2671758007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b024d0 0x561c99b0b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 1 -- 172.21.15.7:0/2671758007 >> 172.21.15.7:0/2671758007 conn(0x561c99afcba0 msgr2=0x561c99afe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 1 -- 172.21.15.7:0/2671758007 shutdown_connections 2024-04-21T08:50:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 1 -- 172.21.15.7:0/2671758007 wait complete. 2024-04-21T08:50:17.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.336+0000 7f4131978740 20 client.16770 trim_cache size 0 max 16384 2024-04-21T08:50:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.350+0000 7f4131978740 1 Processor -- start 2024-04-21T08:50:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.350+0000 7f4131978740 1 -- start start 2024-04-21T08:50:17.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.350+0000 7f4131978740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b024d0 0x561c99b177d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.350+0000 7f4131978740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b02e80 0x561c99b07780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.350+0000 7f4131978740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b0ba00 0x561c99afba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.350+0000 7f4131978740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f41281ad4d0 con 0x561c99b02e80 2024-04-21T08:50:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.350+0000 7f4131978740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41281afd50 con 0x561c99b024d0 2024-04-21T08:50:17.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.350+0000 7f4131978740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41281adb40 con 0x561c99b0ba00 2024-04-21T08:50:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f4127fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b02e80 0x561c99b07780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f4127fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b02e80 0x561c99b07780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41844/0 (socket says 172.21.15.7:41844) 2024-04-21T08:50:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f412ca48640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b024d0 0x561c99b177d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f4127fff640 1 -- 172.21.15.7:0/3043322754 learned_addr learned my addr 172.21.15.7:0/3043322754 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f412d249640 1 --2- 172.21.15.7:0/3043322754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b0ba00 0x561c99afba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f4127fff640 1 -- 172.21.15.7:0/3043322754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b0ba00 msgr2=0x561c99afba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f4127fff640 1 --2- 172.21.15.7:0/3043322754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b0ba00 0x561c99afba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f4127fff640 1 -- 172.21.15.7:0/3043322754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b024d0 msgr2=0x561c99b177d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f4127fff640 1 --2- 172.21.15.7:0/3043322754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b024d0 0x561c99b177d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f4127fff640 1 -- 172.21.15.7:0/3043322754 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c99b02c20 con 0x561c99b02e80 2024-04-21T08:50:17.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f412d249640 1 --2- 172.21.15.7:0/3043322754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b0ba00 0x561c99afba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f4127fff640 1 --2- 172.21.15.7:0/3043322754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b02e80 0x561c99b07780 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f411c09be30 tx=0x7f411c0cdf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6efbacd6de33b53c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f410e7fc640 1 -- 172.21.15.7:0/3043322754 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c0b4070 con 0x561c99b02e80 2024-04-21T08:50:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f410e7fc640 1 -- 172.21.15.7:0/3043322754 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f411c005680 con 0x561c99b02e80 2024-04-21T08:50:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.351+0000 7f410e7fc640 1 -- 172.21.15.7:0/3043322754 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c020040 con 0x561c99b02e80 2024-04-21T08:50:17.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.352+0000 7f4131978740 1 -- 172.21.15.7:0/3043322754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b02e80 msgr2=0x561c99b07780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.352+0000 7f4131978740 1 --2- 172.21.15.7:0/3043322754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b02e80 0x561c99b07780 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f411c09be30 tx=0x7f411c0cdf90 comp rx=0 tx=0).stop 2024-04-21T08:50:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.352+0000 7f4131978740 1 -- 172.21.15.7:0/3043322754 shutdown_connections 2024-04-21T08:50:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.352+0000 7f4131978740 1 --2- 172.21.15.7:0/3043322754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b0ba00 0x561c99afba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.352+0000 7f4131978740 1 --2- 172.21.15.7:0/3043322754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b02e80 0x561c99b07780 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.352+0000 7f4131978740 1 --2- 172.21.15.7:0/3043322754 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b024d0 0x561c99b177d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.352+0000 7f4131978740 1 -- 172.21.15.7:0/3043322754 >> 172.21.15.7:0/3043322754 conn(0x561c99afcba0 msgr2=0x561c99afb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.352+0000 7f4131978740 1 -- 172.21.15.7:0/3043322754 shutdown_connections 2024-04-21T08:50:17.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.352+0000 7f4131978740 1 -- 172.21.15.7:0/3043322754 wait complete. 2024-04-21T08:50:17.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.352+0000 7f4131978740 1 -- start start 2024-04-21T08:50:17.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.352+0000 7f4131978740 1 Processor -- start 2024-04-21T08:50:17.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4131978740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b024d0 0x561c99b13e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4131978740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b02e80 0x561c99b0f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4131978740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 0x561c99b0f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4131978740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f41281ae210 con 0x561c99b0ba00 2024-04-21T08:50:17.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4131978740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41281af680 con 0x561c99b02e80 2024-04-21T08:50:17.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4131978740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41281ae8e0 con 0x561c99b024d0 2024-04-21T08:50:17.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4131978740 10 client.0 _ll_register_callbacks cb 0x7f4120009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:17.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4127fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b02e80 0x561c99b0f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f412d249640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 0x561c99b0f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4127fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b02e80 0x561c99b0f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59238/0 (socket says 172.21.15.7:59238) 2024-04-21T08:50:17.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4127fff640 1 -- 172.21.15.7:0/3748574475 learned_addr learned my addr 172.21.15.7:0/3748574475 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:17.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f412ca48640 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b024d0 0x561c99b13e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4127fff640 1 -- 172.21.15.7:0/3748574475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b024d0 msgr2=0x561c99b13e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4127fff640 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b024d0 0x561c99b13e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4127fff640 1 -- 172.21.15.7:0/3748574475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 msgr2=0x561c99b0f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:17.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4127fff640 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 0x561c99b0f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4127fff640 1 -- 172.21.15.7:0/3748574475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c99b0fdd0 con 0x561c99b02e80 2024-04-21T08:50:17.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f412d249640 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 0x561c99b0f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:17.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f412ca48640 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b024d0 0x561c99b13e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:17.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4127fff640 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b02e80 0x561c99b0f2a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f411c09c0c0 tx=0x7f411c005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5364034a48c33be server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:17.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.353+0000 7f4107fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:50:17.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4107fff640 1 -- 172.21.15.7:0/3748574475 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c07f060 con 0x561c99b02e80 2024-04-21T08:50:17.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4107fff640 1 -- 172.21.15.7:0/3748574475 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f411c0cdb60 con 0x561c99b02e80 2024-04-21T08:50:17.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4131978740 1 -- 172.21.15.7:0/3748574475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f411c005680 con 0x561c99b02e80 2024-04-21T08:50:17.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4107fff640 1 -- 172.21.15.7:0/3748574475 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f411c0bb070 con 0x561c99b02e80 2024-04-21T08:50:17.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4131978740 10 client.26324 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:17.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4131978740 1 -- 172.21.15.7:0/3748574475 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561c99b81e30 con 0x561c99b02e80 2024-04-21T08:50:17.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4107fff640 1 -- 172.21.15.7:0/3748574475 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f411c005680 con 0x561c99b02e80 2024-04-21T08:50:17.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4107fff640 1 -- 172.21.15.7:0/3748574475 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f411c0bc040 con 0x561c99b02e80 2024-04-21T08:50:17.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4131978740 10 client.26324 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:17.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4131978740 10 client.26324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:17.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4131978740 20 client.26324 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:17.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4131978740 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561c99b1bb70 0x561c99b84540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:17.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4131978740 4 client.26324 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:50:17.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f4131978740 1 -- 172.21.15.7:0/3748574475 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561c99a74c10 con 0x561c99b1bb70 2024-04-21T08:50:17.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.354+0000 7f412ca48640 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561c99b1bb70 0x561c99b84540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:17.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.355+0000 7f412ca48640 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561c99b1bb70 0x561c99b84540 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d91cbbb47674db0 server_cookie=b5621cf7a331cd49 in_seq=0 out_seq=0 2024-04-21T08:50:17.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.355+0000 7f4107fff640 10 client.26324 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:17.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4107fff640 1 -- 172.21.15.7:0/3748574475 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561c99a74c10 con 0x561c99b1bb70 2024-04-21T08:50:17.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4107fff640 10 client.26324 handle_command_reply: tid=0 2024-04-21T08:50:17.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 1 client.26324 shutdown 2024-04-21T08:50:17.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 1 -- 172.21.15.7:0/3748574475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b02e80 msgr2=0x561c99b0f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b02e80 0x561c99b0f2a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f411c09c0c0 tx=0x7f411c005b20 comp rx=0 tx=0).stop 2024-04-21T08:50:17.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 1 -- 172.21.15.7:0/3748574475 shutdown_connections 2024-04-21T08:50:17.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561c99b1bb70 0x561c99b84540 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561c99b0ba00 0x561c99b0f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561c99b02e80 0x561c99b0f2a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 1 --2- 172.21.15.7:0/3748574475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c99b024d0 0x561c99b13e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 1 -- 172.21.15.7:0/3748574475 >> 172.21.15.7:0/3748574475 conn(0x561c99afcba0 msgr2=0x561c99afe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4107fff640 0 client.26324 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:17.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 1 -- 172.21.15.7:0/3748574475 shutdown_connections 2024-04-21T08:50:17.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 1 -- 172.21.15.7:0/3748574475 wait complete. 2024-04-21T08:50:17.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.357+0000 7f4131978740 20 client.26324 trim_cache size 0 max 16384 2024-04-21T08:50:17.378 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:50:17.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 -- 172.21.15.7:0/1223240938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41100a3d20 msgr2=0x7f41100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41100a3d20 0x7f41100a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4118011630 tx=0x7f4118011a30 comp rx=0 tx=0).stop 2024-04-21T08:50:17.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 -- 172.21.15.7:0/1223240938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412810c6d0 msgr2=0x7f412819eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:17.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412810c6d0 0x7f412819eba0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f411c00d750 tx=0x7f411c00dc10 comp rx=0 tx=0).stop 2024-04-21T08:50:17.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 -- 172.21.15.7:0/1223240938 shutdown_connections 2024-04-21T08:50:17.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f41100a3d20 0x7f41100a61e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4128198720 0x7f4128198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412810c6d0 0x7f412819eba0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 --2- 172.21.15.7:0/1223240938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41281031c0 0x7f412819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:17.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 -- 172.21.15.7:0/1223240938 >> 172.21.15.7:0/1223240938 conn(0x7f4128077f70 msgr2=0x7f41280fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:17.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 -- 172.21.15.7:0/1223240938 shutdown_connections 2024-04-21T08:50:17.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:17.358+0000 7f412ecac640 1 -- 172.21.15.7:0/1223240938 wait complete. 2024-04-21T08:50:18.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:17 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/970236790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:18.167 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:17 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/970236790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:18.306 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:17 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/970236790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:18.384 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:50:18.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:50:18.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:50:18.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.611+0000 7fa78f26d640 1 -- 172.21.15.7:0/3889039207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa788107260 msgr2=0x7fa78810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.611+0000 7fa78f26d640 1 --2- 172.21.15.7:0/3889039207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa788107260 0x7fa78810ee30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa784009900 tx=0x7fa78402bc80 comp rx=0 tx=0).stop 2024-04-21T08:50:18.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.618+0000 7fa78f26d640 1 -- 172.21.15.7:0/3889039207 shutdown_connections 2024-04-21T08:50:18.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.618+0000 7fa78f26d640 1 --2- 172.21.15.7:0/3889039207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa788107260 0x7fa78810ee30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.618+0000 7fa78f26d640 1 --2- 172.21.15.7:0/3889039207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa788102b40 0x7fa788106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.618+0000 7fa78f26d640 1 --2- 172.21.15.7:0/3889039207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa788102190 0x7fa788102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.618+0000 7fa78f26d640 1 -- 172.21.15.7:0/3889039207 >> 172.21.15.7:0/3889039207 conn(0x7fa7880fda10 msgr2=0x7fa7880ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.622+0000 7fa78f26d640 1 -- 172.21.15.7:0/3889039207 shutdown_connections 2024-04-21T08:50:18.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.624+0000 7fa78f26d640 1 -- 172.21.15.7:0/3889039207 wait complete. 2024-04-21T08:50:18.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.625+0000 7fa78f26d640 1 Processor -- start 2024-04-21T08:50:18.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.628+0000 7fa78f26d640 1 -- start start 2024-04-21T08:50:18.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.628+0000 7fa78f26d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa788102190 0x7fa7881165a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.628+0000 7fa78f26d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa788107260 0x7fa788116ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.628+0000 7fa78f26d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa788111700 0x7fa788111bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.628+0000 7fa78f26d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa788110110 con 0x7fa788111700 2024-04-21T08:50:18.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.628+0000 7fa78f26d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa78810ff90 con 0x7fa788107260 2024-04-21T08:50:18.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.628+0000 7fa78f26d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa788110290 con 0x7fa788102190 2024-04-21T08:50:18.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.630+0000 7fa78d80a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa788111700 0x7fa788111bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.630+0000 7fa78d80a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa788111700 0x7fa788111bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41858/0 (socket says 172.21.15.7:41858) 2024-04-21T08:50:18.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.630+0000 7fa78d80a640 1 -- 172.21.15.7:0/4272437925 learned_addr learned my addr 172.21.15.7:0/4272437925 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:18.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.630+0000 7fa78c808640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa788107260 0x7fa788116ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.634+0000 7fa78d009640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa788102190 0x7fa7881165a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.634+0000 7fa78d80a640 1 -- 172.21.15.7:0/4272437925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa788102190 msgr2=0x7fa7881165a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.635+0000 7fa78d80a640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa788102190 0x7fa7881165a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.635+0000 7fa78d80a640 1 -- 172.21.15.7:0/4272437925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa788107260 msgr2=0x7fa788116ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.635+0000 7fa78d80a640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa788107260 0x7fa788116ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.635+0000 7fa78d80a640 1 -- 172.21.15.7:0/4272437925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7881120f0 con 0x7fa788111700 2024-04-21T08:50:18.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.642+0000 7fa78d80a640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa788111700 0x7fa788111bb0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa784009850 tx=0x7fa7840045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.648+0000 7fa77e7fc640 1 -- 172.21.15.7:0/4272437925 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa784042020 con 0x7fa788111700 2024-04-21T08:50:18.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.648+0000 7fa77e7fc640 1 -- 172.21.15.7:0/4272437925 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7840035f0 con 0x7fa788111700 2024-04-21T08:50:18.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.649+0000 7fa77e7fc640 1 -- 172.21.15.7:0/4272437925 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78403e6c0 con 0x7fa788111700 2024-04-21T08:50:18.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.650+0000 7fa78f26d640 1 -- 172.21.15.7:0/4272437925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa788112380 con 0x7fa788111700 2024-04-21T08:50:18.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.650+0000 7fa78f26d640 1 -- 172.21.15.7:0/4272437925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa78806bb50 con 0x7fa788111700 2024-04-21T08:50:18.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.655+0000 7fa791e69740 1 Processor -- start 2024-04-21T08:50:18.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.659+0000 7fa791e69740 1 -- start start 2024-04-21T08:50:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.659+0000 7fa791e69740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2205428e0 0x55a220542cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.659+0000 7fa791e69740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a220543290 0x55a22054b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.659+0000 7fa791e69740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a22054be10 0x55a22054e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.659+0000 7fa791e69740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a2204a6f00 con 0x55a220543290 2024-04-21T08:50:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.659+0000 7fa791e69740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a220444590 con 0x55a2205428e0 2024-04-21T08:50:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.659+0000 7fa791e69740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a2204a2ec0 con 0x55a22054be10 2024-04-21T08:50:18.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.659+0000 7fa77e7fc640 1 -- 172.21.15.7:0/4272437925 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa784045020 con 0x7fa788111700 2024-04-21T08:50:18.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.660+0000 7fa77e7fc640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa7740a3d00 0x7fa7740a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.660+0000 7fa77e7fc640 1 -- 172.21.15.7:0/4272437925 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa7840f24b0 con 0x7fa788111700 2024-04-21T08:50:18.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.660+0000 7fa78c808640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2205428e0 0x55a220542cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.660+0000 7fa78c808640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2205428e0 0x55a220542cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59268/0 (socket says 172.21.15.7:59268) 2024-04-21T08:50:18.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.660+0000 7fa78c808640 1 -- 172.21.15.7:0/3633522672 learned_addr learned my addr 172.21.15.7:0/3633522672 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.660+0000 7fa78c808640 1 --2- 172.21.15.7:0/3633522672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a22054be10 0x55a22054e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.662+0000 7fa78d80a640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa7740a3d00 0x7fa7740a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.662+0000 7fa78c808640 1 -- 172.21.15.7:0/3633522672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a22054be10 msgr2=0x55a22054e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.662+0000 7fa78c808640 1 --2- 172.21.15.7:0/3633522672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a22054be10 0x55a22054e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.662+0000 7fa78c808640 1 -- 172.21.15.7:0/3633522672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a220543290 msgr2=0x55a22054b840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:50:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.662+0000 7fa78c808640 1 --2- 172.21.15.7:0/3633522672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a220543290 0x55a22054b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.662+0000 7fa78c808640 1 -- 172.21.15.7:0/3633522672 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2202817e0 con 0x55a2205428e0 2024-04-21T08:50:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.667+0000 7fa78c808640 1 --2- 172.21.15.7:0/3633522672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2205428e0 0x55a220542cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa77800afb0 tx=0x7fa77800a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d2cff54d5aba355 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.669+0000 7fa78d80a640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa7740a3d00 0x7fa7740a61c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa7840bde10 tx=0x7fa784009260 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.670+0000 7fa7737fe640 1 -- 172.21.15.7:0/3633522672 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa778016e50 con 0x55a2205428e0 2024-04-21T08:50:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.670+0000 7fa7737fe640 1 -- 172.21.15.7:0/3633522672 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7780188e0 con 0x55a2205428e0 2024-04-21T08:50:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.671+0000 7fa791e69740 1 -- 172.21.15.7:0/3633522672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2205428e0 msgr2=0x55a220542cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.671+0000 7fa791e69740 1 --2- 172.21.15.7:0/3633522672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2205428e0 0x55a220542cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa77800afb0 tx=0x7fa77800a250 comp rx=0 tx=0).stop 2024-04-21T08:50:18.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.678+0000 7fa791e69740 1 -- 172.21.15.7:0/3633522672 shutdown_connections 2024-04-21T08:50:18.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.678+0000 7fa791e69740 1 --2- 172.21.15.7:0/3633522672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a22054be10 0x55a22054e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.678+0000 7fa791e69740 1 --2- 172.21.15.7:0/3633522672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a220543290 0x55a22054b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.678+0000 7fa791e69740 1 --2- 172.21.15.7:0/3633522672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a2205428e0 0x55a220542cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.678+0000 7fa791e69740 1 -- 172.21.15.7:0/3633522672 >> 172.21.15.7:0/3633522672 conn(0x55a22053cfb0 msgr2=0x55a22053d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.686+0000 7fa791e69740 1 -- 172.21.15.7:0/3633522672 shutdown_connections 2024-04-21T08:50:18.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.688+0000 7fa791e69740 1 -- 172.21.15.7:0/3633522672 wait complete. 2024-04-21T08:50:18.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.688+0000 7fa791e69740 1 -- start start 2024-04-21T08:50:18.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.688+0000 7fa791e69740 1 Processor -- start 2024-04-21T08:50:18.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.691+0000 7fa791e69740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220548d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.691+0000 7fa791e69740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a220543290 0x55a220549240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.691+0000 7fa791e69740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a22054be10 0x55a220549780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.691+0000 7fa791e69740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a22054ed70 con 0x55a22054be10 2024-04-21T08:50:18.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.691+0000 7fa791e69740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a220460580 con 0x55a220543290 2024-04-21T08:50:18.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.691+0000 7fa791e69740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a22054eef0 con 0x55a2205428e0 2024-04-21T08:50:18.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.692+0000 7fa791e69740 10 client.0 _ll_register_callbacks cb 0x55a220466650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:18.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.692+0000 7fa78c808640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a22054be10 0x55a220549780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.692+0000 7fa78c808640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a22054be10 0x55a220549780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41864/0 (socket says 172.21.15.7:41864) 2024-04-21T08:50:18.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.692+0000 7fa78c808640 1 -- 172.21.15.7:0/4177796740 learned_addr learned my addr 172.21.15.7:0/4177796740 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:18.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.692+0000 7fa78c808640 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220548d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.697+0000 7fa78c808640 1 -- 172.21.15.7:0/4177796740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a220543290 msgr2=0x55a220549240 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:50:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.697+0000 7fa78c808640 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a220543290 0x55a220549240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.697+0000 7fa78c808640 1 -- 172.21.15.7:0/4177796740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a22054be10 msgr2=0x55a220549780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.697+0000 7fa78c808640 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a22054be10 0x55a220549780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.697+0000 7fa78c808640 1 -- 172.21.15.7:0/4177796740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2202e1f70 con 0x55a2205428e0 2024-04-21T08:50:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.701+0000 7fa78c808640 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220548d00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa778018ab0 tx=0x7fa7780047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e74c22e1d1df3912 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.703+0000 7fa770ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:50:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.703+0000 7fa770ff9640 1 -- 172.21.15.7:0/4177796740 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77802a030 con 0x55a2205428e0 2024-04-21T08:50:18.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.703+0000 7fa791e69740 1 -- 172.21.15.7:0/4177796740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7780188e0 con 0x55a2205428e0 2024-04-21T08:50:18.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.703+0000 7fa770ff9640 1 -- 172.21.15.7:0/4177796740 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa778018de0 con 0x55a2205428e0 2024-04-21T08:50:18.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.709+0000 7fa770ff9640 1 -- 172.21.15.7:0/4177796740 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa7780179b0 con 0x55a2205428e0 2024-04-21T08:50:18.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.711+0000 7fa791e69740 10 client.26305 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:18.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.711+0000 7fa791e69740 1 -- 172.21.15.7:0/4177796740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a220179170 con 0x55a2205428e0 2024-04-21T08:50:18.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.717+0000 7fa770ff9640 1 -- 172.21.15.7:0/4177796740 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7780188e0 con 0x55a2205428e0 2024-04-21T08:50:18.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.717+0000 7fa770ff9640 1 -- 172.21.15.7:0/4177796740 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa778012050 con 0x55a2205428e0 2024-04-21T08:50:18.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.720+0000 7fa791e69740 10 client.26305 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:18.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.720+0000 7fa791e69740 10 client.26305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.720+0000 7fa791e69740 20 client.26305 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.720+0000 7fa791e69740 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a22055c180 0x55a22055e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.720+0000 7fa791e69740 4 client.26305 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.720+0000 7fa791e69740 1 -- 172.21.15.7:0/4177796740 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2204b4920 con 0x55a22055c180 2024-04-21T08:50:18.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.728+0000 7fa78d009640 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a22055c180 0x55a22055e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.743+0000 7fa78d009640 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a22055c180 0x55a22055e570 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32efa156d8d4dfb2 server_cookie=3c79e8205d067c3a in_seq=0 out_seq=0 2024-04-21T08:50:18.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.745+0000 7fa770ff9640 10 client.26305 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:18.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.753+0000 7fa770ff9640 1 -- 172.21.15.7:0/4177796740 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2204b4920 con 0x55a22055c180 2024-04-21T08:50:18.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.753+0000 7fa770ff9640 10 client.26305 handle_command_reply: tid=0 2024-04-21T08:50:18.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.755+0000 7fa791e69740 1 client.26305 shutdown 2024-04-21T08:50:18.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.767+0000 7fa791e69740 1 -- 172.21.15.7:0/4177796740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 msgr2=0x55a220548d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.767+0000 7fa791e69740 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220548d00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa778018ab0 tx=0x7fa7780047d0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.774+0000 7fa791e69740 1 -- 172.21.15.7:0/4177796740 shutdown_connections 2024-04-21T08:50:18.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.774+0000 7fa791e69740 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a22055c180 0x55a22055e570 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.774+0000 7fa791e69740 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a22054be10 0x55a220549780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.774+0000 7fa791e69740 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a220543290 0x55a220549240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.774+0000 7fa791e69740 1 --2- 172.21.15.7:0/4177796740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220548d00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.774+0000 7fa791e69740 1 -- 172.21.15.7:0/4177796740 >> 172.21.15.7:0/4177796740 conn(0x55a22053cfb0 msgr2=0x55a22053ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.774+0000 7fa770ff9640 0 client.26305 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:18.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.777+0000 7fa791e69740 1 -- 172.21.15.7:0/4177796740 shutdown_connections 2024-04-21T08:50:18.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.780+0000 7fa791e69740 1 -- 172.21.15.7:0/4177796740 wait complete. 2024-04-21T08:50:18.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.781+0000 7fa791e69740 20 client.26305 trim_cache size 0 max 16384 2024-04-21T08:50:18.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.821+0000 7fa791e69740 1 Processor -- start 2024-04-21T08:50:18.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.821+0000 7fa791e69740 1 -- start start 2024-04-21T08:50:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.821+0000 7fa791e69740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220557c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.821+0000 7fa791e69740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a220543290 0x55a220563b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.821+0000 7fa791e69740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a22054be10 0x55a2205640b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.822+0000 7fa791e69740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa7880737d0 con 0x55a220543290 2024-04-21T08:50:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.822+0000 7fa791e69740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7881c71b0 con 0x55a22054be10 2024-04-21T08:50:18.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.822+0000 7fa791e69740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa788073e40 con 0x55a2205428e0 2024-04-21T08:50:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.825+0000 7fa78d009640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a220543290 0x55a220563b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.825+0000 7fa78d009640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a220543290 0x55a220563b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41870/0 (socket says 172.21.15.7:41870) 2024-04-21T08:50:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.825+0000 7fa78d009640 1 -- 172.21.15.7:0/1033800598 learned_addr learned my addr 172.21.15.7:0/1033800598 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.826+0000 7fa78c808640 1 --2- 172.21.15.7:0/1033800598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220557c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.826+0000 7fa78c808640 1 --2- 172.21.15.7:0/1033800598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a22054be10 0x55a2205640b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.829+0000 7fa78c808640 1 -- 172.21.15.7:0/1033800598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a22054be10 msgr2=0x55a2205640b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.829+0000 7fa78c808640 1 --2- 172.21.15.7:0/1033800598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a22054be10 0x55a2205640b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.829+0000 7fa78c808640 1 -- 172.21.15.7:0/1033800598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a220543290 msgr2=0x55a220563b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.829+0000 7fa78c808640 1 --2- 172.21.15.7:0/1033800598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a220543290 0x55a220563b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.829+0000 7fa78c808640 1 -- 172.21.15.7:0/1033800598 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a22053db80 con 0x55a2205428e0 2024-04-21T08:50:18.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.830+0000 7fa78c808640 1 --2- 172.21.15.7:0/1033800598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220557c80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa77800aad0 tx=0x7fa778000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=597bdb553f0b899a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.834+0000 7fa772ffd640 1 -- 172.21.15.7:0/1033800598 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77802a070 con 0x55a2205428e0 2024-04-21T08:50:18.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.834+0000 7fa772ffd640 1 -- 172.21.15.7:0/1033800598 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa77802b040 con 0x55a2205428e0 2024-04-21T08:50:18.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.835+0000 7fa791e69740 1 -- 172.21.15.7:0/1033800598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 msgr2=0x55a220557c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.835+0000 7fa791e69740 1 --2- 172.21.15.7:0/1033800598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220557c80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa77800aad0 tx=0x7fa778000bc0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.837+0000 7fa791e69740 1 -- 172.21.15.7:0/1033800598 shutdown_connections 2024-04-21T08:50:18.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.837+0000 7fa791e69740 1 --2- 172.21.15.7:0/1033800598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a22054be10 0x55a2205640b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.837+0000 7fa791e69740 1 --2- 172.21.15.7:0/1033800598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a220543290 0x55a220563b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.837+0000 7fa791e69740 1 --2- 172.21.15.7:0/1033800598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220557c80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.837+0000 7fa791e69740 1 -- 172.21.15.7:0/1033800598 >> 172.21.15.7:0/1033800598 conn(0x55a22053cfb0 msgr2=0x55a22053c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.841+0000 7fa791e69740 1 -- 172.21.15.7:0/1033800598 shutdown_connections 2024-04-21T08:50:18.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.845+0000 7fa791e69740 1 -- 172.21.15.7:0/1033800598 wait complete. 2024-04-21T08:50:18.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.845+0000 7fa791e69740 1 -- start start 2024-04-21T08:50:18.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.845+0000 7fa791e69740 1 Processor -- start 2024-04-21T08:50:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.846+0000 7fa791e69740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220545c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.846+0000 7fa791e69740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a220543290 0x55a220546160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.846+0000 7fa791e69740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a22054be10 0x55a2205466a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.846+0000 7fa791e69740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa7881c5670 con 0x55a22054be10 2024-04-21T08:50:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.846+0000 7fa791e69740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7881c6ae0 con 0x55a220543290 2024-04-21T08:50:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.846+0000 7fa791e69740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa7881c5d40 con 0x55a2205428e0 2024-04-21T08:50:18.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.846+0000 7fa791e69740 10 client.0 _ll_register_callbacks cb 0x7fa77800ac00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:18.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.851+0000 7fa78c808640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220545c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.851+0000 7fa78c808640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220545c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:45410/0 (socket says 172.21.15.7:45410) 2024-04-21T08:50:18.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.851+0000 7fa78c808640 1 -- 172.21.15.7:0/914822407 learned_addr learned my addr 172.21.15.7:0/914822407 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:18.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.851+0000 7fa78c808640 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a22054be10 0x55a2205466a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.853+0000 7fa78d009640 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a220543290 0x55a220546160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.854+0000 7fa78c808640 1 -- 172.21.15.7:0/914822407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a220543290 msgr2=0x55a220546160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.854+0000 7fa78c808640 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a220543290 0x55a220546160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.854+0000 7fa78c808640 1 -- 172.21.15.7:0/914822407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a22054be10 msgr2=0x55a2205466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.854+0000 7fa78c808640 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a22054be10 0x55a2205466a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.854+0000 7fa78c808640 1 -- 172.21.15.7:0/914822407 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2205c1630 con 0x55a2205428e0 2024-04-21T08:50:18.860 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:18 smithi007 ceph-mon[28477]: pgmap v172: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 1.48k op/s 2024-04-21T08:50:18.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.855+0000 7fa78c808640 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220545c20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa7780170a0 tx=0x7fa77800add0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a71c65ddd6588321 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.855+0000 7fa770ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:50:18.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.861+0000 7fa770ff9640 1 -- 172.21.15.7:0/914822407 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77802a070 con 0x55a2205428e0 2024-04-21T08:50:18.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.861+0000 7fa791e69740 1 -- 172.21.15.7:0/914822407 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa77802b040 con 0x55a2205428e0 2024-04-21T08:50:18.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.861+0000 7fa770ff9640 1 -- 172.21.15.7:0/914822407 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa778026e00 con 0x55a2205428e0 2024-04-21T08:50:18.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.868+0000 7fa770ff9640 1 -- 172.21.15.7:0/914822407 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa77800a660 con 0x55a2205428e0 2024-04-21T08:50:18.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.868+0000 7fa791e69740 10 client.26311 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:18.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.868+0000 7fa791e69740 1 -- 172.21.15.7:0/914822407 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a2205c2950 con 0x55a2205428e0 2024-04-21T08:50:18.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.873+0000 7fa770ff9640 1 -- 172.21.15.7:0/914822407 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa77802b040 con 0x55a2205428e0 2024-04-21T08:50:18.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.873+0000 7fa770ff9640 1 -- 172.21.15.7:0/914822407 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa778012070 con 0x55a2205428e0 2024-04-21T08:50:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.873+0000 7fa791e69740 10 client.26311 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.873+0000 7fa791e69740 10 client.26311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.873+0000 7fa791e69740 20 client.26311 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.873+0000 7fa791e69740 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a2205c2c90 0x55a2205c5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.873+0000 7fa791e69740 4 client.26311 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:50:18.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.873+0000 7fa791e69740 1 -- 172.21.15.7:0/914822407 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a2204ae790 con 0x55a2205c2c90 2024-04-21T08:50:18.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.877+0000 7fa78d009640 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a2205c2c90 0x55a2205c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.893+0000 7fa78d009640 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a2205c2c90 0x55a2205c5080 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd2225606503eb0 server_cookie=b76734f8d3f3844b in_seq=0 out_seq=0 2024-04-21T08:50:18.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.895+0000 7fa770ff9640 10 client.26311 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:18.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.899+0000 7fa770ff9640 1 -- 172.21.15.7:0/914822407 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a2204ae790 con 0x55a2205c2c90 2024-04-21T08:50:18.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.899+0000 7fa770ff9640 10 client.26311 handle_command_reply: tid=0 2024-04-21T08:50:18.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.899+0000 7fa791e69740 1 client.26311 shutdown 2024-04-21T08:50:18.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.903+0000 7fa791e69740 1 -- 172.21.15.7:0/914822407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 msgr2=0x55a220545c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.903+0000 7fa791e69740 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220545c20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa7780170a0 tx=0x7fa77800add0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.905+0000 7fa791e69740 1 -- 172.21.15.7:0/914822407 shutdown_connections 2024-04-21T08:50:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.905+0000 7fa791e69740 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a2205c2c90 0x55a2205c5080 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.905+0000 7fa791e69740 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a22054be10 0x55a2205466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.905+0000 7fa791e69740 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a220543290 0x55a220546160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.905+0000 7fa791e69740 1 --2- 172.21.15.7:0/914822407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2205428e0 0x55a220545c20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.905+0000 7fa791e69740 1 -- 172.21.15.7:0/914822407 >> 172.21.15.7:0/914822407 conn(0x55a22053cfb0 msgr2=0x55a22053c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.905+0000 7fa770ff9640 0 client.26311 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:18.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.911+0000 7fa791e69740 1 -- 172.21.15.7:0/914822407 shutdown_connections 2024-04-21T08:50:18.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.912+0000 7fa791e69740 1 -- 172.21.15.7:0/914822407 wait complete. 2024-04-21T08:50:18.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.915+0000 7fa791e69740 20 client.26311 trim_cache size 0 max 16384 2024-04-21T08:50:18.919 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:50:18.919 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:50:18.920 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "512e20f5-23fe-4e80-a3c9-7373ed6b32bd", 2024-04-21T08:50:18.920 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:50:18.920 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:50:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.922+0000 7fa78f26d640 1 -- 172.21.15.7:0/4272437925 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa7740a3d00 msgr2=0x7fa7740a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.922+0000 7fa78f26d640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa7740a3d00 0x7fa7740a61c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa7840bde10 tx=0x7fa784009260 comp rx=0 tx=0).stop 2024-04-21T08:50:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.922+0000 7fa78f26d640 1 -- 172.21.15.7:0/4272437925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa788111700 msgr2=0x7fa788111bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.922+0000 7fa78f26d640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa788111700 0x7fa788111bb0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa784009850 tx=0x7fa7840045e0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.923+0000 7fa78f26d640 1 -- 172.21.15.7:0/4272437925 shutdown_connections 2024-04-21T08:50:18.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.923+0000 7fa78f26d640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa788111700 0x7fa788111bb0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.923+0000 7fa78f26d640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa788107260 0x7fa788116ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.923+0000 7fa78f26d640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa7740a3d00 0x7fa7740a61c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.923+0000 7fa78f26d640 1 --2- 172.21.15.7:0/4272437925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa788102190 0x7fa7881165a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.923+0000 7fa78f26d640 1 -- 172.21.15.7:0/4272437925 >> 172.21.15.7:0/4272437925 conn(0x7fa7880fda10 msgr2=0x7fa7880ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.925+0000 7fa78f26d640 1 -- 172.21.15.7:0/4272437925 shutdown_connections 2024-04-21T08:50:18.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:18.926+0000 7fa78f26d640 1 -- 172.21.15.7:0/4272437925 wait complete. 2024-04-21T08:50:18.950 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:50:18.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:50:19.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:18 smithi077 ceph-mon[31615]: pgmap v172: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 1.48k op/s 2024-04-21T08:50:19.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.166+0000 7f1e1c338640 1 -- 172.21.15.7:0/2480880614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e14101080 msgr2=0x7f1e14101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.166+0000 7f1e1c338640 1 --2- 172.21.15.7:0/2480880614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e14101080 0x7f1e14101460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009ca0 tx=0x7f1e0c02e010 comp rx=0 tx=0).stop 2024-04-21T08:50:19.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.168+0000 7f1e1c338640 1 -- 172.21.15.7:0/2480880614 shutdown_connections 2024-04-21T08:50:19.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.168+0000 7f1e1c338640 1 --2- 172.21.15.7:0/2480880614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14106150 0x7f1e1410dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.168+0000 7f1e1c338640 1 --2- 172.21.15.7:0/2480880614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e14101a30 0x7f1e14105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.168+0000 7f1e1c338640 1 --2- 172.21.15.7:0/2480880614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e14101080 0x7f1e14101460 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.168+0000 7f1e1c338640 1 -- 172.21.15.7:0/2480880614 >> 172.21.15.7:0/2480880614 conn(0x7f1e140fc8a0 msgr2=0x7f1e140fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.170+0000 7f1e1c338640 1 -- 172.21.15.7:0/2480880614 shutdown_connections 2024-04-21T08:50:19.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.171+0000 7f1e1c338640 1 -- 172.21.15.7:0/2480880614 wait complete. 2024-04-21T08:50:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.171+0000 7f1e1c338640 1 Processor -- start 2024-04-21T08:50:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.172+0000 7f1e1c338640 1 -- start start 2024-04-21T08:50:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.172+0000 7f1e1c338640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14101080 0x7f1e14073a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.173+0000 7f1e1a0d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14101080 0x7f1e14073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.173+0000 7f1e1c338640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e14106150 0x7f1e1406ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.173+0000 7f1e1a0d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14101080 0x7f1e14073a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59298/0 (socket says 172.21.15.7:59298) 2024-04-21T08:50:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.173+0000 7f1e1c338640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e1406bf80 0x7f1e1406c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.173+0000 7f1e1c338640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1e141100f0 con 0x7f1e14106150 2024-04-21T08:50:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.173+0000 7f1e1c338640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1410ff70 con 0x7f1e14101080 2024-04-21T08:50:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.173+0000 7f1e1c338640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1e14110270 con 0x7f1e1406bf80 2024-04-21T08:50:19.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.173+0000 7f1e1a0d4640 1 -- 172.21.15.7:0/767131232 learned_addr learned my addr 172.21.15.7:0/767131232 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:19.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.175+0000 7f1e198d3640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e14106150 0x7f1e1406ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.176+0000 7f1e1a8d5640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e1406bf80 0x7f1e1406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.178+0000 7f1e1a8d5640 1 -- 172.21.15.7:0/767131232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14101080 msgr2=0x7f1e14073a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.178+0000 7f1e1a8d5640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14101080 0x7f1e14073a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.178+0000 7f1e1a8d5640 1 -- 172.21.15.7:0/767131232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e14106150 msgr2=0x7f1e1406ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.178+0000 7f1e1a8d5640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e14106150 0x7f1e1406ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.178+0000 7f1e1a8d5640 1 -- 172.21.15.7:0/767131232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e141ab600 con 0x7f1e1406bf80 2024-04-21T08:50:19.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.182+0000 7f1e1a8d5640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e1406bf80 0x7f1e1406c430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009f50 tx=0x7f1e1000b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.186+0000 7f1e077fe640 1 -- 172.21.15.7:0/767131232 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e10021070 con 0x7f1e1406bf80 2024-04-21T08:50:19.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.186+0000 7f1e1c338640 1 -- 172.21.15.7:0/767131232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e141ab8f0 con 0x7f1e1406bf80 2024-04-21T08:50:19.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.186+0000 7f1e1c338640 1 -- 172.21.15.7:0/767131232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e141abe40 con 0x7f1e1406bf80 2024-04-21T08:50:19.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.186+0000 7f1e077fe640 1 -- 172.21.15.7:0/767131232 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e10014d70 con 0x7f1e1406bf80 2024-04-21T08:50:19.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.186+0000 7f1e077fe640 1 -- 172.21.15.7:0/767131232 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1001a740 con 0x7f1e1406bf80 2024-04-21T08:50:19.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.191+0000 7f1e1effd740 1 Processor -- start 2024-04-21T08:50:19.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.192+0000 7f1e077fe640 1 -- 172.21.15.7:0/767131232 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f1e10024070 con 0x7f1e1406bf80 2024-04-21T08:50:19.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.193+0000 7f1e077fe640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1e000a3f30 0x7f1e000a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.193+0000 7f1e077fe640 1 -- 172.21.15.7:0/767131232 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f1e100d7de0 con 0x7f1e1406bf80 2024-04-21T08:50:19.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.195+0000 7f1e198d3640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1e000a3f30 0x7f1e000a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.195+0000 7f1e1effd740 1 -- start start 2024-04-21T08:50:19.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.195+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 0x5630bf5cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.195+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cfea0 0x5630bf5d8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.195+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5dae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.195+0000 7f1e1effd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5630bf5341e0 con 0x5630bf5d8a80 2024-04-21T08:50:19.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.195+0000 7f1e1effd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5630bf4d1390 con 0x5630bf5cf580 2024-04-21T08:50:19.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.195+0000 7f1e1effd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5630bf5301a0 con 0x5630bf5cfea0 2024-04-21T08:50:19.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.199+0000 7f1e1a0d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 0x5630bf5cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.199+0000 7f1e1a0d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 0x5630bf5cf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59300/0 (socket says 172.21.15.7:59300) 2024-04-21T08:50:19.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.199+0000 7f1e1a0d4640 1 -- 172.21.15.7:0/3418648480 learned_addr learned my addr 172.21.15.7:0/3418648480 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:19.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.199+0000 7f1e198d3640 1 --2- 172.21.15.7:0/3418648480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cfea0 0x5630bf5d8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.199+0000 7f1e198d3640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1e000a3f30 0x7f1e000a63f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1e1406d200 tx=0x7f1e08009e40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.200+0000 7f1e1a0d4640 1 -- 172.21.15.7:0/3418648480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cfea0 msgr2=0x5630bf5d8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:19.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.200+0000 7f1e1a8d5640 1 --2- 172.21.15.7:0/3418648480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5dae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.200+0000 7f1e1a0d4640 1 --2- 172.21.15.7:0/3418648480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cfea0 0x5630bf5d8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.200+0000 7f1e1a0d4640 1 -- 172.21.15.7:0/3418648480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 msgr2=0x5630bf5dae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:19.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.200+0000 7f1e1a0d4640 1 --2- 172.21.15.7:0/3418648480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5dae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.201+0000 7f1e1a0d4640 1 -- 172.21.15.7:0/3418648480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630bf3107d0 con 0x5630bf5cf580 2024-04-21T08:50:19.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.201+0000 7f1e1a8d5640 1 --2- 172.21.15.7:0/3418648480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5dae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:19.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.202+0000 7f1e1a0d4640 1 --2- 172.21.15.7:0/3418648480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 0x5630bf5cf960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00d470 tx=0x7f1e0c00d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dab99fbeb66a5b0f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.214+0000 7f1e04bfe640 1 -- 172.21.15.7:0/3418648480 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c044030 con 0x5630bf5cf580 2024-04-21T08:50:19.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.214+0000 7f1e04bfe640 1 -- 172.21.15.7:0/3418648480 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e0c03c040 con 0x5630bf5cf580 2024-04-21T08:50:19.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.214+0000 7f1e04bfe640 1 -- 172.21.15.7:0/3418648480 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c03b070 con 0x5630bf5cf580 2024-04-21T08:50:19.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.216+0000 7f1e1effd740 1 -- 172.21.15.7:0/3418648480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 msgr2=0x5630bf5cf960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.216+0000 7f1e1effd740 1 --2- 172.21.15.7:0/3418648480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 0x5630bf5cf960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00d470 tx=0x7f1e0c00d4a0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.221+0000 7f1e1effd740 1 -- 172.21.15.7:0/3418648480 shutdown_connections 2024-04-21T08:50:19.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.221+0000 7f1e1effd740 1 --2- 172.21.15.7:0/3418648480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5dae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.221+0000 7f1e1effd740 1 --2- 172.21.15.7:0/3418648480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cfea0 0x5630bf5d8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.221+0000 7f1e1effd740 1 --2- 172.21.15.7:0/3418648480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 0x5630bf5cf960 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.221+0000 7f1e1effd740 1 -- 172.21.15.7:0/3418648480 >> 172.21.15.7:0/3418648480 conn(0x5630bf5c9cb0 msgr2=0x5630bf5ca090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.224+0000 7f1e1effd740 1 -- 172.21.15.7:0/3418648480 shutdown_connections 2024-04-21T08:50:19.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.227+0000 7f1e1effd740 1 -- 172.21.15.7:0/3418648480 wait complete. 2024-04-21T08:50:19.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.227+0000 7f1e1effd740 1 -- start start 2024-04-21T08:50:19.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.227+0000 7f1e1effd740 1 Processor -- start 2024-04-21T08:50:19.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.231+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 0x5630bf5d5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.231+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cfea0 0x5630bf5d5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.231+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5d6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.231+0000 7f1e1effd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5630bf5db810 con 0x5630bf5d8a80 2024-04-21T08:50:19.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.231+0000 7f1e1effd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5630bf4ed2f0 con 0x5630bf5cf580 2024-04-21T08:50:19.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.231+0000 7f1e1effd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5630bf5db990 con 0x5630bf5cfea0 2024-04-21T08:50:19.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.231+0000 7f1e1effd740 10 client.0 _ll_register_callbacks cb 0x5630bf4fa580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:19.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.234+0000 7f1e1a0d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 0x5630bf5d5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.234+0000 7f1e1a0d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 0x5630bf5d5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59304/0 (socket says 172.21.15.7:59304) 2024-04-21T08:50:19.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.234+0000 7f1e1a0d4640 1 -- 172.21.15.7:0/1863759778 learned_addr learned my addr 172.21.15.7:0/1863759778 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:19.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.235+0000 7f1e1a8d5640 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5d6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.239+0000 7f1e198d3640 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cfea0 0x5630bf5d5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.240+0000 7f1e198d3640 1 -- 172.21.15.7:0/1863759778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 msgr2=0x5630bf5d5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:19.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.240+0000 7f1e198d3640 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 0x5630bf5d5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.240+0000 7f1e198d3640 1 -- 172.21.15.7:0/1863759778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 msgr2=0x5630bf5d6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:19.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.240+0000 7f1e198d3640 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5d6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.240+0000 7f1e198d3640 1 -- 172.21.15.7:0/1863759778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630bf2463f0 con 0x5630bf5cfea0 2024-04-21T08:50:19.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.241+0000 7f1e198d3640 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cfea0 0x5630bf5d5f40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5630bf5cf050 tx=0x7f1e080046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77943e296caaf766 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.241+0000 7f1dfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:50:19.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.241+0000 7f1dfdffb640 1 -- 172.21.15.7:0/1863759778 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e08025020 con 0x5630bf5cfea0 2024-04-21T08:50:19.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.242+0000 7f1e1effd740 1 -- 172.21.15.7:0/1863759778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e0c03c040 con 0x5630bf5cfea0 2024-04-21T08:50:19.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.242+0000 7f1dfdffb640 1 -- 172.21.15.7:0/1863759778 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f1e08021640 con 0x5630bf5cfea0 2024-04-21T08:50:19.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.244+0000 7f1dfdffb640 1 -- 172.21.15.7:0/1863759778 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1e08030050 con 0x5630bf5cfea0 2024-04-21T08:50:19.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.245+0000 7f1e1effd740 10 client.26323 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:19.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.245+0000 7f1e1effd740 1 -- 172.21.15.7:0/1863759778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5630bf2060f0 con 0x5630bf5cfea0 2024-04-21T08:50:19.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.248+0000 7f1dfdffb640 1 -- 172.21.15.7:0/1863759778 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e0c03c040 con 0x5630bf5cfea0 2024-04-21T08:50:19.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.248+0000 7f1dfdffb640 1 -- 172.21.15.7:0/1863759778 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f1e08020070 con 0x5630bf5cfea0 2024-04-21T08:50:19.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.248+0000 7f1e1effd740 10 client.26323 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:19.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.248+0000 7f1e1effd740 10 client.26323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:19.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.248+0000 7f1e1effd740 20 client.26323 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:19.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.248+0000 7f1e1effd740 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5630bf5e8a80 0x5630bf5eae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.248+0000 7f1e1effd740 4 client.26323 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:19.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.248+0000 7f1e1effd740 1 -- 172.21.15.7:0/1863759778 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630bf541c00 con 0x5630bf5e8a80 2024-04-21T08:50:19.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.250+0000 7f1e1a0d4640 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5630bf5e8a80 0x5630bf5eae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.258+0000 7f1e1a0d4640 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5630bf5e8a80 0x5630bf5eae70 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=918e0a9bc27f45f1 server_cookie=236cd4126dfb2ad6 in_seq=0 out_seq=0 2024-04-21T08:50:19.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.258+0000 7f1dfdffb640 10 client.26323 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:19.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.264+0000 7f1dfdffb640 1 -- 172.21.15.7:0/1863759778 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5630bf541c00 con 0x5630bf5e8a80 2024-04-21T08:50:19.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.265+0000 7f1dfdffb640 10 client.26323 handle_command_reply: tid=0 2024-04-21T08:50:19.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.267+0000 7f1e1effd740 1 client.26323 shutdown 2024-04-21T08:50:19.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.273+0000 7f1e1effd740 1 -- 172.21.15.7:0/1863759778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cfea0 msgr2=0x5630bf5d5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.273+0000 7f1e1effd740 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cfea0 0x5630bf5d5f40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5630bf5cf050 tx=0x7f1e080046c0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.279+0000 7f1e1effd740 1 -- 172.21.15.7:0/1863759778 shutdown_connections 2024-04-21T08:50:19.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.279+0000 7f1e1effd740 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5630bf5e8a80 0x5630bf5eae70 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.279+0000 7f1e1effd740 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5d6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.279+0000 7f1e1effd740 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cfea0 0x5630bf5d5f40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.279+0000 7f1e1effd740 1 --2- 172.21.15.7:0/1863759778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cf580 0x5630bf5d5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.279+0000 7f1e1effd740 1 -- 172.21.15.7:0/1863759778 >> 172.21.15.7:0/1863759778 conn(0x5630bf5c9cb0 msgr2=0x5630bf5cba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.280+0000 7f1dfdffb640 0 client.26323 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:19.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.282+0000 7f1e1effd740 1 -- 172.21.15.7:0/1863759778 shutdown_connections 2024-04-21T08:50:19.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.286+0000 7f1e1effd740 1 -- 172.21.15.7:0/1863759778 wait complete. 2024-04-21T08:50:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.288+0000 7f1e1effd740 20 client.26323 trim_cache size 0 max 16384 2024-04-21T08:50:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:18 smithi016 ceph-mon[31259]: pgmap v172: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 1.48k op/s 2024-04-21T08:50:19.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.324+0000 7f1e1effd740 1 Processor -- start 2024-04-21T08:50:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.325+0000 7f1e1effd740 1 -- start start 2024-04-21T08:50:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.325+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5cf580 0x5630bf5e4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.325+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cfea0 0x5630bf5d62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.325+0000 7f1e1a0d4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5cf580 0x5630bf5e4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.325+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5d8a80 0x5630bf5f0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.326+0000 7f1e1effd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1df800e170 con 0x5630bf5cf580 2024-04-21T08:50:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.326+0000 7f1e1effd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1df80109f0 con 0x5630bf5cfea0 2024-04-21T08:50:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.326+0000 7f1e1effd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1df800e7e0 con 0x5630bf5d8a80 2024-04-21T08:50:19.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.326+0000 7f1e1a8d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5d8a80 0x5630bf5f0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.326+0000 7f1e1a8d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5d8a80 0x5630bf5f0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:45458/0 (socket says 172.21.15.7:45458) 2024-04-21T08:50:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.326+0000 7f1e1a8d5640 1 -- 172.21.15.7:0/511477040 learned_addr learned my addr 172.21.15.7:0/511477040 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.327+0000 7f1e1a8d5640 1 -- 172.21.15.7:0/511477040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cfea0 msgr2=0x5630bf5d62a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:50:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.327+0000 7f1e1a8d5640 1 --2- 172.21.15.7:0/511477040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cfea0 0x5630bf5d62a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.327+0000 7f1e1a8d5640 1 -- 172.21.15.7:0/511477040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5cf580 msgr2=0x5630bf5e4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.327+0000 7f1e1a8d5640 1 --2- 172.21.15.7:0/511477040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5cf580 0x5630bf5e4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.327+0000 7f1e1a8d5640 1 -- 172.21.15.7:0/511477040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630bf5c9ae0 con 0x5630bf5d8a80 2024-04-21T08:50:19.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.327+0000 7f1e1a8d5640 1 --2- 172.21.15.7:0/511477040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5d8a80 0x5630bf5f0760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1e100047b0 tx=0x7f1e100c4040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dad84027eb5a7488 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.330+0000 7f1dfffff640 1 -- 172.21.15.7:0/511477040 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e100c40b0 con 0x5630bf5d8a80 2024-04-21T08:50:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.330+0000 7f1dfffff640 1 -- 172.21.15.7:0/511477040 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e100a3af0 con 0x5630bf5d8a80 2024-04-21T08:50:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.330+0000 7f1dfffff640 1 -- 172.21.15.7:0/511477040 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e100ac220 con 0x5630bf5d8a80 2024-04-21T08:50:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.331+0000 7f1e1effd740 1 -- 172.21.15.7:0/511477040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5d8a80 msgr2=0x5630bf5f0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.331+0000 7f1e1effd740 1 --2- 172.21.15.7:0/511477040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5d8a80 0x5630bf5f0760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1e100047b0 tx=0x7f1e100c4040 comp rx=0 tx=0).stop 2024-04-21T08:50:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.331+0000 7f1e1effd740 1 -- 172.21.15.7:0/511477040 shutdown_connections 2024-04-21T08:50:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.331+0000 7f1e1effd740 1 --2- 172.21.15.7:0/511477040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5d8a80 0x5630bf5f0760 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.331+0000 7f1e1effd740 1 --2- 172.21.15.7:0/511477040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cfea0 0x5630bf5d62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.331+0000 7f1e1effd740 1 --2- 172.21.15.7:0/511477040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5cf580 0x5630bf5e4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.331+0000 7f1e1effd740 1 -- 172.21.15.7:0/511477040 >> 172.21.15.7:0/511477040 conn(0x5630bf5c9cb0 msgr2=0x5630bf5c8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.339+0000 7f1e1effd740 1 -- 172.21.15.7:0/511477040 shutdown_connections 2024-04-21T08:50:19.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.342+0000 7f1e1effd740 1 -- 172.21.15.7:0/511477040 wait complete. 2024-04-21T08:50:19.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.342+0000 7f1e1effd740 1 -- start start 2024-04-21T08:50:19.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.342+0000 7f1e1effd740 1 Processor -- start 2024-04-21T08:50:19.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.343+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cf580 0x5630bf64a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.343+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cfea0 0x5630bf5e0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.343+0000 7f1e1effd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5e11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.343+0000 7f1e1effd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1df800eeb0 con 0x5630bf5d8a80 2024-04-21T08:50:19.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.343+0000 7f1e1effd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8010320 con 0x5630bf5cfea0 2024-04-21T08:50:19.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.343+0000 7f1e1effd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1df800f580 con 0x5630bf5cf580 2024-04-21T08:50:19.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.343+0000 7f1e1effd740 10 client.0 _ll_register_callbacks cb 0x7f1e080119f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.345+0000 7f1e1a8d5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5e11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.345+0000 7f1e1a8d5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5e11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41936/0 (socket says 172.21.15.7:41936) 2024-04-21T08:50:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.345+0000 7f1e1a8d5640 1 -- 172.21.15.7:0/2347953952 learned_addr learned my addr 172.21.15.7:0/2347953952 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.345+0000 7f1e1a0d4640 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cf580 0x5630bf64a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.345+0000 7f1e198d3640 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cfea0 0x5630bf5e0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.348+0000 7f1e198d3640 1 -- 172.21.15.7:0/2347953952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cf580 msgr2=0x5630bf64a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:19.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.348+0000 7f1e198d3640 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cf580 0x5630bf64a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.348+0000 7f1e198d3640 1 -- 172.21.15.7:0/2347953952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 msgr2=0x5630bf5e11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:19.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.348+0000 7f1e198d3640 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5e11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.348+0000 7f1e198d3640 1 -- 172.21.15.7:0/2347953952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630bf647bf0 con 0x5630bf5cfea0 2024-04-21T08:50:19.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.356+0000 7f1e198d3640 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cfea0 0x5630bf5e0c70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x5630bf5d6fd0 tx=0x7f1e08015e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f8df4fa836854f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.357+0000 7f1dfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:50:19.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.360+0000 7f1dfdffb640 1 -- 172.21.15.7:0/2347953952 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e08038020 con 0x5630bf5cfea0 2024-04-21T08:50:19.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.360+0000 7f1e1effd740 1 -- 172.21.15.7:0/2347953952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e100a3af0 con 0x5630bf5cfea0 2024-04-21T08:50:19.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.360+0000 7f1dfdffb640 1 -- 172.21.15.7:0/2347953952 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f1e08015600 con 0x5630bf5cfea0 2024-04-21T08:50:19.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.365+0000 7f1dfdffb640 1 -- 172.21.15.7:0/2347953952 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1e08025070 con 0x5630bf5cfea0 2024-04-21T08:50:19.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.367+0000 7f1e1effd740 10 client.26351 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:19.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.367+0000 7f1e1effd740 1 -- 172.21.15.7:0/2347953952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5630bf6603e0 con 0x5630bf5cfea0 2024-04-21T08:50:19.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.368+0000 7f1dfdffb640 1 -- 172.21.15.7:0/2347953952 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e100a3af0 con 0x5630bf5cfea0 2024-04-21T08:50:19.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.368+0000 7f1dfdffb640 1 -- 172.21.15.7:0/2347953952 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f1e08020040 con 0x5630bf5cfea0 2024-04-21T08:50:19.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.369+0000 7f1e1effd740 10 client.26351 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:19.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.369+0000 7f1e1effd740 10 client.26351 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:19.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.369+0000 7f1e1effd740 20 client.26351 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:19.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.369+0000 7f1e1effd740 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5630bf660b40 0x5630bf662f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.369+0000 7f1e1effd740 4 client.26351 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:50:19.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.369+0000 7f1e1effd740 1 -- 172.21.15.7:0/2347953952 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5630bf53ba70 con 0x5630bf660b40 2024-04-21T08:50:19.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.369+0000 7f1e1a0d4640 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5630bf660b40 0x5630bf662f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.381+0000 7f1e1a0d4640 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5630bf660b40 0x5630bf662f60 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd9ffd592ee7deaa server_cookie=d97f1dbaeaf8b8a4 in_seq=0 out_seq=0 2024-04-21T08:50:19.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.382+0000 7f1dfdffb640 10 client.26351 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.390+0000 7f1dfdffb640 1 -- 172.21.15.7:0/2347953952 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5630bf53ba70 con 0x5630bf660b40 2024-04-21T08:50:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.390+0000 7f1dfdffb640 10 client.26351 handle_command_reply: tid=0 2024-04-21T08:50:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.390+0000 7f1e1effd740 1 client.26351 shutdown 2024-04-21T08:50:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.396+0000 7f1e1effd740 1 -- 172.21.15.7:0/2347953952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cfea0 msgr2=0x5630bf5e0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.396+0000 7f1e1effd740 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cfea0 0x5630bf5e0c70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x5630bf5d6fd0 tx=0x7f1e08015e50 comp rx=0 tx=0).stop 2024-04-21T08:50:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.399+0000 7f1e1effd740 1 -- 172.21.15.7:0/2347953952 shutdown_connections 2024-04-21T08:50:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.399+0000 7f1e1effd740 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5630bf660b40 0x5630bf662f60 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.399+0000 7f1e1effd740 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5630bf5d8a80 0x5630bf5e11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.399+0000 7f1e1effd740 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5630bf5cfea0 0x5630bf5e0c70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.399+0000 7f1e1effd740 1 --2- 172.21.15.7:0/2347953952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5630bf5cf580 0x5630bf64a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.399+0000 7f1e1effd740 1 -- 172.21.15.7:0/2347953952 >> 172.21.15.7:0/2347953952 conn(0x5630bf5c9cb0 msgr2=0x5630bf5cae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.400+0000 7f1dfdffb640 0 client.26351 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:19.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.404+0000 7f1e1effd740 1 -- 172.21.15.7:0/2347953952 shutdown_connections 2024-04-21T08:50:19.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.407+0000 7f1e1effd740 1 -- 172.21.15.7:0/2347953952 wait complete. 2024-04-21T08:50:19.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.408+0000 7f1e1effd740 20 client.26351 trim_cache size 0 max 16384 2024-04-21T08:50:19.408 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:50:19.408 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (2 inodes in the stack)", 2024-04-21T08:50:19.408 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:50:19.408 INFO:teuthology.orchestra.run.smithi007.stdout: "512e20f5-23fe-4e80-a3c9-7373ed6b32bd": { 2024-04-21T08:50:19.408 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:50:19.408 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "512e20f5-23fe-4e80-a3c9-7373ed6b32bd", 2024-04-21T08:50:19.408 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:50:19.408 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:50:19.408 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:50:19.408 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:50:19.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.410+0000 7f1e1c338640 1 -- 172.21.15.7:0/767131232 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1e000a3f30 msgr2=0x7f1e000a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.410+0000 7f1e1c338640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1e000a3f30 0x7f1e000a63f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1e1406d200 tx=0x7f1e08009e40 comp rx=0 tx=0).stop 2024-04-21T08:50:19.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.410+0000 7f1e1c338640 1 -- 172.21.15.7:0/767131232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e1406bf80 msgr2=0x7f1e1406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.410+0000 7f1e1c338640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e1406bf80 0x7f1e1406c430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009f50 tx=0x7f1e1000b870 comp rx=0 tx=0).stop 2024-04-21T08:50:19.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.410+0000 7f1e1c338640 1 -- 172.21.15.7:0/767131232 shutdown_connections 2024-04-21T08:50:19.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.410+0000 7f1e1c338640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f1e000a3f30 0x7f1e000a63f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.410+0000 7f1e1c338640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e1406bf80 0x7f1e1406c430 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.410+0000 7f1e1c338640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e14106150 0x7f1e1406ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.410+0000 7f1e1c338640 1 --2- 172.21.15.7:0/767131232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14101080 0x7f1e14073a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.410+0000 7f1e1c338640 1 -- 172.21.15.7:0/767131232 >> 172.21.15.7:0/767131232 conn(0x7f1e140fc8a0 msgr2=0x7f1e140fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.410+0000 7f1e1c338640 1 -- 172.21.15.7:0/767131232 shutdown_connections 2024-04-21T08:50:19.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:19.413+0000 7f1e1c338640 1 -- 172.21.15.7:0/767131232 wait complete. 2024-04-21T08:50:19.433 INFO:tasks.cephfs.filesystem:scrub status for tag:512e20f5-23fe-4e80-a3c9-7373ed6b32bd - {'path': '/', 'tag': '512e20f5-23fe-4e80-a3c9-7373ed6b32bd', 'options': 'recursive,force'} 2024-04-21T08:50:20.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:19 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:50:20.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:19 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:20.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:19 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:50:20.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:19 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:50:20.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:19 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:20.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:19 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:50:20.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:19 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:50:20.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:19 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:20.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:19 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:50:21.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:20 smithi007 ceph-mon[28477]: pgmap v173: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 1.55k op/s 2024-04-21T08:50:21.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:20 smithi016 ceph-mon[31259]: pgmap v173: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 1.55k op/s 2024-04-21T08:50:21.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:20 smithi077 ceph-mon[31615]: pgmap v173: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 1.55k op/s 2024-04-21T08:50:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.868+0000 7fc278fab640 1 -- 172.21.15.7:0/3943314812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc274106170 msgr2=0x7fc27410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.868+0000 7fc278fab640 1 --2- 172.21.15.7:0/3943314812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc274106170 0x7fc27410dd40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fc26c009900 tx=0x7fc26c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.868+0000 7fc278fab640 1 -- 172.21.15.7:0/3943314812 shutdown_connections 2024-04-21T08:50:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.868+0000 7fc278fab640 1 --2- 172.21.15.7:0/3943314812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc274106170 0x7fc27410dd40 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.868+0000 7fc278fab640 1 --2- 172.21.15.7:0/3943314812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc274101a50 0x7fc274105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.868+0000 7fc278fab640 1 --2- 172.21.15.7:0/3943314812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2741010a0 0x7fc274101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.868+0000 7fc278fab640 1 -- 172.21.15.7:0/3943314812 >> 172.21.15.7:0/3943314812 conn(0x7fc274077f70 msgr2=0x7fc274078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.868+0000 7fc278fab640 1 -- 172.21.15.7:0/3943314812 shutdown_connections 2024-04-21T08:50:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.868+0000 7fc278fab640 1 -- 172.21.15.7:0/3943314812 wait complete. 2024-04-21T08:50:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.869+0000 7fc278fab640 1 Processor -- start 2024-04-21T08:50:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.869+0000 7fc278fab640 1 -- start start 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.869+0000 7fc278fab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2741010a0 0x7fc2741986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.869+0000 7fc278fab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274106170 0x7fc274198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.869+0000 7fc278fab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc27419cfa0 0x7fc27419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.869+0000 7fc278fab640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741110e0 con 0x7fc2741010a0 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.869+0000 7fc278fab640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc274110f60 con 0x7fc274106170 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.869+0000 7fc278fab640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc274111260 con 0x7fc27419cfa0 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.869+0000 7fc27259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2741010a0 0x7fc2741986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.869+0000 7fc27259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2741010a0 0x7fc2741986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41962/0 (socket says 172.21.15.7:41962) 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc271d9b640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274106170 0x7fc274198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc27259c640 1 -- 172.21.15.7:0/1318825974 learned_addr learned my addr 172.21.15.7:0/1318825974 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc272d9d640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc27419cfa0 0x7fc27419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc271d9b640 1 -- 172.21.15.7:0/1318825974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc27419cfa0 msgr2=0x7fc27419d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc271d9b640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc27419cfa0 0x7fc27419d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc271d9b640 1 -- 172.21.15.7:0/1318825974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2741010a0 msgr2=0x7fc2741986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc271d9b640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2741010a0 0x7fc2741986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc271d9b640 1 -- 172.21.15.7:0/1318825974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27419da90 con 0x7fc274106170 2024-04-21T08:50:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc271d9b640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274106170 0x7fc274198c20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc26800e9c0 tx=0x7fc26800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc2637fe640 1 -- 172.21.15.7:0/1318825974 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26800ccc0 con 0x7fc274106170 2024-04-21T08:50:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc278fab640 1 -- 172.21.15.7:0/1318825974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc27419dd80 con 0x7fc274106170 2024-04-21T08:50:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc2637fe640 1 -- 172.21.15.7:0/1318825974 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc268005020 con 0x7fc274106170 2024-04-21T08:50:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc2637fe640 1 -- 172.21.15.7:0/1318825974 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc268004a00 con 0x7fc274106170 2024-04-21T08:50:21.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.870+0000 7fc278fab640 1 -- 172.21.15.7:0/1318825974 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2741a5570 con 0x7fc274106170 2024-04-21T08:50:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.872+0000 7fc2637fe640 1 -- 172.21.15.7:0/1318825974 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fc2680051d0 con 0x7fc274106170 2024-04-21T08:50:21.878 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc27bb9d740 1 Processor -- start 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc27bb9d740 1 -- start start 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc2637fe640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc25c0a3d10 0x7fc25c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e0d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 0x561ea5e0dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e194d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561ea5d728d0 con 0x561ea5e0db60 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ea5d0fe80 con 0x561ea5e0e8d0 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ea5d6e890 con 0x561ea5e170e0 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc272d9d640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc25c0a3d10 0x7fc25c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc271d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 0x561ea5e0dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc2637fe640 1 -- 172.21.15.7:0/1318825974 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc2680d0bc0 con 0x7fc274106170 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc27259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e0d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc27259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e0d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59358/0 (socket says 172.21.15.7:59358) 2024-04-21T08:50:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc27259c640 1 -- 172.21.15.7:0/1237752325 learned_addr learned my addr 172.21.15.7:0/1237752325 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.873+0000 7fc272d9d640 1 --2- 172.21.15.7:0/1237752325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e194d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.874+0000 7fc27259c640 1 -- 172.21.15.7:0/1237752325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 msgr2=0x561ea5e194d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.874+0000 7fc27259c640 1 --2- 172.21.15.7:0/1237752325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e194d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.874+0000 7fc27259c640 1 -- 172.21.15.7:0/1237752325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 msgr2=0x561ea5e0dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.874+0000 7fc27259c640 1 --2- 172.21.15.7:0/1237752325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 0x561ea5e0dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.874+0000 7fc27259c640 1 -- 172.21.15.7:0/1237752325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea5d31320 con 0x561ea5e0e8d0 2024-04-21T08:50:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.874+0000 7fc272d9d640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc25c0a3d10 0x7fc25c0a61d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc26c006010 tx=0x7fc26c02c730 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.874+0000 7fc27259c640 1 --2- 172.21.15.7:0/1237752325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e0d620 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc2640115c0 tx=0x7fc2640119c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46b05b9d2ddd217d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.874+0000 7fc260ff9640 1 -- 172.21.15.7:0/1237752325 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26401e070 con 0x561ea5e0e8d0 2024-04-21T08:50:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.874+0000 7fc260ff9640 1 -- 172.21.15.7:0/1237752325 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc26400fa70 con 0x561ea5e0e8d0 2024-04-21T08:50:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.874+0000 7fc260ff9640 1 -- 172.21.15.7:0/1237752325 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc264019af0 con 0x561ea5e0e8d0 2024-04-21T08:50:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.875+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1237752325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 msgr2=0x561ea5e0d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.875+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/1237752325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e0d620 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc2640115c0 tx=0x7fc2640119c0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.875+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1237752325 shutdown_connections 2024-04-21T08:50:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.875+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/1237752325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e194d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.875+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/1237752325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 0x561ea5e0dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.875+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/1237752325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e0d620 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.875+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1237752325 >> 172.21.15.7:0/1237752325 conn(0x561ea5dd4c90 msgr2=0x561ea5e08850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.875+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1237752325 shutdown_connections 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.875+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1237752325 wait complete. 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.875+0000 7fc27bb9d740 1 -- start start 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.875+0000 7fc27bb9d740 1 Processor -- start 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0db60 0x561ea5e16a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0e8d0 0x561ea5e13e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e143b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561ea5e19e10 con 0x561ea5e0e8d0 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ea5cfaeb0 con 0x561ea5e0db60 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ea5e19f90 con 0x561ea5e170e0 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc27bb9d740 10 client.0 _ll_register_callbacks cb 0x561ea5d27b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc271d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0e8d0 0x561ea5e13e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc271d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0e8d0 0x561ea5e13e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41976/0 (socket says 172.21.15.7:41976) 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc271d9b640 1 -- 172.21.15.7:0/8855525 learned_addr learned my addr 172.21.15.7:0/8855525 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc27259c640 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0db60 0x561ea5e16a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc272d9d640 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e143b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc271d9b640 1 -- 172.21.15.7:0/8855525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 msgr2=0x561ea5e143b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc271d9b640 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e143b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc271d9b640 1 -- 172.21.15.7:0/8855525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0db60 msgr2=0x561ea5e16a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc271d9b640 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0db60 0x561ea5e16a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc271d9b640 1 -- 172.21.15.7:0/8855525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea5a85210 con 0x561ea5e0e8d0 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc272d9d640 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e143b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc27259c640 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0db60 0x561ea5e16a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc271d9b640 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0e8d0 0x561ea5e13e10 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fc2680cdf80 tx=0x7fc26809cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cdb2a7ede1a9b25 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.876+0000 7fc259ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc259ffb640 1 -- 172.21.15.7:0/8855525 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26801c040 con 0x561ea5e0e8d0 2024-04-21T08:50:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc259ffb640 1 -- 172.21.15.7:0/8855525 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc2680ce1a0 con 0x561ea5e0e8d0 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc27bb9d740 1 -- 172.21.15.7:0/8855525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc26400fa70 con 0x561ea5e0e8d0 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc259ffb640 1 -- 172.21.15.7:0/8855525 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc26809c770 con 0x561ea5e0e8d0 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc27bb9d740 10 client.16812 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc27bb9d740 1 -- 172.21.15.7:0/8855525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561ea5b51620 con 0x561ea5e0e8d0 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc259ffb640 1 -- 172.21.15.7:0/8855525 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc26400fa70 con 0x561ea5e0e8d0 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc259ffb640 1 -- 172.21.15.7:0/8855525 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fc26806c440 con 0x561ea5e0e8d0 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc27bb9d740 10 client.16812 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc27bb9d740 10 client.16812 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc27bb9d740 20 client.16812 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ea5e27150 0x561ea5e29540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc27bb9d740 4 client.16812 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc27bb9d740 1 -- 172.21.15.7:0/8855525 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ea5d86880 con 0x561ea5e27150 2024-04-21T08:50:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.877+0000 7fc27259c640 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ea5e27150 0x561ea5e29540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.878+0000 7fc27259c640 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ea5e27150 0x561ea5e29540 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81015755fa164f15 server_cookie=2d54a77fec2f9ccc in_seq=0 out_seq=0 2024-04-21T08:50:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.878+0000 7fc259ffb640 10 client.16812 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc259ffb640 1 -- 172.21.15.7:0/8855525 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ea5d86880 con 0x561ea5e27150 2024-04-21T08:50:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc259ffb640 10 client.16812 handle_command_reply: tid=0 2024-04-21T08:50:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 1 client.16812 shutdown 2024-04-21T08:50:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 1 -- 172.21.15.7:0/8855525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0e8d0 msgr2=0x561ea5e13e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0e8d0 0x561ea5e13e10 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fc2680cdf80 tx=0x7fc26809cf30 comp rx=0 tx=0).stop 2024-04-21T08:50:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 1 -- 172.21.15.7:0/8855525 shutdown_connections 2024-04-21T08:50:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ea5e27150 0x561ea5e29540 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e143b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc259ffb640 0 client.16812 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0e8d0 0x561ea5e13e10 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/8855525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0db60 0x561ea5e16a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 1 -- 172.21.15.7:0/8855525 >> 172.21.15.7:0/8855525 conn(0x561ea5dd4c90 msgr2=0x561ea5e0a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 1 -- 172.21.15.7:0/8855525 shutdown_connections 2024-04-21T08:50:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 1 -- 172.21.15.7:0/8855525 wait complete. 2024-04-21T08:50:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.881+0000 7fc27bb9d740 20 client.16812 trim_cache size 0 max 16384 2024-04-21T08:50:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.896+0000 7fc27bb9d740 1 Processor -- start 2024-04-21T08:50:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.896+0000 7fc27bb9d740 1 -- start start 2024-04-21T08:50:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.896+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 0x561ea5e22d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.896+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e16540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.896+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e16a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.896+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741ad410 con 0x561ea5e0db60 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.896+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741afc90 con 0x561ea5e0e8d0 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.896+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741ada80 con 0x561ea5e170e0 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc27259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 0x561ea5e22d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc27259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 0x561ea5e22d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41988/0 (socket says 172.21.15.7:41988) 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc27259c640 1 -- 172.21.15.7:0/520456980 learned_addr learned my addr 172.21.15.7:0/520456980 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc271d9b640 1 --2- 172.21.15.7:0/520456980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e16540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc272d9d640 1 --2- 172.21.15.7:0/520456980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e16a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc27259c640 1 -- 172.21.15.7:0/520456980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 msgr2=0x561ea5e16a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc27259c640 1 --2- 172.21.15.7:0/520456980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e16a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc27259c640 1 -- 172.21.15.7:0/520456980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 msgr2=0x561ea5e16540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc27259c640 1 --2- 172.21.15.7:0/520456980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e16540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc27259c640 1 -- 172.21.15.7:0/520456980 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea5e0d990 con 0x561ea5e0db60 2024-04-21T08:50:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc27259c640 1 --2- 172.21.15.7:0/520456980 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 0x561ea5e22d10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc264011ec0 tx=0x7fc26400ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c33b3109ba89a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc25bfff640 1 -- 172.21.15.7:0/520456980 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26402c070 con 0x561ea5e0db60 2024-04-21T08:50:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc25bfff640 1 -- 172.21.15.7:0/520456980 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc264003970 con 0x561ea5e0db60 2024-04-21T08:50:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.897+0000 7fc25bfff640 1 -- 172.21.15.7:0/520456980 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26401e040 con 0x561ea5e0db60 2024-04-21T08:50:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.898+0000 7fc27bb9d740 1 -- 172.21.15.7:0/520456980 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 msgr2=0x561ea5e22d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.898+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/520456980 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 0x561ea5e22d10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc264011ec0 tx=0x7fc26400ff30 comp rx=0 tx=0).stop 2024-04-21T08:50:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.898+0000 7fc27bb9d740 1 -- 172.21.15.7:0/520456980 shutdown_connections 2024-04-21T08:50:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.898+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/520456980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e170e0 0x561ea5e16a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.898+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/520456980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e16540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.898+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/520456980 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e0db60 0x561ea5e22d10 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.898+0000 7fc27bb9d740 1 -- 172.21.15.7:0/520456980 >> 172.21.15.7:0/520456980 conn(0x561ea5dd4c90 msgr2=0x561ea5e0efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.898+0000 7fc27bb9d740 1 -- 172.21.15.7:0/520456980 shutdown_connections 2024-04-21T08:50:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.898+0000 7fc27bb9d740 1 -- 172.21.15.7:0/520456980 wait complete. 2024-04-21T08:50:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.898+0000 7fc27bb9d740 1 -- start start 2024-04-21T08:50:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.898+0000 7fc27bb9d740 1 Processor -- start 2024-04-21T08:50:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e0db60 0x561ea5e1abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e1b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc27bb9d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e170e0 0x561ea5e1b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741ae150 con 0x561ea5e170e0 2024-04-21T08:50:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741af5c0 con 0x561ea5e0e8d0 2024-04-21T08:50:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc27bb9d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc2741ae820 con 0x561ea5e0db60 2024-04-21T08:50:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc27bb9d740 10 client.0 _ll_register_callbacks cb 0x7fc268098ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc272d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e170e0 0x561ea5e1b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc272d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e170e0 0x561ea5e1b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42004/0 (socket says 172.21.15.7:42004) 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc272d9d640 1 -- 172.21.15.7:0/1037405229 learned_addr learned my addr 172.21.15.7:0/1037405229 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc271d9b640 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e1b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc27259c640 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e0db60 0x561ea5e1abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc272d9d640 1 -- 172.21.15.7:0/1037405229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e0db60 msgr2=0x561ea5e1abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc272d9d640 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e0db60 0x561ea5e1abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc272d9d640 1 -- 172.21.15.7:0/1037405229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 msgr2=0x561ea5e1b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc272d9d640 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e1b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc272d9d640 1 -- 172.21.15.7:0/1037405229 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea5e8c6c0 con 0x561ea5e170e0 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc271d9b640 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e1b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:21.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc272d9d640 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e170e0 0x561ea5e1b660 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc26c035000 tx=0x7fc26c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c54c31ab057a5ee server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc259ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc259ffb640 1 -- 172.21.15.7:0/1037405229 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26c052020 con 0x561ea5e170e0 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc259ffb640 1 -- 172.21.15.7:0/1037405229 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc26c005260 con 0x561ea5e170e0 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1037405229 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc264003970 con 0x561ea5e170e0 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc259ffb640 1 -- 172.21.15.7:0/1037405229 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc26c005460 con 0x561ea5e170e0 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc27bb9d740 10 client.16818 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.899+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1037405229 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561ea5e8d970 con 0x561ea5e170e0 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.900+0000 7fc259ffb640 1 -- 172.21.15.7:0/1037405229 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc264003970 con 0x561ea5e170e0 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.900+0000 7fc259ffb640 1 -- 172.21.15.7:0/1037405229 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fc26c003710 con 0x561ea5e170e0 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.900+0000 7fc27bb9d740 10 client.16818 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.900+0000 7fc27bb9d740 10 client.16818 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.900+0000 7fc27bb9d740 20 client.16818 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:21.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.900+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ea5e8e170 0x561ea5e90590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.900+0000 7fc27bb9d740 4 client.16818 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:50:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.900+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1037405229 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561ea5d123d0 con 0x561ea5e8e170 2024-04-21T08:50:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.900+0000 7fc27259c640 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ea5e8e170 0x561ea5e90590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.900+0000 7fc259ffb640 10 client.16818 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:21.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:21.900+0000 7fc27259c640 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ea5e8e170 0x561ea5e90590 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e980322218b9a57a server_cookie=b491dade552be63a in_seq=0 out_seq=0 2024-04-21T08:50:23.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:22 smithi007 ceph-mon[28477]: pgmap v174: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-04-21T08:50:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:22 smithi016 ceph-mon[31259]: pgmap v174: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-04-21T08:50:23.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:22 smithi077 ceph-mon[31615]: pgmap v174: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 120 op/s 2024-04-21T08:50:25.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:24 smithi007 ceph-mon[28477]: pgmap v175: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 267 op/s 2024-04-21T08:50:25.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:24 smithi016 ceph-mon[31259]: pgmap v175: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 267 op/s 2024-04-21T08:50:25.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:24 smithi077 ceph-mon[31615]: pgmap v175: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 267 op/s 2024-04-21T08:50:27.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:26 smithi007 ceph-mon[28477]: pgmap v176: 129 pgs: 129 active+clean; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.8 MiB/s wr, 559 op/s 2024-04-21T08:50:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.182+0000 7fc259ffb640 1 -- 172.21.15.7:0/1037405229 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561ea5d123d0 con 0x561ea5e8e170 2024-04-21T08:50:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.182+0000 7fc259ffb640 10 client.16818 handle_command_reply: tid=0 2024-04-21T08:50:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.182+0000 7fc27bb9d740 1 client.16818 shutdown 2024-04-21T08:50:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.182+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1037405229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e170e0 msgr2=0x561ea5e1b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.182+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e170e0 0x561ea5e1b660 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc26c035000 tx=0x7fc26c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.182+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1037405229 shutdown_connections 2024-04-21T08:50:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.182+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ea5e8e170 0x561ea5e90590 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.182+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ea5e170e0 0x561ea5e1b660 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.182+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea5e0e8d0 0x561ea5e1b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc259ffb640 0 client.16818 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc27bb9d740 1 --2- 172.21.15.7:0/1037405229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ea5e0db60 0x561ea5e1abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1037405229 >> 172.21.15.7:0/1037405229 conn(0x561ea5dd4c90 msgr2=0x561ea5e07040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1037405229 shutdown_connections 2024-04-21T08:50:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc27bb9d740 1 -- 172.21.15.7:0/1037405229 wait complete. 2024-04-21T08:50:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc27bb9d740 20 client.16818 trim_cache size 0 max 16384 2024-04-21T08:50:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:50:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:50:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:50:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:50:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 -- 172.21.15.7:0/1318825974 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc25c0a3d10 msgr2=0x7fc25c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc25c0a3d10 0x7fc25c0a61d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc26c006010 tx=0x7fc26c02c730 comp rx=0 tx=0).stop 2024-04-21T08:50:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 -- 172.21.15.7:0/1318825974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274106170 msgr2=0x7fc274198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274106170 0x7fc274198c20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc26800e9c0 tx=0x7fc26800ee80 comp rx=0 tx=0).stop 2024-04-21T08:50:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 -- 172.21.15.7:0/1318825974 shutdown_connections 2024-04-21T08:50:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc25c0a3d10 0x7fc25c0a61d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc27419cfa0 0x7fc27419d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc274106170 0x7fc274198c20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 --2- 172.21.15.7:0/1318825974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2741010a0 0x7fc2741986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 -- 172.21.15.7:0/1318825974 >> 172.21.15.7:0/1318825974 conn(0x7fc274077f70 msgr2=0x7fc274069ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 -- 172.21.15.7:0/1318825974 shutdown_connections 2024-04-21T08:50:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:27.183+0000 7fc278fab640 1 -- 172.21.15.7:0/1318825974 wait complete. 2024-04-21T08:50:27.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:26 smithi016 ceph-mon[31259]: pgmap v176: 129 pgs: 129 active+clean; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.8 MiB/s wr, 559 op/s 2024-04-21T08:50:27.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:26 smithi077 ceph-mon[31615]: pgmap v176: 129 pgs: 129 active+clean; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.8 MiB/s wr, 559 op/s 2024-04-21T08:50:29.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:28 smithi007 ceph-mon[28477]: pgmap v177: 129 pgs: 129 active+clean; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.8 MiB/s wr, 552 op/s 2024-04-21T08:50:29.155 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:28 smithi077 ceph-mon[31615]: pgmap v177: 129 pgs: 129 active+clean; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.8 MiB/s wr, 552 op/s 2024-04-21T08:50:29.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:28 smithi016 ceph-mon[31259]: pgmap v177: 129 pgs: 129 active+clean; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.8 MiB/s wr, 552 op/s 2024-04-21T08:50:31.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:30 smithi007 ceph-mon[28477]: pgmap v178: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.3 MiB/s wr, 1.38k op/s 2024-04-21T08:50:31.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:30 smithi016 ceph-mon[31259]: pgmap v178: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.3 MiB/s wr, 1.38k op/s 2024-04-21T08:50:31.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:30 smithi077 ceph-mon[31615]: pgmap v178: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.3 MiB/s wr, 1.38k op/s 2024-04-21T08:50:33.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:32 smithi007 ceph-mon[28477]: pgmap v179: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.8 MiB/s wr, 1.83k op/s 2024-04-21T08:50:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:32 smithi016 ceph-mon[31259]: pgmap v179: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.8 MiB/s wr, 1.83k op/s 2024-04-21T08:50:33.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:32 smithi077 ceph-mon[31615]: pgmap v179: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.8 MiB/s wr, 1.83k op/s 2024-04-21T08:50:35.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:34 smithi007 ceph-mon[28477]: pgmap v180: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.0 MiB/s wr, 2.70k op/s 2024-04-21T08:50:35.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:34 smithi016 ceph-mon[31259]: pgmap v180: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.0 MiB/s wr, 2.70k op/s 2024-04-21T08:50:35.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:34 smithi077 ceph-mon[31615]: pgmap v180: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.0 MiB/s wr, 2.70k op/s 2024-04-21T08:50:37.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:36 smithi007 ceph-mon[28477]: pgmap v181: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 3.80k op/s 2024-04-21T08:50:37.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:36 smithi016 ceph-mon[31259]: pgmap v181: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 3.80k op/s 2024-04-21T08:50:37.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:36 smithi077 ceph-mon[31615]: pgmap v181: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 3.80k op/s 2024-04-21T08:50:39.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:38 smithi007 ceph-mon[28477]: pgmap v182: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 742 KiB/s wr, 3.46k op/s 2024-04-21T08:50:39.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:38 smithi016 ceph-mon[31259]: pgmap v182: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 742 KiB/s wr, 3.46k op/s 2024-04-21T08:50:39.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:38 smithi077 ceph-mon[31615]: pgmap v182: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 742 KiB/s wr, 3.46k op/s 2024-04-21T08:50:40.345 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.346+0000 7f3d33aa5640 1 -- 172.21.15.7:0/3138942438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d2c106e90 msgr2=0x7f3d2c06a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:40.346 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.346+0000 7f3d33aa5640 1 --2- 172.21.15.7:0/3138942438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d2c106e90 0x7f3d2c06a5d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009ca0 tx=0x7f3d2402c140 comp rx=0 tx=0).stop 2024-04-21T08:50:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.353+0000 7f3d33aa5640 1 -- 172.21.15.7:0/3138942438 shutdown_connections 2024-04-21T08:50:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.353+0000 7f3d33aa5640 1 --2- 172.21.15.7:0/3138942438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d2c104990 0x7f3d2c104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.353+0000 7f3d33aa5640 1 --2- 172.21.15.7:0/3138942438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d2c107300 0x7f3d2c06ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.353+0000 7f3d33aa5640 1 --2- 172.21.15.7:0/3138942438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d2c106e90 0x7f3d2c06a5d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.353+0000 7f3d33aa5640 1 -- 172.21.15.7:0/3138942438 >> 172.21.15.7:0/3138942438 conn(0x7f3d2c077c70 msgr2=0x7f3d2c078080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.359+0000 7f3d33aa5640 1 -- 172.21.15.7:0/3138942438 shutdown_connections 2024-04-21T08:50:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.360+0000 7f3d33aa5640 1 -- 172.21.15.7:0/3138942438 wait complete. 2024-04-21T08:50:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.361+0000 7f3d33aa5640 1 Processor -- start 2024-04-21T08:50:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.365+0000 7f3d33aa5640 1 -- start start 2024-04-21T08:50:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.365+0000 7f3d33aa5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d2c104990 0x7f3d2c1a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.365+0000 7f3d33aa5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d2c106e90 0x7f3d2c1a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.365+0000 7f3d33aa5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d2c107300 0x7f3d2c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.365+0000 7f3d33aa5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c118750 con 0x7f3d2c106e90 2024-04-21T08:50:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.365+0000 7f3d33aa5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c1185d0 con 0x7f3d2c104990 2024-04-21T08:50:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.365+0000 7f3d33aa5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c1188d0 con 0x7f3d2c107300 2024-04-21T08:50:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.365+0000 7f3d31040640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d2c106e90 0x7f3d2c1a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.365+0000 7f3d31841640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d2c104990 0x7f3d2c1a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.365+0000 7f3d31841640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d2c104990 0x7f3d2c1a4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59908/0 (socket says 172.21.15.7:59908) 2024-04-21T08:50:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.365+0000 7f3d31841640 1 -- 172.21.15.7:0/1895311681 learned_addr learned my addr 172.21.15.7:0/1895311681 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.367+0000 7f3d32042640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d2c107300 0x7f3d2c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.368+0000 7f3d31040640 1 -- 172.21.15.7:0/1895311681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d2c107300 msgr2=0x7f3d2c19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.368+0000 7f3d31040640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d2c107300 0x7f3d2c19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.368+0000 7f3d31040640 1 -- 172.21.15.7:0/1895311681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d2c104990 msgr2=0x7f3d2c1a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.368+0000 7f3d31040640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d2c104990 0x7f3d2c1a4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.368+0000 7f3d31040640 1 -- 172.21.15.7:0/1895311681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2c19f340 con 0x7f3d2c106e90 2024-04-21T08:50:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.369+0000 7f3d31040640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d2c106e90 0x7f3d2c1a4fb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00d6d0 tx=0x7f3d1c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.370+0000 7f3d22ffd640 1 -- 172.21.15.7:0/1895311681 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c017070 con 0x7f3d2c106e90 2024-04-21T08:50:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.370+0000 7f3d22ffd640 1 -- 172.21.15.7:0/1895311681 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d1c009820 con 0x7f3d2c106e90 2024-04-21T08:50:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.370+0000 7f3d22ffd640 1 -- 172.21.15.7:0/1895311681 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c004c40 con 0x7f3d2c106e90 2024-04-21T08:50:40.375 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.375+0000 7f3d33aa5640 1 -- 172.21.15.7:0/1895311681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2c19f630 con 0x7f3d2c106e90 2024-04-21T08:50:40.375 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.375+0000 7f3d33aa5640 1 -- 172.21.15.7:0/1895311681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d2c107770 con 0x7f3d2c106e90 2024-04-21T08:50:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.378+0000 7f3d36771740 1 Processor -- start 2024-04-21T08:50:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.379+0000 7f3d36771740 1 -- start start 2024-04-21T08:50:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.379+0000 7f3d36771740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f528d0 0x55e419f51620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.379+0000 7f3d36771740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f51b60 0x55e419f51fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.379+0000 7f3d36771740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f6c0f0 0x55e419f6e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.379+0000 7f3d36771740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e419eb68d0 con 0x55e419f6c0f0 2024-04-21T08:50:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.379+0000 7f3d36771740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e419e53e80 con 0x55e419f528d0 2024-04-21T08:50:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.379+0000 7f3d36771740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e419eb2890 con 0x55e419f51b60 2024-04-21T08:50:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.381+0000 7f3d31841640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f51b60 0x55e419f51fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.381+0000 7f3d31841640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f51b60 0x55e419f51fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:43546/0 (socket says 172.21.15.7:43546) 2024-04-21T08:50:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.381+0000 7f3d31841640 1 -- 172.21.15.7:0/2156167811 learned_addr learned my addr 172.21.15.7:0/2156167811 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.383+0000 7f3d32042640 1 --2- 172.21.15.7:0/2156167811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f528d0 0x55e419f51620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.384+0000 7f3d22ffd640 1 -- 172.21.15.7:0/1895311681 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f3d1c0099d0 con 0x7f3d2c106e90 2024-04-21T08:50:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.384+0000 7f3d31841640 1 -- 172.21.15.7:0/2156167811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f528d0 msgr2=0x55e419f51620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.384+0000 7f3d31841640 1 --2- 172.21.15.7:0/2156167811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f528d0 0x55e419f51620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.384+0000 7f3d31841640 1 -- 172.21.15.7:0/2156167811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f6c0f0 msgr2=0x55e419f6e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.384+0000 7f3d31841640 1 --2- 172.21.15.7:0/2156167811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f6c0f0 0x55e419f6e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.384+0000 7f3d31841640 1 -- 172.21.15.7:0/2156167811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e419e75320 con 0x55e419f51b60 2024-04-21T08:50:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.385+0000 7f3d22ffd640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3d180a3f20 0x7f3d180a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.385+0000 7f3d22ffd640 1 -- 172.21.15.7:0/1895311681 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f3d1c0d0700 con 0x7f3d2c106e90 2024-04-21T08:50:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.388+0000 7f3d32042640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3d180a3f20 0x7f3d180a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.389+0000 7f3d31841640 1 --2- 172.21.15.7:0/2156167811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f51b60 0x55e419f51fe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3d24037040 tx=0x7f3d24032420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd5a2a4d1fb95672 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.391+0000 7f3d17fff640 1 -- 172.21.15.7:0/2156167811 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24043070 con 0x55e419f51b60 2024-04-21T08:50:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.392+0000 7f3d17fff640 1 -- 172.21.15.7:0/2156167811 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d24032970 con 0x55e419f51b60 2024-04-21T08:50:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.393+0000 7f3d36771740 1 -- 172.21.15.7:0/2156167811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f51b60 msgr2=0x55e419f51fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.393+0000 7f3d36771740 1 --2- 172.21.15.7:0/2156167811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f51b60 0x55e419f51fe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3d24037040 tx=0x7f3d24032420 comp rx=0 tx=0).stop 2024-04-21T08:50:40.397 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.396+0000 7f3d36771740 1 -- 172.21.15.7:0/2156167811 shutdown_connections 2024-04-21T08:50:40.397 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.396+0000 7f3d36771740 1 --2- 172.21.15.7:0/2156167811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f6c0f0 0x55e419f6e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.397 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.396+0000 7f3d36771740 1 --2- 172.21.15.7:0/2156167811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f51b60 0x55e419f51fe0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.397 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.396+0000 7f3d36771740 1 --2- 172.21.15.7:0/2156167811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f528d0 0x55e419f51620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.397 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.396+0000 7f3d36771740 1 -- 172.21.15.7:0/2156167811 >> 172.21.15.7:0/2156167811 conn(0x55e419f18c90 msgr2=0x55e419f4c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:40.398 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.398+0000 7f3d32042640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3d180a3f20 0x7f3d180a63e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800aab0 tx=0x7f3d28005d20 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.401+0000 7f3d36771740 1 -- 172.21.15.7:0/2156167811 shutdown_connections 2024-04-21T08:50:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.402+0000 7f3d36771740 1 -- 172.21.15.7:0/2156167811 wait complete. 2024-04-21T08:50:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.402+0000 7f3d36771740 1 -- start start 2024-04-21T08:50:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.402+0000 7f3d36771740 1 Processor -- start 2024-04-21T08:50:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.403+0000 7f3d36771740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f51b60 0x55e419f6b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.403+0000 7f3d36771740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f528d0 0x55e419f68d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.403+0000 7f3d36771740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 0x55e419f692b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.403+0000 7f3d36771740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e419f6ee70 con 0x55e419f528d0 2024-04-21T08:50:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.403+0000 7f3d36771740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e419e3eeb0 con 0x55e419f51b60 2024-04-21T08:50:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.403+0000 7f3d36771740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e419f6eff0 con 0x55e419f6c0f0 2024-04-21T08:50:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.403+0000 7f3d36771740 10 client.0 _ll_register_callbacks cb 0x55e419e6bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.403+0000 7f3d32042640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 0x55e419f692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.403+0000 7f3d32042640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 0x55e419f692b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:43550/0 (socket says 172.21.15.7:43550) 2024-04-21T08:50:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.403+0000 7f3d32042640 1 -- 172.21.15.7:0/2728160109 learned_addr learned my addr 172.21.15.7:0/2728160109 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.403+0000 7f3d31841640 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f51b60 0x55e419f6b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.405 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.405+0000 7f3d31040640 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f528d0 0x55e419f68d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.409+0000 7f3d32042640 1 -- 172.21.15.7:0/2728160109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f51b60 msgr2=0x55e419f6b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.409+0000 7f3d32042640 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f51b60 0x55e419f6b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.409+0000 7f3d32042640 1 -- 172.21.15.7:0/2728160109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f528d0 msgr2=0x55e419f68d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.409+0000 7f3d32042640 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f528d0 0x55e419f68d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.409+0000 7f3d32042640 1 -- 172.21.15.7:0/2728160109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e419bc9210 con 0x55e419f6c0f0 2024-04-21T08:50:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.409+0000 7f3d32042640 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 0x55e419f692b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d280116a0 tx=0x7f3d28004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d939e197694160b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.410+0000 7f3d157fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:50:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.414+0000 7f3d157fa640 1 -- 172.21.15.7:0/2728160109 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d28022070 con 0x55e419f6c0f0 2024-04-21T08:50:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.414+0000 7f3d36771740 1 -- 172.21.15.7:0/2728160109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d24032970 con 0x55e419f6c0f0 2024-04-21T08:50:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.414+0000 7f3d157fa640 1 -- 172.21.15.7:0/2728160109 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f3d28004220 con 0x55e419f6c0f0 2024-04-21T08:50:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.418+0000 7f3d157fa640 1 -- 172.21.15.7:0/2728160109 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3d2802b050 con 0x55e419f6c0f0 2024-04-21T08:50:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.418+0000 7f3d36771740 10 client.26353 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.418+0000 7f3d36771740 1 -- 172.21.15.7:0/2728160109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e419c95620 con 0x55e419f6c0f0 2024-04-21T08:50:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.423+0000 7f3d157fa640 1 -- 172.21.15.7:0/2728160109 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d24032970 con 0x55e419f6c0f0 2024-04-21T08:50:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.423+0000 7f3d157fa640 1 -- 172.21.15.7:0/2728160109 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f3d28003940 con 0x55e419f6c0f0 2024-04-21T08:50:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.423+0000 7f3d36771740 10 client.26353 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:40.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.423+0000 7f3d36771740 10 client.26353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:40.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.423+0000 7f3d36771740 20 client.26353 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:40.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.423+0000 7f3d36771740 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e419f7c5c0 0x55e419f7ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.423+0000 7f3d36771740 4 client.26353 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:40.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.423+0000 7f3d36771740 1 -- 172.21.15.7:0/2728160109 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e419eca880 con 0x55e419f7c5c0 2024-04-21T08:50:40.427 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.426+0000 7f3d31841640 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e419f7c5c0 0x55e419f7ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.430+0000 7f3d31841640 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e419f7c5c0 0x55e419f7ea20 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74040d5f9c057b5a server_cookie=3f0cb884bc1a296b in_seq=0 out_seq=0 2024-04-21T08:50:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.430+0000 7f3d157fa640 10 client.26353 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.437+0000 7f3d157fa640 1 -- 172.21.15.7:0/2728160109 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e419eca880 con 0x55e419f7c5c0 2024-04-21T08:50:40.436 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.437+0000 7f3d157fa640 10 client.26353 handle_command_reply: tid=0 2024-04-21T08:50:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.437+0000 7f3d36771740 1 client.26353 shutdown 2024-04-21T08:50:40.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.443+0000 7f3d36771740 1 -- 172.21.15.7:0/2728160109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 msgr2=0x55e419f692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:40.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.443+0000 7f3d36771740 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 0x55e419f692b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d280116a0 tx=0x7f3d28004730 comp rx=0 tx=0).stop 2024-04-21T08:50:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.449+0000 7f3d36771740 1 -- 172.21.15.7:0/2728160109 shutdown_connections 2024-04-21T08:50:40.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.449+0000 7f3d36771740 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e419f7c5c0 0x55e419f7ea20 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.449+0000 7f3d36771740 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 0x55e419f692b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.449+0000 7f3d36771740 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f528d0 0x55e419f68d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.449+0000 7f3d36771740 1 --2- 172.21.15.7:0/2728160109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f51b60 0x55e419f6b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.449+0000 7f3d36771740 1 -- 172.21.15.7:0/2728160109 >> 172.21.15.7:0/2728160109 conn(0x55e419f18c90 msgr2=0x55e419f4e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:40.450 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.450+0000 7f3d157fa640 0 client.26353 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.451+0000 7f3d36771740 1 -- 172.21.15.7:0/2728160109 shutdown_connections 2024-04-21T08:50:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.452+0000 7f3d36771740 1 -- 172.21.15.7:0/2728160109 wait complete. 2024-04-21T08:50:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.454+0000 7f3d36771740 20 client.26353 trim_cache size 0 max 16384 2024-04-21T08:50:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.487+0000 7f3d36771740 1 Processor -- start 2024-04-21T08:50:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.490+0000 7f3d36771740 1 -- start start 2024-04-21T08:50:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.490+0000 7f3d36771740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f51b60 0x55e419f77dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.490+0000 7f3d36771740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f528d0 0x55e419f6b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.490+0000 7f3d36771740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 0x55e419f6b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.490+0000 7f3d36771740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c19fff0 con 0x55e419f51b60 2024-04-21T08:50:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.490+0000 7f3d36771740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c10ae30 con 0x55e419f528d0 2024-04-21T08:50:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.490+0000 7f3d36771740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c108c40 con 0x55e419f6c0f0 2024-04-21T08:50:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.490+0000 7f3d31841640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f51b60 0x55e419f77dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.490+0000 7f3d31841640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f51b60 0x55e419f77dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37640/0 (socket says 172.21.15.7:37640) 2024-04-21T08:50:40.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.490+0000 7f3d31841640 1 -- 172.21.15.7:0/1774899822 learned_addr learned my addr 172.21.15.7:0/1774899822 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.492+0000 7f3d32042640 1 --2- 172.21.15.7:0/1774899822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 0x55e419f6b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.492+0000 7f3d31841640 1 -- 172.21.15.7:0/1774899822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 msgr2=0x55e419f6b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.492+0000 7f3d31841640 1 --2- 172.21.15.7:0/1774899822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 0x55e419f6b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.492+0000 7f3d31841640 1 -- 172.21.15.7:0/1774899822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f528d0 msgr2=0x55e419f6b4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:50:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.492+0000 7f3d31841640 1 --2- 172.21.15.7:0/1774899822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f528d0 0x55e419f6b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.492+0000 7f3d31841640 1 -- 172.21.15.7:0/1774899822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e419f47f30 con 0x55e419f51b60 2024-04-21T08:50:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.493+0000 7f3d31841640 1 --2- 172.21.15.7:0/1774899822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f51b60 0x55e419f77dc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f3d24003e50 tx=0x7f3d24003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84301f50d61ba04 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.497+0000 7f3d177fe640 1 -- 172.21.15.7:0/1774899822 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24043040 con 0x55e419f51b60 2024-04-21T08:50:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.497+0000 7f3d177fe640 1 -- 172.21.15.7:0/1774899822 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d24032970 con 0x55e419f51b60 2024-04-21T08:50:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.497+0000 7f3d177fe640 1 -- 172.21.15.7:0/1774899822 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24004530 con 0x55e419f51b60 2024-04-21T08:50:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.498+0000 7f3d36771740 1 -- 172.21.15.7:0/1774899822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f51b60 msgr2=0x55e419f77dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.498+0000 7f3d36771740 1 --2- 172.21.15.7:0/1774899822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f51b60 0x55e419f77dc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f3d24003e50 tx=0x7f3d24003e80 comp rx=0 tx=0).stop 2024-04-21T08:50:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.503+0000 7f3d36771740 1 -- 172.21.15.7:0/1774899822 shutdown_connections 2024-04-21T08:50:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.503+0000 7f3d36771740 1 --2- 172.21.15.7:0/1774899822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f6c0f0 0x55e419f6b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.503+0000 7f3d36771740 1 --2- 172.21.15.7:0/1774899822 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f528d0 0x55e419f6b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.504 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.503+0000 7f3d36771740 1 --2- 172.21.15.7:0/1774899822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f51b60 0x55e419f77dc0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.504 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.503+0000 7f3d36771740 1 -- 172.21.15.7:0/1774899822 >> 172.21.15.7:0/1774899822 conn(0x55e419f18c90 msgr2=0x55e419f64090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:40.505 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.506+0000 7f3d36771740 1 -- 172.21.15.7:0/1774899822 shutdown_connections 2024-04-21T08:50:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.510+0000 7f3d36771740 1 -- 172.21.15.7:0/1774899822 wait complete. 2024-04-21T08:50:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.510+0000 7f3d36771740 1 -- start start 2024-04-21T08:50:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.510+0000 7f3d36771740 1 Processor -- start 2024-04-21T08:50:40.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.512+0000 7f3d36771740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f51b60 0x55e419f742d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.513+0000 7f3d36771740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f528d0 0x55e419f74810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.513+0000 7f3d36771740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f6c0f0 0x55e419f66010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.513+0000 7f3d36771740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c1092f0 con 0x55e419f51b60 2024-04-21T08:50:40.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.513+0000 7f3d36771740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c10a760 con 0x55e419f6c0f0 2024-04-21T08:50:40.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.513+0000 7f3d36771740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c1099c0 con 0x55e419f528d0 2024-04-21T08:50:40.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.513+0000 7f3d36771740 10 client.0 _ll_register_callbacks cb 0x7f3d28015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.515+0000 7f3d31040640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f528d0 0x55e419f74810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.516+0000 7f3d31040640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f528d0 0x55e419f74810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:43562/0 (socket says 172.21.15.7:43562) 2024-04-21T08:50:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.516+0000 7f3d31040640 1 -- 172.21.15.7:0/762370625 learned_addr learned my addr 172.21.15.7:0/762370625 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.516+0000 7f3d31040640 1 -- 172.21.15.7:0/762370625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f6c0f0 msgr2=0x55e419f66010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:50:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.516+0000 7f3d31040640 1 --2- 172.21.15.7:0/762370625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f6c0f0 0x55e419f66010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.516+0000 7f3d31040640 1 -- 172.21.15.7:0/762370625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f51b60 msgr2=0x55e419f742d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.516+0000 7f3d31040640 1 --2- 172.21.15.7:0/762370625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f51b60 0x55e419f742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.516+0000 7f3d31040640 1 -- 172.21.15.7:0/762370625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e419f66840 con 0x55e419f528d0 2024-04-21T08:50:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.517+0000 7f3d31040640 1 --2- 172.21.15.7:0/762370625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f528d0 0x55e419f74810 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c098ca0 tx=0x7f3d1c09bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c30a5abead94310 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.517+0000 7f3d157fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:50:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.517+0000 7f3d157fa640 1 -- 172.21.15.7:0/762370625 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c09c9d0 con 0x55e419f528d0 2024-04-21T08:50:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.517+0000 7f3d36771740 1 -- 172.21.15.7:0/762370625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d24032970 con 0x55e419f528d0 2024-04-21T08:50:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.517+0000 7f3d157fa640 1 -- 172.21.15.7:0/762370625 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f3d1c0cdc80 con 0x55e419f528d0 2024-04-21T08:50:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.518+0000 7f3d157fa640 1 -- 172.21.15.7:0/762370625 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3d1c0cde80 con 0x55e419f528d0 2024-04-21T08:50:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.519+0000 7f3d36771740 10 client.26356 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.519+0000 7f3d36771740 1 -- 172.21.15.7:0/762370625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e419f6cfc0 con 0x55e419f528d0 2024-04-21T08:50:40.525 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.525+0000 7f3d157fa640 1 -- 172.21.15.7:0/762370625 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d24032970 con 0x55e419f528d0 2024-04-21T08:50:40.525 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.526+0000 7f3d157fa640 1 -- 172.21.15.7:0/762370625 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f3d1c0d0340 con 0x55e419f528d0 2024-04-21T08:50:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.528+0000 7f3d36771740 10 client.26356 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.528+0000 7f3d36771740 10 client.26356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.528+0000 7f3d36771740 20 client.26356 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.528+0000 7f3d36771740 1 --2- 172.21.15.7:0/762370625 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e419fe2db0 0x55e419fe51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.528+0000 7f3d36771740 4 client.26356 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:50:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.528+0000 7f3d36771740 1 -- 172.21.15.7:0/762370625 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e419e563d0 con 0x55e419fe2db0 2024-04-21T08:50:40.529 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.529+0000 7f3d31841640 1 --2- 172.21.15.7:0/762370625 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e419fe2db0 0x55e419fe51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:40.531 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.532+0000 7f3d31841640 1 --2- 172.21.15.7:0/762370625 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e419fe2db0 0x55e419fe51d0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f442a4feb5b8cd9 server_cookie=95307549ec0929f7 in_seq=0 out_seq=0 2024-04-21T08:50:40.532 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:40.532+0000 7f3d157fa640 10 client.26356 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:40 smithi007 ceph-mon[28477]: pgmap v183: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 4.99k op/s 2024-04-21T08:50:41.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:40 smithi016 ceph-mon[31259]: pgmap v183: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 4.99k op/s 2024-04-21T08:50:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:40 smithi077 ceph-mon[31615]: pgmap v183: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 4.99k op/s 2024-04-21T08:50:43.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:42 smithi007 ceph-mon[28477]: pgmap v184: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 992 KiB/s wr, 5.07k op/s 2024-04-21T08:50:43.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:42 smithi016 ceph-mon[31259]: pgmap v184: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 992 KiB/s wr, 5.07k op/s 2024-04-21T08:50:43.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:42 smithi077 ceph-mon[31615]: pgmap v184: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 992 KiB/s wr, 5.07k op/s 2024-04-21T08:50:45.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:44 smithi007 ceph-mon[28477]: pgmap v185: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.2 MiB/s wr, 4.77k op/s 2024-04-21T08:50:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:44 smithi016 ceph-mon[31259]: pgmap v185: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.2 MiB/s wr, 4.77k op/s 2024-04-21T08:50:45.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:44 smithi077 ceph-mon[31615]: pgmap v185: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 2.2 MiB/s wr, 4.77k op/s 2024-04-21T08:50:45.511 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.510+0000 7f3d157fa640 1 -- 172.21.15.7:0/762370625 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e419e563d0 con 0x55e419fe2db0 2024-04-21T08:50:45.511 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.510+0000 7f3d157fa640 10 client.26356 handle_command_reply: tid=0 2024-04-21T08:50:45.511 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.510+0000 7f3d36771740 1 client.26356 shutdown 2024-04-21T08:50:45.511 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d36771740 1 -- 172.21.15.7:0/762370625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f528d0 msgr2=0x55e419f74810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:45.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d36771740 1 --2- 172.21.15.7:0/762370625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f528d0 0x55e419f74810 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c098ca0 tx=0x7f3d1c09bed0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d36771740 1 -- 172.21.15.7:0/762370625 shutdown_connections 2024-04-21T08:50:45.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d36771740 1 --2- 172.21.15.7:0/762370625 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e419fe2db0 0x55e419fe51d0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d36771740 1 --2- 172.21.15.7:0/762370625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e419f6c0f0 0x55e419f66010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d36771740 1 --2- 172.21.15.7:0/762370625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e419f528d0 0x55e419f74810 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d157fa640 0 client.26356 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:45.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d36771740 1 --2- 172.21.15.7:0/762370625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e419f51b60 0x55e419f742d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d36771740 1 -- 172.21.15.7:0/762370625 >> 172.21.15.7:0/762370625 conn(0x55e419f18c90 msgr2=0x55e419f4b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:45.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d36771740 1 -- 172.21.15.7:0/762370625 shutdown_connections 2024-04-21T08:50:45.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d36771740 1 -- 172.21.15.7:0/762370625 wait complete. 2024-04-21T08:50:45.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.511+0000 7f3d36771740 20 client.26356 trim_cache size 0 max 16384 2024-04-21T08:50:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:50:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:50:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:50:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:50:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 -- 172.21.15.7:0/1895311681 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3d180a3f20 msgr2=0x7f3d180a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3d180a3f20 0x7f3d180a63e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800aab0 tx=0x7f3d28005d20 comp rx=0 tx=0).stop 2024-04-21T08:50:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 -- 172.21.15.7:0/1895311681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d2c106e90 msgr2=0x7f3d2c1a4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d2c106e90 0x7f3d2c1a4fb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00d6d0 tx=0x7f3d1c00db90 comp rx=0 tx=0).stop 2024-04-21T08:50:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 -- 172.21.15.7:0/1895311681 shutdown_connections 2024-04-21T08:50:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3d180a3f20 0x7f3d180a63e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d2c107300 0x7f3d2c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d2c106e90 0x7f3d2c1a4fb0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 --2- 172.21.15.7:0/1895311681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d2c104990 0x7f3d2c1a4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 -- 172.21.15.7:0/1895311681 >> 172.21.15.7:0/1895311681 conn(0x7f3d2c077c70 msgr2=0x7f3d2c103830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 -- 172.21.15.7:0/1895311681 shutdown_connections 2024-04-21T08:50:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:45.512+0000 7f3d33aa5640 1 -- 172.21.15.7:0/1895311681 wait complete. 2024-04-21T08:50:47.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:46 smithi007 ceph-mon[28477]: pgmap v186: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.1 MiB/s wr, 4.44k op/s 2024-04-21T08:50:47.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:46 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:46 smithi016 ceph-mon[31259]: pgmap v186: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.1 MiB/s wr, 4.44k op/s 2024-04-21T08:50:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:46 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:47.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:46 smithi077 ceph-mon[31615]: pgmap v186: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.1 MiB/s wr, 4.44k op/s 2024-04-21T08:50:47.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:46 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:48.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:47 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:50:48.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:47 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:50:48.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:47 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:50:49.305 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:48 smithi016 ceph-mon[31259]: pgmap v187: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 3.19k op/s 2024-04-21T08:50:49.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:48 smithi077 ceph-mon[31615]: pgmap v187: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 3.19k op/s 2024-04-21T08:50:49.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:48 smithi007 ceph-mon[28477]: pgmap v187: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 3.19k op/s 2024-04-21T08:50:49.435 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:50:49.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:50:49.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.552+0000 7fa0cb29a640 1 -- 172.21.15.7:0/3273674263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c4107240 msgr2=0x7fa0c410ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.552+0000 7fa0cb29a640 1 --2- 172.21.15.7:0/3273674263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c4107240 0x7fa0c410ee10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009ca0 tx=0x7fa0bc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.552+0000 7fa0cb29a640 1 -- 172.21.15.7:0/3273674263 shutdown_connections 2024-04-21T08:50:49.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.552+0000 7fa0cb29a640 1 --2- 172.21.15.7:0/3273674263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c4107240 0x7fa0c410ee10 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.552+0000 7fa0cb29a640 1 --2- 172.21.15.7:0/3273674263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0c4102b20 0x7fa0c4106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.552+0000 7fa0cb29a640 1 --2- 172.21.15.7:0/3273674263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c4102170 0x7fa0c4102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.552+0000 7fa0cb29a640 1 -- 172.21.15.7:0/3273674263 >> 172.21.15.7:0/3273674263 conn(0x7fa0c40fd9b0 msgr2=0x7fa0c40ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:49.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.553+0000 7fa0cb29a640 1 -- 172.21.15.7:0/3273674263 shutdown_connections 2024-04-21T08:50:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.553+0000 7fa0cb29a640 1 -- 172.21.15.7:0/3273674263 wait complete. 2024-04-21T08:50:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.553+0000 7fa0cb29a640 1 Processor -- start 2024-04-21T08:50:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.553+0000 7fa0cb29a640 1 -- start start 2024-04-21T08:50:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.553+0000 7fa0cb29a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c4102170 0x7fa0c4074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.553+0000 7fa0cb29a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0c4107240 0x7fa0c4074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.553+0000 7fa0cb29a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c4074cd0 0x7fa0c41a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.553+0000 7fa0cb29a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c4112190 con 0x7fa0c4102170 2024-04-21T08:50:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.553+0000 7fa0cb29a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c4112010 con 0x7fa0c4074cd0 2024-04-21T08:50:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.553+0000 7fa0cb29a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c4112310 con 0x7fa0c4107240 2024-04-21T08:50:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0c9036640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c4102170 0x7fa0c4074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0c9036640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c4102170 0x7fa0c4074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38536/0 (socket says 172.21.15.7:38536) 2024-04-21T08:50:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0c9036640 1 -- 172.21.15.7:0/4271102272 learned_addr learned my addr 172.21.15.7:0/4271102272 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0c8835640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0c4107240 0x7fa0c4074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0c9837640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c4074cd0 0x7fa0c41a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0c9036640 1 -- 172.21.15.7:0/4271102272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0c4107240 msgr2=0x7fa0c4074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0c9036640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0c4107240 0x7fa0c4074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0c9036640 1 -- 172.21.15.7:0/4271102272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c4074cd0 msgr2=0x7fa0c41a6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0c9036640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c4074cd0 0x7fa0c41a6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0c9036640 1 -- 172.21.15.7:0/4271102272 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c41a6f20 con 0x7fa0c4102170 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0c9036640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c4102170 0x7fa0c4074250 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000d950 tx=0x7fa0c000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0ba7fc640 1 -- 172.21.15.7:0/4271102272 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0017030 con 0x7fa0c4102170 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0cb29a640 1 -- 172.21.15.7:0/4271102272 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c41a7210 con 0x7fa0c4102170 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0ba7fc640 1 -- 172.21.15.7:0/4271102272 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c0004020 con 0x7fa0c4102170 2024-04-21T08:50:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0ba7fc640 1 -- 172.21.15.7:0/4271102272 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0005560 con 0x7fa0c4102170 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.554+0000 7fa0cb29a640 1 -- 172.21.15.7:0/4271102272 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c41a7760 con 0x7fa0c4102170 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.557+0000 7fa0ba7fc640 1 -- 172.21.15.7:0/4271102272 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa0c0009a30 con 0x7fa0c4102170 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.557+0000 7fa0cdf6c740 1 Processor -- start 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.557+0000 7fa0cdf6c740 1 -- start start 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.557+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da6c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.557+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da6cea0 0x56111da75540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.557+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da75a80 0x56111da77e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.557+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56111d9d11e0 con 0x56111da6cea0 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.557+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56111d96e390 con 0x56111da75a80 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.557+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56111d9cd1a0 con 0x56111da6c580 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.557+0000 7fa0ba7fc640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa0b00a3d50 0x7fa0b00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9036640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da6cea0 0x56111da75540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9036640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da6cea0 0x56111da75540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38550/0 (socket says 172.21.15.7:38550) 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9036640 1 -- 172.21.15.7:0/1160202767 learned_addr learned my addr 172.21.15.7:0/1160202767 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9837640 1 --2- 172.21.15.7:0/1160202767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da6c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c8835640 1 --2- 172.21.15.7:0/1160202767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da75a80 0x56111da77e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9837640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa0b00a3d50 0x7fa0b00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0ba7fc640 1 -- 172.21.15.7:0/4271102272 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa0c00d0460 con 0x7fa0c4102170 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9036640 1 -- 172.21.15.7:0/1160202767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 msgr2=0x56111da6c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9036640 1 --2- 172.21.15.7:0/1160202767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da6c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9036640 1 -- 172.21.15.7:0/1160202767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da75a80 msgr2=0x56111da77e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9036640 1 --2- 172.21.15.7:0/1160202767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da75a80 0x56111da77e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9036640 1 -- 172.21.15.7:0/1160202767 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56111d7ad7d0 con 0x56111da6cea0 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c8835640 1 --2- 172.21.15.7:0/1160202767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da75a80 0x56111da77e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9036640 1 --2- 172.21.15.7:0/1160202767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da6cea0 0x56111da75540 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa0c009bd10 tx=0x7fa0c006eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74892019db3c220 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0c9837640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa0b00a3d50 0x7fa0b00a6210 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc004af0 tx=0x7fa0bc005c50 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.558+0000 7fa0af7fe640 1 -- 172.21.15.7:0/1160202767 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c009c710 con 0x56111da6cea0 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.559+0000 7fa0af7fe640 1 -- 172.21.15.7:0/1160202767 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c009c010 con 0x56111da6cea0 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.559+0000 7fa0af7fe640 1 -- 172.21.15.7:0/1160202767 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00041d0 con 0x56111da6cea0 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.559+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1160202767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da6cea0 msgr2=0x56111da75540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.559+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/1160202767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da6cea0 0x56111da75540 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa0c009bd10 tx=0x7fa0c006eb00 comp rx=0 tx=0).stop 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.559+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1160202767 shutdown_connections 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.559+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/1160202767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da75a80 0x56111da77e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.559+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/1160202767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da6cea0 0x56111da75540 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.559+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/1160202767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da6c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.559+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1160202767 >> 172.21.15.7:0/1160202767 conn(0x56111da66cb0 msgr2=0x56111da67090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.559+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1160202767 shutdown_connections 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.560+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1160202767 wait complete. 2024-04-21T08:50:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.560+0000 7fa0cdf6c740 1 -- start start 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.560+0000 7fa0cdf6c740 1 Processor -- start 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.560+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da6c580 0x56111da72b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.560+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da73080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.560+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da735c0 0x56111da81c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.560+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56111da78810 con 0x56111da75a80 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.560+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56111d98a2f0 con 0x56111da6c580 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.560+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56111da78990 con 0x56111da735c0 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.560+0000 7fa0cdf6c740 10 client.0 _ll_register_callbacks cb 0x56111d997580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.560+0000 7fa0c8835640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da73080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0c8835640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da73080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38556/0 (socket says 172.21.15.7:38556) 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0c8835640 1 -- 172.21.15.7:0/1639316052 learned_addr learned my addr 172.21.15.7:0/1639316052 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0c9036640 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da6c580 0x56111da72b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0c9837640 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da735c0 0x56111da81c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0c8835640 1 -- 172.21.15.7:0/1639316052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da735c0 msgr2=0x56111da81c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0c8835640 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da735c0 0x56111da81c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0c8835640 1 -- 172.21.15.7:0/1639316052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da6c580 msgr2=0x56111da72b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0c8835640 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da6c580 0x56111da72b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0c8835640 1 -- 172.21.15.7:0/1639316052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56111d6e33f0 con 0x56111da75a80 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0c8835640 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da73080 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40107f0 tx=0x7fa0b4010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e6bb67f49d132 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0acff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0acff9640 1 -- 172.21.15.7:0/1639316052 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b401e060 con 0x56111da75a80 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0acff9640 1 -- 172.21.15.7:0/1639316052 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa0b400fb30 con 0x56111da75a80 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1639316052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0c009c010 con 0x56111da75a80 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0acff9640 1 -- 172.21.15.7:0/1639316052 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa0b4025050 con 0x56111da75a80 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0cdf6c740 10 client.16842 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.561+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1639316052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56111d6a30f0 con 0x56111da75a80 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.562+0000 7fa0acff9640 1 -- 172.21.15.7:0/1639316052 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0c009c010 con 0x56111da75a80 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.562+0000 7fa0acff9640 1 -- 172.21.15.7:0/1639316052 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa0b4018370 con 0x56111da75a80 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.562+0000 7fa0cdf6c740 10 client.16842 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.562+0000 7fa0cdf6c740 10 client.16842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.562+0000 7fa0cdf6c740 20 client.16842 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.562+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56111da85f90 0x56111da883c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.562+0000 7fa0cdf6c740 4 client.16842 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.562+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1639316052 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56111d9dec00 con 0x56111da85f90 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.562+0000 7fa0c9036640 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56111da85f90 0x56111da883c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.562+0000 7fa0c9036640 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56111da85f90 0x56111da883c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db7b4b2f09203a35 server_cookie=e0046d630544ff8a in_seq=0 out_seq=0 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.562+0000 7fa0acff9640 10 client.16842 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0acff9640 1 -- 172.21.15.7:0/1639316052 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56111d9dec00 con 0x56111da85f90 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0acff9640 10 client.16842 handle_command_reply: tid=0 2024-04-21T08:50:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0cdf6c740 1 client.16842 shutdown 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1639316052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 msgr2=0x56111da73080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da73080 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40107f0 tx=0x7fa0b4010cb0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1639316052 shutdown_connections 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56111da85f90 0x56111da883c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da735c0 0x56111da81c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da73080 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0acff9640 0 client.16842 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/1639316052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da6c580 0x56111da72b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.563+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1639316052 >> 172.21.15.7:0/1639316052 conn(0x56111da66cb0 msgr2=0x56111da68a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.564+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1639316052 shutdown_connections 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.564+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/1639316052 wait complete. 2024-04-21T08:50:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.564+0000 7fa0cdf6c740 20 client.16842 trim_cache size 0 max 16384 2024-04-21T08:50:49.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0cdf6c740 1 Processor -- start 2024-04-21T08:50:49.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0cdf6c740 1 -- start start 2024-04-21T08:50:49.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da65bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da735c0 0x56111da8dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da8e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41af6f0 con 0x56111da75a80 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41b1f70 con 0x56111da735c0 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41afd60 con 0x56111da6c580 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0c9837640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da8e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0c9837640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da8e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38558/0 (socket says 172.21.15.7:38558) 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0c9837640 1 -- 172.21.15.7:0/2924255519 learned_addr learned my addr 172.21.15.7:0/2924255519 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0c8835640 1 --2- 172.21.15.7:0/2924255519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da735c0 0x56111da8dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0c9036640 1 --2- 172.21.15.7:0/2924255519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da65bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0c9837640 1 -- 172.21.15.7:0/2924255519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 msgr2=0x56111da65bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0c9837640 1 --2- 172.21.15.7:0/2924255519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da65bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0c9837640 1 -- 172.21.15.7:0/2924255519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da735c0 msgr2=0x56111da8dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0c9837640 1 --2- 172.21.15.7:0/2924255519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da735c0 0x56111da8dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.583+0000 7fa0c9837640 1 -- 172.21.15.7:0/2924255519 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56111da667e0 con 0x56111da75a80 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.584+0000 7fa0c9837640 1 --2- 172.21.15.7:0/2924255519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da8e1d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009ca0 tx=0x7fa0bc02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6747ede3a9afe1a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.584+0000 7fa0aeffd640 1 -- 172.21.15.7:0/2924255519 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc039040 con 0x56111da75a80 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.584+0000 7fa0aeffd640 1 -- 172.21.15.7:0/2924255519 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0bc055070 con 0x56111da75a80 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.584+0000 7fa0aeffd640 1 -- 172.21.15.7:0/2924255519 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0506c0 con 0x56111da75a80 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.584+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2924255519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 msgr2=0x56111da8e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.584+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/2924255519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da8e1d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009ca0 tx=0x7fa0bc02e040 comp rx=0 tx=0).stop 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2924255519 shutdown_connections 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/2924255519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da8e1d0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/2924255519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da735c0 0x56111da8dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/2924255519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da65bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2924255519 >> 172.21.15.7:0/2924255519 conn(0x56111da66cb0 msgr2=0x56111da654e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:49.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2924255519 shutdown_connections 2024-04-21T08:50:49.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2924255519 wait complete. 2024-04-21T08:50:49.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 -- start start 2024-04-21T08:50:49.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 Processor -- start 2024-04-21T08:50:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da7dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da735c0 0x56111da7e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da7abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41b0430 con 0x56111da75a80 2024-04-21T08:50:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41b18a0 con 0x56111da735c0 2024-04-21T08:50:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.585+0000 7fa0cdf6c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41b0b00 con 0x56111da6c580 2024-04-21T08:50:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0cdf6c740 10 client.0 _ll_register_callbacks cb 0x7fa0b4010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0c9036640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da7dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0c9837640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da7abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0c9036640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da7dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:50370/0 (socket says 172.21.15.7:50370) 2024-04-21T08:50:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0c9036640 1 -- 172.21.15.7:0/2894342266 learned_addr learned my addr 172.21.15.7:0/2894342266 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0c8835640 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da735c0 0x56111da7e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0c9837640 1 -- 172.21.15.7:0/2894342266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 msgr2=0x56111da7dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0c9837640 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da7dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0c9837640 1 -- 172.21.15.7:0/2894342266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da735c0 msgr2=0x56111da7e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0c9837640 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da735c0 0x56111da7e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0c9837640 1 -- 172.21.15.7:0/2894342266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56111da7b3e0 con 0x56111da75a80 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0c9837640 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da7abb0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x56111da73c90 tx=0x7fa0bc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d35f4e10dea9a7d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0acff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0acff9640 1 -- 172.21.15.7:0/2894342266 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc05e060 con 0x56111da75a80 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2894342266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0bc055070 con 0x56111da75a80 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0acff9640 1 -- 172.21.15.7:0/2894342266 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa0bc004410 con 0x56111da75a80 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0acff9640 1 -- 172.21.15.7:0/2894342266 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa0bc039040 con 0x56111da75a80 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0cdf6c740 10 client.16848 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.586+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2894342266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56111daed000 con 0x56111da75a80 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.587+0000 7fa0acff9640 1 -- 172.21.15.7:0/2894342266 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0bc055070 con 0x56111da75a80 2024-04-21T08:50:49.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.587+0000 7fa0acff9640 1 -- 172.21.15.7:0/2894342266 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa0bc0507d0 con 0x56111da75a80 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.587+0000 7fa0cdf6c740 10 client.16848 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.587+0000 7fa0cdf6c740 10 client.16848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.587+0000 7fa0cdf6c740 20 client.16848 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.587+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56111daed620 0x56111daefa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.587+0000 7fa0cdf6c740 4 client.16848 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.587+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2894342266 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56111d9d8a70 con 0x56111daed620 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.587+0000 7fa0c9036640 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56111daed620 0x56111daefa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.587+0000 7fa0c9036640 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56111daed620 0x56111daefa10 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3870e38a1545ee2 server_cookie=96cf1f8895045d31 in_seq=0 out_seq=0 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.587+0000 7fa0acff9640 10 client.16848 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0acff9640 1 -- 172.21.15.7:0/2894342266 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56111d9d8a70 con 0x56111daed620 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0acff9640 10 client.16848 handle_command_reply: tid=0 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 1 client.16848 shutdown 2024-04-21T08:50:49.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2894342266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 msgr2=0x56111da7abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da7abb0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x56111da73c90 tx=0x7fa0bc0095c0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2894342266 shutdown_connections 2024-04-21T08:50:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56111daed620 0x56111daefa10 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56111da75a80 0x56111da7abb0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56111da735c0 0x56111da7e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 1 --2- 172.21.15.7:0/2894342266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56111da6c580 0x56111da7dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2894342266 >> 172.21.15.7:0/2894342266 conn(0x56111da66cb0 msgr2=0x56111da68900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0acff9640 0 client.16848 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2894342266 shutdown_connections 2024-04-21T08:50:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 1 -- 172.21.15.7:0/2894342266 wait complete. 2024-04-21T08:50:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.588+0000 7fa0cdf6c740 20 client.16848 trim_cache size 0 max 16384 2024-04-21T08:50:49.595 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:50:49.595 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:50:49.595 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:50:49.595 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:50:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 -- 172.21.15.7:0/4271102272 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa0b00a3d50 msgr2=0x7fa0b00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa0b00a3d50 0x7fa0b00a6210 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc004af0 tx=0x7fa0bc005c50 comp rx=0 tx=0).stop 2024-04-21T08:50:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 -- 172.21.15.7:0/4271102272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c4102170 msgr2=0x7fa0c4074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c4102170 0x7fa0c4074250 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000d950 tx=0x7fa0c000dd50 comp rx=0 tx=0).stop 2024-04-21T08:50:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 -- 172.21.15.7:0/4271102272 shutdown_connections 2024-04-21T08:50:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0c4074cd0 0x7fa0c41a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa0b00a3d50 0x7fa0b00a6210 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa0c4107240 0x7fa0c4074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 --2- 172.21.15.7:0/4271102272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c4102170 0x7fa0c4074250 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 -- 172.21.15.7:0/4271102272 >> 172.21.15.7:0/4271102272 conn(0x7fa0c40fd9b0 msgr2=0x7fa0c40ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 -- 172.21.15.7:0/4271102272 shutdown_connections 2024-04-21T08:50:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.589+0000 7fa0cb29a640 1 -- 172.21.15.7:0/4271102272 wait complete. 2024-04-21T08:50:49.600 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:50:49.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:50:49.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.703+0000 7f5403f32640 1 -- 172.21.15.7:0/1721471313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53fc10e8e0 msgr2=0x7f53fc110cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.703+0000 7f5403f32640 1 --2- 172.21.15.7:0/1721471313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53fc10e8e0 0x7f53fc110cd0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009ca0 tx=0x7f53f402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.703+0000 7f5403f32640 1 -- 172.21.15.7:0/1721471313 shutdown_connections 2024-04-21T08:50:49.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.703+0000 7f5403f32640 1 --2- 172.21.15.7:0/1721471313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53fc10e8e0 0x7f53fc110cd0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.703+0000 7f5403f32640 1 --2- 172.21.15.7:0/1721471313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53fc101650 0x7f53fc10e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.703+0000 7f5403f32640 1 --2- 172.21.15.7:0/1721471313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53fc100d30 0x7f53fc101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.703+0000 7f5403f32640 1 -- 172.21.15.7:0/1721471313 >> 172.21.15.7:0/1721471313 conn(0x7f53fc0771d0 msgr2=0x7f53fc0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:49.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.703+0000 7f5403f32640 1 -- 172.21.15.7:0/1721471313 shutdown_connections 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.703+0000 7f5403f32640 1 -- 172.21.15.7:0/1721471313 wait complete. 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.703+0000 7f5403f32640 1 Processor -- start 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.703+0000 7f5403f32640 1 -- start start 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f5403f32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53fc100d30 0x7f53fc1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f5403f32640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53fc101650 0x7f53fc1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f5403f32640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53fc10e8e0 0x7f53fc19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f5403f32640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc1143c0 con 0x7f53fc101650 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f5403f32640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc114240 con 0x7f53fc10e8e0 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f5403f32640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc114540 con 0x7f53fc100d30 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f54014cd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53fc101650 0x7f53fc1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f54014cd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53fc101650 0x7f53fc1a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38594/0 (socket says 172.21.15.7:38594) 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f54014cd640 1 -- 172.21.15.7:0/3382802058 learned_addr learned my addr 172.21.15.7:0/3382802058 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:49.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f54024cf640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53fc10e8e0 0x7f53fc19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f5401cce640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53fc100d30 0x7f53fc1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f54014cd640 1 -- 172.21.15.7:0/3382802058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53fc100d30 msgr2=0x7f53fc1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f54014cd640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53fc100d30 0x7f53fc1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f54014cd640 1 -- 172.21.15.7:0/3382802058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53fc10e8e0 msgr2=0x7f53fc19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f54014cd640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53fc10e8e0 0x7f53fc19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f54014cd640 1 -- 172.21.15.7:0/3382802058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53fc19afe0 con 0x7f53fc101650 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f54014cd640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53fc101650 0x7f53fc1a0ba0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00d770 tx=0x7f53ec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.704+0000 7f53f2ffd640 1 -- 172.21.15.7:0/3382802058 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec017070 con 0x7f53fc101650 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.705+0000 7f53f2ffd640 1 -- 172.21.15.7:0/3382802058 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53ec0054d0 con 0x7f53fc101650 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.705+0000 7f5403f32640 1 -- 172.21.15.7:0/3382802058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53fc19b2d0 con 0x7f53fc101650 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.705+0000 7f53f2ffd640 1 -- 172.21.15.7:0/3382802058 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec004c30 con 0x7f53fc101650 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.705+0000 7f5403f32640 1 -- 172.21.15.7:0/3382802058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53fc1a75e0 con 0x7f53fc101650 2024-04-21T08:50:49.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.705+0000 7f5403f32640 1 -- 172.21.15.7:0/3382802058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53fc069e90 con 0x7f53fc101650 2024-04-21T08:50:49.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.707+0000 7f53f2ffd640 1 -- 172.21.15.7:0/3382802058 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f53ec004dd0 con 0x7f53fc101650 2024-04-21T08:50:49.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.707+0000 7f53f2ffd640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f53e80a3da0 0x7f53e80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:49.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.707+0000 7f5401cce640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f53e80a3da0 0x7f53e80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:49.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.707+0000 7f53f2ffd640 1 -- 172.21.15.7:0/3382802058 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f53ec0d09b0 con 0x7f53fc101650 2024-04-21T08:50:49.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.708+0000 7f5401cce640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f53e80a3da0 0x7f53e80a6260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009b60 tx=0x7f53f8005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.711+0000 7f53f2ffd640 1 -- 172.21.15.7:0/3382802058 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f53ec09cce0 con 0x7f53fc101650 2024-04-21T08:50:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.937+0000 7f5403f32640 1 -- 172.21.15.7:0/3382802058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53fc19bc50 con 0x7f53fc101650 2024-04-21T08:50:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.938+0000 7f53f2ffd640 1 -- 172.21.15.7:0/3382802058 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7f53ec09c5e0 con 0x7f53fc101650 2024-04-21T08:50:49.938 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:50:49.938 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:50:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:50:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 -- 172.21.15.7:0/3382802058 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f53e80a3da0 msgr2=0x7f53e80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f53e80a3da0 0x7f53e80a6260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009b60 tx=0x7f53f8005f90 comp rx=0 tx=0).stop 2024-04-21T08:50:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 -- 172.21.15.7:0/3382802058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53fc101650 msgr2=0x7f53fc1a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53fc101650 0x7f53fc1a0ba0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00d770 tx=0x7f53ec00dc30 comp rx=0 tx=0).stop 2024-04-21T08:50:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 -- 172.21.15.7:0/3382802058 shutdown_connections 2024-04-21T08:50:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53fc10e8e0 0x7f53fc19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53fc101650 0x7f53fc1a0ba0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f53e80a3da0 0x7f53e80a6260 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 --2- 172.21.15.7:0/3382802058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f53fc100d30 0x7f53fc1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 -- 172.21.15.7:0/3382802058 >> 172.21.15.7:0/3382802058 conn(0x7f53fc0771d0 msgr2=0x7f53fc10ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 -- 172.21.15.7:0/3382802058 shutdown_connections 2024-04-21T08:50:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:49.940+0000 7f5403f32640 1 -- 172.21.15.7:0/3382802058 wait complete. 2024-04-21T08:50:49.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:50:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.054+0000 7f748efb2640 1 -- 172.21.15.7:0/2280957905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7488103b50 msgr2=0x7f7488103fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.054+0000 7f748efb2640 1 --2- 172.21.15.7:0/2280957905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7488103b50 0x7f7488103fb0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f7470009930 tx=0x7f747002bc20 comp rx=0 tx=0).stop 2024-04-21T08:50:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.054+0000 7f748efb2640 1 -- 172.21.15.7:0/2280957905 shutdown_connections 2024-04-21T08:50:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.054+0000 7f748efb2640 1 --2- 172.21.15.7:0/2280957905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74881044f0 0x7f748810eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.054+0000 7f748efb2640 1 --2- 172.21.15.7:0/2280957905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7488103b50 0x7f7488103fb0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.054+0000 7f748efb2640 1 --2- 172.21.15.7:0/2280957905 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7488102270 0x7f7488102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.054+0000 7f748efb2640 1 -- 172.21.15.7:0/2280957905 >> 172.21.15.7:0/2280957905 conn(0x7f74880fda10 msgr2=0x7f74880ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.054+0000 7f748efb2640 1 -- 172.21.15.7:0/2280957905 shutdown_connections 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.054+0000 7f748efb2640 1 -- 172.21.15.7:0/2280957905 wait complete. 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748efb2640 1 Processor -- start 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748efb2640 1 -- start start 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748efb2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7488102270 0x7f7488198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748efb2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7488103b50 0x7f7488198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748efb2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74881044f0 0x7f748819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748efb2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f74881122c0 con 0x7f74881044f0 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748efb2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7488112140 con 0x7f7488102270 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748efb2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7488112440 con 0x7f7488103b50 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748d54f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74881044f0 0x7f748819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748d54f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74881044f0 0x7f748819d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38612/0 (socket says 172.21.15.7:38612) 2024-04-21T08:50:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748d54f640 1 -- 172.21.15.7:0/3074328436 learned_addr learned my addr 172.21.15.7:0/3074328436 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748cd4e640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7488102270 0x7f7488198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f7487fff640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7488103b50 0x7f7488198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748d54f640 1 -- 172.21.15.7:0/3074328436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7488103b50 msgr2=0x7f7488198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748d54f640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7488103b50 0x7f7488198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748d54f640 1 -- 172.21.15.7:0/3074328436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7488102270 msgr2=0x7f7488198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748d54f640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7488102270 0x7f7488198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f748d54f640 1 -- 172.21.15.7:0/3074328436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f748819d730 con 0x7f74881044f0 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.055+0000 7f7487fff640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7488103b50 0x7f7488198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.056+0000 7f748d54f640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74881044f0 0x7f748819d000 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f747800c970 tx=0x7f747800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.056+0000 7f7485ffb640 1 -- 172.21.15.7:0/3074328436 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7478007bb0 con 0x7f74881044f0 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.056+0000 7f748efb2640 1 -- 172.21.15.7:0/3074328436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f748819da20 con 0x7f74881044f0 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.056+0000 7f7485ffb640 1 -- 172.21.15.7:0/3074328436 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7478007d50 con 0x7f74881044f0 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.056+0000 7f7485ffb640 1 -- 172.21.15.7:0/3074328436 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7478018770 con 0x7f74881044f0 2024-04-21T08:50:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.056+0000 7f748efb2640 1 -- 172.21.15.7:0/3074328436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74881a52b0 con 0x7f74881044f0 2024-04-21T08:50:50.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.058+0000 7f7485ffb640 1 -- 172.21.15.7:0/3074328436 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f747800f990 con 0x7f74881044f0 2024-04-21T08:50:50.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.058+0000 7f7491ba0740 1 Processor -- start 2024-04-21T08:50:50.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.058+0000 7f7491ba0740 1 -- start start 2024-04-21T08:50:50.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7485ffb640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f74740a3d20 0x7f74740a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 0x556749eb28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749ebba00 0x556749ebde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7491ba0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556749e171f0 con 0x556749eb2e80 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7491ba0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556749db44b0 con 0x556749eb24d0 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7491ba0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556749e131b0 con 0x556749ebba00 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7485ffb640 1 -- 172.21.15.7:0/3074328436 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f74780d0ce0 con 0x7f74881044f0 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7487fff640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f74740a3d20 0x7f74740a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7487fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7487fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38614/0 (socket says 172.21.15.7:38614) 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7487fff640 1 -- 172.21.15.7:0/131501733 learned_addr learned my addr 172.21.15.7:0/131501733 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f748cd4e640 1 --2- 172.21.15.7:0/131501733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 0x556749eb28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f748d54f640 1 --2- 172.21.15.7:0/131501733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749ebba00 0x556749ebde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7487fff640 1 -- 172.21.15.7:0/131501733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749ebba00 msgr2=0x556749ebde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7487fff640 1 --2- 172.21.15.7:0/131501733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749ebba00 0x556749ebde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7487fff640 1 -- 172.21.15.7:0/131501733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 msgr2=0x556749eb28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7487fff640 1 --2- 172.21.15.7:0/131501733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 0x556749eb28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f7487fff640 1 -- 172.21.15.7:0/131501733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556749da61b0 con 0x556749eb2e80 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.059+0000 7f748cd4e640 1 --2- 172.21.15.7:0/131501733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 0x556749eb28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.060+0000 7f7487fff640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f74740a3d20 0x7f74740a61e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7470005ec0 tx=0x7f74700040d0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.060+0000 7f7487fff640 1 --2- 172.21.15.7:0/131501733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebb430 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f747002c230 tx=0x7f7470046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d9faaefc4cbae6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.060+0000 7f747effd640 1 -- 172.21.15.7:0/131501733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7470052070 con 0x556749eb2e80 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.060+0000 7f747effd640 1 -- 172.21.15.7:0/131501733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f747002cab0 con 0x556749eb2e80 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.060+0000 7f747effd640 1 -- 172.21.15.7:0/131501733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7470003850 con 0x556749eb2e80 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.060+0000 7f7491ba0740 1 -- 172.21.15.7:0/131501733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 msgr2=0x556749ebb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.060+0000 7f7491ba0740 1 --2- 172.21.15.7:0/131501733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebb430 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f747002c230 tx=0x7f7470046000 comp rx=0 tx=0).stop 2024-04-21T08:50:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.060+0000 7f7491ba0740 1 -- 172.21.15.7:0/131501733 shutdown_connections 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.060+0000 7f7491ba0740 1 --2- 172.21.15.7:0/131501733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749ebba00 0x556749ebde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.060+0000 7f7491ba0740 1 --2- 172.21.15.7:0/131501733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebb430 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.060+0000 7f7491ba0740 1 --2- 172.21.15.7:0/131501733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 0x556749eb28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 1 -- 172.21.15.7:0/131501733 >> 172.21.15.7:0/131501733 conn(0x556749eacba0 msgr2=0x556749eacf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 1 -- 172.21.15.7:0/131501733 shutdown_connections 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 1 -- 172.21.15.7:0/131501733 wait complete. 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 1 -- start start 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 1 Processor -- start 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749eb24d0 0x556749ebb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb2e80 0x556749eb86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749ebba00 0x556749eb8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556749ebe910 con 0x556749ebba00 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556749dd0500 con 0x556749eb2e80 2024-04-21T08:50:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556749ebea90 con 0x556749eb24d0 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f7491ba0740 10 client.0 _ll_register_callbacks cb 0x556749d9eb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.061+0000 7f748d54f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749ebba00 0x556749eb8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f748d54f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749ebba00 0x556749eb8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38628/0 (socket says 172.21.15.7:38628) 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f748d54f640 1 -- 172.21.15.7:0/3911554696 learned_addr learned my addr 172.21.15.7:0/3911554696 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f748cd4e640 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749eb24d0 0x556749ebb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f7487fff640 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb2e80 0x556749eb86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f748d54f640 1 -- 172.21.15.7:0/3911554696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749eb24d0 msgr2=0x556749ebb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f748d54f640 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749eb24d0 0x556749ebb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f748d54f640 1 -- 172.21.15.7:0/3911554696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb2e80 msgr2=0x556749eb86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f748d54f640 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb2e80 0x556749eb86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f748d54f640 1 -- 172.21.15.7:0/3911554696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556749b293f0 con 0x556749ebba00 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f748d54f640 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749ebba00 0x556749eb8cd0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7478098570 tx=0x7f747809c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b28d1e0bd3e327 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f7467fff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f7467fff640 1 -- 172.21.15.7:0/3911554696 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74780a0c40 con 0x556749ebba00 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f7467fff640 1 -- 172.21.15.7:0/3911554696 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f74780870b0 con 0x556749ebba00 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f7491ba0740 1 -- 172.21.15.7:0/3911554696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f747002cab0 con 0x556749ebba00 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f7467fff640 1 -- 172.21.15.7:0/3911554696 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f74780a5500 con 0x556749ebba00 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f7491ba0740 10 client.16866 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f7491ba0740 1 -- 172.21.15.7:0/3911554696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556749bf37d0 con 0x556749ebba00 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.062+0000 7f7467fff640 1 -- 172.21.15.7:0/3911554696 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f747002cab0 con 0x556749ebba00 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.063+0000 7f7467fff640 1 -- 172.21.15.7:0/3911554696 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f7478061040 con 0x556749ebba00 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.063+0000 7f7491ba0740 10 client.16866 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.063+0000 7f7491ba0740 10 client.16866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.063+0000 7f7491ba0740 20 client.16866 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.063+0000 7f7491ba0740 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556749ecbbe0 0x556749ece000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.063+0000 7f7491ba0740 4 client.16866 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.063+0000 7f7491ba0740 1 -- 172.21.15.7:0/3911554696 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556749db6930 con 0x556749ecbbe0 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.063+0000 7f748cd4e640 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556749ecbbe0 0x556749ece000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.063+0000 7f7467fff640 10 client.16866 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.063+0000 7f748cd4e640 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556749ecbbe0 0x556749ece000 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e243c41014f70d4 server_cookie=7dcb8d17860b9892 in_seq=0 out_seq=0 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7467fff640 1 -- 172.21.15.7:0/3911554696 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556749db6930 con 0x556749ecbbe0 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7467fff640 10 client.16866 handle_command_reply: tid=0 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 1 client.16866 shutdown 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 1 -- 172.21.15.7:0/3911554696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749ebba00 msgr2=0x556749eb8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749ebba00 0x556749eb8cd0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7478098570 tx=0x7f747809c320 comp rx=0 tx=0).stop 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 1 -- 172.21.15.7:0/3911554696 shutdown_connections 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556749ecbbe0 0x556749ece000 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749ebba00 0x556749eb8cd0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb2e80 0x556749eb86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 1 --2- 172.21.15.7:0/3911554696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749eb24d0 0x556749ebb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 1 -- 172.21.15.7:0/3911554696 >> 172.21.15.7:0/3911554696 conn(0x556749eacba0 msgr2=0x556749eae8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:50.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7467fff640 0 client.16866 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:50.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 1 -- 172.21.15.7:0/3911554696 shutdown_connections 2024-04-21T08:50:50.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 1 -- 172.21.15.7:0/3911554696 wait complete. 2024-04-21T08:50:50.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.064+0000 7f7491ba0740 20 client.16866 trim_cache size 0 max 16384 2024-04-21T08:50:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f7491ba0740 1 Processor -- start 2024-04-21T08:50:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f7491ba0740 1 -- start start 2024-04-21T08:50:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb24d0 0x556749ec7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749eb2e80 0x556749eaba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749ebba00 0x556749ebadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f7491ba0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f74881ad3a0 con 0x556749eb24d0 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f7491ba0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f74881afc20 con 0x556749ebba00 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f7491ba0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f74881ada10 con 0x556749eb2e80 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f748cd4e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb24d0 0x556749ec7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f748cd4e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb24d0 0x556749ec7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38632/0 (socket says 172.21.15.7:38632) 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f748cd4e640 1 -- 172.21.15.7:0/44013577 learned_addr learned my addr 172.21.15.7:0/44013577 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f748cd4e640 1 -- 172.21.15.7:0/44013577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749eb2e80 msgr2=0x556749eaba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f748cd4e640 1 --2- 172.21.15.7:0/44013577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749eb2e80 0x556749eaba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f748cd4e640 1 -- 172.21.15.7:0/44013577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749ebba00 msgr2=0x556749ebadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f748d54f640 1 --2- 172.21.15.7:0/44013577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749ebba00 0x556749ebadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f748cd4e640 1 --2- 172.21.15.7:0/44013577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749ebba00 0x556749ebadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.078+0000 7f748cd4e640 1 -- 172.21.15.7:0/44013577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556749ebe1c0 con 0x556749eb24d0 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f748cd4e640 1 --2- 172.21.15.7:0/44013577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb24d0 0x556749ec7830 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7480003ba0 tx=0x7f7480012bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4ba5a539a20067 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f747e7fc640 1 -- 172.21.15.7:0/44013577 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7480022020 con 0x556749eb24d0 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f747e7fc640 1 -- 172.21.15.7:0/44013577 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f748001e8d0 con 0x556749eb24d0 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f747e7fc640 1 -- 172.21.15.7:0/44013577 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7480011ea0 con 0x556749eb24d0 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f7491ba0740 1 -- 172.21.15.7:0/44013577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb24d0 msgr2=0x556749ec7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f7491ba0740 1 --2- 172.21.15.7:0/44013577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb24d0 0x556749ec7830 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7480003ba0 tx=0x7f7480012bb0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f7491ba0740 1 -- 172.21.15.7:0/44013577 shutdown_connections 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f7491ba0740 1 --2- 172.21.15.7:0/44013577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749ebba00 0x556749ebadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f7491ba0740 1 --2- 172.21.15.7:0/44013577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749eb2e80 0x556749eaba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f7491ba0740 1 --2- 172.21.15.7:0/44013577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb24d0 0x556749ec7830 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f7491ba0740 1 -- 172.21.15.7:0/44013577 >> 172.21.15.7:0/44013577 conn(0x556749eacba0 msgr2=0x556749eab370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f7491ba0740 1 -- 172.21.15.7:0/44013577 shutdown_connections 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.079+0000 7f7491ba0740 1 -- 172.21.15.7:0/44013577 wait complete. 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7491ba0740 1 -- start start 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7491ba0740 1 Processor -- start 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 0x556749ec3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7491ba0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749ebba00 0x556749ebf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7491ba0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f74881ae0e0 con 0x556749eb2e80 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7491ba0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f74881af550 con 0x556749eb24d0 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7491ba0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f74881ae7b0 con 0x556749ebba00 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7491ba0740 10 client.0 _ll_register_callbacks cb 0x7f747809c460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7487fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7487fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38646/0 (socket says 172.21.15.7:38646) 2024-04-21T08:50:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f7487fff640 1 -- 172.21.15.7:0/2216230312 learned_addr learned my addr 172.21.15.7:0/2216230312 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.080+0000 7f748cd4e640 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 0x556749ec3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f748d54f640 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749ebba00 0x556749ebf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7487fff640 1 -- 172.21.15.7:0/2216230312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749ebba00 msgr2=0x556749ebf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7487fff640 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749ebba00 0x556749ebf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7487fff640 1 -- 172.21.15.7:0/2216230312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 msgr2=0x556749ec3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7487fff640 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 0x556749ec3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7487fff640 1 -- 172.21.15.7:0/2216230312 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556749ec0010 con 0x556749eb2e80 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f748cd4e640 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 0x556749ec3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7487fff640 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebf3e0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7470050040 tx=0x7f747002c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4ada5b47630f4b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7467fff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7467fff640 1 -- 172.21.15.7:0/2216230312 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7470052040 con 0x556749eb2e80 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7467fff640 1 -- 172.21.15.7:0/2216230312 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f7470005610 con 0x556749eb2e80 2024-04-21T08:50:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7491ba0740 1 -- 172.21.15.7:0/2216230312 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f748001e8d0 con 0x556749eb2e80 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7467fff640 1 -- 172.21.15.7:0/2216230312 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7470064050 con 0x556749eb2e80 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7491ba0740 10 client.16872 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7491ba0740 1 -- 172.21.15.7:0/2216230312 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556749f32090 con 0x556749eb2e80 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7467fff640 1 -- 172.21.15.7:0/2216230312 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f748001e8d0 con 0x556749eb2e80 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7467fff640 1 -- 172.21.15.7:0/2216230312 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f7470040040 con 0x556749eb2e80 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7491ba0740 10 client.16872 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.081+0000 7f7491ba0740 10 client.16872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.082+0000 7f7491ba0740 20 client.16872 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.082+0000 7f7491ba0740 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556749f32430 0x556749f34820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.082+0000 7f7491ba0740 4 client.16872 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.082+0000 7f7491ba0740 1 -- 172.21.15.7:0/2216230312 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556749e24c10 con 0x556749f32430 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.082+0000 7f748cd4e640 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556749f32430 0x556749f34820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.082+0000 7f748cd4e640 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556749f32430 0x556749f34820 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10cbce90259db881 server_cookie=e0fb3fb99a401108 in_seq=0 out_seq=0 2024-04-21T08:50:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.082+0000 7f7467fff640 10 client.16872 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.082+0000 7f7467fff640 1 -- 172.21.15.7:0/2216230312 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556749e24c10 con 0x556749f32430 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.082+0000 7f7467fff640 10 client.16872 handle_command_reply: tid=0 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.082+0000 7f7491ba0740 1 client.16872 shutdown 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7491ba0740 1 -- 172.21.15.7:0/2216230312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 msgr2=0x556749ebf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7491ba0740 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebf3e0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7470050040 tx=0x7f747002c500 comp rx=0 tx=0).stop 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7491ba0740 1 -- 172.21.15.7:0/2216230312 shutdown_connections 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7491ba0740 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556749f32430 0x556749f34820 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7491ba0740 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556749ebba00 0x556749ebf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7491ba0740 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556749eb2e80 0x556749ebf3e0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7491ba0740 1 --2- 172.21.15.7:0/2216230312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556749eb24d0 0x556749ec3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7467fff640 0 client.16872 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7491ba0740 1 -- 172.21.15.7:0/2216230312 >> 172.21.15.7:0/2216230312 conn(0x556749eacba0 msgr2=0x556749eabc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7491ba0740 1 -- 172.21.15.7:0/2216230312 shutdown_connections 2024-04-21T08:50:50.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7491ba0740 1 -- 172.21.15.7:0/2216230312 wait complete. 2024-04-21T08:50:50.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f7491ba0740 20 client.16872 trim_cache size 0 max 16384 2024-04-21T08:50:50.090 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:50:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f748efb2640 1 -- 172.21.15.7:0/3074328436 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f74740a3d20 msgr2=0x7f74740a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f748efb2640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f74740a3d20 0x7f74740a61e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7470005ec0 tx=0x7f74700040d0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f748efb2640 1 -- 172.21.15.7:0/3074328436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74881044f0 msgr2=0x7f748819d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f748efb2640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74881044f0 0x7f748819d000 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f747800c970 tx=0x7f747800ce30 comp rx=0 tx=0).stop 2024-04-21T08:50:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f748efb2640 1 -- 172.21.15.7:0/3074328436 shutdown_connections 2024-04-21T08:50:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f748efb2640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74881044f0 0x7f748819d000 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f748efb2640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f74740a3d20 0x7f74740a61e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f748efb2640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7488103b50 0x7f7488198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f748efb2640 1 --2- 172.21.15.7:0/3074328436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7488102270 0x7f7488198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.083+0000 7f748efb2640 1 -- 172.21.15.7:0/3074328436 >> 172.21.15.7:0/3074328436 conn(0x7f74880fda10 msgr2=0x7f74880fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:50.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.084+0000 7f748efb2640 1 -- 172.21.15.7:0/3074328436 shutdown_connections 2024-04-21T08:50:50.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:50.084+0000 7f748efb2640 1 -- 172.21.15.7:0/3074328436 wait complete. 2024-04-21T08:50:51.095 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:50:51.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:50:51.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:50:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.214+0000 7f5bdbec1640 1 -- 172.21.15.7:0/4045109306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bd41031c0 msgr2=0x7f5bd41035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.214+0000 7f5bdbec1640 1 --2- 172.21.15.7:0/4045109306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bd41031c0 0x7f5bd41035a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009900 tx=0x7f5bd002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.214+0000 7f5bdbec1640 1 -- 172.21.15.7:0/4045109306 shutdown_connections 2024-04-21T08:50:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.214+0000 7f5bdbec1640 1 --2- 172.21.15.7:0/4045109306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd410c500 0x7f5bd410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.214+0000 7f5bdbec1640 1 --2- 172.21.15.7:0/4045109306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bd4103ae0 0x7f5bd410bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.214+0000 7f5bdbec1640 1 --2- 172.21.15.7:0/4045109306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bd41031c0 0x7f5bd41035a0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.214+0000 7f5bdbec1640 1 -- 172.21.15.7:0/4045109306 >> 172.21.15.7:0/4045109306 conn(0x7f5bd4078000 msgr2=0x7f5bd4078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.214+0000 7f5bdbec1640 1 -- 172.21.15.7:0/4045109306 shutdown_connections 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.214+0000 7f5bdbec1640 1 -- 172.21.15.7:0/4045109306 wait complete. 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bdbec1640 1 Processor -- start 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bdbec1640 1 -- start start 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bdbec1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bd41031c0 0x7f5bd4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bdbec1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd4103ae0 0x7f5bd4198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bdbec1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bd410c500 0x7f5bd419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bdbec1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd4112170 con 0x7f5bd41031c0 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bdbec1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd4111ff0 con 0x7f5bd4103ae0 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bdbec1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41122f0 con 0x7f5bd410c500 2024-04-21T08:50:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bd9c5d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bd41031c0 0x7f5bd4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bd9c5d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bd41031c0 0x7f5bd4198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38662/0 (socket says 172.21.15.7:38662) 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/301028726 learned_addr learned my addr 172.21.15.7:0/301028726 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bd945c640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd4103ae0 0x7f5bd4198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.215+0000 7f5bda45e640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bd410c500 0x7f5bd419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.216+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/301028726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bd410c500 msgr2=0x7f5bd419cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.216+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bd410c500 0x7f5bd419cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.216+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/301028726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd4103ae0 msgr2=0x7f5bd4198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.216+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd4103ae0 0x7f5bd4198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.216+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/301028726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd419d720 con 0x7f5bd41031c0 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.216+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bd41031c0 0x7f5bd4198730 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0005940 tx=0x7f5bd00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.216+0000 7f5bcaffd640 1 -- 172.21.15.7:0/301028726 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0043020 con 0x7f5bd41031c0 2024-04-21T08:50:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.216+0000 7f5bcaffd640 1 -- 172.21.15.7:0/301028726 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd002dba0 con 0x7f5bd41031c0 2024-04-21T08:50:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.216+0000 7f5bdbec1640 1 -- 172.21.15.7:0/301028726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd419d9b0 con 0x7f5bd41031c0 2024-04-21T08:50:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.216+0000 7f5bcaffd640 1 -- 172.21.15.7:0/301028726 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0003930 con 0x7f5bd41031c0 2024-04-21T08:50:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.216+0000 7f5bdbec1640 1 -- 172.21.15.7:0/301028726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd41a52a0 con 0x7f5bd41031c0 2024-04-21T08:50:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.218+0000 7f5bcaffd640 1 -- 172.21.15.7:0/301028726 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f5bd0046020 con 0x7f5bd41031c0 2024-04-21T08:50:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.218+0000 7f5bdeb91740 1 Processor -- start 2024-04-21T08:50:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bdeb91740 1 -- start start 2024-04-21T08:50:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b1ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1d290 0x561ee8b25840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b28260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561ee8a80f00 con 0x561ee8b1c8e0 2024-04-21T08:50:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ee8a1e590 con 0x561ee8b25e10 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ee8a7cec0 con 0x561ee8b1d290 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bda45e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b1ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bda45e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b1ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38678/0 (socket says 172.21.15.7:38678) 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bda45e640 1 -- 172.21.15.7:0/3164944683 learned_addr learned my addr 172.21.15.7:0/3164944683 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bd945c640 1 --2- 172.21.15.7:0/3164944683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b28260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bcaffd640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5bc00a3da0 0x7f5bc00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/3164944683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1d290 0x561ee8b25840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bda45e640 1 -- 172.21.15.7:0/3164944683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1d290 msgr2=0x561ee8b25840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.219+0000 7f5bda45e640 1 --2- 172.21.15.7:0/3164944683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1d290 0x561ee8b25840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.220+0000 7f5bda45e640 1 -- 172.21.15.7:0/3164944683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 msgr2=0x561ee8b28260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.220+0000 7f5bda45e640 1 --2- 172.21.15.7:0/3164944683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b28260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.220+0000 7f5bda45e640 1 -- 172.21.15.7:0/3164944683 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ee885b7e0 con 0x561ee8b1c8e0 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.220+0000 7f5bcaffd640 1 -- 172.21.15.7:0/301028726 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f5bd00f2af0 con 0x7f5bd41031c0 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.220+0000 7f5bda45e640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5bc00a3da0 0x7f5bc00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.220+0000 7f5bda45e640 1 --2- 172.21.15.7:0/3164944683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b1ccc0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc011430 tx=0x7f5bcc0118f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33bacf058bf06073 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.220+0000 7f5bbffff640 1 -- 172.21.15.7:0/3164944683 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc014a10 con 0x561ee8b1c8e0 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.220+0000 7f5bbffff640 1 -- 172.21.15.7:0/3164944683 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bcc014db0 con 0x561ee8b1c8e0 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.220+0000 7f5bbffff640 1 -- 172.21.15.7:0/3164944683 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc01c9f0 con 0x561ee8b1c8e0 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3164944683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 msgr2=0x561ee8b1ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bda45e640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5bc00a3da0 0x7f5bc00a6260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc020000 tx=0x7f5bcc01d000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/3164944683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b1ccc0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc011430 tx=0x7f5bcc0118f0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3164944683 shutdown_connections 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/3164944683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b28260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/3164944683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1d290 0x561ee8b25840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/3164944683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b1ccc0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3164944683 >> 172.21.15.7:0/3164944683 conn(0x561ee8b16fb0 msgr2=0x561ee8b17390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3164944683 shutdown_connections 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3164944683 wait complete. 2024-04-21T08:50:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bdeb91740 1 -- start start 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.221+0000 7f5bdeb91740 1 Processor -- start 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1c8e0 0x561ee8b22d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1d290 0x561ee8b23270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b237b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561ee8b28d80 con 0x561ee8b1d290 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ee8a3a580 con 0x561ee8b25e10 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ee8b28f00 con 0x561ee8b1c8e0 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bdeb91740 10 client.0 _ll_register_callbacks cb 0x561ee8a40650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bd945c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1d290 0x561ee8b23270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bd945c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1d290 0x561ee8b23270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38690/0 (socket says 172.21.15.7:38690) 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bd945c640 1 -- 172.21.15.7:0/3424963680 learned_addr learned my addr 172.21.15.7:0/3424963680 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1c8e0 0x561ee8b22d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bda45e640 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b237b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bd945c640 1 -- 172.21.15.7:0/3424963680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1c8e0 msgr2=0x561ee8b22d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bd945c640 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1c8e0 0x561ee8b22d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bd945c640 1 -- 172.21.15.7:0/3424963680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 msgr2=0x561ee8b237b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bd945c640 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b237b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bd945c640 1 -- 172.21.15.7:0/3424963680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ee88bbf70 con 0x561ee8b1d290 2024-04-21T08:50:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bda45e640 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b237b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bd945c640 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1d290 0x561ee8b23270 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400bf70 tx=0x7f5bc40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55eb5cb8467c9590 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.222+0000 7f5bbd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/3424963680 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4004020 con 0x561ee8b1d290 2024-04-21T08:50:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/3424963680 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f5bc40041c0 con 0x561ee8b1d290 2024-04-21T08:50:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3424963680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bcc014db0 con 0x561ee8b1d290 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/3424963680 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5bc4025050 con 0x561ee8b1d290 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bdeb91740 10 client.16884 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3424963680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561ee8753170 con 0x561ee8b1d290 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/3424963680 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bcc014db0 con 0x561ee8b1d290 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/3424963680 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f5bc40038c0 con 0x561ee8b1d290 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bdeb91740 10 client.16884 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bdeb91740 10 client.16884 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bdeb91740 20 client.16884 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ee8b36120 0x561ee8b38560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bdeb91740 4 client.16884 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.223+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3424963680 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ee8a8e920 con 0x561ee8b36120 2024-04-21T08:50:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.224+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ee8b36120 0x561ee8b38560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.224+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ee8b36120 0x561ee8b38560 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a74e042348add3eb server_cookie=5f55a3b8e18a66dd in_seq=0 out_seq=0 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.224+0000 7f5bbd7fa640 10 client.16884 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/3424963680 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ee8a8e920 con 0x561ee8b36120 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bbd7fa640 10 client.16884 handle_command_reply: tid=0 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 1 client.16884 shutdown 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3424963680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1d290 msgr2=0x561ee8b23270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1d290 0x561ee8b23270 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400bf70 tx=0x7f5bc40118d0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3424963680 shutdown_connections 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ee8b36120 0x561ee8b38560 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b237b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1d290 0x561ee8b23270 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/3424963680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1c8e0 0x561ee8b22d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bbd7fa640 0 client.16884 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:51.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3424963680 >> 172.21.15.7:0/3424963680 conn(0x561ee8b16fb0 msgr2=0x561ee8b18d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3424963680 shutdown_connections 2024-04-21T08:50:51.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 1 -- 172.21.15.7:0/3424963680 wait complete. 2024-04-21T08:50:51.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.225+0000 7f5bdeb91740 20 client.16884 trim_cache size 0 max 16384 2024-04-21T08:50:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.243+0000 7f5bdeb91740 1 Processor -- start 2024-04-21T08:50:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.243+0000 7f5bdeb91740 1 -- start start 2024-04-21T08:50:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b25460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b1d290 0x561ee8b21c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b25e10 0x561ee8b31c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41ad460 con 0x561ee8b1c8e0 2024-04-21T08:50:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41afce0 con 0x561ee8b1d290 2024-04-21T08:50:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41adad0 con 0x561ee8b25e10 2024-04-21T08:50:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bd9c5d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b25460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bd9c5d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b25460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38694/0 (socket says 172.21.15.7:38694) 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/2956297854 learned_addr learned my addr 172.21.15.7:0/2956297854 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bda45e640 1 --2- 172.21.15.7:0/2956297854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b25e10 0x561ee8b31c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bd945c640 1 --2- 172.21.15.7:0/2956297854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b1d290 0x561ee8b21c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/2956297854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b25e10 msgr2=0x561ee8b31c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/2956297854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b25e10 0x561ee8b31c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/2956297854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b1d290 msgr2=0x561ee8b21c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/2956297854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b1d290 0x561ee8b21c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/2956297854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ee8b1cff0 con 0x561ee8b1c8e0 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/2956297854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b25460 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00bb2a0 tx=0x7f5bd00f5070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e4c529b306f68a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bbf7fe640 1 -- 172.21.15.7:0/2956297854 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd00c7540 con 0x561ee8b1c8e0 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bbf7fe640 1 -- 172.21.15.7:0/2956297854 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd00be8d0 con 0x561ee8b1c8e0 2024-04-21T08:50:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.244+0000 7f5bbf7fe640 1 -- 172.21.15.7:0/2956297854 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd00baf40 con 0x561ee8b1c8e0 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.245+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2956297854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 msgr2=0x561ee8b25460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.245+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/2956297854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b25460 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00bb2a0 tx=0x7f5bd00f5070 comp rx=0 tx=0).stop 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.245+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2956297854 shutdown_connections 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.245+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/2956297854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b25e10 0x561ee8b31c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.245+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/2956297854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b1d290 0x561ee8b21c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.245+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/2956297854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b25460 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.245+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2956297854 >> 172.21.15.7:0/2956297854 conn(0x561ee8b16fb0 msgr2=0x561ee8b18c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.245+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2956297854 shutdown_connections 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.245+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2956297854 wait complete. 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.245+0000 7f5bdeb91740 1 -- start start 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bdeb91740 1 Processor -- start 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b2e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1d290 0x561ee8b2a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bdeb91740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b2ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41ae1a0 con 0x561ee8b1c8e0 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41af610 con 0x561ee8b25e10 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bdeb91740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41ae870 con 0x561ee8b1d290 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bdeb91740 10 client.0 _ll_register_callbacks cb 0x7f5bc400bb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bd9c5d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b2e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bd9c5d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b2e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38702/0 (socket says 172.21.15.7:38702) 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/2977421968 learned_addr learned my addr 172.21.15.7:0/2977421968 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bda45e640 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b2ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bd945c640 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1d290 0x561ee8b2a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/2977421968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1d290 msgr2=0x561ee8b2a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1d290 0x561ee8b2a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/2977421968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 msgr2=0x561ee8b2ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b2ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bd9c5d640 1 -- 172.21.15.7:0/2977421968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ee8b2b1c0 con 0x561ee8b1c8e0 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.246+0000 7f5bd945c640 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1d290 0x561ee8b2a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bd9c5d640 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b2e2f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00f0450 tx=0x7f5bd0005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e9e43887662d11 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bbd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/2977421968 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd002dd50 con 0x561ee8b1c8e0 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/2977421968 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f5bd00baf40 con 0x561ee8b1c8e0 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2977421968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bd00be8d0 con 0x561ee8b1c8e0 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/2977421968 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5bd00c7540 con 0x561ee8b1c8e0 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bdeb91740 10 client.16890 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2977421968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561ee8b9d4c0 con 0x561ee8b1c8e0 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/2977421968 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bd00be8d0 con 0x561ee8b1c8e0 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/2977421968 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f5bd00c7140 con 0x561ee8b1c8e0 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bdeb91740 10 client.16890 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:51.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bdeb91740 10 client.16890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:51.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bdeb91740 20 client.16890 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:51.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ee8b9db50 0x561ee8b9ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bdeb91740 4 client.16890 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:50:51.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.247+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2977421968 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561ee8a88790 con 0x561ee8b9db50 2024-04-21T08:50:51.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.248+0000 7f5bd945c640 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ee8b9db50 0x561ee8b9ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.248+0000 7f5bd945c640 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ee8b9db50 0x561ee8b9ff70 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=720658dd35508416 server_cookie=c4f757c3816b81f in_seq=0 out_seq=0 2024-04-21T08:50:51.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.248+0000 7f5bbd7fa640 10 client.16890 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:51.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.249+0000 7f5bbd7fa640 1 -- 172.21.15.7:0/2977421968 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561ee8a88790 con 0x561ee8b9db50 2024-04-21T08:50:51.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.249+0000 7f5bbd7fa640 10 client.16890 handle_command_reply: tid=0 2024-04-21T08:50:51.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.249+0000 7f5bdeb91740 1 client.16890 shutdown 2024-04-21T08:50:51.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.249+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2977421968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 msgr2=0x561ee8b2e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.249+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b2e2f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00f0450 tx=0x7f5bd0005cb0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.249+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2977421968 shutdown_connections 2024-04-21T08:50:51.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ee8b9db50 0x561ee8b9ff70 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ee8b25e10 0x561ee8b2ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ee8b1d290 0x561ee8b2a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdeb91740 1 --2- 172.21.15.7:0/2977421968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ee8b1c8e0 0x561ee8b2e2f0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2977421968 >> 172.21.15.7:0/2977421968 conn(0x561ee8b16fb0 msgr2=0x561ee8b22cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bbd7fa640 0 client.16890 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:51.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2977421968 shutdown_connections 2024-04-21T08:50:51.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdeb91740 1 -- 172.21.15.7:0/2977421968 wait complete. 2024-04-21T08:50:51.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdeb91740 20 client.16890 trim_cache size 0 max 16384 2024-04-21T08:50:51.256 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:50:51.257 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:50:51.257 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "81ebd966-d9b6-44a6-b87d-1c305ebffa72", 2024-04-21T08:50:51.257 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:50:51.257 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:50:51.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 -- 172.21.15.7:0/301028726 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5bc00a3da0 msgr2=0x7f5bc00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5bc00a3da0 0x7f5bc00a6260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc020000 tx=0x7f5bcc01d000 comp rx=0 tx=0).stop 2024-04-21T08:50:51.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 -- 172.21.15.7:0/301028726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bd41031c0 msgr2=0x7f5bd4198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bd41031c0 0x7f5bd4198730 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0005940 tx=0x7f5bd00045e0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 -- 172.21.15.7:0/301028726 shutdown_connections 2024-04-21T08:50:51.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5bc00a3da0 0x7f5bc00a6260 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bd410c500 0x7f5bd419cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd4103ae0 0x7f5bd4198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 --2- 172.21.15.7:0/301028726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bd41031c0 0x7f5bd4198730 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 -- 172.21.15.7:0/301028726 >> 172.21.15.7:0/301028726 conn(0x7f5bd4078000 msgr2=0x7f5bd40fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 -- 172.21.15.7:0/301028726 shutdown_connections 2024-04-21T08:50:51.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.250+0000 7f5bdbec1640 1 -- 172.21.15.7:0/301028726 wait complete. 2024-04-21T08:50:51.260 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:50 smithi007 ceph-mon[28477]: pgmap v188: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 4.05k op/s 2024-04-21T08:50:51.260 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:50 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3382802058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:51.261 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:50:51.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:50:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:50 smithi016 ceph-mon[31259]: pgmap v188: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 4.05k op/s 2024-04-21T08:50:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:50 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3382802058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:51.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:50 smithi077 ceph-mon[31615]: pgmap v188: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 4.05k op/s 2024-04-21T08:50:51.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:50 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3382802058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:51.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.368+0000 7f3ebcb96640 1 -- 172.21.15.7:0/3419168894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3eb81072c0 msgr2=0x7f3eb810ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.368+0000 7f3ebcb96640 1 --2- 172.21.15.7:0/3419168894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3eb81072c0 0x7f3eb810ee70 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0009ca0 tx=0x7f3eb002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.369+0000 7f3ebcb96640 1 -- 172.21.15.7:0/3419168894 shutdown_connections 2024-04-21T08:50:51.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.369+0000 7f3ebcb96640 1 --2- 172.21.15.7:0/3419168894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3eb81072c0 0x7f3eb810ee70 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.369+0000 7f3ebcb96640 1 --2- 172.21.15.7:0/3419168894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb806b300 0x7f3eb8106be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.369+0000 7f3ebcb96640 1 --2- 172.21.15.7:0/3419168894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb806a950 0x7f3eb806ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.369+0000 7f3ebcb96640 1 -- 172.21.15.7:0/3419168894 >> 172.21.15.7:0/3419168894 conn(0x7f3eb8077f40 msgr2=0x7f3eb8078350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.369+0000 7f3ebcb96640 1 -- 172.21.15.7:0/3419168894 shutdown_connections 2024-04-21T08:50:51.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.369+0000 7f3ebcb96640 1 -- 172.21.15.7:0/3419168894 wait complete. 2024-04-21T08:50:51.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.369+0000 7f3ebcb96640 1 Processor -- start 2024-04-21T08:50:51.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.369+0000 7f3ebcb96640 1 -- start start 2024-04-21T08:50:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.369+0000 7f3ebcb96640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb806a950 0x7f3eb81986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3ebcb96640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3eb81072c0 0x7f3eb8198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3ebcb96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb819cf40 0x7f3eb819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3ebcb96640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81121d0 con 0x7f3eb81072c0 2024-04-21T08:50:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3ebcb96640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb8112050 con 0x7f3eb806a950 2024-04-21T08:50:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3ebcb96640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb8112350 con 0x7f3eb819cf40 2024-04-21T08:50:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3eb81072c0 0x7f3eb8198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3eb81072c0 0x7f3eb8198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38726/0 (socket says 172.21.15.7:38726) 2024-04-21T08:50:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eb659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb806a950 0x7f3eb81986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaffff640 1 -- 172.21.15.7:0/1027456838 learned_addr learned my addr 172.21.15.7:0/1027456838 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eb6d9d640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb819cf40 0x7f3eb819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaffff640 1 -- 172.21.15.7:0/1027456838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb819cf40 msgr2=0x7f3eb819d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaffff640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb819cf40 0x7f3eb819d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaffff640 1 -- 172.21.15.7:0/1027456838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb806a950 msgr2=0x7f3eb81986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaffff640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb806a950 0x7f3eb81986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaffff640 1 -- 172.21.15.7:0/1027456838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb819da30 con 0x7f3eb81072c0 2024-04-21T08:50:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaffff640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3eb81072c0 0x7f3eb8198c10 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000e9f0 tx=0x7f3ea000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaf7fe640 1 -- 172.21.15.7:0/1027456838 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea000ccc0 con 0x7f3eb81072c0 2024-04-21T08:50:51.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaf7fe640 1 -- 172.21.15.7:0/1027456838 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea00108e0 con 0x7f3eb81072c0 2024-04-21T08:50:51.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3ebcb96640 1 -- 172.21.15.7:0/1027456838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb819dd20 con 0x7f3eb81072c0 2024-04-21T08:50:51.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3eaf7fe640 1 -- 172.21.15.7:0/1027456838 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0018860 con 0x7f3eb81072c0 2024-04-21T08:50:51.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.370+0000 7f3ebcb96640 1 -- 172.21.15.7:0/1027456838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb81a5560 con 0x7f3eb81072c0 2024-04-21T08:50:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3eaf7fe640 1 -- 172.21.15.7:0/1027456838 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f3ea0010430 con 0x7f3eb81072c0 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3ebf787740 1 Processor -- start 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3ebf787740 1 -- start start 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a31606990 0x562a31606d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a316072b0 0x562a3160f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a31612280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3ebf787740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562a315671e0 con 0x562a31606990 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3ebf787740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562a31504390 con 0x562a3160fe90 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3ebf787740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562a315631a0 con 0x562a316072b0 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3eb6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a31606990 0x562a31606d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a31612280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3eb6d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a31606990 0x562a31606d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38740/0 (socket says 172.21.15.7:38740) 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3eb6d9d640 1 -- 172.21.15.7:0/3954087857 learned_addr learned my addr 172.21.15.7:0/3954087857 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3eb659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a316072b0 0x562a3160f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.373+0000 7f3eaf7fe640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ea40a3da0 0x7f3ea40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eb6d9d640 1 -- 172.21.15.7:0/3954087857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a316072b0 msgr2=0x562a3160f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eb6d9d640 1 --2- 172.21.15.7:0/3954087857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a316072b0 0x562a3160f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eb6d9d640 1 -- 172.21.15.7:0/3954087857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 msgr2=0x562a31612280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eb6d9d640 1 --2- 172.21.15.7:0/3954087857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a31612280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eb6d9d640 1 -- 172.21.15.7:0/3954087857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a313437d0 con 0x562a31606990 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eaf7fe640 1 -- 172.21.15.7:0/1027456838 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f3ea00d0e00 con 0x7f3eb81072c0 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eb6d9d640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ea40a3da0 0x7f3ea40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eb6d9d640 1 --2- 172.21.15.7:0/3954087857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a31606990 0x562a31606d70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0037040 tx=0x7f3eb0005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14cf5dd821323c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eacff9640 1 -- 172.21.15.7:0/3954087857 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0039040 con 0x562a31606990 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eacff9640 1 -- 172.21.15.7:0/3954087857 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3eb0003e20 con 0x562a31606990 2024-04-21T08:50:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eacff9640 1 -- 172.21.15.7:0/3954087857 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0034440 con 0x562a31606990 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.374+0000 7f3eb6d9d640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ea40a3da0 0x7f3ea40a6260 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3eb004c000 tx=0x7f3eb0045000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.375+0000 7f3ebf787740 1 -- 172.21.15.7:0/3954087857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a31606990 msgr2=0x562a31606d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.375+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3954087857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a31606990 0x562a31606d70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0037040 tx=0x7f3eb0005f70 comp rx=0 tx=0).stop 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.375+0000 7f3ebf787740 1 -- 172.21.15.7:0/3954087857 shutdown_connections 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.375+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3954087857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a31612280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.375+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3954087857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a316072b0 0x562a3160f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.375+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3954087857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a31606990 0x562a31606d70 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.375+0000 7f3ebf787740 1 -- 172.21.15.7:0/3954087857 >> 172.21.15.7:0/3954087857 conn(0x562a316010c0 msgr2=0x562a316014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.375+0000 7f3ebf787740 1 -- 172.21.15.7:0/3954087857 shutdown_connections 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.375+0000 7f3ebf787740 1 -- 172.21.15.7:0/3954087857 wait complete. 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.375+0000 7f3ebf787740 1 -- start start 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.375+0000 7f3ebf787740 1 Processor -- start 2024-04-21T08:50:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a31606990 0x562a3160cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3160d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a3160d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3ebf787740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562a31612c80 con 0x562a316072b0 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3ebf787740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562a315202f0 con 0x562a3160fe90 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3ebf787740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562a31612e00 con 0x562a31606990 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3ebf787740 10 client.0 _ll_register_callbacks cb 0x562a3152d580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3160d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3160d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38748/0 (socket says 172.21.15.7:38748) 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eb659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a31606990 0x562a3160cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eb6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a3160d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eaffff640 1 -- 172.21.15.7:0/3430559942 learned_addr learned my addr 172.21.15.7:0/3430559942 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eaffff640 1 -- 172.21.15.7:0/3430559942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a31606990 msgr2=0x562a3160cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eaffff640 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a31606990 0x562a3160cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eaffff640 1 -- 172.21.15.7:0/3430559942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 msgr2=0x562a3160d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eaffff640 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a3160d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eaffff640 1 -- 172.21.15.7:0/3430559942 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a312793f0 con 0x562a316072b0 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eb659c640 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a31606990 0x562a3160cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3eaffff640 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3160d460 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000c6a0 tx=0x7f3ea00096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3244424775aa4ed server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3e9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3e9dffb640 1 -- 172.21.15.7:0/3430559942 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0010720 con 0x562a316072b0 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3e9dffb640 1 -- 172.21.15.7:0/3430559942 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f3ea00d4020 con 0x562a316072b0 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.376+0000 7f3ebf787740 1 -- 172.21.15.7:0/3430559942 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3eb0003e20 con 0x562a316072b0 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3e9dffb640 1 -- 172.21.15.7:0/3430559942 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3ea009ca60 con 0x562a316072b0 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3ebf787740 10 client.16902 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3ebf787740 1 -- 172.21.15.7:0/3430559942 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562a312390f0 con 0x562a316072b0 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3e9dffb640 1 -- 172.21.15.7:0/3430559942 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3eb0003e20 con 0x562a316072b0 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3e9dffb640 1 -- 172.21.15.7:0/3430559942 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f3ea009dce0 con 0x562a316072b0 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3ebf787740 10 client.16902 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3ebf787740 10 client.16902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3ebf787740 20 client.16902 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a3161fe80 0x562a31622290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3ebf787740 4 client.16902 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3ebf787740 1 -- 172.21.15.7:0/3430559942 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a31574c00 con 0x562a3161fe80 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.377+0000 7f3eb659c640 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a3161fe80 0x562a31622290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.378+0000 7f3eb659c640 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a3161fe80 0x562a31622290 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b73a11f42dfc1750 server_cookie=4796fedcf61212ea in_seq=0 out_seq=0 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.378+0000 7f3e9dffb640 10 client.16902 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3e9dffb640 1 -- 172.21.15.7:0/3430559942 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a31574c00 con 0x562a3161fe80 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3e9dffb640 10 client.16902 handle_command_reply: tid=0 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 1 client.16902 shutdown 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 1 -- 172.21.15.7:0/3430559942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 msgr2=0x562a3160d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3160d460 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000c6a0 tx=0x7f3ea00096a0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 1 -- 172.21.15.7:0/3430559942 shutdown_connections 2024-04-21T08:50:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a3161fe80 0x562a31622290 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a3160d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3160d460 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3430559942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a31606990 0x562a3160cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 1 -- 172.21.15.7:0/3430559942 >> 172.21.15.7:0/3430559942 conn(0x562a316010c0 msgr2=0x562a31602e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3e9dffb640 0 client.16902 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 1 -- 172.21.15.7:0/3430559942 shutdown_connections 2024-04-21T08:50:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 1 -- 172.21.15.7:0/3430559942 wait complete. 2024-04-21T08:50:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.380+0000 7f3ebf787740 20 client.16902 trim_cache size 0 max 16384 2024-04-21T08:50:51.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3ebf787740 1 Processor -- start 2024-04-21T08:50:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3ebf787740 1 -- start start 2024-04-21T08:50:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a31606990 0x562a3160f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3161bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a3160fe90 0x562a31627b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3ebf787740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81ab1e0 con 0x562a316072b0 2024-04-21T08:50:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3ebf787740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81ada60 con 0x562a31606990 2024-04-21T08:50:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3ebf787740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81ab850 con 0x562a3160fe90 2024-04-21T08:50:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3161bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3161bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38750/0 (socket says 172.21.15.7:38750) 2024-04-21T08:50:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3eaffff640 1 -- 172.21.15.7:0/3807647122 learned_addr learned my addr 172.21.15.7:0/3807647122 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.398+0000 7f3eb659c640 1 --2- 172.21.15.7:0/3807647122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a31606990 0x562a3160f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.399+0000 7f3eb6d9d640 1 --2- 172.21.15.7:0/3807647122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a3160fe90 0x562a31627b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.399+0000 7f3eaffff640 1 -- 172.21.15.7:0/3807647122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a3160fe90 msgr2=0x562a31627b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.399+0000 7f3eaffff640 1 --2- 172.21.15.7:0/3807647122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a3160fe90 0x562a31627b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.399+0000 7f3eaffff640 1 -- 172.21.15.7:0/3807647122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a31606990 msgr2=0x562a3160f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.399+0000 7f3eaffff640 1 --2- 172.21.15.7:0/3807647122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a31606990 0x562a3160f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.399+0000 7f3eaffff640 1 -- 172.21.15.7:0/3807647122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a3160b470 con 0x562a316072b0 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.399+0000 7f3eaffff640 1 --2- 172.21.15.7:0/3807647122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3161bb40 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000e9c0 tx=0x7f3ea000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09cd108ac4e77e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.399+0000 7f3e9ffff640 1 -- 172.21.15.7:0/3807647122 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0010720 con 0x562a316072b0 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.399+0000 7f3e9ffff640 1 -- 172.21.15.7:0/3807647122 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea00d4460 con 0x562a316072b0 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.399+0000 7f3e9ffff640 1 -- 172.21.15.7:0/3807647122 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0087340 con 0x562a316072b0 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 -- 172.21.15.7:0/3807647122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 msgr2=0x562a3161bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3807647122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3161bb40 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000e9c0 tx=0x7f3ea000ce60 comp rx=0 tx=0).stop 2024-04-21T08:50:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 -- 172.21.15.7:0/3807647122 shutdown_connections 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3807647122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a3160fe90 0x562a31627b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3807647122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a3161bb40 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 --2- 172.21.15.7:0/3807647122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a31606990 0x562a3160f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 -- 172.21.15.7:0/3807647122 >> 172.21.15.7:0/3807647122 conn(0x562a316010c0 msgr2=0x562a3160b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 -- 172.21.15.7:0/3807647122 shutdown_connections 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 -- 172.21.15.7:0/3807647122 wait complete. 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 -- start start 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 Processor -- start 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a31606990 0x562a31618340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a31609810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a31609d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81abf20 con 0x562a316072b0 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81ad390 con 0x562a3160fe90 2024-04-21T08:50:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb81ac5f0 con 0x562a31606990 2024-04-21T08:50:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.400+0000 7f3ebf787740 10 client.0 _ll_register_callbacks cb 0x7f3ea00d4640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a31609810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a31609810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38762/0 (socket says 172.21.15.7:38762) 2024-04-21T08:50:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3eaffff640 1 -- 172.21.15.7:0/240824075 learned_addr learned my addr 172.21.15.7:0/240824075 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3eb6d9d640 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a31609d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3eb659c640 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a31606990 0x562a31618340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3eaffff640 1 -- 172.21.15.7:0/240824075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a31606990 msgr2=0x562a31618340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3eaffff640 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a31606990 0x562a31618340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3eaffff640 1 -- 172.21.15.7:0/240824075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 msgr2=0x562a31609d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3eaffff640 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a31609d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3eaffff640 1 -- 172.21.15.7:0/240824075 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a3160a410 con 0x562a316072b0 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3eaffff640 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a31609810 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00a57e0 tx=0x7f3ea009c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfdde5ba02ccff7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3e9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3e9dffb640 1 -- 172.21.15.7:0/240824075 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0010720 con 0x562a316072b0 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3e9dffb640 1 -- 172.21.15.7:0/240824075 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f3ea00a1ef0 con 0x562a316072b0 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3ebf787740 1 -- 172.21.15.7:0/240824075 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ea00d4460 con 0x562a316072b0 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3e9dffb640 1 -- 172.21.15.7:0/240824075 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3ea00a5620 con 0x562a316072b0 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3ebf787740 10 client.16908 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.401+0000 7f3ebf787740 1 -- 172.21.15.7:0/240824075 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562a316862d0 con 0x562a316072b0 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.402+0000 7f3e9dffb640 1 -- 172.21.15.7:0/240824075 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea00b8030 con 0x562a316072b0 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.402+0000 7f3e9dffb640 1 -- 172.21.15.7:0/240824075 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f3ea009dce0 con 0x562a316072b0 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.402+0000 7f3ebf787740 10 client.16908 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.402+0000 7f3ebf787740 10 client.16908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.402+0000 7f3ebf787740 20 client.16908 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.402+0000 7f3ebf787740 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a31686810 0x562a31688c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.402+0000 7f3ebf787740 4 client.16908 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:50:51.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.402+0000 7f3ebf787740 1 -- 172.21.15.7:0/240824075 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562a3156ea70 con 0x562a31686810 2024-04-21T08:50:51.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.402+0000 7f3eb659c640 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a31686810 0x562a31688c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.402+0000 7f3eb659c640 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a31686810 0x562a31688c30 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3d7f37dde085007 server_cookie=a9bcd8b6337e81e8 in_seq=0 out_seq=0 2024-04-21T08:50:51.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.402+0000 7f3e9dffb640 10 client.16908 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:51.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3e9dffb640 1 -- 172.21.15.7:0/240824075 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x562a3156ea70 con 0x562a31686810 2024-04-21T08:50:51.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3e9dffb640 10 client.16908 handle_command_reply: tid=0 2024-04-21T08:50:51.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3ebf787740 1 client.16908 shutdown 2024-04-21T08:50:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3ebf787740 1 -- 172.21.15.7:0/240824075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 msgr2=0x562a31609810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3ebf787740 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a31609810 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00a57e0 tx=0x7f3ea009c0e0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3ebf787740 1 -- 172.21.15.7:0/240824075 shutdown_connections 2024-04-21T08:50:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3ebf787740 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a31686810 0x562a31688c30 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3ebf787740 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a3160fe90 0x562a31609d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3e9dffb640 0 client.16908 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3ebf787740 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a316072b0 0x562a31609810 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3ebf787740 1 --2- 172.21.15.7:0/240824075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a31606990 0x562a31618340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3ebf787740 1 -- 172.21.15.7:0/240824075 >> 172.21.15.7:0/240824075 conn(0x562a316010c0 msgr2=0x562a315fb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3ebf787740 1 -- 172.21.15.7:0/240824075 shutdown_connections 2024-04-21T08:50:51.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.404+0000 7f3ebf787740 1 -- 172.21.15.7:0/240824075 wait complete. 2024-04-21T08:50:51.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebf787740 20 client.16908 trim_cache size 0 max 16384 2024-04-21T08:50:51.411 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:50:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (6838 inodes in the stack)", 2024-04-21T08:50:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:50:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "81ebd966-d9b6-44a6-b87d-1c305ebffa72": { 2024-04-21T08:50:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:50:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "81ebd966-d9b6-44a6-b87d-1c305ebffa72", 2024-04-21T08:50:51.411 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:50:51.412 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:50:51.412 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:50:51.412 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:50:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 -- 172.21.15.7:0/1027456838 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ea40a3da0 msgr2=0x7f3ea40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ea40a3da0 0x7f3ea40a6260 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3eb004c000 tx=0x7f3eb0045000 comp rx=0 tx=0).stop 2024-04-21T08:50:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 -- 172.21.15.7:0/1027456838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3eb81072c0 msgr2=0x7f3eb8198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3eb81072c0 0x7f3eb8198c10 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000e9f0 tx=0x7f3ea000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 -- 172.21.15.7:0/1027456838 shutdown_connections 2024-04-21T08:50:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f3ea40a3da0 0x7f3ea40a6260 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3eb819cf40 0x7f3eb819d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3eb81072c0 0x7f3eb8198c10 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 --2- 172.21.15.7:0/1027456838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eb806a950 0x7f3eb81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 -- 172.21.15.7:0/1027456838 >> 172.21.15.7:0/1027456838 conn(0x7f3eb8077f40 msgr2=0x7f3eb80ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 -- 172.21.15.7:0/1027456838 shutdown_connections 2024-04-21T08:50:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:50:51.405+0000 7f3ebcb96640 1 -- 172.21.15.7:0/1027456838 wait complete. 2024-04-21T08:50:51.415 INFO:tasks.cephfs.filesystem:scrub status for tag:81ebd966-d9b6-44a6-b87d-1c305ebffa72 - {'path': '/', 'tag': '81ebd966-d9b6-44a6-b87d-1c305ebffa72', 'options': 'recursive,force'} 2024-04-21T08:50:52.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:51 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:50:52.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:51 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:52.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:51 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:50:52.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:51 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:50:52.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:51 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:52.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:51 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:50:52.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:51 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:50:52.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:51 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:52.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:51 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:50:53.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:52 smithi016 ceph-mon[31259]: pgmap v189: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 2.81k op/s 2024-04-21T08:50:53.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:52 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:50:53.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:52 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:53.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:52 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:53.205 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:52 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:50:53.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:52 smithi077 ceph-mon[31615]: pgmap v189: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 2.81k op/s 2024-04-21T08:50:53.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:52 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:50:53.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:52 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:53.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:52 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:53.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:52 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:50:53.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:52 smithi007 ceph-mon[28477]: pgmap v189: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 2.81k op/s 2024-04-21T08:50:53.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:52 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:50:53.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:52 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:53.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:52 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:53.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:52 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:50:55.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:54 smithi016 ceph-mon[31259]: pgmap v190: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 2.08k op/s 2024-04-21T08:50:55.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:54 smithi077 ceph-mon[31615]: pgmap v190: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 2.08k op/s 2024-04-21T08:50:55.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:54 smithi007 ceph-mon[28477]: pgmap v190: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 2.08k op/s 2024-04-21T08:50:57.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:56 smithi016 ceph-mon[31259]: pgmap v191: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 2.37k op/s 2024-04-21T08:50:57.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:56 smithi077 ceph-mon[31615]: pgmap v191: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 2.37k op/s 2024-04-21T08:50:57.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:56 smithi007 ceph-mon[28477]: pgmap v191: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 2.37k op/s 2024-04-21T08:50:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.606+0000 7f6c76775640 1 -- 172.21.15.7:0/595005755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c701010a0 msgr2=0x7f6c70101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.606+0000 7f6c76775640 1 --2- 172.21.15.7:0/595005755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c701010a0 0x7f6c70101480 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009900 tx=0x7f6c60030040 comp rx=0 tx=0).stop 2024-04-21T08:50:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.606+0000 7f6c76775640 1 -- 172.21.15.7:0/595005755 shutdown_connections 2024-04-21T08:50:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.606+0000 7f6c76775640 1 --2- 172.21.15.7:0/595005755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c70106170 0x7f6c7010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.606+0000 7f6c76775640 1 --2- 172.21.15.7:0/595005755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c70101a50 0x7f6c70105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.606+0000 7f6c76775640 1 --2- 172.21.15.7:0/595005755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c701010a0 0x7f6c70101480 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.606+0000 7f6c76775640 1 -- 172.21.15.7:0/595005755 >> 172.21.15.7:0/595005755 conn(0x7f6c70077f70 msgr2=0x7f6c70078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.606+0000 7f6c76775640 1 -- 172.21.15.7:0/595005755 shutdown_connections 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.606+0000 7f6c76775640 1 -- 172.21.15.7:0/595005755 wait complete. 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.606+0000 7f6c76775640 1 Processor -- start 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c76775640 1 -- start start 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c76775640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c701010a0 0x7f6c70196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c76775640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c70106170 0x7f6c70196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c76775640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c7019adf0 0x7f6c7019b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c76775640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7010ffd0 con 0x7f6c7019adf0 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c76775640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7010fe50 con 0x7f6c701010a0 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c76775640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6c70110150 con 0x7f6c70106170 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c74d12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c7019adf0 0x7f6c7019b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c74d12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c7019adf0 0x7f6c7019b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55088/0 (socket says 172.21.15.7:55088) 2024-04-21T08:50:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c6ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c701010a0 0x7f6c70196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c74d12640 1 -- 172.21.15.7:0/3133326145 learned_addr learned my addr 172.21.15.7:0/3133326145 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c6f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c70106170 0x7f6c70196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c74d12640 1 -- 172.21.15.7:0/3133326145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c70106170 msgr2=0x7f6c70196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c74d12640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c70106170 0x7f6c70196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c74d12640 1 -- 172.21.15.7:0/3133326145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c701010a0 msgr2=0x7f6c70196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c74d12640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c701010a0 0x7f6c70196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c74d12640 1 -- 172.21.15.7:0/3133326145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7019b8e0 con 0x7f6c7019adf0 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.607+0000 7f6c6ffff640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c701010a0 0x7f6c70196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.608+0000 7f6c74d12640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c7019adf0 0x7f6c7019b2a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800da90 tx=0x7f6c6800df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.608+0000 7f6c6d7fa640 1 -- 172.21.15.7:0/3133326145 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6801b020 con 0x7f6c7019adf0 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.608+0000 7f6c6d7fa640 1 -- 172.21.15.7:0/3133326145 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c6800bc30 con 0x7f6c7019adf0 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.608+0000 7f6c76775640 1 -- 172.21.15.7:0/3133326145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7019bbd0 con 0x7f6c7019adf0 2024-04-21T08:50:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.608+0000 7f6c6d7fa640 1 -- 172.21.15.7:0/3133326145 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68005950 con 0x7f6c7019adf0 2024-04-21T08:50:58.611 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.608+0000 7f6c76775640 1 -- 172.21.15.7:0/3133326145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c701a3370 con 0x7f6c7019adf0 2024-04-21T08:50:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.610+0000 7f6c6d7fa640 1 -- 172.21.15.7:0/3133326145 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f6c68004a40 con 0x7f6c7019adf0 2024-04-21T08:50:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c78ffd740 1 Processor -- start 2024-04-21T08:50:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c78ffd740 1 -- start start 2024-04-21T08:50:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c6d7fa640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6c580a3d10 0x7f6c580a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b438d0 0x560524b42620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b42b60 0x560524b42fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 0x560524b4e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x560524aa78d0 con 0x560524b438d0 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560524a44e80 con 0x560524b42b60 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560524aa3890 con 0x560524b4c0e0 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c6f7fe640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6c580a3d10 0x7f6c580a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c6d7fa640 1 -- 172.21.15.7:0/3133326145 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f6c680d09c0 con 0x7f6c7019adf0 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c6f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b42b60 0x560524b42fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c6ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b438d0 0x560524b42620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.611+0000 7f6c74d12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 0x560524b4e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c6f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b42b60 0x560524b42fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:44064/0 (socket says 172.21.15.7:44064) 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c6f7fe640 1 -- 172.21.15.7:0/212349607 learned_addr learned my addr 172.21.15.7:0/212349607 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:58.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c74d12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 0x560524b4e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:35566/0 (socket says 172.21.15.7:35566) 2024-04-21T08:50:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c6ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b438d0 0x560524b42620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55090/0 (socket says 172.21.15.7:55090) 2024-04-21T08:50:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c74d12640 1 -- 172.21.15.7:0/212349607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b42b60 msgr2=0x560524b42fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c74d12640 1 --2- 172.21.15.7:0/212349607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b42b60 0x560524b42fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c74d12640 1 -- 172.21.15.7:0/212349607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b438d0 msgr2=0x560524b42620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c74d12640 1 --2- 172.21.15.7:0/212349607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b438d0 0x560524b42620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.618 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c74d12640 1 -- 172.21.15.7:0/212349607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560524a66320 con 0x560524b4c0e0 2024-04-21T08:50:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c6f7fe640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6c580a3d10 0x7f6c580a61d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6c64013570 tx=0x7f6c640098b0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c74d12640 1 --2- 172.21.15.7:0/212349607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 0x560524b4e4d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6c680cdd20 tx=0x7f6c68098260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=431a816775e2a9d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c567fc640 1 -- 172.21.15.7:0/212349607 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68098040 con 0x560524b4c0e0 2024-04-21T08:50:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c567fc640 1 -- 172.21.15.7:0/212349607 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c6809c600 con 0x560524b4c0e0 2024-04-21T08:50:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.612+0000 7f6c567fc640 1 -- 172.21.15.7:0/212349607 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c680a1b40 con 0x560524b4c0e0 2024-04-21T08:50:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.613+0000 7f6c78ffd740 1 -- 172.21.15.7:0/212349607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 msgr2=0x560524b4e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.613+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/212349607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 0x560524b4e4d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6c680cdd20 tx=0x7f6c68098260 comp rx=0 tx=0).stop 2024-04-21T08:50:58.619 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.613+0000 7f6c78ffd740 1 -- 172.21.15.7:0/212349607 shutdown_connections 2024-04-21T08:50:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.613+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/212349607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 0x560524b4e4d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.613+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/212349607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b42b60 0x560524b42fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.613+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/212349607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b438d0 0x560524b42620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.613+0000 7f6c78ffd740 1 -- 172.21.15.7:0/212349607 >> 172.21.15.7:0/212349607 conn(0x560524b09c90 msgr2=0x560524b3d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.613+0000 7f6c78ffd740 1 -- 172.21.15.7:0/212349607 shutdown_connections 2024-04-21T08:50:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.613+0000 7f6c78ffd740 1 -- 172.21.15.7:0/212349607 wait complete. 2024-04-21T08:50:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c78ffd740 1 -- start start 2024-04-21T08:50:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c78ffd740 1 Processor -- start 2024-04-21T08:50:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b4ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b48dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 0x560524b49320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x560524b4ee10 con 0x560524b4c0e0 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x560524a2feb0 con 0x560524b438d0 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560524b4ef90 con 0x560524b42b60 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c78ffd740 10 client.0 _ll_register_callbacks cb 0x560524a5cb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c6ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b4ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c6ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b4ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:35574/0 (socket says 172.21.15.7:35574) 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c74d12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 0x560524b49320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.614+0000 7f6c6ffff640 1 -- 172.21.15.7:0/1367408446 learned_addr learned my addr 172.21.15.7:0/1367408446 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c6f7fe640 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b48dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c6ffff640 1 -- 172.21.15.7:0/1367408446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 msgr2=0x560524b48dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c6ffff640 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b48dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c6ffff640 1 -- 172.21.15.7:0/1367408446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 msgr2=0x560524b49320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c6ffff640 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 0x560524b49320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c6ffff640 1 -- 172.21.15.7:0/1367408446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605247ba210 con 0x560524b42b60 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c6f7fe640 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b48dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c74d12640 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 0x560524b49320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c6ffff640 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b4ba30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009900 tx=0x7f6c60004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe3857b5d12f5383 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/1367408446 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c60048070 con 0x560524b42b60 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c78ffd740 1 -- 172.21.15.7:0/1367408446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c6809c600 con 0x560524b42b60 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.615+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/1367408446 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f6c60038460 con 0x560524b42b60 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/1367408446 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6c60038790 con 0x560524b42b60 2024-04-21T08:50:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c78ffd740 10 client.26437 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c78ffd740 1 -- 172.21.15.7:0/1367408446 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x560524886620 con 0x560524b42b60 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/1367408446 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c6809c600 con 0x560524b42b60 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/1367408446 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6c600039e0 con 0x560524b42b60 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c78ffd740 10 client.26437 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c78ffd740 10 client.26437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c78ffd740 20 client.26437 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560524b5c490 0x560524b5e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c78ffd740 4 client.26437 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c78ffd740 1 -- 172.21.15.7:0/1367408446 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560524abb880 con 0x560524b5c490 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.616+0000 7f6c6f7fe640 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560524b5c490 0x560524b5e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.617+0000 7f6c6f7fe640 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560524b5c490 0x560524b5e900 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ee2fcf3b50f72df server_cookie=aedb4535bfcf2283 in_seq=0 out_seq=0 2024-04-21T08:50:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.617+0000 7f6c4f7fe640 10 client.26437 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:58.626 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/1367408446 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560524abb880 con 0x560524b5c490 2024-04-21T08:50:58.626 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c4f7fe640 10 client.26437 handle_command_reply: tid=0 2024-04-21T08:50:58.626 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 1 client.26437 shutdown 2024-04-21T08:50:58.626 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 1 -- 172.21.15.7:0/1367408446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 msgr2=0x560524b4ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.626 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b4ba30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009900 tx=0x7f6c60004270 comp rx=0 tx=0).stop 2024-04-21T08:50:58.626 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 1 -- 172.21.15.7:0/1367408446 shutdown_connections 2024-04-21T08:50:58.626 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560524b5c490 0x560524b5e900 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.626 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 0x560524b49320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.626 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b48dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.627 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c4f7fe640 0 client.26437 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:58.627 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/1367408446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b4ba30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.627 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 1 -- 172.21.15.7:0/1367408446 >> 172.21.15.7:0/1367408446 conn(0x560524b09c90 msgr2=0x560524b3f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.627 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 1 -- 172.21.15.7:0/1367408446 shutdown_connections 2024-04-21T08:50:58.627 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 1 -- 172.21.15.7:0/1367408446 wait complete. 2024-04-21T08:50:58.627 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.618+0000 7f6c78ffd740 20 client.26437 trim_cache size 0 max 16384 2024-04-21T08:50:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.635+0000 7f6c78ffd740 1 Processor -- start 2024-04-21T08:50:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.635+0000 7f6c78ffd740 1 -- start start 2024-04-21T08:50:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b57d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b4b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 0x560524b4ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701ab230 con 0x560524b4c0e0 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701adab0 con 0x560524b438d0 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701ab8a0 con 0x560524b42b60 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c6ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b57d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c74d12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 0x560524b4ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c6ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b57d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:35578/0 (socket says 172.21.15.7:35578) 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c6ffff640 1 -- 172.21.15.7:0/70923007 learned_addr learned my addr 172.21.15.7:0/70923007 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c6f7fe640 1 --2- 172.21.15.7:0/70923007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b4b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c6ffff640 1 -- 172.21.15.7:0/70923007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 msgr2=0x560524b4b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c6ffff640 1 --2- 172.21.15.7:0/70923007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b4b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c6ffff640 1 -- 172.21.15.7:0/70923007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 msgr2=0x560524b4ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c6ffff640 1 --2- 172.21.15.7:0/70923007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 0x560524b4ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c6ffff640 1 -- 172.21.15.7:0/70923007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560524b41e30 con 0x560524b42b60 2024-04-21T08:50:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c74d12640 1 --2- 172.21.15.7:0/70923007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 0x560524b4ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:58.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.636+0000 7f6c6ffff640 1 --2- 172.21.15.7:0/70923007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b57d10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009250 tx=0x7f6c60030040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db4ae31ab5dfe83c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.637+0000 7f6c55ffb640 1 -- 172.21.15.7:0/70923007 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c60048070 con 0x560524b42b60 2024-04-21T08:50:58.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.637+0000 7f6c55ffb640 1 -- 172.21.15.7:0/70923007 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c60038de0 con 0x560524b42b60 2024-04-21T08:50:58.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.637+0000 7f6c55ffb640 1 -- 172.21.15.7:0/70923007 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6004db30 con 0x560524b42b60 2024-04-21T08:50:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.637+0000 7f6c78ffd740 1 -- 172.21.15.7:0/70923007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 msgr2=0x560524b57d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.637+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/70923007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b57d10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009250 tx=0x7f6c60030040 comp rx=0 tx=0).stop 2024-04-21T08:50:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.637+0000 7f6c78ffd740 1 -- 172.21.15.7:0/70923007 shutdown_connections 2024-04-21T08:50:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.637+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/70923007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b4c0e0 0x560524b4ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.637+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/70923007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b4b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.637+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/70923007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b42b60 0x560524b57d10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.637+0000 7f6c78ffd740 1 -- 172.21.15.7:0/70923007 >> 172.21.15.7:0/70923007 conn(0x560524b09c90 msgr2=0x560524b476d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.638+0000 7f6c78ffd740 1 -- 172.21.15.7:0/70923007 shutdown_connections 2024-04-21T08:50:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.638+0000 7f6c78ffd740 1 -- 172.21.15.7:0/70923007 wait complete. 2024-04-21T08:50:58.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.638+0000 7f6c78ffd740 1 -- start start 2024-04-21T08:50:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.638+0000 7f6c78ffd740 1 Processor -- start 2024-04-21T08:50:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.638+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b42b60 0x560524b54220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.638+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b54760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.638+0000 7f6c78ffd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 0x560524b45210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.638+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701abf70 con 0x560524b42b60 2024-04-21T08:50:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.638+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701ad3e0 con 0x560524b438d0 2024-04-21T08:50:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.638+0000 7f6c78ffd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701ac640 con 0x560524b4c0e0 2024-04-21T08:50:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.638+0000 7f6c78ffd740 10 client.0 _ll_register_callbacks cb 0x7f6c60009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c6ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b42b60 0x560524b54220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c6f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b54760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c74d12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 0x560524b45210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c6ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b42b60 0x560524b54220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55116/0 (socket says 172.21.15.7:55116) 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c6ffff640 1 -- 172.21.15.7:0/2242112414 learned_addr learned my addr 172.21.15.7:0/2242112414 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c74d12640 1 -- 172.21.15.7:0/2242112414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 msgr2=0x560524b54760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c74d12640 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b54760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c74d12640 1 -- 172.21.15.7:0/2242112414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b42b60 msgr2=0x560524b54220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c74d12640 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b42b60 0x560524b54220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c74d12640 1 -- 172.21.15.7:0/2242112414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560524b45a10 con 0x560524b4c0e0 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c6ffff640 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b42b60 0x560524b54220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c74d12640 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 0x560524b45210 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6c6809c300 tx=0x7f6c68005af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fae83ec95f4dc12 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/2242112414 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c680b9070 con 0x560524b4c0e0 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/2242112414 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f6c680bb2f0 con 0x560524b4c0e0 2024-04-21T08:50:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.639+0000 7f6c78ffd740 1 -- 172.21.15.7:0/2242112414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560524a473d0 con 0x560524b4c0e0 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/2242112414 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6c68082050 con 0x560524b4c0e0 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c78ffd740 10 client.26443 fetch_fsmap learned FSMap version 23 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c78ffd740 1 -- 172.21.15.7:0/2242112414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x560524bc2510 con 0x560524b4c0e0 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/2242112414 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560524a473d0 con 0x560524b4c0e0 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/2242112414 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6c68016070 con 0x560524b4c0e0 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c78ffd740 10 client.26443 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c78ffd740 10 client.26443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c78ffd740 20 client.26443 populate_metadata read hostname 'smithi007' 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560524bc27f0 0x560524bc4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c78ffd740 4 client.26443 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c78ffd740 1 -- 172.21.15.7:0/2242112414 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560524ab52f0 con 0x560524bc27f0 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.640+0000 7f6c6ffff640 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560524bc27f0 0x560524bc4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.641+0000 7f6c4f7fe640 10 client.26443 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:50:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:50:58.641+0000 7f6c6ffff640 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560524bc27f0 0x560524bc4be0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5aaed1012b7d928d server_cookie=5a9a2d4295bbc1e7 in_seq=0 out_seq=0 2024-04-21T08:50:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:50:58 smithi016 ceph-mon[31259]: pgmap v192: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 274 KiB/s wr, 1.83k op/s 2024-04-21T08:50:59.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:50:58 smithi077 ceph-mon[31615]: pgmap v192: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 274 KiB/s wr, 1.83k op/s 2024-04-21T08:50:59.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:50:58 smithi007 ceph-mon[28477]: pgmap v192: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 274 KiB/s wr, 1.83k op/s 2024-04-21T08:51:01.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:00 smithi016 ceph-mon[31259]: pgmap v193: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 425 KiB/s wr, 2.16k op/s 2024-04-21T08:51:01.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:00 smithi077 ceph-mon[31615]: pgmap v193: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 425 KiB/s wr, 2.16k op/s 2024-04-21T08:51:01.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:00 smithi007 ceph-mon[28477]: pgmap v193: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 425 KiB/s wr, 2.16k op/s 2024-04-21T08:51:03.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:02 smithi016 ceph-mon[31259]: pgmap v194: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 818 KiB/s wr, 1.71k op/s 2024-04-21T08:51:03.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:02 smithi077 ceph-mon[31615]: pgmap v194: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 818 KiB/s wr, 1.71k op/s 2024-04-21T08:51:03.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:02 smithi007 ceph-mon[28477]: pgmap v194: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 818 KiB/s wr, 1.71k op/s 2024-04-21T08:51:05.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:04 smithi016 ceph-mon[31259]: pgmap v195: 129 pgs: 129 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 2.02k op/s 2024-04-21T08:51:05.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:04 smithi077 ceph-mon[31615]: pgmap v195: 129 pgs: 129 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 2.02k op/s 2024-04-21T08:51:05.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:04 smithi007 ceph-mon[28477]: pgmap v195: 129 pgs: 129 active+clean; 1.3 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 2.02k op/s 2024-04-21T08:51:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c4f7fe640 1 -- 172.21.15.7:0/2242112414 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560524ab52f0 con 0x560524bc27f0 2024-04-21T08:51:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c4f7fe640 10 client.26443 handle_command_reply: tid=0 2024-04-21T08:51:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 1 client.26443 shutdown 2024-04-21T08:51:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 1 -- 172.21.15.7:0/2242112414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 msgr2=0x560524b45210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 0x560524b45210 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6c6809c300 tx=0x7f6c68005af0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 1 -- 172.21.15.7:0/2242112414 shutdown_connections 2024-04-21T08:51:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x560524bc27f0 0x560524bc4be0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c4f7fe640 0 client.26443 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560524b4c0e0 0x560524b45210 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x560524b438d0 0x560524b54760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 1 --2- 172.21.15.7:0/2242112414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560524b42b60 0x560524b54220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 1 -- 172.21.15.7:0/2242112414 >> 172.21.15.7:0/2242112414 conn(0x560524b09c90 msgr2=0x560524b3c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 1 -- 172.21.15.7:0/2242112414 shutdown_connections 2024-04-21T08:51:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 1 -- 172.21.15.7:0/2242112414 wait complete. 2024-04-21T08:51:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.612+0000 7f6c78ffd740 20 client.26443 trim_cache size 0 max 16384 2024-04-21T08:51:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:51:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:51:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:51:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:51:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 -- 172.21.15.7:0/3133326145 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6c580a3d10 msgr2=0x7f6c580a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6c580a3d10 0x7f6c580a61d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6c64013570 tx=0x7f6c640098b0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 -- 172.21.15.7:0/3133326145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c7019adf0 msgr2=0x7f6c7019b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c7019adf0 0x7f6c7019b2a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800da90 tx=0x7f6c6800df50 comp rx=0 tx=0).stop 2024-04-21T08:51:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 -- 172.21.15.7:0/3133326145 shutdown_connections 2024-04-21T08:51:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c7019adf0 0x7f6c7019b2a0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6c580a3d10 0x7f6c580a61d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c70106170 0x7f6c70196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 --2- 172.21.15.7:0/3133326145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c701010a0 0x7f6c70196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 -- 172.21.15.7:0/3133326145 >> 172.21.15.7:0/3133326145 conn(0x7f6c70077f70 msgr2=0x7f6c700ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 -- 172.21.15.7:0/3133326145 shutdown_connections 2024-04-21T08:51:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:06.613+0000 7f6c76775640 1 -- 172.21.15.7:0/3133326145 wait complete. 2024-04-21T08:51:07.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:06 smithi016 ceph-mon[31259]: pgmap v196: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 2.57k op/s 2024-04-21T08:51:07.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:06 smithi077 ceph-mon[31615]: pgmap v196: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 2.57k op/s 2024-04-21T08:51:07.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:06 smithi007 ceph-mon[28477]: pgmap v196: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.7 MiB/s wr, 2.57k op/s 2024-04-21T08:51:09.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:08 smithi016 ceph-mon[31259]: pgmap v197: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 2.06k op/s 2024-04-21T08:51:09.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:08 smithi077 ceph-mon[31615]: pgmap v197: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 2.06k op/s 2024-04-21T08:51:09.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:08 smithi007 ceph-mon[28477]: pgmap v197: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 2.06k op/s 2024-04-21T08:51:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:10 smithi016 ceph-mon[31259]: pgmap v198: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.6 MiB/s wr, 2.25k op/s 2024-04-21T08:51:11.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:10 smithi077 ceph-mon[31615]: pgmap v198: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.6 MiB/s wr, 2.25k op/s 2024-04-21T08:51:11.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:10 smithi007 ceph-mon[28477]: pgmap v198: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.6 MiB/s wr, 2.25k op/s 2024-04-21T08:51:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:12 smithi016 ceph-mon[31259]: pgmap v199: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.2 MiB/s wr, 2.09k op/s 2024-04-21T08:51:13.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:12 smithi077 ceph-mon[31615]: pgmap v199: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.2 MiB/s wr, 2.09k op/s 2024-04-21T08:51:13.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:12 smithi007 ceph-mon[28477]: pgmap v199: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.2 MiB/s wr, 2.09k op/s 2024-04-21T08:51:14.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:13 smithi007 ceph-mon[28477]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:51:14.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:13 smithi016 ceph-mon[31259]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:51:14.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:13 smithi077 ceph-mon[31615]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:51:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:14 smithi016 ceph-mon[31259]: pgmap v201: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 1.46k op/s 2024-04-21T08:51:15.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:14 smithi077 ceph-mon[31615]: pgmap v201: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 1.46k op/s 2024-04-21T08:51:15.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:14 smithi007 ceph-mon[28477]: pgmap v201: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 6.2 MiB/s wr, 1.46k op/s 2024-04-21T08:51:16.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:15 smithi016 ceph-mon[31259]: pgmap v202: 129 pgs: 13 active+clean+snaptrim, 86 active+clean, 30 active+clean+snaptrim_wait; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 951 op/s 2024-04-21T08:51:16.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:15 smithi077 ceph-mon[31615]: pgmap v202: 129 pgs: 13 active+clean+snaptrim, 86 active+clean, 30 active+clean+snaptrim_wait; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 951 op/s 2024-04-21T08:51:16.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:15 smithi007 ceph-mon[28477]: pgmap v202: 129 pgs: 13 active+clean+snaptrim, 86 active+clean, 30 active+clean+snaptrim_wait; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 951 op/s 2024-04-21T08:51:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:18 smithi016 ceph-mon[31259]: pgmap v203: 129 pgs: 13 active+clean+snaptrim, 86 active+clean, 30 active+clean+snaptrim_wait; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 951 op/s 2024-04-21T08:51:18.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:18 smithi077 ceph-mon[31615]: pgmap v203: 129 pgs: 13 active+clean+snaptrim, 86 active+clean, 30 active+clean+snaptrim_wait; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 951 op/s 2024-04-21T08:51:18.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:18 smithi007 ceph-mon[28477]: pgmap v203: 129 pgs: 13 active+clean+snaptrim, 86 active+clean, 30 active+clean+snaptrim_wait; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.7 MiB/s wr, 951 op/s 2024-04-21T08:51:19.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.763+0000 7fe42a9f4640 1 -- 172.21.15.7:0/1509322178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe424100820 msgr2=0x7fe424100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.763+0000 7fe42a9f4640 1 --2- 172.21.15.7:0/1509322178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe424100820 0x7fe424100c00 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fe41801a910 tx=0x7fe41803cd20 comp rx=0 tx=0).stop 2024-04-21T08:51:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.769+0000 7fe42a9f4640 1 -- 172.21.15.7:0/1509322178 shutdown_connections 2024-04-21T08:51:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.769+0000 7fe42a9f4640 1 --2- 172.21.15.7:0/1509322178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe424112cb0 0x7fe4241150a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.769+0000 7fe42a9f4640 1 --2- 172.21.15.7:0/1509322178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101140 0x7fe424112770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.769+0000 7fe42a9f4640 1 --2- 172.21.15.7:0/1509322178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe424100820 0x7fe424100c00 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.769+0000 7fe42a9f4640 1 -- 172.21.15.7:0/1509322178 >> 172.21.15.7:0/1509322178 conn(0x7fe4240761a0 msgr2=0x7fe4240765b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.773+0000 7fe42a9f4640 1 -- 172.21.15.7:0/1509322178 shutdown_connections 2024-04-21T08:51:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.774+0000 7fe42a9f4640 1 -- 172.21.15.7:0/1509322178 wait complete. 2024-04-21T08:51:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.775+0000 7fe42a9f4640 1 Processor -- start 2024-04-21T08:51:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.779+0000 7fe42a9f4640 1 -- start start 2024-04-21T08:51:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.779+0000 7fe42a9f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe424100820 0x7fe424112320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.779+0000 7fe42a9f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101140 0x7fe42410e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.779+0000 7fe42a9f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe424112cb0 0x7fe4240739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.779+0000 7fe42a9f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe42410a820 con 0x7fe424100820 2024-04-21T08:51:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.779+0000 7fe42a9f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe42410a6a0 con 0x7fe424101140 2024-04-21T08:51:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.779+0000 7fe42a9f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe42410a9a0 con 0x7fe424112cb0 2024-04-21T08:51:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.780+0000 7fe428f91640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe424112cb0 0x7fe4240739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.780+0000 7fe4237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101140 0x7fe42410e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.780+0000 7fe4237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101140 0x7fe42410e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:49334/0 (socket says 172.21.15.7:49334) 2024-04-21T08:51:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.780+0000 7fe4237fe640 1 -- 172.21.15.7:0/2201868206 learned_addr learned my addr 172.21.15.7:0/2201868206 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.780+0000 7fe423fff640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe424100820 0x7fe424112320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.781+0000 7fe423fff640 1 -- 172.21.15.7:0/2201868206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe424112cb0 msgr2=0x7fe4240739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.781+0000 7fe423fff640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe424112cb0 0x7fe4240739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.781+0000 7fe423fff640 1 -- 172.21.15.7:0/2201868206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101140 msgr2=0x7fe42410e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.781+0000 7fe423fff640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101140 0x7fe42410e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.781+0000 7fe423fff640 1 -- 172.21.15.7:0/2201868206 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe424073fb0 con 0x7fe424100820 2024-04-21T08:51:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.782+0000 7fe423fff640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe424100820 0x7fe424112320 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fe41803d330 tx=0x7fe418045670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.782+0000 7fe4217fa640 1 -- 172.21.15.7:0/2201868206 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41804a040 con 0x7fe424100820 2024-04-21T08:51:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.782+0000 7fe4217fa640 1 -- 172.21.15.7:0/2201868206 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe418045c90 con 0x7fe424100820 2024-04-21T08:51:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.783+0000 7fe4217fa640 1 -- 172.21.15.7:0/2201868206 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41804ecc0 con 0x7fe424100820 2024-04-21T08:51:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.785+0000 7fe42a9f4640 1 -- 172.21.15.7:0/2201868206 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42410e9b0 con 0x7fe424100820 2024-04-21T08:51:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.785+0000 7fe42a9f4640 1 -- 172.21.15.7:0/2201868206 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe42410ee50 con 0x7fe424100820 2024-04-21T08:51:19.791 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.790+0000 7fe42d5dd740 1 Processor -- start 2024-04-21T08:51:19.791 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.790+0000 7fe42d5dd740 1 -- start start 2024-04-21T08:51:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.791+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f84868d0 0x55f2f8485620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.791+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f8485b60 0x55f2f8485fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.791+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f848f0e0 0x55f2f84914d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.791+0000 7fe42d5dd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f2f83ea8d0 con 0x55f2f84868d0 2024-04-21T08:51:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.791+0000 7fe42d5dd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f2f8387e80 con 0x55f2f8485b60 2024-04-21T08:51:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.791+0000 7fe42d5dd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f2f83e6890 con 0x55f2f848f0e0 2024-04-21T08:51:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.790+0000 7fe4217fa640 1 -- 172.21.15.7:0/2201868206 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fe41804e3f0 con 0x7fe424100820 2024-04-21T08:51:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.791+0000 7fe423fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f8485b60 0x55f2f8485fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.793 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.791+0000 7fe423fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f8485b60 0x55f2f8485fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:49344/0 (socket says 172.21.15.7:49344) 2024-04-21T08:51:19.793 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.791+0000 7fe423fff640 1 -- 172.21.15.7:0/174043671 learned_addr learned my addr 172.21.15.7:0/174043671 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:19.793 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.792+0000 7fe423fff640 1 -- 172.21.15.7:0/174043671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f848f0e0 msgr2=0x55f2f84914d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:51:19.793 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.792+0000 7fe423fff640 1 --2- 172.21.15.7:0/174043671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f848f0e0 0x55f2f84914d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.793 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.792+0000 7fe423fff640 1 -- 172.21.15.7:0/174043671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f84868d0 msgr2=0x55f2f8485620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.793 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.792+0000 7fe423fff640 1 --2- 172.21.15.7:0/174043671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f84868d0 0x55f2f8485620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.793 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.792+0000 7fe423fff640 1 -- 172.21.15.7:0/174043671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2f83a9320 con 0x55f2f8485b60 2024-04-21T08:51:19.793 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.792+0000 7fe423fff640 1 --2- 172.21.15.7:0/174043671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f8485b60 0x55f2f8485fe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe4180ffb80 tx=0x7fe418108b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=924497ffe19e5194 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.794 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.794+0000 7fe40a7fc640 1 -- 172.21.15.7:0/174043671 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418111070 con 0x55f2f8485b60 2024-04-21T08:51:19.794 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.794+0000 7fe40a7fc640 1 -- 172.21.15.7:0/174043671 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4181040d0 con 0x55f2f8485b60 2024-04-21T08:51:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.794+0000 7fe4217fa640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe40c0a3e00 0x7fe40c0a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.794+0000 7fe4217fa640 1 -- 172.21.15.7:0/2201868206 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7fe4181028e0 con 0x7fe424100820 2024-04-21T08:51:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.795+0000 7fe42d5dd740 1 -- 172.21.15.7:0/174043671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f8485b60 msgr2=0x55f2f8485fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.795+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/174043671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f8485b60 0x55f2f8485fe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe4180ffb80 tx=0x7fe418108b20 comp rx=0 tx=0).stop 2024-04-21T08:51:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.795+0000 7fe428f91640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe40c0a3e00 0x7fe40c0a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.801+0000 7fe42d5dd740 1 -- 172.21.15.7:0/174043671 shutdown_connections 2024-04-21T08:51:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.801+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/174043671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f848f0e0 0x55f2f84914d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.801+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/174043671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f8485b60 0x55f2f8485fe0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.801+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/174043671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f84868d0 0x55f2f8485620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.801+0000 7fe42d5dd740 1 -- 172.21.15.7:0/174043671 >> 172.21.15.7:0/174043671 conn(0x55f2f844cc90 msgr2=0x55f2f8480850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.801+0000 7fe428f91640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe40c0a3e00 0x7fe40c0a62c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe41c005fd0 tx=0x7fe41c009b10 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.802+0000 7fe42d5dd740 1 -- 172.21.15.7:0/174043671 shutdown_connections 2024-04-21T08:51:19.804 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.803+0000 7fe42d5dd740 1 -- 172.21.15.7:0/174043671 wait complete. 2024-04-21T08:51:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.804+0000 7fe42d5dd740 1 -- start start 2024-04-21T08:51:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.804+0000 7fe42d5dd740 1 Processor -- start 2024-04-21T08:51:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.805+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f8485b60 0x55f2f848ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.805+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f84868d0 0x55f2f848bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.805+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f848f0e0 0x55f2f848c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.805+0000 7fe42d5dd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f2f8491bd0 con 0x55f2f848f0e0 2024-04-21T08:51:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.805+0000 7fe42d5dd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f2f8372eb0 con 0x55f2f84868d0 2024-04-21T08:51:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.805+0000 7fe42d5dd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f2f8491d50 con 0x55f2f8485b60 2024-04-21T08:51:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.805+0000 7fe42d5dd740 10 client.0 _ll_register_callbacks cb 0x55f2f839fb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.806+0000 7fe428f91640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f848f0e0 0x55f2f848c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.806+0000 7fe428f91640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f848f0e0 0x55f2f848c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51370/0 (socket says 172.21.15.7:51370) 2024-04-21T08:51:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.806+0000 7fe428f91640 1 -- 172.21.15.7:0/743793943 learned_addr learned my addr 172.21.15.7:0/743793943 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.806+0000 7fe428f91640 1 -- 172.21.15.7:0/743793943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f8485b60 msgr2=0x55f2f848ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.806+0000 7fe428f91640 1 --2- 172.21.15.7:0/743793943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f8485b60 0x55f2f848ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.806+0000 7fe428f91640 1 -- 172.21.15.7:0/743793943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f84868d0 msgr2=0x55f2f848bdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.806+0000 7fe428f91640 1 --2- 172.21.15.7:0/743793943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f84868d0 0x55f2f848bdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.806+0000 7fe428f91640 1 -- 172.21.15.7:0/743793943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2f80fd210 con 0x55f2f848f0e0 2024-04-21T08:51:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.808+0000 7fe428f91640 1 --2- 172.21.15.7:0/743793943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f848f0e0 0x55f2f848c330 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fe41c01cb90 tx=0x7fe41c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782022972aff49c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.808+0000 7fe4037fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:51:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.810+0000 7fe4037fe640 1 -- 172.21.15.7:0/743793943 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c00fb40 con 0x55f2f848f0e0 2024-04-21T08:51:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.810+0000 7fe42d5dd740 1 -- 172.21.15.7:0/743793943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4181040d0 con 0x55f2f848f0e0 2024-04-21T08:51:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.810+0000 7fe4037fe640 1 -- 172.21.15.7:0/743793943 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7fe41c00fce0 con 0x55f2f848f0e0 2024-04-21T08:51:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.814+0000 7fe4037fe640 1 -- 172.21.15.7:0/743793943 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe41c028480 con 0x55f2f848f0e0 2024-04-21T08:51:19.817 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.816+0000 7fe42d5dd740 10 client.16935 fetch_fsmap learned FSMap version 23 2024-04-21T08:51:19.817 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.816+0000 7fe42d5dd740 1 -- 172.21.15.7:0/743793943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f2f81c9620 con 0x55f2f848f0e0 2024-04-21T08:51:19.818 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.817+0000 7fe4037fe640 1 -- 172.21.15.7:0/743793943 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4181040d0 con 0x55f2f848f0e0 2024-04-21T08:51:19.818 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.817+0000 7fe4037fe640 1 -- 172.21.15.7:0/743793943 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fe41c01d410 con 0x55f2f848f0e0 2024-04-21T08:51:19.818 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.817+0000 7fe42d5dd740 10 client.16935 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:51:19.818 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.817+0000 7fe42d5dd740 10 client.16935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:19.818 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.818+0000 7fe42d5dd740 20 client.16935 populate_metadata read hostname 'smithi007' 2024-04-21T08:51:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.818+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/743793943 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f2f849f350 0x55f2f84a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.818+0000 7fe42d5dd740 4 client.16935 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.818+0000 7fe42d5dd740 1 -- 172.21.15.7:0/743793943 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2f83fe880 con 0x55f2f849f350 2024-04-21T08:51:19.820 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.819+0000 7fe423fff640 1 --2- 172.21.15.7:0/743793943 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f2f849f350 0x55f2f84a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.830+0000 7fe423fff640 1 --2- 172.21.15.7:0/743793943 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f2f849f350 0x55f2f84a17f0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb4d0a8b10acfa7e server_cookie=e3e3602fb9f5d81f in_seq=0 out_seq=0 2024-04-21T08:51:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.830+0000 7fe4037fe640 10 client.16935 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.836+0000 7fe4037fe640 1 -- 172.21.15.7:0/743793943 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f2f83fe880 con 0x55f2f849f350 2024-04-21T08:51:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.836+0000 7fe4037fe640 10 client.16935 handle_command_reply: tid=0 2024-04-21T08:51:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.836+0000 7fe42d5dd740 1 client.16935 shutdown 2024-04-21T08:51:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.840+0000 7fe42d5dd740 1 -- 172.21.15.7:0/743793943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f848f0e0 msgr2=0x55f2f848c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.840+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/743793943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f848f0e0 0x55f2f848c330 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fe41c01cb90 tx=0x7fe41c009500 comp rx=0 tx=0).stop 2024-04-21T08:51:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.841+0000 7fe42d5dd740 1 -- 172.21.15.7:0/743793943 shutdown_connections 2024-04-21T08:51:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.841+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/743793943 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f2f849f350 0x55f2f84a17f0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.841+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/743793943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f848f0e0 0x55f2f848c330 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.841+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/743793943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f84868d0 0x55f2f848bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.841+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/743793943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f8485b60 0x55f2f848ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.841+0000 7fe42d5dd740 1 -- 172.21.15.7:0/743793943 >> 172.21.15.7:0/743793943 conn(0x55f2f844cc90 msgr2=0x55f2f84825a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.841+0000 7fe4037fe640 0 client.16935 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.845+0000 7fe42d5dd740 1 -- 172.21.15.7:0/743793943 shutdown_connections 2024-04-21T08:51:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.846+0000 7fe42d5dd740 1 -- 172.21.15.7:0/743793943 wait complete. 2024-04-21T08:51:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.847+0000 7fe42d5dd740 20 client.16935 trim_cache size 0 max 16384 2024-04-21T08:51:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.879+0000 7fe42d5dd740 1 Processor -- start 2024-04-21T08:51:19.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.879+0000 7fe42d5dd740 1 -- start start 2024-04-21T08:51:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.880+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f8485b60 0x55f2f849aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.880+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f84868d0 0x55f2f848e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.880+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 0x55f2f848ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.880+0000 7fe42d5dd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b3b80 con 0x55f2f84868d0 2024-04-21T08:51:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.880+0000 7fe42d5dd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b63e0 con 0x55f2f848f0e0 2024-04-21T08:51:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.880+0000 7fe428f91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 0x55f2f848ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.880+0000 7fe428f91640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 0x55f2f848ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:49362/0 (socket says 172.21.15.7:49362) 2024-04-21T08:51:19.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.880+0000 7fe428f91640 1 -- 172.21.15.7:0/791894634 learned_addr learned my addr 172.21.15.7:0/791894634 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.880+0000 7fe42d5dd740 1 -- 172.21.15.7:0/791894634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b41d0 con 0x55f2f8485b60 2024-04-21T08:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.881+0000 7fe4237fe640 1 --2- 172.21.15.7:0/791894634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f84868d0 0x55f2f848e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.881+0000 7fe423fff640 1 --2- 172.21.15.7:0/791894634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f8485b60 0x55f2f849aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.882+0000 7fe428f91640 1 -- 172.21.15.7:0/791894634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f8485b60 msgr2=0x55f2f849aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.882 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.882+0000 7fe428f91640 1 --2- 172.21.15.7:0/791894634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f8485b60 0x55f2f849aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.882+0000 7fe428f91640 1 -- 172.21.15.7:0/791894634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f84868d0 msgr2=0x55f2f848e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.882+0000 7fe428f91640 1 --2- 172.21.15.7:0/791894634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f84868d0 0x55f2f848e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.883 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.882+0000 7fe428f91640 1 -- 172.21.15.7:0/791894634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2f849dad0 con 0x55f2f848f0e0 2024-04-21T08:51:19.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.885+0000 7fe428f91640 1 --2- 172.21.15.7:0/791894634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 0x55f2f848ea40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe41c016040 tx=0x7fe41c01c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fc4beb95c78705c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.887+0000 7fe409ffb640 1 -- 172.21.15.7:0/791894634 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c01da50 con 0x55f2f848f0e0 2024-04-21T08:51:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.887+0000 7fe409ffb640 1 -- 172.21.15.7:0/791894634 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe41c003e20 con 0x55f2f848f0e0 2024-04-21T08:51:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.888+0000 7fe42d5dd740 1 -- 172.21.15.7:0/791894634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 msgr2=0x55f2f848ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.888+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/791894634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 0x55f2f848ea40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe41c016040 tx=0x7fe41c01c740 comp rx=0 tx=0).stop 2024-04-21T08:51:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.890+0000 7fe42d5dd740 1 -- 172.21.15.7:0/791894634 shutdown_connections 2024-04-21T08:51:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.890+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/791894634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 0x55f2f848ea40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.890+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/791894634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f84868d0 0x55f2f848e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.890+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/791894634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f8485b60 0x55f2f849aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.890+0000 7fe42d5dd740 1 -- 172.21.15.7:0/791894634 >> 172.21.15.7:0/791894634 conn(0x55f2f844cc90 msgr2=0x55f2f8487060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.891+0000 7fe42d5dd740 1 -- 172.21.15.7:0/791894634 shutdown_connections 2024-04-21T08:51:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.891+0000 7fe42d5dd740 1 -- 172.21.15.7:0/791894634 wait complete. 2024-04-21T08:51:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.891+0000 7fe42d5dd740 1 -- start start 2024-04-21T08:51:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.891+0000 7fe42d5dd740 1 Processor -- start 2024-04-21T08:51:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.892+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f8485b60 0x55f2f8496ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.892+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f84868d0 0x55f2f8497530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.892+0000 7fe42d5dd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 0x55f2f8488040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.892+0000 7fe42d5dd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b48a0 con 0x55f2f8485b60 2024-04-21T08:51:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.892+0000 7fe42d5dd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b5d10 con 0x55f2f848f0e0 2024-04-21T08:51:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.892+0000 7fe42d5dd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241b4f70 con 0x55f2f84868d0 2024-04-21T08:51:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.892+0000 7fe42d5dd740 10 client.0 _ll_register_callbacks cb 0x7fe41c009ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.892+0000 7fe4237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f84868d0 0x55f2f8497530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.892+0000 7fe423fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f8485b60 0x55f2f8496ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.892+0000 7fe423fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f8485b60 0x55f2f8496ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51390/0 (socket says 172.21.15.7:51390) 2024-04-21T08:51:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.892+0000 7fe423fff640 1 -- 172.21.15.7:0/3198920428 learned_addr learned my addr 172.21.15.7:0/3198920428 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.893+0000 7fe428f91640 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 0x55f2f8488040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.893+0000 7fe423fff640 1 -- 172.21.15.7:0/3198920428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f84868d0 msgr2=0x55f2f8497530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.893+0000 7fe423fff640 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f84868d0 0x55f2f8497530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.893+0000 7fe423fff640 1 -- 172.21.15.7:0/3198920428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 msgr2=0x55f2f8488040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.896 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.893+0000 7fe423fff640 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 0x55f2f8488040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.893+0000 7fe423fff640 1 -- 172.21.15.7:0/3198920428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2f8488580 con 0x55f2f8485b60 2024-04-21T08:51:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.894+0000 7fe423fff640 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f8485b60 0x55f2f8496ff0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fe41801a910 tx=0x7fe418005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8ae7c44ac25345 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.894+0000 7fe4037fe640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:51:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.894+0000 7fe4037fe640 1 -- 172.21.15.7:0/3198920428 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418111040 con 0x55f2f8485b60 2024-04-21T08:51:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.894+0000 7fe42d5dd740 1 -- 172.21.15.7:0/3198920428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe41c003e20 con 0x55f2f8485b60 2024-04-21T08:51:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.894+0000 7fe4037fe640 1 -- 172.21.15.7:0/3198920428 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7fe4180ce7a0 con 0x55f2f8485b60 2024-04-21T08:51:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.894+0000 7fe4037fe640 1 -- 172.21.15.7:0/3198920428 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe41804ee60 con 0x55f2f8485b60 2024-04-21T08:51:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.895+0000 7fe42d5dd740 10 client.16941 fetch_fsmap learned FSMap version 23 2024-04-21T08:51:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.895+0000 7fe42d5dd740 1 -- 172.21.15.7:0/3198920428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f2f8505140 con 0x55f2f8485b60 2024-04-21T08:51:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.896+0000 7fe4037fe640 1 -- 172.21.15.7:0/3198920428 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe41c003e20 con 0x55f2f8485b60 2024-04-21T08:51:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.896+0000 7fe4037fe640 1 -- 172.21.15.7:0/3198920428 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fe418102040 con 0x55f2f8485b60 2024-04-21T08:51:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.896+0000 7fe42d5dd740 10 client.16941 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:51:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.896+0000 7fe42d5dd740 10 client.16941 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.896+0000 7fe42d5dd740 20 client.16941 populate_metadata read hostname 'smithi007' 2024-04-21T08:51:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.896+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f2f8505880 0x55f2f8507ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.896+0000 7fe42d5dd740 4 client.16941 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:51:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.896+0000 7fe42d5dd740 1 -- 172.21.15.7:0/3198920428 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f2f838a3d0 con 0x55f2f8505880 2024-04-21T08:51:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.901+0000 7fe4237fe640 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f2f8505880 0x55f2f8507ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.911+0000 7fe4237fe640 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f2f8505880 0x55f2f8507ca0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61fe3a50eb55b987 server_cookie=c274b3df6dca7369 in_seq=0 out_seq=0 2024-04-21T08:51:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:19.911+0000 7fe4037fe640 10 client.16941 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:20.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:20 smithi016 ceph-mon[31259]: pgmap v204: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.3 MiB/s wr, 1.00k op/s 2024-04-21T08:51:20.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:20 smithi077 ceph-mon[31615]: pgmap v204: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.3 MiB/s wr, 1.00k op/s 2024-04-21T08:51:20.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:20 smithi007 ceph-mon[28477]: pgmap v204: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.3 MiB/s wr, 1.00k op/s 2024-04-21T08:51:21.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:51:21.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:51:21.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.614+0000 7fd216b79640 1 -- 172.21.15.7:0/3235094173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20809ace0 msgr2=0x7fd208004730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.615+0000 7fd216b79640 1 --2- 172.21.15.7:0/3235094173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20809ace0 0x7fd208004730 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd210067dc0 tx=0x7fd210098000 comp rx=0 tx=0).stop 2024-04-21T08:51:21.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.618+0000 7fd216b79640 1 -- 172.21.15.7:0/3235094173 shutdown_connections 2024-04-21T08:51:21.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.618+0000 7fd216b79640 1 --2- 172.21.15.7:0/3235094173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2080986d0 0x7fd208098b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.618+0000 7fd216b79640 1 --2- 172.21.15.7:0/3235094173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd208004c70 0x7fd2080050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.618+0000 7fd216b79640 1 --2- 172.21.15.7:0/3235094173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd20809ace0 0x7fd208004730 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.618+0000 7fd216b79640 1 -- 172.21.15.7:0/3235094173 >> 172.21.15.7:0/3235094173 conn(0x7fd20800a730 msgr2=0x7fd20800ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:21.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.623+0000 7fd216b79640 1 -- 172.21.15.7:0/3235094173 shutdown_connections 2024-04-21T08:51:21.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.624+0000 7fd216b79640 1 -- 172.21.15.7:0/3235094173 wait complete. 2024-04-21T08:51:21.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.625+0000 7fd216b79640 1 Processor -- start 2024-04-21T08:51:21.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.629+0000 7fd216b79640 1 -- start start 2024-04-21T08:51:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.630+0000 7fd216b79640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd208004c70 0x7fd2080a9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.630+0000 7fd216b79640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2080986d0 0x7fd2080a3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.630+0000 7fd216b79640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd20809ace0 0x7fd2080a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.630+0000 7fd216b79640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd20809bd60 con 0x7fd208004c70 2024-04-21T08:51:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.630+0000 7fd216b79640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd20809bbe0 con 0x7fd2080986d0 2024-04-21T08:51:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.630+0000 7fd216b79640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd20809bee0 con 0x7fd20809ace0 2024-04-21T08:51:21.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.630+0000 7fd215376640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2080986d0 0x7fd2080a3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.630+0000 7fd215376640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2080986d0 0x7fd2080a3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:49396/0 (socket says 172.21.15.7:49396) 2024-04-21T08:51:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.630+0000 7fd215376640 1 -- 172.21.15.7:0/1308421039 learned_addr learned my addr 172.21.15.7:0/1308421039 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.630+0000 7fd215b77640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd208004c70 0x7fd2080a9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.631+0000 7fd215b77640 1 -- 172.21.15.7:0/1308421039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd20809ace0 msgr2=0x7fd2080a3950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:51:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.631+0000 7fd215b77640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd20809ace0 0x7fd2080a3950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.631+0000 7fd215b77640 1 -- 172.21.15.7:0/1308421039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2080986d0 msgr2=0x7fd2080a3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.631+0000 7fd215b77640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2080986d0 0x7fd2080a3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.631+0000 7fd215b77640 1 -- 172.21.15.7:0/1308421039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2080a4030 con 0x7fd208004c70 2024-04-21T08:51:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.631+0000 7fd215b77640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd208004c70 0x7fd2080a9350 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd21002db50 tx=0x7fd210098f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.633+0000 7fd206ffd640 1 -- 172.21.15.7:0/1308421039 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21006bcc0 con 0x7fd208004c70 2024-04-21T08:51:21.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.634+0000 7fd216b79640 1 -- 172.21.15.7:0/1308421039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2081588b0 con 0x7fd208004c70 2024-04-21T08:51:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.634+0000 7fd216b79640 1 -- 172.21.15.7:0/1308421039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd208158cb0 con 0x7fd208004c70 2024-04-21T08:51:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.634+0000 7fd206ffd640 1 -- 172.21.15.7:0/1308421039 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd210068570 con 0x7fd208004c70 2024-04-21T08:51:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.634+0000 7fd206ffd640 1 -- 172.21.15.7:0/1308421039 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21006be60 con 0x7fd208004c70 2024-04-21T08:51:21.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.638+0000 7fd21a9cf740 1 Processor -- start 2024-04-21T08:51:21.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.640+0000 7fd21a9cf740 1 -- start start 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.640+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2a710 0x55a708e2aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.640+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2b030 0x55a708e336d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.640+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 0x55a708e36000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.640+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a708d8f1e0 con 0x55a708e2a710 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.640+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a708d2c390 con 0x55a708e2b030 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.640+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a708d8b1a0 con 0x55a708e33c10 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.640+0000 7fd206ffd640 1 -- 172.21.15.7:0/1308421039 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd2100a19d0 con 0x7fd208004c70 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.640+0000 7fd215376640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 0x55a708e36000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.641+0000 7fd215376640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 0x55a708e36000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:56556/0 (socket says 172.21.15.7:56556) 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.641+0000 7fd215376640 1 -- 172.21.15.7:0/3523210748 learned_addr learned my addr 172.21.15.7:0/3523210748 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.641+0000 7fd215376640 1 -- 172.21.15.7:0/3523210748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2b030 msgr2=0x55a708e336d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:51:21.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.641+0000 7fd215376640 1 --2- 172.21.15.7:0/3523210748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2b030 0x55a708e336d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.641+0000 7fd215376640 1 -- 172.21.15.7:0/3523210748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2a710 msgr2=0x55a708e2aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.641+0000 7fd215376640 1 --2- 172.21.15.7:0/3523210748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2a710 0x55a708e2aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.641+0000 7fd215376640 1 -- 172.21.15.7:0/3523210748 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a708b6b7d0 con 0x55a708e33c10 2024-04-21T08:51:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.641+0000 7fd215376640 1 --2- 172.21.15.7:0/3523210748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 0x55a708e36000 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00bd40 tx=0x7fd20c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=757b98e534f96398 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.642+0000 7fd206ffd640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1fc0a3ef0 0x7fd1fc0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.642+0000 7fd1fbfff640 1 -- 172.21.15.7:0/3523210748 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c004230 con 0x55a708e33c10 2024-04-21T08:51:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.642+0000 7fd1fbfff640 1 -- 172.21.15.7:0/3523210748 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd20c0043d0 con 0x55a708e33c10 2024-04-21T08:51:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.643+0000 7fd206ffd640 1 -- 172.21.15.7:0/1308421039 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7fd210155400 con 0x7fd208004c70 2024-04-21T08:51:21.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.643+0000 7fd216378640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1fc0a3ef0 0x7fd1fc0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.643+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3523210748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 msgr2=0x55a708e36000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.643+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/3523210748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 0x55a708e36000 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00bd40 tx=0x7fd20c0107a0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.644+0000 7fd216378640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1fc0a3ef0 0x7fd1fc0a63b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd200006fd0 tx=0x7fd200006c40 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.647+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3523210748 shutdown_connections 2024-04-21T08:51:21.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.647+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/3523210748 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 0x55a708e36000 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.647+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/3523210748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2b030 0x55a708e336d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.647+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/3523210748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2a710 0x55a708e2aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.647+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3523210748 >> 172.21.15.7:0/3523210748 conn(0x55a708e24e40 msgr2=0x55a708e25220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:21.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.648+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3523210748 shutdown_connections 2024-04-21T08:51:21.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.652+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3523210748 wait complete. 2024-04-21T08:51:21.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.652+0000 7fd21a9cf740 1 -- start start 2024-04-21T08:51:21.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.653+0000 7fd21a9cf740 1 Processor -- start 2024-04-21T08:51:21.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.653+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2a710 0x55a708e30b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.653+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2b030 0x55a708e31080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.653+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 0x55a708e315c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.653+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a708e36a50 con 0x55a708e2b030 2024-04-21T08:51:21.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.653+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a708d482f0 con 0x55a708e2a710 2024-04-21T08:51:21.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.653+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a708e36bd0 con 0x55a708e33c10 2024-04-21T08:51:21.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.653+0000 7fd21a9cf740 10 client.0 _ll_register_callbacks cb 0x55a708d55580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:21.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.654+0000 7fd215376640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2b030 0x55a708e31080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.654+0000 7fd215376640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2b030 0x55a708e31080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51414/0 (socket says 172.21.15.7:51414) 2024-04-21T08:51:21.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.654+0000 7fd215376640 1 -- 172.21.15.7:0/4219637471 learned_addr learned my addr 172.21.15.7:0/4219637471 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:21.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.655+0000 7fd216378640 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 0x55a708e315c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.658+0000 7fd215376640 1 -- 172.21.15.7:0/4219637471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 msgr2=0x55a708e315c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:21.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.658+0000 7fd215376640 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 0x55a708e315c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.658+0000 7fd215376640 1 -- 172.21.15.7:0/4219637471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2a710 msgr2=0x55a708e30b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:21.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.658+0000 7fd215376640 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2a710 0x55a708e30b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.658+0000 7fd215376640 1 -- 172.21.15.7:0/4219637471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a708aa13f0 con 0x55a708e2b030 2024-04-21T08:51:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.662+0000 7fd215376640 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2b030 0x55a708e31080 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fd20c003ee0 tx=0x7fd20c004150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f00103b3e54ee37 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.662+0000 7fd1f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:51:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.663+0000 7fd1f97fa640 1 -- 172.21.15.7:0/4219637471 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c02b060 con 0x55a708e2b030 2024-04-21T08:51:21.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.664+0000 7fd21a9cf740 1 -- 172.21.15.7:0/4219637471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd20c0043d0 con 0x55a708e2b030 2024-04-21T08:51:21.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.664+0000 7fd1f97fa640 1 -- 172.21.15.7:0/4219637471 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7fd20c011c90 con 0x55a708e2b030 2024-04-21T08:51:21.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.669+0000 7fd1f97fa640 1 -- 172.21.15.7:0/4219637471 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd20c032050 con 0x55a708e2b030 2024-04-21T08:51:21.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.670+0000 7fd21a9cf740 10 client.16947 fetch_fsmap learned FSMap version 23 2024-04-21T08:51:21.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.670+0000 7fd21a9cf740 1 -- 172.21.15.7:0/4219637471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a708a610f0 con 0x55a708e2b030 2024-04-21T08:51:21.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.673+0000 7fd1f97fa640 1 -- 172.21.15.7:0/4219637471 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd20c0043d0 con 0x55a708e2b030 2024-04-21T08:51:21.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.673+0000 7fd1f97fa640 1 -- 172.21.15.7:0/4219637471 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fd20c01b6d0 con 0x55a708e2b030 2024-04-21T08:51:21.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.673+0000 7fd21a9cf740 10 client.16947 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.673+0000 7fd21a9cf740 10 client.16947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.673+0000 7fd21a9cf740 20 client.16947 populate_metadata read hostname 'smithi007' 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.673+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a708e44040 0x55a708e464a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.673+0000 7fd21a9cf740 4 client.16947 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.673+0000 7fd21a9cf740 1 -- 172.21.15.7:0/4219637471 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a708d9cc00 con 0x55a708e44040 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.674+0000 7fd215b77640 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a708e44040 0x55a708e464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.679+0000 7fd1f97fa640 10 client.16947 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:21.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:21.680+0000 7fd215b77640 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a708e44040 0x55a708e464a0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=646fd61fc2e7b2df server_cookie=d60a47c1a1b42e79 in_seq=0 out_seq=0 2024-04-21T08:51:22.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:22 smithi016 ceph-mon[31259]: pgmap v205: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.2 MiB/s wr, 964 op/s 2024-04-21T08:51:22.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:22 smithi077 ceph-mon[31615]: pgmap v205: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.2 MiB/s wr, 964 op/s 2024-04-21T08:51:22.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:22 smithi007 ceph-mon[28477]: pgmap v205: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.2 MiB/s wr, 964 op/s 2024-04-21T08:51:24.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:24 smithi016 ceph-mon[31259]: pgmap v206: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 960 op/s 2024-04-21T08:51:24.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:24 smithi077 ceph-mon[31615]: pgmap v206: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 960 op/s 2024-04-21T08:51:24.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:24 smithi007 ceph-mon[28477]: pgmap v206: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.7 MiB/s wr, 960 op/s 2024-04-21T08:51:26.506 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe4037fe640 1 -- 172.21.15.7:0/3198920428 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f2f838a3d0 con 0x55f2f8505880 2024-04-21T08:51:26.506 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe4037fe640 10 client.16941 handle_command_reply: tid=0 2024-04-21T08:51:26.507 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe42d5dd740 1 client.16941 shutdown 2024-04-21T08:51:26.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fd1f97fa640 1 -- 172.21.15.7:0/4219637471 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a708d9cc00 con 0x55a708e44040 2024-04-21T08:51:26.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fd1f97fa640 10 client.16947 handle_command_reply: tid=0 2024-04-21T08:51:26.507 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe42d5dd740 1 -- 172.21.15.7:0/3198920428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f8485b60 msgr2=0x55f2f8496ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f8485b60 0x55f2f8496ff0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fe41801a910 tx=0x7fe418005fd0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe42d5dd740 1 -- 172.21.15.7:0/3198920428 shutdown_connections 2024-04-21T08:51:26.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55f2f8505880 0x55f2f8507ca0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f2f848f0e0 0x55f2f8488040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2f84868d0 0x55f2f8497530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe42d5dd740 1 --2- 172.21.15.7:0/3198920428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f2f8485b60 0x55f2f8496ff0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe42d5dd740 1 -- 172.21.15.7:0/3198920428 >> 172.21.15.7:0/3198920428 conn(0x55f2f844cc90 msgr2=0x55f2f847f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe4037fe640 0 client.16941 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fd21a9cf740 1 client.16947 shutdown 2024-04-21T08:51:26.509 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.506+0000 7fe42d5dd740 1 -- 172.21.15.7:0/3198920428 shutdown_connections 2024-04-21T08:51:26.509 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fe42d5dd740 1 -- 172.21.15.7:0/3198920428 wait complete. 2024-04-21T08:51:26.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd21a9cf740 1 -- 172.21.15.7:0/4219637471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2b030 msgr2=0x55a708e31080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2b030 0x55a708e31080 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fd20c003ee0 tx=0x7fd20c004150 comp rx=0 tx=0).stop 2024-04-21T08:51:26.510 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fe42d5dd740 20 client.16941 trim_cache size 0 max 16384 2024-04-21T08:51:26.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd21a9cf740 1 -- 172.21.15.7:0/4219637471 shutdown_connections 2024-04-21T08:51:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a708e44040 0x55a708e464a0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e33c10 0x55a708e315c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2b030 0x55a708e31080 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/4219637471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2a710 0x55a708e30b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd21a9cf740 1 -- 172.21.15.7:0/4219637471 >> 172.21.15.7:0/4219637471 conn(0x55a708e24e40 msgr2=0x55a708e24120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd1f97fa640 0 client.16947 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd21a9cf740 1 -- 172.21.15.7:0/4219637471 shutdown_connections 2024-04-21T08:51:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd21a9cf740 1 -- 172.21.15.7:0/4219637471 wait complete. 2024-04-21T08:51:26.511 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:51:26.511 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:51:26.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:51:26.512 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:51:26.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.507+0000 7fd21a9cf740 20 client.16947 trim_cache size 0 max 16384 2024-04-21T08:51:26.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.508+0000 7fe42a9f4640 1 -- 172.21.15.7:0/2201868206 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe40c0a3e00 msgr2=0x7fe40c0a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.508+0000 7fe42a9f4640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe40c0a3e00 0x7fe40c0a62c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe41c005fd0 tx=0x7fe41c009b10 comp rx=0 tx=0).stop 2024-04-21T08:51:26.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.508+0000 7fe42a9f4640 1 -- 172.21.15.7:0/2201868206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe424100820 msgr2=0x7fe424112320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.508+0000 7fe42a9f4640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe424100820 0x7fe424112320 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fe41803d330 tx=0x7fe418045670 comp rx=0 tx=0).stop 2024-04-21T08:51:26.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.513+0000 7fe42a9f4640 1 -- 172.21.15.7:0/2201868206 shutdown_connections 2024-04-21T08:51:26.513 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.513+0000 7fe42a9f4640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe40c0a3e00 0x7fe40c0a62c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.513+0000 7fe42a9f4640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe424112cb0 0x7fe4240739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.513+0000 7fe42a9f4640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe424101140 0x7fe42410e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.513+0000 7fe42a9f4640 1 --2- 172.21.15.7:0/2201868206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe424100820 0x7fe424112320 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.513+0000 7fe42a9f4640 1 -- 172.21.15.7:0/2201868206 >> 172.21.15.7:0/2201868206 conn(0x7fe4240761a0 msgr2=0x7fe424113270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.513+0000 7fe42a9f4640 1 -- 172.21.15.7:0/2201868206 shutdown_connections 2024-04-21T08:51:26.514 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:26.513+0000 7fe42a9f4640 1 -- 172.21.15.7:0/2201868206 wait complete. 2024-04-21T08:51:26.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.546+0000 7fd21a9cf740 1 Processor -- start 2024-04-21T08:51:26.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd21a9cf740 1 -- start start 2024-04-21T08:51:26.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2a710 0x55a708e3f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e2fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e33c10 0x55a708e313e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f400e360 con 0x55a708e33c10 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f4010bc0 con 0x55a708e2a710 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f400e9b0 con 0x55a708e2b030 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd215376640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e2fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd215376640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e2fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:33182/0 (socket says 172.21.15.7:33182) 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd215376640 1 -- 172.21.15.7:0/1667193781 learned_addr learned my addr 172.21.15.7:0/1667193781 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd215376640 1 -- 172.21.15.7:0/1667193781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2a710 msgr2=0x55a708e3f9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd215376640 1 --2- 172.21.15.7:0/1667193781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2a710 0x55a708e3f9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd215376640 1 -- 172.21.15.7:0/1667193781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e33c10 msgr2=0x55a708e313e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.547+0000 7fd215376640 1 --2- 172.21.15.7:0/1667193781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e33c10 0x55a708e313e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.548+0000 7fd215376640 1 -- 172.21.15.7:0/1667193781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a708e29a90 con 0x55a708e2b030 2024-04-21T08:51:26.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.550+0000 7fd215376640 1 --2- 172.21.15.7:0/1667193781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e2fca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00bc60 tx=0x7fd20c0119c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b244d0ccbee0561 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.551+0000 7fd1fb7fe640 1 -- 172.21.15.7:0/1667193781 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c01f030 con 0x55a708e2b030 2024-04-21T08:51:26.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.551+0000 7fd1fb7fe640 1 -- 172.21.15.7:0/1667193781 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd20c00b480 con 0x55a708e2b030 2024-04-21T08:51:26.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.551+0000 7fd1fb7fe640 1 -- 172.21.15.7:0/1667193781 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c02b070 con 0x55a708e2b030 2024-04-21T08:51:26.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.552+0000 7fd21a9cf740 1 -- 172.21.15.7:0/1667193781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 msgr2=0x55a708e2fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.552+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/1667193781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e2fca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00bc60 tx=0x7fd20c0119c0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.553+0000 7fd21a9cf740 1 -- 172.21.15.7:0/1667193781 shutdown_connections 2024-04-21T08:51:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.553+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/1667193781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e33c10 0x55a708e313e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.553+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/1667193781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e2fca0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.553+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/1667193781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e2a710 0x55a708e3f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.553+0000 7fd21a9cf740 1 -- 172.21.15.7:0/1667193781 >> 172.21.15.7:0/1667193781 conn(0x55a708e24e40 msgr2=0x55a708e23890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.557+0000 7fd21a9cf740 1 -- 172.21.15.7:0/1667193781 shutdown_connections 2024-04-21T08:51:26.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.562+0000 7fd21a9cf740 1 -- 172.21.15.7:0/1667193781 wait complete. 2024-04-21T08:51:26.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.562+0000 7fd21a9cf740 1 -- start start 2024-04-21T08:51:26.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.563+0000 7fd21a9cf740 1 Processor -- start 2024-04-21T08:51:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.563+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2a710 0x55a708e36e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.563+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e373d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.563+0000 7fd21a9cf740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e33c10 0x55a708e37910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.563+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f400f080 con 0x55a708e2a710 2024-04-21T08:51:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.563+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f40104f0 con 0x55a708e33c10 2024-04-21T08:51:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.563+0000 7fd21a9cf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f400f750 con 0x55a708e2b030 2024-04-21T08:51:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.563+0000 7fd21a9cf740 10 client.0 _ll_register_callbacks cb 0x7fd20c00bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.564+0000 7fd215376640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.564+0000 7fd215376640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e373d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:33198/0 (socket says 172.21.15.7:33198) 2024-04-21T08:51:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.564+0000 7fd215376640 1 -- 172.21.15.7:0/3449817991 learned_addr learned my addr 172.21.15.7:0/3449817991 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.564+0000 7fd215376640 1 -- 172.21.15.7:0/3449817991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e33c10 msgr2=0x55a708e37910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.564+0000 7fd215376640 1 --2- 172.21.15.7:0/3449817991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e33c10 0x55a708e37910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.564+0000 7fd215376640 1 -- 172.21.15.7:0/3449817991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2a710 msgr2=0x55a708e36e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:51:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.564+0000 7fd215376640 1 --2- 172.21.15.7:0/3449817991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2a710 0x55a708e36e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.564+0000 7fd215376640 1 -- 172.21.15.7:0/3449817991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a708ea8810 con 0x55a708e2b030 2024-04-21T08:51:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.565+0000 7fd215376640 1 --2- 172.21.15.7:0/3449817991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e373d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd20c01bb90 tx=0x7fd20c00b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c27f636fe8b824b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.565+0000 7fd1f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:51:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.565+0000 7fd1f97fa640 1 -- 172.21.15.7:0/3449817991 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c02b040 con 0x55a708e2b030 2024-04-21T08:51:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.565+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3449817991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd20c00b480 con 0x55a708e2b030 2024-04-21T08:51:26.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.565+0000 7fd1f97fa640 1 -- 172.21.15.7:0/3449817991 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7fd20c01b420 con 0x55a708e2b030 2024-04-21T08:51:26.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.566+0000 7fd1f97fa640 1 -- 172.21.15.7:0/3449817991 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd20c01b660 con 0x55a708e2b030 2024-04-21T08:51:26.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.567+0000 7fd21a9cf740 10 client.26467 fetch_fsmap learned FSMap version 23 2024-04-21T08:51:26.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.567+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3449817991 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a708ea9ae0 con 0x55a708e2b030 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.570+0000 7fd1f97fa640 1 -- 172.21.15.7:0/3449817991 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd20c00b480 con 0x55a708e2b030 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.570+0000 7fd1f97fa640 1 -- 172.21.15.7:0/3449817991 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fd20c069030 con 0x55a708e2b030 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.570+0000 7fd21a9cf740 10 client.26467 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.570+0000 7fd21a9cf740 10 client.26467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.570+0000 7fd21a9cf740 20 client.26467 populate_metadata read hostname 'smithi007' 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.570+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/3449817991 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a708eaa1c0 0x55a708eac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.570+0000 7fd21a9cf740 4 client.26467 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.570+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3449817991 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a708d96a70 con 0x55a708eaa1c0 2024-04-21T08:51:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.575+0000 7fd215b77640 1 --2- 172.21.15.7:0/3449817991 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a708eaa1c0 0x55a708eac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.582+0000 7fd215b77640 1 --2- 172.21.15.7:0/3449817991 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a708eaa1c0 0x55a708eac5e0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64dd5aed6d8580dc server_cookie=5218dccc742dd36c in_seq=0 out_seq=0 2024-04-21T08:51:26.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.582+0000 7fd1f97fa640 10 client.26467 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:26.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.585+0000 7fd1f97fa640 1 -- 172.21.15.7:0/3449817991 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x55a708d96a70 con 0x55a708eaa1c0 2024-04-21T08:51:26.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.585+0000 7fd1f97fa640 10 client.26467 handle_command_reply: tid=0 2024-04-21T08:51:26.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.585+0000 7fd21a9cf740 1 client.26467 shutdown 2024-04-21T08:51:26.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.587+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3449817991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 msgr2=0x55a708e373d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.587+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/3449817991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e373d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd20c01bb90 tx=0x7fd20c00b990 comp rx=0 tx=0).stop 2024-04-21T08:51:26.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.588+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3449817991 shutdown_connections 2024-04-21T08:51:26.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.588+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/3449817991 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a708eaa1c0 0x55a708eac5e0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.588+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/3449817991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a708e33c10 0x55a708e37910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.588+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/3449817991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a708e2b030 0x55a708e373d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.588+0000 7fd21a9cf740 1 --2- 172.21.15.7:0/3449817991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a708e2a710 0x55a708e36e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.588+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3449817991 >> 172.21.15.7:0/3449817991 conn(0x55a708e24e40 msgr2=0x55a708e26120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.588+0000 7fd1f97fa640 0 client.26467 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:26.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.590+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3449817991 shutdown_connections 2024-04-21T08:51:26.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.591+0000 7fd21a9cf740 1 -- 172.21.15.7:0/3449817991 wait complete. 2024-04-21T08:51:26.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.595+0000 7fd21a9cf740 20 client.26467 trim_cache size 0 max 16384 2024-04-21T08:51:26.596 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:51:26.597 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (15049 inodes in the stack)", 2024-04-21T08:51:26.597 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:51:26.597 INFO:teuthology.orchestra.run.smithi007.stdout: "81ebd966-d9b6-44a6-b87d-1c305ebffa72": { 2024-04-21T08:51:26.597 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:51:26.597 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "81ebd966-d9b6-44a6-b87d-1c305ebffa72", 2024-04-21T08:51:26.597 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:51:26.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:51:26.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:51:26.597 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:51:26.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.598+0000 7fd216b79640 1 -- 172.21.15.7:0/1308421039 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1fc0a3ef0 msgr2=0x7fd1fc0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.598+0000 7fd216b79640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1fc0a3ef0 0x7fd1fc0a63b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd200006fd0 tx=0x7fd200006c40 comp rx=0 tx=0).stop 2024-04-21T08:51:26.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.598+0000 7fd216b79640 1 -- 172.21.15.7:0/1308421039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd208004c70 msgr2=0x7fd2080a9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.598+0000 7fd216b79640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd208004c70 0x7fd2080a9350 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd21002db50 tx=0x7fd210098f40 comp rx=0 tx=0).stop 2024-04-21T08:51:26.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.601+0000 7fd216b79640 1 -- 172.21.15.7:0/1308421039 shutdown_connections 2024-04-21T08:51:26.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.601+0000 7fd216b79640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1fc0a3ef0 0x7fd1fc0a63b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.601+0000 7fd216b79640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd20809ace0 0x7fd2080a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.601+0000 7fd216b79640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2080986d0 0x7fd2080a3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.601+0000 7fd216b79640 1 --2- 172.21.15.7:0/1308421039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd208004c70 0x7fd2080a9350 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.601+0000 7fd216b79640 1 -- 172.21.15.7:0/1308421039 >> 172.21.15.7:0/1308421039 conn(0x7fd20800a730 msgr2=0x7fd2080922d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.603+0000 7fd216b79640 1 -- 172.21.15.7:0/1308421039 shutdown_connections 2024-04-21T08:51:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:26.603+0000 7fd216b79640 1 -- 172.21.15.7:0/1308421039 wait complete. 2024-04-21T08:51:26.624 INFO:tasks.cephfs.filesystem:scrub status for tag:81ebd966-d9b6-44a6-b87d-1c305ebffa72 - {'path': '/', 'tag': '81ebd966-d9b6-44a6-b87d-1c305ebffa72', 'options': 'recursive,force'} 2024-04-21T08:51:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:26 smithi016 ceph-mon[31259]: pgmap v207: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 1.39k op/s 2024-04-21T08:51:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:26 smithi077 ceph-mon[31615]: pgmap v207: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 1.39k op/s 2024-04-21T08:51:26.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:26 smithi007 ceph-mon[28477]: pgmap v207: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.5 MiB/s wr, 1.39k op/s 2024-04-21T08:51:28.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:28 smithi016 ceph-mon[31259]: pgmap v208: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 5.1 MiB/s wr, 1.09k op/s 2024-04-21T08:51:28.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:28 smithi077 ceph-mon[31615]: pgmap v208: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 5.1 MiB/s wr, 1.09k op/s 2024-04-21T08:51:28.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:28 smithi007 ceph-mon[28477]: pgmap v208: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 5.1 MiB/s wr, 1.09k op/s 2024-04-21T08:51:30.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:30 smithi016 ceph-mon[31259]: pgmap v209: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 5.2 MiB/s wr, 1.46k op/s 2024-04-21T08:51:30.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:30 smithi077 ceph-mon[31615]: pgmap v209: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 5.2 MiB/s wr, 1.46k op/s 2024-04-21T08:51:30.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:30 smithi007 ceph-mon[28477]: pgmap v209: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 5.2 MiB/s wr, 1.46k op/s 2024-04-21T08:51:32.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:32 smithi016 ceph-mon[31259]: pgmap v210: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 1.45k op/s 2024-04-21T08:51:32.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:32 smithi077 ceph-mon[31615]: pgmap v210: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 1.45k op/s 2024-04-21T08:51:32.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:32 smithi007 ceph-mon[28477]: pgmap v210: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 1.45k op/s 2024-04-21T08:51:34.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:34 smithi016 ceph-mon[31259]: pgmap v211: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 1.34k op/s 2024-04-21T08:51:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:34 smithi077 ceph-mon[31615]: pgmap v211: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 1.34k op/s 2024-04-21T08:51:34.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:34 smithi007 ceph-mon[28477]: pgmap v211: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 1.34k op/s 2024-04-21T08:51:36.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:36 smithi016 ceph-mon[31259]: pgmap v212: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 1.1 MiB/s wr, 1.26k op/s 2024-04-21T08:51:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:36 smithi077 ceph-mon[31615]: pgmap v212: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 1.1 MiB/s wr, 1.26k op/s 2024-04-21T08:51:36.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:36 smithi007 ceph-mon[28477]: pgmap v212: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 1.1 MiB/s wr, 1.26k op/s 2024-04-21T08:51:38.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:38 smithi016 ceph-mon[31259]: pgmap v213: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 557 KiB/s wr, 716 op/s 2024-04-21T08:51:38.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:38 smithi077 ceph-mon[31615]: pgmap v213: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 557 KiB/s wr, 716 op/s 2024-04-21T08:51:38.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:38 smithi007 ceph-mon[28477]: pgmap v213: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 557 KiB/s wr, 716 op/s 2024-04-21T08:51:39.679 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.678+0000 7f04a1a0e640 1 -- 172.21.15.7:0/1512810116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f049409acf0 msgr2=0x7f0494004740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:39.679 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.678+0000 7f04a1a0e640 1 --2- 172.21.15.7:0/1512810116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f049409acf0 0x7f0494004740 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f049c0676c0 tx=0x7f049c098260 comp rx=0 tx=0).stop 2024-04-21T08:51:39.679 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.678+0000 7f04a1a0e640 1 -- 172.21.15.7:0/1512810116 shutdown_connections 2024-04-21T08:51:39.679 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.678+0000 7f04a1a0e640 1 --2- 172.21.15.7:0/1512810116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04940986e0 0x7f0494098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.679 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.678+0000 7f04a1a0e640 1 --2- 172.21.15.7:0/1512810116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0494004c80 0x7f0494005100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.679 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.678+0000 7f04a1a0e640 1 --2- 172.21.15.7:0/1512810116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f049409acf0 0x7f0494004740 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.679 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.678+0000 7f04a1a0e640 1 -- 172.21.15.7:0/1512810116 >> 172.21.15.7:0/1512810116 conn(0x7f049400a740 msgr2=0x7f049400ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:39.683 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.683+0000 7f04a1a0e640 1 -- 172.21.15.7:0/1512810116 shutdown_connections 2024-04-21T08:51:39.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.684+0000 7f04a1a0e640 1 -- 172.21.15.7:0/1512810116 wait complete. 2024-04-21T08:51:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.684+0000 7f04a1a0e640 1 Processor -- start 2024-04-21T08:51:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.684+0000 7f04a1a0e640 1 -- start start 2024-04-21T08:51:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.685+0000 7f04a1a0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0494004c80 0x7f04941496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.685+0000 7f04a1a0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04940986e0 0x7f0494149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.685+0000 7f04a1a0e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049409acf0 0x7f0494143880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.685+0000 7f04a1a0e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f04940ac490 con 0x7f049409acf0 2024-04-21T08:51:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.685+0000 7f04a1a0e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f04940ac310 con 0x7f0494004c80 2024-04-21T08:51:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.685+0000 7f04a1a0e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f04940ac610 con 0x7f04940986e0 2024-04-21T08:51:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.685+0000 7f04a0a0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0494004c80 0x7f04941496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.685+0000 7f04a0a0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0494004c80 0x7f04941496d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:52212/0 (socket says 172.21.15.7:52212) 2024-04-21T08:51:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.685+0000 7f04a0a0c640 1 -- 172.21.15.7:0/686416724 learned_addr learned my addr 172.21.15.7:0/686416724 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:39.686 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.686+0000 7f04a120d640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049409acf0 0x7f0494143880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.686 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.686+0000 7f04a0a0c640 1 -- 172.21.15.7:0/686416724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04940986e0 msgr2=0x7f0494149c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:51:39.686 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.686+0000 7f04a0a0c640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04940986e0 0x7f0494149c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.686 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.686+0000 7f04a0a0c640 1 -- 172.21.15.7:0/686416724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049409acf0 msgr2=0x7f0494143880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:39.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.686+0000 7f04a0a0c640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049409acf0 0x7f0494143880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.686+0000 7f04a0a0c640 1 -- 172.21.15.7:0/686416724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0494144000 con 0x7f0494004c80 2024-04-21T08:51:39.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.687+0000 7f04a120d640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049409acf0 0x7f0494143880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:39.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.687+0000 7f04a0a0c640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0494004c80 0x7f04941496d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f049c0657a0 tx=0x7f049c06a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:39.690 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.689+0000 7f0499ffb640 1 -- 172.21.15.7:0/686416724 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c06a740 con 0x7f0494004c80 2024-04-21T08:51:39.690 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.689+0000 7f0499ffb640 1 -- 172.21.15.7:0/686416724 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f049c075040 con 0x7f0494004c80 2024-04-21T08:51:39.690 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.690+0000 7f0499ffb640 1 -- 172.21.15.7:0/686416724 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c0a03d0 con 0x7f0494004c80 2024-04-21T08:51:39.690 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.691+0000 7f04a1a0e640 1 -- 172.21.15.7:0/686416724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04941442f0 con 0x7f0494004c80 2024-04-21T08:51:39.691 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.691+0000 7f04a1a0e640 1 -- 172.21.15.7:0/686416724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0494150590 con 0x7f0494004c80 2024-04-21T08:51:39.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.700+0000 7f04a5859740 1 Processor -- start 2024-04-21T08:51:39.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.702+0000 7f04a5859740 1 -- start start 2024-04-21T08:51:39.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.702+0000 7f04a5859740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db326aac0 0x562db3269810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.702 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.702+0000 7f04a5859740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 0x562db326a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.702 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.702+0000 7f04a5859740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db32732d0 0x562db32756c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.702 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.702+0000 7f04a5859740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562db31ce8d0 con 0x562db32732d0 2024-04-21T08:51:39.702 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.702+0000 7f04a5859740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562db316be80 con 0x562db3269d50 2024-04-21T08:51:39.702 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.702+0000 7f04a5859740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562db31ca890 con 0x562db326aac0 2024-04-21T08:51:39.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.702+0000 7f04a0a0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 0x562db326a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.702+0000 7f04a0a0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 0x562db326a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:52228/0 (socket says 172.21.15.7:52228) 2024-04-21T08:51:39.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.702+0000 7f04a0a0c640 1 -- 172.21.15.7:0/1456998894 learned_addr learned my addr 172.21.15.7:0/1456998894 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:39.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.703+0000 7f0499ffb640 1 -- 172.21.15.7:0/686416724 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f049c0a8060 con 0x7f0494004c80 2024-04-21T08:51:39.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.703+0000 7f0499ffb640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f048c0a3d80 0x7f048c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.705 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.704+0000 7f0499ffb640 1 -- 172.21.15.7:0/686416724 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7f049c1550d0 con 0x7f0494004c80 2024-04-21T08:51:39.705 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.705+0000 7f04a120d640 1 --2- 172.21.15.7:0/1456998894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db326aac0 0x562db3269810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.707+0000 7f049bfff640 1 --2- 172.21.15.7:0/1456998894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db32732d0 0x562db32756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.707+0000 7f04a120d640 1 -- 172.21.15.7:0/1456998894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 msgr2=0x562db326a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:39.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.707+0000 7f04a120d640 1 --2- 172.21.15.7:0/1456998894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 0x562db326a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.707+0000 7f04a120d640 1 -- 172.21.15.7:0/1456998894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db32732d0 msgr2=0x562db32756c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:39.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.707+0000 7f04a120d640 1 --2- 172.21.15.7:0/1456998894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db32732d0 0x562db32756c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.707+0000 7f04a120d640 1 -- 172.21.15.7:0/1456998894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562db318d320 con 0x562db326aac0 2024-04-21T08:51:39.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.707+0000 7f04a120d640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f048c0a3d80 0x7f048c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.707+0000 7f049bfff640 1 --2- 172.21.15.7:0/1456998894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db32732d0 0x562db32756c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:39.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.708+0000 7f04a120d640 1 --2- 172.21.15.7:0/1456998894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db326aac0 0x562db3269810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f048800a9d0 tx=0x7f048800add0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee066053912658ed server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:39.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.709+0000 7f04a120d640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f048c0a3d80 0x7f048c0a6240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f048801e000 tx=0x7f048801b000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:39.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.710+0000 7f0486ffd640 1 -- 172.21.15.7:0/1456998894 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048800f040 con 0x562db326aac0 2024-04-21T08:51:39.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.710+0000 7f0486ffd640 1 -- 172.21.15.7:0/1456998894 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04880138c0 con 0x562db326aac0 2024-04-21T08:51:39.710 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.710+0000 7f0486ffd640 1 -- 172.21.15.7:0/1456998894 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048801c700 con 0x562db326aac0 2024-04-21T08:51:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.711+0000 7f04a5859740 1 -- 172.21.15.7:0/1456998894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db326aac0 msgr2=0x562db3269810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.711+0000 7f04a5859740 1 --2- 172.21.15.7:0/1456998894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db326aac0 0x562db3269810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f048800a9d0 tx=0x7f048800add0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.711+0000 7f04a5859740 1 -- 172.21.15.7:0/1456998894 shutdown_connections 2024-04-21T08:51:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.711+0000 7f04a5859740 1 --2- 172.21.15.7:0/1456998894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db32732d0 0x562db32756c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.712+0000 7f04a5859740 1 --2- 172.21.15.7:0/1456998894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 0x562db326a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.712+0000 7f04a5859740 1 --2- 172.21.15.7:0/1456998894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db326aac0 0x562db3269810 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.712+0000 7f04a5859740 1 -- 172.21.15.7:0/1456998894 >> 172.21.15.7:0/1456998894 conn(0x562db3230c90 msgr2=0x562db3264a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:39.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.715+0000 7f04a5859740 1 -- 172.21.15.7:0/1456998894 shutdown_connections 2024-04-21T08:51:39.716 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.716+0000 7f04a5859740 1 -- 172.21.15.7:0/1456998894 wait complete. 2024-04-21T08:51:39.716 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.716+0000 7f04a5859740 1 -- start start 2024-04-21T08:51:39.716 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.716+0000 7f04a5859740 1 Processor -- start 2024-04-21T08:51:39.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.717+0000 7f04a5859740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db3269d50 0x562db3272c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.717+0000 7f04a5859740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 0x562db326ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.717+0000 7f04a5859740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db32732d0 0x562db3270510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.717+0000 7f04a5859740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562db3276000 con 0x562db3269d50 2024-04-21T08:51:39.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.718+0000 7f04a5859740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562db3156eb0 con 0x562db326aac0 2024-04-21T08:51:39.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.718+0000 7f04a5859740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562db3276180 con 0x562db32732d0 2024-04-21T08:51:39.718 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.718+0000 7f04a5859740 10 client.0 _ll_register_callbacks cb 0x562db3183b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:39.719 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.718+0000 7f049bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 0x562db326ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.719 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.718+0000 7f049bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 0x562db326ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:52234/0 (socket says 172.21.15.7:52234) 2024-04-21T08:51:39.719 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.718+0000 7f049bfff640 1 -- 172.21.15.7:0/3064498939 learned_addr learned my addr 172.21.15.7:0/3064498939 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:39.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.721+0000 7f049bfff640 1 -- 172.21.15.7:0/3064498939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db32732d0 msgr2=0x562db3270510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:51:39.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.721+0000 7f049bfff640 1 --2- 172.21.15.7:0/3064498939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db32732d0 0x562db3270510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.721+0000 7f049bfff640 1 -- 172.21.15.7:0/3064498939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db3269d50 msgr2=0x562db3272c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:39.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.721+0000 7f049bfff640 1 --2- 172.21.15.7:0/3064498939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db3269d50 0x562db3272c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.721 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.721+0000 7f049bfff640 1 -- 172.21.15.7:0/3064498939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562db2ee1210 con 0x562db326aac0 2024-04-21T08:51:39.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.726+0000 7f049bfff640 1 --2- 172.21.15.7:0/3064498939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 0x562db326ffb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0490005fd0 tx=0x7f049000e800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=981765bf6fd6cf68 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.727+0000 7f047bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:51:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.728+0000 7f047bfff640 1 -- 172.21.15.7:0/3064498939 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490004230 con 0x562db326aac0 2024-04-21T08:51:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.728+0000 7f047bfff640 1 -- 172.21.15.7:0/3064498939 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7f0490004030 con 0x562db326aac0 2024-04-21T08:51:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.728+0000 7f04a5859740 1 -- 172.21.15.7:0/3064498939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04880138c0 con 0x562db326aac0 2024-04-21T08:51:39.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.732+0000 7f047bfff640 1 -- 172.21.15.7:0/3064498939 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0490004230 con 0x562db326aac0 2024-04-21T08:51:39.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.732+0000 7f04a5859740 10 client.26495 fetch_fsmap learned FSMap version 23 2024-04-21T08:51:39.734 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.732+0000 7f04a5859740 1 -- 172.21.15.7:0/3064498939 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562db2fad620 con 0x562db326aac0 2024-04-21T08:51:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.736+0000 7f047bfff640 1 -- 172.21.15.7:0/3064498939 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04880138c0 con 0x562db326aac0 2024-04-21T08:51:39.739 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.736+0000 7f047bfff640 1 -- 172.21.15.7:0/3064498939 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f049000f040 con 0x562db326aac0 2024-04-21T08:51:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.736+0000 7f04a5859740 10 client.26495 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:51:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.736+0000 7f04a5859740 10 client.26495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.736+0000 7f04a5859740 20 client.26495 populate_metadata read hostname 'smithi007' 2024-04-21T08:51:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.736+0000 7f04a5859740 1 --2- 172.21.15.7:0/3064498939 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562db3283850 0x562db3285cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.736+0000 7f04a5859740 4 client.26495 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:39.740 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.736+0000 7f04a5859740 1 -- 172.21.15.7:0/3064498939 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562db31e2880 con 0x562db3283850 2024-04-21T08:51:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.739+0000 7f04a0a0c640 1 --2- 172.21.15.7:0/3064498939 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562db3283850 0x562db3285cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.751 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.751+0000 7f04a0a0c640 1 --2- 172.21.15.7:0/3064498939 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562db3283850 0x562db3285cf0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a6ffa5ef17d593b server_cookie=d113dd0799dc219e in_seq=0 out_seq=0 2024-04-21T08:51:39.751 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.752+0000 7f047bfff640 10 client.26495 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.755+0000 7f047bfff640 1 -- 172.21.15.7:0/3064498939 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562db31e2880 con 0x562db3283850 2024-04-21T08:51:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.755+0000 7f047bfff640 10 client.26495 handle_command_reply: tid=0 2024-04-21T08:51:39.758 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.756+0000 7f04a5859740 1 client.26495 shutdown 2024-04-21T08:51:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.759+0000 7f04a5859740 1 -- 172.21.15.7:0/3064498939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 msgr2=0x562db326ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:39.760 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.759+0000 7f04a5859740 1 --2- 172.21.15.7:0/3064498939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 0x562db326ffb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0490005fd0 tx=0x7f049000e800 comp rx=0 tx=0).stop 2024-04-21T08:51:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.762+0000 7f04a5859740 1 -- 172.21.15.7:0/3064498939 shutdown_connections 2024-04-21T08:51:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.762+0000 7f04a5859740 1 --2- 172.21.15.7:0/3064498939 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562db3283850 0x562db3285cf0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.762+0000 7f04a5859740 1 --2- 172.21.15.7:0/3064498939 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db32732d0 0x562db3270510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.762+0000 7f04a5859740 1 --2- 172.21.15.7:0/3064498939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 0x562db326ffb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.762+0000 7f04a5859740 1 --2- 172.21.15.7:0/3064498939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db3269d50 0x562db3272c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.762+0000 7f04a5859740 1 -- 172.21.15.7:0/3064498939 >> 172.21.15.7:0/3064498939 conn(0x562db3230c90 msgr2=0x562db3266790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:39.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.762+0000 7f047bfff640 0 client.26495 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:39.771 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.769+0000 7f04a5859740 1 -- 172.21.15.7:0/3064498939 shutdown_connections 2024-04-21T08:51:39.771 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.770+0000 7f04a5859740 1 -- 172.21.15.7:0/3064498939 wait complete. 2024-04-21T08:51:39.771 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.770+0000 7f04a5859740 20 client.26495 trim_cache size 0 max 16384 2024-04-21T08:51:39.805 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.805+0000 7f04a5859740 1 Processor -- start 2024-04-21T08:51:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.807+0000 7f04a5859740 1 -- start start 2024-04-21T08:51:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.807+0000 7f04a5859740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 0x562db327ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.807+0000 7f04a5859740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db326aac0 0x562db32726e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.808+0000 7f04a5859740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db32732d0 0x562db3272c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.808+0000 7f04a5859740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0494158960 con 0x562db326aac0 2024-04-21T08:51:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.808+0000 7f04a5859740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f049415b240 con 0x562db3269d50 2024-04-21T08:51:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.808+0000 7f04a5859740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0494159030 con 0x562db32732d0 2024-04-21T08:51:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.808+0000 7f049bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db326aac0 0x562db32726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.808+0000 7f049bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db326aac0 0x562db32726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40222/0 (socket says 172.21.15.7:40222) 2024-04-21T08:51:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.808+0000 7f049bfff640 1 -- 172.21.15.7:0/4085837855 learned_addr learned my addr 172.21.15.7:0/4085837855 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.808+0000 7f04a0a0c640 1 --2- 172.21.15.7:0/4085837855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 0x562db327ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.808+0000 7f04a120d640 1 --2- 172.21.15.7:0/4085837855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db32732d0 0x562db3272c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.809+0000 7f04a0a0c640 1 -- 172.21.15.7:0/4085837855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db32732d0 msgr2=0x562db3272c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.809+0000 7f04a0a0c640 1 --2- 172.21.15.7:0/4085837855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db32732d0 0x562db3272c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.809+0000 7f04a0a0c640 1 -- 172.21.15.7:0/4085837855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db326aac0 msgr2=0x562db32726e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.809+0000 7f04a0a0c640 1 --2- 172.21.15.7:0/4085837855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db326aac0 0x562db32726e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.810+0000 7f04a0a0c640 1 -- 172.21.15.7:0/4085837855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562db32759f0 con 0x562db3269d50 2024-04-21T08:51:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.810+0000 7f04a0a0c640 1 --2- 172.21.15.7:0/4085837855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 0x562db327ef90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x562db326b220 tx=0x7f049c068390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58b7d6cdb810e326 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.812+0000 7f04867fc640 1 -- 172.21.15.7:0/4085837855 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c1205e0 con 0x562db3269d50 2024-04-21T08:51:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.812+0000 7f04867fc640 1 -- 172.21.15.7:0/4085837855 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f049c09c040 con 0x562db3269d50 2024-04-21T08:51:39.811 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.812+0000 7f04867fc640 1 -- 172.21.15.7:0/4085837855 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c11edc0 con 0x562db3269d50 2024-04-21T08:51:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.814+0000 7f04a5859740 1 -- 172.21.15.7:0/4085837855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 msgr2=0x562db327ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.814+0000 7f04a5859740 1 --2- 172.21.15.7:0/4085837855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 0x562db327ef90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x562db326b220 tx=0x7f049c068390 comp rx=0 tx=0).stop 2024-04-21T08:51:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.818+0000 7f04a5859740 1 -- 172.21.15.7:0/4085837855 shutdown_connections 2024-04-21T08:51:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.818+0000 7f04a5859740 1 --2- 172.21.15.7:0/4085837855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db32732d0 0x562db3272c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.818+0000 7f04a5859740 1 --2- 172.21.15.7:0/4085837855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db326aac0 0x562db32726e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.818+0000 7f04a5859740 1 --2- 172.21.15.7:0/4085837855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db3269d50 0x562db327ef90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.818+0000 7f04a5859740 1 -- 172.21.15.7:0/4085837855 >> 172.21.15.7:0/4085837855 conn(0x562db3230c90 msgr2=0x562db326b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.823+0000 7f04a5859740 1 -- 172.21.15.7:0/4085837855 shutdown_connections 2024-04-21T08:51:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.824+0000 7f04a5859740 1 -- 172.21.15.7:0/4085837855 wait complete. 2024-04-21T08:51:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.824+0000 7f04a5859740 1 -- start start 2024-04-21T08:51:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.824+0000 7f04a5859740 1 Processor -- start 2024-04-21T08:51:39.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f04a5859740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db3269d50 0x562db327b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.826 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f04a5859740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 0x562db3276ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f04a5859740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db32732d0 0x562db32772f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f04a5859740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0494159700 con 0x562db32732d0 2024-04-21T08:51:39.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f04a5859740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f049415ab70 con 0x562db326aac0 2024-04-21T08:51:39.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f04a5859740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0494159dd0 con 0x562db3269d50 2024-04-21T08:51:39.827 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f04a5859740 10 client.0 _ll_register_callbacks cb 0x7f04900092c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:39.828 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f04a0a0c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db3269d50 0x562db327b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.828 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f049bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 0x562db3276ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.828 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f04a0a0c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db3269d50 0x562db327b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:34078/0 (socket says 172.21.15.7:34078) 2024-04-21T08:51:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f04a0a0c640 1 -- 172.21.15.7:0/3287592776 learned_addr learned my addr 172.21.15.7:0/3287592776 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.827+0000 7f04a120d640 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db32732d0 0x562db32772f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.828+0000 7f049bfff640 1 -- 172.21.15.7:0/3287592776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db3269d50 msgr2=0x562db327b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.828+0000 7f049bfff640 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db3269d50 0x562db327b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.828+0000 7f049bfff640 1 -- 172.21.15.7:0/3287592776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db32732d0 msgr2=0x562db32772f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.828+0000 7f049bfff640 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db32732d0 0x562db32772f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.828+0000 7f049bfff640 1 -- 172.21.15.7:0/3287592776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562db3277af0 con 0x562db326aac0 2024-04-21T08:51:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.836+0000 7f049bfff640 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 0x562db3276ce0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f049000ed00 tx=0x7f0490009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56ced1a66d122d45 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.837+0000 7f047bfff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:51:39.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.839+0000 7f047bfff640 1 -- 172.21.15.7:0/3287592776 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490004230 con 0x562db326aac0 2024-04-21T08:51:39.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.839+0000 7f047bfff640 1 -- 172.21.15.7:0/3287592776 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7f0490010970 con 0x562db326aac0 2024-04-21T08:51:39.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.840+0000 7f04a5859740 1 -- 172.21.15.7:0/3287592776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f049c09c040 con 0x562db326aac0 2024-04-21T08:51:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.841+0000 7f047bfff640 1 -- 172.21.15.7:0/3287592776 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0490010bb0 con 0x562db326aac0 2024-04-21T08:51:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.842+0000 7f04a5859740 10 client.26501 fetch_fsmap learned FSMap version 23 2024-04-21T08:51:39.841 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.842+0000 7f04a5859740 1 -- 172.21.15.7:0/3287592776 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562db32e9f80 con 0x562db326aac0 2024-04-21T08:51:39.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.844+0000 7f047bfff640 1 -- 172.21.15.7:0/3287592776 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f049c09c040 con 0x562db326aac0 2024-04-21T08:51:39.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.844+0000 7f047bfff640 1 -- 172.21.15.7:0/3287592776 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f049002e060 con 0x562db326aac0 2024-04-21T08:51:39.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.844+0000 7f04a5859740 10 client.26501 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:51:39.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.844+0000 7f04a5859740 10 client.26501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:39.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.844+0000 7f04a5859740 20 client.26501 populate_metadata read hostname 'smithi007' 2024-04-21T08:51:39.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.844+0000 7f04a5859740 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562db32ea6b0 0x562db32ecad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:39.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.844+0000 7f04a5859740 4 client.26501 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:51:39.844 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.844+0000 7f04a5859740 1 -- 172.21.15.7:0/3287592776 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562db316e3d0 con 0x562db32ea6b0 2024-04-21T08:51:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.848+0000 7f04a0a0c640 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562db32ea6b0 0x562db32ecad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.849+0000 7f04a0a0c640 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562db32ea6b0 0x562db32ecad0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38d9e9bdf19a8889 server_cookie=b1ac372dfc0740bc in_seq=0 out_seq=0 2024-04-21T08:51:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:39.849+0000 7f047bfff640 10 client.26501 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:40.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:40 smithi016 ceph-mon[31259]: pgmap v214: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.8 MiB/s wr, 858 op/s 2024-04-21T08:51:40.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:40 smithi077 ceph-mon[31615]: pgmap v214: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.8 MiB/s wr, 858 op/s 2024-04-21T08:51:40.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:40 smithi007 ceph-mon[28477]: pgmap v214: 129 pgs: 20 active+clean+snaptrim, 63 active+clean, 46 active+clean+snaptrim_wait; 1.3 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.8 MiB/s wr, 858 op/s 2024-04-21T08:51:42.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:42 smithi016 ceph-mon[31259]: pgmap v215: 129 pgs: 19 active+clean+snaptrim, 64 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 3.2 MiB/s wr, 626 op/s 2024-04-21T08:51:42.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:42 smithi077 ceph-mon[31615]: pgmap v215: 129 pgs: 19 active+clean+snaptrim, 64 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 3.2 MiB/s wr, 626 op/s 2024-04-21T08:51:42.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:42 smithi007 ceph-mon[28477]: pgmap v215: 129 pgs: 19 active+clean+snaptrim, 64 active+clean, 46 active+clean+snaptrim_wait; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 3.2 MiB/s wr, 626 op/s 2024-04-21T08:51:44.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:44 smithi016 ceph-mon[31259]: pgmap v216: 129 pgs: 15 active+clean+snaptrim, 78 active+clean, 36 active+clean+snaptrim_wait; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 450 op/s 2024-04-21T08:51:44.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:44 smithi077 ceph-mon[31615]: pgmap v216: 129 pgs: 15 active+clean+snaptrim, 78 active+clean, 36 active+clean+snaptrim_wait; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 450 op/s 2024-04-21T08:51:44.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:44 smithi007 ceph-mon[28477]: pgmap v216: 129 pgs: 15 active+clean+snaptrim, 78 active+clean, 36 active+clean+snaptrim_wait; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 450 op/s 2024-04-21T08:51:46.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:46 smithi016 ceph-mon[31259]: pgmap v217: 129 pgs: 14 active+clean+snaptrim, 84 active+clean, 31 active+clean+snaptrim_wait; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 686 op/s 2024-04-21T08:51:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:46 smithi077 ceph-mon[31615]: pgmap v217: 129 pgs: 14 active+clean+snaptrim, 84 active+clean, 31 active+clean+snaptrim_wait; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 686 op/s 2024-04-21T08:51:46.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:46 smithi007 ceph-mon[28477]: pgmap v217: 129 pgs: 14 active+clean+snaptrim, 84 active+clean, 31 active+clean+snaptrim_wait; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 686 op/s 2024-04-21T08:51:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.482+0000 7f047bfff640 1 -- 172.21.15.7:0/3287592776 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562db316e3d0 con 0x562db32ea6b0 2024-04-21T08:51:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.482+0000 7f047bfff640 10 client.26501 handle_command_reply: tid=0 2024-04-21T08:51:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.482+0000 7f04a5859740 1 client.26501 shutdown 2024-04-21T08:51:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f04a5859740 1 -- 172.21.15.7:0/3287592776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 msgr2=0x562db3276ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f04a5859740 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 0x562db3276ce0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f049000ed00 tx=0x7f0490009620 comp rx=0 tx=0).stop 2024-04-21T08:51:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f04a5859740 1 -- 172.21.15.7:0/3287592776 shutdown_connections 2024-04-21T08:51:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f04a5859740 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562db32ea6b0 0x562db32ecad0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f04a5859740 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562db32732d0 0x562db32772f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f04a5859740 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562db326aac0 0x562db3276ce0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f04a5859740 1 --2- 172.21.15.7:0/3287592776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562db3269d50 0x562db327b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f04a5859740 1 -- 172.21.15.7:0/3287592776 >> 172.21.15.7:0/3287592776 conn(0x562db3230c90 msgr2=0x562db3263250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f047bfff640 0 client.26501 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f04a5859740 1 -- 172.21.15.7:0/3287592776 shutdown_connections 2024-04-21T08:51:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f04a5859740 1 -- 172.21.15.7:0/3287592776 wait complete. 2024-04-21T08:51:47.484 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.483+0000 7f04a5859740 20 client.26501 trim_cache size 0 max 16384 2024-04-21T08:51:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:51:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:51:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:51:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.485+0000 7f04a1a0e640 1 -- 172.21.15.7:0/686416724 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f048c0a3d80 msgr2=0x7f048c0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.485+0000 7f04a1a0e640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f048c0a3d80 0x7f048c0a6240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f048801e000 tx=0x7f048801b000 comp rx=0 tx=0).stop 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.485+0000 7f04a1a0e640 1 -- 172.21.15.7:0/686416724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0494004c80 msgr2=0x7f04941496d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.485+0000 7f04a1a0e640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0494004c80 0x7f04941496d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f049c0657a0 tx=0x7f049c06a300 comp rx=0 tx=0).stop 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.485+0000 7f04a1a0e640 1 -- 172.21.15.7:0/686416724 shutdown_connections 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.485+0000 7f04a1a0e640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049409acf0 0x7f0494143880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.485+0000 7f04a1a0e640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f048c0a3d80 0x7f048c0a6240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.485+0000 7f04a1a0e640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04940986e0 0x7f0494149c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.485+0000 7f04a1a0e640 1 --2- 172.21.15.7:0/686416724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0494004c80 0x7f04941496d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.485+0000 7f04a1a0e640 1 -- 172.21.15.7:0/686416724 >> 172.21.15.7:0/686416724 conn(0x7f049400a740 msgr2=0x7f04940976c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.486+0000 7f04a1a0e640 1 -- 172.21.15.7:0/686416724 shutdown_connections 2024-04-21T08:51:47.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:51:47.486+0000 7f04a1a0e640 1 -- 172.21.15.7:0/686416724 wait complete. 2024-04-21T08:51:48.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:48 smithi016 ceph-mon[31259]: pgmap v218: 129 pgs: 14 active+clean+snaptrim, 85 active+clean, 30 active+clean+snaptrim_wait; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 3.4 MiB/s wr, 676 op/s 2024-04-21T08:51:48.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:48 smithi077 ceph-mon[31615]: pgmap v218: 129 pgs: 14 active+clean+snaptrim, 85 active+clean, 30 active+clean+snaptrim_wait; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 3.4 MiB/s wr, 676 op/s 2024-04-21T08:51:48.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:48 smithi007 ceph-mon[28477]: pgmap v218: 129 pgs: 14 active+clean+snaptrim, 85 active+clean, 30 active+clean+snaptrim_wait; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 3.4 MiB/s wr, 676 op/s 2024-04-21T08:51:50.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:50 smithi016 ceph-mon[31259]: pgmap v219: 129 pgs: 14 active+clean+snaptrim, 85 active+clean, 30 active+clean+snaptrim_wait; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.6 MiB/s wr, 1.18k op/s 2024-04-21T08:51:50.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:50 smithi077 ceph-mon[31615]: pgmap v219: 129 pgs: 14 active+clean+snaptrim, 85 active+clean, 30 active+clean+snaptrim_wait; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.6 MiB/s wr, 1.18k op/s 2024-04-21T08:51:50.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:50 smithi007 ceph-mon[28477]: pgmap v219: 129 pgs: 14 active+clean+snaptrim, 85 active+clean, 30 active+clean+snaptrim_wait; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.6 MiB/s wr, 1.18k op/s 2024-04-21T08:51:52.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:52 smithi016 ceph-mon[31259]: pgmap v220: 129 pgs: 13 active+clean+snaptrim, 91 active+clean, 25 active+clean+snaptrim_wait; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 1.45k op/s 2024-04-21T08:51:52.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:52 smithi077 ceph-mon[31615]: pgmap v220: 129 pgs: 13 active+clean+snaptrim, 91 active+clean, 25 active+clean+snaptrim_wait; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 1.45k op/s 2024-04-21T08:51:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:52 smithi007 ceph-mon[28477]: pgmap v220: 129 pgs: 13 active+clean+snaptrim, 91 active+clean, 25 active+clean+snaptrim_wait; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 1.45k op/s 2024-04-21T08:51:53.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:53 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:53.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:53 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:53 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:54.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:54 smithi016 ceph-mon[31259]: pgmap v221: 129 pgs: 13 active+clean+snaptrim, 91 active+clean, 25 active+clean+snaptrim_wait; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 1.51k op/s 2024-04-21T08:51:54.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:54 smithi077 ceph-mon[31615]: pgmap v221: 129 pgs: 13 active+clean+snaptrim, 91 active+clean, 25 active+clean+snaptrim_wait; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 1.51k op/s 2024-04-21T08:51:54.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:54 smithi007 ceph-mon[28477]: pgmap v221: 129 pgs: 13 active+clean+snaptrim, 91 active+clean, 25 active+clean+snaptrim_wait; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 1.51k op/s 2024-04-21T08:51:55.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:55 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:55.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:55 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:55.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:55 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:51:55.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:55 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:55.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:55 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:55.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:55 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:51:55.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:55 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:55.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:55 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:55.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:55 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:51:56.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:51:56.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:51:56.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:56 smithi016 ceph-mon[31259]: pgmap v222: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 1.82k op/s 2024-04-21T08:51:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.808+0000 7f4698ee0640 1 -- 172.21.15.7:0/111287210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46941106a0 msgr2=0x7f4694109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.808+0000 7f4698ee0640 1 --2- 172.21.15.7:0/111287210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46941106a0 0x7f4694109150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4688009900 tx=0x7f468802bc80 comp rx=0 tx=0).stop 2024-04-21T08:51:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.809+0000 7f4698ee0640 1 -- 172.21.15.7:0/111287210 shutdown_connections 2024-04-21T08:51:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.809+0000 7f4698ee0640 1 --2- 172.21.15.7:0/111287210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f469410e1b0 0x7f469410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.809+0000 7f4698ee0640 1 --2- 172.21.15.7:0/111287210 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4694110b10 0x7f4694109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.809+0000 7f4698ee0640 1 --2- 172.21.15.7:0/111287210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46941106a0 0x7f4694109150 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.809+0000 7f4698ee0640 1 -- 172.21.15.7:0/111287210 >> 172.21.15.7:0/111287210 conn(0x7f469406e360 msgr2=0x7f469406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.812+0000 7f4698ee0640 1 -- 172.21.15.7:0/111287210 shutdown_connections 2024-04-21T08:51:56.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.813+0000 7f4698ee0640 1 -- 172.21.15.7:0/111287210 wait complete. 2024-04-21T08:51:56.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.814+0000 7f4698ee0640 1 Processor -- start 2024-04-21T08:51:56.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.814+0000 7f4698ee0640 1 -- start start 2024-04-21T08:51:56.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.815+0000 7f4698ee0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f469410e1b0 0x7f46941a7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.815+0000 7f4698ee0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46941106a0 0x7f46941a7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.815+0000 7f4698ee0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4694110b10 0x7f46941abef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.815+0000 7f4698ee0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4694121140 con 0x7f469410e1b0 2024-04-21T08:51:56.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.815+0000 7f4698ee0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4694120fc0 con 0x7f4694110b10 2024-04-21T08:51:56.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.816+0000 7f4692d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4694110b10 0x7f46941abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.816+0000 7f4692d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4694110b10 0x7f46941abef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59900/0 (socket says 172.21.15.7:59900) 2024-04-21T08:51:56.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.816+0000 7f4692d9d640 1 -- 172.21.15.7:0/741670280 learned_addr learned my addr 172.21.15.7:0/741670280 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:56.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.816+0000 7f4691d9b640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46941106a0 0x7f46941a7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.816+0000 7f4698ee0640 1 -- 172.21.15.7:0/741670280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f46941212c0 con 0x7f46941106a0 2024-04-21T08:51:56.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.817+0000 7f4691d9b640 1 -- 172.21.15.7:0/741670280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4694110b10 msgr2=0x7f46941abef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.817+0000 7f4691d9b640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4694110b10 0x7f46941abef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.817+0000 7f4691d9b640 1 -- 172.21.15.7:0/741670280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f469410e1b0 msgr2=0x7f46941a7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.817+0000 7f4691d9b640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f469410e1b0 0x7f46941a7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.817+0000 7f4691d9b640 1 -- 172.21.15.7:0/741670280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46941ac620 con 0x7f46941106a0 2024-04-21T08:51:56.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.817+0000 7f4692d9d640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4694110b10 0x7f46941abef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:56.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.818+0000 7f4691d9b640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46941106a0 0x7f46941a7bc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4694108f70 tx=0x7f468400b850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.818+0000 7f46837fe640 1 -- 172.21.15.7:0/741670280 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684015020 con 0x7f46941106a0 2024-04-21T08:51:56.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.818+0000 7f46837fe640 1 -- 172.21.15.7:0/741670280 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4684011ce0 con 0x7f46941106a0 2024-04-21T08:51:56.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.818+0000 7f46837fe640 1 -- 172.21.15.7:0/741670280 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684010da0 con 0x7f46941106a0 2024-04-21T08:51:56.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.819+0000 7f4698ee0640 1 -- 172.21.15.7:0/741670280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46941ac910 con 0x7f46941106a0 2024-04-21T08:51:56.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.819+0000 7f4698ee0640 1 -- 172.21.15.7:0/741670280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46941b41a0 con 0x7f46941106a0 2024-04-21T08:51:56.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.823+0000 7f469bbac740 1 Processor -- start 2024-04-21T08:51:56.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.824+0000 7f46837fe640 1 -- 172.21.15.7:0/741670280 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f4684018020 con 0x7f46941106a0 2024-04-21T08:51:56.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.825+0000 7f46837fe640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f467c0a3ef0 0x7f467c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.825+0000 7f46837fe640 1 -- 172.21.15.7:0/741670280 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7f46840d0290 con 0x7f46941106a0 2024-04-21T08:51:56.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.827+0000 7f469bbac740 1 -- start start 2024-04-21T08:51:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.827+0000 7f469bbac740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156580 0x559064156960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.827+0000 7f469bbac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559064156ea0 0x55906415f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.827+0000 7f469bbac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55906415fa80 0x559064161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.827+0000 7f469bbac740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5590640bb1e0 con 0x559064156580 2024-04-21T08:51:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.827+0000 7f469bbac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559064058390 con 0x55906415fa80 2024-04-21T08:51:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.827+0000 7f469bbac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5590640b71a0 con 0x559064156ea0 2024-04-21T08:51:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.827+0000 7f4692d9d640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f467c0a3ef0 0x7f467c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.827+0000 7f4692d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55906415fa80 0x559064161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.827+0000 7f4692d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55906415fa80 0x559064161e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59908/0 (socket says 172.21.15.7:59908) 2024-04-21T08:51:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.827+0000 7f4692d9d640 1 -- 172.21.15.7:0/1240362588 learned_addr learned my addr 172.21.15.7:0/1240362588 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.828+0000 7f469259c640 1 --2- 172.21.15.7:0/1240362588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156580 0x559064156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.828+0000 7f4691d9b640 1 --2- 172.21.15.7:0/1240362588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559064156ea0 0x55906415f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.828+0000 7f4692d9d640 1 -- 172.21.15.7:0/1240362588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559064156ea0 msgr2=0x55906415f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.828+0000 7f4692d9d640 1 --2- 172.21.15.7:0/1240362588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559064156ea0 0x55906415f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.828+0000 7f4692d9d640 1 -- 172.21.15.7:0/1240362588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156580 msgr2=0x559064156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.828+0000 7f4692d9d640 1 --2- 172.21.15.7:0/1240362588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156580 0x559064156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.828+0000 7f4692d9d640 1 -- 172.21.15.7:0/1240362588 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559063e977d0 con 0x55906415fa80 2024-04-21T08:51:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.828+0000 7f4692d9d640 1 --2- 172.21.15.7:0/1240362588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55906415fa80 0x559064161e70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f468c013f50 tx=0x7f468c011750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52010383b918eadf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.829+0000 7f4680ff9640 1 -- 172.21.15.7:0/1240362588 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c00dd70 con 0x55906415fa80 2024-04-21T08:51:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.829+0000 7f4680ff9640 1 -- 172.21.15.7:0/1240362588 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f468c025070 con 0x55906415fa80 2024-04-21T08:51:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.829+0000 7f4680ff9640 1 -- 172.21.15.7:0/1240362588 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c029410 con 0x55906415fa80 2024-04-21T08:51:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.829+0000 7f4692d9d640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f467c0a3ef0 0x7f467c0a63b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f468c011c50 tx=0x7f468c015000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.831+0000 7f469bbac740 1 -- 172.21.15.7:0/1240362588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55906415fa80 msgr2=0x559064161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.831+0000 7f469bbac740 1 --2- 172.21.15.7:0/1240362588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55906415fa80 0x559064161e70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f468c013f50 tx=0x7f468c011750 comp rx=0 tx=0).stop 2024-04-21T08:51:56.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.832+0000 7f469bbac740 1 -- 172.21.15.7:0/1240362588 shutdown_connections 2024-04-21T08:51:56.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.832+0000 7f469bbac740 1 --2- 172.21.15.7:0/1240362588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55906415fa80 0x559064161e70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.832+0000 7f469bbac740 1 --2- 172.21.15.7:0/1240362588 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559064156ea0 0x55906415f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.832+0000 7f469bbac740 1 --2- 172.21.15.7:0/1240362588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156580 0x559064156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.832+0000 7f469bbac740 1 -- 172.21.15.7:0/1240362588 >> 172.21.15.7:0/1240362588 conn(0x559064150cb0 msgr2=0x559064151090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.835+0000 7f469bbac740 1 -- 172.21.15.7:0/1240362588 shutdown_connections 2024-04-21T08:51:56.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.837+0000 7f469bbac740 1 -- 172.21.15.7:0/1240362588 wait complete. 2024-04-21T08:51:56.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.837+0000 7f469bbac740 1 -- start start 2024-04-21T08:51:56.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.837+0000 7f469bbac740 1 Processor -- start 2024-04-21T08:51:56.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.841+0000 7f469bbac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 0x55906415cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.841+0000 7f469bbac740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 0x55906415d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.841+0000 7f469bbac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.841+0000 7f469bbac740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x559064162810 con 0x559064156ea0 2024-04-21T08:51:56.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.841+0000 7f469bbac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5590640742f0 con 0x559064156580 2024-04-21T08:51:56.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.841+0000 7f469bbac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559064162990 con 0x55906415fa80 2024-04-21T08:51:56.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.841+0000 7f469bbac740 10 client.0 _ll_register_callbacks cb 0x559064081580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:56.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.842+0000 7f4692d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.842+0000 7f4692d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:49598/0 (socket says 172.21.15.7:49598) 2024-04-21T08:51:56.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.842+0000 7f4692d9d640 1 -- 172.21.15.7:0/3077761998 learned_addr learned my addr 172.21.15.7:0/3077761998 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:56.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.843+0000 7f4692d9d640 1 -- 172.21.15.7:0/3077761998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 msgr2=0x55906415cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:56.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.843+0000 7f4692d9d640 1 --2- 172.21.15.7:0/3077761998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 0x55906415cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.843+0000 7f4692d9d640 1 -- 172.21.15.7:0/3077761998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 msgr2=0x55906415d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:56.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.843+0000 7f4692d9d640 1 --2- 172.21.15.7:0/3077761998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 0x55906415d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.843+0000 7f4692d9d640 1 -- 172.21.15.7:0/3077761998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559063dcd3f0 con 0x55906415fa80 2024-04-21T08:51:56.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.844+0000 7f4692d9d640 1 --2- 172.21.15.7:0/3077761998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f468c009540 tx=0x7f468c021f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ce379a80103eae9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.844+0000 7f4679ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:51:56.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.844+0000 7f4679ffb640 1 -- 172.21.15.7:0/3077761998 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c013b40 con 0x55906415fa80 2024-04-21T08:51:56.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.844+0000 7f4679ffb640 1 -- 172.21.15.7:0/3077761998 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7f468c013250 con 0x55906415fa80 2024-04-21T08:51:56.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.844+0000 7f469bbac740 1 -- 172.21.15.7:0/3077761998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f468c025070 con 0x55906415fa80 2024-04-21T08:51:56.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.845+0000 7f4679ffb640 1 -- 172.21.15.7:0/3077761998 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f468c029410 con 0x55906415fa80 2024-04-21T08:51:56.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.846+0000 7f469bbac740 10 client.26488 fetch_fsmap learned FSMap version 23 2024-04-21T08:51:56.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.846+0000 7f469bbac740 1 -- 172.21.15.7:0/3077761998 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559063d8d0f0 con 0x55906415fa80 2024-04-21T08:51:56.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.849+0000 7f4679ffb640 1 -- 172.21.15.7:0/3077761998 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f468c025070 con 0x55906415fa80 2024-04-21T08:51:56.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.849+0000 7f4679ffb640 1 -- 172.21.15.7:0/3077761998 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f468c01f070 con 0x55906415fa80 2024-04-21T08:51:56.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.849+0000 7f469bbac740 10 client.26488 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:51:56.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.849+0000 7f469bbac740 10 client.26488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:56.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.849+0000 7f469bbac740 20 client.26488 populate_metadata read hostname 'smithi007' 2024-04-21T08:51:56.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.849+0000 7f469bbac740 1 --2- 172.21.15.7:0/3077761998 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559064180cf0 0x559064183160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.849+0000 7f469bbac740 4 client.26488 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:56.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.849+0000 7f469bbac740 1 -- 172.21.15.7:0/3077761998 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590640c8c00 con 0x559064180cf0 2024-04-21T08:51:56.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.851+0000 7f469259c640 1 --2- 172.21.15.7:0/3077761998 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559064180cf0 0x559064183160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.853 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:56 smithi077 ceph-mon[31615]: pgmap v222: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 1.82k op/s 2024-04-21T08:51:56.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.854+0000 7f469259c640 1 --2- 172.21.15.7:0/3077761998 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559064180cf0 0x559064183160 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50ac0e425d098012 server_cookie=8b0e7ee79d237f99 in_seq=0 out_seq=0 2024-04-21T08:51:56.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.854+0000 7f4679ffb640 10 client.26488 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:56.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.856+0000 7f4679ffb640 1 -- 172.21.15.7:0/3077761998 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590640c8c00 con 0x559064180cf0 2024-04-21T08:51:56.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.856+0000 7f4679ffb640 10 client.26488 handle_command_reply: tid=0 2024-04-21T08:51:56.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.856+0000 7f469bbac740 1 client.26488 shutdown 2024-04-21T08:51:56.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.857+0000 7f469bbac740 1 -- 172.21.15.7:0/3077761998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 msgr2=0x55906415d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.857+0000 7f469bbac740 1 --2- 172.21.15.7:0/3077761998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f468c009540 tx=0x7f468c021f00 comp rx=0 tx=0).stop 2024-04-21T08:51:56.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.857+0000 7f469bbac740 1 -- 172.21.15.7:0/3077761998 shutdown_connections 2024-04-21T08:51:56.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.857+0000 7f469bbac740 1 --2- 172.21.15.7:0/3077761998 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559064180cf0 0x559064183160 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.857+0000 7f469bbac740 1 --2- 172.21.15.7:0/3077761998 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d610 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.857+0000 7f469bbac740 1 --2- 172.21.15.7:0/3077761998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 0x55906415d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.857+0000 7f469bbac740 1 --2- 172.21.15.7:0/3077761998 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 0x55906415cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.857+0000 7f469bbac740 1 -- 172.21.15.7:0/3077761998 >> 172.21.15.7:0/3077761998 conn(0x559064150cb0 msgr2=0x559064152a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.857+0000 7f4679ffb640 0 client.26488 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:56.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.857+0000 7f469bbac740 1 -- 172.21.15.7:0/3077761998 shutdown_connections 2024-04-21T08:51:56.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.858+0000 7f469bbac740 1 -- 172.21.15.7:0/3077761998 wait complete. 2024-04-21T08:51:56.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.858+0000 7f469bbac740 20 client.26488 trim_cache size 0 max 16384 2024-04-21T08:51:56.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.891+0000 7f469bbac740 1 Processor -- start 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.891+0000 7f469bbac740 1 -- start start 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.891+0000 7f469bbac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 0x55906415f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.891+0000 7f469bbac740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 0x55906416b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.891+0000 7f469bbac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.891+0000 7f469bbac740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4694068e00 con 0x559064156ea0 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.891+0000 7f469bbac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46941b6720 con 0x559064156580 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.891+0000 7f469bbac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f46941b4510 con 0x55906415fa80 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.892+0000 7f4692d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.892+0000 7f4692d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:49602/0 (socket says 172.21.15.7:49602) 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.892+0000 7f4692d9d640 1 -- 172.21.15.7:0/3398105093 learned_addr learned my addr 172.21.15.7:0/3398105093 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.892+0000 7f469259c640 1 --2- 172.21.15.7:0/3398105093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 0x55906415f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.893+0000 7f4692d9d640 1 -- 172.21.15.7:0/3398105093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 msgr2=0x55906415f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:56.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.893+0000 7f4692d9d640 1 --2- 172.21.15.7:0/3398105093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 0x55906415f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.893+0000 7f4692d9d640 1 -- 172.21.15.7:0/3398105093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 msgr2=0x55906416b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:56.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.893+0000 7f4692d9d640 1 --2- 172.21.15.7:0/3398105093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 0x55906416b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.893+0000 7f4692d9d640 1 -- 172.21.15.7:0/3398105093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590641834d0 con 0x55906415fa80 2024-04-21T08:51:56.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.893+0000 7f4692d9d640 1 --2- 172.21.15.7:0/3398105093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d260 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f468c005c50 tx=0x7f468c005c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2a9b0ecda98dde7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.893+0000 7f467bfff640 1 -- 172.21.15.7:0/3398105093 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c044030 con 0x55906415fa80 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.893+0000 7f467bfff640 1 -- 172.21.15.7:0/3398105093 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f468c025040 con 0x55906415fa80 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.894+0000 7f469bbac740 1 -- 172.21.15.7:0/3398105093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 msgr2=0x55906415d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.894+0000 7f469bbac740 1 --2- 172.21.15.7:0/3398105093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d260 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f468c005c50 tx=0x7f468c005c80 comp rx=0 tx=0).stop 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.895+0000 7f469bbac740 1 -- 172.21.15.7:0/3398105093 shutdown_connections 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.895+0000 7f469bbac740 1 --2- 172.21.15.7:0/3398105093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415d260 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.895+0000 7f469bbac740 1 --2- 172.21.15.7:0/3398105093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 0x55906416b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.895+0000 7f469bbac740 1 --2- 172.21.15.7:0/3398105093 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 0x55906415f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.895+0000 7f469bbac740 1 -- 172.21.15.7:0/3398105093 >> 172.21.15.7:0/3398105093 conn(0x559064150cb0 msgr2=0x55906415cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.895+0000 7f469bbac740 1 -- 172.21.15.7:0/3398105093 shutdown_connections 2024-04-21T08:51:56.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.896+0000 7f469bbac740 1 -- 172.21.15.7:0/3398105093 wait complete. 2024-04-21T08:51:56.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.896+0000 7f469bbac740 1 -- start start 2024-04-21T08:51:56.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.896+0000 7f469bbac740 1 Processor -- start 2024-04-21T08:51:56.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.896+0000 7f469bbac740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 0x559064159810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.896+0000 7f469bbac740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 0x559064159d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.896+0000 7f469bbac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.896+0000 7f469bbac740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f46941b4be0 con 0x559064156ea0 2024-04-21T08:51:56.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.896+0000 7f469bbac740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46941b6050 con 0x559064156580 2024-04-21T08:51:56.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.896+0000 7f469bbac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f46941b52b0 con 0x55906415fa80 2024-04-21T08:51:56.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.896+0000 7f469bbac740 10 client.0 _ll_register_callbacks cb 0x7f468c013da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:56.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.897+0000 7f4692d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.897+0000 7f4692d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:49618/0 (socket says 172.21.15.7:49618) 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.897+0000 7f4692d9d640 1 -- 172.21.15.7:0/1297092526 learned_addr learned my addr 172.21.15.7:0/1297092526 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.897+0000 7f469259c640 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 0x559064159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.897+0000 7f4692d9d640 1 -- 172.21.15.7:0/1297092526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 msgr2=0x559064159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.897+0000 7f4692d9d640 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 0x559064159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.897+0000 7f4692d9d640 1 -- 172.21.15.7:0/1297092526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 msgr2=0x559064159d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.897+0000 7f4692d9d640 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 0x559064159d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.897+0000 7f4692d9d640 1 -- 172.21.15.7:0/1297092526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559064156cd0 con 0x55906415fa80 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.898+0000 7f4692d9d640 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415a290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f468c020960 tx=0x7f468c029f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d16a2bb25d106065 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.899+0000 7f4679ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.899+0000 7f4679ffb640 1 -- 172.21.15.7:0/1297092526 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c04f060 con 0x55906415fa80 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.899+0000 7f469bbac740 1 -- 172.21.15.7:0/1297092526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f468c025040 con 0x55906415fa80 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.899+0000 7f4679ffb640 1 -- 172.21.15.7:0/1297092526 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7f468c029950 con 0x55906415fa80 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.899+0000 7f4679ffb640 1 -- 172.21.15.7:0/1297092526 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f468c044040 con 0x55906415fa80 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.899+0000 7f469bbac740 10 client.26494 fetch_fsmap learned FSMap version 23 2024-04-21T08:51:56.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.899+0000 7f469bbac740 1 -- 172.21.15.7:0/1297092526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559064173eb0 con 0x55906415fa80 2024-04-21T08:51:56.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.901+0000 7f4679ffb640 1 -- 172.21.15.7:0/1297092526 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f468c025040 con 0x55906415fa80 2024-04-21T08:51:56.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.901+0000 7f4679ffb640 1 -- 172.21.15.7:0/1297092526 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f468c01f040 con 0x55906415fa80 2024-04-21T08:51:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.902+0000 7f469bbac740 10 client.26494 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:51:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.902+0000 7f469bbac740 10 client.26494 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.902+0000 7f469bbac740 20 client.26494 populate_metadata read hostname 'smithi007' 2024-04-21T08:51:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.902+0000 7f469bbac740 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559064174560 0x559064176980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.902+0000 7f469bbac740 4 client.26494 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:51:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.902+0000 7f469bbac740 1 -- 172.21.15.7:0/1297092526 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5590640c2a70 con 0x559064174560 2024-04-21T08:51:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.902+0000 7f469259c640 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559064174560 0x559064176980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:56 smithi007 ceph-mon[28477]: pgmap v222: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 1.82k op/s 2024-04-21T08:51:56.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.903+0000 7f469259c640 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559064174560 0x559064176980 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7878ad6d13a2ac53 server_cookie=1f9ee2e3f208a3fe in_seq=0 out_seq=0 2024-04-21T08:51:56.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.903+0000 7f4679ffb640 10 client.26494 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:56.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.904+0000 7f4679ffb640 1 -- 172.21.15.7:0/1297092526 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x5590640c2a70 con 0x559064174560 2024-04-21T08:51:56.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.904+0000 7f4679ffb640 10 client.26494 handle_command_reply: tid=0 2024-04-21T08:51:56.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.904+0000 7f469bbac740 1 client.26494 shutdown 2024-04-21T08:51:56.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.905+0000 7f469bbac740 1 -- 172.21.15.7:0/1297092526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 msgr2=0x55906415a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.905+0000 7f469bbac740 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415a290 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f468c020960 tx=0x7f468c029f70 comp rx=0 tx=0).stop 2024-04-21T08:51:56.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.905+0000 7f469bbac740 1 -- 172.21.15.7:0/1297092526 shutdown_connections 2024-04-21T08:51:56.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.905+0000 7f469bbac740 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x559064174560 0x559064176980 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.905+0000 7f469bbac740 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55906415fa80 0x55906415a290 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.905+0000 7f469bbac740 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559064156ea0 0x559064159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.905+0000 7f469bbac740 1 --2- 172.21.15.7:0/1297092526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559064156580 0x559064159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.905+0000 7f469bbac740 1 -- 172.21.15.7:0/1297092526 >> 172.21.15.7:0/1297092526 conn(0x559064150cb0 msgr2=0x55906414fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.905+0000 7f4679ffb640 0 client.26494 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:51:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.905+0000 7f469bbac740 1 -- 172.21.15.7:0/1297092526 shutdown_connections 2024-04-21T08:51:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.905+0000 7f469bbac740 1 -- 172.21.15.7:0/1297092526 wait complete. 2024-04-21T08:51:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.908+0000 7f469bbac740 20 client.26494 trim_cache size 0 max 16384 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (15533 inodes in the stack)", 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi007.stdout: "81ebd966-d9b6-44a6-b87d-1c305ebffa72": { 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "81ebd966-d9b6-44a6-b87d-1c305ebffa72", 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:51:56.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.915+0000 7f4698ee0640 1 -- 172.21.15.7:0/741670280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f467c0a3ef0 msgr2=0x7f467c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.915+0000 7f4698ee0640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f467c0a3ef0 0x7f467c0a63b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f468c011c50 tx=0x7f468c015000 comp rx=0 tx=0).stop 2024-04-21T08:51:56.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.915+0000 7f4698ee0640 1 -- 172.21.15.7:0/741670280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46941106a0 msgr2=0x7f46941a7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.915+0000 7f4698ee0640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46941106a0 0x7f46941a7bc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4694108f70 tx=0x7f468400b850 comp rx=0 tx=0).stop 2024-04-21T08:51:56.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.915+0000 7f4698ee0640 1 -- 172.21.15.7:0/741670280 shutdown_connections 2024-04-21T08:51:56.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.915+0000 7f4698ee0640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4694110b10 0x7f46941abef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.915+0000 7f4698ee0640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f467c0a3ef0 0x7f467c0a63b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.915+0000 7f4698ee0640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f46941106a0 0x7f46941a7bc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.915+0000 7f4698ee0640 1 --2- 172.21.15.7:0/741670280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f469410e1b0 0x7f46941a7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.915+0000 7f4698ee0640 1 -- 172.21.15.7:0/741670280 >> 172.21.15.7:0/741670280 conn(0x7f469406e360 msgr2=0x7f469410a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.918+0000 7f4698ee0640 1 -- 172.21.15.7:0/741670280 shutdown_connections 2024-04-21T08:51:56.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:51:56.918+0000 7f4698ee0640 1 -- 172.21.15.7:0/741670280 wait complete. 2024-04-21T08:51:56.941 INFO:tasks.cephfs.filesystem:scrub status for tag:81ebd966-d9b6-44a6-b87d-1c305ebffa72 - {'path': '/', 'tag': '81ebd966-d9b6-44a6-b87d-1c305ebffa72', 'options': 'recursive,force'} 2024-04-21T08:51:58.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:51:58 smithi016 ceph-mon[31259]: pgmap v223: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 1.60k op/s 2024-04-21T08:51:58.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:51:58 smithi077 ceph-mon[31615]: pgmap v223: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 1.60k op/s 2024-04-21T08:51:58.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:51:58 smithi007 ceph-mon[28477]: pgmap v223: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 1.60k op/s 2024-04-21T08:52:00.633 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.633+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3392030265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59a0074380 msgr2=0x7f59a0074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.633 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.633+0000 7f59a6fe6640 1 --2- 172.21.15.7:0/3392030265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59a0074380 0x7f59a0074760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5998009ca0 tx=0x7f599802c180 comp rx=0 tx=0).stop 2024-04-21T08:52:00.633 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.633+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3392030265 shutdown_connections 2024-04-21T08:52:00.633 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.633+0000 7f59a6fe6640 1 --2- 172.21.15.7:0/3392030265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59a010e780 0x7f59a0110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.633 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.633+0000 7f59a6fe6640 1 --2- 172.21.15.7:0/3392030265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59a0074ca0 0x7f59a010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.633 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.633+0000 7f59a6fe6640 1 --2- 172.21.15.7:0/3392030265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59a0074380 0x7f59a0074760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.634 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.633+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3392030265 >> 172.21.15.7:0/3392030265 conn(0x7f59a00fda30 msgr2=0x7f59a00ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:00.634 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.634+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3392030265 shutdown_connections 2024-04-21T08:52:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.635+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3392030265 wait complete. 2024-04-21T08:52:00.635 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.636+0000 7f59a6fe6640 1 Processor -- start 2024-04-21T08:52:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.636+0000 7f59a6fe6640 1 -- start start 2024-04-21T08:52:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.636+0000 7f59a6fe6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59a0074380 0x7f59a010b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.636+0000 7f59a6fe6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59a0074ca0 0x7f59a010b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.636+0000 7f59a6fe6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59a010e780 0x7f59a010be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.636+0000 7f59a6fe6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f59a0114400 con 0x7f59a0074ca0 2024-04-21T08:52:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.636+0000 7f59a6fe6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f59a0114280 con 0x7f59a010e780 2024-04-21T08:52:00.636 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.636+0000 7f59a6fe6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59a0114580 con 0x7f59a0074380 2024-04-21T08:52:00.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.637+0000 7f599ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59a0074ca0 0x7f59a010b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.637 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.637+0000 7f599ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59a0074ca0 0x7f59a010b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59666/0 (socket says 172.21.15.7:59666) 2024-04-21T08:52:00.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.637+0000 7f599ffff640 1 -- 172.21.15.7:0/3751647506 learned_addr learned my addr 172.21.15.7:0/3751647506 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:00.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.637+0000 7f59a5583640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59a010e780 0x7f59a010be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.637+0000 7f599ffff640 1 -- 172.21.15.7:0/3751647506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59a0074380 msgr2=0x7f59a010b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:52:00.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.637+0000 7f599ffff640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59a0074380 0x7f59a010b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.637+0000 7f599ffff640 1 -- 172.21.15.7:0/3751647506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59a010e780 msgr2=0x7f59a010be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.637+0000 7f599ffff640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59a010e780 0x7f59a010be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.637+0000 7f599ffff640 1 -- 172.21.15.7:0/3751647506 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a006b970 con 0x7f59a0074ca0 2024-04-21T08:52:00.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.637+0000 7f59a5583640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59a010e780 0x7f59a010be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:00.638 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.638+0000 7f599ffff640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59a0074ca0 0x7f59a010b8d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f599000ea20 tx=0x7f599000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.639+0000 7f599dffb640 1 -- 172.21.15.7:0/3751647506 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599000cd80 con 0x7f59a0074ca0 2024-04-21T08:52:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.639+0000 7f599dffb640 1 -- 172.21.15.7:0/3751647506 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59900108e0 con 0x7f59a0074ca0 2024-04-21T08:52:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.639+0000 7f599dffb640 1 -- 172.21.15.7:0/3751647506 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5990018840 con 0x7f59a0074ca0 2024-04-21T08:52:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.639+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3751647506 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59a006bc60 con 0x7f59a0074ca0 2024-04-21T08:52:00.639 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.639+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3751647506 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59a006c180 con 0x7f59a0074ca0 2024-04-21T08:52:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.643+0000 7f59a9bd3740 1 Processor -- start 2024-04-21T08:52:00.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.644+0000 7f59a9bd3740 1 -- start start 2024-04-21T08:52:00.644 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.644+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd37620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.644+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd37b60 0x55d5cdd37fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.645+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd410e0 0x55d5cdd434d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.645+0000 7f59a9bd3740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d5cdc9c8d0 con 0x55d5cdd37b60 2024-04-21T08:52:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.645+0000 7f59a9bd3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d5cdc39e80 con 0x55d5cdd388d0 2024-04-21T08:52:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.645+0000 7f59a9bd3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d5cdc98890 con 0x55d5cdd410e0 2024-04-21T08:52:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.645+0000 7f59a4d82640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd37b60 0x55d5cdd37fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.645+0000 7f59a4d82640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd37b60 0x55d5cdd37fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59678/0 (socket says 172.21.15.7:59678) 2024-04-21T08:52:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.645+0000 7f59a4d82640 1 -- 172.21.15.7:0/239700427 learned_addr learned my addr 172.21.15.7:0/239700427 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.645+0000 7f59a5583640 1 --2- 172.21.15.7:0/239700427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd37620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.646+0000 7f599dffb640 1 -- 172.21.15.7:0/3751647506 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f5990010430 con 0x7f59a0074ca0 2024-04-21T08:52:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.646+0000 7f59a4d82640 1 -- 172.21.15.7:0/239700427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd410e0 msgr2=0x55d5cdd434d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:52:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.646+0000 7f59a4d82640 1 --2- 172.21.15.7:0/239700427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd410e0 0x55d5cdd434d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.646+0000 7f59a4d82640 1 -- 172.21.15.7:0/239700427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 msgr2=0x55d5cdd37620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.646+0000 7f59a4d82640 1 --2- 172.21.15.7:0/239700427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd37620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.646+0000 7f59a4d82640 1 -- 172.21.15.7:0/239700427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5cdc5b320 con 0x55d5cdd37b60 2024-04-21T08:52:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.646+0000 7f59a4d82640 1 --2- 172.21.15.7:0/239700427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd37b60 0x55d5cdd37fe0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f599802c710 tx=0x7f59980323f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8640f8308dcae4c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.647+0000 7f599dffb640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59880a3eb0 0x7f59880a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.647+0000 7f5986ffd640 1 -- 172.21.15.7:0/239700427 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998041020 con 0x55d5cdd37b60 2024-04-21T08:52:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.647+0000 7f599dffb640 1 -- 172.21.15.7:0/3751647506 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7f5990010d70 con 0x7f59a0074ca0 2024-04-21T08:52:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.647+0000 7f5986ffd640 1 -- 172.21.15.7:0/239700427 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5998003e20 con 0x55d5cdd37b60 2024-04-21T08:52:00.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.647+0000 7f5986ffd640 1 -- 172.21.15.7:0/239700427 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998039040 con 0x55d5cdd37b60 2024-04-21T08:52:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.648+0000 7f59a9bd3740 1 -- 172.21.15.7:0/239700427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd37b60 msgr2=0x55d5cdd37fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.648+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/239700427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd37b60 0x55d5cdd37fe0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f599802c710 tx=0x7f59980323f0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.649+0000 7f59a5583640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59880a3eb0 0x7f59880a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.649+0000 7f59a9bd3740 1 -- 172.21.15.7:0/239700427 shutdown_connections 2024-04-21T08:52:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.649+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/239700427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd410e0 0x55d5cdd434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.649+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/239700427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd37b60 0x55d5cdd37fe0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.649+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/239700427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd37620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.649+0000 7f59a9bd3740 1 -- 172.21.15.7:0/239700427 >> 172.21.15.7:0/239700427 conn(0x55d5cdcfec90 msgr2=0x55d5cdd32850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:00.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.650+0000 7f59a9bd3740 1 -- 172.21.15.7:0/239700427 shutdown_connections 2024-04-21T08:52:00.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.650+0000 7f59a9bd3740 1 -- 172.21.15.7:0/239700427 wait complete. 2024-04-21T08:52:00.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.650+0000 7f59a9bd3740 1 -- start start 2024-04-21T08:52:00.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.651+0000 7f59a9bd3740 1 Processor -- start 2024-04-21T08:52:00.652 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.652+0000 7f59a5583640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59880a3eb0 0x7f59880a6370 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5994009c30 tx=0x7f59940093d0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.652 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.652+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd37b60 0x55d5cdd40ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.653 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.653+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd3de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.653 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.653+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd410e0 0x55d5cdd3e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.653 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.653+0000 7f599ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd3de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.654 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.653+0000 7f599ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd3de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:59972/0 (socket says 172.21.15.7:59972) 2024-04-21T08:52:00.654 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.653+0000 7f59a5583640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd410e0 0x55d5cdd3e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.654 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.653+0000 7f59a5583640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd410e0 0x55d5cdd3e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:49644/0 (socket says 172.21.15.7:49644) 2024-04-21T08:52:00.654 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.653+0000 7f59a5583640 1 -- 172.21.15.7:0/2818348420 learned_addr learned my addr 172.21.15.7:0/2818348420 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:00.654 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.654+0000 7f59a9bd3740 1 -- 172.21.15.7:0/2818348420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d5cdd43db0 con 0x55d5cdd37b60 2024-04-21T08:52:00.654 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.654+0000 7f59a9bd3740 1 -- 172.21.15.7:0/2818348420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d5cdc24eb0 con 0x55d5cdd388d0 2024-04-21T08:52:00.654 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.654+0000 7f59a9bd3740 1 -- 172.21.15.7:0/2818348420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d5cdd43f30 con 0x55d5cdd410e0 2024-04-21T08:52:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.654+0000 7f59a9bd3740 10 client.0 _ll_register_callbacks cb 0x55d5cdc51b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.654+0000 7f599ffff640 1 -- 172.21.15.7:0/2818348420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd410e0 msgr2=0x55d5cdd3e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.655 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.654+0000 7f599ffff640 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd410e0 0x55d5cdd3e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.655+0000 7f599ffff640 1 -- 172.21.15.7:0/2818348420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd37b60 msgr2=0x55d5cdd40ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.655+0000 7f599ffff640 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd37b60 0x55d5cdd40ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.655+0000 7f599ffff640 1 -- 172.21.15.7:0/2818348420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5cd9af210 con 0x55d5cdd388d0 2024-04-21T08:52:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.655+0000 7f59a9bd3740 1 -- 172.21.15.7:0/2818348420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5998003e20 con 0x55d5cdd388d0 2024-04-21T08:52:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.655+0000 7f599ffff640 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd3de70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f59900ce4a0 tx=0x7f599009c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c8904c23d0bfb2e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.656 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.655+0000 7f59a5583640 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd410e0 0x55d5cdd3e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:00.657 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.658+0000 7f597ffff640 10 client.26522 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:52:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.658+0000 7f597ffff640 1 -- 172.21.15.7:0/2818348420 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59900b6260 con 0x55d5cdd388d0 2024-04-21T08:52:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.658+0000 7f597ffff640 1 -- 172.21.15.7:0/2818348420 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7f59900ce720 con 0x55d5cdd388d0 2024-04-21T08:52:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.658+0000 7f597ffff640 1 -- 172.21.15.7:0/2818348420 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f59900ce920 con 0x55d5cdd388d0 2024-04-21T08:52:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.660+0000 7f59a9bd3740 10 client.26522 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.660+0000 7f59a9bd3740 1 -- 172.21.15.7:0/2818348420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({fsmap=23}) -- 0x55d5cda7b620 con 0x55d5cdd388d0 2024-04-21T08:52:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.664+0000 7f597ffff640 1 -- 172.21.15.7:0/2818348420 <== mon.2 v2:172.21.15.16:3300/0 4 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f59900bd0b0 con 0x55d5cdd388d0 2024-04-21T08:52:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.665+0000 7f59a9bd3740 10 client.26522 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.665+0000 7f59a9bd3740 10 client.26522 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.665+0000 7f59a9bd3740 20 client.26522 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.665+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5cdd50cd0 0x55d5cdd530e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.665+0000 7f59a9bd3740 4 client.26522 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.665+0000 7f59a9bd3740 1 -- 172.21.15.7:0/2818348420 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5cdcb0880 con 0x55d5cdd50cd0 2024-04-21T08:52:00.665 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.665+0000 7f59a4d82640 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5cdd50cd0 0x55d5cdd530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.666 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.666+0000 7f59a4d82640 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5cdd50cd0 0x55d5cdd530e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c672b4630f7a29aa server_cookie=9375c035e1679469 in_seq=0 out_seq=0 2024-04-21T08:52:00.667 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.667+0000 7f597ffff640 10 client.26522 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.668+0000 7f597ffff640 1 -- 172.21.15.7:0/2818348420 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d5cdcb0880 con 0x55d5cdd50cd0 2024-04-21T08:52:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.668+0000 7f597ffff640 10 client.26522 handle_command_reply: tid=0 2024-04-21T08:52:00.668 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.668+0000 7f59a9bd3740 1 client.26522 shutdown 2024-04-21T08:52:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.669+0000 7f59a9bd3740 1 -- 172.21.15.7:0/2818348420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 msgr2=0x55d5cdd3de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.669+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd3de70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f59900ce4a0 tx=0x7f599009c940 comp rx=0 tx=0).stop 2024-04-21T08:52:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.669+0000 7f59a9bd3740 1 -- 172.21.15.7:0/2818348420 shutdown_connections 2024-04-21T08:52:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.669+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5cdd50cd0 0x55d5cdd530e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.669+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd410e0 0x55d5cdd3e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.669+0000 7f597ffff640 0 client.26522 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.669+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd3de70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.669+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/2818348420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd37b60 0x55d5cdd40ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.670+0000 7f59a9bd3740 1 -- 172.21.15.7:0/2818348420 >> 172.21.15.7:0/2818348420 conn(0x55d5cdcfec90 msgr2=0x55d5cdd345a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:00.670 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.670+0000 7f59a9bd3740 1 -- 172.21.15.7:0/2818348420 shutdown_connections 2024-04-21T08:52:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.670+0000 7f59a9bd3740 1 -- 172.21.15.7:0/2818348420 wait complete. 2024-04-21T08:52:00.671 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.671+0000 7f59a9bd3740 20 client.26522 trim_cache size 0 max 16384 2024-04-21T08:52:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.699+0000 7f59a9bd3740 1 Processor -- start 2024-04-21T08:52:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.699+0000 7f59a9bd3740 1 -- start start 2024-04-21T08:52:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.700+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 0x55d5cdd329f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.700+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd4ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.700+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd410e0 0x55d5cdd405a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.700+0000 7f59a9bd3740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f59a0073e10 con 0x55d5cdd410e0 2024-04-21T08:52:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.700+0000 7f59a9bd3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f59a01c76d0 con 0x55d5cdd388d0 2024-04-21T08:52:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.700+0000 7f59a9bd3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59a01c54c0 con 0x55d5cdd37b60 2024-04-21T08:52:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.700+0000 7f59a4d82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 0x55d5cdd329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.700+0000 7f59a4d82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 0x55d5cdd329f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:49952/0 (socket says 172.21.15.7:49656) 2024-04-21T08:52:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.700+0000 7f59a4d82640 1 -- 172.21.15.7:0/1423192336 learned_addr learned my addr 172.21.15.7:0/1423192336 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.700+0000 7f599ffff640 1 --2- 172.21.15.7:0/1423192336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd4ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.701+0000 7f59a4d82640 1 -- 172.21.15.7:0/1423192336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 msgr2=0x55d5cdd4ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.701+0000 7f59a4d82640 1 --2- 172.21.15.7:0/1423192336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd4ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.701+0000 7f59a4d82640 1 -- 172.21.15.7:0/1423192336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd410e0 msgr2=0x55d5cdd405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.701+0000 7f59a4d82640 1 --2- 172.21.15.7:0/1423192336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd410e0 0x55d5cdd405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.701+0000 7f59a4d82640 1 -- 172.21.15.7:0/1423192336 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5cdd2e120 con 0x55d5cdd37b60 2024-04-21T08:52:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.701+0000 7f59a4d82640 1 --2- 172.21.15.7:0/1423192336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 0x55d5cdd329f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f599802f020 tx=0x7f599803f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45e6d11951b98a86 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.703+0000 7f59867fc640 1 -- 172.21.15.7:0/1423192336 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599804d020 con 0x55d5cdd37b60 2024-04-21T08:52:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.703+0000 7f59867fc640 1 -- 172.21.15.7:0/1423192336 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5998005bd0 con 0x55d5cdd37b60 2024-04-21T08:52:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.703+0000 7f59867fc640 1 -- 172.21.15.7:0/1423192336 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998039040 con 0x55d5cdd37b60 2024-04-21T08:52:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.704+0000 7f59a9bd3740 1 -- 172.21.15.7:0/1423192336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 msgr2=0x55d5cdd329f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.704+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/1423192336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 0x55d5cdd329f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f599802f020 tx=0x7f599803f040 comp rx=0 tx=0).stop 2024-04-21T08:52:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.705+0000 7f59a9bd3740 1 -- 172.21.15.7:0/1423192336 shutdown_connections 2024-04-21T08:52:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.705+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/1423192336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd410e0 0x55d5cdd405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.705+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/1423192336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd388d0 0x55d5cdd4ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.706+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/1423192336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 0x55d5cdd329f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.706+0000 7f59a9bd3740 1 -- 172.21.15.7:0/1423192336 >> 172.21.15.7:0/1423192336 conn(0x55d5cdcfec90 msgr2=0x55d5cdd3c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.706+0000 7f59a9bd3740 1 -- 172.21.15.7:0/1423192336 shutdown_connections 2024-04-21T08:52:00.707 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.706+0000 7f59a9bd3740 1 -- 172.21.15.7:0/1423192336 wait complete. 2024-04-21T08:52:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.707+0000 7f59a9bd3740 1 -- start start 2024-04-21T08:52:00.708 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.707+0000 7f59a9bd3740 1 Processor -- start 2024-04-21T08:52:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.708+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 0x55d5cdd493b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.708+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd388d0 0x55d5cdd3b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.708+0000 7f59a9bd3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd410e0 0x55d5cdd3b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.708+0000 7f59a9bd3740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f59a01c5b90 con 0x55d5cdd388d0 2024-04-21T08:52:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.708+0000 7f59a9bd3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f59a01c7000 con 0x55d5cdd410e0 2024-04-21T08:52:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.708+0000 7f59a9bd3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59a01c6260 con 0x55d5cdd37b60 2024-04-21T08:52:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.708+0000 7f59a9bd3740 10 client.0 _ll_register_callbacks cb 0x7f5990099030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.709+0000 7f599ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd388d0 0x55d5cdd3b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.709+0000 7f599ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd388d0 0x55d5cdd3b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59706/0 (socket says 172.21.15.7:59706) 2024-04-21T08:52:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.709+0000 7f59a4d82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 0x55d5cdd493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.709+0000 7f599ffff640 1 -- 172.21.15.7:0/643267420 learned_addr learned my addr 172.21.15.7:0/643267420 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.709+0000 7f59a5583640 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd410e0 0x55d5cdd3b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.709+0000 7f59a4d82640 1 -- 172.21.15.7:0/643267420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd410e0 msgr2=0x55d5cdd3b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.709+0000 7f59a4d82640 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd410e0 0x55d5cdd3b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.709+0000 7f59a4d82640 1 -- 172.21.15.7:0/643267420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd388d0 msgr2=0x55d5cdd3b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.709+0000 7f59a4d82640 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd388d0 0x55d5cdd3b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.709+0000 7f59a4d82640 1 -- 172.21.15.7:0/643267420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5cdd3bb70 con 0x55d5cdd37b60 2024-04-21T08:52:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.709+0000 7f59a4d82640 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 0x55d5cdd493b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5998009620 tx=0x7f5998004910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5454971fa9ca2ab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.710+0000 7f597ffff640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:52:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.710+0000 7f597ffff640 1 -- 172.21.15.7:0/643267420 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998039040 con 0x55d5cdd37b60 2024-04-21T08:52:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.710+0000 7f597ffff640 1 -- 172.21.15.7:0/643267420 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9700+0+0 (secure 0 0 0) 0x7f5998003fd0 con 0x55d5cdd37b60 2024-04-21T08:52:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.710+0000 7f59a9bd3740 1 -- 172.21.15.7:0/643267420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5998005bd0 con 0x55d5cdd37b60 2024-04-21T08:52:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.710+0000 7f597ffff640 1 -- 172.21.15.7:0/643267420 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5998060050 con 0x55d5cdd37b60 2024-04-21T08:52:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.710+0000 7f59a9bd3740 10 client.26506 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.710+0000 7f59a9bd3740 1 -- 172.21.15.7:0/643267420 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d5cdd42670 con 0x55d5cdd37b60 2024-04-21T08:52:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.711+0000 7f597ffff640 1 -- 172.21.15.7:0/643267420 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5998005bd0 con 0x55d5cdd37b60 2024-04-21T08:52:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.711+0000 7f597ffff640 1 -- 172.21.15.7:0/643267420 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f599803c030 con 0x55d5cdd37b60 2024-04-21T08:52:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.711+0000 7f59a9bd3740 10 client.26506 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.711+0000 7f59a9bd3740 10 client.26506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.711+0000 7f59a9bd3740 20 client.26506 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.711+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5cdd50cd0 0x55d5cddb9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.711+0000 7f59a9bd3740 4 client.26506 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:52:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.711+0000 7f59a9bd3740 1 -- 172.21.15.7:0/643267420 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d5cdc3c3d0 con 0x55d5cdd50cd0 2024-04-21T08:52:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.712+0000 7f599ffff640 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5cdd50cd0 0x55d5cddb9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.713+0000 7f599ffff640 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5cdd50cd0 0x55d5cddb9550 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41c5b2df2daf1b86 server_cookie=847cfc5e4147609d in_seq=0 out_seq=0 2024-04-21T08:52:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:00.713+0000 7f597ffff640 10 client.26506 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:00.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:00 smithi077 ceph-mon[31615]: pgmap v224: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 1.76k op/s 2024-04-21T08:52:00.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:00 smithi007 ceph-mon[28477]: pgmap v224: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 1.76k op/s 2024-04-21T08:52:01.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:00 smithi016 ceph-mon[31259]: pgmap v224: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 1.76k op/s 2024-04-21T08:52:02.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:02 smithi077 ceph-mon[31615]: pgmap v225: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 1.55k op/s 2024-04-21T08:52:02.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:02 smithi007 ceph-mon[28477]: pgmap v225: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 1.55k op/s 2024-04-21T08:52:03.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:02 smithi016 ceph-mon[31259]: pgmap v225: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 1.55k op/s 2024-04-21T08:52:04.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:04 smithi077 ceph-mon[31615]: pgmap v226: 129 pgs: 5 active+clean+snaptrim, 117 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 1.49k op/s 2024-04-21T08:52:04.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:04 smithi007 ceph-mon[28477]: pgmap v226: 129 pgs: 5 active+clean+snaptrim, 117 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 1.49k op/s 2024-04-21T08:52:05.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:04 smithi016 ceph-mon[31259]: pgmap v226: 129 pgs: 5 active+clean+snaptrim, 117 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 1.49k op/s 2024-04-21T08:52:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f597ffff640 1 -- 172.21.15.7:0/643267420 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d5cdc3c3d0 con 0x55d5cdd50cd0 2024-04-21T08:52:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f597ffff640 10 client.26506 handle_command_reply: tid=0 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 1 client.26506 shutdown 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 1 -- 172.21.15.7:0/643267420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 msgr2=0x55d5cdd493b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 0x55d5cdd493b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5998009620 tx=0x7f5998004910 comp rx=0 tx=0).stop 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 1 -- 172.21.15.7:0/643267420 shutdown_connections 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d5cdd50cd0 0x55d5cddb9550 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d5cdd410e0 0x55d5cdd3b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d5cdd388d0 0x55d5cdd3b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 1 --2- 172.21.15.7:0/643267420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d5cdd37b60 0x55d5cdd493b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f597ffff640 0 client.26506 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 1 -- 172.21.15.7:0/643267420 >> 172.21.15.7:0/643267420 conn(0x55d5cdcfec90 msgr2=0x55d5cdd3d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 1 -- 172.21.15.7:0/643267420 shutdown_connections 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 1 -- 172.21.15.7:0/643267420 wait complete. 2024-04-21T08:52:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.870+0000 7f59a9bd3740 20 client.26506 trim_cache size 0 max 16384 2024-04-21T08:52:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:52:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:52:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:52:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:52:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3751647506 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59880a3eb0 msgr2=0x7f59880a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59880a3eb0 0x7f59880a6370 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5994009c30 tx=0x7f59940093d0 comp rx=0 tx=0).stop 2024-04-21T08:52:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3751647506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59a0074ca0 msgr2=0x7f59a010b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59a0074ca0 0x7f59a010b8d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f599000ea20 tx=0x7f599000eee0 comp rx=0 tx=0).stop 2024-04-21T08:52:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3751647506 shutdown_connections 2024-04-21T08:52:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59a010e780 0x7f59a010be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59a0074ca0 0x7f59a010b8d0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f59880a3eb0 0x7f59880a6370 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 --2- 172.21.15.7:0/3751647506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59a0074380 0x7f59a010b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3751647506 >> 172.21.15.7:0/3751647506 conn(0x7f59a00fda30 msgr2=0x7f59a00fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3751647506 shutdown_connections 2024-04-21T08:52:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:05.871+0000 7f59a6fe6640 1 -- 172.21.15.7:0/3751647506 wait complete. 2024-04-21T08:52:06.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:06 smithi077 ceph-mon[31615]: pgmap v227: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 1.79k op/s 2024-04-21T08:52:06.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:06 smithi007 ceph-mon[28477]: pgmap v227: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 1.79k op/s 2024-04-21T08:52:07.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:06 smithi016 ceph-mon[31259]: pgmap v227: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 1.79k op/s 2024-04-21T08:52:08.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:08 smithi077 ceph-mon[31615]: pgmap v228: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 1.53k op/s 2024-04-21T08:52:08.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:08 smithi007 ceph-mon[28477]: pgmap v228: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 1.53k op/s 2024-04-21T08:52:09.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:08 smithi016 ceph-mon[31259]: pgmap v228: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 1.53k op/s 2024-04-21T08:52:10.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:10 smithi077 ceph-mon[31615]: pgmap v229: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 1.54k op/s 2024-04-21T08:52:10.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:10 smithi007 ceph-mon[28477]: pgmap v229: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 1.54k op/s 2024-04-21T08:52:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:10 smithi016 ceph-mon[31259]: pgmap v229: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 1.54k op/s 2024-04-21T08:52:12.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:12 smithi077 ceph-mon[31615]: pgmap v230: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 1.50k op/s 2024-04-21T08:52:12.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:12 smithi007 ceph-mon[28477]: pgmap v230: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 1.50k op/s 2024-04-21T08:52:13.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:12 smithi016 ceph-mon[31259]: pgmap v230: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 1.50k op/s 2024-04-21T08:52:13.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:13 smithi077 ceph-mon[31615]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:52:13.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:13 smithi007 ceph-mon[28477]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:52:14.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:13 smithi016 ceph-mon[31259]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:52:14.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:14 smithi077 ceph-mon[31615]: pgmap v232: 129 pgs: 8 active+clean+snaptrim, 98 active+clean, 23 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 1.17k op/s 2024-04-21T08:52:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:14 smithi007 ceph-mon[28477]: pgmap v232: 129 pgs: 8 active+clean+snaptrim, 98 active+clean, 23 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 1.17k op/s 2024-04-21T08:52:15.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:14 smithi016 ceph-mon[31259]: pgmap v232: 129 pgs: 8 active+clean+snaptrim, 98 active+clean, 23 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 1.17k op/s 2024-04-21T08:52:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:16 smithi077 ceph-mon[31615]: pgmap v233: 129 pgs: 16 active+clean+snaptrim, 63 active+clean, 50 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 652 op/s 2024-04-21T08:52:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:16 smithi007 ceph-mon[28477]: pgmap v233: 129 pgs: 16 active+clean+snaptrim, 63 active+clean, 50 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 652 op/s 2024-04-21T08:52:17.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:16 smithi016 ceph-mon[31259]: pgmap v233: 129 pgs: 16 active+clean+snaptrim, 63 active+clean, 50 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 652 op/s 2024-04-21T08:52:18.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:18 smithi007 ceph-mon[28477]: pgmap v234: 129 pgs: 19 active+clean+snaptrim, 54 active+clean, 56 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 622 op/s 2024-04-21T08:52:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.020+0000 7f6f85281640 1 -- 172.21.15.7:0/2475174465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f8011b730 msgr2=0x7f6f8011db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.020+0000 7f6f85281640 1 --2- 172.21.15.7:0/2475174465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f8011b730 0x7f6f8011db20 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009ca0 tx=0x7f6f7802c140 comp rx=0 tx=0).stop 2024-04-21T08:52:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.021+0000 7f6f85281640 1 -- 172.21.15.7:0/2475174465 shutdown_connections 2024-04-21T08:52:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.021+0000 7f6f85281640 1 --2- 172.21.15.7:0/2475174465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f8011b730 0x7f6f8011db20 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.021+0000 7f6f85281640 1 --2- 172.21.15.7:0/2475174465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8010e7d0 0x7f6f8010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.021+0000 7f6f85281640 1 --2- 172.21.15.7:0/2475174465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f801106f0 0x7f6f8010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.021+0000 7f6f85281640 1 -- 172.21.15.7:0/2475174465 >> 172.21.15.7:0/2475174465 conn(0x7f6f8006d030 msgr2=0x7f6f8006d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.024+0000 7f6f85281640 1 -- 172.21.15.7:0/2475174465 shutdown_connections 2024-04-21T08:52:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.025+0000 7f6f85281640 1 -- 172.21.15.7:0/2475174465 wait complete. 2024-04-21T08:52:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.026+0000 7f6f85281640 1 Processor -- start 2024-04-21T08:52:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.029+0000 7f6f85281640 1 -- start start 2024-04-21T08:52:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.029+0000 7f6f85281640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f8010e7d0 0x7f6f80114b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.029+0000 7f6f85281640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f801106f0 0x7f6f801150d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.029+0000 7f6f85281640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f8011b730 0x7f6f80119400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.029+0000 7f6f85281640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6f801113e0 con 0x7f6f8010e7d0 2024-04-21T08:52:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.029+0000 7f6f85281640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f80111260 con 0x7f6f8011b730 2024-04-21T08:52:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.029+0000 7f6f85281640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6f80111560 con 0x7f6f801106f0 2024-04-21T08:52:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.030+0000 7f6f7e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f801106f0 0x7f6f801150d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.030+0000 7f6f7e7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f801106f0 0x7f6f801150d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:49126/0 (socket says 172.21.15.7:49126) 2024-04-21T08:52:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.030+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/1249002391 learned_addr learned my addr 172.21.15.7:0/1249002391 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.030+0000 7f6f7f7fe640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f8011b730 0x7f6f80119400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.030+0000 7f6f7effd640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f8010e7d0 0x7f6f80114b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.031+0000 7f6f7f7fe640 1 -- 172.21.15.7:0/1249002391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f801106f0 msgr2=0x7f6f801150d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.031+0000 7f6f7f7fe640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f801106f0 0x7f6f801150d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.031+0000 7f6f7f7fe640 1 -- 172.21.15.7:0/1249002391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f8010e7d0 msgr2=0x7f6f80114b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.031+0000 7f6f7f7fe640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f8010e7d0 0x7f6f80114b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.031+0000 7f6f7f7fe640 1 -- 172.21.15.7:0/1249002391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f80119b30 con 0x7f6f8011b730 2024-04-21T08:52:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.037+0000 7f6f7f7fe640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f8011b730 0x7f6f80119400 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6f78037040 tx=0x7f6f78035ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.039+0000 7f6f67fff640 1 -- 172.21.15.7:0/1249002391 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f78039040 con 0x7f6f8011b730 2024-04-21T08:52:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.039+0000 7f6f67fff640 1 -- 172.21.15.7:0/1249002391 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f78035e50 con 0x7f6f8011b730 2024-04-21T08:52:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.039+0000 7f6f67fff640 1 -- 172.21.15.7:0/1249002391 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7803f440 con 0x7f6f8011b730 2024-04-21T08:52:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.041+0000 7f6f85281640 1 -- 172.21.15.7:0/1249002391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f80119dc0 con 0x7f6f8011b730 2024-04-21T08:52:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.043+0000 7f6f85281640 1 -- 172.21.15.7:0/1249002391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8011a230 con 0x7f6f8011b730 2024-04-21T08:52:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.049+0000 7f6f87e75740 1 Processor -- start 2024-04-21T08:52:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.050+0000 7f6f87e75740 1 -- start start 2024-04-21T08:52:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.050+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce78620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.050+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce78b60 0x55df8ce78fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.050+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce820e0 0x55df8ce844d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.050+0000 7f6f87e75740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55df8cddd8d0 con 0x55df8ce798d0 2024-04-21T08:52:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.050+0000 7f6f87e75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55df8cd7ae80 con 0x55df8ce78b60 2024-04-21T08:52:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.050+0000 7f6f87e75740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55df8cdd9890 con 0x55df8ce820e0 2024-04-21T08:52:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.053+0000 7f6f67fff640 1 -- 172.21.15.7:0/1249002391 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f6f78046030 con 0x7f6f8011b730 2024-04-21T08:52:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.053+0000 7f6f7e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce78620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.053+0000 7f6f7e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce78620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42468/0 (socket says 172.21.15.7:42468) 2024-04-21T08:52:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.053+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/371916043 learned_addr learned my addr 172.21.15.7:0/371916043 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.054+0000 7f6f67fff640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f680a3f20 0x7f6f680a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.054+0000 7f6f7f7fe640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f680a3f20 0x7f6f680a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:18 smithi016 ceph-mon[31259]: pgmap v234: 129 pgs: 19 active+clean+snaptrim, 54 active+clean, 56 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 622 op/s 2024-04-21T08:52:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.054+0000 7f6f67fff640 1 -- 172.21.15.7:0/1249002391 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9732+0+0 (secure 0 0 0) 0x7f6f780f2890 con 0x7f6f8011b730 2024-04-21T08:52:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.056+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/371916043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce820e0 msgr2=0x55df8ce844d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.056+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/371916043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce820e0 0x55df8ce844d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.056+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/371916043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce78b60 msgr2=0x55df8ce78fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:52:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.056+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/371916043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce78b60 0x55df8ce78fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.056+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/371916043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df8cd9c320 con 0x55df8ce798d0 2024-04-21T08:52:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.056+0000 7f6f7f7fe640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f680a3f20 0x7f6f680a63e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f780c6b60 tx=0x7f6f780e9530 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.056+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/371916043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce78620 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6f7000ad10 tx=0x7f6f70011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720275d8b7059745 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.058+0000 7f6f657fa640 1 -- 172.21.15.7:0/371916043 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f70005220 con 0x55df8ce798d0 2024-04-21T08:52:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.058+0000 7f6f657fa640 1 -- 172.21.15.7:0/371916043 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f70015070 con 0x55df8ce798d0 2024-04-21T08:52:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.059+0000 7f6f87e75740 1 -- 172.21.15.7:0/371916043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 msgr2=0x55df8ce78620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.059+0000 7f6f87e75740 1 --2- 172.21.15.7:0/371916043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce78620 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6f7000ad10 tx=0x7f6f70011810 comp rx=0 tx=0).stop 2024-04-21T08:52:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.062+0000 7f6f87e75740 1 -- 172.21.15.7:0/371916043 shutdown_connections 2024-04-21T08:52:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.062+0000 7f6f87e75740 1 --2- 172.21.15.7:0/371916043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce820e0 0x55df8ce844d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.062+0000 7f6f87e75740 1 --2- 172.21.15.7:0/371916043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce78b60 0x55df8ce78fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.062+0000 7f6f87e75740 1 --2- 172.21.15.7:0/371916043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce78620 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.063+0000 7f6f87e75740 1 -- 172.21.15.7:0/371916043 >> 172.21.15.7:0/371916043 conn(0x55df8ce3fc90 msgr2=0x55df8ce73850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.069+0000 7f6f87e75740 1 -- 172.21.15.7:0/371916043 shutdown_connections 2024-04-21T08:52:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.072+0000 7f6f87e75740 1 -- 172.21.15.7:0/371916043 wait complete. 2024-04-21T08:52:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.072+0000 7f6f87e75740 1 -- start start 2024-04-21T08:52:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.072+0000 7f6f87e75740 1 Processor -- start 2024-04-21T08:52:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.072+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce78b60 0x55df8ce81a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.072+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce7ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.073+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce820e0 0x55df8ce7f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.073+0000 7f6f87e75740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55df8ce84e60 con 0x55df8ce798d0 2024-04-21T08:52:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.073+0000 7f6f87e75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55df8cd65eb0 con 0x55df8ce78b60 2024-04-21T08:52:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.073+0000 7f6f87e75740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55df8ce84fe0 con 0x55df8ce820e0 2024-04-21T08:52:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.073+0000 7f6f87e75740 10 client.0 _ll_register_callbacks cb 0x55df8cd92b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.074+0000 7f6f7effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce7ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.074+0000 7f6f7effd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce7ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42480/0 (socket says 172.21.15.7:42480) 2024-04-21T08:52:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.074+0000 7f6f7effd640 1 -- 172.21.15.7:0/3602720895 learned_addr learned my addr 172.21.15.7:0/3602720895 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.075+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce78b60 0x55df8ce81a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.077+0000 7f6f7effd640 1 -- 172.21.15.7:0/3602720895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce820e0 msgr2=0x55df8ce7f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.077+0000 7f6f7effd640 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce820e0 0x55df8ce7f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.077+0000 7f6f7effd640 1 -- 172.21.15.7:0/3602720895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce78b60 msgr2=0x55df8ce81a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.077+0000 7f6f7effd640 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce78b60 0x55df8ce81a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.078+0000 7f6f7effd640 1 -- 172.21.15.7:0/3602720895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df8caf0210 con 0x55df8ce798d0 2024-04-21T08:52:19.089 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.090+0000 7f6f7effd640 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce7ed90 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400d1e0 tx=0x7f6f7400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c55b3a6ebb6bec4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.091+0000 7f6f62ff5640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:52:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.091+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3602720895 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7400de60 con 0x55df8ce798d0 2024-04-21T08:52:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.092+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3602720895 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9732+0+0 (secure 0 0 0) 0x7f6f740125f0 con 0x55df8ce798d0 2024-04-21T08:52:19.092 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.092+0000 7f6f87e75740 1 -- 172.21.15.7:0/3602720895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f70015070 con 0x55df8ce798d0 2024-04-21T08:52:19.092 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.093+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3602720895 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6f740127f0 con 0x55df8ce798d0 2024-04-21T08:52:19.094 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.094+0000 7f6f87e75740 10 client.16983 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:19.094 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.094+0000 7f6f87e75740 1 -- 172.21.15.7:0/3602720895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55df8cbbc620 con 0x55df8ce798d0 2024-04-21T08:52:19.095 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.096+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3602720895 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f70015070 con 0x55df8ce798d0 2024-04-21T08:52:19.095 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.096+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3602720895 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6f74013ce0 con 0x55df8ce798d0 2024-04-21T08:52:19.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:18 smithi077 ceph-mon[31615]: pgmap v234: 129 pgs: 19 active+clean+snaptrim, 54 active+clean, 56 active+clean+snaptrim_wait; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 622 op/s 2024-04-21T08:52:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.098+0000 7f6f87e75740 10 client.16983 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.098+0000 7f6f87e75740 10 client.16983 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.098+0000 7f6f87e75740 20 client.16983 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.098+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55df8ce921e0 0x55df8ce94650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.098+0000 7f6f87e75740 4 client.16983 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.098+0000 7f6f87e75740 1 -- 172.21.15.7:0/3602720895 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df8cdf1880 con 0x55df8ce921e0 2024-04-21T08:52:19.105 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.101+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55df8ce921e0 0x55df8ce94650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.116+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55df8ce921e0 0x55df8ce94650 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0f77c7aec400155 server_cookie=e551cb81a4cbd54b in_seq=0 out_seq=0 2024-04-21T08:52:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.118+0000 7f6f62ff5640 10 client.16983 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.121+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3602720895 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df8cdf1880 con 0x55df8ce921e0 2024-04-21T08:52:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.121+0000 7f6f62ff5640 10 client.16983 handle_command_reply: tid=0 2024-04-21T08:52:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.121+0000 7f6f87e75740 1 client.16983 shutdown 2024-04-21T08:52:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.125+0000 7f6f87e75740 1 -- 172.21.15.7:0/3602720895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 msgr2=0x55df8ce7ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.125+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce7ed90 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400d1e0 tx=0x7f6f7400bf90 comp rx=0 tx=0).stop 2024-04-21T08:52:19.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.127+0000 7f6f87e75740 1 -- 172.21.15.7:0/3602720895 shutdown_connections 2024-04-21T08:52:19.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.127+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55df8ce921e0 0x55df8ce94650 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.127+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce820e0 0x55df8ce7f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.127+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce7ed90 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.127+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3602720895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce78b60 0x55df8ce81a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.127+0000 7f6f87e75740 1 -- 172.21.15.7:0/3602720895 >> 172.21.15.7:0/3602720895 conn(0x55df8ce3fc90 msgr2=0x55df8ce755a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.127+0000 7f6f62ff5640 0 client.16983 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.134+0000 7f6f87e75740 1 -- 172.21.15.7:0/3602720895 shutdown_connections 2024-04-21T08:52:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.135+0000 7f6f87e75740 1 -- 172.21.15.7:0/3602720895 wait complete. 2024-04-21T08:52:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.135+0000 7f6f87e75740 20 client.16983 trim_cache size 0 max 16384 2024-04-21T08:52:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.165+0000 7f6f87e75740 1 Processor -- start 2024-04-21T08:52:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.166+0000 7f6f87e75740 1 -- start start 2024-04-21T08:52:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.166+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce3fc90 0x55df8ce8dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.166+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce78b60 0x55df8ce73380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.166+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.166+0000 7f6f87e75740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6f801c4c10 con 0x55df8ce798d0 2024-04-21T08:52:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.166+0000 7f6f87e75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f801c7470 con 0x55df8ce3fc90 2024-04-21T08:52:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.166+0000 7f6f87e75740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6f801c5260 con 0x55df8ce78b60 2024-04-21T08:52:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.167+0000 7f6f7e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.167+0000 7f6f7e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce814c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42482/0 (socket says 172.21.15.7:42482) 2024-04-21T08:52:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.167+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/668240309 learned_addr learned my addr 172.21.15.7:0/668240309 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.167+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/668240309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce3fc90 0x55df8ce8dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.167+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/668240309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce78b60 msgr2=0x55df8ce73380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:52:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.167+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/668240309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce78b60 0x55df8ce73380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.167+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/668240309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce3fc90 msgr2=0x55df8ce8dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.167+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/668240309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce3fc90 0x55df8ce8dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.167+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/668240309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df8ce90d80 con 0x55df8ce798d0 2024-04-21T08:52:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.168+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/668240309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce814c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6f70011d10 tx=0x7f6f70013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5bc6295eeea08f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.169+0000 7f6f64ff9640 1 -- 172.21.15.7:0/668240309 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f70030070 con 0x55df8ce798d0 2024-04-21T08:52:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.169+0000 7f6f64ff9640 1 -- 172.21.15.7:0/668240309 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f70015070 con 0x55df8ce798d0 2024-04-21T08:52:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.169+0000 7f6f64ff9640 1 -- 172.21.15.7:0/668240309 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7002ae60 con 0x55df8ce798d0 2024-04-21T08:52:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.170+0000 7f6f87e75740 1 -- 172.21.15.7:0/668240309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 msgr2=0x55df8ce814c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.170+0000 7f6f87e75740 1 --2- 172.21.15.7:0/668240309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce814c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6f70011d10 tx=0x7f6f70013040 comp rx=0 tx=0).stop 2024-04-21T08:52:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.171+0000 7f6f87e75740 1 -- 172.21.15.7:0/668240309 shutdown_connections 2024-04-21T08:52:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.171+0000 7f6f87e75740 1 --2- 172.21.15.7:0/668240309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce798d0 0x55df8ce814c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.171+0000 7f6f87e75740 1 --2- 172.21.15.7:0/668240309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce78b60 0x55df8ce73380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.171+0000 7f6f87e75740 1 --2- 172.21.15.7:0/668240309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce3fc90 0x55df8ce8dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.171+0000 7f6f87e75740 1 -- 172.21.15.7:0/668240309 >> 172.21.15.7:0/668240309 conn(0x55df8ce7f430 msgr2=0x55df8ce7d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:19.176 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.177+0000 7f6f87e75740 1 -- 172.21.15.7:0/668240309 shutdown_connections 2024-04-21T08:52:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.181+0000 7f6f87e75740 1 -- 172.21.15.7:0/668240309 wait complete. 2024-04-21T08:52:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.181+0000 7f6f87e75740 1 -- start start 2024-04-21T08:52:19.181 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.181+0000 7f6f87e75740 1 Processor -- start 2024-04-21T08:52:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.182+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce3fc90 0x55df8ce8a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.182+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce78b60 0x55df8ce7b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.182+0000 7f6f87e75740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce798d0 0x55df8ce7bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.182+0000 7f6f87e75740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6f801c5930 con 0x55df8ce3fc90 2024-04-21T08:52:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.182+0000 7f6f87e75740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f801c6da0 con 0x55df8ce798d0 2024-04-21T08:52:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.182+0000 7f6f87e75740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6f801c6000 con 0x55df8ce78b60 2024-04-21T08:52:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.182+0000 7f6f87e75740 10 client.0 _ll_register_callbacks cb 0x7f6f7400b1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.183+0000 7f6f7e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce3fc90 0x55df8ce8a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.183+0000 7f6f7e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce3fc90 0x55df8ce8a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42488/0 (socket says 172.21.15.7:42488) 2024-04-21T08:52:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.183+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/3118192718 learned_addr learned my addr 172.21.15.7:0/3118192718 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.183+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce798d0 0x55df8ce7bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.183+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/3118192718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce78b60 msgr2=0x55df8ce7b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.183+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce78b60 0x55df8ce7b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.183+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/3118192718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce798d0 msgr2=0x55df8ce7bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.183+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce798d0 0x55df8ce7bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.183+0000 7f6f7e7fc640 1 -- 172.21.15.7:0/3118192718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df8ce77e30 con 0x55df8ce3fc90 2024-04-21T08:52:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.184+0000 7f6f7e7fc640 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce3fc90 0x55df8ce8a7c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6f7000a960 tx=0x7f6f7000a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68b1d51f43a3823 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.184+0000 7f6f62ff5640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:52:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.185+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3118192718 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f70030040 con 0x55df8ce3fc90 2024-04-21T08:52:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.185+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3118192718 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9732+0+0 (secure 0 0 0) 0x7f6f7000fd40 con 0x55df8ce3fc90 2024-04-21T08:52:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.188+0000 7f6f87e75740 1 -- 172.21.15.7:0/3118192718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f70015070 con 0x55df8ce3fc90 2024-04-21T08:52:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.189+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3118192718 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6f7003c050 con 0x55df8ce3fc90 2024-04-21T08:52:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.191+0000 7f6f87e75740 10 client.16989 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.191+0000 7f6f87e75740 1 -- 172.21.15.7:0/3118192718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55df8cef8710 con 0x55df8ce3fc90 2024-04-21T08:52:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.192+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3118192718 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f70015070 con 0x55df8ce3fc90 2024-04-21T08:52:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.193+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3118192718 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6f70004c70 con 0x55df8ce3fc90 2024-04-21T08:52:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.197+0000 7f6f87e75740 10 client.16989 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.197+0000 7f6f87e75740 10 client.16989 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.197+0000 7f6f87e75740 20 client.16989 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.197+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55df8ce820e0 0x55df8cefac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.197+0000 7f6f87e75740 4 client.16989 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:52:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.197+0000 7f6f87e75740 1 -- 172.21.15.7:0/3118192718 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55df8cd7d3d0 con 0x55df8ce820e0 2024-04-21T08:52:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.204+0000 7f6f7effd640 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55df8ce820e0 0x55df8cefac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.215+0000 7f6f7effd640 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55df8ce820e0 0x55df8cefac40 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d24700489dae96a3 server_cookie=a1dba2ac2dbdbcef in_seq=0 out_seq=0 2024-04-21T08:52:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:19.215+0000 7f6f62ff5640 10 client.16989 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:19.566 INFO:tasks.workunit.client.0.smithi007.stderr:+ cd linux-6.5.11 2024-04-21T08:52:19.591 INFO:tasks.workunit.client.0.smithi007.stderr:+ make defconfig 2024-04-21T08:52:20.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:20 smithi007 ceph-mon[28477]: pgmap v235: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 578 op/s 2024-04-21T08:52:21.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:20 smithi016 ceph-mon[31259]: pgmap v235: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 578 op/s 2024-04-21T08:52:21.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:20 smithi077 ceph-mon[31615]: pgmap v235: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 578 op/s 2024-04-21T08:52:22.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:22 smithi007 ceph-mon[28477]: pgmap v236: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 465 op/s 2024-04-21T08:52:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:22 smithi016 ceph-mon[31259]: pgmap v236: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 465 op/s 2024-04-21T08:52:23.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:22 smithi077 ceph-mon[31615]: pgmap v236: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.6 MiB/s wr, 465 op/s 2024-04-21T08:52:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.907+0000 7f6f62ff5640 1 -- 172.21.15.7:0/3118192718 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55df8cd7d3d0 con 0x55df8ce820e0 2024-04-21T08:52:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.907+0000 7f6f62ff5640 10 client.16989 handle_command_reply: tid=0 2024-04-21T08:52:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.907+0000 7f6f87e75740 1 client.16989 shutdown 2024-04-21T08:52:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.908+0000 7f6f87e75740 1 -- 172.21.15.7:0/3118192718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce3fc90 msgr2=0x55df8ce8a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.908+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce3fc90 0x55df8ce8a7c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6f7000a960 tx=0x7f6f7000a8f0 comp rx=0 tx=0).stop 2024-04-21T08:52:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.909+0000 7f6f87e75740 1 -- 172.21.15.7:0/3118192718 shutdown_connections 2024-04-21T08:52:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.909+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55df8ce820e0 0x55df8cefac40 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.909+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55df8ce798d0 0x55df8ce7bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.909+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55df8ce78b60 0x55df8ce7b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.909+0000 7f6f87e75740 1 --2- 172.21.15.7:0/3118192718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55df8ce3fc90 0x55df8ce8a7c0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.909+0000 7f6f87e75740 1 -- 172.21.15.7:0/3118192718 >> 172.21.15.7:0/3118192718 conn(0x55df8ce7f430 msgr2=0x55df8ce72040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.909+0000 7f6f62ff5640 0 client.16989 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.909+0000 7f6f87e75740 1 -- 172.21.15.7:0/3118192718 shutdown_connections 2024-04-21T08:52:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.909+0000 7f6f87e75740 1 -- 172.21.15.7:0/3118192718 wait complete. 2024-04-21T08:52:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.910+0000 7f6f87e75740 20 client.16989 trim_cache size 0 max 16384 2024-04-21T08:52:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:52:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:52:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:52:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:52:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.912+0000 7f6f85281640 1 -- 172.21.15.7:0/1249002391 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f680a3f20 msgr2=0x7f6f680a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.912+0000 7f6f85281640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f680a3f20 0x7f6f680a63e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f780c6b60 tx=0x7f6f780e9530 comp rx=0 tx=0).stop 2024-04-21T08:52:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.912+0000 7f6f85281640 1 -- 172.21.15.7:0/1249002391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f8011b730 msgr2=0x7f6f80119400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.912+0000 7f6f85281640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f8011b730 0x7f6f80119400 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6f78037040 tx=0x7f6f78035ce0 comp rx=0 tx=0).stop 2024-04-21T08:52:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.912+0000 7f6f85281640 1 -- 172.21.15.7:0/1249002391 shutdown_connections 2024-04-21T08:52:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.912+0000 7f6f85281640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f8011b730 0x7f6f80119400 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.912+0000 7f6f85281640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f680a3f20 0x7f6f680a63e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.912+0000 7f6f85281640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f801106f0 0x7f6f801150d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.912+0000 7f6f85281640 1 --2- 172.21.15.7:0/1249002391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f8010e7d0 0x7f6f80114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.912+0000 7f6f85281640 1 -- 172.21.15.7:0/1249002391 >> 172.21.15.7:0/1249002391 conn(0x7f6f8006d030 msgr2=0x7f6f8011c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.915+0000 7f6f85281640 1 -- 172.21.15.7:0/1249002391 shutdown_connections 2024-04-21T08:52:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:23.915+0000 7f6f85281640 1 -- 172.21.15.7:0/1249002391 wait complete. 2024-04-21T08:52:24.736 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/basic/fixdep 2024-04-21T08:52:24.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:24 smithi007 ceph-mon[28477]: pgmap v237: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 862 op/s 2024-04-21T08:52:25.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:24 smithi016 ceph-mon[31259]: pgmap v237: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 862 op/s 2024-04-21T08:52:25.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:24 smithi077 ceph-mon[31615]: pgmap v237: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 862 op/s 2024-04-21T08:52:26.066 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/kconfig/conf.o 2024-04-21T08:52:26.463 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/kconfig/confdata.o 2024-04-21T08:52:26.895 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/kconfig/expr.o 2024-04-21T08:52:26.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:26 smithi007 ceph-mon[28477]: pgmap v238: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 958 op/s 2024-04-21T08:52:26.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:26 smithi007 ceph-mon[28477]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T08:52:26.941 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:52:26.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:52:27.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:26 smithi016 ceph-mon[31259]: pgmap v238: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 958 op/s 2024-04-21T08:52:27.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:26 smithi016 ceph-mon[31259]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T08:52:27.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:26 smithi077 ceph-mon[31615]: pgmap v238: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 958 op/s 2024-04-21T08:52:27.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:26 smithi077 ceph-mon[31615]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T08:52:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.178+0000 7faefb0b3640 1 -- 172.21.15.7:0/877840002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4105f80 msgr2=0x7faef410db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.178+0000 7faefb0b3640 1 --2- 172.21.15.7:0/877840002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4105f80 0x7faef410db50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faeec009ca0 tx=0x7faeec02c140 comp rx=0 tx=0).stop 2024-04-21T08:52:27.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.179+0000 7faefb0b3640 1 -- 172.21.15.7:0/877840002 shutdown_connections 2024-04-21T08:52:27.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.179+0000 7faefb0b3640 1 --2- 172.21.15.7:0/877840002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4105f80 0x7faef410db50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.179+0000 7faefb0b3640 1 --2- 172.21.15.7:0/877840002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faef4101860 0x7faef41058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.179+0000 7faefb0b3640 1 --2- 172.21.15.7:0/877840002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faef4100eb0 0x7faef4101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.179+0000 7faefb0b3640 1 -- 172.21.15.7:0/877840002 >> 172.21.15.7:0/877840002 conn(0x7faef4077a00 msgr2=0x7faef40fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.188+0000 7faefb0b3640 1 -- 172.21.15.7:0/877840002 shutdown_connections 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.193+0000 7faefb0b3640 1 -- 172.21.15.7:0/877840002 wait complete. 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.193+0000 7faefb0b3640 1 Processor -- start 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.195+0000 7faefb0b3640 1 -- start start 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.195+0000 7faefb0b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4100eb0 0x7faef419e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.195+0000 7faefb0b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faef4105f80 0x7faef4196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.195+0000 7faefb0b3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faef41969d0 0x7faef4196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.195+0000 7faefb0b3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7faef4110fc0 con 0x7faef41969d0 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.195+0000 7faefb0b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faef4110e40 con 0x7faef4100eb0 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.195+0000 7faefb0b3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faef4111140 con 0x7faef4105f80 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.196+0000 7faef98b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faef4105f80 0x7faef4196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.196+0000 7faef98b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faef4105f80 0x7faef4196490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:56312/0 (socket says 172.21.15.7:56312) 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.196+0000 7faef98b0640 1 -- 172.21.15.7:0/28489978 learned_addr learned my addr 172.21.15.7:0/28489978 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.196+0000 7faefa8b2640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faef41969d0 0x7faef4196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.197+0000 7faefa8b2640 1 -- 172.21.15.7:0/28489978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faef4105f80 msgr2=0x7faef4196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.197+0000 7faefa8b2640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faef4105f80 0x7faef4196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.197+0000 7faefa8b2640 1 -- 172.21.15.7:0/28489978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4100eb0 msgr2=0x7faef419e300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:52:27.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.197+0000 7faefa8b2640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4100eb0 0x7faef419e300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.197+0000 7faefa8b2640 1 -- 172.21.15.7:0/28489978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef406b970 con 0x7faef41969d0 2024-04-21T08:52:27.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.199+0000 7faefa8b2640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faef41969d0 0x7faef4196e80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7faeec02c750 tx=0x7faeec035d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.200+0000 7faeeb7fe640 1 -- 172.21.15.7:0/28489978 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec039040 con 0x7faef41969d0 2024-04-21T08:52:27.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.200+0000 7faeeb7fe640 1 -- 172.21.15.7:0/28489978 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeec032a00 con 0x7faef41969d0 2024-04-21T08:52:27.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.200+0000 7faeeb7fe640 1 -- 172.21.15.7:0/28489978 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec004310 con 0x7faef41969d0 2024-04-21T08:52:27.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.200+0000 7faefb0b3640 1 -- 172.21.15.7:0/28489978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef406bc00 con 0x7faef41969d0 2024-04-21T08:52:27.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.201+0000 7faefb0b3640 1 -- 172.21.15.7:0/28489978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef406c0a0 con 0x7faef41969d0 2024-04-21T08:52:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.206+0000 7faefefe5740 1 Processor -- start 2024-04-21T08:52:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.207+0000 7faeeb7fe640 1 -- 172.21.15.7:0/28489978 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7faeec032550 con 0x7faef41969d0 2024-04-21T08:52:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.207+0000 7faefefe5740 1 -- start start 2024-04-21T08:52:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.210+0000 7faeeb7fe640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7faee00a3ea0 0x7faee00a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.210+0000 7faeeb7fe640 1 -- 172.21.15.7:0/28489978 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9672+0+0 (secure 0 0 0) 0x7faeec0f2860 con 0x7faef41969d0 2024-04-21T08:52:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.210+0000 7faefefe5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaebae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.210+0000 7faefefe5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaec020 0x5604edaf46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.210+0000 7faef98b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaec020 0x5604edaf46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.212+0000 7faefefe5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaf4c00 0x5604edaf6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.212+0000 7faefefe5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5604eda501e0 con 0x5604edaf4c00 2024-04-21T08:52:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.212+0000 7faefefe5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5604ed9ed390 con 0x5604edaeb700 2024-04-21T08:52:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.212+0000 7faefefe5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5604eda4c1a0 con 0x5604edaec020 2024-04-21T08:52:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.212+0000 7faef98b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaec020 0x5604edaf46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:56328/0 (socket says 172.21.15.7:56328) 2024-04-21T08:52:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.212+0000 7faef98b0640 1 -- 172.21.15.7:0/442583849 learned_addr learned my addr 172.21.15.7:0/442583849 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.212+0000 7faef98b0640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7faee00a3ea0 0x7faee00a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.210+0000 7faefa0b1640 1 --2- 172.21.15.7:0/442583849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaebae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.214+0000 7faefa0b1640 1 -- 172.21.15.7:0/442583849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaec020 msgr2=0x5604edaf46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.214+0000 7faefa0b1640 1 --2- 172.21.15.7:0/442583849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaec020 0x5604edaf46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.214+0000 7faefa0b1640 1 -- 172.21.15.7:0/442583849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaf4c00 msgr2=0x5604edaf6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.214+0000 7faefa0b1640 1 --2- 172.21.15.7:0/442583849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaf4c00 0x5604edaf6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.214+0000 7faefa0b1640 1 -- 172.21.15.7:0/442583849 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604ed82c7d0 con 0x5604edaeb700 2024-04-21T08:52:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.215+0000 7faef98b0640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7faee00a3ea0 0x7faee00a6360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faee4006fd0 tx=0x7faee400a5a0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.215+0000 7faefa0b1640 1 --2- 172.21.15.7:0/442583849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaebae0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7faef0009660 tx=0x7faef0009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e82798495e63205e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.218+0000 7faee8bfe640 1 -- 172.21.15.7:0/442583849 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0015020 con 0x5604edaeb700 2024-04-21T08:52:27.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.218+0000 7faee8bfe640 1 -- 172.21.15.7:0/442583849 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef000bd70 con 0x5604edaeb700 2024-04-21T08:52:27.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.219+0000 7faefefe5740 1 -- 172.21.15.7:0/442583849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 msgr2=0x5604edaebae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.219+0000 7faefefe5740 1 --2- 172.21.15.7:0/442583849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaebae0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7faef0009660 tx=0x7faef0009b20 comp rx=0 tx=0).stop 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.219+0000 7faefefe5740 1 -- 172.21.15.7:0/442583849 shutdown_connections 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.219+0000 7faefefe5740 1 --2- 172.21.15.7:0/442583849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaf4c00 0x5604edaf6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.219+0000 7faefefe5740 1 --2- 172.21.15.7:0/442583849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaec020 0x5604edaf46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.219+0000 7faefefe5740 1 --2- 172.21.15.7:0/442583849 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaebae0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.219+0000 7faefefe5740 1 -- 172.21.15.7:0/442583849 >> 172.21.15.7:0/442583849 conn(0x5604edae5e30 msgr2=0x5604edae6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.221+0000 7faefefe5740 1 -- 172.21.15.7:0/442583849 shutdown_connections 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.221+0000 7faefefe5740 1 -- 172.21.15.7:0/442583849 wait complete. 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.221+0000 7faefefe5740 1 -- start start 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.221+0000 7faefefe5740 1 Processor -- start 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.222+0000 7faefefe5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaf1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.222+0000 7faefefe5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 0x5604edaf20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.222+0000 7faefefe5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaf4c00 0x5604edaf25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.222+0000 7faefefe5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5604edaf7a90 con 0x5604edaec020 2024-04-21T08:52:27.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.222+0000 7faefefe5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5604eda092f0 con 0x5604edaeb700 2024-04-21T08:52:27.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.222+0000 7faefefe5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5604edaf7c10 con 0x5604edaf4c00 2024-04-21T08:52:27.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.222+0000 7faefefe5740 10 client.0 _ll_register_callbacks cb 0x5604eda16580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:27.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.222+0000 7faef98b0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 0x5604edaf20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.222+0000 7faef98b0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 0x5604edaf20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43586/0 (socket says 172.21.15.7:43586) 2024-04-21T08:52:27.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.222+0000 7faef98b0640 1 -- 172.21.15.7:0/667335475 learned_addr learned my addr 172.21.15.7:0/667335475 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:27.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.223+0000 7faef98b0640 1 -- 172.21.15.7:0/667335475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaf4c00 msgr2=0x5604edaf25e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:27.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.223+0000 7faef98b0640 1 --2- 172.21.15.7:0/667335475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaf4c00 0x5604edaf25e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.223+0000 7faef98b0640 1 -- 172.21.15.7:0/667335475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 msgr2=0x5604edaf1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.223+0000 7faef98b0640 1 --2- 172.21.15.7:0/667335475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaf1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.223+0000 7faef98b0640 1 -- 172.21.15.7:0/667335475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604ed7623f0 con 0x5604edaec020 2024-04-21T08:52:27.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.223+0000 7faef98b0640 1 --2- 172.21.15.7:0/667335475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 0x5604edaf20a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faee4017bb0 tx=0x7faee400a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d33d363b68313 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.225+0000 7faeddffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:52:27.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.225+0000 7faeddffb640 1 -- 172.21.15.7:0/667335475 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4025060 con 0x5604edaec020 2024-04-21T08:52:27.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.225+0000 7faeddffb640 1 -- 172.21.15.7:0/667335475 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9672+0+0 (secure 0 0 0) 0x7faee400f070 con 0x5604edaec020 2024-04-21T08:52:27.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.226+0000 7faefefe5740 1 -- 172.21.15.7:0/667335475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faef000bd70 con 0x5604edaec020 2024-04-21T08:52:27.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.234+0000 7faeddffb640 1 -- 172.21.15.7:0/667335475 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7faee4016ba0 con 0x5604edaec020 2024-04-21T08:52:27.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.235+0000 7faefefe5740 10 client.16995 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:27.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.235+0000 7faefefe5740 1 -- 172.21.15.7:0/667335475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5604ed7220f0 con 0x5604edaec020 2024-04-21T08:52:27.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.237+0000 7faeddffb640 1 -- 172.21.15.7:0/667335475 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef000bd70 con 0x5604edaec020 2024-04-21T08:52:27.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.237+0000 7faeddffb640 1 -- 172.21.15.7:0/667335475 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7faee4015040 con 0x5604edaec020 2024-04-21T08:52:27.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.240+0000 7faefefe5740 10 client.16995 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:27.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.240+0000 7faefefe5740 10 client.16995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:27.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.240+0000 7faefefe5740 20 client.16995 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:27.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.240+0000 7faefefe5740 1 --2- 172.21.15.7:0/667335475 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5604edb04cd0 0x5604edb070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.240+0000 7faefefe5740 4 client.16995 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:27.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.240+0000 7faefefe5740 1 -- 172.21.15.7:0/667335475 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604eda5dc00 con 0x5604edb04cd0 2024-04-21T08:52:27.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.241+0000 7faefa0b1640 1 --2- 172.21.15.7:0/667335475 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5604edb04cd0 0x5604edb070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.251+0000 7faefa0b1640 1 --2- 172.21.15.7:0/667335475 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5604edb04cd0 0x5604edb070e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a36d2cdc1b86406b server_cookie=2cbff9edac63785a in_seq=0 out_seq=0 2024-04-21T08:52:27.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.251+0000 7faeddffb640 10 client.16995 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.256+0000 7faeddffb640 1 -- 172.21.15.7:0/667335475 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604eda5dc00 con 0x5604edb04cd0 2024-04-21T08:52:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.256+0000 7faeddffb640 10 client.16995 handle_command_reply: tid=0 2024-04-21T08:52:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.258+0000 7faefefe5740 1 client.16995 shutdown 2024-04-21T08:52:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.260+0000 7faefefe5740 1 -- 172.21.15.7:0/667335475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 msgr2=0x5604edaf20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.260+0000 7faefefe5740 1 --2- 172.21.15.7:0/667335475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 0x5604edaf20a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faee4017bb0 tx=0x7faee400a330 comp rx=0 tx=0).stop 2024-04-21T08:52:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.261+0000 7faefefe5740 1 -- 172.21.15.7:0/667335475 shutdown_connections 2024-04-21T08:52:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.261+0000 7faefefe5740 1 --2- 172.21.15.7:0/667335475 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5604edb04cd0 0x5604edb070e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.261+0000 7faefefe5740 1 --2- 172.21.15.7:0/667335475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaf4c00 0x5604edaf25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.261+0000 7faefefe5740 1 --2- 172.21.15.7:0/667335475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 0x5604edaf20a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.261+0000 7faefefe5740 1 --2- 172.21.15.7:0/667335475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaf1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.261+0000 7faefefe5740 1 -- 172.21.15.7:0/667335475 >> 172.21.15.7:0/667335475 conn(0x5604edae5e30 msgr2=0x5604edae7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.261+0000 7faeddffb640 0 client.16995 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:27.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.263+0000 7faefefe5740 1 -- 172.21.15.7:0/667335475 shutdown_connections 2024-04-21T08:52:27.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.263+0000 7faefefe5740 1 -- 172.21.15.7:0/667335475 wait complete. 2024-04-21T08:52:27.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.265+0000 7faefefe5740 20 client.16995 trim_cache size 0 max 16384 2024-04-21T08:52:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.304+0000 7faefefe5740 1 Processor -- start 2024-04-21T08:52:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.308+0000 7faefefe5740 1 -- start start 2024-04-21T08:52:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.308+0000 7faefefe5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edae5e30 0x5604edaf4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.308+0000 7faefefe5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaf2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.308+0000 7faefefe5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 0x5604edae7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.308+0000 7faefefe5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7faed800e2f0 con 0x5604edaec020 2024-04-21T08:52:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.308+0000 7faefefe5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faed8010b70 con 0x5604edaeb700 2024-04-21T08:52:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.308+0000 7faefefe5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faed800e960 con 0x5604edae5e30 2024-04-21T08:52:27.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.309+0000 7faefa0b1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edae5e30 0x5604edaf4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.309+0000 7faefa0b1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edae5e30 0x5604edaf4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:56332/0 (socket says 172.21.15.7:56332) 2024-04-21T08:52:27.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.309+0000 7faefa0b1640 1 -- 172.21.15.7:0/2192744088 learned_addr learned my addr 172.21.15.7:0/2192744088 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:27.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.309+0000 7faef98b0640 1 --2- 172.21.15.7:0/2192744088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaf2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.309+0000 7faefa8b2640 1 --2- 172.21.15.7:0/2192744088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 0x5604edae7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.311+0000 7faefa0b1640 1 -- 172.21.15.7:0/2192744088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 msgr2=0x5604edaf2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.311+0000 7faefa0b1640 1 --2- 172.21.15.7:0/2192744088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaf2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.311+0000 7faefa0b1640 1 -- 172.21.15.7:0/2192744088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 msgr2=0x5604edae7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.311+0000 7faefa0b1640 1 --2- 172.21.15.7:0/2192744088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 0x5604edae7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.311+0000 7faefa0b1640 1 -- 172.21.15.7:0/2192744088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604edaf00b0 con 0x5604edae5e30 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.316+0000 7faefa0b1640 1 --2- 172.21.15.7:0/2192744088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edae5e30 0x5604edaf4290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faef00119f0 tx=0x7faef00090e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90382e76f006b2e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.317+0000 7faedffff640 1 -- 172.21.15.7:0/2192744088 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0015040 con 0x5604edae5e30 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.317+0000 7faedffff640 1 -- 172.21.15.7:0/2192744088 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef00092f0 con 0x5604edae5e30 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.317+0000 7faedffff640 1 -- 172.21.15.7:0/2192744088 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0022170 con 0x5604edae5e30 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.318+0000 7faefefe5740 1 -- 172.21.15.7:0/2192744088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edae5e30 msgr2=0x5604edaf4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.318+0000 7faefefe5740 1 --2- 172.21.15.7:0/2192744088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edae5e30 0x5604edaf4290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faef00119f0 tx=0x7faef00090e0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.321+0000 7faefefe5740 1 -- 172.21.15.7:0/2192744088 shutdown_connections 2024-04-21T08:52:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.321+0000 7faefefe5740 1 --2- 172.21.15.7:0/2192744088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaec020 0x5604edae7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.321+0000 7faefefe5740 1 --2- 172.21.15.7:0/2192744088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edaeb700 0x5604edaf2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.321+0000 7faefefe5740 1 --2- 172.21.15.7:0/2192744088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edae5e30 0x5604edaf4290 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.321+0000 7faefefe5740 1 -- 172.21.15.7:0/2192744088 >> 172.21.15.7:0/2192744088 conn(0x5604edaf07a0 msgr2=0x5604edae7280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.324+0000 7faefefe5740 1 -- 172.21.15.7:0/2192744088 shutdown_connections 2024-04-21T08:52:27.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.325+0000 7faefefe5740 1 -- 172.21.15.7:0/2192744088 wait complete. 2024-04-21T08:52:27.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.326+0000 7faefefe5740 1 -- start start 2024-04-21T08:52:27.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.326+0000 7faefefe5740 1 Processor -- start 2024-04-21T08:52:27.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.326+0000 7faefefe5740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edae5e30 0x5604edafd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.326+0000 7faefefe5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaeb700 0x5604edaf9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.326+0000 7faefefe5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaec020 0x5604edaf9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.326+0000 7faefefe5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7faed800f030 con 0x5604edaeb700 2024-04-21T08:52:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.326+0000 7faefefe5740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faed80104a0 con 0x5604edae5e30 2024-04-21T08:52:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.326+0000 7faefefe5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faed800f700 con 0x5604edaec020 2024-04-21T08:52:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.326+0000 7faefefe5740 10 client.0 _ll_register_callbacks cb 0x7faee4017850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.327+0000 7faef98b0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaeb700 0x5604edaf9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.327+0000 7faef98b0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaeb700 0x5604edaf9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43610/0 (socket says 172.21.15.7:43610) 2024-04-21T08:52:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.327+0000 7faef98b0640 1 -- 172.21.15.7:0/3325251891 learned_addr learned my addr 172.21.15.7:0/3325251891 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.327+0000 7faefa0b1640 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edae5e30 0x5604edafd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.327+0000 7faef98b0640 1 -- 172.21.15.7:0/3325251891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaec020 msgr2=0x5604edaf9940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.327+0000 7faef98b0640 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaec020 0x5604edaf9940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.327+0000 7faef98b0640 1 -- 172.21.15.7:0/3325251891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edae5e30 msgr2=0x5604edafd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.328+0000 7faef98b0640 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edae5e30 0x5604edafd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.328+0000 7faef98b0640 1 -- 172.21.15.7:0/3325251891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604edaf9eb0 con 0x5604edaeb700 2024-04-21T08:52:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.328+0000 7faef98b0640 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaeb700 0x5604edaf9400 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7faee4006810 tx=0x7faee4006960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0526f35a6c04e9c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.328+0000 7faeddffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:52:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.328+0000 7faeddffb640 1 -- 172.21.15.7:0/3325251891 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4025040 con 0x5604edaeb700 2024-04-21T08:52:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.328+0000 7faeddffb640 1 -- 172.21.15.7:0/3325251891 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9672+0+0 (secure 0 0 0) 0x7faee400f050 con 0x5604edaeb700 2024-04-21T08:52:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.329+0000 7faefefe5740 1 -- 172.21.15.7:0/3325251891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faef00092f0 con 0x5604edaeb700 2024-04-21T08:52:27.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.329+0000 7faeddffb640 1 -- 172.21.15.7:0/3325251891 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7faee4004020 con 0x5604edaeb700 2024-04-21T08:52:27.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.329+0000 7faefefe5740 10 client.17001 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:27.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.329+0000 7faefefe5740 1 -- 172.21.15.7:0/3325251891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5604edb6c020 con 0x5604edaeb700 2024-04-21T08:52:27.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.330+0000 7faeddffb640 1 -- 172.21.15.7:0/3325251891 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef00092f0 con 0x5604edaeb700 2024-04-21T08:52:27.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.330+0000 7faeddffb640 1 -- 172.21.15.7:0/3325251891 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7faee402a6b0 con 0x5604edaeb700 2024-04-21T08:52:27.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.331+0000 7faefefe5740 10 client.17001 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:27.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.331+0000 7faefefe5740 10 client.17001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:27.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.331+0000 7faefefe5740 20 client.17001 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:27.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.331+0000 7faefefe5740 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5604edaf4c00 0x5604edb6e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.331+0000 7faefefe5740 4 client.17001 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:52:27.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.331+0000 7faefefe5740 1 -- 172.21.15.7:0/3325251891 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5604eda57a70 con 0x5604edaf4c00 2024-04-21T08:52:27.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.331+0000 7faefa0b1640 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5604edaf4c00 0x5604edb6e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.336+0000 7faefa0b1640 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5604edaf4c00 0x5604edb6e7c0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=556449c300bd798e server_cookie=fda44701e1d662e3 in_seq=0 out_seq=0 2024-04-21T08:52:27.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.336+0000 7faeddffb640 10 client.17001 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:27.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.342+0000 7faeddffb640 1 -- 172.21.15.7:0/3325251891 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x5604eda57a70 con 0x5604edaf4c00 2024-04-21T08:52:27.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.342+0000 7faeddffb640 10 client.17001 handle_command_reply: tid=0 2024-04-21T08:52:27.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.342+0000 7faefefe5740 1 client.17001 shutdown 2024-04-21T08:52:27.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.344+0000 7faefefe5740 1 -- 172.21.15.7:0/3325251891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaeb700 msgr2=0x5604edaf9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.344+0000 7faefefe5740 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaeb700 0x5604edaf9400 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7faee4006810 tx=0x7faee4006960 comp rx=0 tx=0).stop 2024-04-21T08:52:27.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.345+0000 7faefefe5740 1 -- 172.21.15.7:0/3325251891 shutdown_connections 2024-04-21T08:52:27.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.345+0000 7faefefe5740 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5604edaf4c00 0x5604edb6e7c0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.345+0000 7faefefe5740 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5604edaec020 0x5604edaf9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.345+0000 7faefefe5740 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5604edaeb700 0x5604edaf9400 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.345+0000 7faefefe5740 1 --2- 172.21.15.7:0/3325251891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5604edae5e30 0x5604edafd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.345+0000 7faefefe5740 1 -- 172.21.15.7:0/3325251891 >> 172.21.15.7:0/3325251891 conn(0x5604edaf07a0 msgr2=0x5604edae79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.345+0000 7faeddffb640 0 client.17001 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:27.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.349+0000 7faefefe5740 1 -- 172.21.15.7:0/3325251891 shutdown_connections 2024-04-21T08:52:27.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.349+0000 7faefefe5740 1 -- 172.21.15.7:0/3325251891 wait complete. 2024-04-21T08:52:27.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.350+0000 7faefefe5740 20 client.17001 trim_cache size 0 max 16384 2024-04-21T08:52:27.352 INFO:tasks.workunit.client.0.smithi007.stdout: LEX scripts/kconfig/lexer.lex.c 2024-04-21T08:52:27.355 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:52:27.355 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (11899 inodes in the stack)", 2024-04-21T08:52:27.355 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:52:27.356 INFO:teuthology.orchestra.run.smithi007.stdout: "81ebd966-d9b6-44a6-b87d-1c305ebffa72": { 2024-04-21T08:52:27.356 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:52:27.356 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "81ebd966-d9b6-44a6-b87d-1c305ebffa72", 2024-04-21T08:52:27.356 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:52:27.356 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:52:27.356 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:52:27.356 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:52:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.359+0000 7faefb0b3640 1 -- 172.21.15.7:0/28489978 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7faee00a3ea0 msgr2=0x7faee00a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.359+0000 7faefb0b3640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7faee00a3ea0 0x7faee00a6360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faee4006fd0 tx=0x7faee400a5a0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.359+0000 7faefb0b3640 1 -- 172.21.15.7:0/28489978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faef41969d0 msgr2=0x7faef4196e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.359+0000 7faefb0b3640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faef41969d0 0x7faef4196e80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7faeec02c750 tx=0x7faeec035d50 comp rx=0 tx=0).stop 2024-04-21T08:52:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.359+0000 7faefb0b3640 1 -- 172.21.15.7:0/28489978 shutdown_connections 2024-04-21T08:52:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.359+0000 7faefb0b3640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faef41969d0 0x7faef4196e80 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.359+0000 7faefb0b3640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7faee00a3ea0 0x7faee00a6360 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.359+0000 7faefb0b3640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faef4105f80 0x7faef4196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.359+0000 7faefb0b3640 1 --2- 172.21.15.7:0/28489978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef4100eb0 0x7faef419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.359+0000 7faefb0b3640 1 -- 172.21.15.7:0/28489978 >> 172.21.15.7:0/28489978 conn(0x7faef4077a00 msgr2=0x7faef4104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.361+0000 7faefb0b3640 1 -- 172.21.15.7:0/28489978 shutdown_connections 2024-04-21T08:52:27.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:27.361+0000 7faefb0b3640 1 -- 172.21.15.7:0/28489978 wait complete. 2024-04-21T08:52:27.389 INFO:tasks.cephfs.filesystem:scrub status for tag:81ebd966-d9b6-44a6-b87d-1c305ebffa72 - {'path': '/', 'tag': '81ebd966-d9b6-44a6-b87d-1c305ebffa72', 'options': 'recursive,force'} 2024-04-21T08:52:27.390 INFO:tasks.workunit.client.0.smithi007.stdout: YACC scripts/kconfig/parser.tab.[ch] 2024-04-21T08:52:27.543 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/kconfig/lexer.lex.o 2024-04-21T08:52:28.046 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/kconfig/menu.o 2024-04-21T08:52:28.390 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/kconfig/parser.tab.o 2024-04-21T08:52:28.827 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/kconfig/preprocess.o 2024-04-21T08:52:28.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:28 smithi007 ceph-mon[28477]: pgmap v240: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 1.26k op/s 2024-04-21T08:52:29.041 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/kconfig/symbol.o 2024-04-21T08:52:29.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:28 smithi016 ceph-mon[31259]: pgmap v240: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 1.26k op/s 2024-04-21T08:52:29.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:28 smithi077 ceph-mon[31615]: pgmap v240: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 1.26k op/s 2024-04-21T08:52:30.265 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/kconfig/util.o 2024-04-21T08:52:30.359 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTLD scripts/kconfig/conf 2024-04-21T08:52:30.400 INFO:tasks.workunit.client.0.smithi007.stdout:*** Default configuration is based on 'x86_64_defconfig' 2024-04-21T08:52:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:30 smithi007 ceph-mon[28477]: pgmap v241: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.4 MiB/s wr, 1.32k op/s 2024-04-21T08:52:30.905 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:30 smithi077 ceph-mon[31615]: pgmap v241: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.4 MiB/s wr, 1.32k op/s 2024-04-21T08:52:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:30 smithi016 ceph-mon[31259]: pgmap v241: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.4 MiB/s wr, 1.32k op/s 2024-04-21T08:52:33.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:32 smithi016 ceph-mon[31259]: pgmap v242: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.4 MiB/s wr, 1.37k op/s 2024-04-21T08:52:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:32 smithi077 ceph-mon[31615]: pgmap v242: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.4 MiB/s wr, 1.37k op/s 2024-04-21T08:52:33.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:32 smithi007 ceph-mon[28477]: pgmap v242: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 1.4 MiB/s wr, 1.37k op/s 2024-04-21T08:52:35.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:34 smithi016 ceph-mon[31259]: pgmap v243: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 1.1 MiB/s wr, 1.03k op/s 2024-04-21T08:52:35.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:34 smithi077 ceph-mon[31615]: pgmap v243: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 1.1 MiB/s wr, 1.03k op/s 2024-04-21T08:52:35.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:34 smithi007 ceph-mon[28477]: pgmap v243: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 1.1 MiB/s wr, 1.03k op/s 2024-04-21T08:52:37.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:36 smithi016 ceph-mon[31259]: pgmap v244: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 930 KiB/s wr, 718 op/s 2024-04-21T08:52:37.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:36 smithi016 ceph-mon[31259]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T08:52:37.086 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.087+0000 7fe03e670640 1 -- 172.21.15.7:0/2957823038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe038101130 msgr2=0x7fe038101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:37.086 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.087+0000 7fe03e670640 1 --2- 172.21.15.7:0/2957823038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe038101130 0x7fe038101510 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fe02800d910 tx=0x7fe02802dbf0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.092+0000 7fe03e670640 1 -- 172.21.15.7:0/2957823038 shutdown_connections 2024-04-21T08:52:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.092+0000 7fe03e670640 1 --2- 172.21.15.7:0/2957823038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe038106200 0x7fe03810ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.092+0000 7fe03e670640 1 --2- 172.21.15.7:0/2957823038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe038101ae0 0x7fe038105b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.092 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.092+0000 7fe03e670640 1 --2- 172.21.15.7:0/2957823038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe038101130 0x7fe038101510 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.092+0000 7fe03e670640 1 -- 172.21.15.7:0/2957823038 >> 172.21.15.7:0/2957823038 conn(0x7fe0380fc970 msgr2=0x7fe0380fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:37.093 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.094+0000 7fe03e670640 1 -- 172.21.15.7:0/2957823038 shutdown_connections 2024-04-21T08:52:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.098+0000 7fe03e670640 1 -- 172.21.15.7:0/2957823038 wait complete. 2024-04-21T08:52:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.098+0000 7fe03e670640 1 Processor -- start 2024-04-21T08:52:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.099+0000 7fe03e670640 1 -- start start 2024-04-21T08:52:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.099+0000 7fe03e670640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe038101130 0x7fe03819d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.099+0000 7fe03e670640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe038106200 0x7fe03819d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.099+0000 7fe03e670640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0381964d0 0x7fe038196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.099+0000 7fe03e670640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe038110120 con 0x7fe038101130 2024-04-21T08:52:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.099+0000 7fe03e670640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe03810ffa0 con 0x7fe038106200 2024-04-21T08:52:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.099+0000 7fe03e670640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe0381102a0 con 0x7fe0381964d0 2024-04-21T08:52:37.101 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.101+0000 7fe0377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe038106200 0x7fe03819d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.101 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.101+0000 7fe0377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe038106200 0x7fe03819d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:56006/0 (socket says 172.21.15.7:56006) 2024-04-21T08:52:37.101 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.101+0000 7fe0377fe640 1 -- 172.21.15.7:0/1291627070 learned_addr learned my addr 172.21.15.7:0/1291627070 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:37.101 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.101+0000 7fe03cc0d640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0381964d0 0x7fe038196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.102 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.103+0000 7fe037fff640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe038101130 0x7fe03819d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:36 smithi077 ceph-mon[31615]: pgmap v244: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 930 KiB/s wr, 718 op/s 2024-04-21T08:52:37.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:36 smithi077 ceph-mon[31615]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T08:52:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.107+0000 7fe03cc0d640 1 -- 172.21.15.7:0/1291627070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe038106200 msgr2=0x7fe03819d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.107+0000 7fe03cc0d640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe038106200 0x7fe03819d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.107+0000 7fe03cc0d640 1 -- 172.21.15.7:0/1291627070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe038101130 msgr2=0x7fe03819d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.107+0000 7fe03cc0d640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe038101130 0x7fe03819d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.107+0000 7fe03cc0d640 1 -- 172.21.15.7:0/1291627070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03819ae80 con 0x7fe0381964d0 2024-04-21T08:52:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.108+0000 7fe037fff640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe038101130 0x7fe03819d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.111+0000 7fe03cc0d640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0381964d0 0x7fe038196980 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe03000be20 tx=0x7fe03000bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.113+0000 7fe0357fa640 1 -- 172.21.15.7:0/1291627070 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe030019070 con 0x7fe0381964d0 2024-04-21T08:52:37.113 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.113+0000 7fe0357fa640 1 -- 172.21.15.7:0/1291627070 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0300148f0 con 0x7fe0381964d0 2024-04-21T08:52:37.113 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.113+0000 7fe0357fa640 1 -- 172.21.15.7:0/1291627070 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe030012860 con 0x7fe0381964d0 2024-04-21T08:52:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.117+0000 7fe03e670640 1 -- 172.21.15.7:0/1291627070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe03819b170 con 0x7fe0381964d0 2024-04-21T08:52:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.117+0000 7fe03e670640 1 -- 172.21.15.7:0/1291627070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe03819b640 con 0x7fe0381964d0 2024-04-21T08:52:37.121 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.121+0000 7fe04125c740 1 Processor -- start 2024-04-21T08:52:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.127+0000 7fe04125c740 1 -- start start 2024-04-21T08:52:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.127+0000 7fe04125c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634af8d0 0x5593634ae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.127+0000 7fe04125c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634aeb60 0x5593634aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.127+0000 7fe04125c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b80e0 0x5593634ba4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.127+0000 7fe04125c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5593634138d0 con 0x5593634af8d0 2024-04-21T08:52:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.127+0000 7fe04125c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5593633b0e80 con 0x5593634aeb60 2024-04-21T08:52:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.127+0000 7fe04125c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55936340f890 con 0x5593634b80e0 2024-04-21T08:52:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.128+0000 7fe037fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634aeb60 0x5593634aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.128+0000 7fe037fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634aeb60 0x5593634aefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:56022/0 (socket says 172.21.15.7:56022) 2024-04-21T08:52:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.128+0000 7fe037fff640 1 -- 172.21.15.7:0/1516695145 learned_addr learned my addr 172.21.15.7:0/1516695145 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.128+0000 7fe037fff640 1 -- 172.21.15.7:0/1516695145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b80e0 msgr2=0x5593634ba4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.128+0000 7fe037fff640 1 --2- 172.21.15.7:0/1516695145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b80e0 0x5593634ba4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.128+0000 7fe037fff640 1 -- 172.21.15.7:0/1516695145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634af8d0 msgr2=0x5593634ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.129+0000 7fe037fff640 1 --2- 172.21.15.7:0/1516695145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634af8d0 0x5593634ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.129+0000 7fe037fff640 1 -- 172.21.15.7:0/1516695145 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593633d2320 con 0x5593634aeb60 2024-04-21T08:52:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.129+0000 7fe037fff640 1 --2- 172.21.15.7:0/1516695145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634aeb60 0x5593634aefe0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe02800d910 tx=0x7fe028004a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bc5002c3609e509 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.129+0000 7fe0357fa640 1 -- 172.21.15.7:0/1291627070 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fe030014aa0 con 0x7fe0381964d0 2024-04-21T08:52:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.130+0000 7fe0357fa640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0240a3f20 0x7fe0240a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.131+0000 7fe0377fe640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0240a3f20 0x7fe0240a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.131+0000 7fe02e7fc640 1 -- 172.21.15.7:0/1516695145 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02803b040 con 0x5593634aeb60 2024-04-21T08:52:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.131+0000 7fe02e7fc640 1 -- 172.21.15.7:0/1516695145 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe028044070 con 0x5593634aeb60 2024-04-21T08:52:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.131+0000 7fe0357fa640 1 -- 172.21.15.7:0/1291627070 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7fe0300d0350 con 0x7fe0381964d0 2024-04-21T08:52:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.132+0000 7fe04125c740 1 -- 172.21.15.7:0/1516695145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634aeb60 msgr2=0x5593634aefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.132+0000 7fe04125c740 1 --2- 172.21.15.7:0/1516695145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634aeb60 0x5593634aefe0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe02800d910 tx=0x7fe028004a10 comp rx=0 tx=0).stop 2024-04-21T08:52:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.134+0000 7fe04125c740 1 -- 172.21.15.7:0/1516695145 shutdown_connections 2024-04-21T08:52:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.134+0000 7fe04125c740 1 --2- 172.21.15.7:0/1516695145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b80e0 0x5593634ba4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.134+0000 7fe04125c740 1 --2- 172.21.15.7:0/1516695145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634aeb60 0x5593634aefe0 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe02800d910 tx=0x7fe028004a10 comp rx=0 tx=0).stop 2024-04-21T08:52:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.134+0000 7fe04125c740 1 --2- 172.21.15.7:0/1516695145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634af8d0 0x5593634ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.134+0000 7fe04125c740 1 -- 172.21.15.7:0/1516695145 >> 172.21.15.7:0/1516695145 conn(0x559363475c90 msgr2=0x5593634a9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.138+0000 7fe04125c740 1 -- 172.21.15.7:0/1516695145 shutdown_connections 2024-04-21T08:52:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.143+0000 7fe04125c740 1 -- 172.21.15.7:0/1516695145 wait complete. 2024-04-21T08:52:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.143+0000 7fe04125c740 1 -- start start 2024-04-21T08:52:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.143+0000 7fe04125c740 1 Processor -- start 2024-04-21T08:52:37.151 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.150+0000 7fe04125c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634af8d0 0x5593634b7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.151 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.150+0000 7fe04125c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b80e0 0x5593634b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.150+0000 7fe04125c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b5430 0x5593634b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.150+0000 7fe04125c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5593634bae00 con 0x5593634b80e0 2024-04-21T08:52:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.150+0000 7fe04125c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55936339beb0 con 0x5593634b5430 2024-04-21T08:52:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.150+0000 7fe04125c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5593634baf80 con 0x5593634af8d0 2024-04-21T08:52:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.150+0000 7fe04125c740 10 client.0 _ll_register_callbacks cb 0x5593633c8b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:37.152 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.152+0000 7fe037fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634af8d0 0x5593634b7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:36 smithi007 ceph-mon[28477]: pgmap v244: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 930 KiB/s wr, 718 op/s 2024-04-21T08:52:37.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:36 smithi007 ceph-mon[28477]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T08:52:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.152+0000 7fe037fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634af8d0 0x5593634b7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:50884/0 (socket says 172.21.15.7:50884) 2024-04-21T08:52:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.152+0000 7fe037fff640 1 -- 172.21.15.7:0/377389559 learned_addr learned my addr 172.21.15.7:0/377389559 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.153+0000 7fe03cc0d640 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b5430 0x5593634b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.154+0000 7fe037fff640 1 -- 172.21.15.7:0/377389559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b5430 msgr2=0x5593634b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.154+0000 7fe037fff640 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b5430 0x5593634b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.154+0000 7fe037fff640 1 -- 172.21.15.7:0/377389559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b80e0 msgr2=0x5593634b4e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:52:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.154+0000 7fe037fff640 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b80e0 0x5593634b4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.154+0000 7fe037fff640 1 -- 172.21.15.7:0/377389559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559363126210 con 0x5593634af8d0 2024-04-21T08:52:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.157+0000 7fe037fff640 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634af8d0 0x5593634b7af0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe028005ab0 tx=0x7fe02802e690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b0dbc1e793cabc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:37.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.157+0000 7fe0177fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:52:37.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.161+0000 7fe0177fe640 1 -- 172.21.15.7:0/377389559 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02802e7f0 con 0x5593634af8d0 2024-04-21T08:52:37.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.161+0000 7fe04125c740 1 -- 172.21.15.7:0/377389559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe028044070 con 0x5593634af8d0 2024-04-21T08:52:37.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.161+0000 7fe0177fe640 1 -- 172.21.15.7:0/377389559 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7fe02802e990 con 0x5593634af8d0 2024-04-21T08:52:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.165+0000 7fe0377fe640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0240a3f20 0x7fe0240a63e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x5593634aa9e0 tx=0x7fe0200046c0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.168+0000 7fe0177fe640 1 -- 172.21.15.7:0/377389559 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe02802eb90 con 0x5593634af8d0 2024-04-21T08:52:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.168+0000 7fe04125c740 10 client.26524 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.168+0000 7fe04125c740 1 -- 172.21.15.7:0/377389559 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5593631f2620 con 0x5593634af8d0 2024-04-21T08:52:37.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.169+0000 7fe0177fe640 1 -- 172.21.15.7:0/377389559 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe028044070 con 0x5593634af8d0 2024-04-21T08:52:37.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.169+0000 7fe0177fe640 1 -- 172.21.15.7:0/377389559 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fe02803e070 con 0x5593634af8d0 2024-04-21T08:52:37.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.169+0000 7fe04125c740 10 client.26524 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.169+0000 7fe04125c740 10 client.26524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.169+0000 7fe04125c740 20 client.26524 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.169+0000 7fe04125c740 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5593634c8610 0x5593634caa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.169+0000 7fe04125c740 4 client.26524 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.169+0000 7fe04125c740 1 -- 172.21.15.7:0/377389559 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559363427880 con 0x5593634c8610 2024-04-21T08:52:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.170+0000 7fe0377fe640 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5593634c8610 0x5593634caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.185+0000 7fe0377fe640 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5593634c8610 0x5593634caa90 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afef2ff540b13f1f server_cookie=d9c3b88a9a05e90b in_seq=0 out_seq=0 2024-04-21T08:52:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.186+0000 7fe0177fe640 10 client.26524 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.190+0000 7fe0177fe640 1 -- 172.21.15.7:0/377389559 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559363427880 con 0x5593634c8610 2024-04-21T08:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.190+0000 7fe0177fe640 10 client.26524 handle_command_reply: tid=0 2024-04-21T08:52:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.190+0000 7fe04125c740 1 client.26524 shutdown 2024-04-21T08:52:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.194+0000 7fe04125c740 1 -- 172.21.15.7:0/377389559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634af8d0 msgr2=0x5593634b7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.194+0000 7fe04125c740 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634af8d0 0x5593634b7af0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe028005ab0 tx=0x7fe02802e690 comp rx=0 tx=0).stop 2024-04-21T08:52:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.195+0000 7fe04125c740 1 -- 172.21.15.7:0/377389559 shutdown_connections 2024-04-21T08:52:37.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.195+0000 7fe04125c740 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5593634c8610 0x5593634caa90 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.195+0000 7fe04125c740 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b5430 0x5593634b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.195+0000 7fe04125c740 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b80e0 0x5593634b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.195+0000 7fe04125c740 1 --2- 172.21.15.7:0/377389559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634af8d0 0x5593634b7af0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe028005ab0 tx=0x7fe02802e690 comp rx=0 tx=0).stop 2024-04-21T08:52:37.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.195+0000 7fe04125c740 1 -- 172.21.15.7:0/377389559 >> 172.21.15.7:0/377389559 conn(0x559363475c90 msgr2=0x5593634ab5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:37.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.195+0000 7fe0177fe640 0 client.26524 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.202+0000 7fe04125c740 1 -- 172.21.15.7:0/377389559 shutdown_connections 2024-04-21T08:52:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.203+0000 7fe04125c740 1 -- 172.21.15.7:0/377389559 wait complete. 2024-04-21T08:52:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.205+0000 7fe04125c740 20 client.26524 trim_cache size 0 max 16384 2024-04-21T08:52:37.240 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.240+0000 7fe04125c740 1 Processor -- start 2024-04-21T08:52:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.241+0000 7fe04125c740 1 -- start start 2024-04-21T08:52:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.242+0000 7fe04125c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b5430 0x5593634c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.242+0000 7fe04125c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b80e0 0x5593634b7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.242+0000 7fe04125c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b7bd0 0x5593634d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.242+0000 7fe04125c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe038069e30 con 0x5593634b80e0 2024-04-21T08:52:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.242+0000 7fe04125c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe0381c74d0 con 0x5593634b7bd0 2024-04-21T08:52:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.242+0000 7fe04125c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe038197910 con 0x5593634b5430 2024-04-21T08:52:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.243+0000 7fe037fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b5430 0x5593634c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.245+0000 7fe037fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b5430 0x5593634c3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:50892/0 (socket says 172.21.15.7:50892) 2024-04-21T08:52:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.245+0000 7fe037fff640 1 -- 172.21.15.7:0/3314018011 learned_addr learned my addr 172.21.15.7:0/3314018011 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.246+0000 7fe0377fe640 1 --2- 172.21.15.7:0/3314018011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b80e0 0x5593634b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.249+0000 7fe03cc0d640 1 --2- 172.21.15.7:0/3314018011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b7bd0 0x5593634d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.249+0000 7fe0377fe640 1 -- 172.21.15.7:0/3314018011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b5430 msgr2=0x5593634c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.249+0000 7fe0377fe640 1 --2- 172.21.15.7:0/3314018011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b5430 0x5593634c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.249+0000 7fe0377fe640 1 -- 172.21.15.7:0/3314018011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b7bd0 msgr2=0x5593634d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.249+0000 7fe0377fe640 1 --2- 172.21.15.7:0/3314018011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b7bd0 0x5593634d03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.249+0000 7fe0377fe640 1 -- 172.21.15.7:0/3314018011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593634ade30 con 0x5593634b80e0 2024-04-21T08:52:37.261 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.262+0000 7fe0377fe640 1 --2- 172.21.15.7:0/3314018011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b80e0 0x5593634b7690 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x5593634aa700 tx=0x7fe020012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b216b05216cbb191 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:37.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.267+0000 7fe02dffb640 1 -- 172.21.15.7:0/3314018011 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02002a030 con 0x5593634b80e0 2024-04-21T08:52:37.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.267+0000 7fe02dffb640 1 -- 172.21.15.7:0/3314018011 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0200254b0 con 0x5593634b80e0 2024-04-21T08:52:37.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.268+0000 7fe04125c740 1 -- 172.21.15.7:0/3314018011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b80e0 msgr2=0x5593634b7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:37.268 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.268+0000 7fe04125c740 1 --2- 172.21.15.7:0/3314018011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b80e0 0x5593634b7690 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x5593634aa700 tx=0x7fe020012000 comp rx=0 tx=0).stop 2024-04-21T08:52:37.272 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.272+0000 7fe04125c740 1 -- 172.21.15.7:0/3314018011 shutdown_connections 2024-04-21T08:52:37.272 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.272+0000 7fe04125c740 1 --2- 172.21.15.7:0/3314018011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b7bd0 0x5593634d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.272 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.272+0000 7fe04125c740 1 --2- 172.21.15.7:0/3314018011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b80e0 0x5593634b7690 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.272 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.272+0000 7fe04125c740 1 --2- 172.21.15.7:0/3314018011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b5430 0x5593634c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.272 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.272+0000 7fe04125c740 1 -- 172.21.15.7:0/3314018011 >> 172.21.15.7:0/3314018011 conn(0x559363475c90 msgr2=0x5593634a9380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.275+0000 7fe04125c740 1 -- 172.21.15.7:0/3314018011 shutdown_connections 2024-04-21T08:52:37.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.277+0000 7fe04125c740 1 -- 172.21.15.7:0/3314018011 wait complete. 2024-04-21T08:52:37.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.278+0000 7fe04125c740 1 -- start start 2024-04-21T08:52:37.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.278+0000 7fe04125c740 1 Processor -- start 2024-04-21T08:52:37.282 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.282+0000 7fe04125c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b5430 0x5593634c05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.282 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.282+0000 7fe04125c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b7bd0 0x5593634b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.282 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.282+0000 7fe04125c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b80e0 0x5593634b9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.282 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.282+0000 7fe04125c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe038073cf0 con 0x5593634b5430 2024-04-21T08:52:37.282 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.282+0000 7fe04125c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe0381c6e00 con 0x5593634b80e0 2024-04-21T08:52:37.282 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.282+0000 7fe04125c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe0381c6060 con 0x5593634b7bd0 2024-04-21T08:52:37.282 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.282+0000 7fe04125c740 10 client.0 _ll_register_callbacks cb 0x7fe0280048e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:37.283 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.283+0000 7fe03cc0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b80e0 0x5593634b9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.283 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.283+0000 7fe037fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b5430 0x5593634c05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.283 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.284+0000 7fe037fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b5430 0x5593634c05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59012/0 (socket says 172.21.15.7:59012) 2024-04-21T08:52:37.283 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.284+0000 7fe037fff640 1 -- 172.21.15.7:0/3591797224 learned_addr learned my addr 172.21.15.7:0/3591797224 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:37.283 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.284+0000 7fe0377fe640 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b7bd0 0x5593634b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.286+0000 7fe03cc0d640 1 -- 172.21.15.7:0/3591797224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b7bd0 msgr2=0x5593634b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.286+0000 7fe03cc0d640 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b7bd0 0x5593634b8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.286+0000 7fe03cc0d640 1 -- 172.21.15.7:0/3591797224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b5430 msgr2=0x5593634c05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.286+0000 7fe03cc0d640 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b5430 0x5593634c05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.286+0000 7fe03cc0d640 1 -- 172.21.15.7:0/3591797224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593634b97e0 con 0x5593634b80e0 2024-04-21T08:52:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.290+0000 7fe03cc0d640 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b80e0 0x5593634b9000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe0300cd6d0 tx=0x7fe03009bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75872825229e4a7e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.290+0000 7fe0177fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:52:37.297 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.296+0000 7fe0177fe640 1 -- 172.21.15.7:0/3591797224 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0300b4070 con 0x5593634b80e0 2024-04-21T08:52:37.297 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.296+0000 7fe0177fe640 1 -- 172.21.15.7:0/3591797224 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7fe0300cd840 con 0x5593634b80e0 2024-04-21T08:52:37.297 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.298+0000 7fe04125c740 1 -- 172.21.15.7:0/3591797224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0200254b0 con 0x5593634b80e0 2024-04-21T08:52:37.304 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.305+0000 7fe0177fe640 1 -- 172.21.15.7:0/3591797224 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe0300cda40 con 0x5593634b80e0 2024-04-21T08:52:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.308+0000 7fe04125c740 10 client.26561 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.308+0000 7fe04125c740 1 -- 172.21.15.7:0/3591797224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55936352ec10 con 0x5593634b80e0 2024-04-21T08:52:37.313 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.314+0000 7fe0177fe640 1 -- 172.21.15.7:0/3591797224 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0200254b0 con 0x5593634b80e0 2024-04-21T08:52:37.313 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.314+0000 7fe0177fe640 1 -- 172.21.15.7:0/3591797224 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fe030012a00 con 0x5593634b80e0 2024-04-21T08:52:37.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.316+0000 7fe04125c740 10 client.26561 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:37.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.317+0000 7fe04125c740 10 client.26561 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:37.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.317+0000 7fe04125c740 20 client.26561 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:37.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.317+0000 7fe04125c740 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55936352f290 0x5593635316b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:37.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.317+0000 7fe04125c740 4 client.26561 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:52:37.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.317+0000 7fe04125c740 1 -- 172.21.15.7:0/3591797224 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5593633b33d0 con 0x55936352f290 2024-04-21T08:52:37.320 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.320+0000 7fe037fff640 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55936352f290 0x5593635316b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:37.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.335+0000 7fe037fff640 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55936352f290 0x5593635316b0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce478b163cddda70 server_cookie=c5e7384686a5b35e in_seq=0 out_seq=0 2024-04-21T08:52:37.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:37.336+0000 7fe0177fe640 10 client.26561 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:38.805 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe0177fe640 1 -- 172.21.15.7:0/3591797224 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5593633b33d0 con 0x55936352f290 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe0177fe640 10 client.26561 handle_command_reply: tid=0 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe04125c740 1 client.26561 shutdown 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe04125c740 1 -- 172.21.15.7:0/3591797224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b80e0 msgr2=0x5593634b9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe04125c740 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b80e0 0x5593634b9000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe0300cd6d0 tx=0x7fe03009bfd0 comp rx=0 tx=0).stop 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe04125c740 1 -- 172.21.15.7:0/3591797224 shutdown_connections 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe04125c740 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55936352f290 0x5593635316b0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe04125c740 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5593634b80e0 0x5593634b9000 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe04125c740 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5593634b7bd0 0x5593634b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe04125c740 1 --2- 172.21.15.7:0/3591797224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5593634b5430 0x5593634c05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe04125c740 1 -- 172.21.15.7:0/3591797224 >> 172.21.15.7:0/3591797224 conn(0x559363475c90 msgr2=0x5593634b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.805+0000 7fe0177fe640 0 client.26561 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.806+0000 7fe04125c740 1 -- 172.21.15.7:0/3591797224 shutdown_connections 2024-04-21T08:52:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.806+0000 7fe04125c740 1 -- 172.21.15.7:0/3591797224 wait complete. 2024-04-21T08:52:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.806+0000 7fe04125c740 20 client.26561 trim_cache size 0 max 16384 2024-04-21T08:52:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:52:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:52:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:52:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.807+0000 7fe03e670640 1 -- 172.21.15.7:0/1291627070 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0240a3f20 msgr2=0x7fe0240a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.807+0000 7fe03e670640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0240a3f20 0x7fe0240a63e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x5593634aa9e0 tx=0x7fe0200046c0 comp rx=0 tx=0).stop 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.807+0000 7fe03e670640 1 -- 172.21.15.7:0/1291627070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0381964d0 msgr2=0x7fe038196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.807+0000 7fe03e670640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0381964d0 0x7fe038196980 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe03000be20 tx=0x7fe03000bf10 comp rx=0 tx=0).stop 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.807+0000 7fe03e670640 1 -- 172.21.15.7:0/1291627070 shutdown_connections 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.807+0000 7fe03e670640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0240a3f20 0x7fe0240a63e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.807+0000 7fe03e670640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0381964d0 0x7fe038196980 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.807+0000 7fe03e670640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe038106200 0x7fe03819d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.807+0000 7fe03e670640 1 --2- 172.21.15.7:0/1291627070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe038101130 0x7fe03819d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.808+0000 7fe03e670640 1 -- 172.21.15.7:0/1291627070 >> 172.21.15.7:0/1291627070 conn(0x7fe0380fc970 msgr2=0x7fe0380fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.808+0000 7fe03e670640 1 -- 172.21.15.7:0/1291627070 shutdown_connections 2024-04-21T08:52:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:38.808+0000 7fe03e670640 1 -- 172.21.15.7:0/1291627070 wait complete. 2024-04-21T08:52:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:38 smithi016 ceph-mon[31259]: pgmap v246: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 783 KiB/s wr, 582 op/s 2024-04-21T08:52:39.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:38 smithi077 ceph-mon[31615]: pgmap v246: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 783 KiB/s wr, 582 op/s 2024-04-21T08:52:39.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:38 smithi007 ceph-mon[28477]: pgmap v246: 129 pgs: 20 active+clean+snaptrim, 51 active+clean, 58 active+clean+snaptrim_wait; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 783 KiB/s wr, 582 op/s 2024-04-21T08:52:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:40 smithi016 ceph-mon[31259]: pgmap v247: 129 pgs: 19 active+clean+snaptrim, 56 active+clean, 54 active+clean+snaptrim_wait; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 611 op/s 2024-04-21T08:52:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:40 smithi077 ceph-mon[31615]: pgmap v247: 129 pgs: 19 active+clean+snaptrim, 56 active+clean, 54 active+clean+snaptrim_wait; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 611 op/s 2024-04-21T08:52:41.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:40 smithi007 ceph-mon[28477]: pgmap v247: 129 pgs: 19 active+clean+snaptrim, 56 active+clean, 54 active+clean+snaptrim_wait; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 611 op/s 2024-04-21T08:52:42.297 INFO:tasks.workunit.client.0.smithi007.stdout:# 2024-04-21T08:52:42.298 INFO:tasks.workunit.client.0.smithi007.stdout:# configuration written to .config 2024-04-21T08:52:42.298 INFO:tasks.workunit.client.0.smithi007.stdout:# 2024-04-21T08:52:42.300 INFO:tasks.workunit.client.0.smithi007.stderr:++ grep -c processor /proc/cpuinfo 2024-04-21T08:52:42.302 INFO:tasks.workunit.client.0.smithi007.stderr:+ make -j8 2024-04-21T08:52:43.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:42 smithi016 ceph-mon[31259]: pgmap v248: 129 pgs: 18 active+clean+snaptrim, 57 active+clean, 54 active+clean+snaptrim_wait; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 923 KiB/s wr, 637 op/s 2024-04-21T08:52:43.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:42 smithi077 ceph-mon[31615]: pgmap v248: 129 pgs: 18 active+clean+snaptrim, 57 active+clean, 54 active+clean+snaptrim_wait; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 923 KiB/s wr, 637 op/s 2024-04-21T08:52:43.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:42 smithi007 ceph-mon[28477]: pgmap v248: 129 pgs: 18 active+clean+snaptrim, 57 active+clean, 54 active+clean+snaptrim_wait; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 923 KiB/s wr, 637 op/s 2024-04-21T08:52:44.954 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:44 smithi007 ceph-mon[28477]: pgmap v249: 129 pgs: 15 active+clean+snaptrim, 66 active+clean, 48 active+clean+snaptrim_wait; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 654 KiB/s wr, 686 op/s 2024-04-21T08:52:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:44 smithi016 ceph-mon[31259]: pgmap v249: 129 pgs: 15 active+clean+snaptrim, 66 active+clean, 48 active+clean+snaptrim_wait; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 654 KiB/s wr, 686 op/s 2024-04-21T08:52:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:44 smithi077 ceph-mon[31615]: pgmap v249: 129 pgs: 15 active+clean+snaptrim, 66 active+clean, 48 active+clean+snaptrim_wait; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 654 KiB/s wr, 686 op/s 2024-04-21T08:52:45.120 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/bpf_perf_event.h 2024-04-21T08:52:45.124 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/errno.h 2024-04-21T08:52:45.126 INFO:tasks.workunit.client.0.smithi007.stdout: GEN arch/x86/include/generated/asm/orc_hash.h 2024-04-21T08:52:45.126 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/fcntl.h 2024-04-21T08:52:45.128 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/ioctl.h 2024-04-21T08:52:45.130 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/ioctls.h 2024-04-21T08:52:45.133 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/ipcbuf.h 2024-04-21T08:52:45.135 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/param.h 2024-04-21T08:52:45.138 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/poll.h 2024-04-21T08:52:45.139 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/resource.h 2024-04-21T08:52:45.143 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/socket.h 2024-04-21T08:52:45.146 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/sockios.h 2024-04-21T08:52:45.150 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/termios.h 2024-04-21T08:52:45.153 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/termbits.h 2024-04-21T08:52:45.155 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/uapi/asm/types.h 2024-04-21T08:52:45.187 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/asm/early_ioremap.h 2024-04-21T08:52:45.188 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/asm/export.h 2024-04-21T08:52:45.195 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/asm/mcs_spinlock.h 2024-04-21T08:52:45.196 INFO:tasks.workunit.client.0.smithi007.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_32.h 2024-04-21T08:52:45.198 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/asm/irq_regs.h 2024-04-21T08:52:45.207 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/asm/kmap_size.h 2024-04-21T08:52:45.207 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC arch/x86/tools/relocs_32.o 2024-04-21T08:52:45.209 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/asm/local64.h 2024-04-21T08:52:45.214 INFO:tasks.workunit.client.0.smithi007.stdout: UPD include/config/kernel.release 2024-04-21T08:52:45.214 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/asm/mmiowb.h 2024-04-21T08:52:45.228 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/asm/module.lds.h 2024-04-21T08:52:45.228 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/asm/rwonce.h 2024-04-21T08:52:45.245 INFO:tasks.workunit.client.0.smithi007.stdout: WRAP arch/x86/include/generated/asm/unaligned.h 2024-04-21T08:52:45.247 INFO:tasks.workunit.client.0.smithi007.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_64.h 2024-04-21T08:52:45.254 INFO:tasks.workunit.client.0.smithi007.stdout: UPD include/generated/uapi/linux/version.h 2024-04-21T08:52:45.254 INFO:tasks.workunit.client.0.smithi007.stdout: SYSTBL arch/x86/include/generated/asm/syscalls_32.h 2024-04-21T08:52:45.255 INFO:tasks.workunit.client.0.smithi007.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_x32.h 2024-04-21T08:52:45.297 INFO:tasks.workunit.client.0.smithi007.stdout: SYSHDR arch/x86/include/generated/asm/unistd_32_ia32.h 2024-04-21T08:52:45.305 INFO:tasks.workunit.client.0.smithi007.stdout: UPD include/generated/compile.h 2024-04-21T08:52:45.319 INFO:tasks.workunit.client.0.smithi007.stdout: SYSHDR arch/x86/include/generated/asm/unistd_64_x32.h 2024-04-21T08:52:45.324 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC arch/x86/tools/relocs_64.o 2024-04-21T08:52:45.370 INFO:tasks.workunit.client.0.smithi007.stdout: SYSTBL arch/x86/include/generated/asm/syscalls_64.h 2024-04-21T08:52:45.414 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/kallsyms 2024-04-21T08:52:45.480 INFO:tasks.workunit.client.0.smithi007.stdout: UPD include/generated/utsrelease.h 2024-04-21T08:52:45.484 INFO:tasks.workunit.client.0.smithi007.stdout: DESCEND objtool 2024-04-21T08:52:45.517 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/selinux/genheaders/genheaders 2024-04-21T08:52:45.517 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC arch/x86/tools/relocs_common.o 2024-04-21T08:52:45.679 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/selinux/mdp/mdp 2024-04-21T08:52:45.701 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/sorttable 2024-04-21T08:52:45.744 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/asn1_compiler 2024-04-21T08:52:45.878 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTLD arch/x86/tools/relocs 2024-04-21T08:52:46.045 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/fixdep.o 2024-04-21T08:52:46.242 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTLD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/fixdep-in.o 2024-04-21T08:52:46.248 INFO:tasks.workunit.client.0.smithi007.stdout: LINK /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/fixdep 2024-04-21T08:52:46.351 INFO:tasks.workunit.client.0.smithi007.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/exec-cmd.h 2024-04-21T08:52:46.353 INFO:tasks.workunit.client.0.smithi007.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/help.h 2024-04-21T08:52:46.353 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/exec-cmd.o 2024-04-21T08:52:46.355 INFO:tasks.workunit.client.0.smithi007.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/pager.h 2024-04-21T08:52:46.356 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/help.o 2024-04-21T08:52:46.367 INFO:tasks.workunit.client.0.smithi007.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/parse-options.h 2024-04-21T08:52:46.373 INFO:tasks.workunit.client.0.smithi007.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/run-command.h 2024-04-21T08:52:46.377 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/pager.o 2024-04-21T08:52:46.385 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/parse-options.o 2024-04-21T08:52:46.386 INFO:tasks.workunit.client.0.smithi007.stdout: INSTALL libsubcmd_headers 2024-04-21T08:52:46.389 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/run-command.o 2024-04-21T08:52:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/sigchain.o 2024-04-21T08:52:46.449 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/subcmd-config.o 2024-04-21T08:52:47.042 INFO:tasks.workunit.client.0.smithi007.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/libsubcmd-in.o 2024-04-21T08:52:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:46 smithi016 ceph-mon[31259]: pgmap v250: 129 pgs: 12 active+clean+snaptrim, 83 active+clean, 34 active+clean+snaptrim_wait; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 809 KiB/s wr, 739 op/s 2024-04-21T08:52:47.065 INFO:tasks.workunit.client.0.smithi007.stdout: AR /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/libsubcmd.a 2024-04-21T08:52:47.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:46 smithi077 ceph-mon[31615]: pgmap v250: 129 pgs: 12 active+clean+snaptrim, 83 active+clean, 34 active+clean+snaptrim_wait; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 809 KiB/s wr, 739 op/s 2024-04-21T08:52:47.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:46 smithi007 ceph-mon[28477]: pgmap v250: 129 pgs: 12 active+clean+snaptrim, 83 active+clean, 34 active+clean+snaptrim_wait; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 809 KiB/s wr, 739 op/s 2024-04-21T08:52:48.271 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/weak.o 2024-04-21T08:52:48.273 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/check.o 2024-04-21T08:52:48.273 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/special.o 2024-04-21T08:52:48.274 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/builtin-check.o 2024-04-21T08:52:48.276 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/elf.o 2024-04-21T08:52:48.277 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/objtool.o 2024-04-21T08:52:48.305 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/special.o 2024-04-21T08:52:48.460 INFO:tasks.workunit.client.0.smithi007.stdout: MKDIR /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/lib/ 2024-04-21T08:52:48.467 INFO:tasks.workunit.client.0.smithi007.stdout: GEN /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/lib/inat-tables.c 2024-04-21T08:52:48.469 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/orc_gen.o 2024-04-21T08:52:48.505 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/orc_dump.o 2024-04-21T08:52:48.518 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/decode.o 2024-04-21T08:52:48.579 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libstring.o 2024-04-21T08:52:48.581 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libctype.o 2024-04-21T08:52:48.693 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/str_error_r.o 2024-04-21T08:52:48.756 INFO:tasks.workunit.client.0.smithi007.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/librbtree.o 2024-04-21T08:52:49.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:48 smithi016 ceph-mon[31259]: pgmap v251: 129 pgs: 11 active+clean+snaptrim, 91 active+clean, 27 active+clean+snaptrim_wait; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 928 KiB/s wr, 858 op/s 2024-04-21T08:52:49.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:48 smithi077 ceph-mon[31615]: pgmap v251: 129 pgs: 11 active+clean+snaptrim, 91 active+clean, 27 active+clean+snaptrim_wait; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 928 KiB/s wr, 858 op/s 2024-04-21T08:52:49.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:48 smithi007 ceph-mon[28477]: pgmap v251: 129 pgs: 11 active+clean+snaptrim, 91 active+clean, 27 active+clean+snaptrim_wait; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 928 KiB/s wr, 858 op/s 2024-04-21T08:52:49.305 INFO:tasks.workunit.client.0.smithi007.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/objtool-in.o 2024-04-21T08:52:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:50 smithi016 ceph-mon[31259]: pgmap v252: 129 pgs: 10 active+clean+snaptrim, 100 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 889 KiB/s wr, 910 op/s 2024-04-21T08:52:51.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:50 smithi077 ceph-mon[31615]: pgmap v252: 129 pgs: 10 active+clean+snaptrim, 100 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 889 KiB/s wr, 910 op/s 2024-04-21T08:52:51.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:50 smithi007 ceph-mon[28477]: pgmap v252: 129 pgs: 10 active+clean+snaptrim, 100 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 889 KiB/s wr, 910 op/s 2024-04-21T08:52:51.436 INFO:tasks.workunit.client.0.smithi007.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/objtool-in.o 2024-04-21T08:52:51.458 INFO:tasks.workunit.client.0.smithi007.stdout: LINK /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/objtool 2024-04-21T08:52:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.953+0000 7f6bdd258640 1 -- 172.21.15.7:0/621145469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd8100ed0 msgr2=0x7f6bd81012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.953+0000 7f6bdd258640 1 --2- 172.21.15.7:0/621145469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd8100ed0 0x7f6bd81012b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009900 tx=0x7f6bc802bc50 comp rx=0 tx=0).stop 2024-04-21T08:52:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.954+0000 7f6bdd258640 1 -- 172.21.15.7:0/621145469 shutdown_connections 2024-04-21T08:52:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.954+0000 7f6bdd258640 1 --2- 172.21.15.7:0/621145469 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bd8105fa0 0x7f6bd810db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.954+0000 7f6bdd258640 1 --2- 172.21.15.7:0/621145469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd8101880 0x7f6bd81058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.954+0000 7f6bdd258640 1 --2- 172.21.15.7:0/621145469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd8100ed0 0x7f6bd81012b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.954+0000 7f6bdd258640 1 -- 172.21.15.7:0/621145469 >> 172.21.15.7:0/621145469 conn(0x7f6bd80fc730 msgr2=0x7f6bd80feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.954+0000 7f6bdd258640 1 -- 172.21.15.7:0/621145469 shutdown_connections 2024-04-21T08:52:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.954+0000 7f6bdd258640 1 -- 172.21.15.7:0/621145469 wait complete. 2024-04-21T08:52:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.954+0000 7f6bdd258640 1 Processor -- start 2024-04-21T08:52:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.955+0000 7f6bdd258640 1 -- start start 2024-04-21T08:52:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.955+0000 7f6bdd258640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd8100ed0 0x7f6bd810cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.955+0000 7f6bdd258640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd8105fa0 0x7f6bd810d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.955+0000 7f6bdd258640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bd81a3330 0x7f6bd81a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.955+0000 7f6bd77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd8105fa0 0x7f6bd810d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.955+0000 7f6bd77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd8105fa0 0x7f6bd810d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49092/0 (socket says 172.21.15.7:49092) 2024-04-21T08:52:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.955+0000 7f6bd77fe640 1 -- 172.21.15.7:0/2906861297 learned_addr learned my addr 172.21.15.7:0/2906861297 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.955+0000 7f6bdd258640 1 -- 172.21.15.7:0/2906861297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd81120b0 con 0x7f6bd8105fa0 2024-04-21T08:52:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.955+0000 7f6bdd258640 1 -- 172.21.15.7:0/2906861297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd8111f30 con 0x7f6bd8100ed0 2024-04-21T08:52:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.955+0000 7f6bdd258640 1 -- 172.21.15.7:0/2906861297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd8112230 con 0x7f6bd81a3330 2024-04-21T08:52:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.956+0000 7f6bd7fff640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd8100ed0 0x7f6bd810cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.956+0000 7f6bdca57640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bd81a3330 0x7f6bd81a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.956+0000 7f6bd77fe640 1 -- 172.21.15.7:0/2906861297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bd81a3330 msgr2=0x7f6bd81a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.956+0000 7f6bd77fe640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bd81a3330 0x7f6bd81a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.956+0000 7f6bd77fe640 1 -- 172.21.15.7:0/2906861297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd8100ed0 msgr2=0x7f6bd810cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.956+0000 7f6bd77fe640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd8100ed0 0x7f6bd810cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.956+0000 7f6bd77fe640 1 -- 172.21.15.7:0/2906861297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd81a3df0 con 0x7f6bd8105fa0 2024-04-21T08:52:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.956+0000 7f6bd7fff640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd8100ed0 0x7f6bd810cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.957+0000 7f6bd77fe640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd8105fa0 0x7f6bd810d2e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00b450 tx=0x7f6bcc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.961+0000 7f6bd57fa640 1 -- 172.21.15.7:0/2906861297 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc015020 con 0x7f6bd8105fa0 2024-04-21T08:52:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.961+0000 7f6bd57fa640 1 -- 172.21.15.7:0/2906861297 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bcc011d40 con 0x7f6bd8105fa0 2024-04-21T08:52:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.961+0000 7f6bd57fa640 1 -- 172.21.15.7:0/2906861297 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00f340 con 0x7f6bd8105fa0 2024-04-21T08:52:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.962+0000 7f6bdd258640 1 -- 172.21.15.7:0/2906861297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd81a40e0 con 0x7f6bd8105fa0 2024-04-21T08:52:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.962+0000 7f6bdd258640 1 -- 172.21.15.7:0/2906861297 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd81ab880 con 0x7f6bd8105fa0 2024-04-21T08:52:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.965+0000 7f6be1188740 1 Processor -- start 2024-04-21T08:52:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.967+0000 7f6be1188740 1 -- start start 2024-04-21T08:52:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.968+0000 7f6be1188740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cc7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.968+0000 7f6be1188740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc7d50 0x564269cc81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.968+0000 7f6be1188740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 0x564269cd36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.968+0000 7f6be1188740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564269c2c8d0 con 0x564269cd12d0 2024-04-21T08:52:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.968+0000 7f6be1188740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564269bc9e80 con 0x564269cc7d50 2024-04-21T08:52:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.968+0000 7f6be1188740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564269c28890 con 0x564269cc8ac0 2024-04-21T08:52:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.968+0000 7f6bdca57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cc7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.968+0000 7f6bdca57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cc7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:35296/0 (socket says 172.21.15.7:35296) 2024-04-21T08:52:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.968+0000 7f6bdca57640 1 -- 172.21.15.7:0/249206474 learned_addr learned my addr 172.21.15.7:0/249206474 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.970+0000 7f6bd57fa640 1 -- 172.21.15.7:0/2906861297 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f6bcc018020 con 0x7f6bd8105fa0 2024-04-21T08:52:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.970+0000 7f6bd7fff640 1 --2- 172.21.15.7:0/249206474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc7d50 0x564269cc81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.970+0000 7f6bdca57640 1 -- 172.21.15.7:0/249206474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc7d50 msgr2=0x564269cc81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.970+0000 7f6bdca57640 1 --2- 172.21.15.7:0/249206474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc7d50 0x564269cc81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.970+0000 7f6bdca57640 1 -- 172.21.15.7:0/249206474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 msgr2=0x564269cd36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.970+0000 7f6bdca57640 1 --2- 172.21.15.7:0/249206474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 0x564269cd36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.970+0000 7f6bdca57640 1 -- 172.21.15.7:0/249206474 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564269beb320 con 0x564269cc8ac0 2024-04-21T08:52:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.971+0000 7f6bdca57640 1 --2- 172.21.15.7:0/249206474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cc7810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0009ee0 tx=0x7f6bd0010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e4d9097f36c01dc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.971+0000 7f6bd57fa640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bc00a3f30 0x7f6bc00a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.971+0000 7f6bbe7fc640 1 -- 172.21.15.7:0/249206474 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0016070 con 0x564269cc8ac0 2024-04-21T08:52:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.971+0000 7f6bbe7fc640 1 -- 172.21.15.7:0/249206474 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bd0004d20 con 0x564269cc8ac0 2024-04-21T08:52:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.971+0000 7f6bd57fa640 1 -- 172.21.15.7:0/2906861297 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f6bcc0d0500 con 0x7f6bd8105fa0 2024-04-21T08:52:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.971+0000 7f6bbe7fc640 1 -- 172.21.15.7:0/249206474 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0012730 con 0x564269cc8ac0 2024-04-21T08:52:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.972+0000 7f6bd7fff640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bc00a3f30 0x7f6bc00a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.972+0000 7f6be1188740 1 -- 172.21.15.7:0/249206474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 msgr2=0x564269cc7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.972+0000 7f6be1188740 1 --2- 172.21.15.7:0/249206474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cc7810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0009ee0 tx=0x7f6bd0010740 comp rx=0 tx=0).stop 2024-04-21T08:52:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.973+0000 7f6bd7fff640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bc00a3f30 0x7f6bc00a63f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x564269cc9220 tx=0x7f6bc8000bc0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.975+0000 7f6be1188740 1 -- 172.21.15.7:0/249206474 shutdown_connections 2024-04-21T08:52:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.975+0000 7f6be1188740 1 --2- 172.21.15.7:0/249206474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 0x564269cd36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.975+0000 7f6be1188740 1 --2- 172.21.15.7:0/249206474 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc7d50 0x564269cc81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.975+0000 7f6be1188740 1 --2- 172.21.15.7:0/249206474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cc7810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.975+0000 7f6be1188740 1 -- 172.21.15.7:0/249206474 >> 172.21.15.7:0/249206474 conn(0x564269c8ec90 msgr2=0x564269cc2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.976+0000 7f6be1188740 1 -- 172.21.15.7:0/249206474 shutdown_connections 2024-04-21T08:52:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.979+0000 7f6be1188740 1 -- 172.21.15.7:0/249206474 wait complete. 2024-04-21T08:52:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.979+0000 7f6be1188740 1 -- start start 2024-04-21T08:52:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.980+0000 7f6be1188740 1 Processor -- start 2024-04-21T08:52:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.984+0000 7f6be1188740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc7d50 0x564269cd0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.984+0000 7f6be1188740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc8ac0 0x564269cce020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.984+0000 7f6be1188740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 0x564269cce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.984+0000 7f6be1188740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564269cd4000 con 0x564269cd12d0 2024-04-21T08:52:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.984+0000 7f6be1188740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564269bb4eb0 con 0x564269cc8ac0 2024-04-21T08:52:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.984+0000 7f6be1188740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564269cd4180 con 0x564269cc7d50 2024-04-21T08:52:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.984+0000 7f6be1188740 10 client.0 _ll_register_callbacks cb 0x564269be1b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.984+0000 7f6bd77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc8ac0 0x564269cce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.986+0000 7f6bd7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc7d50 0x564269cd0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.986+0000 7f6bd7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc7d50 0x564269cd0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:35304/0 (socket says 172.21.15.7:35304) 2024-04-21T08:52:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.986+0000 7f6bd7fff640 1 -- 172.21.15.7:0/2943201940 learned_addr learned my addr 172.21.15.7:0/2943201940 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.987+0000 7f6bd7fff640 1 -- 172.21.15.7:0/2943201940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc8ac0 msgr2=0x564269cce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.987+0000 7f6bd7fff640 1 --2- 172.21.15.7:0/2943201940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc8ac0 0x564269cce020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.987+0000 7f6bd7fff640 1 -- 172.21.15.7:0/2943201940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 msgr2=0x564269cce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.987+0000 7f6bd7fff640 1 --2- 172.21.15.7:0/2943201940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 0x564269cce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.987+0000 7f6bd7fff640 1 -- 172.21.15.7:0/2943201940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56426993f210 con 0x564269cc7d50 2024-04-21T08:52:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.989+0000 7f6bd7fff640 1 --2- 172.21.15.7:0/2943201940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc7d50 0x564269cd0c20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8035000 tx=0x7f6bc802bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12f7406d8170fd42 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.989+0000 7f6bb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:52:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.989+0000 7f6bb77fe640 1 -- 172.21.15.7:0/2943201940 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc804f070 con 0x564269cc7d50 2024-04-21T08:52:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.989+0000 7f6bb77fe640 1 -- 172.21.15.7:0/2943201940 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f6bc8003a00 con 0x564269cc7d50 2024-04-21T08:52:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.989+0000 7f6be1188740 1 -- 172.21.15.7:0/2943201940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bd0004d20 con 0x564269cc7d50 2024-04-21T08:52:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.990+0000 7f6bb77fe640 1 -- 172.21.15.7:0/2943201940 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6bc805d050 con 0x564269cc7d50 2024-04-21T08:52:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.993+0000 7f6be1188740 10 client.26539 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.993+0000 7f6be1188740 1 -- 172.21.15.7:0/2943201940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564269a0b620 con 0x564269cc7d50 2024-04-21T08:52:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.996+0000 7f6bb77fe640 1 -- 172.21.15.7:0/2943201940 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bd0004d20 con 0x564269cc7d50 2024-04-21T08:52:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.997+0000 7f6bb77fe640 1 -- 172.21.15.7:0/2943201940 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6bc804b420 con 0x564269cc7d50 2024-04-21T08:52:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.997+0000 7f6be1188740 10 client.26539 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.997+0000 7f6be1188740 10 client.26539 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.997+0000 7f6be1188740 20 client.26539 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.997+0000 7f6be1188740 1 --2- 172.21.15.7:0/2943201940 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564269cf27e0 0x564269cf4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.997+0000 7f6be1188740 4 client.26539 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.997+0000 7f6be1188740 1 -- 172.21.15.7:0/2943201940 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564269c40880 con 0x564269cf27e0 2024-04-21T08:52:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:51.999+0000 7f6bd77fe640 1 --2- 172.21.15.7:0/2943201940 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564269cf27e0 0x564269cf4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.001+0000 7f6bd77fe640 1 --2- 172.21.15.7:0/2943201940 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564269cf27e0 0x564269cf4c40 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62978c518c9df16f server_cookie=7e5358062a248f34 in_seq=0 out_seq=0 2024-04-21T08:52:52.002 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.001+0000 7f6bb77fe640 10 client.26539 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.003+0000 7f6bb77fe640 1 -- 172.21.15.7:0/2943201940 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564269c40880 con 0x564269cf27e0 2024-04-21T08:52:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.003+0000 7f6bb77fe640 10 client.26539 handle_command_reply: tid=0 2024-04-21T08:52:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.003+0000 7f6be1188740 1 client.26539 shutdown 2024-04-21T08:52:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.005+0000 7f6be1188740 1 -- 172.21.15.7:0/2943201940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc7d50 msgr2=0x564269cd0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.005+0000 7f6be1188740 1 --2- 172.21.15.7:0/2943201940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc7d50 0x564269cd0c20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8035000 tx=0x7f6bc802bc50 comp rx=0 tx=0).stop 2024-04-21T08:52:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.015+0000 7f6be1188740 1 -- 172.21.15.7:0/2943201940 shutdown_connections 2024-04-21T08:52:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.015+0000 7f6be1188740 1 --2- 172.21.15.7:0/2943201940 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564269cf27e0 0x564269cf4c40 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.015+0000 7f6be1188740 1 --2- 172.21.15.7:0/2943201940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 0x564269cce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.015+0000 7f6be1188740 1 --2- 172.21.15.7:0/2943201940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc8ac0 0x564269cce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.015+0000 7f6be1188740 1 --2- 172.21.15.7:0/2943201940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc7d50 0x564269cd0c20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.015+0000 7f6be1188740 1 -- 172.21.15.7:0/2943201940 >> 172.21.15.7:0/2943201940 conn(0x564269c8ec90 msgr2=0x564269cc4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.015+0000 7f6bb77fe640 0 client.26539 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.016+0000 7f6be1188740 1 -- 172.21.15.7:0/2943201940 shutdown_connections 2024-04-21T08:52:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.017+0000 7f6be1188740 1 -- 172.21.15.7:0/2943201940 wait complete. 2024-04-21T08:52:52.019 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.019+0000 7f6be1188740 20 client.26539 trim_cache size 0 max 16384 2024-04-21T08:52:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.053+0000 7f6be1188740 1 Processor -- start 2024-04-21T08:52:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.053+0000 7f6be1188740 1 -- start start 2024-04-21T08:52:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.054+0000 7f6be1188740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cc7d50 0x564269cdcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.054+0000 7f6be1188740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc8ac0 0x564269cbd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.054+0000 7f6be1188740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cd12d0 0x564269cd0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.054+0000 7f6be1188740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc400e1e0 con 0x564269cc7d50 2024-04-21T08:52:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.054+0000 7f6be1188740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc4010a60 con 0x564269cc8ac0 2024-04-21T08:52:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.054+0000 7f6bdca57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cd12d0 0x564269cd0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.054+0000 7f6bdca57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cd12d0 0x564269cd0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:35308/0 (socket says 172.21.15.7:35308) 2024-04-21T08:52:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.054+0000 7f6bdca57640 1 -- 172.21.15.7:0/3489907718 learned_addr learned my addr 172.21.15.7:0/3489907718 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.054+0000 7f6be1188740 1 -- 172.21.15.7:0/3489907718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc400e850 con 0x564269cd12d0 2024-04-21T08:52:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.054+0000 7f6bd7fff640 1 --2- 172.21.15.7:0/3489907718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cc7d50 0x564269cdcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.055+0000 7f6bd77fe640 1 --2- 172.21.15.7:0/3489907718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc8ac0 0x564269cbd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:52.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:51 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.055+0000 7f6bdca57640 1 -- 172.21.15.7:0/3489907718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc8ac0 msgr2=0x564269cbd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.055+0000 7f6bdca57640 1 --2- 172.21.15.7:0/3489907718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc8ac0 0x564269cbd970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.055+0000 7f6bdca57640 1 -- 172.21.15.7:0/3489907718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cc7d50 msgr2=0x564269cdcf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.055+0000 7f6bdca57640 1 --2- 172.21.15.7:0/3489907718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cc7d50 0x564269cdcf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.055+0000 7f6bdca57640 1 -- 172.21.15.7:0/3489907718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564269cce930 con 0x564269cd12d0 2024-04-21T08:52:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.055+0000 7f6bdca57640 1 --2- 172.21.15.7:0/3489907718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cd12d0 0x564269cd0750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0009c60 tx=0x7f6bd00095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=820bd65026193410 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.056+0000 7f6bbdffb640 1 -- 172.21.15.7:0/3489907718 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0027070 con 0x564269cd12d0 2024-04-21T08:52:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.056+0000 7f6bbdffb640 1 -- 172.21.15.7:0/3489907718 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bd0004c60 con 0x564269cd12d0 2024-04-21T08:52:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.056+0000 7f6bbdffb640 1 -- 172.21.15.7:0/3489907718 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0016050 con 0x564269cd12d0 2024-04-21T08:52:52.058 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.057+0000 7f6be1188740 1 -- 172.21.15.7:0/3489907718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cd12d0 msgr2=0x564269cd0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:52.058 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.057+0000 7f6be1188740 1 --2- 172.21.15.7:0/3489907718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cd12d0 0x564269cd0750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0009c60 tx=0x7f6bd00095c0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.060+0000 7f6be1188740 1 -- 172.21.15.7:0/3489907718 shutdown_connections 2024-04-21T08:52:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.060+0000 7f6be1188740 1 --2- 172.21.15.7:0/3489907718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cd12d0 0x564269cd0750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.060+0000 7f6be1188740 1 --2- 172.21.15.7:0/3489907718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc8ac0 0x564269cbd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.060+0000 7f6be1188740 1 --2- 172.21.15.7:0/3489907718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cc7d50 0x564269cdcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.060+0000 7f6be1188740 1 -- 172.21.15.7:0/3489907718 >> 172.21.15.7:0/3489907718 conn(0x564269c8ec90 msgr2=0x564269ccc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:52.062 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.062+0000 7f6be1188740 1 -- 172.21.15.7:0/3489907718 shutdown_connections 2024-04-21T08:52:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.064+0000 7f6be1188740 1 -- 172.21.15.7:0/3489907718 wait complete. 2024-04-21T08:52:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.065+0000 7f6be1188740 1 -- start start 2024-04-21T08:52:52.066 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.065+0000 7f6be1188740 1 Processor -- start 2024-04-21T08:52:52.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6be1188740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc7d50 0x564269cd4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:52.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6be1188740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cd4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:52.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6be1188740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 0x564269d56db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:52.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6bd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc7d50 0x564269cd4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:52.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6bd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc7d50 0x564269cd4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39832/0 (socket says 172.21.15.7:39832) 2024-04-21T08:52:52.069 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6bd77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cd4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6bd77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cd4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:35324/0 (socket says 172.21.15.7:35324) 2024-04-21T08:52:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6bd7fff640 1 -- 172.21.15.7:0/1434308358 learned_addr learned my addr 172.21.15.7:0/1434308358 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6be1188740 1 -- 172.21.15.7:0/1434308358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc400ef20 con 0x564269cd12d0 2024-04-21T08:52:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6be1188740 1 -- 172.21.15.7:0/1434308358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc4010390 con 0x564269cc7d50 2024-04-21T08:52:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6be1188740 1 -- 172.21.15.7:0/1434308358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc400f5f0 con 0x564269cc8ac0 2024-04-21T08:52:52.070 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.069+0000 7f6be1188740 10 client.0 _ll_register_callbacks cb 0x7f6bc8043050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.074+0000 7f6bdca57640 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 0x564269d56db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.075+0000 7f6bd77fe640 1 -- 172.21.15.7:0/1434308358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc7d50 msgr2=0x564269cd4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.075+0000 7f6bd77fe640 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc7d50 0x564269cd4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.075+0000 7f6bd77fe640 1 -- 172.21.15.7:0/1434308358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 msgr2=0x564269d56db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.075+0000 7f6bd77fe640 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 0x564269d56db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:52.075 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.075+0000 7f6bd77fe640 1 -- 172.21.15.7:0/1434308358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564269cd5230 con 0x564269cc8ac0 2024-04-21T08:52:52.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.076+0000 7f6bd77fe640 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cd4970 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0cd880 tx=0x7f6bcc010000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=237397f0662a6309 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:52.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.076+0000 7f6bb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:52:52.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.076+0000 7f6bb77fe640 1 -- 172.21.15.7:0/1434308358 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc0b4070 con 0x564269cc8ac0 2024-04-21T08:52:52.076 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.077+0000 7f6bb77fe640 1 -- 172.21.15.7:0/1434308358 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f6bcc098740 con 0x564269cc8ac0 2024-04-21T08:52:52.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.077+0000 7f6be1188740 1 -- 172.21.15.7:0/1434308358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bd0004c60 con 0x564269cc8ac0 2024-04-21T08:52:52.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.079+0000 7f6bb77fe640 1 -- 172.21.15.7:0/1434308358 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6bcc0a4d40 con 0x564269cc8ac0 2024-04-21T08:52:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.080+0000 7f6be1188740 10 client.26545 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:52.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.080+0000 7f6be1188740 1 -- 172.21.15.7:0/1434308358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564269d58430 con 0x564269cc8ac0 2024-04-21T08:52:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.083+0000 7f6bb77fe640 1 -- 172.21.15.7:0/1434308358 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bd0004c60 con 0x564269cc8ac0 2024-04-21T08:52:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.083+0000 7f6bb77fe640 1 -- 172.21.15.7:0/1434308358 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6bcc0b0b70 con 0x564269cc8ac0 2024-04-21T08:52:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.083+0000 7f6be1188740 10 client.26545 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.083+0000 7f6be1188740 10 client.26545 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:52.083 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.083+0000 7f6be1188740 20 client.26545 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.084+0000 7f6be1188740 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564269d58b10 0x564269d5af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.084+0000 7f6be1188740 4 client.26545 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:52:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.084+0000 7f6be1188740 1 -- 172.21.15.7:0/1434308358 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564269bcc3d0 con 0x564269d58b10 2024-04-21T08:52:52.086 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.085+0000 7f6bd7fff640 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564269d58b10 0x564269d5af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.091+0000 7f6bd7fff640 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564269d58b10 0x564269d5af30 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=857783e33d0b9d10 server_cookie=1ae7366417ad0dc in_seq=0 out_seq=0 2024-04-21T08:52:52.091 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:52.091+0000 7f6bb77fe640 10 client.26545 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:52.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:51 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:52.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:51 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:52 smithi016 ceph-mon[31259]: pgmap v253: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 917 KiB/s wr, 1.17k op/s 2024-04-21T08:52:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:52 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:52:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:52 smithi077 ceph-mon[31615]: pgmap v253: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 917 KiB/s wr, 1.17k op/s 2024-04-21T08:52:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:52 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:52:53.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:52 smithi007 ceph-mon[28477]: pgmap v253: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 917 KiB/s wr, 1.17k op/s 2024-04-21T08:52:53.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:52 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:52:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.734+0000 7f6bb77fe640 1 -- 172.21.15.7:0/1434308358 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564269bcc3d0 con 0x564269d58b10 2024-04-21T08:52:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.734+0000 7f6bb77fe640 10 client.26545 handle_command_reply: tid=0 2024-04-21T08:52:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.734+0000 7f6be1188740 1 client.26545 shutdown 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.734+0000 7f6be1188740 1 -- 172.21.15.7:0/1434308358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 msgr2=0x564269cd4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.734+0000 7f6be1188740 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cd4970 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0cd880 tx=0x7f6bcc010000 comp rx=0 tx=0).stop 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.735+0000 7f6be1188740 1 -- 172.21.15.7:0/1434308358 shutdown_connections 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.735+0000 7f6be1188740 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564269d58b10 0x564269d5af30 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.735+0000 7f6be1188740 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564269cd12d0 0x564269d56db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.735+0000 7f6bb77fe640 0 client.26545 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.735+0000 7f6be1188740 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564269cc8ac0 0x564269cd4970 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.735+0000 7f6be1188740 1 --2- 172.21.15.7:0/1434308358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564269cc7d50 0x564269cd4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.735+0000 7f6be1188740 1 -- 172.21.15.7:0/1434308358 >> 172.21.15.7:0/1434308358 conn(0x564269c8ec90 msgr2=0x564269cc1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.735+0000 7f6be1188740 1 -- 172.21.15.7:0/1434308358 shutdown_connections 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.735+0000 7f6be1188740 1 -- 172.21.15.7:0/1434308358 wait complete. 2024-04-21T08:52:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.736+0000 7f6be1188740 20 client.26545 trim_cache size 0 max 16384 2024-04-21T08:52:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:52:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:52:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:52:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:52:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.736+0000 7f6bdd258640 1 -- 172.21.15.7:0/2906861297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bc00a3f30 msgr2=0x7f6bc00a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.736+0000 7f6bdd258640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bc00a3f30 0x7f6bc00a63f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x564269cc9220 tx=0x7f6bc8000bc0 comp rx=0 tx=0).stop 2024-04-21T08:52:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.736+0000 7f6bdd258640 1 -- 172.21.15.7:0/2906861297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd8105fa0 msgr2=0x7f6bd810d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.736+0000 7f6bdd258640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd8105fa0 0x7f6bd810d2e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00b450 tx=0x7f6bcc00b910 comp rx=0 tx=0).stop 2024-04-21T08:52:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.737+0000 7f6bdd258640 1 -- 172.21.15.7:0/2906861297 shutdown_connections 2024-04-21T08:52:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.737+0000 7f6bdd258640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bc00a3f30 0x7f6bc00a63f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.737+0000 7f6bdd258640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bd81a3330 0x7f6bd81a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.737+0000 7f6bdd258640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd8105fa0 0x7f6bd810d2e0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.737+0000 7f6bdd258640 1 --2- 172.21.15.7:0/2906861297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bd8100ed0 0x7f6bd810cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.737+0000 7f6bdd258640 1 -- 172.21.15.7:0/2906861297 >> 172.21.15.7:0/2906861297 conn(0x7f6bd80fc730 msgr2=0x7f6bd80fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.737+0000 7f6bdd258640 1 -- 172.21.15.7:0/2906861297 shutdown_connections 2024-04-21T08:52:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:52:54.737+0000 7f6bdd258640 1 -- 172.21.15.7:0/2906861297 wait complete. 2024-04-21T08:52:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:54 smithi016 ceph-mon[31259]: pgmap v254: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 1.26k op/s 2024-04-21T08:52:55.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:54 smithi077 ceph-mon[31615]: pgmap v254: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 1.26k op/s 2024-04-21T08:52:55.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:54 smithi007 ceph-mon[28477]: pgmap v254: 129 pgs: 9 active+clean+snaptrim, 101 active+clean, 19 active+clean+snaptrim_wait; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 1.26k op/s 2024-04-21T08:52:56.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:55 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:52:56.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:55 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:52:56.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:55 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:52:57.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:56 smithi016 ceph-mon[31259]: pgmap v255: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 1.26k op/s 2024-04-21T08:52:57.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:56 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:52:57.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:56 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:52:57.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:56 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:52:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:56 smithi077 ceph-mon[31615]: pgmap v255: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 1.26k op/s 2024-04-21T08:52:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:56 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:52:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:56 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:52:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:56 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:52:57.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:56 smithi007 ceph-mon[28477]: pgmap v255: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 1.26k op/s 2024-04-21T08:52:57.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:56 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:52:57.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:56 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:52:57.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:56 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:52:57.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:52:57.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:52:57.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.568+0000 7f7ba32b3640 1 -- 172.21.15.7:0/3591876344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b9c101640 msgr2=0x7f7b9c101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.568+0000 7f7ba32b3640 1 --2- 172.21.15.7:0/3591876344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b9c101640 0x7f7b9c101a20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009900 tx=0x7f7b9802bc30 comp rx=0 tx=0).stop 2024-04-21T08:52:57.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.568+0000 7f7ba32b3640 1 -- 172.21.15.7:0/3591876344 shutdown_connections 2024-04-21T08:52:57.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.568+0000 7f7ba32b3640 1 --2- 172.21.15.7:0/3591876344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c105fe0 0x7f7b9c10dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.568+0000 7f7ba32b3640 1 --2- 172.21.15.7:0/3591876344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b9c101ff0 0x7f7b9c105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.568+0000 7f7ba32b3640 1 --2- 172.21.15.7:0/3591876344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b9c101640 0x7f7b9c101a20 secure :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009900 tx=0x7f7b9802bc30 comp rx=0 tx=0).stop 2024-04-21T08:52:57.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.568+0000 7f7ba32b3640 1 -- 172.21.15.7:0/3591876344 >> 172.21.15.7:0/3591876344 conn(0x7f7b9c077190 msgr2=0x7f7b9c0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:57.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.569+0000 7f7ba32b3640 1 -- 172.21.15.7:0/3591876344 shutdown_connections 2024-04-21T08:52:57.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.569+0000 7f7ba32b3640 1 -- 172.21.15.7:0/3591876344 wait complete. 2024-04-21T08:52:57.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.570+0000 7f7ba32b3640 1 Processor -- start 2024-04-21T08:52:57.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.570+0000 7f7ba32b3640 1 -- start start 2024-04-21T08:52:57.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.570+0000 7f7ba32b3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b9c105fe0 0x7f7b9c19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.570+0000 7f7ba32b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c19eab0 0x7f7b9c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.571+0000 7f7ba32b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b9c198b70 0x7f7b9c199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.571+0000 7f7ba1ab0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c19eab0 0x7f7b9c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.571+0000 7f7ba1ab0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c19eab0 0x7f7b9c198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:51870/0 (socket says 172.21.15.7:51870) 2024-04-21T08:52:57.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.571+0000 7f7ba1ab0640 1 -- 172.21.15.7:0/4087951492 learned_addr learned my addr 172.21.15.7:0/4087951492 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:57.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.571+0000 7f7ba2ab2640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b9c198b70 0x7f7b9c199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.571+0000 7f7ba22b1640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b9c105fe0 0x7f7b9c19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.571+0000 7f7ba32b3640 1 -- 172.21.15.7:0/4087951492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c1120f0 con 0x7f7b9c105fe0 2024-04-21T08:52:57.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.571+0000 7f7ba32b3640 1 -- 172.21.15.7:0/4087951492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c111f70 con 0x7f7b9c198b70 2024-04-21T08:52:57.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.571+0000 7f7ba32b3640 1 -- 172.21.15.7:0/4087951492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c112270 con 0x7f7b9c19eab0 2024-04-21T08:52:57.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.572+0000 7f7ba1ab0640 1 -- 172.21.15.7:0/4087951492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b9c198b70 msgr2=0x7f7b9c199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.572+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b9c198b70 0x7f7b9c199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.572+0000 7f7ba1ab0640 1 -- 172.21.15.7:0/4087951492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b9c105fe0 msgr2=0x7f7b9c19e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.572+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b9c105fe0 0x7f7b9c19e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.572+0000 7f7ba1ab0640 1 -- 172.21.15.7:0/4087951492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9c1a5310 con 0x7f7b9c19eab0 2024-04-21T08:52:57.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.572+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c19eab0 0x7f7b9c198630 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c009ee0 tx=0x7f7b8c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.574+0000 7f7b937fe640 1 -- 172.21.15.7:0/4087951492 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c00ccc0 con 0x7f7b9c19eab0 2024-04-21T08:52:57.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.574+0000 7f7b937fe640 1 -- 172.21.15.7:0/4087951492 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b8c0054d0 con 0x7f7b9c19eab0 2024-04-21T08:52:57.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.574+0000 7f7ba32b3640 1 -- 172.21.15.7:0/4087951492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b9c10e310 con 0x7f7b9c19eab0 2024-04-21T08:52:57.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.574+0000 7f7ba32b3640 1 -- 172.21.15.7:0/4087951492 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b9c10e810 con 0x7f7b9c19eab0 2024-04-21T08:52:57.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.575+0000 7f7b937fe640 1 -- 172.21.15.7:0/4087951492 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0047a0 con 0x7f7b9c19eab0 2024-04-21T08:52:57.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.585+0000 7f7b937fe640 1 -- 172.21.15.7:0/4087951492 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f7b8c004940 con 0x7f7b9c19eab0 2024-04-21T08:52:57.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.585+0000 7f7ba71e2740 1 Processor -- start 2024-04-21T08:52:57.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.587+0000 7f7ba71e2740 1 -- start start 2024-04-21T08:52:57.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.587+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be1f20 0x556ac2be2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.588+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2be2840 0x556ac2beaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.588+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2beb420 0x556ac2bed810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.588+0000 7f7ba71e2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556ac2b3e1e0 con 0x556ac2beb420 2024-04-21T08:52:57.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.588+0000 7f7ba71e2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556ac2adb390 con 0x556ac2be2840 2024-04-21T08:52:57.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.588+0000 7f7ba71e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556ac2b3a1a0 con 0x556ac2be1f20 2024-04-21T08:52:57.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.589+0000 7f7ba1ab0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2beb420 0x556ac2bed810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.589+0000 7f7b937fe640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7b880a3ee0 0x7f7b880a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.589+0000 7f7ba1ab0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2beb420 0x556ac2bed810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50994/0 (socket says 172.21.15.7:50994) 2024-04-21T08:52:57.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.589+0000 7f7ba1ab0640 1 -- 172.21.15.7:0/3177789858 learned_addr learned my addr 172.21.15.7:0/3177789858 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:57.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.589+0000 7f7b937fe640 1 -- 172.21.15.7:0/4087951492 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f7b8c0d4d90 con 0x7f7b9c19eab0 2024-04-21T08:52:57.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.590+0000 7f7ba1ab0640 1 -- 172.21.15.7:0/3177789858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be1f20 msgr2=0x556ac2be2300 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:57.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.590+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/3177789858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be1f20 0x556ac2be2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.590+0000 7f7ba1ab0640 1 -- 172.21.15.7:0/3177789858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2be2840 msgr2=0x556ac2beaee0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:57.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.590+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/3177789858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2be2840 0x556ac2beaee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.590+0000 7f7ba1ab0640 1 -- 172.21.15.7:0/3177789858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac291a7d0 con 0x556ac2beb420 2024-04-21T08:52:57.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.590+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/3177789858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2beb420 0x556ac2bed810 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00c6a0 tx=0x7f7b8c0a0860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de19366b1a2c17d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.590+0000 7f7b90bfe640 1 -- 172.21.15.7:0/3177789858 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0a4f70 con 0x556ac2beb420 2024-04-21T08:52:57.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.591+0000 7f7b90bfe640 1 -- 172.21.15.7:0/3177789858 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b8c0a4d10 con 0x556ac2beb420 2024-04-21T08:52:57.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.591+0000 7f7b90bfe640 1 -- 172.21.15.7:0/3177789858 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0110d0 con 0x556ac2beb420 2024-04-21T08:52:57.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.592+0000 7f7ba2ab2640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7b880a3ee0 0x7f7b880a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.592+0000 7f7ba71e2740 1 -- 172.21.15.7:0/3177789858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2beb420 msgr2=0x556ac2bed810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.592+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/3177789858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2beb420 0x556ac2bed810 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00c6a0 tx=0x7f7b8c0a0860 comp rx=0 tx=0).stop 2024-04-21T08:52:57.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.593+0000 7f7ba2ab2640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7b880a3ee0 0x7f7b880a63a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7b94005fd0 tx=0x7f7b94009500 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.595+0000 7f7ba71e2740 1 -- 172.21.15.7:0/3177789858 shutdown_connections 2024-04-21T08:52:57.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.595+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/3177789858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2beb420 0x556ac2bed810 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.595+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/3177789858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2be2840 0x556ac2beaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.595+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/3177789858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be1f20 0x556ac2be2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.595+0000 7f7ba71e2740 1 -- 172.21.15.7:0/3177789858 >> 172.21.15.7:0/3177789858 conn(0x556ac2bdc650 msgr2=0x556ac2bdca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:57.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.597+0000 7f7ba71e2740 1 -- 172.21.15.7:0/3177789858 shutdown_connections 2024-04-21T08:52:57.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.598+0000 7f7ba71e2740 1 -- 172.21.15.7:0/3177789858 wait complete. 2024-04-21T08:52:57.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.598+0000 7f7ba71e2740 1 -- start start 2024-04-21T08:52:57.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.598+0000 7f7ba71e2740 1 Processor -- start 2024-04-21T08:52:57.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.601+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 0x556ac2be8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.601+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be2840 0x556ac2be88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.601+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2beb420 0x556ac2be8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.601+0000 7f7ba71e2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556ac2bee210 con 0x556ac2be1f20 2024-04-21T08:52:57.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.601+0000 7f7ba71e2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556ac2af72f0 con 0x556ac2beb420 2024-04-21T08:52:57.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.601+0000 7f7ba71e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556ac2bee390 con 0x556ac2be2840 2024-04-21T08:52:57.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.601+0000 7f7ba71e2740 10 client.0 _ll_register_callbacks cb 0x556ac2b04580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:57.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.604+0000 7f7ba22b1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 0x556ac2be8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.604+0000 7f7ba22b1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 0x556ac2be8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50996/0 (socket says 172.21.15.7:50996) 2024-04-21T08:52:57.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.604+0000 7f7ba22b1640 1 -- 172.21.15.7:0/403982599 learned_addr learned my addr 172.21.15.7:0/403982599 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:57.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.604+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be2840 0x556ac2be88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.604+0000 7f7ba2ab2640 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2beb420 0x556ac2be8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.606+0000 7f7ba22b1640 1 -- 172.21.15.7:0/403982599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be2840 msgr2=0x556ac2be88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:57.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.606+0000 7f7ba22b1640 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be2840 0x556ac2be88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.606+0000 7f7ba22b1640 1 -- 172.21.15.7:0/403982599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2beb420 msgr2=0x556ac2be8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:57.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.606+0000 7f7ba22b1640 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2beb420 0x556ac2be8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.606+0000 7f7ba22b1640 1 -- 172.21.15.7:0/403982599 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ac28503f0 con 0x556ac2be1f20 2024-04-21T08:52:57.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.607+0000 7f7ba22b1640 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 0x556ac2be8390 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7b9802c240 tx=0x7f7b9802ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ebdf1c9b5a3ba95 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.608+0000 7f7b85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:52:57.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.608+0000 7f7b85ffb640 1 -- 172.21.15.7:0/403982599 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98045020 con 0x556ac2be1f20 2024-04-21T08:52:57.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.608+0000 7f7b85ffb640 1 -- 172.21.15.7:0/403982599 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f7b98005040 con 0x556ac2be1f20 2024-04-21T08:52:57.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.608+0000 7f7ba71e2740 1 -- 172.21.15.7:0/403982599 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556ac2b4bc00 con 0x556ac2be1f20 2024-04-21T08:52:57.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.609+0000 7f7b85ffb640 1 -- 172.21.15.7:0/403982599 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7b9802c5d0 con 0x556ac2be1f20 2024-04-21T08:52:57.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.609+0000 7f7ba71e2740 10 client.17034 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:57.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.609+0000 7f7ba71e2740 1 -- 172.21.15.7:0/403982599 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556ac28100f0 con 0x556ac2be1f20 2024-04-21T08:52:57.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.610+0000 7f7b85ffb640 1 -- 172.21.15.7:0/403982599 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556ac2b4bc00 con 0x556ac2be1f20 2024-04-21T08:52:57.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.610+0000 7f7b85ffb640 1 -- 172.21.15.7:0/403982599 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f7b98004ba0 con 0x556ac2be1f20 2024-04-21T08:52:57.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.610+0000 7f7ba71e2740 10 client.17034 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:57.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.610+0000 7f7ba71e2740 10 client.17034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:57.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.610+0000 7f7ba71e2740 20 client.17034 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:57.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.610+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556ac2bd77c0 0x556ac2bd9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.610+0000 7f7ba71e2740 4 client.17034 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:57.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.610+0000 7f7ba71e2740 1 -- 172.21.15.7:0/403982599 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ac2b45a70 con 0x556ac2bd77c0 2024-04-21T08:52:57.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.611+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556ac2bd77c0 0x556ac2bd9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.612+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556ac2bd77c0 0x556ac2bd9c60 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=803a19154ba34e24 server_cookie=2098b40b9046928f in_seq=0 out_seq=0 2024-04-21T08:52:57.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.612+0000 7f7b85ffb640 10 client.17034 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:57.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7b85ffb640 1 -- 172.21.15.7:0/403982599 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556ac2b45a70 con 0x556ac2bd77c0 2024-04-21T08:52:57.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7b85ffb640 10 client.17034 handle_command_reply: tid=0 2024-04-21T08:52:57.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7ba71e2740 1 client.17034 shutdown 2024-04-21T08:52:57.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7ba71e2740 1 -- 172.21.15.7:0/403982599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 msgr2=0x556ac2be8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 0x556ac2be8390 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7b9802c240 tx=0x7f7b9802ca10 comp rx=0 tx=0).stop 2024-04-21T08:52:57.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7ba71e2740 1 -- 172.21.15.7:0/403982599 shutdown_connections 2024-04-21T08:52:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556ac2bd77c0 0x556ac2bd9c60 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2beb420 0x556ac2be8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be2840 0x556ac2be88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/403982599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 0x556ac2be8390 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7ba71e2740 1 -- 172.21.15.7:0/403982599 >> 172.21.15.7:0/403982599 conn(0x556ac2bdc650 msgr2=0x556ac2bddb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7b85ffb640 0 client.17034 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7ba71e2740 1 -- 172.21.15.7:0/403982599 shutdown_connections 2024-04-21T08:52:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.614+0000 7f7ba71e2740 1 -- 172.21.15.7:0/403982599 wait complete. 2024-04-21T08:52:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.619+0000 7f7ba71e2740 20 client.17034 trim_cache size 0 max 16384 2024-04-21T08:52:57.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.652+0000 7f7ba71e2740 1 Processor -- start 2024-04-21T08:52:57.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.653+0000 7f7ba71e2740 1 -- start start 2024-04-21T08:52:57.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.653+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2be1f20 0x556ac2beaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.653+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be2840 0x556ac2bf7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.653+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2beb420 0x556ac2c0d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.654+0000 7f7ba71e2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8000e2a0 con 0x556ac2be2840 2024-04-21T08:52:57.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.654+0000 7f7ba71e2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b80010b20 con 0x556ac2be1f20 2024-04-21T08:52:57.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.654+0000 7f7ba2ab2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2beb420 0x556ac2c0d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.654+0000 7f7ba2ab2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2beb420 0x556ac2c0d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:51894/0 (socket says 172.21.15.7:51894) 2024-04-21T08:52:57.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.654+0000 7f7ba2ab2640 1 -- 172.21.15.7:0/740774436 learned_addr learned my addr 172.21.15.7:0/740774436 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:57.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.654+0000 7f7ba71e2740 1 -- 172.21.15.7:0/740774436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8000e910 con 0x556ac2beb420 2024-04-21T08:52:57.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.654+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/740774436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be2840 0x556ac2bf7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.654+0000 7f7ba22b1640 1 --2- 172.21.15.7:0/740774436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2be1f20 0x556ac2beaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.655+0000 7f7ba1ab0640 1 -- 172.21.15.7:0/740774436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2beb420 msgr2=0x556ac2c0d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.655+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/740774436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2beb420 0x556ac2c0d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.655+0000 7f7ba1ab0640 1 -- 172.21.15.7:0/740774436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2be1f20 msgr2=0x556ac2beaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.655+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/740774436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2be1f20 0x556ac2beaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.655+0000 7f7ba1ab0640 1 -- 172.21.15.7:0/740774436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac2be68e0 con 0x556ac2be2840 2024-04-21T08:52:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.655+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/740774436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be2840 0x556ac2bf7170 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00cfd0 tx=0x7f7b8c0a0860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97dbd9f16617859 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.656+0000 7f7b87fff640 1 -- 172.21.15.7:0/740774436 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0a02c0 con 0x556ac2be2840 2024-04-21T08:52:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.656+0000 7f7b87fff640 1 -- 172.21.15.7:0/740774436 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b8c0a1060 con 0x556ac2be2840 2024-04-21T08:52:57.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.657+0000 7f7b87fff640 1 -- 172.21.15.7:0/740774436 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0a93a0 con 0x556ac2be2840 2024-04-21T08:52:57.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.657+0000 7f7ba71e2740 1 -- 172.21.15.7:0/740774436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be2840 msgr2=0x556ac2bf7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.657+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/740774436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be2840 0x556ac2bf7170 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00cfd0 tx=0x7f7b8c0a0860 comp rx=0 tx=0).stop 2024-04-21T08:52:57.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.658+0000 7f7ba71e2740 1 -- 172.21.15.7:0/740774436 shutdown_connections 2024-04-21T08:52:57.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.658+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/740774436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2beb420 0x556ac2c0d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.658+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/740774436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be2840 0x556ac2bf7170 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.658+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/740774436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2be1f20 0x556ac2beaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.658+0000 7f7ba71e2740 1 -- 172.21.15.7:0/740774436 >> 172.21.15.7:0/740774436 conn(0x556ac2bdc650 msgr2=0x556ac2be3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:57.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.658+0000 7f7ba71e2740 1 -- 172.21.15.7:0/740774436 shutdown_connections 2024-04-21T08:52:57.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.658+0000 7f7ba71e2740 1 -- 172.21.15.7:0/740774436 wait complete. 2024-04-21T08:52:57.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.659+0000 7f7ba71e2740 1 -- start start 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.659+0000 7f7ba71e2740 1 Processor -- start 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.659+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 0x556ac2bf39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.659+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be2840 0x556ac2be5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.659+0000 7f7ba71e2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2beb420 0x556ac2be5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.659+0000 7f7ba71e2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8000efe0 con 0x556ac2be1f20 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.659+0000 7f7ba71e2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b80010450 con 0x556ac2beb420 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.659+0000 7f7ba71e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8000f6b0 con 0x556ac2be2840 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.659+0000 7f7ba71e2740 10 client.0 _ll_register_callbacks cb 0x7f7b940093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.660+0000 7f7ba2ab2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2beb420 0x556ac2be5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.660+0000 7f7ba2ab2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2beb420 0x556ac2be5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:42208/0 (socket says 172.21.15.7:42208) 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.660+0000 7f7ba2ab2640 1 -- 172.21.15.7:0/1833934045 learned_addr learned my addr 172.21.15.7:0/1833934045 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.660+0000 7f7ba22b1640 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 0x556ac2bf39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.660+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be2840 0x556ac2be5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.660+0000 7f7ba22b1640 1 -- 172.21.15.7:0/1833934045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be2840 msgr2=0x556ac2be5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:57.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.660+0000 7f7ba22b1640 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be2840 0x556ac2be5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.660+0000 7f7ba22b1640 1 -- 172.21.15.7:0/1833934045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2beb420 msgr2=0x556ac2be5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:57.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.660+0000 7f7ba22b1640 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2beb420 0x556ac2be5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.660+0000 7f7ba22b1640 1 -- 172.21.15.7:0/1833934045 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ac2be62a0 con 0x556ac2be1f20 2024-04-21T08:52:57.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.661+0000 7f7ba22b1640 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 0x556ac2bf39b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7b9802c170 tx=0x7f7b98005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c658f454dac139de server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.661+0000 7f7b85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:52:57.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.661+0000 7f7b85ffb640 1 -- 172.21.15.7:0/1833934045 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98056030 con 0x556ac2be1f20 2024-04-21T08:52:57.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.661+0000 7f7ba71e2740 1 -- 172.21.15.7:0/1833934045 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b8c0a1060 con 0x556ac2be1f20 2024-04-21T08:52:57.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.661+0000 7f7b85ffb640 1 -- 172.21.15.7:0/1833934045 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f7b98004fa0 con 0x556ac2be1f20 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.662+0000 7f7b85ffb640 1 -- 172.21.15.7:0/1833934045 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7b98060050 con 0x556ac2be1f20 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.662+0000 7f7ba71e2740 10 client.17040 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.662+0000 7f7ba71e2740 1 -- 172.21.15.7:0/1833934045 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556ac2c7afd0 con 0x556ac2be1f20 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.663+0000 7f7b85ffb640 1 -- 172.21.15.7:0/1833934045 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b8c0a1060 con 0x556ac2be1f20 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.663+0000 7f7b85ffb640 1 -- 172.21.15.7:0/1833934045 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f7b980049b0 con 0x556ac2be1f20 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.663+0000 7f7ba71e2740 10 client.17040 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.663+0000 7f7ba71e2740 10 client.17040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.663+0000 7f7ba71e2740 20 client.17040 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.663+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556ac2c7b6c0 0x556ac2c7dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.663+0000 7f7ba71e2740 4 client.17040 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.663+0000 7f7ba71e2740 1 -- 172.21.15.7:0/1833934045 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556ac2ac1590 con 0x556ac2c7b6c0 2024-04-21T08:52:57.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.663+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556ac2c7b6c0 0x556ac2c7dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.664+0000 7f7ba1ab0640 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556ac2c7b6c0 0x556ac2c7dae0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb2d52fa2c93ada2 server_cookie=77f9d1f484a8917c in_seq=0 out_seq=0 2024-04-21T08:52:57.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.664+0000 7f7b85ffb640 10 client.17040 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:57.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.667+0000 7f7b85ffb640 1 -- 172.21.15.7:0/1833934045 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556ac2ac1590 con 0x556ac2c7b6c0 2024-04-21T08:52:57.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.667+0000 7f7b85ffb640 10 client.17040 handle_command_reply: tid=0 2024-04-21T08:52:57.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.667+0000 7f7ba71e2740 1 client.17040 shutdown 2024-04-21T08:52:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.669+0000 7f7ba71e2740 1 -- 172.21.15.7:0/1833934045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 msgr2=0x556ac2bf39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.669+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 0x556ac2bf39b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7b9802c170 tx=0x7f7b98005bc0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.670+0000 7f7ba71e2740 1 -- 172.21.15.7:0/1833934045 shutdown_connections 2024-04-21T08:52:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.670+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556ac2c7b6c0 0x556ac2c7dae0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.670+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556ac2beb420 0x556ac2be5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.670+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556ac2be2840 0x556ac2be5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.670+0000 7f7ba71e2740 1 --2- 172.21.15.7:0/1833934045 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ac2be1f20 0x556ac2bf39b0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.670+0000 7f7ba71e2740 1 -- 172.21.15.7:0/1833934045 >> 172.21.15.7:0/1833934045 conn(0x556ac2bdc650 msgr2=0x556ac2bd2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.670+0000 7f7b85ffb640 0 client.17040 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.670+0000 7f7ba71e2740 1 -- 172.21.15.7:0/1833934045 shutdown_connections 2024-04-21T08:52:57.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.670+0000 7f7ba71e2740 1 -- 172.21.15.7:0/1833934045 wait complete. 2024-04-21T08:52:57.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.670+0000 7f7ba71e2740 20 client.17040 trim_cache size 0 max 16384 2024-04-21T08:52:57.671 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:52:57.672 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:52:57.672 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:52:57.672 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:52:57.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.672+0000 7f7ba32b3640 1 -- 172.21.15.7:0/4087951492 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7b880a3ee0 msgr2=0x7f7b880a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.672+0000 7f7ba32b3640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7b880a3ee0 0x7f7b880a63a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7b94005fd0 tx=0x7f7b94009500 comp rx=0 tx=0).stop 2024-04-21T08:52:57.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.672+0000 7f7ba32b3640 1 -- 172.21.15.7:0/4087951492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c19eab0 msgr2=0x7f7b9c198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.672+0000 7f7ba32b3640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c19eab0 0x7f7b9c198630 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c009ee0 tx=0x7f7b8c00edc0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.672+0000 7f7ba32b3640 1 -- 172.21.15.7:0/4087951492 shutdown_connections 2024-04-21T08:52:57.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.672+0000 7f7ba32b3640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b9c198b70 0x7f7b9c199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.672+0000 7f7ba32b3640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7b880a3ee0 0x7f7b880a63a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.672+0000 7f7ba32b3640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b9c19eab0 0x7f7b9c198630 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.672+0000 7f7ba32b3640 1 --2- 172.21.15.7:0/4087951492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b9c105fe0 0x7f7b9c19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.672+0000 7f7ba32b3640 1 -- 172.21.15.7:0/4087951492 >> 172.21.15.7:0/4087951492 conn(0x7f7b9c077190 msgr2=0x7f7b9c0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:57.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.673+0000 7f7ba32b3640 1 -- 172.21.15.7:0/4087951492 shutdown_connections 2024-04-21T08:52:57.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.673+0000 7f7ba32b3640 1 -- 172.21.15.7:0/4087951492 wait complete. 2024-04-21T08:52:57.690 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:52:57.690 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:52:57.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.862+0000 7ff506775640 1 -- 172.21.15.7:0/2957630539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff500106ed0 msgr2=0x7ff5000692f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.862+0000 7ff506775640 1 --2- 172.21.15.7:0/2957630539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff500106ed0 0x7ff5000692f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff4f401a940 tx=0x7ff4f403cc80 comp rx=0 tx=0).stop 2024-04-21T08:52:57.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.862+0000 7ff506775640 1 -- 172.21.15.7:0/2957630539 shutdown_connections 2024-04-21T08:52:57.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.862+0000 7ff506775640 1 --2- 172.21.15.7:0/2957630539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff500104a40 0x7ff500104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.862+0000 7ff506775640 1 --2- 172.21.15.7:0/2957630539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff500069830 0x7ff500069c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.862+0000 7ff506775640 1 --2- 172.21.15.7:0/2957630539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff500106ed0 0x7ff5000692f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.862+0000 7ff506775640 1 -- 172.21.15.7:0/2957630539 >> 172.21.15.7:0/2957630539 conn(0x7ff5000771d0 msgr2=0x7ff5000775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:57.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.863+0000 7ff506775640 1 -- 172.21.15.7:0/2957630539 shutdown_connections 2024-04-21T08:52:57.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.863+0000 7ff506775640 1 -- 172.21.15.7:0/2957630539 wait complete. 2024-04-21T08:52:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.863+0000 7ff506775640 1 Processor -- start 2024-04-21T08:52:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.864+0000 7ff506775640 1 -- start start 2024-04-21T08:52:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.864+0000 7ff506775640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff500069830 0x7ff500115200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.864+0000 7ff506775640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff500104a40 0x7ff500111290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.864+0000 7ff506775640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff500106ed0 0x7ff5001117d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.864+0000 7ff506775640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff500109c60 con 0x7ff500106ed0 2024-04-21T08:52:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.864+0000 7ff506775640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff500109ae0 con 0x7ff500069830 2024-04-21T08:52:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.864+0000 7ff506775640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff500109de0 con 0x7ff500104a40 2024-04-21T08:52:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.864+0000 7ff4fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff500069830 0x7ff500115200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.864+0000 7ff4fffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff500069830 0x7ff500115200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:42210/0 (socket says 172.21.15.7:42210) 2024-04-21T08:52:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.864+0000 7ff4fffff640 1 -- 172.21.15.7:0/2192333751 learned_addr learned my addr 172.21.15.7:0/2192333751 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.864+0000 7ff4ff7fe640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff500104a40 0x7ff500111290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.865+0000 7ff4fffff640 1 -- 172.21.15.7:0/2192333751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff500104a40 msgr2=0x7ff500111290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.865+0000 7ff4fffff640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff500104a40 0x7ff500111290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.865+0000 7ff4fffff640 1 -- 172.21.15.7:0/2192333751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff500106ed0 msgr2=0x7ff5001117d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.865+0000 7ff4fffff640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff500106ed0 0x7ff5001117d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.865+0000 7ff4fffff640 1 -- 172.21.15.7:0/2192333751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff500111d90 con 0x7ff500069830 2024-04-21T08:52:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.865+0000 7ff4fffff640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff500069830 0x7ff500115200 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4047040 tx=0x7ff4f4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.867+0000 7ff4fd7fa640 1 -- 172.21.15.7:0/2192333751 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4011020 con 0x7ff500069830 2024-04-21T08:52:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.867+0000 7ff506775640 1 -- 172.21.15.7:0/2192333751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff500112020 con 0x7ff500069830 2024-04-21T08:52:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.867+0000 7ff506775640 1 -- 172.21.15.7:0/2192333751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5001c5020 con 0x7ff500069830 2024-04-21T08:52:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.868+0000 7ff4fd7fa640 1 -- 172.21.15.7:0/2192333751 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4f4005400 con 0x7ff500069830 2024-04-21T08:52:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.868+0000 7ff4fd7fa640 1 -- 172.21.15.7:0/2192333751 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f403f710 con 0x7ff500069830 2024-04-21T08:52:57.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.871+0000 7ff506775640 1 -- 172.21.15.7:0/2192333751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4ec005290 con 0x7ff500069830 2024-04-21T08:52:57.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.871+0000 7ff4fd7fa640 1 -- 172.21.15.7:0/2192333751 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7ff4f4014020 con 0x7ff500069830 2024-04-21T08:52:57.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.872+0000 7ff4fd7fa640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff4e80a3e70 0x7ff4e80a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.872+0000 7ff4fd7fa640 1 -- 172.21.15.7:0/2192333751 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7ff4f40f2a70 con 0x7ff500069830 2024-04-21T08:52:57.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.873+0000 7ff4ff7fe640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff4e80a3e70 0x7ff4e80a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.873+0000 7ff4ff7fe640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff4e80a3e70 0x7ff4e80a6330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff500112a50 tx=0x7ff4f00090e0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:57.881+0000 7ff4fd7fa640 1 -- 172.21.15.7:0/2192333751 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff4f40bed40 con 0x7ff500069830 2024-04-21T08:52:58.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.314+0000 7ff506775640 1 -- 172.21.15.7:0/2192333751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4ec005850 con 0x7ff500069830 2024-04-21T08:52:58.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.315+0000 7ff4fd7fa640 1 -- 172.21.15.7:0/2192333751 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7ff4f40ba020 con 0x7ff500069830 2024-04-21T08:52:58.315 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:52:58.315 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:52:58.315 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:52:58.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 -- 172.21.15.7:0/2192333751 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff4e80a3e70 msgr2=0x7ff4e80a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:58.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff4e80a3e70 0x7ff4e80a6330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff500112a50 tx=0x7ff4f00090e0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 -- 172.21.15.7:0/2192333751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff500069830 msgr2=0x7ff500115200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:58.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff500069830 0x7ff500115200 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4047040 tx=0x7ff4f4004270 comp rx=0 tx=0).stop 2024-04-21T08:52:58.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 -- 172.21.15.7:0/2192333751 shutdown_connections 2024-04-21T08:52:58.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff500106ed0 0x7ff5001117d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7ff4e80a3e70 0x7ff4e80a6330 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff500104a40 0x7ff500111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 --2- 172.21.15.7:0/2192333751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff500069830 0x7ff500115200 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 -- 172.21.15.7:0/2192333751 >> 172.21.15.7:0/2192333751 conn(0x7ff5000771d0 msgr2=0x7ff5000fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:58.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 -- 172.21.15.7:0/2192333751 shutdown_connections 2024-04-21T08:52:58.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.319+0000 7ff506775640 1 -- 172.21.15.7:0/2192333751 wait complete. 2024-04-21T08:52:58.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:52:58.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.495+0000 7f24a3493640 1 -- 172.21.15.7:0/1106026381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249c10c530 msgr2=0x7f249c10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:58.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.495+0000 7f24a3493640 1 --2- 172.21.15.7:0/1106026381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249c10c530 0x7f249c10e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2494009ca0 tx=0x7f249402c190 comp rx=0 tx=0).stop 2024-04-21T08:52:58.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.495+0000 7f24a3493640 1 -- 172.21.15.7:0/1106026381 shutdown_connections 2024-04-21T08:52:58.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.495+0000 7f24a3493640 1 --2- 172.21.15.7:0/1106026381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249c10c530 0x7f249c10e9a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.495+0000 7f24a3493640 1 --2- 172.21.15.7:0/1106026381 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f249c103a80 0x7f249c10bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.495+0000 7f24a3493640 1 --2- 172.21.15.7:0/1106026381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c103160 0x7f249c103540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.495+0000 7f24a3493640 1 -- 172.21.15.7:0/1106026381 >> 172.21.15.7:0/1106026381 conn(0x7f249c078060 msgr2=0x7f249c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:58.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.495+0000 7f24a3493640 1 -- 172.21.15.7:0/1106026381 shutdown_connections 2024-04-21T08:52:58.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.495+0000 7f24a3493640 1 -- 172.21.15.7:0/1106026381 wait complete. 2024-04-21T08:52:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.496+0000 7f24a3493640 1 Processor -- start 2024-04-21T08:52:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.496+0000 7f24a3493640 1 -- start start 2024-04-21T08:52:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a3493640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f249c103160 0x7f249c103fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a3493640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249c103a80 0x7f249c104510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a3493640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c10c530 0x7f249c104a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a3493640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f249c112190 con 0x7f249c10c530 2024-04-21T08:52:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a3493640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f249c112010 con 0x7f249c103160 2024-04-21T08:52:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a3493640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f249c112310 con 0x7f249c103a80 2024-04-21T08:52:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a0a2e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249c103a80 0x7f249c104510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a122f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f249c103160 0x7f249c103fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a122f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f249c103160 0x7f249c103fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:42236/0 (socket says 172.21.15.7:42236) 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a0a2e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249c103a80 0x7f249c104510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:51918/0 (socket says 172.21.15.7:51918) 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a122f640 1 -- 172.21.15.7:0/3136401883 learned_addr learned my addr 172.21.15.7:0/3136401883 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a1a30640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c10c530 0x7f249c104a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a122f640 1 -- 172.21.15.7:0/3136401883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249c103a80 msgr2=0x7f249c104510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a122f640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249c103a80 0x7f249c104510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a122f640 1 -- 172.21.15.7:0/3136401883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c10c530 msgr2=0x7f249c104a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a122f640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c10c530 0x7f249c104a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a122f640 1 -- 172.21.15.7:0/3136401883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249c1aafb0 con 0x7f249c103160 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.497+0000 7f24a0a2e640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249c103a80 0x7f249c104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.498+0000 7f24a122f640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f249c103160 0x7f249c103fd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f249800ea20 tx=0x7f249800eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.498+0000 7f24927fc640 1 -- 172.21.15.7:0/3136401883 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249800cd80 con 0x7f249c103160 2024-04-21T08:52:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.498+0000 7f24927fc640 1 -- 172.21.15.7:0/3136401883 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24980108e0 con 0x7f249c103160 2024-04-21T08:52:58.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.498+0000 7f24927fc640 1 -- 172.21.15.7:0/3136401883 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498018840 con 0x7f249c103160 2024-04-21T08:52:58.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.498+0000 7f24a3493640 1 -- 172.21.15.7:0/3136401883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f249c1ab2a0 con 0x7f249c103160 2024-04-21T08:52:58.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.498+0000 7f24a3493640 1 -- 172.21.15.7:0/3136401883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f249c1ab7f0 con 0x7f249c103160 2024-04-21T08:52:58.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.501+0000 7f24927fc640 1 -- 172.21.15.7:0/3136401883 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f2498010430 con 0x7f249c103160 2024-04-21T08:52:58.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.501+0000 7f24a6164740 1 Processor -- start 2024-04-21T08:52:58.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.501+0000 7f24a6164740 1 -- start start 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.501+0000 7f24a6164740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 0x558095fb38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.501+0000 7f24a6164740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 0x558095fbc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.501+0000 7f24a6164740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fbee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.501+0000 7f24a6164740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558095f181f0 con 0x558095fbca00 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.501+0000 7f24a6164740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558095eb54b0 con 0x558095fb34d0 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.501+0000 7f24a6164740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558095f141b0 con 0x558095fb3e80 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a0a2e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fbee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a1a30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 0x558095fb38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a122f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 0x558095fbc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a1a30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 0x558095fb38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:42244/0 (socket says 172.21.15.7:42244) 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a1a30640 1 -- 172.21.15.7:0/2163177574 learned_addr learned my addr 172.21.15.7:0/2163177574 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a0a2e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fbee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51056/0 (socket says 172.21.15.7:51056) 2024-04-21T08:52:58.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24927fc640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f24880a3d20 0x7f24880a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a1a30640 1 -- 172.21.15.7:0/2163177574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 msgr2=0x558095fbc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a1a30640 1 --2- 172.21.15.7:0/2163177574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 0x558095fbc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a1a30640 1 -- 172.21.15.7:0/2163177574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 msgr2=0x558095fbee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a1a30640 1 --2- 172.21.15.7:0/2163177574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fbee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a1a30640 1 -- 172.21.15.7:0/2163177574 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558095ea71b0 con 0x558095fb34d0 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24927fc640 1 -- 172.21.15.7:0/3136401883 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f24980d0fc0 con 0x7f249c103160 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a1a30640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f24880a3d20 0x7f24880a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.502+0000 7f24a1a30640 1 --2- 172.21.15.7:0/2163177574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 0x558095fb38b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2494004af0 tx=0x7f2494009910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39626765f15c570 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.503+0000 7f24877fe640 1 -- 172.21.15.7:0/2163177574 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494039040 con 0x558095fb34d0 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.503+0000 7f24a1a30640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f24880a3d20 0x7f24880a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2494006010 tx=0x7f2494045000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.503+0000 7f24877fe640 1 -- 172.21.15.7:0/2163177574 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2494004590 con 0x558095fb34d0 2024-04-21T08:52:58.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.503+0000 7f24877fe640 1 -- 172.21.15.7:0/2163177574 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249404a660 con 0x558095fb34d0 2024-04-21T08:52:58.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.503+0000 7f24a6164740 1 -- 172.21.15.7:0/2163177574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 msgr2=0x558095fb38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:58.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.503+0000 7f24a6164740 1 --2- 172.21.15.7:0/2163177574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 0x558095fb38b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2494004af0 tx=0x7f2494009910 comp rx=0 tx=0).stop 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.503+0000 7f24a6164740 1 -- 172.21.15.7:0/2163177574 shutdown_connections 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.503+0000 7f24a6164740 1 --2- 172.21.15.7:0/2163177574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fbee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.503+0000 7f24a6164740 1 --2- 172.21.15.7:0/2163177574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 0x558095fbc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.503+0000 7f24a6164740 1 --2- 172.21.15.7:0/2163177574 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 0x558095fb38b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.503+0000 7f24a6164740 1 -- 172.21.15.7:0/2163177574 >> 172.21.15.7:0/2163177574 conn(0x558095fadba0 msgr2=0x558095fadf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.504+0000 7f24a6164740 1 -- 172.21.15.7:0/2163177574 shutdown_connections 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.504+0000 7f24a6164740 1 -- 172.21.15.7:0/2163177574 wait complete. 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.504+0000 7f24a6164740 1 -- start start 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.504+0000 7f24a6164740 1 Processor -- start 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.505+0000 7f24a6164740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 0x558095fbc350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.505+0000 7f24a6164740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fb3e80 0x558095fb96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.505+0000 7f24a122f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 0x558095fbc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.505+0000 7f24a122f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 0x558095fbc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:51930/0 (socket says 172.21.15.7:51930) 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.505+0000 7f24a122f640 1 -- 172.21.15.7:0/3126052840 learned_addr learned my addr 172.21.15.7:0/3126052840 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.505+0000 7f24a6164740 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fbca00 0x558095fb9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.505+0000 7f24a6164740 1 -- 172.21.15.7:0/3126052840 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558095fbf970 con 0x558095fb3e80 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.505+0000 7f24a6164740 1 -- 172.21.15.7:0/3126052840 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558095ed1500 con 0x558095fbca00 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.505+0000 7f24a6164740 1 -- 172.21.15.7:0/3126052840 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558095fbfaf0 con 0x558095fb34d0 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.505+0000 7f24a6164740 10 client.0 _ll_register_callbacks cb 0x558095e9fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.505+0000 7f24a1a30640 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fbca00 0x558095fb9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f24a0a2e640 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fb3e80 0x558095fb96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f24a122f640 1 -- 172.21.15.7:0/3126052840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fbca00 msgr2=0x558095fb9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f24a122f640 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fbca00 0x558095fb9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f24a122f640 1 -- 172.21.15.7:0/3126052840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fb3e80 msgr2=0x558095fb96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f24a122f640 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fb3e80 0x558095fb96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f24a122f640 1 -- 172.21.15.7:0/3126052840 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558095c2a3f0 con 0x558095fb34d0 2024-04-21T08:52:58.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f24a0a2e640 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fb3e80 0x558095fb96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:58.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f24a1a30640 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fbca00 0x558095fb9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:58.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f24a122f640 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 0x558095fbc350 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f249800c6a0 tx=0x7f249809c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ea9a1ff2aeffa0e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:58.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f2484ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:52:58.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f2484ff9640 1 -- 172.21.15.7:0/3126052840 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24980ce6b0 con 0x558095fb34d0 2024-04-21T08:52:58.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f24a6164740 1 -- 172.21.15.7:0/3126052840 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2494004590 con 0x558095fb34d0 2024-04-21T08:52:58.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.506+0000 7f2484ff9640 1 -- 172.21.15.7:0/3126052840 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f24980a5260 con 0x558095fb34d0 2024-04-21T08:52:58.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.507+0000 7f2484ff9640 1 -- 172.21.15.7:0/3126052840 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f249809cb30 con 0x558095fb34d0 2024-04-21T08:52:58.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.507+0000 7f24a6164740 10 client.26578 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:58.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.507+0000 7f24a6164740 1 -- 172.21.15.7:0/3126052840 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558095cf47d0 con 0x558095fb34d0 2024-04-21T08:52:58.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.507+0000 7f2484ff9640 1 -- 172.21.15.7:0/3126052840 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24980a0e30 con 0x558095fb34d0 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.508+0000 7f2484ff9640 1 -- 172.21.15.7:0/3126052840 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f24980989a0 con 0x558095fb34d0 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.508+0000 7f24a6164740 10 client.26578 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.508+0000 7f24a6164740 10 client.26578 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.508+0000 7f24a6164740 20 client.26578 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.508+0000 7f24a6164740 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558095fcd160 0x558095fcf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.508+0000 7f24a6164740 4 client.26578 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.508+0000 7f24a6164740 1 -- 172.21.15.7:0/3126052840 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2494004590 con 0x558095fcd160 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.508+0000 7f24a0a2e640 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558095fcd160 0x558095fcf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.509+0000 7f24a0a2e640 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558095fcd160 0x558095fcf5c0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef7602611b581acd server_cookie=5e0d750312c60496 in_seq=0 out_seq=0 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.509+0000 7f2484ff9640 10 client.26578 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.510+0000 7f2484ff9640 1 -- 172.21.15.7:0/3126052840 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f2494004590 con 0x558095fcd160 2024-04-21T08:52:58.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.510+0000 7f2484ff9640 10 client.26578 handle_command_reply: tid=0 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.510+0000 7f24a6164740 1 client.26578 shutdown 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f24a6164740 1 -- 172.21.15.7:0/3126052840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 msgr2=0x558095fbc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f24a6164740 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 0x558095fbc350 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f249800c6a0 tx=0x7f249809c9c0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f24a6164740 1 -- 172.21.15.7:0/3126052840 shutdown_connections 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f24a6164740 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x558095fcd160 0x558095fcf5c0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f24a6164740 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fbca00 0x558095fb9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f24a6164740 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fb3e80 0x558095fb96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f24a6164740 1 --2- 172.21.15.7:0/3126052840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 0x558095fbc350 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f24a6164740 1 -- 172.21.15.7:0/3126052840 >> 172.21.15.7:0/3126052840 conn(0x558095fadba0 msgr2=0x558095faf8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f2484ff9640 0 client.26578 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f24a6164740 1 -- 172.21.15.7:0/3126052840 shutdown_connections 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f24a6164740 1 -- 172.21.15.7:0/3126052840 wait complete. 2024-04-21T08:52:58.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.511+0000 7f24a6164740 20 client.26578 trim_cache size 0 max 16384 2024-04-21T08:52:58.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.535+0000 7f24a6164740 1 Processor -- start 2024-04-21T08:52:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.536+0000 7f24a6164740 1 -- start start 2024-04-21T08:52:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.536+0000 7f24a6164740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 0x558095fc8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.536+0000 7f24a6164740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 0x558095fb9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.536+0000 7f24a6164740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fbbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.536+0000 7f24a6164740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1b3d70 con 0x558095fbca00 2024-04-21T08:52:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.536+0000 7f24a6164740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1b65f0 con 0x558095fb34d0 2024-04-21T08:52:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.536+0000 7f24a6164740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1b43e0 con 0x558095fb3e80 2024-04-21T08:52:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.536+0000 7f24a1a30640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fbbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.536+0000 7f24a0a2e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 0x558095fb9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.536+0000 7f24a0a2e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 0x558095fb9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:51946/0 (socket says 172.21.15.7:51946) 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.536+0000 7f24a0a2e640 1 -- 172.21.15.7:0/166776589 learned_addr learned my addr 172.21.15.7:0/166776589 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.537+0000 7f24a122f640 1 --2- 172.21.15.7:0/166776589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 0x558095fc8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.537+0000 7f24a0a2e640 1 -- 172.21.15.7:0/166776589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 msgr2=0x558095fc8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.537+0000 7f24a0a2e640 1 --2- 172.21.15.7:0/166776589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 0x558095fc8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.537+0000 7f24a0a2e640 1 -- 172.21.15.7:0/166776589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 msgr2=0x558095fbbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.537+0000 7f24a0a2e640 1 --2- 172.21.15.7:0/166776589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fbbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.537+0000 7f24a0a2e640 1 -- 172.21.15.7:0/166776589 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558095fb7c70 con 0x558095fb3e80 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.537+0000 7f24a0a2e640 1 --2- 172.21.15.7:0/166776589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 0x558095fb9680 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f248c000c00 tx=0x7f248c012e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed84e0492731f5df server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.537+0000 7f2486ffd640 1 -- 172.21.15.7:0/166776589 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c0122d0 con 0x558095fb3e80 2024-04-21T08:52:58.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.537+0000 7f2486ffd640 1 -- 172.21.15.7:0/166776589 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f248c012470 con 0x558095fb3e80 2024-04-21T08:52:58.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.537+0000 7f2486ffd640 1 -- 172.21.15.7:0/166776589 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c01f330 con 0x558095fb3e80 2024-04-21T08:52:58.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.538+0000 7f24a6164740 1 -- 172.21.15.7:0/166776589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 msgr2=0x558095fb9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:58.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.538+0000 7f24a6164740 1 --2- 172.21.15.7:0/166776589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 0x558095fb9680 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f248c000c00 tx=0x7f248c012e00 comp rx=0 tx=0).stop 2024-04-21T08:52:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.538+0000 7f24a6164740 1 -- 172.21.15.7:0/166776589 shutdown_connections 2024-04-21T08:52:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.538+0000 7f24a6164740 1 --2- 172.21.15.7:0/166776589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fbbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.538+0000 7f24a6164740 1 --2- 172.21.15.7:0/166776589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb3e80 0x558095fb9680 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.538+0000 7f24a6164740 1 --2- 172.21.15.7:0/166776589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb34d0 0x558095fc8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.538+0000 7f24a6164740 1 -- 172.21.15.7:0/166776589 >> 172.21.15.7:0/166776589 conn(0x558095fadba0 msgr2=0x558095fb8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.539+0000 7f24a6164740 1 -- 172.21.15.7:0/166776589 shutdown_connections 2024-04-21T08:52:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.539+0000 7f24a6164740 1 -- 172.21.15.7:0/166776589 wait complete. 2024-04-21T08:52:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.539+0000 7f24a6164740 1 -- start start 2024-04-21T08:52:58.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.540+0000 7f24a6164740 1 Processor -- start 2024-04-21T08:52:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.540+0000 7f24a6164740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 0x558095fc4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.540+0000 7f24a6164740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb3e80 0x558095fc5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.540+0000 7f24a6164740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fb5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.540+0000 7f24a6164740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1b4ab0 con 0x558095fbca00 2024-04-21T08:52:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.540+0000 7f24a6164740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1b5f20 con 0x558095fb3e80 2024-04-21T08:52:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.540+0000 7f24a6164740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1b5180 con 0x558095fb34d0 2024-04-21T08:52:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.540+0000 7f24a6164740 10 client.0 _ll_register_callbacks cb 0x7f24980d4760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.541+0000 7f24a1a30640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fb5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.541+0000 7f24a1a30640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fb5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51088/0 (socket says 172.21.15.7:51088) 2024-04-21T08:52:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.541+0000 7f24a1a30640 1 -- 172.21.15.7:0/1572605931 learned_addr learned my addr 172.21.15.7:0/1572605931 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:58.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.541+0000 7f24a122f640 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 0x558095fc4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.541+0000 7f24a0a2e640 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb3e80 0x558095fc5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.541+0000 7f24a1a30640 1 -- 172.21.15.7:0/1572605931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 msgr2=0x558095fc4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.541+0000 7f24a1a30640 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 0x558095fc4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.541+0000 7f24a1a30640 1 -- 172.21.15.7:0/1572605931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb3e80 msgr2=0x558095fc5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.541+0000 7f24a1a30640 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb3e80 0x558095fc5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.541+0000 7f24a1a30640 1 -- 172.21.15.7:0/1572605931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558095fb6370 con 0x558095fbca00 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.542+0000 7f24a0a2e640 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb3e80 0x558095fc5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.542+0000 7f24a1a30640 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fb5b70 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f24940098e0 tx=0x7f2494032440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc420d06b7d8630 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.542+0000 7f2484ff9640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.542+0000 7f2484ff9640 1 -- 172.21.15.7:0/1572605931 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494059070 con 0x558095fbca00 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.542+0000 7f2484ff9640 1 -- 172.21.15.7:0/1572605931 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f2494004790 con 0x558095fbca00 2024-04-21T08:52:58.543 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.542+0000 7f24a6164740 1 -- 172.21.15.7:0/1572605931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f248c012470 con 0x558095fbca00 2024-04-21T08:52:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.543+0000 7f2484ff9640 1 -- 172.21.15.7:0/1572605931 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2494065050 con 0x558095fbca00 2024-04-21T08:52:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.543+0000 7f24a6164740 10 client.17052 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.543+0000 7f24a6164740 1 -- 172.21.15.7:0/1572605931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558096044280 con 0x558095fbca00 2024-04-21T08:52:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.543+0000 7f2484ff9640 1 -- 172.21.15.7:0/1572605931 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f248c012470 con 0x558095fbca00 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.543+0000 7f2484ff9640 1 -- 172.21.15.7:0/1572605931 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f249403f040 con 0x558095fbca00 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.543+0000 7f24a6164740 10 client.17052 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.543+0000 7f24a6164740 10 client.17052 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.543+0000 7f24a6164740 20 client.17052 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.543+0000 7f24a6164740 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5580960445b0 0x5580960469a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.543+0000 7f24a6164740 4 client.17052 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.543+0000 7f24a6164740 1 -- 172.21.15.7:0/1572605931 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558095eb7930 con 0x5580960445b0 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.544+0000 7f24a122f640 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5580960445b0 0x5580960469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.544+0000 7f24a122f640 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5580960445b0 0x5580960469a0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f33f05558ff616dc server_cookie=420771aae552651c in_seq=0 out_seq=0 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.544+0000 7f2484ff9640 10 client.17052 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.544+0000 7f2484ff9640 1 -- 172.21.15.7:0/1572605931 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558095eb7930 con 0x5580960445b0 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f2484ff9640 10 client.17052 handle_command_reply: tid=0 2024-04-21T08:52:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 1 client.17052 shutdown 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 1 -- 172.21.15.7:0/1572605931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 msgr2=0x558095fb5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fb5b70 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f24940098e0 tx=0x7f2494032440 comp rx=0 tx=0).stop 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 1 -- 172.21.15.7:0/1572605931 shutdown_connections 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5580960445b0 0x5580960469a0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558095fbca00 0x558095fb5b70 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558095fb3e80 0x558095fc5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 1 --2- 172.21.15.7:0/1572605931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558095fb34d0 0x558095fc4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 1 -- 172.21.15.7:0/1572605931 >> 172.21.15.7:0/1572605931 conn(0x558095fadba0 msgr2=0x558095fac370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f2484ff9640 0 client.17052 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 1 -- 172.21.15.7:0/1572605931 shutdown_connections 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 1 -- 172.21.15.7:0/1572605931 wait complete. 2024-04-21T08:52:58.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.545+0000 7f24a6164740 20 client.17052 trim_cache size 0 max 16384 2024-04-21T08:52:58.548 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.546+0000 7f24a3493640 1 -- 172.21.15.7:0/3136401883 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f24880a3d20 msgr2=0x7f24880a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.546+0000 7f24a3493640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f24880a3d20 0x7f24880a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2494006010 tx=0x7f2494045000 comp rx=0 tx=0).stop 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.546+0000 7f24a3493640 1 -- 172.21.15.7:0/3136401883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f249c103160 msgr2=0x7f249c103fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.546+0000 7f24a3493640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f249c103160 0x7f249c103fd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f249800ea20 tx=0x7f249800eee0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.546+0000 7f24a3493640 1 -- 172.21.15.7:0/3136401883 shutdown_connections 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.546+0000 7f24a3493640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f249c10c530 0x7f249c104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.546+0000 7f24a3493640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f24880a3d20 0x7f24880a61e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.546+0000 7f24a3493640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f249c103a80 0x7f249c104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.546+0000 7f24a3493640 1 --2- 172.21.15.7:0/3136401883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f249c103160 0x7f249c103fd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.546+0000 7f24a3493640 1 -- 172.21.15.7:0/3136401883 >> 172.21.15.7:0/3136401883 conn(0x7f249c078060 msgr2=0x7f249c108450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.546+0000 7f24a3493640 1 -- 172.21.15.7:0/3136401883 shutdown_connections 2024-04-21T08:52:58.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:58.547+0000 7f24a3493640 1 -- 172.21.15.7:0/3136401883 wait complete. 2024-04-21T08:52:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:58 smithi016 ceph-mon[31259]: pgmap v256: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 892 KiB/s wr, 1.34k op/s 2024-04-21T08:52:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:52:58 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2192333751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:52:59.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:58 smithi077 ceph-mon[31615]: pgmap v256: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 892 KiB/s wr, 1.34k op/s 2024-04-21T08:52:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:52:58 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2192333751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:52:59.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:58 smithi007 ceph-mon[28477]: pgmap v256: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 892 KiB/s wr, 1.34k op/s 2024-04-21T08:52:59.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:52:58 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2192333751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:52:59.563 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:52:59.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:52:59.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:52:59.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.707+0000 7f898dc14640 1 -- 172.21.15.7:0/1465399969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8988103ae0 msgr2=0x7f898810bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.707+0000 7f898dc14640 1 --2- 172.21.15.7:0/1465399969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8988103ae0 0x7f898810bfc0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f897c009860 tx=0x7f897c02bc10 comp rx=0 tx=0).stop 2024-04-21T08:52:59.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.708+0000 7f898dc14640 1 -- 172.21.15.7:0/1465399969 shutdown_connections 2024-04-21T08:52:59.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.708+0000 7f898dc14640 1 --2- 172.21.15.7:0/1465399969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f898810c500 0x7f898810e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.708+0000 7f898dc14640 1 --2- 172.21.15.7:0/1465399969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8988103ae0 0x7f898810bfc0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.708+0000 7f898dc14640 1 --2- 172.21.15.7:0/1465399969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89881031c0 0x7f89881035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.708+0000 7f898dc14640 1 -- 172.21.15.7:0/1465399969 >> 172.21.15.7:0/1465399969 conn(0x7f8988078000 msgr2=0x7f8988078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.708+0000 7f898dc14640 1 -- 172.21.15.7:0/1465399969 shutdown_connections 2024-04-21T08:52:59.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.708+0000 7f898dc14640 1 -- 172.21.15.7:0/1465399969 wait complete. 2024-04-21T08:52:59.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.709+0000 7f898dc14640 1 Processor -- start 2024-04-21T08:52:59.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.709+0000 7f898dc14640 1 -- start start 2024-04-21T08:52:59.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.709+0000 7f898dc14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89881031c0 0x7f898819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.709+0000 7f898dc14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8988103ae0 0x7f898819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.709+0000 7f898dc14640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f898810c500 0x7f89881985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.709+0000 7f898dc14640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f89881121c0 con 0x7f898810c500 2024-04-21T08:52:59.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.709+0000 7f898dc14640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8988112040 con 0x7f89881031c0 2024-04-21T08:52:59.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.709+0000 7f898dc14640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8988112340 con 0x7f8988103ae0 2024-04-21T08:52:59.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.709+0000 7f8986ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8988103ae0 0x7f898819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.709+0000 7f8986ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8988103ae0 0x7f898819e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:51974/0 (socket says 172.21.15.7:51974) 2024-04-21T08:52:59.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.709+0000 7f8986ffd640 1 -- 172.21.15.7:0/2453204139 learned_addr learned my addr 172.21.15.7:0/2453204139 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:59.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.710+0000 7f89877fe640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89881031c0 0x7f898819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.710+0000 7f8987fff640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f898810c500 0x7f89881985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.710+0000 7f8986ffd640 1 -- 172.21.15.7:0/2453204139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89881031c0 msgr2=0x7f898819e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.710+0000 7f8986ffd640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89881031c0 0x7f898819e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.710+0000 7f8986ffd640 1 -- 172.21.15.7:0/2453204139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f898810c500 msgr2=0x7f89881985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.710+0000 7f8986ffd640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f898810c500 0x7f89881985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.710+0000 7f8986ffd640 1 -- 172.21.15.7:0/2453204139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8988198d40 con 0x7f8988103ae0 2024-04-21T08:52:59.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.711+0000 7f8986ffd640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8988103ae0 0x7f898819e950 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f897c037040 tx=0x7f897c02cee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.712+0000 7f8984ff9640 1 -- 172.21.15.7:0/2453204139 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c039040 con 0x7f8988103ae0 2024-04-21T08:52:59.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.712+0000 7f8984ff9640 1 -- 172.21.15.7:0/2453204139 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f897c004480 con 0x7f8988103ae0 2024-04-21T08:52:59.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.712+0000 7f8984ff9640 1 -- 172.21.15.7:0/2453204139 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c0035f0 con 0x7f8988103ae0 2024-04-21T08:52:59.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.712+0000 7f898dc14640 1 -- 172.21.15.7:0/2453204139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8988198fd0 con 0x7f8988103ae0 2024-04-21T08:52:59.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.712+0000 7f898dc14640 1 -- 172.21.15.7:0/2453204139 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89881993b0 con 0x7f8988103ae0 2024-04-21T08:52:59.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.715+0000 7f8984ff9640 1 -- 172.21.15.7:0/2453204139 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f897c03d5f0 con 0x7f8988103ae0 2024-04-21T08:52:59.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.715+0000 7f8990807740 1 Processor -- start 2024-04-21T08:52:59.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f8990807740 1 -- start start 2024-04-21T08:52:59.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f8990807740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x559072689c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f8990807740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907268a230 0x55907269b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f8990807740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907269b620 0x55907269da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f8990807740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5590725edf00 con 0x55907268a230 2024-04-21T08:52:59.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f8990807740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55907258b590 con 0x559072689880 2024-04-21T08:52:59.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f8990807740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5590725e9ec0 con 0x55907269b620 2024-04-21T08:52:59.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f8984ff9640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89700a3f20 0x7f89700a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f89877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907268a230 0x55907269b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f89877fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907268a230 0x55907269b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51110/0 (socket says 172.21.15.7:51110) 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f89877fe640 1 -- 172.21.15.7:0/2214217637 learned_addr learned my addr 172.21.15.7:0/2214217637 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f8987fff640 1 --2- 172.21.15.7:0/2214217637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x559072689c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.716+0000 7f8984ff9640 1 -- 172.21.15.7:0/2453204139 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f897c0f6110 con 0x7f8988103ae0 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.717+0000 7f8987fff640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89700a3f20 0x7f89700a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.717+0000 7f89877fe640 1 -- 172.21.15.7:0/2214217637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907269b620 msgr2=0x55907269da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.717+0000 7f89877fe640 1 --2- 172.21.15.7:0/2214217637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907269b620 0x55907269da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.717+0000 7f89877fe640 1 -- 172.21.15.7:0/2214217637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 msgr2=0x559072689c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.717+0000 7f89877fe640 1 --2- 172.21.15.7:0/2214217637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x559072689c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.717+0000 7f89877fe640 1 -- 172.21.15.7:0/2214217637 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590723c87e0 con 0x55907268a230 2024-04-21T08:52:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.717+0000 7f8987fff640 1 --2- 172.21.15.7:0/2214217637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x559072689c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:59.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.717+0000 7f89877fe640 1 --2- 172.21.15.7:0/2214217637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907268a230 0x55907269b050 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f897800b740 tx=0x7f897800bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb54f71b01060255 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.717+0000 7f8987fff640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89700a3f20 0x7f89700a63e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8980009fd0 tx=0x7f8980005bd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.718+0000 7f896dffb640 1 -- 172.21.15.7:0/2214217637 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8978003dd0 con 0x55907268a230 2024-04-21T08:52:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.718+0000 7f896dffb640 1 -- 172.21.15.7:0/2214217637 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8978015070 con 0x55907268a230 2024-04-21T08:52:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.718+0000 7f896dffb640 1 -- 172.21.15.7:0/2214217637 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8978011730 con 0x55907268a230 2024-04-21T08:52:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.719+0000 7f8990807740 1 -- 172.21.15.7:0/2214217637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907268a230 msgr2=0x55907269b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.719+0000 7f8990807740 1 --2- 172.21.15.7:0/2214217637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907268a230 0x55907269b050 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f897800b740 tx=0x7f897800bc00 comp rx=0 tx=0).stop 2024-04-21T08:52:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.719+0000 7f8990807740 1 -- 172.21.15.7:0/2214217637 shutdown_connections 2024-04-21T08:52:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.719+0000 7f8990807740 1 --2- 172.21.15.7:0/2214217637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907269b620 0x55907269da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.719+0000 7f8990807740 1 --2- 172.21.15.7:0/2214217637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907268a230 0x55907269b050 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.719+0000 7f8990807740 1 --2- 172.21.15.7:0/2214217637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x559072689c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.719+0000 7f8990807740 1 -- 172.21.15.7:0/2214217637 >> 172.21.15.7:0/2214217637 conn(0x559072683fb0 msgr2=0x559072684390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.719+0000 7f8990807740 1 -- 172.21.15.7:0/2214217637 shutdown_connections 2024-04-21T08:52:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.719+0000 7f8990807740 1 -- 172.21.15.7:0/2214217637 wait complete. 2024-04-21T08:52:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.720+0000 7f8990807740 1 -- start start 2024-04-21T08:52:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.720+0000 7f8990807740 1 Processor -- start 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f8990807740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x5590726952d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f8990807740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x55907269aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f8990807740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726906d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f8990807740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55907268abe0 con 0x55907269b620 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f8990807740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5590725a7580 con 0x559072689880 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f8990807740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55907268ad60 con 0x55907268a230 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f8990807740 10 client.0 _ll_register_callbacks cb 0x5590725ad650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f8987fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726906d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f89877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x5590726952d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f8987fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726906d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51116/0 (socket says 172.21.15.7:51116) 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f8987fff640 1 -- 172.21.15.7:0/1559229618 learned_addr learned my addr 172.21.15.7:0/1559229618 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.721+0000 7f8986ffd640 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x55907269aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8987fff640 1 -- 172.21.15.7:0/1559229618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 msgr2=0x55907269aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8987fff640 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x55907269aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8987fff640 1 -- 172.21.15.7:0/1559229618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 msgr2=0x5590726952d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8987fff640 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x5590726952d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8987fff640 1 -- 172.21.15.7:0/1559229618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559072428f70 con 0x55907269b620 2024-04-21T08:52:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f89877fe640 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x5590726952d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8986ffd640 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x55907269aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8987fff640 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726906d0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f898000db40 tx=0x7f8980004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3c75b24db8239b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8966ffd640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:52:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8966ffd640 1 -- 172.21.15.7:0/1559229618 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8980024030 con 0x55907269b620 2024-04-21T08:52:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8966ffd640 1 -- 172.21.15.7:0/1559229618 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f898001b070 con 0x55907269b620 2024-04-21T08:52:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8990807740 1 -- 172.21.15.7:0/1559229618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8978015070 con 0x55907269b620 2024-04-21T08:52:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8966ffd640 1 -- 172.21.15.7:0/1559229618 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8980031050 con 0x55907269b620 2024-04-21T08:52:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.722+0000 7f8990807740 10 client.17064 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.723+0000 7f8990807740 1 -- 172.21.15.7:0/1559229618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5590722c0170 con 0x55907269b620 2024-04-21T08:52:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.723+0000 7f8966ffd640 1 -- 172.21.15.7:0/1559229618 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8978015070 con 0x55907269b620 2024-04-21T08:52:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.723+0000 7f8966ffd640 1 -- 172.21.15.7:0/1559229618 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f898001e040 con 0x55907269b620 2024-04-21T08:52:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.723+0000 7f8990807740 10 client.17064 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.723+0000 7f8990807740 10 client.17064 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.723+0000 7f8990807740 20 client.17064 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.723+0000 7f8990807740 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590726b4400 0x5590726b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.723+0000 7f8990807740 4 client.17064 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.723+0000 7f8990807740 1 -- 172.21.15.7:0/1559229618 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590725fb920 con 0x5590726b4400 2024-04-21T08:52:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.724+0000 7f89877fe640 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590726b4400 0x5590726b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.724+0000 7f8966ffd640 10 client.17064 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.724+0000 7f89877fe640 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590726b4400 0x5590726b6810 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c11ceea5207ec145 server_cookie=dce316418e0ba825 in_seq=0 out_seq=0 2024-04-21T08:52:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.725+0000 7f8966ffd640 1 -- 172.21.15.7:0/1559229618 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590725fb920 con 0x5590726b4400 2024-04-21T08:52:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.725+0000 7f8966ffd640 10 client.17064 handle_command_reply: tid=0 2024-04-21T08:52:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.725+0000 7f8990807740 1 client.17064 shutdown 2024-04-21T08:52:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.725+0000 7f8990807740 1 -- 172.21.15.7:0/1559229618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 msgr2=0x5590726906d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.725+0000 7f8990807740 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726906d0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f898000db40 tx=0x7f8980004a40 comp rx=0 tx=0).stop 2024-04-21T08:52:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.726+0000 7f8990807740 1 -- 172.21.15.7:0/1559229618 shutdown_connections 2024-04-21T08:52:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.726+0000 7f8990807740 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590726b4400 0x5590726b6810 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.726+0000 7f8990807740 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726906d0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.726+0000 7f8966ffd640 0 client.17064 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.726+0000 7f8990807740 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x55907269aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.726+0000 7f8990807740 1 --2- 172.21.15.7:0/1559229618 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x5590726952d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.726+0000 7f8990807740 1 -- 172.21.15.7:0/1559229618 >> 172.21.15.7:0/1559229618 conn(0x559072683fb0 msgr2=0x559072685d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.726+0000 7f8990807740 1 -- 172.21.15.7:0/1559229618 shutdown_connections 2024-04-21T08:52:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.726+0000 7f8990807740 1 -- 172.21.15.7:0/1559229618 wait complete. 2024-04-21T08:52:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.726+0000 7f8990807740 20 client.17064 trim_cache size 0 max 16384 2024-04-21T08:52:59.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.751+0000 7f8990807740 1 Processor -- start 2024-04-21T08:52:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.751+0000 7f8990807740 1 -- start start 2024-04-21T08:52:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.752+0000 7f8990807740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x559072685b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.752+0000 7f8990807740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x5590726bc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.752+0000 7f8990807740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726bc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.752+0000 7f89877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x559072685b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.752+0000 7f89877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x559072685b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:42320/0 (socket says 172.21.15.7:42320) 2024-04-21T08:52:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.752+0000 7f8987fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726bc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.752+0000 7f8987fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726bc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51124/0 (socket says 172.21.15.7:51124) 2024-04-21T08:52:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.752+0000 7f89877fe640 1 -- 172.21.15.7:0/3474870379 learned_addr learned my addr 172.21.15.7:0/3474870379 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.752+0000 7f8990807740 1 -- 172.21.15.7:0/3474870379 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f89881ab3f0 con 0x55907269b620 2024-04-21T08:52:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.752+0000 7f8990807740 1 -- 172.21.15.7:0/3474870379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89881adc70 con 0x559072689880 2024-04-21T08:52:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.752+0000 7f8990807740 1 -- 172.21.15.7:0/3474870379 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f89881aba60 con 0x55907268a230 2024-04-21T08:52:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.753+0000 7f8986ffd640 1 --2- 172.21.15.7:0/3474870379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x5590726bc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.753+0000 7f89877fe640 1 -- 172.21.15.7:0/3474870379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 msgr2=0x5590726bc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.753+0000 7f89877fe640 1 --2- 172.21.15.7:0/3474870379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x5590726bc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.753+0000 7f89877fe640 1 -- 172.21.15.7:0/3474870379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 msgr2=0x5590726bc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.753+0000 7f89877fe640 1 --2- 172.21.15.7:0/3474870379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726bc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.753+0000 7f89877fe640 1 -- 172.21.15.7:0/3474870379 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559072684b80 con 0x559072689880 2024-04-21T08:52:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.753+0000 7f89877fe640 1 --2- 172.21.15.7:0/3474870379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x559072685b10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f897801af00 tx=0x7f897801ae90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc43c3d598d0f2c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.754+0000 7f896d7fa640 1 -- 172.21.15.7:0/3474870379 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8978005040 con 0x559072689880 2024-04-21T08:52:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.754+0000 7f896d7fa640 1 -- 172.21.15.7:0/3474870379 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8978015030 con 0x559072689880 2024-04-21T08:52:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.754+0000 7f896d7fa640 1 -- 172.21.15.7:0/3474870379 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8978028860 con 0x559072689880 2024-04-21T08:52:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.755+0000 7f8990807740 1 -- 172.21.15.7:0/3474870379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 msgr2=0x559072685b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.755+0000 7f8990807740 1 --2- 172.21.15.7:0/3474870379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x559072685b10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f897801af00 tx=0x7f897801ae90 comp rx=0 tx=0).stop 2024-04-21T08:52:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.755+0000 7f8990807740 1 -- 172.21.15.7:0/3474870379 shutdown_connections 2024-04-21T08:52:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.755+0000 7f8990807740 1 --2- 172.21.15.7:0/3474870379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726bc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.755+0000 7f8990807740 1 --2- 172.21.15.7:0/3474870379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x5590726bc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.755+0000 7f8990807740 1 --2- 172.21.15.7:0/3474870379 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x559072685b10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.755+0000 7f8990807740 1 -- 172.21.15.7:0/3474870379 >> 172.21.15.7:0/3474870379 conn(0x559072683fb0 msgr2=0x559072685400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.755+0000 7f8990807740 1 -- 172.21.15.7:0/3474870379 shutdown_connections 2024-04-21T08:52:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.755+0000 7f8990807740 1 -- 172.21.15.7:0/3474870379 wait complete. 2024-04-21T08:52:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.755+0000 7f8990807740 1 -- start start 2024-04-21T08:52:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.756+0000 7f8990807740 1 Processor -- start 2024-04-21T08:52:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.756+0000 7f8990807740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x5590726c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.756+0000 7f8990807740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x5590726c2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f8990807740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726c33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f89877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x5590726c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f89877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x5590726c7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:42330/0 (socket says 172.21.15.7:42330) 2024-04-21T08:52:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f8986ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x5590726c2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f8986ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x5590726c2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:52006/0 (socket says 172.21.15.7:52006) 2024-04-21T08:52:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f89877fe640 1 -- 172.21.15.7:0/2404479394 learned_addr learned my addr 172.21.15.7:0/2404479394 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f8990807740 1 -- 172.21.15.7:0/2404479394 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f89881ac130 con 0x55907269b620 2024-04-21T08:52:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f8990807740 1 -- 172.21.15.7:0/2404479394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89881ad5a0 con 0x559072689880 2024-04-21T08:52:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f8990807740 1 -- 172.21.15.7:0/2404479394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f89881ac800 con 0x55907268a230 2024-04-21T08:52:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f8990807740 10 client.0 _ll_register_callbacks cb 0x7f89800127e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f8987fff640 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726c33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f89877fe640 1 -- 172.21.15.7:0/2404479394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 msgr2=0x5590726c2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f89877fe640 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x5590726c2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f89877fe640 1 -- 172.21.15.7:0/2404479394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 msgr2=0x5590726c33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f89877fe640 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726c33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.757+0000 7f89877fe640 1 -- 172.21.15.7:0/2404479394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559072691640 con 0x559072689880 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.758+0000 7f89877fe640 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x5590726c7aa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f897800b4e0 tx=0x7f8978028230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a1b43aa7f843867 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.758+0000 7f8966ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.758+0000 7f8966ffd640 1 -- 172.21.15.7:0/2404479394 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8978028760 con 0x559072689880 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.758+0000 7f8966ffd640 1 -- 172.21.15.7:0/2404479394 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7f8978005030 con 0x559072689880 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.758+0000 7f8990807740 1 -- 172.21.15.7:0/2404479394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8978015030 con 0x559072689880 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.758+0000 7f8966ffd640 1 -- 172.21.15.7:0/2404479394 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8978028900 con 0x559072689880 2024-04-21T08:52:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.758+0000 7f8990807740 10 client.26624 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:59.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.758+0000 7f8990807740 1 -- 172.21.15.7:0/2404479394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55907271a2a0 con 0x559072689880 2024-04-21T08:52:59.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.758+0000 7f8966ffd640 1 -- 172.21.15.7:0/2404479394 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f897800be50 con 0x559072689880 2024-04-21T08:52:59.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.759+0000 7f8966ffd640 1 -- 172.21.15.7:0/2404479394 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f8978004600 con 0x559072689880 2024-04-21T08:52:59.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.759+0000 7f8990807740 10 client.26624 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:59.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.759+0000 7f8990807740 10 client.26624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:59.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.759+0000 7f8990807740 20 client.26624 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:59.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.759+0000 7f8990807740 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590726b4400 0x55907271c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.760+0000 7f8990807740 4 client.26624 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:52:59.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.760+0000 7f8990807740 1 -- 172.21.15.7:0/2404479394 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f8978015030 con 0x5590726b4400 2024-04-21T08:52:59.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.760+0000 7f8986ffd640 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590726b4400 0x55907271c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.760+0000 7f8986ffd640 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590726b4400 0x55907271c730 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdeee29d2506ea0b server_cookie=431dddd4af27285e in_seq=0 out_seq=0 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.760+0000 7f8966ffd640 10 client.26624 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8966ffd640 1 -- 172.21.15.7:0/2404479394 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f8978015030 con 0x5590726b4400 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8966ffd640 10 client.26624 handle_command_reply: tid=0 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8990807740 1 client.26624 shutdown 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8990807740 1 -- 172.21.15.7:0/2404479394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 msgr2=0x5590726c7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8990807740 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x5590726c7aa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f897800b4e0 tx=0x7f8978028230 comp rx=0 tx=0).stop 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8990807740 1 -- 172.21.15.7:0/2404479394 shutdown_connections 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8990807740 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5590726b4400 0x55907271c730 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8966ffd640 0 client.26624 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8990807740 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55907269b620 0x5590726c33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8990807740 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55907268a230 0x5590726c2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8990807740 1 --2- 172.21.15.7:0/2404479394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559072689880 0x5590726c7aa0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.763+0000 7f8990807740 1 -- 172.21.15.7:0/2404479394 >> 172.21.15.7:0/2404479394 conn(0x559072683fb0 msgr2=0x559072685c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.764+0000 7f8990807740 1 -- 172.21.15.7:0/2404479394 shutdown_connections 2024-04-21T08:52:59.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.764+0000 7f8990807740 1 -- 172.21.15.7:0/2404479394 wait complete. 2024-04-21T08:52:59.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.764+0000 7f8990807740 20 client.26624 trim_cache size 0 max 16384 2024-04-21T08:52:59.766 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:52:59.766 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:52:59.766 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "2f44a19d-b0a7-4681-b27b-fb7529461620", 2024-04-21T08:52:59.766 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:52:59.767 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:52:59.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 -- 172.21.15.7:0/2453204139 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89700a3f20 msgr2=0x7f89700a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89700a3f20 0x7f89700a63e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8980009fd0 tx=0x7f8980005bd0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 -- 172.21.15.7:0/2453204139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8988103ae0 msgr2=0x7f898819e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8988103ae0 0x7f898819e950 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f897c037040 tx=0x7f897c02cee0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 -- 172.21.15.7:0/2453204139 shutdown_connections 2024-04-21T08:52:59.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f898810c500 0x7f89881985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89700a3f20 0x7f89700a63e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8988103ae0 0x7f898819e950 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 --2- 172.21.15.7:0/2453204139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89881031c0 0x7f898819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 -- 172.21.15.7:0/2453204139 >> 172.21.15.7:0/2453204139 conn(0x7f8988078000 msgr2=0x7f89880fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 -- 172.21.15.7:0/2453204139 shutdown_connections 2024-04-21T08:52:59.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.765+0000 7f898dc14640 1 -- 172.21.15.7:0/2453204139 wait complete. 2024-04-21T08:52:59.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:52:59.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:52:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.924+0000 7fb64d42e640 1 -- 172.21.15.7:0/1580313791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb64806abb0 msgr2=0x7fb64810e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.924+0000 7fb64d42e640 1 --2- 172.21.15.7:0/1580313791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb64806abb0 0x7fb64810e3b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb630009900 tx=0x7fb63002bc30 comp rx=0 tx=0).stop 2024-04-21T08:52:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.924+0000 7fb64d42e640 1 -- 172.21.15.7:0/1580313791 shutdown_connections 2024-04-21T08:52:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.924+0000 7fb64d42e640 1 --2- 172.21.15.7:0/1580313791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64810e8f0 0x7fb648110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.924+0000 7fb64d42e640 1 --2- 172.21.15.7:0/1580313791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb64806abb0 0x7fb64810e3b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.924+0000 7fb64d42e640 1 --2- 172.21.15.7:0/1580313791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb64806a290 0x7fb64806a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.924+0000 7fb64d42e640 1 -- 172.21.15.7:0/1580313791 >> 172.21.15.7:0/1580313791 conn(0x7fb648077f40 msgr2=0x7fb648078350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.924+0000 7fb64d42e640 1 -- 172.21.15.7:0/1580313791 shutdown_connections 2024-04-21T08:52:59.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.924+0000 7fb64d42e640 1 -- 172.21.15.7:0/1580313791 wait complete. 2024-04-21T08:52:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.925+0000 7fb64d42e640 1 Processor -- start 2024-04-21T08:52:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.925+0000 7fb64d42e640 1 -- start start 2024-04-21T08:52:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.925+0000 7fb64d42e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb64806a290 0x7fb6481a7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.925+0000 7fb64d42e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64806abb0 0x7fb6481a7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.925+0000 7fb64d42e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb64810e8f0 0x7fb6481abe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.925+0000 7fb64d42e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb6481057e0 con 0x7fb64806abb0 2024-04-21T08:52:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.925+0000 7fb64d42e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb648105660 con 0x7fb64806a290 2024-04-21T08:52:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.925+0000 7fb64d42e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb648105960 con 0x7fb64810e8f0 2024-04-21T08:52:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb6467fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64806abb0 0x7fb6481a7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb64810e8f0 0x7fb6481abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb6467fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64806abb0 0x7fb6481a7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51156/0 (socket says 172.21.15.7:51156) 2024-04-21T08:52:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb6467fc640 1 -- 172.21.15.7:0/992575715 learned_addr learned my addr 172.21.15.7:0/992575715 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb646ffd640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb64806a290 0x7fb6481a7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb6467fc640 1 -- 172.21.15.7:0/992575715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb64810e8f0 msgr2=0x7fb6481abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb6467fc640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb64810e8f0 0x7fb6481abe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb6467fc640 1 -- 172.21.15.7:0/992575715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb64806a290 msgr2=0x7fb6481a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb6467fc640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb64806a290 0x7fb6481a7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb6467fc640 1 -- 172.21.15.7:0/992575715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6481ac580 con 0x7fb64806abb0 2024-04-21T08:52:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb6477fe640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb64810e8f0 0x7fb6481abe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:59.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.926+0000 7fb6467fc640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64806abb0 0x7fb6481a7f80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb630005ab0 tx=0x7fb630004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.927+0000 7fb63ffff640 1 -- 172.21.15.7:0/992575715 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb630043020 con 0x7fb64806abb0 2024-04-21T08:52:59.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.927+0000 7fb63ffff640 1 -- 172.21.15.7:0/992575715 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb63002ca70 con 0x7fb64806abb0 2024-04-21T08:52:59.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.927+0000 7fb64d42e640 1 -- 172.21.15.7:0/992575715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6481ac810 con 0x7fb64806abb0 2024-04-21T08:52:59.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.927+0000 7fb63ffff640 1 -- 172.21.15.7:0/992575715 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6300039e0 con 0x7fb64806abb0 2024-04-21T08:52:59.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.927+0000 7fb64d42e640 1 -- 172.21.15.7:0/992575715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6481b4100 con 0x7fb64806abb0 2024-04-21T08:52:59.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.930+0000 7fb650016740 1 Processor -- start 2024-04-21T08:52:59.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.930+0000 7fb650016740 1 -- start start 2024-04-21T08:52:59.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.930+0000 7fb63ffff640 1 -- 172.21.15.7:0/992575715 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fb630046020 con 0x7fb64806abb0 2024-04-21T08:52:59.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb650016740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4ee580 0x56109e4ee960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb650016740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e4f7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb650016740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4f7a80 0x56109e4f9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb650016740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56109e4531e0 con 0x56109e4eeea0 2024-04-21T08:52:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb650016740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56109e3f0390 con 0x56109e4f7a80 2024-04-21T08:52:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb650016740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56109e44f1a0 con 0x56109e4ee580 2024-04-21T08:52:59.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4ee580 0x56109e4ee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4ee580 0x56109e4ee960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:52032/0 (socket says 172.21.15.7:52032) 2024-04-21T08:52:59.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb6477fe640 1 -- 172.21.15.7:0/754035594 learned_addr learned my addr 172.21.15.7:0/754035594 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:59.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb6467fc640 1 --2- 172.21.15.7:0/754035594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4f7a80 0x56109e4f9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb646ffd640 1 --2- 172.21.15.7:0/754035594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e4f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb6477fe640 1 -- 172.21.15.7:0/754035594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4f7a80 msgr2=0x56109e4f9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb6477fe640 1 --2- 172.21.15.7:0/754035594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4f7a80 0x56109e4f9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb6477fe640 1 -- 172.21.15.7:0/754035594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 msgr2=0x56109e4f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb6477fe640 1 --2- 172.21.15.7:0/754035594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e4f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.931+0000 7fb6477fe640 1 -- 172.21.15.7:0/754035594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56109e22f7d0 con 0x56109e4ee580 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.932+0000 7fb6477fe640 1 --2- 172.21.15.7:0/754035594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4ee580 0x56109e4ee960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb640011790 tx=0x7fb640011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a64348f6fdcc3422 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.932+0000 7fb63ffff640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb6340a3de0 0x7fb6340a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.932+0000 7fb63d7fa640 1 -- 172.21.15.7:0/754035594 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640003c30 con 0x56109e4ee580 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.932+0000 7fb63d7fa640 1 -- 172.21.15.7:0/754035594 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb64000fe50 con 0x56109e4ee580 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.932+0000 7fb63d7fa640 1 -- 172.21.15.7:0/754035594 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640004840 con 0x56109e4ee580 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.932+0000 7fb63ffff640 1 -- 172.21.15.7:0/992575715 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7fb6300f2b80 con 0x7fb64806abb0 2024-04-21T08:52:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.932+0000 7fb646ffd640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb6340a3de0 0x7fb6340a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.933+0000 7fb650016740 1 -- 172.21.15.7:0/754035594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4ee580 msgr2=0x56109e4ee960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.933+0000 7fb650016740 1 --2- 172.21.15.7:0/754035594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4ee580 0x56109e4ee960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb640011790 tx=0x7fb640011c50 comp rx=0 tx=0).stop 2024-04-21T08:52:59.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.933+0000 7fb646ffd640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb6340a3de0 0x7fb6340a62a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x56109e4ee050 tx=0x7fb638015040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.933+0000 7fb650016740 1 -- 172.21.15.7:0/754035594 shutdown_connections 2024-04-21T08:52:59.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.933+0000 7fb650016740 1 --2- 172.21.15.7:0/754035594 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4f7a80 0x56109e4f9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.933+0000 7fb650016740 1 --2- 172.21.15.7:0/754035594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e4f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.933+0000 7fb650016740 1 --2- 172.21.15.7:0/754035594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4ee580 0x56109e4ee960 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.933+0000 7fb650016740 1 -- 172.21.15.7:0/754035594 >> 172.21.15.7:0/754035594 conn(0x56109e4e8cb0 msgr2=0x56109e4e9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.933+0000 7fb650016740 1 -- 172.21.15.7:0/754035594 shutdown_connections 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.934+0000 7fb650016740 1 -- 172.21.15.7:0/754035594 wait complete. 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.934+0000 7fb650016740 1 -- start start 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.934+0000 7fb650016740 1 Processor -- start 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.934+0000 7fb650016740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e4f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.934+0000 7fb650016740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e4f5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.934+0000 7fb650016740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 0x56109e4f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.934+0000 7fb650016740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56109e4fa870 con 0x56109e4eeea0 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.934+0000 7fb650016740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56109e40c2f0 con 0x56109e4ee580 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb650016740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56109e4fa9f0 con 0x56109e4f7a80 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb650016740 10 client.0 _ll_register_callbacks cb 0x56109e419580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:59.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb6467fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e4f5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 0x56109e4f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb646ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e4f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb6467fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e4f5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51172/0 (socket says 172.21.15.7:51172) 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb6467fc640 1 -- 172.21.15.7:0/4290998040 learned_addr learned my addr 172.21.15.7:0/4290998040 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb6467fc640 1 -- 172.21.15.7:0/4290998040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 msgr2=0x56109e4f55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb6467fc640 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 0x56109e4f55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb6467fc640 1 -- 172.21.15.7:0/4290998040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 msgr2=0x56109e4f4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb6467fc640 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e4f4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb6467fc640 1 -- 172.21.15.7:0/4290998040 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56109e1653f0 con 0x56109e4eeea0 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb6477fe640 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 0x56109e4f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb6467fc640 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e4f5080 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb63003e040 tx=0x7fb6300ba450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d41b547b4c3b57b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb62e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb62e7fc640 1 -- 172.21.15.7:0/4290998040 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6300be2f0 con 0x56109e4eeea0 2024-04-21T08:52:59.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb650016740 1 -- 172.21.15.7:0/4290998040 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb64000fe50 con 0x56109e4eeea0 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.935+0000 7fb62e7fc640 1 -- 172.21.15.7:0/4290998040 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7fb6300f00d0 con 0x56109e4eeea0 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb62e7fc640 1 -- 172.21.15.7:0/4290998040 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb6300f6020 con 0x56109e4eeea0 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb650016740 10 client.17082 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb650016740 1 -- 172.21.15.7:0/4290998040 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56109e1250f0 con 0x56109e4eeea0 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb62e7fc640 1 -- 172.21.15.7:0/4290998040 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb64000fe50 con 0x56109e4eeea0 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb62e7fc640 1 -- 172.21.15.7:0/4290998040 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb6300f9020 con 0x56109e4eeea0 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb650016740 10 client.17082 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb650016740 10 client.17082 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb650016740 20 client.17082 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb650016740 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56109e5077d0 0x56109e509bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb650016740 4 client.17082 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb650016740 1 -- 172.21.15.7:0/4290998040 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56109e460c00 con 0x56109e5077d0 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.936+0000 7fb646ffd640 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56109e5077d0 0x56109e509bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.937+0000 7fb646ffd640 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56109e5077d0 0x56109e509bc0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ebe5d88ece5de71 server_cookie=6c695e85c1d55a89 in_seq=0 out_seq=0 2024-04-21T08:52:59.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.937+0000 7fb62e7fc640 10 client.17082 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb62e7fc640 1 -- 172.21.15.7:0/4290998040 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56109e460c00 con 0x56109e5077d0 2024-04-21T08:52:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb62e7fc640 10 client.17082 handle_command_reply: tid=0 2024-04-21T08:52:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb650016740 1 client.17082 shutdown 2024-04-21T08:52:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb650016740 1 -- 172.21.15.7:0/4290998040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 msgr2=0x56109e4f5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb650016740 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e4f5080 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb63003e040 tx=0x7fb6300ba450 comp rx=0 tx=0).stop 2024-04-21T08:52:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb650016740 1 -- 172.21.15.7:0/4290998040 shutdown_connections 2024-04-21T08:52:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb650016740 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56109e5077d0 0x56109e509bc0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb650016740 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 0x56109e4f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb650016740 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e4f5080 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb650016740 1 --2- 172.21.15.7:0/4290998040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e4f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb62e7fc640 0 client.17082 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.938+0000 7fb650016740 1 -- 172.21.15.7:0/4290998040 >> 172.21.15.7:0/4290998040 conn(0x56109e4e8cb0 msgr2=0x56109e4eaa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.939+0000 7fb650016740 1 -- 172.21.15.7:0/4290998040 shutdown_connections 2024-04-21T08:52:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.939+0000 7fb650016740 1 -- 172.21.15.7:0/4290998040 wait complete. 2024-04-21T08:52:59.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.939+0000 7fb650016740 20 client.17082 trim_cache size 0 max 16384 2024-04-21T08:52:59.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.966+0000 7fb650016740 1 Processor -- start 2024-04-21T08:52:59.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.966+0000 7fb650016740 1 -- start start 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.966+0000 7fb650016740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e4f7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.966+0000 7fb650016740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e503730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.966+0000 7fb650016740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 0x56109e4f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.966+0000 7fb650016740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb6481bc470 con 0x56109e4eeea0 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.966+0000 7fb650016740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb6481becf0 con 0x56109e4ee580 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.966+0000 7fb650016740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb6481bcae0 con 0x56109e4f7a80 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.966+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 0x56109e4f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb6467fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e503730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 0x56109e4f3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:52044/0 (socket says 172.21.15.7:52044) 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb6477fe640 1 -- 172.21.15.7:0/3546339864 learned_addr learned my addr 172.21.15.7:0/3546339864 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb646ffd640 1 --2- 172.21.15.7:0/3546339864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e4f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb6477fe640 1 -- 172.21.15.7:0/3546339864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 msgr2=0x56109e4f7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb6477fe640 1 --2- 172.21.15.7:0/3546339864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e4f7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb6477fe640 1 -- 172.21.15.7:0/3546339864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 msgr2=0x56109e503730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb6477fe640 1 --2- 172.21.15.7:0/3546339864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e503730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb6477fe640 1 -- 172.21.15.7:0/3546339864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56109e50a240 con 0x56109e4f7a80 2024-04-21T08:52:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb6477fe640 1 --2- 172.21.15.7:0/3546339864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 0x56109e4f3a80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb640014020 tx=0x7fb640005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a73444d59457deea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb63cff9640 1 -- 172.21.15.7:0/3546339864 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64001f040 con 0x56109e4f7a80 2024-04-21T08:52:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb63cff9640 1 -- 172.21.15.7:0/3546339864 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb640004050 con 0x56109e4f7a80 2024-04-21T08:52:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.967+0000 7fb63cff9640 1 -- 172.21.15.7:0/3546339864 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6400052b0 con 0x56109e4f7a80 2024-04-21T08:52:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.968+0000 7fb650016740 1 -- 172.21.15.7:0/3546339864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 msgr2=0x56109e4f3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.968+0000 7fb650016740 1 --2- 172.21.15.7:0/3546339864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 0x56109e4f3a80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb640014020 tx=0x7fb640005bd0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.968+0000 7fb650016740 1 -- 172.21.15.7:0/3546339864 shutdown_connections 2024-04-21T08:52:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.968+0000 7fb650016740 1 --2- 172.21.15.7:0/3546339864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4f7a80 0x56109e4f3a80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.968+0000 7fb650016740 1 --2- 172.21.15.7:0/3546339864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4eeea0 0x56109e503730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.968+0000 7fb650016740 1 --2- 172.21.15.7:0/3546339864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e4f7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.969+0000 7fb650016740 1 -- 172.21.15.7:0/3546339864 >> 172.21.15.7:0/3546339864 conn(0x56109e4e8cb0 msgr2=0x56109e4e7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.969+0000 7fb650016740 1 -- 172.21.15.7:0/3546339864 shutdown_connections 2024-04-21T08:52:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.969+0000 7fb650016740 1 -- 172.21.15.7:0/3546339864 wait complete. 2024-04-21T08:52:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.970+0000 7fb650016740 1 -- start start 2024-04-21T08:52:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.970+0000 7fb650016740 1 Processor -- start 2024-04-21T08:52:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.970+0000 7fb650016740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e5001a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.970+0000 7fb650016740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4eeea0 0x56109e4f1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.971+0000 7fb650016740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4f7a80 0x56109e4f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.971+0000 7fb646ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e5001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.971+0000 7fb646ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e5001a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:42370/0 (socket says 172.21.15.7:42370) 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.971+0000 7fb6467fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4eeea0 0x56109e4f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.971+0000 7fb6467fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4eeea0 0x56109e4f1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:52058/0 (socket says 172.21.15.7:52058) 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.971+0000 7fb646ffd640 1 -- 172.21.15.7:0/2232523311 learned_addr learned my addr 172.21.15.7:0/2232523311 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.971+0000 7fb650016740 1 -- 172.21.15.7:0/2232523311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb6481bd1b0 con 0x56109e4f7a80 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.971+0000 7fb650016740 1 -- 172.21.15.7:0/2232523311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb6481be620 con 0x56109e4ee580 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.971+0000 7fb6477fe640 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4f7a80 0x56109e4f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.971+0000 7fb650016740 1 -- 172.21.15.7:0/2232523311 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb6481bd880 con 0x56109e4eeea0 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb650016740 10 client.0 _ll_register_callbacks cb 0x7fb630005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb6477fe640 1 -- 172.21.15.7:0/2232523311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4eeea0 msgr2=0x56109e4f1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb6477fe640 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4eeea0 0x56109e4f1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb6477fe640 1 -- 172.21.15.7:0/2232523311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 msgr2=0x56109e5001a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb6477fe640 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e5001a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb6477fe640 1 -- 172.21.15.7:0/2232523311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56109e56cd40 con 0x56109e4f7a80 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb646ffd640 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e5001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb6467fc640 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4eeea0 0x56109e4f1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb650016740 1 -- 172.21.15.7:0/2232523311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb640004050 con 0x56109e4f7a80 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb62e7fc640 10 client.17088 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb6477fe640 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4f7a80 0x56109e4f1dc0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x56109e50a0d0 tx=0x7fb640000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1559c4201b4f68cd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb62e7fc640 1 -- 172.21.15.7:0/2232523311 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640034060 con 0x56109e4f7a80 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.972+0000 7fb62e7fc640 1 -- 172.21.15.7:0/2232523311 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9672+0+0 (secure 0 0 0) 0x7fb640003d30 con 0x56109e4f7a80 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.973+0000 7fb62e7fc640 1 -- 172.21.15.7:0/2232523311 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb64001f040 con 0x56109e4f7a80 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.973+0000 7fb650016740 10 client.17088 fetch_fsmap learned FSMap version 23 2024-04-21T08:52:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.973+0000 7fb650016740 1 -- 172.21.15.7:0/2232523311 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({fsmap=23}) -- 0x56109e56ddb0 con 0x56109e4f7a80 2024-04-21T08:52:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.973+0000 7fb62e7fc640 1 -- 172.21.15.7:0/2232523311 <== mon.0 v2:172.21.15.7:3300/0 4 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fb640037050 con 0x56109e4f7a80 2024-04-21T08:52:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.973+0000 7fb650016740 10 client.17088 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:52:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.973+0000 7fb650016740 10 client.17088 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.973+0000 7fb650016740 20 client.17088 populate_metadata read hostname 'smithi007' 2024-04-21T08:52:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.973+0000 7fb650016740 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56109e56e180 0x56109e570570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.973+0000 7fb650016740 4 client.17088 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:52:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.973+0000 7fb650016740 1 -- 172.21.15.7:0/2232523311 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56109e45aa70 con 0x56109e56e180 2024-04-21T08:52:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.973+0000 7fb646ffd640 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56109e56e180 0x56109e570570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.974+0000 7fb62e7fc640 10 client.17088 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.974+0000 7fb646ffd640 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56109e56e180 0x56109e570570 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2a3e51f61a2d292 server_cookie=15f719384b558c04 in_seq=0 out_seq=0 2024-04-21T08:52:59.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.979+0000 7fb62e7fc640 1 -- 172.21.15.7:0/2232523311 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x56109e45aa70 con 0x56109e56e180 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.979+0000 7fb62e7fc640 10 client.17088 handle_command_reply: tid=0 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.979+0000 7fb650016740 1 client.17088 shutdown 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.979+0000 7fb650016740 1 -- 172.21.15.7:0/2232523311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4f7a80 msgr2=0x56109e4f1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.979+0000 7fb650016740 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4f7a80 0x56109e4f1dc0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x56109e50a0d0 tx=0x7fb640000f40 comp rx=0 tx=0).stop 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.979+0000 7fb650016740 1 -- 172.21.15.7:0/2232523311 shutdown_connections 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.979+0000 7fb650016740 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56109e56e180 0x56109e570570 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.980+0000 7fb650016740 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56109e4f7a80 0x56109e4f1dc0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.980+0000 7fb650016740 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56109e4eeea0 0x56109e4f1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.980+0000 7fb650016740 1 --2- 172.21.15.7:0/2232523311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56109e4ee580 0x56109e5001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.980+0000 7fb650016740 1 -- 172.21.15.7:0/2232523311 >> 172.21.15.7:0/2232523311 conn(0x56109e4e8cb0 msgr2=0x56109e4f4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.980+0000 7fb62e7fc640 0 client.17088 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.980+0000 7fb650016740 1 -- 172.21.15.7:0/2232523311 shutdown_connections 2024-04-21T08:52:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.980+0000 7fb650016740 1 -- 172.21.15.7:0/2232523311 wait complete. 2024-04-21T08:52:59.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.980+0000 7fb650016740 20 client.17088 trim_cache size 0 max 16384 2024-04-21T08:52:59.981 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:52:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (3777 inodes in the stack)", 2024-04-21T08:52:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:52:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "2f44a19d-b0a7-4681-b27b-fb7529461620": { 2024-04-21T08:52:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:52:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "2f44a19d-b0a7-4681-b27b-fb7529461620", 2024-04-21T08:52:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:52:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:52:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:52:59.982 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:52:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.981+0000 7fb64d42e640 1 -- 172.21.15.7:0/992575715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb6340a3de0 msgr2=0x7fb6340a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.981+0000 7fb64d42e640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb6340a3de0 0x7fb6340a62a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x56109e4ee050 tx=0x7fb638015040 comp rx=0 tx=0).stop 2024-04-21T08:52:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.981+0000 7fb64d42e640 1 -- 172.21.15.7:0/992575715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64806abb0 msgr2=0x7fb6481a7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.981+0000 7fb64d42e640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64806abb0 0x7fb6481a7f80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb630005ab0 tx=0x7fb630004550 comp rx=0 tx=0).stop 2024-04-21T08:52:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.981+0000 7fb64d42e640 1 -- 172.21.15.7:0/992575715 shutdown_connections 2024-04-21T08:52:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.981+0000 7fb64d42e640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fb6340a3de0 0x7fb6340a62a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.981+0000 7fb64d42e640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb64810e8f0 0x7fb6481abe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.981+0000 7fb64d42e640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb64806abb0 0x7fb6481a7f80 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.981+0000 7fb64d42e640 1 --2- 172.21.15.7:0/992575715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb64806a290 0x7fb6481a7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.981+0000 7fb64d42e640 1 -- 172.21.15.7:0/992575715 >> 172.21.15.7:0/992575715 conn(0x7fb648077f40 msgr2=0x7fb64810fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:59.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.982+0000 7fb64d42e640 1 -- 172.21.15.7:0/992575715 shutdown_connections 2024-04-21T08:52:59.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:52:59.982+0000 7fb64d42e640 1 -- 172.21.15.7:0/992575715 wait complete. 2024-04-21T08:52:59.997 INFO:tasks.cephfs.filesystem:scrub status for tag:2f44a19d-b0a7-4681-b27b-fb7529461620 - {'path': '/', 'tag': '2f44a19d-b0a7-4681-b27b-fb7529461620', 'options': 'recursive,force'} 2024-04-21T08:53:01.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:00 smithi016 ceph-mon[31259]: pgmap v257: 129 pgs: 6 active+clean+snaptrim, 116 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 836 KiB/s wr, 1.22k op/s 2024-04-21T08:53:01.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:00 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:53:01.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:00 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:01.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:00 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:53:01.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:00 smithi077 ceph-mon[31615]: pgmap v257: 129 pgs: 6 active+clean+snaptrim, 116 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 836 KiB/s wr, 1.22k op/s 2024-04-21T08:53:01.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:00 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:53:01.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:00 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:01.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:00 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:53:01.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:00 smithi007 ceph-mon[28477]: pgmap v257: 129 pgs: 6 active+clean+snaptrim, 116 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 836 KiB/s wr, 1.22k op/s 2024-04-21T08:53:01.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:00 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:53:01.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:00 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:01.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:00 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:53:03.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:02 smithi016 ceph-mon[31259]: pgmap v258: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 809 KiB/s wr, 1.08k op/s 2024-04-21T08:53:03.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:02 smithi077 ceph-mon[31615]: pgmap v258: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 809 KiB/s wr, 1.08k op/s 2024-04-21T08:53:03.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:02 smithi007 ceph-mon[28477]: pgmap v258: 129 pgs: 4 active+clean+snaptrim, 118 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 809 KiB/s wr, 1.08k op/s 2024-04-21T08:53:05.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:04 smithi016 ceph-mon[31259]: pgmap v259: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 873 op/s 2024-04-21T08:53:05.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:04 smithi077 ceph-mon[31615]: pgmap v259: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 873 op/s 2024-04-21T08:53:05.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:04 smithi007 ceph-mon[28477]: pgmap v259: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 873 op/s 2024-04-21T08:53:07.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:06 smithi016 ceph-mon[31259]: pgmap v260: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 720 op/s 2024-04-21T08:53:07.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:06 smithi077 ceph-mon[31615]: pgmap v260: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 720 op/s 2024-04-21T08:53:07.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:06 smithi007 ceph-mon[28477]: pgmap v260: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 720 op/s 2024-04-21T08:53:07.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.886+0000 7f154e8f2640 1 -- 172.21.15.7:0/326373069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15481018d0 msgr2=0x7f1548105d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.886+0000 7f154e8f2640 1 --2- 172.21.15.7:0/326373069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15481018d0 0x7f1548105d90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1540009cd0 tx=0x7f154002c1b0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.886+0000 7f154e8f2640 1 -- 172.21.15.7:0/326373069 shutdown_connections 2024-04-21T08:53:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.886+0000 7f154e8f2640 1 --2- 172.21.15.7:0/326373069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15481063b0 0x7f1548106860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.886+0000 7f154e8f2640 1 --2- 172.21.15.7:0/326373069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15481018d0 0x7f1548105d90 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.886+0000 7f154e8f2640 1 --2- 172.21.15.7:0/326373069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1548100fb0 0x7f1548101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.886+0000 7f154e8f2640 1 -- 172.21.15.7:0/326373069 >> 172.21.15.7:0/326373069 conn(0x7f1548077f70 msgr2=0x7f1548078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.889+0000 7f154e8f2640 1 -- 172.21.15.7:0/326373069 shutdown_connections 2024-04-21T08:53:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.889+0000 7f154e8f2640 1 -- 172.21.15.7:0/326373069 wait complete. 2024-04-21T08:53:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.890+0000 7f154e8f2640 1 Processor -- start 2024-04-21T08:53:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.890+0000 7f154e8f2640 1 -- start start 2024-04-21T08:53:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.890+0000 7f154e8f2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1548100fb0 0x7f1548115220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.890+0000 7f154e8f2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15481018d0 0x7f154810e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.890+0000 7f154e8f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15481063b0 0x7f154810e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.890+0000 7f154e8f2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f15481187a0 con 0x7f1548100fb0 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.890+0000 7f154e8f2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1548118620 con 0x7f15481063b0 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.890+0000 7f154e8f2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1548118920 con 0x7f15481018d0 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.891+0000 7f1547fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1548100fb0 0x7f1548115220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.891+0000 7f1547fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1548100fb0 0x7f1548115220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44618/0 (socket says 172.21.15.7:44618) 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.891+0000 7f1547fff640 1 -- 172.21.15.7:0/3582624550 learned_addr learned my addr 172.21.15.7:0/3582624550 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.891+0000 7f15477fe640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15481018d0 0x7f154810e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.891+0000 7f1547fff640 1 -- 172.21.15.7:0/3582624550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15481018d0 msgr2=0x7f154810e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.891+0000 7f1547fff640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15481018d0 0x7f154810e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.891+0000 7f1547fff640 1 -- 172.21.15.7:0/3582624550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15481063b0 msgr2=0x7f154810e830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.891+0000 7f1547fff640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15481063b0 0x7f154810e830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.891+0000 7f1547fff640 1 -- 172.21.15.7:0/3582624550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154810eea0 con 0x7f1548100fb0 2024-04-21T08:53:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.891+0000 7f1547fff640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1548100fb0 0x7f1548115220 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c890 tx=0x7f153c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.892+0000 7f15457fa640 1 -- 172.21.15.7:0/3582624550 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c004220 con 0x7f1548100fb0 2024-04-21T08:53:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.892+0000 7f15457fa640 1 -- 172.21.15.7:0/3582624550 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f153c0043c0 con 0x7f1548100fb0 2024-04-21T08:53:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.892+0000 7f15457fa640 1 -- 172.21.15.7:0/3582624550 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c0049e0 con 0x7f1548100fb0 2024-04-21T08:53:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.892+0000 7f154e8f2640 1 -- 172.21.15.7:0/3582624550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1548112d40 con 0x7f1548100fb0 2024-04-21T08:53:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.892+0000 7f154e8f2640 1 -- 172.21.15.7:0/3582624550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1548113240 con 0x7f1548100fb0 2024-04-21T08:53:07.895 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.896+0000 7f15515c0740 1 Processor -- start 2024-04-21T08:53:07.896 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.896+0000 7f15515c0740 1 -- start start 2024-04-21T08:53:07.896 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.897+0000 7f15515c0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff9a8d0 0x55d0fff99620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.896 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.897+0000 7f15515c0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff99b60 0x55d0fff99fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.897+0000 7f15515c0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fffa30e0 0x55d0fffa54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.897+0000 7f15515c0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d0ffefe8d0 con 0x55d0fff99b60 2024-04-21T08:53:07.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.897+0000 7f15515c0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d0ffe9be80 con 0x55d0fffa30e0 2024-04-21T08:53:07.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.897+0000 7f15515c0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d0ffefa890 con 0x55d0fff9a8d0 2024-04-21T08:53:07.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.897+0000 7f15457fa640 1 -- 172.21.15.7:0/3582624550 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f153c0079b0 con 0x7f1548100fb0 2024-04-21T08:53:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.898+0000 7f15457fa640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f15300a3f10 0x7f15300a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.898+0000 7f15457fa640 1 -- 172.21.15.7:0/3582624550 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9764+0+0 (secure 0 0 0) 0x7f153c0d09e0 con 0x7f1548100fb0 2024-04-21T08:53:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.898+0000 7f154ce8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff9a8d0 0x55d0fff99620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.899+0000 7f154ce8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff9a8d0 0x55d0fff99620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:39188/0 (socket says 172.21.15.7:39188) 2024-04-21T08:53:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.899+0000 7f154ce8f640 1 -- 172.21.15.7:0/61642438 learned_addr learned my addr 172.21.15.7:0/61642438 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.899+0000 7f154ce8f640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f15300a3f10 0x7f15300a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.899+0000 7f154ce8f640 1 -- 172.21.15.7:0/61642438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fffa30e0 msgr2=0x55d0fffa54d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.899+0000 7f154ce8f640 1 --2- 172.21.15.7:0/61642438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fffa30e0 0x55d0fffa54d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.899+0000 7f154ce8f640 1 -- 172.21.15.7:0/61642438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff99b60 msgr2=0x55d0fff99fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.899+0000 7f154ce8f640 1 --2- 172.21.15.7:0/61642438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff99b60 0x55d0fff99fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.899+0000 7f154ce8f640 1 -- 172.21.15.7:0/61642438 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0ffebd320 con 0x55d0fff9a8d0 2024-04-21T08:53:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.900+0000 7f154ce8f640 1 --2- 172.21.15.7:0/61642438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff9a8d0 0x55d0fff99620 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1538014540 tx=0x7f1538014a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6d2c1c3721679db server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.901+0000 7f154ce8f640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f15300a3f10 0x7f15300a63d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1538014f00 tx=0x7f153801b000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.901+0000 7f152e7fc640 1 -- 172.21.15.7:0/61642438 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538011e50 con 0x55d0fff9a8d0 2024-04-21T08:53:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.901+0000 7f152e7fc640 1 -- 172.21.15.7:0/61642438 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15380054e0 con 0x55d0fff9a8d0 2024-04-21T08:53:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.903+0000 7f15515c0740 1 -- 172.21.15.7:0/61642438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff9a8d0 msgr2=0x55d0fff99620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.903+0000 7f15515c0740 1 --2- 172.21.15.7:0/61642438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff9a8d0 0x55d0fff99620 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1538014540 tx=0x7f1538014a00 comp rx=0 tx=0).stop 2024-04-21T08:53:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.903+0000 7f15515c0740 1 -- 172.21.15.7:0/61642438 shutdown_connections 2024-04-21T08:53:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.903+0000 7f15515c0740 1 --2- 172.21.15.7:0/61642438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fffa30e0 0x55d0fffa54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.903+0000 7f15515c0740 1 --2- 172.21.15.7:0/61642438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff99b60 0x55d0fff99fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.903+0000 7f15515c0740 1 --2- 172.21.15.7:0/61642438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff9a8d0 0x55d0fff99620 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.903+0000 7f15515c0740 1 -- 172.21.15.7:0/61642438 >> 172.21.15.7:0/61642438 conn(0x55d0fff60c90 msgr2=0x55d0fff94850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.905+0000 7f15515c0740 1 -- 172.21.15.7:0/61642438 shutdown_connections 2024-04-21T08:53:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.905+0000 7f15515c0740 1 -- 172.21.15.7:0/61642438 wait complete. 2024-04-21T08:53:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.906+0000 7f15515c0740 1 -- start start 2024-04-21T08:53:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.906+0000 7f15515c0740 1 Processor -- start 2024-04-21T08:53:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.906+0000 7f15515c0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff99b60 0x55d0fffa2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.906+0000 7f15515c0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff9a8d0 0x55d0fff9fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.906+0000 7f15515c0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fffa30e0 0x55d0fffa02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.906+0000 7f15515c0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d0fffa5da0 con 0x55d0fff9a8d0 2024-04-21T08:53:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.906+0000 7f15515c0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d0ffe86eb0 con 0x55d0fffa30e0 2024-04-21T08:53:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.906+0000 7f15515c0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d0fffa5f20 con 0x55d0fff99b60 2024-04-21T08:53:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.907+0000 7f15515c0740 10 client.0 _ll_register_callbacks cb 0x55d0ffeb3b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.907+0000 7f1547fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff99b60 0x55d0fffa2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.907+0000 7f1547fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff99b60 0x55d0fffa2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:39190/0 (socket says 172.21.15.7:39190) 2024-04-21T08:53:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.907+0000 7f1547fff640 1 -- 172.21.15.7:0/3012937110 learned_addr learned my addr 172.21.15.7:0/3012937110 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.907+0000 7f154ce8f640 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fffa30e0 0x55d0fffa02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.909+0000 7f15477fe640 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff9a8d0 0x55d0fff9fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.909+0000 7f1547fff640 1 -- 172.21.15.7:0/3012937110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fffa30e0 msgr2=0x55d0fffa02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.909+0000 7f1547fff640 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fffa30e0 0x55d0fffa02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.909+0000 7f1547fff640 1 -- 172.21.15.7:0/3012937110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff9a8d0 msgr2=0x55d0fff9fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.909+0000 7f1547fff640 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff9a8d0 0x55d0fff9fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.909+0000 7f1547fff640 1 -- 172.21.15.7:0/3012937110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0ffc11210 con 0x55d0fff99b60 2024-04-21T08:53:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.909+0000 7f1547fff640 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff99b60 0x55d0fffa2a10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f153c09c140 tx=0x7f153c09c070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=721d087cd7be286e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.912+0000 7f15277fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:53:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.913+0000 7f15277fe640 1 -- 172.21.15.7:0/3012937110 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c0b5060 con 0x55d0fff99b60 2024-04-21T08:53:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.914+0000 7f15277fe640 1 -- 172.21.15.7:0/3012937110 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9764+0+0 (secure 0 0 0) 0x7f153c09b890 con 0x55d0fff99b60 2024-04-21T08:53:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.914+0000 7f15515c0740 1 -- 172.21.15.7:0/3012937110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15380054e0 con 0x55d0fff99b60 2024-04-21T08:53:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.918+0000 7f15277fe640 1 -- 172.21.15.7:0/3012937110 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f153c0ba050 con 0x55d0fff99b60 2024-04-21T08:53:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.926+0000 7f15515c0740 10 client.26623 fetch_fsmap learned FSMap version 23 2024-04-21T08:53:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.926+0000 7f15515c0740 1 -- 172.21.15.7:0/3012937110 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d0ffcdd620 con 0x55d0fff99b60 2024-04-21T08:53:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.930+0000 7f15277fe640 1 -- 172.21.15.7:0/3012937110 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15380054e0 con 0x55d0fff99b60 2024-04-21T08:53:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.930+0000 7f15277fe640 1 -- 172.21.15.7:0/3012937110 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f153c0bb240 con 0x55d0fff99b60 2024-04-21T08:53:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.934+0000 7f15515c0740 10 client.26623 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:53:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.934+0000 7f15515c0740 10 client.26623 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.934+0000 7f15515c0740 20 client.26623 populate_metadata read hostname 'smithi007' 2024-04-21T08:53:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.934+0000 7f15515c0740 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0fffb30c0 0x55d0fffb54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.934+0000 7f15515c0740 4 client.26623 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.934+0000 7f15515c0740 1 -- 172.21.15.7:0/3012937110 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0fff12880 con 0x55d0fffb30c0 2024-04-21T08:53:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.936+0000 7f15477fe640 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0fffb30c0 0x55d0fffb54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.939+0000 7f15477fe640 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0fffb30c0 0x55d0fffb54b0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cefb0fc84dbc46a server_cookie=5bbe852883379be9 in_seq=0 out_seq=0 2024-04-21T08:53:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.940+0000 7f15277fe640 10 client.26623 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.941+0000 7f15277fe640 1 -- 172.21.15.7:0/3012937110 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0fff12880 con 0x55d0fffb30c0 2024-04-21T08:53:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.941+0000 7f15277fe640 10 client.26623 handle_command_reply: tid=0 2024-04-21T08:53:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.941+0000 7f15515c0740 1 client.26623 shutdown 2024-04-21T08:53:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.941+0000 7f15515c0740 1 -- 172.21.15.7:0/3012937110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff99b60 msgr2=0x55d0fffa2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.941+0000 7f15515c0740 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff99b60 0x55d0fffa2a10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f153c09c140 tx=0x7f153c09c070 comp rx=0 tx=0).stop 2024-04-21T08:53:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.942+0000 7f15515c0740 1 -- 172.21.15.7:0/3012937110 shutdown_connections 2024-04-21T08:53:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.942+0000 7f15515c0740 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0fffb30c0 0x55d0fffb54b0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.942+0000 7f15515c0740 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fffa30e0 0x55d0fffa02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.942+0000 7f15515c0740 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff9a8d0 0x55d0fff9fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.942+0000 7f15515c0740 1 --2- 172.21.15.7:0/3012937110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fff99b60 0x55d0fffa2a10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.942+0000 7f15515c0740 1 -- 172.21.15.7:0/3012937110 >> 172.21.15.7:0/3012937110 conn(0x55d0fff60c90 msgr2=0x55d0fff965a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.942+0000 7f15277fe640 0 client.26623 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.943+0000 7f15515c0740 1 -- 172.21.15.7:0/3012937110 shutdown_connections 2024-04-21T08:53:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.943+0000 7f15515c0740 1 -- 172.21.15.7:0/3012937110 wait complete. 2024-04-21T08:53:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.943+0000 7f15515c0740 20 client.26623 trim_cache size 0 max 16384 2024-04-21T08:53:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.972+0000 7f15515c0740 1 Processor -- start 2024-04-21T08:53:07.972 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.973+0000 7f15515c0740 1 -- start start 2024-04-21T08:53:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.973+0000 7f15515c0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff99b60 0x55d0fffaecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.973+0000 7f15515c0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff9a8d0 0x55d0fff9ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.973 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.973+0000 7f15515c0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fffa30e0 0x55d0fff96340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.974+0000 7f15515c0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f154810f7d0 con 0x55d0fff99b60 2024-04-21T08:53:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.974+0000 7f15477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff9a8d0 0x55d0fff9ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.974+0000 7f15477fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff9a8d0 0x55d0fff9ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33484/0 (socket says 172.21.15.7:33484) 2024-04-21T08:53:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.974+0000 7f15477fe640 1 -- 172.21.15.7:0/2622701030 learned_addr learned my addr 172.21.15.7:0/2622701030 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.974+0000 7f154ce8f640 1 --2- 172.21.15.7:0/2622701030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fffa30e0 0x55d0fff96340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.974+0000 7f15515c0740 1 -- 172.21.15.7:0/2622701030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15481c77b0 con 0x55d0fff9a8d0 2024-04-21T08:53:07.974 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.974+0000 7f15515c0740 1 -- 172.21.15.7:0/2622701030 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1548073aa0 con 0x55d0fffa30e0 2024-04-21T08:53:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.975+0000 7f15477fe640 1 -- 172.21.15.7:0/2622701030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fffa30e0 msgr2=0x55d0fff96340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.975+0000 7f15477fe640 1 --2- 172.21.15.7:0/2622701030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fffa30e0 0x55d0fff96340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.975+0000 7f15477fe640 1 -- 172.21.15.7:0/2622701030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff99b60 msgr2=0x55d0fffaecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.975+0000 7f15477fe640 1 --2- 172.21.15.7:0/2622701030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff99b60 0x55d0fffaecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.975+0000 7f15477fe640 1 -- 172.21.15.7:0/2622701030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0fff98e30 con 0x55d0fff9a8d0 2024-04-21T08:53:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.975+0000 7f15477fe640 1 --2- 172.21.15.7:0/2622701030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff9a8d0 0x55d0fff9ef00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f154002c7c0 tx=0x7f154002c1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ca65fed4b4b8c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:07.975 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.975+0000 7f152dffb640 1 -- 172.21.15.7:0/2622701030 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1540049070 con 0x55d0fff9a8d0 2024-04-21T08:53:07.976 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.976+0000 7f152dffb640 1 -- 172.21.15.7:0/2622701030 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1540032340 con 0x55d0fff9a8d0 2024-04-21T08:53:07.976 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.977+0000 7f15515c0740 1 -- 172.21.15.7:0/2622701030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff9a8d0 msgr2=0x55d0fff9ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:07.976 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.977+0000 7f15515c0740 1 --2- 172.21.15.7:0/2622701030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff9a8d0 0x55d0fff9ef00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f154002c7c0 tx=0x7f154002c1b0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.976 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.977+0000 7f15515c0740 1 -- 172.21.15.7:0/2622701030 shutdown_connections 2024-04-21T08:53:07.977 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.977+0000 7f15515c0740 1 --2- 172.21.15.7:0/2622701030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fffa30e0 0x55d0fff96340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.977 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.977+0000 7f15515c0740 1 --2- 172.21.15.7:0/2622701030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff9a8d0 0x55d0fff9ef00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.977 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.977+0000 7f15515c0740 1 --2- 172.21.15.7:0/2622701030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fff99b60 0x55d0fffaecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.977 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.977+0000 7f15515c0740 1 -- 172.21.15.7:0/2622701030 >> 172.21.15.7:0/2622701030 conn(0x55d0fff60c90 msgr2=0x55d0fff95c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:07.978 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.978+0000 7f15515c0740 1 -- 172.21.15.7:0/2622701030 shutdown_connections 2024-04-21T08:53:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.978+0000 7f15515c0740 1 -- 172.21.15.7:0/2622701030 wait complete. 2024-04-21T08:53:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.979+0000 7f15515c0740 1 -- start start 2024-04-21T08:53:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.979+0000 7f15515c0740 1 Processor -- start 2024-04-21T08:53:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.984+0000 7f15515c0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff99b60 0x55d0fffab620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.984+0000 7f15515c0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fffa30e0 0x55d0fffa68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.984+0000 7f15515c0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fffa6de0 0x55d0fffa7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.984+0000 7f15515c0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f15481c5cc0 con 0x55d0fffa6de0 2024-04-21T08:53:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.984+0000 7f15515c0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15481c70e0 con 0x55d0fff99b60 2024-04-21T08:53:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.984+0000 7f15515c0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f15481c6340 con 0x55d0fffa30e0 2024-04-21T08:53:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.984+0000 7f15515c0740 10 client.0 _ll_register_callbacks cb 0x7f153c0057c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.985+0000 7f1547fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff99b60 0x55d0fffab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.985+0000 7f1547fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff99b60 0x55d0fffab620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:33498/0 (socket says 172.21.15.7:33498) 2024-04-21T08:53:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.985+0000 7f1547fff640 1 -- 172.21.15.7:0/1609652670 learned_addr learned my addr 172.21.15.7:0/1609652670 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.986+0000 7f1547fff640 1 -- 172.21.15.7:0/1609652670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fffa30e0 msgr2=0x55d0fffa68a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.986+0000 7f1547fff640 1 --2- 172.21.15.7:0/1609652670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fffa30e0 0x55d0fffa68a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.986+0000 7f1547fff640 1 -- 172.21.15.7:0/1609652670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fffa6de0 msgr2=0x55d0fffa7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.986+0000 7f1547fff640 1 --2- 172.21.15.7:0/1609652670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fffa6de0 0x55d0fffa7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.986+0000 7f1547fff640 1 -- 172.21.15.7:0/1609652670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d100018660 con 0x55d0fff99b60 2024-04-21T08:53:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.987+0000 7f1547fff640 1 --2- 172.21.15.7:0/1609652670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff99b60 0x55d0fffab620 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f153c09a6f0 tx=0x7f153c007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dde4acd92890b778 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.988+0000 7f15277fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:53:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.988+0000 7f15277fe640 1 -- 172.21.15.7:0/1609652670 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153c0b5070 con 0x55d0fff99b60 2024-04-21T08:53:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.988+0000 7f15277fe640 1 -- 172.21.15.7:0/1609652670 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9764+0+0 (secure 0 0 0) 0x7f153c09ba90 con 0x55d0fff99b60 2024-04-21T08:53:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.989+0000 7f15515c0740 1 -- 172.21.15.7:0/1609652670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d0ffe9e3d0 con 0x55d0fff99b60 2024-04-21T08:53:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.989+0000 7f15277fe640 1 -- 172.21.15.7:0/1609652670 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f153c01d040 con 0x55d0fff99b60 2024-04-21T08:53:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.989+0000 7f15515c0740 10 client.26654 fetch_fsmap learned FSMap version 23 2024-04-21T08:53:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.989+0000 7f15515c0740 1 -- 172.21.15.7:0/1609652670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d1000199c0 con 0x55d0fff99b60 2024-04-21T08:53:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.990+0000 7f15277fe640 1 -- 172.21.15.7:0/1609652670 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d0ffe9e3d0 con 0x55d0fff99b60 2024-04-21T08:53:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.990+0000 7f15277fe640 1 -- 172.21.15.7:0/1609652670 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f153c0a1980 con 0x55d0fff99b60 2024-04-21T08:53:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.990+0000 7f15515c0740 10 client.26654 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:53:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.990+0000 7f15515c0740 10 client.26654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.990+0000 7f15515c0740 20 client.26654 populate_metadata read hostname 'smithi007' 2024-04-21T08:53:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.990+0000 7f15515c0740 1 --2- 172.21.15.7:0/1609652670 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d10001a030 0x55d10001c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.990+0000 7f15515c0740 4 client.26654 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:53:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.990+0000 7f15515c0740 1 -- 172.21.15.7:0/1609652670 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d0fff0c2f0 con 0x55d10001a030 2024-04-21T08:53:07.991 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.991+0000 7f15477fe640 1 --2- 172.21.15.7:0/1609652670 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d10001a030 0x55d10001c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.994+0000 7f15477fe640 1 --2- 172.21.15.7:0/1609652670 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d10001a030 0x55d10001c470 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6835425be05bf5bb server_cookie=6cab7929a892fa35 in_seq=0 out_seq=0 2024-04-21T08:53:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:07.994+0000 7f15277fe640 10 client.26654 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:08.989 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.989+0000 7f15277fe640 1 -- 172.21.15.7:0/1609652670 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d0fff0c2f0 con 0x55d10001a030 2024-04-21T08:53:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.989+0000 7f15277fe640 10 client.26654 handle_command_reply: tid=0 2024-04-21T08:53:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.989+0000 7f15515c0740 1 client.26654 shutdown 2024-04-21T08:53:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.990+0000 7f15515c0740 1 -- 172.21.15.7:0/1609652670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff99b60 msgr2=0x55d0fffab620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:08.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.990+0000 7f15515c0740 1 --2- 172.21.15.7:0/1609652670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff99b60 0x55d0fffab620 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f153c09a6f0 tx=0x7f153c007ed0 comp rx=0 tx=0).stop 2024-04-21T08:53:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.990+0000 7f15515c0740 1 -- 172.21.15.7:0/1609652670 shutdown_connections 2024-04-21T08:53:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.990+0000 7f15515c0740 1 --2- 172.21.15.7:0/1609652670 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d10001a030 0x55d10001c470 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.990+0000 7f15515c0740 1 --2- 172.21.15.7:0/1609652670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0fffa6de0 0x55d0fffa7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:08.993 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.990+0000 7f15515c0740 1 --2- 172.21.15.7:0/1609652670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0fffa30e0 0x55d0fffa68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.990+0000 7f15515c0740 1 --2- 172.21.15.7:0/1609652670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0fff99b60 0x55d0fffab620 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.990+0000 7f15515c0740 1 -- 172.21.15.7:0/1609652670 >> 172.21.15.7:0/1609652670 conn(0x55d0fff60c90 msgr2=0x55d0fff95ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.990+0000 7f15277fe640 0 client.26654 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.990+0000 7f15515c0740 1 -- 172.21.15.7:0/1609652670 shutdown_connections 2024-04-21T08:53:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.990+0000 7f15515c0740 1 -- 172.21.15.7:0/1609652670 wait complete. 2024-04-21T08:53:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.991+0000 7f15515c0740 20 client.26654 trim_cache size 0 max 16384 2024-04-21T08:53:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:53:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:53:08.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:53:08.995 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:53:08.995 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.991+0000 7f154e8f2640 1 -- 172.21.15.7:0/3582624550 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f15300a3f10 msgr2=0x7f15300a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:08.995 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.991+0000 7f154e8f2640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f15300a3f10 0x7f15300a63d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1538014f00 tx=0x7f153801b000 comp rx=0 tx=0).stop 2024-04-21T08:53:08.995 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.991+0000 7f154e8f2640 1 -- 172.21.15.7:0/3582624550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1548100fb0 msgr2=0x7f1548115220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:08.995 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.991+0000 7f154e8f2640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1548100fb0 0x7f1548115220 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c890 tx=0x7f153c00cd50 comp rx=0 tx=0).stop 2024-04-21T08:53:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.992+0000 7f154e8f2640 1 -- 172.21.15.7:0/3582624550 shutdown_connections 2024-04-21T08:53:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.992+0000 7f154e8f2640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15481063b0 0x7f154810e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.992+0000 7f154e8f2640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f15300a3f10 0x7f15300a63d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.992+0000 7f154e8f2640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15481018d0 0x7f154810e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.992+0000 7f154e8f2640 1 --2- 172.21.15.7:0/3582624550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1548100fb0 0x7f1548115220 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.992+0000 7f154e8f2640 1 -- 172.21.15.7:0/3582624550 >> 172.21.15.7:0/3582624550 conn(0x7f1548077f70 msgr2=0x7f1548103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.992+0000 7f154e8f2640 1 -- 172.21.15.7:0/3582624550 shutdown_connections 2024-04-21T08:53:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:08.992+0000 7f154e8f2640 1 -- 172.21.15.7:0/3582624550 wait complete. 2024-04-21T08:53:09.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:08 smithi016 ceph-mon[31259]: pgmap v261: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 841 op/s 2024-04-21T08:53:09.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:08 smithi016 ceph-mon[31259]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T08:53:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:08 smithi077 ceph-mon[31615]: pgmap v261: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 841 op/s 2024-04-21T08:53:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:08 smithi077 ceph-mon[31615]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T08:53:09.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:08 smithi007 ceph-mon[28477]: pgmap v261: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 841 op/s 2024-04-21T08:53:09.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:08 smithi007 ceph-mon[28477]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T08:53:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:10 smithi016 ceph-mon[31259]: pgmap v263: 129 pgs: 9 active+clean+snaptrim, 94 active+clean, 26 active+clean+snaptrim_wait; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 1.01k op/s 2024-04-21T08:53:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:10 smithi077 ceph-mon[31615]: pgmap v263: 129 pgs: 9 active+clean+snaptrim, 94 active+clean, 26 active+clean+snaptrim_wait; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 1.01k op/s 2024-04-21T08:53:11.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:10 smithi007 ceph-mon[28477]: pgmap v263: 129 pgs: 9 active+clean+snaptrim, 94 active+clean, 26 active+clean+snaptrim_wait; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 1.01k op/s 2024-04-21T08:53:12.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:12 smithi016 ceph-mon[31259]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T08:53:12.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:12 smithi016 ceph-mon[31259]: pgmap v265: 129 pgs: 18 active+clean+snaptrim, 60 active+clean, 51 active+clean+snaptrim_wait; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 1.61k op/s 2024-04-21T08:53:12.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:12 smithi077 ceph-mon[31615]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T08:53:12.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:12 smithi077 ceph-mon[31615]: pgmap v265: 129 pgs: 18 active+clean+snaptrim, 60 active+clean, 51 active+clean+snaptrim_wait; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 1.61k op/s 2024-04-21T08:53:12.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:12 smithi007 ceph-mon[28477]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T08:53:12.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:12 smithi007 ceph-mon[28477]: pgmap v265: 129 pgs: 18 active+clean+snaptrim, 60 active+clean, 51 active+clean+snaptrim_wait; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 1.61k op/s 2024-04-21T08:53:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:14 smithi016 ceph-mon[31259]: pgmap v266: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 1.85k op/s 2024-04-21T08:53:14.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:14 smithi077 ceph-mon[31615]: pgmap v266: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 1.85k op/s 2024-04-21T08:53:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:14 smithi007 ceph-mon[28477]: pgmap v266: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 1.85k op/s 2024-04-21T08:53:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:16 smithi016 ceph-mon[31259]: pgmap v267: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 1.55k op/s 2024-04-21T08:53:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:16 smithi016 ceph-mon[31259]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T08:53:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:16 smithi077 ceph-mon[31615]: pgmap v267: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 1.55k op/s 2024-04-21T08:53:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:16 smithi077 ceph-mon[31615]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T08:53:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:16 smithi007 ceph-mon[28477]: pgmap v267: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 1.55k op/s 2024-04-21T08:53:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:16 smithi007 ceph-mon[28477]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T08:53:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:18 smithi016 ceph-mon[31259]: pgmap v269: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 1.31k op/s 2024-04-21T08:53:18.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:18 smithi077 ceph-mon[31615]: pgmap v269: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 1.31k op/s 2024-04-21T08:53:18.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:18 smithi007 ceph-mon[28477]: pgmap v269: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 1.31k op/s 2024-04-21T08:53:20.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:20 smithi016 ceph-mon[31259]: pgmap v270: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 1.02k op/s 2024-04-21T08:53:20.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:20 smithi077 ceph-mon[31615]: pgmap v270: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 1.02k op/s 2024-04-21T08:53:20.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:20 smithi007 ceph-mon[28477]: pgmap v270: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 1.02k op/s 2024-04-21T08:53:22.160 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.159+0000 7f25ec796640 1 -- 172.21.15.7:0/595321359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25e41010c0 msgr2=0x7f25e41014a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:22.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.159+0000 7f25ec796640 1 --2- 172.21.15.7:0/595321359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25e41010c0 0x7f25e41014a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f25e0009860 tx=0x7f25e002bbb0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.160+0000 7f25ec796640 1 -- 172.21.15.7:0/595321359 shutdown_connections 2024-04-21T08:53:22.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.160+0000 7f25ec796640 1 --2- 172.21.15.7:0/595321359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e4106190 0x7f25e406a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.160+0000 7f25ec796640 1 --2- 172.21.15.7:0/595321359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25e4101a70 0x7f25e4105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.160+0000 7f25ec796640 1 --2- 172.21.15.7:0/595321359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25e41010c0 0x7f25e41014a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.160+0000 7f25ec796640 1 -- 172.21.15.7:0/595321359 >> 172.21.15.7:0/595321359 conn(0x7f25e4077fa0 msgr2=0x7f25e40783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:22.161 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.161+0000 7f25ec796640 1 -- 172.21.15.7:0/595321359 shutdown_connections 2024-04-21T08:53:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.161+0000 7f25ec796640 1 -- 172.21.15.7:0/595321359 wait complete. 2024-04-21T08:53:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.162+0000 7f25ec796640 1 Processor -- start 2024-04-21T08:53:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.162+0000 7f25ec796640 1 -- start start 2024-04-21T08:53:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.162+0000 7f25ec796640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e41010c0 0x7f25e4110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.162+0000 7f25ec796640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25e4106190 0x7f25e4110580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.162+0000 7f25ec796640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25e4110ac0 0x7f25e4072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.162+0000 7f25ec796640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f25e406baf0 con 0x7f25e41010c0 2024-04-21T08:53:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.162+0000 7f25ec796640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25e406b970 con 0x7f25e4106190 2024-04-21T08:53:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.162+0000 7f25ec796640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25e406bc70 con 0x7f25e4110ac0 2024-04-21T08:53:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.163+0000 7f25ea532640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e41010c0 0x7f25e4110040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.163+0000 7f25ea532640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e41010c0 0x7f25e4110040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59554/0 (socket says 172.21.15.7:59554) 2024-04-21T08:53:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.163+0000 7f25ea532640 1 -- 172.21.15.7:0/1595898429 learned_addr learned my addr 172.21.15.7:0/1595898429 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.163+0000 7f25e9d31640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25e4106190 0x7f25e4110580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.164+0000 7f25ea532640 1 -- 172.21.15.7:0/1595898429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25e4110ac0 msgr2=0x7f25e4072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.164+0000 7f25ea532640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25e4110ac0 0x7f25e4072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.164+0000 7f25ea532640 1 -- 172.21.15.7:0/1595898429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25e4106190 msgr2=0x7f25e4110580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.164+0000 7f25ea532640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25e4106190 0x7f25e4110580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.164+0000 7f25ea532640 1 -- 172.21.15.7:0/1595898429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e4073510 con 0x7f25e41010c0 2024-04-21T08:53:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.165+0000 7f25ea532640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e41010c0 0x7f25e4110040 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f25e0009830 tx=0x7f25e002cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.166+0000 7f25db7fe640 1 -- 172.21.15.7:0/1595898429 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e0041030 con 0x7f25e41010c0 2024-04-21T08:53:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.166+0000 7f25db7fe640 1 -- 172.21.15.7:0/1595898429 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25e0005140 con 0x7f25e41010c0 2024-04-21T08:53:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.166+0000 7f25db7fe640 1 -- 172.21.15.7:0/1595898429 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e0039040 con 0x7f25e41010c0 2024-04-21T08:53:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.166+0000 7f25ec796640 1 -- 172.21.15.7:0/1595898429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e4073740 con 0x7f25e41010c0 2024-04-21T08:53:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.166+0000 7f25ec796640 1 -- 172.21.15.7:0/1595898429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e4073be0 con 0x7f25e41010c0 2024-04-21T08:53:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.170+0000 7f25ef386740 1 Processor -- start 2024-04-21T08:53:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.171+0000 7f25ef386740 1 -- start start 2024-04-21T08:53:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.171+0000 7f25ef386740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88d8d0 0x5568ab88c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.171+0000 7f25ef386740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab88cb60 0x5568ab88cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.171+0000 7f25ef386740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab8960e0 0x5568ab8984d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.171 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.171+0000 7f25ef386740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5568ab7f18d0 con 0x5568ab8960e0 2024-04-21T08:53:22.172 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.171+0000 7f25ef386740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5568ab78ee80 con 0x5568ab88cb60 2024-04-21T08:53:22.172 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.171+0000 7f25ef386740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5568ab7ed890 con 0x5568ab88d8d0 2024-04-21T08:53:22.172 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.171+0000 7f25ead33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88d8d0 0x5568ab88c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.172 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.171+0000 7f25ead33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88d8d0 0x5568ab88c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:33954/0 (socket says 172.21.15.7:33954) 2024-04-21T08:53:22.172 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.171+0000 7f25ead33640 1 -- 172.21.15.7:0/3092898238 learned_addr learned my addr 172.21.15.7:0/3092898238 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.172+0000 7f25ead33640 1 -- 172.21.15.7:0/3092898238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab88cb60 msgr2=0x5568ab88cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.172+0000 7f25ead33640 1 --2- 172.21.15.7:0/3092898238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab88cb60 0x5568ab88cfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.172+0000 7f25ead33640 1 -- 172.21.15.7:0/3092898238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab8960e0 msgr2=0x5568ab8984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.172+0000 7f25ead33640 1 --2- 172.21.15.7:0/3092898238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab8960e0 0x5568ab8984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.172+0000 7f25ead33640 1 -- 172.21.15.7:0/3092898238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568ab7b0320 con 0x5568ab88d8d0 2024-04-21T08:53:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.174+0000 7f25ead33640 1 --2- 172.21.15.7:0/3092898238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88d8d0 0x5568ab88c620 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f25dc005c40 tx=0x7f25dc0307d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d3f3ff8d20755d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:22.177 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.175+0000 7f25d8ff9640 1 -- 172.21.15.7:0/3092898238 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc003b70 con 0x5568ab88d8d0 2024-04-21T08:53:22.177 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.176+0000 7f25db7fe640 1 -- 172.21.15.7:0/1595898429 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f25e0044070 con 0x7f25e41010c0 2024-04-21T08:53:22.177 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.177+0000 7f25db7fe640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f25d00a3f20 0x7f25d00a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.178+0000 7f25db7fe640 1 -- 172.21.15.7:0/1595898429 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9716+0+0 (secure 0 0 0) 0x7f25e00f1680 con 0x7f25e41010c0 2024-04-21T08:53:22.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.178+0000 7f25d8ff9640 1 -- 172.21.15.7:0/3092898238 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25dc003d10 con 0x5568ab88d8d0 2024-04-21T08:53:22.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.179+0000 7f25ef386740 1 -- 172.21.15.7:0/3092898238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88d8d0 msgr2=0x5568ab88c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:22.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.179+0000 7f25ef386740 1 --2- 172.21.15.7:0/3092898238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88d8d0 0x5568ab88c620 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f25dc005c40 tx=0x7f25dc0307d0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.180 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.180+0000 7f25ea532640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f25d00a3f20 0x7f25d00a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.181+0000 7f25ef386740 1 -- 172.21.15.7:0/3092898238 shutdown_connections 2024-04-21T08:53:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.181+0000 7f25ef386740 1 --2- 172.21.15.7:0/3092898238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab8960e0 0x5568ab8984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.181+0000 7f25ef386740 1 --2- 172.21.15.7:0/3092898238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab88cb60 0x5568ab88cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.181+0000 7f25ef386740 1 --2- 172.21.15.7:0/3092898238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88d8d0 0x5568ab88c620 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.181+0000 7f25ef386740 1 -- 172.21.15.7:0/3092898238 >> 172.21.15.7:0/3092898238 conn(0x5568ab853c90 msgr2=0x5568ab887850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.181+0000 7f25ea532640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f25d00a3f20 0x7f25d00a63e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x5568ab88e030 tx=0x7f25e00bb2b0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:22.189 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.189+0000 7f25ef386740 1 -- 172.21.15.7:0/3092898238 shutdown_connections 2024-04-21T08:53:22.191 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.192+0000 7f25ef386740 1 -- 172.21.15.7:0/3092898238 wait complete. 2024-04-21T08:53:22.193 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.192+0000 7f25ef386740 1 -- start start 2024-04-21T08:53:22.193 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.193+0000 7f25ef386740 1 Processor -- start 2024-04-21T08:53:22.193 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.193+0000 7f25ef386740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab895920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.193+0000 7f25ef386740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab892cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.194+0000 7f25ead33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab895920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.194+0000 7f25ead33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab895920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:33964/0 (socket says 172.21.15.7:33964) 2024-04-21T08:53:22.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.194+0000 7f25ef386740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab8931f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.194+0000 7f25ead33640 1 -- 172.21.15.7:0/2376853439 learned_addr learned my addr 172.21.15.7:0/2376853439 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.194+0000 7f25ef386740 1 -- 172.21.15.7:0/2376853439 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5568ab898eb0 con 0x5568ab88d8d0 2024-04-21T08:53:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.194+0000 7f25ef386740 1 -- 172.21.15.7:0/2376853439 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5568ab779eb0 con 0x5568ab8960e0 2024-04-21T08:53:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.194+0000 7f25ef386740 1 -- 172.21.15.7:0/2376853439 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5568ab899030 con 0x5568ab88cb60 2024-04-21T08:53:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.194+0000 7f25ead33640 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab8931f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.194+0000 7f25ef386740 10 client.0 _ll_register_callbacks cb 0x5568ab7a6b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.195+0000 7f25ead33640 1 -- 172.21.15.7:0/2376853439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 msgr2=0x5568ab8931f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.195+0000 7f25ead33640 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab8931f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.195+0000 7f25ead33640 1 -- 172.21.15.7:0/2376853439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 msgr2=0x5568ab892cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:22.195 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.195+0000 7f25ead33640 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab892cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.195+0000 7f25ead33640 1 -- 172.21.15.7:0/2376853439 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568ab504210 con 0x5568ab88cb60 2024-04-21T08:53:22.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.197+0000 7f25ead33640 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab895920 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f25dc004d80 tx=0x7f25dc0095a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42ba1eec87a140e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.202+0000 7f25cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:53:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.202+0000 7f25cdffb640 1 -- 172.21.15.7:0/2376853439 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc004430 con 0x5568ab88cb60 2024-04-21T08:53:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.202+0000 7f25ef386740 1 -- 172.21.15.7:0/2376853439 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25dc003d10 con 0x5568ab88cb60 2024-04-21T08:53:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.202+0000 7f25cdffb640 1 -- 172.21.15.7:0/2376853439 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9716+0+0 (secure 0 0 0) 0x7f25dc0045d0 con 0x5568ab88cb60 2024-04-21T08:53:22.202 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.203+0000 7f25cdffb640 1 -- 172.21.15.7:0/2376853439 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f25dc055050 con 0x5568ab88cb60 2024-04-21T08:53:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.206+0000 7f25ef386740 10 client.26635 fetch_fsmap learned FSMap version 23 2024-04-21T08:53:22.205 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.206+0000 7f25ef386740 1 -- 172.21.15.7:0/2376853439 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5568ab5d0620 con 0x5568ab88cb60 2024-04-21T08:53:22.209 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.209+0000 7f25cdffb640 1 -- 172.21.15.7:0/2376853439 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25dc047900 con 0x5568ab88cb60 2024-04-21T08:53:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.210+0000 7f25cdffb640 1 -- 172.21.15.7:0/2376853439 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f25dc044040 con 0x5568ab88cb60 2024-04-21T08:53:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.211+0000 7f25ef386740 10 client.26635 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:53:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.211+0000 7f25ef386740 10 client.26635 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.211+0000 7f25ef386740 20 client.26635 populate_metadata read hostname 'smithi007' 2024-04-21T08:53:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.211+0000 7f25ef386740 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5568ab8a64a0 0x5568ab8a8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.211+0000 7f25ef386740 4 client.26635 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:22.211 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.211+0000 7f25ef386740 1 -- 172.21.15.7:0/2376853439 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f25dc003d10 con 0x5568ab8a64a0 2024-04-21T08:53:22.218 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.218+0000 7f25e9d31640 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5568ab8a64a0 0x5568ab8a8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.240 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.241+0000 7f25e9d31640 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5568ab8a64a0 0x5568ab8a8890 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26b73968f357fe5d server_cookie=f8dd66bebb97419f in_seq=0 out_seq=0 2024-04-21T08:53:22.241 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.241+0000 7f25cdffb640 10 client.26635 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.245+0000 7f25cdffb640 1 -- 172.21.15.7:0/2376853439 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f25dc003d10 con 0x5568ab8a64a0 2024-04-21T08:53:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.245+0000 7f25cdffb640 10 client.26635 handle_command_reply: tid=0 2024-04-21T08:53:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.245+0000 7f25ef386740 1 client.26635 shutdown 2024-04-21T08:53:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.248+0000 7f25ef386740 1 -- 172.21.15.7:0/2376853439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 msgr2=0x5568ab895920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:22.247 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.248+0000 7f25ef386740 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab895920 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f25dc004d80 tx=0x7f25dc0095a0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.251+0000 7f25ef386740 1 -- 172.21.15.7:0/2376853439 shutdown_connections 2024-04-21T08:53:22.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.251+0000 7f25ef386740 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5568ab8a64a0 0x5568ab8a8890 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.251+0000 7f25ef386740 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab8931f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.251+0000 7f25ef386740 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab892cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.251+0000 7f25ef386740 1 --2- 172.21.15.7:0/2376853439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab895920 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.251 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.251+0000 7f25ef386740 1 -- 172.21.15.7:0/2376853439 >> 172.21.15.7:0/2376853439 conn(0x5568ab853c90 msgr2=0x5568ab8895a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:22.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.253+0000 7f25cdffb640 0 client.26635 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:22.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.254+0000 7f25ef386740 1 -- 172.21.15.7:0/2376853439 shutdown_connections 2024-04-21T08:53:22.256 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.256+0000 7f25ef386740 1 -- 172.21.15.7:0/2376853439 wait complete. 2024-04-21T08:53:22.257 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.258+0000 7f25ef386740 20 client.26635 trim_cache size 0 max 16384 2024-04-21T08:53:22.288 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.289+0000 7f25ef386740 1 Processor -- start 2024-04-21T08:53:22.289 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.290+0000 7f25ef386740 1 -- start start 2024-04-21T08:53:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.290+0000 7f25ef386740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab8a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.290+0000 7f25ef386740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab8953e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.290+0000 7f25ef386740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab895920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.290 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.290+0000 7f25ef386740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41b72c0 con 0x5568ab88d8d0 2024-04-21T08:53:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.291+0000 7f25ef386740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41b6520 con 0x5568ab8960e0 2024-04-21T08:53:22.291 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.291+0000 7f25ef386740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41b2360 con 0x5568ab88cb60 2024-04-21T08:53:22.292 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.293+0000 7f25e9d31640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab8953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.293 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.293+0000 7f25ead33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab8a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.293 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.293+0000 7f25e9d31640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab8953e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59568/0 (socket says 172.21.15.7:59568) 2024-04-21T08:53:22.293 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.293+0000 7f25e9d31640 1 -- 172.21.15.7:0/2736558202 learned_addr learned my addr 172.21.15.7:0/2736558202 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:22.293 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.294+0000 7f25ead33640 1 --2- 172.21.15.7:0/2736558202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab895920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.294+0000 7f25e9d31640 1 -- 172.21.15.7:0/2736558202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 msgr2=0x5568ab8a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.294+0000 7f25e9d31640 1 --2- 172.21.15.7:0/2736558202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab8a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.294+0000 7f25e9d31640 1 -- 172.21.15.7:0/2736558202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 msgr2=0x5568ab895920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.294+0000 7f25e9d31640 1 --2- 172.21.15.7:0/2736558202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab895920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.295 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.294+0000 7f25e9d31640 1 -- 172.21.15.7:0/2736558202 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568ab882f30 con 0x5568ab88d8d0 2024-04-21T08:53:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.295+0000 7f25e9d31640 1 --2- 172.21.15.7:0/2736558202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab8953e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f25d40058c0 tx=0x7f25d4005d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beff461cd6adec7d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.295+0000 7f25cffff640 1 -- 172.21.15.7:0/2736558202 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d401c030 con 0x5568ab88d8d0 2024-04-21T08:53:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.295+0000 7f25cffff640 1 -- 172.21.15.7:0/2736558202 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25d40054d0 con 0x5568ab88d8d0 2024-04-21T08:53:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.296+0000 7f25ef386740 1 -- 172.21.15.7:0/2736558202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 msgr2=0x5568ab8953e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:22.296 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.296+0000 7f25ef386740 1 --2- 172.21.15.7:0/2736558202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab8953e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f25d40058c0 tx=0x7f25d4005d00 comp rx=0 tx=0).stop 2024-04-21T08:53:22.303 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.302+0000 7f25ef386740 1 -- 172.21.15.7:0/2736558202 shutdown_connections 2024-04-21T08:53:22.303 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.302+0000 7f25ef386740 1 --2- 172.21.15.7:0/2736558202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab895920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.303 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.302+0000 7f25ef386740 1 --2- 172.21.15.7:0/2736558202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab8953e0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.303 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.302+0000 7f25ef386740 1 --2- 172.21.15.7:0/2736558202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab8a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.303 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.302+0000 7f25ef386740 1 -- 172.21.15.7:0/2736558202 >> 172.21.15.7:0/2736558202 conn(0x5568ab853c90 msgr2=0x5568ab88e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:22.304 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.305+0000 7f25ef386740 1 -- 172.21.15.7:0/2736558202 shutdown_connections 2024-04-21T08:53:22.304 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.305+0000 7f25ef386740 1 -- 172.21.15.7:0/2736558202 wait complete. 2024-04-21T08:53:22.305 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.305+0000 7f25ef386740 1 -- start start 2024-04-21T08:53:22.305 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.305+0000 7f25ef386740 1 Processor -- start 2024-04-21T08:53:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.309+0000 7f25ef386740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab89e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.309+0000 7f25ef386740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab890030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.309+0000 7f25ef386740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab890770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.309+0000 7f25ef386740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41b49e0 con 0x5568ab88d8d0 2024-04-21T08:53:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.309+0000 7f25ef386740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41b5e50 con 0x5568ab8960e0 2024-04-21T08:53:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.309+0000 7f25ef386740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41b50b0 con 0x5568ab88cb60 2024-04-21T08:53:22.309 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.309+0000 7f25ef386740 10 client.0 _ll_register_callbacks cb 0x7f25dc009e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:22.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.311+0000 7f25ead33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab89e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.311+0000 7f25ead33640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab89e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:33970/0 (socket says 172.21.15.7:33970) 2024-04-21T08:53:22.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.311+0000 7f25ead33640 1 -- 172.21.15.7:0/1322876294 learned_addr learned my addr 172.21.15.7:0/1322876294 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:22.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.311+0000 7f25e9d31640 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab890030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.311 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.311+0000 7f25ead33640 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab890770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.314+0000 7f25ead33640 1 -- 172.21.15.7:0/1322876294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 msgr2=0x5568ab890770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:22.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.314+0000 7f25ead33640 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab890770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.314+0000 7f25ead33640 1 -- 172.21.15.7:0/1322876294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 msgr2=0x5568ab890030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:22.318 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.314+0000 7f25ead33640 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab890030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.318 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.314+0000 7f25ead33640 1 -- 172.21.15.7:0/1322876294 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568ab88ef40 con 0x5568ab88cb60 2024-04-21T08:53:22.318 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.318+0000 7f25ead33640 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab89e890 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f25dc002230 tx=0x7f25dc033040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c97600f69e9dc9e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:22.324 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.323+0000 7f25cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:53:22.324 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.323+0000 7f25cdffb640 1 -- 172.21.15.7:0/1322876294 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25dc038ba0 con 0x5568ab88cb60 2024-04-21T08:53:22.324 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.323+0000 7f25cdffb640 1 -- 172.21.15.7:0/1322876294 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9716+0+0 (secure 0 0 0) 0x7f25dc038d40 con 0x5568ab88cb60 2024-04-21T08:53:22.324 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.324+0000 7f25ef386740 1 -- 172.21.15.7:0/1322876294 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25d40054d0 con 0x5568ab88cb60 2024-04-21T08:53:22.326 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.327+0000 7f25cdffb640 1 -- 172.21.15.7:0/1322876294 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f25dc004230 con 0x5568ab88cb60 2024-04-21T08:53:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.329+0000 7f25ef386740 10 client.26641 fetch_fsmap learned FSMap version 23 2024-04-21T08:53:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.329+0000 7f25ef386740 1 -- 172.21.15.7:0/1322876294 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5568ab90bc20 con 0x5568ab88cb60 2024-04-21T08:53:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.333+0000 7f25cdffb640 1 -- 172.21.15.7:0/1322876294 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25d40054d0 con 0x5568ab88cb60 2024-04-21T08:53:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.334+0000 7f25cdffb640 1 -- 172.21.15.7:0/1322876294 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f25dc00b040 con 0x5568ab88cb60 2024-04-21T08:53:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.334+0000 7f25ef386740 10 client.26641 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:53:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.334+0000 7f25ef386740 10 client.26641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.334+0000 7f25ef386740 20 client.26641 populate_metadata read hostname 'smithi007' 2024-04-21T08:53:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.334+0000 7f25ef386740 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5568ab90c280 0x5568ab90e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.335+0000 7f25ef386740 4 client.26641 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:53:22.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.335+0000 7f25ef386740 1 -- 172.21.15.7:0/1322876294 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5568ab805880 con 0x5568ab90c280 2024-04-21T08:53:22.335 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.335+0000 7f25e9d31640 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5568ab90c280 0x5568ab90e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:22.341 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.341+0000 7f25e9d31640 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5568ab90c280 0x5568ab90e6a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb69256c7a8f2ae2 server_cookie=173acc3ccd829a1d in_seq=0 out_seq=0 2024-04-21T08:53:22.341 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.341+0000 7f25cdffb640 10 client.26641 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:22.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.683+0000 7f25cdffb640 1 -- 172.21.15.7:0/1322876294 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5568ab805880 con 0x5568ab90c280 2024-04-21T08:53:22.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.683+0000 7f25cdffb640 10 client.26641 handle_command_reply: tid=0 2024-04-21T08:53:22.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.683+0000 7f25ef386740 1 client.26641 shutdown 2024-04-21T08:53:22.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.683+0000 7f25ef386740 1 -- 172.21.15.7:0/1322876294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 msgr2=0x5568ab89e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:22.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.683+0000 7f25ef386740 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab89e890 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f25dc002230 tx=0x7f25dc033040 comp rx=0 tx=0).stop 2024-04-21T08:53:22.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.684+0000 7f25ef386740 1 -- 172.21.15.7:0/1322876294 shutdown_connections 2024-04-21T08:53:22.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.684+0000 7f25ef386740 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5568ab90c280 0x5568ab90e6a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.684+0000 7f25ef386740 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5568ab8960e0 0x5568ab890770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.684+0000 7f25ef386740 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5568ab88d8d0 0x5568ab890030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.684+0000 7f25ef386740 1 --2- 172.21.15.7:0/1322876294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5568ab88cb60 0x5568ab89e890 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.684+0000 7f25ef386740 1 -- 172.21.15.7:0/1322876294 >> 172.21.15.7:0/1322876294 conn(0x5568ab853c90 msgr2=0x5568ab888c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:22.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.684+0000 7f25cdffb640 0 client.26641 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:22.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.684+0000 7f25ef386740 1 -- 172.21.15.7:0/1322876294 shutdown_connections 2024-04-21T08:53:22.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.684+0000 7f25ef386740 1 -- 172.21.15.7:0/1322876294 wait complete. 2024-04-21T08:53:22.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.684+0000 7f25ef386740 20 client.26641 trim_cache size 0 max 16384 2024-04-21T08:53:22.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:53:22.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:53:22.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:53:22.686 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:53:22.686 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.685+0000 7f25ec796640 1 -- 172.21.15.7:0/1595898429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f25d00a3f20 msgr2=0x7f25d00a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:22.686 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.685+0000 7f25ec796640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f25d00a3f20 0x7f25d00a63e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x5568ab88e030 tx=0x7f25e00bb2b0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.685+0000 7f25ec796640 1 -- 172.21.15.7:0/1595898429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e41010c0 msgr2=0x7f25e4110040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.685+0000 7f25ec796640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e41010c0 0x7f25e4110040 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f25e0009830 tx=0x7f25e002cb40 comp rx=0 tx=0).stop 2024-04-21T08:53:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.685+0000 7f25ec796640 1 -- 172.21.15.7:0/1595898429 shutdown_connections 2024-04-21T08:53:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.685+0000 7f25ec796640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f25d00a3f20 0x7f25d00a63e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.685+0000 7f25ec796640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25e4110ac0 0x7f25e4072fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.685+0000 7f25ec796640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25e4106190 0x7f25e4110580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.685+0000 7f25ec796640 1 --2- 172.21.15.7:0/1595898429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e41010c0 0x7f25e4110040 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.685+0000 7f25ec796640 1 -- 172.21.15.7:0/1595898429 >> 172.21.15.7:0/1595898429 conn(0x7f25e4077fa0 msgr2=0x7f25e4104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.685+0000 7f25ec796640 1 -- 172.21.15.7:0/1595898429 shutdown_connections 2024-04-21T08:53:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:22.686+0000 7f25ec796640 1 -- 172.21.15.7:0/1595898429 wait complete. 2024-04-21T08:53:22.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:22 smithi016 ceph-mon[31259]: pgmap v271: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 925 KiB/s wr, 827 op/s 2024-04-21T08:53:22.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:22 smithi077 ceph-mon[31615]: pgmap v271: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 925 KiB/s wr, 827 op/s 2024-04-21T08:53:22.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:22 smithi007 ceph-mon[28477]: pgmap v271: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 925 KiB/s wr, 827 op/s 2024-04-21T08:53:24.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:24 smithi016 ceph-mon[31259]: pgmap v272: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 408 KiB/s wr, 685 op/s 2024-04-21T08:53:24.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:24 smithi077 ceph-mon[31615]: pgmap v272: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 408 KiB/s wr, 685 op/s 2024-04-21T08:53:24.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:24 smithi007 ceph-mon[28477]: pgmap v272: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 408 KiB/s wr, 685 op/s 2024-04-21T08:53:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:26 smithi016 ceph-mon[31259]: pgmap v273: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 425 KiB/s wr, 654 op/s 2024-04-21T08:53:26.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:26 smithi077 ceph-mon[31615]: pgmap v273: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 425 KiB/s wr, 654 op/s 2024-04-21T08:53:26.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:26 smithi007 ceph-mon[28477]: pgmap v273: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.4 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 425 KiB/s wr, 654 op/s 2024-04-21T08:53:28.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:28 smithi016 ceph-mon[31259]: pgmap v274: 129 pgs: 20 active+clean+snaptrim, 58 active+clean, 51 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 466 KiB/s wr, 740 op/s 2024-04-21T08:53:28.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:28 smithi077 ceph-mon[31615]: pgmap v274: 129 pgs: 20 active+clean+snaptrim, 58 active+clean, 51 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 466 KiB/s wr, 740 op/s 2024-04-21T08:53:28.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:28 smithi007 ceph-mon[28477]: pgmap v274: 129 pgs: 20 active+clean+snaptrim, 58 active+clean, 51 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 466 KiB/s wr, 740 op/s 2024-04-21T08:53:29.998 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:53:29.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:53:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.179+0000 7f93a617e640 1 -- 172.21.15.7:0/2286601112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93a0107270 msgr2=0x7f93a010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:30.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.179+0000 7f93a617e640 1 --2- 172.21.15.7:0/2286601112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93a0107270 0x7f93a010ee40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f939801acb0 tx=0x7f939803d150 comp rx=0 tx=0).stop 2024-04-21T08:53:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.180+0000 7f93a617e640 1 -- 172.21.15.7:0/2286601112 shutdown_connections 2024-04-21T08:53:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.180+0000 7f93a617e640 1 --2- 172.21.15.7:0/2286601112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93a0107270 0x7f93a010ee40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.180+0000 7f93a617e640 1 --2- 172.21.15.7:0/2286601112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a0102b50 0x7f93a0106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.180+0000 7f93a617e640 1 --2- 172.21.15.7:0/2286601112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93a01021a0 0x7f93a0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.180+0000 7f93a617e640 1 -- 172.21.15.7:0/2286601112 >> 172.21.15.7:0/2286601112 conn(0x7f93a00fda40 msgr2=0x7f93a00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:30.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.184+0000 7f93a617e640 1 -- 172.21.15.7:0/2286601112 shutdown_connections 2024-04-21T08:53:30.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.184+0000 7f93a617e640 1 -- 172.21.15.7:0/2286601112 wait complete. 2024-04-21T08:53:30.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.185+0000 7f93a617e640 1 Processor -- start 2024-04-21T08:53:30.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.189+0000 7f93a617e640 1 -- start start 2024-04-21T08:53:30.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.189+0000 7f93a617e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93a01021a0 0x7f93a0075820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.189+0000 7f93a617e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93a0107270 0x7f93a0075d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.189+0000 7f93a617e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a006f8e0 0x7f93a006fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.189+0000 7f93a617e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f93a006e2e0 con 0x7f93a0107270 2024-04-21T08:53:30.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.189+0000 7f93a617e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f93a006e160 con 0x7f93a01021a0 2024-04-21T08:53:30.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.189+0000 7f93a617e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f93a006e460 con 0x7f93a006f8e0 2024-04-21T08:53:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.192+0000 7f939f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93a01021a0 0x7f93a0075820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.192+0000 7f939ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a006f8e0 0x7f93a006fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.192+0000 7f939ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a006f8e0 0x7f93a006fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:50556/0 (socket says 172.21.15.7:50556) 2024-04-21T08:53:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.192+0000 7f939ffff640 1 -- 172.21.15.7:0/2666522388 learned_addr learned my addr 172.21.15.7:0/2666522388 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:30.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.193+0000 7f9397fff640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93a0107270 0x7f93a0075d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.194+0000 7f939ffff640 1 -- 172.21.15.7:0/2666522388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93a01021a0 msgr2=0x7f93a0075820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:30.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.194+0000 7f939ffff640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93a01021a0 0x7f93a0075820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.194+0000 7f939ffff640 1 -- 172.21.15.7:0/2666522388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93a0107270 msgr2=0x7f93a0075d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:30.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.194+0000 7f939ffff640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93a0107270 0x7f93a0075d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.194+0000 7f939ffff640 1 -- 172.21.15.7:0/2666522388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a0070600 con 0x7f93a006f8e0 2024-04-21T08:53:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.198+0000 7f939ffff640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a006f8e0 0x7f93a006fd40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f939803d6e0 tx=0x7f9398003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.198+0000 7f939d7fa640 1 -- 172.21.15.7:0/2666522388 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9398052020 con 0x7f93a006f8e0 2024-04-21T08:53:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.198+0000 7f939d7fa640 1 -- 172.21.15.7:0/2666522388 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9398004570 con 0x7f93a006f8e0 2024-04-21T08:53:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.198+0000 7f939d7fa640 1 -- 172.21.15.7:0/2666522388 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939804a040 con 0x7f93a006f8e0 2024-04-21T08:53:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.199+0000 7f93a617e640 1 -- 172.21.15.7:0/2666522388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93a019a6d0 con 0x7f93a006f8e0 2024-04-21T08:53:30.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.199+0000 7f93a617e640 1 -- 172.21.15.7:0/2666522388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93a019abd0 con 0x7f93a006f8e0 2024-04-21T08:53:30.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.203+0000 7f93a8d6c740 1 Processor -- start 2024-04-21T08:53:30.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.203+0000 7f93a8d6c740 1 -- start start 2024-04-21T08:53:30.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.203+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821bf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.203+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bfea0 0x562a821c8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.203+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821cae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.203+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562a821241e0 con 0x562a821c8a80 2024-04-21T08:53:30.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.203+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562a820c1390 con 0x562a821bfea0 2024-04-21T08:53:30.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.203+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562a821201a0 con 0x562a821bf580 2024-04-21T08:53:30.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.205+0000 7f939d7fa640 1 -- 172.21.15.7:0/2666522388 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f9398055020 con 0x7f93a006f8e0 2024-04-21T08:53:30.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.206+0000 7f939d7fa640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f938c0a3e60 0x7f938c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.206+0000 7f939f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bfea0 0x562a821c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.207+0000 7f9397fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821bf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.207+0000 7f9397fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821bf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:50570/0 (socket says 172.21.15.7:50570) 2024-04-21T08:53:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.207+0000 7f9397fff640 1 -- 172.21.15.7:0/663587979 learned_addr learned my addr 172.21.15.7:0/663587979 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.207+0000 7f9397fff640 1 --2- 172.21.15.7:0/663587979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.207+0000 7f939ffff640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f938c0a3e60 0x7f938c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.207+0000 7f939d7fa640 1 -- 172.21.15.7:0/2666522388 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9716+0+0 (secure 0 0 0) 0x7f9398103d10 con 0x7f93a006f8e0 2024-04-21T08:53:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.208+0000 7f939f7fe640 1 -- 172.21.15.7:0/663587979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 msgr2=0x562a821bf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.208+0000 7f939f7fe640 1 --2- 172.21.15.7:0/663587979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821bf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.208+0000 7f939f7fe640 1 -- 172.21.15.7:0/663587979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 msgr2=0x562a821cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.208+0000 7f939f7fe640 1 --2- 172.21.15.7:0/663587979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821cae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.208+0000 7f939f7fe640 1 -- 172.21.15.7:0/663587979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a81f007d0 con 0x562a821bfea0 2024-04-21T08:53:30.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.208+0000 7f939f7fe640 1 --2- 172.21.15.7:0/663587979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bfea0 0x562a821c8540 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f93900099a0 tx=0x7f9390009e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de9c737468517202 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:30.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.208+0000 7f939ffff640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f938c0a3e60 0x7f938c0a6320 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9398044000 tx=0x7f939803d150 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.210+0000 7f9395ffb640 1 -- 172.21.15.7:0/663587979 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9390017070 con 0x562a821bfea0 2024-04-21T08:53:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.210+0000 7f9395ffb640 1 -- 172.21.15.7:0/663587979 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9390013470 con 0x562a821bfea0 2024-04-21T08:53:30.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.210+0000 7f9395ffb640 1 -- 172.21.15.7:0/663587979 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9390012630 con 0x562a821bfea0 2024-04-21T08:53:30.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.212+0000 7f93a8d6c740 1 -- 172.21.15.7:0/663587979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bfea0 msgr2=0x562a821c8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:30.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.212+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/663587979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bfea0 0x562a821c8540 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f93900099a0 tx=0x7f9390009e60 comp rx=0 tx=0).stop 2024-04-21T08:53:30.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.219+0000 7f93a8d6c740 1 -- 172.21.15.7:0/663587979 shutdown_connections 2024-04-21T08:53:30.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.219+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/663587979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.219+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/663587979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bfea0 0x562a821c8540 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.219+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/663587979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821bf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.219+0000 7f93a8d6c740 1 -- 172.21.15.7:0/663587979 >> 172.21.15.7:0/663587979 conn(0x562a821b9cb0 msgr2=0x562a821ba090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:30.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.219+0000 7f93a8d6c740 1 -- 172.21.15.7:0/663587979 shutdown_connections 2024-04-21T08:53:30.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.220+0000 7f93a8d6c740 1 -- 172.21.15.7:0/663587979 wait complete. 2024-04-21T08:53:30.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.220+0000 7f93a8d6c740 1 -- start start 2024-04-21T08:53:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.220+0000 7f93a8d6c740 1 Processor -- start 2024-04-21T08:53:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.221+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.221+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821bfea0 0x562a821c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.222+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821c8a80 0x562a821c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.222+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562a821cb870 con 0x562a821bfea0 2024-04-21T08:53:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.222+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562a820dd2f0 con 0x562a821c8a80 2024-04-21T08:53:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.222+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562a821cb9f0 con 0x562a821bf580 2024-04-21T08:53:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.222+0000 7f93a8d6c740 10 client.0 _ll_register_callbacks cb 0x562a820ea580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:30.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.225+0000 7f9397fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.225+0000 7f9397fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:50580/0 (socket says 172.21.15.7:50580) 2024-04-21T08:53:30.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.225+0000 7f9397fff640 1 -- 172.21.15.7:0/2407313551 learned_addr learned my addr 172.21.15.7:0/2407313551 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.226+0000 7f9397fff640 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821c8a80 0x562a821c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.227+0000 7f939f7fe640 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821bfea0 0x562a821c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.227+0000 7f9397fff640 1 -- 172.21.15.7:0/2407313551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821c8a80 msgr2=0x562a821c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.227+0000 7f9397fff640 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821c8a80 0x562a821c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.227+0000 7f9397fff640 1 -- 172.21.15.7:0/2407313551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821bfea0 msgr2=0x562a821c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.227+0000 7f9397fff640 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821bfea0 0x562a821c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.227+0000 7f9397fff640 1 -- 172.21.15.7:0/2407313551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a81e363f0 con 0x562a821bf580 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.228+0000 7f9397fff640 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821c5b70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f938800a010 tx=0x7f93880046e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27a16d078f75069e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.228+0000 7f937effd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.228+0000 7f937effd640 1 -- 172.21.15.7:0/2407313551 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93880148d0 con 0x562a821bf580 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.228+0000 7f937effd640 1 -- 172.21.15.7:0/2407313551 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9716+0+0 (secure 0 0 0) 0x7f9388014a70 con 0x562a821bf580 2024-04-21T08:53:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.229+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2407313551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9390013470 con 0x562a821bf580 2024-04-21T08:53:30.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.231+0000 7f937effd640 1 -- 172.21.15.7:0/2407313551 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9388014c70 con 0x562a821bf580 2024-04-21T08:53:30.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.231+0000 7f93a8d6c740 10 client.26650 fetch_fsmap learned FSMap version 23 2024-04-21T08:53:30.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.231+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2407313551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562a81df60f0 con 0x562a821bf580 2024-04-21T08:53:30.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.232+0000 7f937effd640 1 -- 172.21.15.7:0/2407313551 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f938801d070 con 0x562a821bf580 2024-04-21T08:53:30.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.232+0000 7f937effd640 1 -- 172.21.15.7:0/2407313551 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f9388012070 con 0x562a821bf580 2024-04-21T08:53:30.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.232+0000 7f93a8d6c740 10 client.26650 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:53:30.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.232+0000 7f93a8d6c740 10 client.26650 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:30.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.232+0000 7f93a8d6c740 20 client.26650 populate_metadata read hostname 'smithi007' 2024-04-21T08:53:30.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.232+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a821d8f70 0x562a821db3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.232+0000 7f93a8d6c740 4 client.26650 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.232+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2407313551 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9390013470 con 0x562a821d8f70 2024-04-21T08:53:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.233+0000 7f939f7fe640 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a821d8f70 0x562a821db3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.233+0000 7f939f7fe640 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a821d8f70 0x562a821db3d0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1442cae20920a1c3 server_cookie=139460bd96654503 in_seq=0 out_seq=0 2024-04-21T08:53:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.233+0000 7f937effd640 10 client.26650 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.234+0000 7f937effd640 1 -- 172.21.15.7:0/2407313551 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f9390013470 con 0x562a821d8f70 2024-04-21T08:53:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.234+0000 7f937effd640 10 client.26650 handle_command_reply: tid=0 2024-04-21T08:53:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.234+0000 7f93a8d6c740 1 client.26650 shutdown 2024-04-21T08:53:30.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.237+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2407313551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 msgr2=0x562a821c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:30.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.237+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821c5b70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f938800a010 tx=0x7f93880046e0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.242+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2407313551 shutdown_connections 2024-04-21T08:53:30.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.242+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a821d8f70 0x562a821db3d0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.242+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821c8a80 0x562a821c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.242+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821bfea0 0x562a821c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.242+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2407313551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821c5b70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.242+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2407313551 >> 172.21.15.7:0/2407313551 conn(0x562a821b9cb0 msgr2=0x562a821bba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:30.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.242+0000 7f937effd640 0 client.26650 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:30.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.243+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2407313551 shutdown_connections 2024-04-21T08:53:30.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.243+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2407313551 wait complete. 2024-04-21T08:53:30.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.243+0000 7f93a8d6c740 20 client.26650 trim_cache size 0 max 16384 2024-04-21T08:53:30.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.283+0000 7f93a8d6c740 1 Processor -- start 2024-04-21T08:53:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.283+0000 7f93a8d6c740 1 -- start start 2024-04-21T08:53:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.283+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821d4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.283+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bfea0 0x562a821c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.283+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821e0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.283+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f93a01b3be0 con 0x562a821c8a80 2024-04-21T08:53:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.283+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f93a01b6440 con 0x562a821bfea0 2024-04-21T08:53:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.283+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f93a01b4230 con 0x562a821bf580 2024-04-21T08:53:30.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.285+0000 7f9397fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821d4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.285+0000 7f9397fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821d4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:50590/0 (socket says 172.21.15.7:50590) 2024-04-21T08:53:30.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.285+0000 7f9397fff640 1 -- 172.21.15.7:0/3756700611 learned_addr learned my addr 172.21.15.7:0/3756700611 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:30.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.285+0000 7f9397fff640 1 --2- 172.21.15.7:0/3756700611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821e0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.286+0000 7f9397fff640 1 -- 172.21.15.7:0/3756700611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bfea0 msgr2=0x562a821c6410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:30.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.286+0000 7f9397fff640 1 --2- 172.21.15.7:0/3756700611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bfea0 0x562a821c6410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.286+0000 7f9397fff640 1 -- 172.21.15.7:0/3756700611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 msgr2=0x562a821e0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:30.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.286+0000 7f9397fff640 1 --2- 172.21.15.7:0/3756700611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821e0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.286+0000 7f9397fff640 1 -- 172.21.15.7:0/3756700611 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a821d8d20 con 0x562a821bf580 2024-04-21T08:53:30.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.286+0000 7f9397fff640 1 --2- 172.21.15.7:0/3756700611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821d4730 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9388014f90 tx=0x7f9388004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af0393c16ba77b25 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:30.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.289+0000 7f93957fa640 1 -- 172.21.15.7:0/3756700611 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9388014920 con 0x562a821bf580 2024-04-21T08:53:30.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.289+0000 7f93957fa640 1 -- 172.21.15.7:0/3756700611 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f938801d040 con 0x562a821bf580 2024-04-21T08:53:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.290+0000 7f93a8d6c740 1 -- 172.21.15.7:0/3756700611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 msgr2=0x562a821d4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.290+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/3756700611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821d4730 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9388014f90 tx=0x7f9388004750 comp rx=0 tx=0).stop 2024-04-21T08:53:30.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.291+0000 7f93a8d6c740 1 -- 172.21.15.7:0/3756700611 shutdown_connections 2024-04-21T08:53:30.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.292+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/3756700611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821e0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.292+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/3756700611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bfea0 0x562a821c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.292+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/3756700611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bf580 0x562a821d4730 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.292+0000 7f93a8d6c740 1 -- 172.21.15.7:0/3756700611 >> 172.21.15.7:0/3756700611 conn(0x562a821b9cb0 msgr2=0x562a821b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.295+0000 7f93a8d6c740 1 -- 172.21.15.7:0/3756700611 shutdown_connections 2024-04-21T08:53:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.295+0000 7f93a8d6c740 1 -- 172.21.15.7:0/3756700611 wait complete. 2024-04-21T08:53:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.295+0000 7f93a8d6c740 1 -- start start 2024-04-21T08:53:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.295+0000 7f93a8d6c740 1 Processor -- start 2024-04-21T08:53:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.296+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bf580 0x562a821d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.296+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bfea0 0x562a821d1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.296+0000 7f93a8d6c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821c23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.296+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f93a01b4900 con 0x562a821c8a80 2024-04-21T08:53:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.296+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f93a01b5d70 con 0x562a821bf580 2024-04-21T08:53:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.296+0000 7f93a8d6c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f93a01b4fd0 con 0x562a821bfea0 2024-04-21T08:53:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.296+0000 7f93a8d6c740 10 client.0 _ll_register_callbacks cb 0x7f938800a7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:30.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.296+0000 7f9397fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821c23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.296+0000 7f9397fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bf580 0x562a821d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.296+0000 7f9397fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bf580 0x562a821d0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:40170/0 (socket says 172.21.15.7:40170) 2024-04-21T08:53:30.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.296+0000 7f9397fff640 1 -- 172.21.15.7:0/2787685511 learned_addr learned my addr 172.21.15.7:0/2787685511 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:30.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.297+0000 7f9397fff640 1 -- 172.21.15.7:0/2787685511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bfea0 msgr2=0x562a821d1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:30.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.297+0000 7f9397fff640 1 --2- 172.21.15.7:0/2787685511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bfea0 0x562a821d1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.297+0000 7f9397fff640 1 -- 172.21.15.7:0/2787685511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 msgr2=0x562a821c23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:30.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.297+0000 7f9397fff640 1 --2- 172.21.15.7:0/2787685511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821c23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.297+0000 7f9397fff640 1 -- 172.21.15.7:0/2787685511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a821c2bf0 con 0x562a821bf580 2024-04-21T08:53:30.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.297+0000 7f9397fff640 1 --2- 172.21.15.7:0/2787685511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bf580 0x562a821d0c40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9388018470 tx=0x7f93880176c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6c57e2f44adcf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:30.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.297+0000 7f937effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:53:30.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.297+0000 7f937effd640 1 -- 172.21.15.7:0/2787685511 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9388032040 con 0x562a821bf580 2024-04-21T08:53:30.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.297+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2787685511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f938801d040 con 0x562a821bf580 2024-04-21T08:53:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.299+0000 7f937effd640 1 -- 172.21.15.7:0/2787685511 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9716+0+0 (secure 0 0 0) 0x7f9388017020 con 0x562a821bf580 2024-04-21T08:53:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.299+0000 7f937effd640 1 -- 172.21.15.7:0/2787685511 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9388017240 con 0x562a821bf580 2024-04-21T08:53:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.299+0000 7f93a8d6c740 10 client.26678 fetch_fsmap learned FSMap version 23 2024-04-21T08:53:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.299+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2787685511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562a82261170 con 0x562a821bf580 2024-04-21T08:53:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.303+0000 7f937effd640 1 -- 172.21.15.7:0/2787685511 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f938801d040 con 0x562a821bf580 2024-04-21T08:53:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.303+0000 7f937effd640 1 -- 172.21.15.7:0/2787685511 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f9388012070 con 0x562a821bf580 2024-04-21T08:53:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.303+0000 7f93a8d6c740 10 client.26678 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:53:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.303+0000 7f93a8d6c740 10 client.26678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.303+0000 7f93a8d6c740 20 client.26678 populate_metadata read hostname 'smithi007' 2024-04-21T08:53:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.303+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2787685511 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a82261860 0x562a82263c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.303+0000 7f93a8d6c740 4 client.26678 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:53:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.303+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2787685511 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562a82131c00 con 0x562a82261860 2024-04-21T08:53:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.305+0000 7f939f7fe640 1 --2- 172.21.15.7:0/2787685511 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a82261860 0x562a82263c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:30.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.317+0000 7f939f7fe640 1 --2- 172.21.15.7:0/2787685511 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a82261860 0x562a82263c80 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1318caea3500fc87 server_cookie=40630a2d4ad740a6 in_seq=0 out_seq=0 2024-04-21T08:53:30.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.319+0000 7f937effd640 10 client.26678 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:30.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.323+0000 7f937effd640 1 -- 172.21.15.7:0/2787685511 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x562a82131c00 con 0x562a82261860 2024-04-21T08:53:30.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.323+0000 7f937effd640 10 client.26678 handle_command_reply: tid=0 2024-04-21T08:53:30.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.323+0000 7f93a8d6c740 1 client.26678 shutdown 2024-04-21T08:53:30.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.328+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2787685511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bf580 msgr2=0x562a821d0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:30.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.328+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2787685511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bf580 0x562a821d0c40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9388018470 tx=0x7f93880176c0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.328+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2787685511 shutdown_connections 2024-04-21T08:53:30.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.328+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2787685511 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562a82261860 0x562a82263c80 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.328+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2787685511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562a821c8a80 0x562a821c23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.328+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2787685511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562a821bfea0 0x562a821d1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.328+0000 7f93a8d6c740 1 --2- 172.21.15.7:0/2787685511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562a821bf580 0x562a821d0c40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.328+0000 7f937effd640 0 client.26678 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:30.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.328+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2787685511 >> 172.21.15.7:0/2787685511 conn(0x562a821b9cb0 msgr2=0x562a821b8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:30.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.334+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2787685511 shutdown_connections 2024-04-21T08:53:30.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.335+0000 7f93a8d6c740 1 -- 172.21.15.7:0/2787685511 wait complete. 2024-04-21T08:53:30.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.336+0000 7f93a8d6c740 20 client.26678 trim_cache size 0 max 16384 2024-04-21T08:53:30.337 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:53:30.337 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (21996 inodes in the stack)", 2024-04-21T08:53:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:53:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "2f44a19d-b0a7-4681-b27b-fb7529461620": { 2024-04-21T08:53:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:53:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "2f44a19d-b0a7-4681-b27b-fb7529461620", 2024-04-21T08:53:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:53:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:53:30.338 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:53:30.338 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:53:30.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.337+0000 7f93a617e640 1 -- 172.21.15.7:0/2666522388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f938c0a3e60 msgr2=0x7f938c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:30.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.337+0000 7f93a617e640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f938c0a3e60 0x7f938c0a6320 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9398044000 tx=0x7f939803d150 comp rx=0 tx=0).stop 2024-04-21T08:53:30.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.337+0000 7f93a617e640 1 -- 172.21.15.7:0/2666522388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a006f8e0 msgr2=0x7f93a006fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:30.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.337+0000 7f93a617e640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a006f8e0 0x7f93a006fd40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f939803d6e0 tx=0x7f9398003f90 comp rx=0 tx=0).stop 2024-04-21T08:53:30.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.341+0000 7f93a617e640 1 -- 172.21.15.7:0/2666522388 shutdown_connections 2024-04-21T08:53:30.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.341+0000 7f93a617e640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f938c0a3e60 0x7f938c0a6320 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.341+0000 7f93a617e640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93a006f8e0 0x7f93a006fd40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.341+0000 7f93a617e640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f93a0107270 0x7f93a0075d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.341+0000 7f93a617e640 1 --2- 172.21.15.7:0/2666522388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93a01021a0 0x7f93a0075820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:30.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.341+0000 7f93a617e640 1 -- 172.21.15.7:0/2666522388 >> 172.21.15.7:0/2666522388 conn(0x7f93a00fda40 msgr2=0x7f93a00fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.342+0000 7f93a617e640 1 -- 172.21.15.7:0/2666522388 shutdown_connections 2024-04-21T08:53:30.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:53:30.342+0000 7f93a617e640 1 -- 172.21.15.7:0/2666522388 wait complete. 2024-04-21T08:53:30.365 INFO:tasks.cephfs.filesystem:scrub status for tag:2f44a19d-b0a7-4681-b27b-fb7529461620 - {'path': '/', 'tag': '2f44a19d-b0a7-4681-b27b-fb7529461620', 'options': 'recursive,force'} 2024-04-21T08:53:30.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:30 smithi007 ceph-mon[28477]: pgmap v275: 129 pgs: 20 active+clean+snaptrim, 58 active+clean, 51 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 603 KiB/s wr, 715 op/s 2024-04-21T08:53:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:30 smithi077 ceph-mon[31615]: pgmap v275: 129 pgs: 20 active+clean+snaptrim, 58 active+clean, 51 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 603 KiB/s wr, 715 op/s 2024-04-21T08:53:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:30 smithi016 ceph-mon[31259]: pgmap v275: 129 pgs: 20 active+clean+snaptrim, 58 active+clean, 51 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 603 KiB/s wr, 715 op/s 2024-04-21T08:53:32.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:32 smithi077 ceph-mon[31615]: pgmap v276: 129 pgs: 18 active+clean+snaptrim, 71 active+clean, 40 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 452 KiB/s wr, 702 op/s 2024-04-21T08:53:32.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:32 smithi007 ceph-mon[28477]: pgmap v276: 129 pgs: 18 active+clean+snaptrim, 71 active+clean, 40 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 452 KiB/s wr, 702 op/s 2024-04-21T08:53:33.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:32 smithi016 ceph-mon[31259]: pgmap v276: 129 pgs: 18 active+clean+snaptrim, 71 active+clean, 40 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 452 KiB/s wr, 702 op/s 2024-04-21T08:53:34.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:34 smithi077 ceph-mon[31615]: pgmap v277: 129 pgs: 13 active+clean+snaptrim, 82 active+clean, 34 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 677 KiB/s wr, 774 op/s 2024-04-21T08:53:34.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:34 smithi007 ceph-mon[28477]: pgmap v277: 129 pgs: 13 active+clean+snaptrim, 82 active+clean, 34 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 677 KiB/s wr, 774 op/s 2024-04-21T08:53:35.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:34 smithi016 ceph-mon[31259]: pgmap v277: 129 pgs: 13 active+clean+snaptrim, 82 active+clean, 34 active+clean+snaptrim_wait; 1.4 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 677 KiB/s wr, 774 op/s 2024-04-21T08:53:35.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.837+0000 7f786e152640 1 -- 172.21.15.7:0/1620119505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7868100fb0 msgr2=0x7f7868101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:35.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.837+0000 7f786e152640 1 --2- 172.21.15.7:0/1620119505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7868100fb0 0x7f7868101390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f785801a940 tx=0x7f785803cd20 comp rx=0 tx=0).stop 2024-04-21T08:53:35.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.837+0000 7f786e152640 1 -- 172.21.15.7:0/1620119505 shutdown_connections 2024-04-21T08:53:35.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.837+0000 7f786e152640 1 --2- 172.21.15.7:0/1620119505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78681063b0 0x7f7868106860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.837+0000 7f786e152640 1 --2- 172.21.15.7:0/1620119505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78681018d0 0x7f7868105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.837+0000 7f786e152640 1 --2- 172.21.15.7:0/1620119505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7868100fb0 0x7f7868101390 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.837+0000 7f786e152640 1 -- 172.21.15.7:0/1620119505 >> 172.21.15.7:0/1620119505 conn(0x7f7868077f70 msgr2=0x7f7868078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.839+0000 7f786e152640 1 -- 172.21.15.7:0/1620119505 shutdown_connections 2024-04-21T08:53:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.842+0000 7f786e152640 1 -- 172.21.15.7:0/1620119505 wait complete. 2024-04-21T08:53:35.846 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.843+0000 7f786e152640 1 Processor -- start 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.847+0000 7f786e152640 1 -- start start 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.847+0000 7f786e152640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7868100fb0 0x7f7868072af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.847+0000 7f786e152640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78681018d0 0x7f7868073030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.847+0000 7f786e152640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78681063b0 0x7f786810d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.847+0000 7f786e152640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7868109cb0 con 0x7f78681018d0 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.847+0000 7f786e152640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7868109b30 con 0x7f78681063b0 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.847+0000 7f786e152640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7868109e30 con 0x7f7868100fb0 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.849+0000 7f7866ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78681018d0 0x7f7868073030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.849+0000 7f7866ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78681018d0 0x7f7868073030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34710/0 (socket says 172.21.15.7:34710) 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.849+0000 7f7866ffd640 1 -- 172.21.15.7:0/2177397074 learned_addr learned my addr 172.21.15.7:0/2177397074 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.850+0000 7f7866ffd640 1 -- 172.21.15.7:0/2177397074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7868100fb0 msgr2=0x7f7868072af0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.850+0000 7f7866ffd640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7868100fb0 0x7f7868072af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.850+0000 7f7866ffd640 1 -- 172.21.15.7:0/2177397074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78681063b0 msgr2=0x7f786810d690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:53:35.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.850+0000 7f7866ffd640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78681063b0 0x7f786810d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.850+0000 7f7866ffd640 1 -- 172.21.15.7:0/2177397074 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786810dbd0 con 0x7f78681018d0 2024-04-21T08:53:35.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.850+0000 7f7866ffd640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78681018d0 0x7f7868073030 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f785c00b450 tx=0x7f785c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:35.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.851+0000 7f7864ff9640 1 -- 172.21.15.7:0/2177397074 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c015020 con 0x7f78681018d0 2024-04-21T08:53:35.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.851+0000 7f786e152640 1 -- 172.21.15.7:0/2177397074 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786810dec0 con 0x7f78681018d0 2024-04-21T08:53:35.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.851+0000 7f786e152640 1 -- 172.21.15.7:0/2177397074 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786810e3e0 con 0x7f78681018d0 2024-04-21T08:53:35.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.851+0000 7f7864ff9640 1 -- 172.21.15.7:0/2177397074 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f785c011d40 con 0x7f78681018d0 2024-04-21T08:53:35.854 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.851+0000 7f7864ff9640 1 -- 172.21.15.7:0/2177397074 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c00f340 con 0x7f78681018d0 2024-04-21T08:53:35.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.856+0000 7f7864ff9640 1 -- 172.21.15.7:0/2177397074 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f785c018020 con 0x7f78681018d0 2024-04-21T08:53:35.858 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.858+0000 7f7870e18740 1 Processor -- start 2024-04-21T08:53:35.859 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.858+0000 7f7870e18740 1 -- start start 2024-04-21T08:53:35.859 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.858+0000 7f7864ff9640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78500a3e60 0x7f78500a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.859 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.858+0000 7f7870e18740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce666858d0 0x55ce66684620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.859 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.858+0000 7f7870e18740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce66684b60 0x55ce66684fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.859 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.858+0000 7f7870e18740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6668e0e0 0x55ce666904d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.859 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.858+0000 7f7870e18740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ce665e98d0 con 0x55ce666858d0 2024-04-21T08:53:35.859 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.858+0000 7f7870e18740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ce66586e80 con 0x55ce66684b60 2024-04-21T08:53:35.859 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.858+0000 7f7870e18740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ce665e5890 con 0x55ce6668e0e0 2024-04-21T08:53:35.860 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.858+0000 7f7864ff9640 1 -- 172.21.15.7:0/2177397074 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9716+0+0 (secure 0 0 0) 0x7f785c0d0410 con 0x7f78681018d0 2024-04-21T08:53:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.859+0000 7f7867fff640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78500a3e60 0x7f78500a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.859+0000 7f7866ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce66684b60 0x55ce66684fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.859+0000 7f7867fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6668e0e0 0x55ce666904d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.859+0000 7f7867fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6668e0e0 0x55ce666904d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:50118/0 (socket says 172.21.15.7:50118) 2024-04-21T08:53:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.859+0000 7f7867fff640 1 -- 172.21.15.7:0/1930797841 learned_addr learned my addr 172.21.15.7:0/1930797841 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.859+0000 7f78677fe640 1 --2- 172.21.15.7:0/1930797841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce666858d0 0x55ce66684620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.859+0000 7f7866ffd640 1 -- 172.21.15.7:0/1930797841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6668e0e0 msgr2=0x55ce666904d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.859+0000 7f7866ffd640 1 --2- 172.21.15.7:0/1930797841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6668e0e0 0x55ce666904d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.859+0000 7f7866ffd640 1 -- 172.21.15.7:0/1930797841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce666858d0 msgr2=0x55ce66684620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.859+0000 7f7866ffd640 1 --2- 172.21.15.7:0/1930797841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce666858d0 0x55ce66684620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.859+0000 7f7866ffd640 1 -- 172.21.15.7:0/1930797841 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce665a8320 con 0x55ce66684b60 2024-04-21T08:53:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.860+0000 7f7866ffd640 1 --2- 172.21.15.7:0/1930797841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce66684b60 0x55ce66684fe0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f785c0cd7c0 tx=0x7f785c09c010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1404a6782a87dcdf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.860+0000 7f784dffb640 1 -- 172.21.15.7:0/1930797841 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c09a310 con 0x55ce66684b60 2024-04-21T08:53:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.860+0000 7f784dffb640 1 -- 172.21.15.7:0/1930797841 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f785c0d0180 con 0x55ce66684b60 2024-04-21T08:53:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.861+0000 7f784dffb640 1 -- 172.21.15.7:0/1930797841 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c096b80 con 0x55ce66684b60 2024-04-21T08:53:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.862+0000 7f7870e18740 1 -- 172.21.15.7:0/1930797841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce66684b60 msgr2=0x55ce66684fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:35.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.862+0000 7f7870e18740 1 --2- 172.21.15.7:0/1930797841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce66684b60 0x55ce66684fe0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f785c0cd7c0 tx=0x7f785c09c010 comp rx=0 tx=0).stop 2024-04-21T08:53:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.863+0000 7f7870e18740 1 -- 172.21.15.7:0/1930797841 shutdown_connections 2024-04-21T08:53:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.863+0000 7f7870e18740 1 --2- 172.21.15.7:0/1930797841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6668e0e0 0x55ce666904d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.863+0000 7f7870e18740 1 --2- 172.21.15.7:0/1930797841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce66684b60 0x55ce66684fe0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.863+0000 7f7870e18740 1 --2- 172.21.15.7:0/1930797841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce666858d0 0x55ce66684620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.863+0000 7f7870e18740 1 -- 172.21.15.7:0/1930797841 >> 172.21.15.7:0/1930797841 conn(0x55ce6664bc90 msgr2=0x55ce6667f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.863+0000 7f7870e18740 1 -- 172.21.15.7:0/1930797841 shutdown_connections 2024-04-21T08:53:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.863+0000 7f7867fff640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78500a3e60 0x7f78500a6320 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7860011550 tx=0x7f786000ca30 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.863+0000 7f7870e18740 1 -- 172.21.15.7:0/1930797841 wait complete. 2024-04-21T08:53:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.864+0000 7f7870e18740 1 -- start start 2024-04-21T08:53:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.864+0000 7f7870e18740 1 Processor -- start 2024-04-21T08:53:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.864+0000 7f7870e18740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce66684b60 0x55ce6668d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.864+0000 7f7870e18740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce666858d0 0x55ce6668acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.864+0000 7f7870e18740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 0x55ce6668b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.864+0000 7f7870e18740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ce66690e00 con 0x55ce66684b60 2024-04-21T08:53:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.864+0000 7f7870e18740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ce66571eb0 con 0x55ce6668e0e0 2024-04-21T08:53:35.865 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.864+0000 7f7870e18740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ce66690f80 con 0x55ce666858d0 2024-04-21T08:53:35.866 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.864+0000 7f7870e18740 10 client.0 _ll_register_callbacks cb 0x55ce6659eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.865+0000 7f7866ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce666858d0 0x55ce6668acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.865+0000 7f7866ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce666858d0 0x55ce6668acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:50120/0 (socket says 172.21.15.7:50120) 2024-04-21T08:53:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.865+0000 7f7866ffd640 1 -- 172.21.15.7:0/972900894 learned_addr learned my addr 172.21.15.7:0/972900894 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.865+0000 7f78677fe640 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce66684b60 0x55ce6668d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.867 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.865+0000 7f7867fff640 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 0x55ce6668b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.865+0000 7f7866ffd640 1 -- 172.21.15.7:0/972900894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 msgr2=0x55ce6668b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.865+0000 7f7866ffd640 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 0x55ce6668b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.865+0000 7f7866ffd640 1 -- 172.21.15.7:0/972900894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce66684b60 msgr2=0x55ce6668d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.865+0000 7f7866ffd640 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce66684b60 0x55ce6668d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.865+0000 7f7866ffd640 1 -- 172.21.15.7:0/972900894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce662fc210 con 0x55ce666858d0 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.865+0000 7f7867fff640 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 0x55ce6668b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.866+0000 7f7866ffd640 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce666858d0 0x55ce6668acc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f785c09bfa0 tx=0x7f785c09a520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f299c1f5ceb6be18 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.866+0000 7f7846ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.866+0000 7f7846ffd640 1 -- 172.21.15.7:0/972900894 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c098680 con 0x55ce666858d0 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.866+0000 7f7846ffd640 1 -- 172.21.15.7:0/972900894 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9716+0+0 (secure 0 0 0) 0x7f785c0bc040 con 0x55ce666858d0 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.866+0000 7f7870e18740 1 -- 172.21.15.7:0/972900894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f785c0d0180 con 0x55ce666858d0 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.866+0000 7f7846ffd640 1 -- 172.21.15.7:0/972900894 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f785c011890 con 0x55ce666858d0 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.867+0000 7f7870e18740 10 client.26662 fetch_fsmap learned FSMap version 23 2024-04-21T08:53:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.867+0000 7f7870e18740 1 -- 172.21.15.7:0/972900894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ce663c8620 con 0x55ce666858d0 2024-04-21T08:53:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.867+0000 7f7846ffd640 1 -- 172.21.15.7:0/972900894 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f785c0d0180 con 0x55ce666858d0 2024-04-21T08:53:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.867+0000 7f7846ffd640 1 -- 172.21.15.7:0/972900894 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f785c097d60 con 0x55ce666858d0 2024-04-21T08:53:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.867+0000 7f7870e18740 10 client.26662 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:53:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.867+0000 7f7870e18740 10 client.26662 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.867+0000 7f7870e18740 20 client.26662 populate_metadata read hostname 'smithi007' 2024-04-21T08:53:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.867+0000 7f7870e18740 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ce6669dec0 0x55ce666a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.867+0000 7f7870e18740 4 client.26662 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.867+0000 7f7870e18740 1 -- 172.21.15.7:0/972900894 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce665fd880 con 0x55ce6669dec0 2024-04-21T08:53:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.868+0000 7f78677fe640 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ce6669dec0 0x55ce666a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.869+0000 7f78677fe640 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ce6669dec0 0x55ce666a02b0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc3fd2dd535dbef0 server_cookie=370ae203face1cea in_seq=0 out_seq=0 2024-04-21T08:53:35.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.869+0000 7f7846ffd640 10 client.26662 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.871+0000 7f7846ffd640 1 -- 172.21.15.7:0/972900894 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce665fd880 con 0x55ce6669dec0 2024-04-21T08:53:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.871+0000 7f7846ffd640 10 client.26662 handle_command_reply: tid=0 2024-04-21T08:53:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.871+0000 7f7870e18740 1 client.26662 shutdown 2024-04-21T08:53:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.871+0000 7f7870e18740 1 -- 172.21.15.7:0/972900894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce666858d0 msgr2=0x55ce6668acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.871+0000 7f7870e18740 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce666858d0 0x55ce6668acc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f785c09bfa0 tx=0x7f785c09a520 comp rx=0 tx=0).stop 2024-04-21T08:53:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.872+0000 7f7870e18740 1 -- 172.21.15.7:0/972900894 shutdown_connections 2024-04-21T08:53:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.872+0000 7f7870e18740 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ce6669dec0 0x55ce666a02b0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.872+0000 7f7870e18740 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 0x55ce6668b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.872+0000 7f7870e18740 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce666858d0 0x55ce6668acc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.874 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.872+0000 7f7870e18740 1 --2- 172.21.15.7:0/972900894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce66684b60 0x55ce6668d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.872+0000 7f7870e18740 1 -- 172.21.15.7:0/972900894 >> 172.21.15.7:0/972900894 conn(0x55ce6664bc90 msgr2=0x55ce666815a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.872+0000 7f7846ffd640 0 client.26662 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.872+0000 7f7870e18740 1 -- 172.21.15.7:0/972900894 shutdown_connections 2024-04-21T08:53:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.872+0000 7f7870e18740 1 -- 172.21.15.7:0/972900894 wait complete. 2024-04-21T08:53:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.872+0000 7f7870e18740 20 client.26662 trim_cache size 0 max 16384 2024-04-21T08:53:35.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.896+0000 7f7870e18740 1 Processor -- start 2024-04-21T08:53:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.897+0000 7f7870e18740 1 -- start start 2024-04-21T08:53:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.897+0000 7f7870e18740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce66684b60 0x55ce66699db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.897+0000 7f7870e18740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce666858d0 0x55ce6668d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.897+0000 7f7870e18740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6668e0e0 0x55ce6668d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.898+0000 7f78677fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce66684b60 0x55ce66699db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.898+0000 7f78677fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce66684b60 0x55ce66699db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34728/0 (socket says 172.21.15.7:34728) 2024-04-21T08:53:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.898+0000 7f7866ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce666858d0 0x55ce6668d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.898+0000 7f7866ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce666858d0 0x55ce6668d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:34304/0 (socket says 172.21.15.7:34304) 2024-04-21T08:53:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.898+0000 7f78677fe640 1 -- 172.21.15.7:0/3643590976 learned_addr learned my addr 172.21.15.7:0/3643590976 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.898+0000 7f7870e18740 1 -- 172.21.15.7:0/3643590976 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f785400e230 con 0x55ce66684b60 2024-04-21T08:53:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.898+0000 7f7870e18740 1 -- 172.21.15.7:0/3643590976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7854010a90 con 0x55ce666858d0 2024-04-21T08:53:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.898+0000 7f7870e18740 1 -- 172.21.15.7:0/3643590976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f785400e880 con 0x55ce6668e0e0 2024-04-21T08:53:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.898+0000 7f7867fff640 1 --2- 172.21.15.7:0/3643590976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6668e0e0 0x55ce6668d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.899+0000 7f7866ffd640 1 -- 172.21.15.7:0/3643590976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6668e0e0 msgr2=0x55ce6668d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.899+0000 7f7866ffd640 1 --2- 172.21.15.7:0/3643590976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6668e0e0 0x55ce6668d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.899+0000 7f7866ffd640 1 -- 172.21.15.7:0/3643590976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce66684b60 msgr2=0x55ce66699db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.899+0000 7f7866ffd640 1 --2- 172.21.15.7:0/3643590976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce66684b60 0x55ce66699db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.899+0000 7f7866ffd640 1 -- 172.21.15.7:0/3643590976 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce66683e30 con 0x55ce666858d0 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.899+0000 7f7866ffd640 1 --2- 172.21.15.7:0/3643590976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce666858d0 0x55ce6668d3f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f785c0cdcb0 tx=0x7f785c0cd870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=215919d43bf998c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.899+0000 7f784d7fa640 1 -- 172.21.15.7:0/3643590976 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c0a9080 con 0x55ce666858d0 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.899+0000 7f784d7fa640 1 -- 172.21.15.7:0/3643590976 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f785c0d0180 con 0x55ce666858d0 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.899+0000 7f784d7fa640 1 -- 172.21.15.7:0/3643590976 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c098680 con 0x55ce666858d0 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.900+0000 7f7870e18740 1 -- 172.21.15.7:0/3643590976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce666858d0 msgr2=0x55ce6668d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.900+0000 7f7870e18740 1 --2- 172.21.15.7:0/3643590976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce666858d0 0x55ce6668d3f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f785c0cdcb0 tx=0x7f785c0cd870 comp rx=0 tx=0).stop 2024-04-21T08:53:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.900+0000 7f7870e18740 1 -- 172.21.15.7:0/3643590976 shutdown_connections 2024-04-21T08:53:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.900+0000 7f7870e18740 1 --2- 172.21.15.7:0/3643590976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce6668e0e0 0x55ce6668d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.900+0000 7f7870e18740 1 --2- 172.21.15.7:0/3643590976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce666858d0 0x55ce6668d3f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.900+0000 7f7870e18740 1 --2- 172.21.15.7:0/3643590976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce66684b60 0x55ce66699db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.900+0000 7f7870e18740 1 -- 172.21.15.7:0/3643590976 >> 172.21.15.7:0/3643590976 conn(0x55ce6664bc90 msgr2=0x55ce6667e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.901+0000 7f7870e18740 1 -- 172.21.15.7:0/3643590976 shutdown_connections 2024-04-21T08:53:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.901+0000 7f7870e18740 1 -- 172.21.15.7:0/3643590976 wait complete. 2024-04-21T08:53:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.901+0000 7f7870e18740 1 -- start start 2024-04-21T08:53:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.901+0000 7f7870e18740 1 Processor -- start 2024-04-21T08:53:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.902+0000 7f7870e18740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce66684b60 0x55ce666967e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.902+0000 7f7870e18740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce666858d0 0x55ce666877b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.902+0000 7f78677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce66684b60 0x55ce666967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.902+0000 7f78677fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce66684b60 0x55ce666967e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:50130/0 (socket says 172.21.15.7:50130) 2024-04-21T08:53:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f78677fe640 1 -- 172.21.15.7:0/2219028677 learned_addr learned my addr 172.21.15.7:0/2219028677 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7866ffd640 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce666858d0 0x55ce666877b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7870e18740 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 0x55ce66687ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7870e18740 1 -- 172.21.15.7:0/2219028677 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f785400ef50 con 0x55ce666858d0 2024-04-21T08:53:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7870e18740 1 -- 172.21.15.7:0/2219028677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78540103c0 con 0x55ce6668e0e0 2024-04-21T08:53:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7870e18740 1 -- 172.21.15.7:0/2219028677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f785400f620 con 0x55ce66684b60 2024-04-21T08:53:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7867fff640 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 0x55ce66687ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7870e18740 10 client.0 _ll_register_callbacks cb 0x7f785c010f80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7867fff640 1 -- 172.21.15.7:0/2219028677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce66684b60 msgr2=0x55ce666967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7867fff640 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce66684b60 0x55ce666967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7867fff640 1 -- 172.21.15.7:0/2219028677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce666858d0 msgr2=0x55ce666877b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7867fff640 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce666858d0 0x55ce666877b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f7867fff640 1 -- 172.21.15.7:0/2219028677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce667030c0 con 0x55ce6668e0e0 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.903+0000 7f78677fe640 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce66684b60 0x55ce666967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.904+0000 7f7866ffd640 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce666858d0 0x55ce666877b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.904+0000 7f7867fff640 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 0x55ce66687ef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f786000c870 tx=0x7f7860016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a6c7d49477a4782 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.904+0000 7f7846ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.904+0000 7f7846ffd640 1 -- 172.21.15.7:0/2219028677 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786000e740 con 0x55ce6668e0e0 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.904+0000 7f7870e18740 1 -- 172.21.15.7:0/2219028677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f785c0d0180 con 0x55ce6668e0e0 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.904+0000 7f7846ffd640 1 -- 172.21.15.7:0/2219028677 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9716+0+0 (secure 0 0 0) 0x7f7860017040 con 0x55ce6668e0e0 2024-04-21T08:53:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.905+0000 7f7846ffd640 1 -- 172.21.15.7:0/2219028677 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f786000e8e0 con 0x55ce6668e0e0 2024-04-21T08:53:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.905+0000 7f7870e18740 10 client.26690 fetch_fsmap learned FSMap version 23 2024-04-21T08:53:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.905+0000 7f7870e18740 1 -- 172.21.15.7:0/2219028677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ce66704190 con 0x55ce6668e0e0 2024-04-21T08:53:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.908+0000 7f7846ffd640 1 -- 172.21.15.7:0/2219028677 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f785c0d0180 con 0x55ce6668e0e0 2024-04-21T08:53:35.909 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.908+0000 7f7846ffd640 1 -- 172.21.15.7:0/2219028677 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f7860006660 con 0x55ce6668e0e0 2024-04-21T08:53:35.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.908+0000 7f7870e18740 10 client.26690 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:53:35.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.908+0000 7f7870e18740 10 client.26690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:35.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.908+0000 7f7870e18740 20 client.26690 populate_metadata read hostname 'smithi007' 2024-04-21T08:53:35.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.909+0000 7f7870e18740 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ce66704940 0x55ce66706d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:35.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.909+0000 7f7870e18740 4 client.26690 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:53:35.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.909+0000 7f7870e18740 1 -- 172.21.15.7:0/2219028677 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ce665893d0 con 0x55ce66704940 2024-04-21T08:53:35.910 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.910+0000 7f78677fe640 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ce66704940 0x55ce66706d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:35.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.911+0000 7f78677fe640 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ce66704940 0x55ce66706d60 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12eb3e3f58dad1ea server_cookie=533f17a15955d52c in_seq=0 out_seq=0 2024-04-21T08:53:35.911 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:35.911+0000 7f7846ffd640 10 client.26690 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.193+0000 7f7846ffd640 1 -- 172.21.15.7:0/2219028677 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ce665893d0 con 0x55ce66704940 2024-04-21T08:53:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.193+0000 7f7846ffd640 10 client.26690 handle_command_reply: tid=0 2024-04-21T08:53:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.193+0000 7f7870e18740 1 client.26690 shutdown 2024-04-21T08:53:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.193+0000 7f7870e18740 1 -- 172.21.15.7:0/2219028677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 msgr2=0x55ce66687ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.193+0000 7f7870e18740 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 0x55ce66687ef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f786000c870 tx=0x7f7860016000 comp rx=0 tx=0).stop 2024-04-21T08:53:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.193+0000 7f7870e18740 1 -- 172.21.15.7:0/2219028677 shutdown_connections 2024-04-21T08:53:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.193+0000 7f7870e18740 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ce66704940 0x55ce66706d60 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.194+0000 7f7870e18740 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ce6668e0e0 0x55ce66687ef0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.194+0000 7f7846ffd640 0 client.26690 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.194+0000 7f7870e18740 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ce666858d0 0x55ce666877b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.194+0000 7f7870e18740 1 --2- 172.21.15.7:0/2219028677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce66684b60 0x55ce666967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.194+0000 7f7870e18740 1 -- 172.21.15.7:0/2219028677 >> 172.21.15.7:0/2219028677 conn(0x55ce6664bc90 msgr2=0x55ce6668ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.194+0000 7f7870e18740 1 -- 172.21.15.7:0/2219028677 shutdown_connections 2024-04-21T08:53:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.194+0000 7f7870e18740 1 -- 172.21.15.7:0/2219028677 wait complete. 2024-04-21T08:53:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.194+0000 7f7870e18740 20 client.26690 trim_cache size 0 max 16384 2024-04-21T08:53:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:53:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:53:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:53:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:53:36.198 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.195+0000 7f786e152640 1 -- 172.21.15.7:0/2177397074 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78500a3e60 msgr2=0x7f78500a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:36.198 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.195+0000 7f786e152640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78500a3e60 0x7f78500a6320 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7860011550 tx=0x7f786000ca30 comp rx=0 tx=0).stop 2024-04-21T08:53:36.198 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.195+0000 7f786e152640 1 -- 172.21.15.7:0/2177397074 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78681018d0 msgr2=0x7f7868073030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:36.198 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.196+0000 7f786e152640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78681018d0 0x7f7868073030 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f785c00b450 tx=0x7f785c00b910 comp rx=0 tx=0).stop 2024-04-21T08:53:36.198 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.196+0000 7f786e152640 1 -- 172.21.15.7:0/2177397074 shutdown_connections 2024-04-21T08:53:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.196+0000 7f786e152640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78681063b0 0x7f786810d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.196+0000 7f786e152640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78681018d0 0x7f7868073030 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.196+0000 7f786e152640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78500a3e60 0x7f78500a6320 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.196+0000 7f786e152640 1 --2- 172.21.15.7:0/2177397074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7868100fb0 0x7f7868072af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.196+0000 7f786e152640 1 -- 172.21.15.7:0/2177397074 >> 172.21.15.7:0/2177397074 conn(0x7f7868077f70 msgr2=0x7f7868103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.196+0000 7f786e152640 1 -- 172.21.15.7:0/2177397074 shutdown_connections 2024-04-21T08:53:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:36.196+0000 7f786e152640 1 -- 172.21.15.7:0/2177397074 wait complete. 2024-04-21T08:53:36.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:36 smithi077 ceph-mon[31615]: pgmap v278: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 556 KiB/s wr, 718 op/s 2024-04-21T08:53:36.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:36 smithi007 ceph-mon[28477]: pgmap v278: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 556 KiB/s wr, 718 op/s 2024-04-21T08:53:37.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:36 smithi016 ceph-mon[31259]: pgmap v278: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 1.4 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 556 KiB/s wr, 718 op/s 2024-04-21T08:53:38.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:38 smithi077 ceph-mon[31615]: pgmap v279: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 1.4 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 617 KiB/s wr, 976 op/s 2024-04-21T08:53:38.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:38 smithi007 ceph-mon[28477]: pgmap v279: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 1.4 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 617 KiB/s wr, 976 op/s 2024-04-21T08:53:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:38 smithi016 ceph-mon[31259]: pgmap v279: 129 pgs: 9 active+clean+snaptrim, 100 active+clean, 20 active+clean+snaptrim_wait; 1.4 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 617 KiB/s wr, 976 op/s 2024-04-21T08:53:40.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:40 smithi077 ceph-mon[31615]: pgmap v280: 129 pgs: 9 active+clean+snaptrim, 102 active+clean, 18 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 953 KiB/s wr, 1.03k op/s 2024-04-21T08:53:40.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:40 smithi007 ceph-mon[28477]: pgmap v280: 129 pgs: 9 active+clean+snaptrim, 102 active+clean, 18 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 953 KiB/s wr, 1.03k op/s 2024-04-21T08:53:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:40 smithi016 ceph-mon[31259]: pgmap v280: 129 pgs: 9 active+clean+snaptrim, 102 active+clean, 18 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 953 KiB/s wr, 1.03k op/s 2024-04-21T08:53:42.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:42 smithi077 ceph-mon[31615]: pgmap v281: 129 pgs: 9 active+clean+snaptrim, 102 active+clean, 18 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 990 KiB/s wr, 1.14k op/s 2024-04-21T08:53:42.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:42 smithi007 ceph-mon[28477]: pgmap v281: 129 pgs: 9 active+clean+snaptrim, 102 active+clean, 18 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 990 KiB/s wr, 1.14k op/s 2024-04-21T08:53:43.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:42 smithi016 ceph-mon[31259]: pgmap v281: 129 pgs: 9 active+clean+snaptrim, 102 active+clean, 18 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 990 KiB/s wr, 1.14k op/s 2024-04-21T08:53:43.987 INFO:tasks.workunit.client.0.smithi007.stdout: CC scripts/mod/empty.o 2024-04-21T08:53:43.987 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/mod/mk_elfconfig 2024-04-21T08:53:43.992 INFO:tasks.workunit.client.0.smithi007.stdout: CC scripts/mod/devicetable-offsets.s 2024-04-21T08:53:44.074 INFO:tasks.workunit.client.0.smithi007.stdout: MKELF scripts/mod/elfconfig.h 2024-04-21T08:53:44.082 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/mod/modpost.o 2024-04-21T08:53:44.084 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/mod/sumversion.o 2024-04-21T08:53:44.113 INFO:tasks.workunit.client.0.smithi007.stdout: UPD scripts/mod/devicetable-offsets.h 2024-04-21T08:53:44.120 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC scripts/mod/file2alias.o 2024-04-21T08:53:44.831 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTLD scripts/mod/modpost 2024-04-21T08:53:44.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:44 smithi077 ceph-mon[31615]: pgmap v282: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 994 KiB/s wr, 1.35k op/s 2024-04-21T08:53:44.887 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/bounds.s 2024-04-21T08:53:44.889 INFO:tasks.workunit.client.0.smithi007.stdout: CHKSHA1 include/linux/atomic/atomic-arch-fallback.h 2024-04-21T08:53:44.891 INFO:tasks.workunit.client.0.smithi007.stdout: CHKSHA1 include/linux/atomic/atomic-instrumented.h 2024-04-21T08:53:44.893 INFO:tasks.workunit.client.0.smithi007.stdout: CHKSHA1 include/linux/atomic/atomic-long.h 2024-04-21T08:53:44.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:44 smithi007 ceph-mon[28477]: pgmap v282: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 994 KiB/s wr, 1.35k op/s 2024-04-21T08:53:44.908 INFO:tasks.workunit.client.0.smithi007.stdout: UPD include/generated/timeconst.h 2024-04-21T08:53:44.973 INFO:tasks.workunit.client.0.smithi007.stdout: UPD include/generated/bounds.h 2024-04-21T08:53:44.982 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/asm-offsets.s 2024-04-21T08:53:45.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:44 smithi016 ceph-mon[31259]: pgmap v282: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 994 KiB/s wr, 1.35k op/s 2024-04-21T08:53:45.574 INFO:tasks.workunit.client.0.smithi007.stdout: UPD include/generated/asm-offsets.h 2024-04-21T08:53:45.579 INFO:tasks.workunit.client.0.smithi007.stdout: CALL scripts/checksyscalls.sh 2024-04-21T08:53:45.627 INFO:tasks.workunit.client.0.smithi007.stdout: LDS scripts/module.lds 2024-04-21T08:53:45.738 INFO:tasks.workunit.client.0.smithi007.stdout: CC init/main.o 2024-04-21T08:53:45.740 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC usr/gen_init_cpio 2024-04-21T08:53:45.741 INFO:tasks.workunit.client.0.smithi007.stdout: CC init/do_mounts.o 2024-04-21T08:53:45.746 INFO:tasks.workunit.client.0.smithi007.stdout: UPD init/utsversion-tmp.h 2024-04-21T08:53:45.758 INFO:tasks.workunit.client.0.smithi007.stdout: CC certs/system_keyring.o 2024-04-21T08:53:45.762 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/filemap.o 2024-04-21T08:53:45.876 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/vsyscall/vsyscall_64.o 2024-04-21T08:53:45.936 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/vdso/vma.o 2024-04-21T08:53:46.040 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/sched/core.o 2024-04-21T08:53:46.240 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC certs/extract-cert 2024-04-21T08:53:46.293 INFO:tasks.workunit.client.0.smithi007.stdout: GEN usr/initramfs_data.cpio 2024-04-21T08:53:46.398 INFO:tasks.workunit.client.0.smithi007.stdout: COPY usr/initramfs_inc_data 2024-04-21T08:53:46.426 INFO:tasks.workunit.client.0.smithi007.stdout: AS usr/initramfs_data.o 2024-04-21T08:53:46.612 INFO:tasks.workunit.client.0.smithi007.stdout: AR usr/built-in.a 2024-04-21T08:53:46.737 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/entry/entry.o 2024-04-21T08:53:46.763 INFO:tasks.workunit.client.0.smithi007.stdout: CERT certs/x509_certificate_list 2024-04-21T08:53:46.780 INFO:tasks.workunit.client.0.smithi007.stdout: CERT certs/signing_key.x509 2024-04-21T08:53:46.800 INFO:tasks.workunit.client.0.smithi007.stdout: AS certs/system_certificates.o 2024-04-21T08:53:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:46 smithi077 ceph-mon[31615]: pgmap v283: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 769 KiB/s wr, 1.25k op/s 2024-04-21T08:53:46.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:46 smithi077 ceph-mon[31615]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T08:53:46.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:46 smithi007 ceph-mon[28477]: pgmap v283: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 769 KiB/s wr, 1.25k op/s 2024-04-21T08:53:46.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:46 smithi007 ceph-mon[28477]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T08:53:46.903 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/entry/vsyscall/vsyscall_emu_64.o 2024-04-21T08:53:46.925 INFO:tasks.workunit.client.0.smithi007.stdout: AR certs/built-in.a 2024-04-21T08:53:46.965 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/entry/entry_64.o 2024-04-21T08:53:46.982 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/entry/vsyscall/built-in.a 2024-04-21T08:53:47.034 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/notify/dnotify/dnotify.o 2024-04-21T08:53:47.050 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/syscall_64.o 2024-04-21T08:53:47.052 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs_common/nfsacl.o 2024-04-21T08:53:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:46 smithi016 ceph-mon[31259]: pgmap v283: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.4 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 769 KiB/s wr, 1.25k op/s 2024-04-21T08:53:47.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:46 smithi016 ceph-mon[31259]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T08:53:47.092 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/vdso/extable.o 2024-04-21T08:53:47.094 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs_common/grace.o 2024-04-21T08:53:47.712 INFO:tasks.workunit.client.0.smithi007.stdout: CC init/do_mounts_initrd.o 2024-04-21T08:53:47.855 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/notify/dnotify/built-in.a 2024-04-21T08:53:47.926 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/notify/inotify/inotify_fsnotify.o 2024-04-21T08:53:47.986 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/vdso/vdso32-setup.o 2024-04-21T08:53:48.006 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/notify/inotify/inotify_user.o 2024-04-21T08:53:48.169 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/nfs_common/built-in.a 2024-04-21T08:53:48.213 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/notify/fanotify/built-in.a 2024-04-21T08:53:48.253 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/iomap/trace.o 2024-04-21T08:53:48.273 INFO:tasks.workunit.client.0.smithi007.stdout: LDS arch/x86/entry/vdso/vdso.lds 2024-04-21T08:53:48.364 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/iomap/iter.o 2024-04-21T08:53:48.404 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/entry/vdso/vdso-note.o 2024-04-21T08:53:48.468 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/vdso/vclock_gettime.o 2024-04-21T08:53:48.535 INFO:tasks.workunit.client.0.smithi007.stdout: CC init/initramfs.o 2024-04-21T08:53:48.562 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/iomap/buffered-io.o 2024-04-21T08:53:48.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:48 smithi007 ceph-mon[28477]: pgmap v285: 129 pgs: 5 active+clean+snaptrim, 117 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 1.63k op/s 2024-04-21T08:53:48.984 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/notify/inotify/built-in.a 2024-04-21T08:53:49.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:48 smithi016 ceph-mon[31259]: pgmap v285: 129 pgs: 5 active+clean+snaptrim, 117 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 1.63k op/s 2024-04-21T08:53:49.060 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/notify/fsnotify.o 2024-04-21T08:53:49.067 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/vdso/vgetcpu.o 2024-04-21T08:53:49.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:48 smithi077 ceph-mon[31615]: pgmap v285: 129 pgs: 5 active+clean+snaptrim, 117 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 1.63k op/s 2024-04-21T08:53:49.171 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mempool.o 2024-04-21T08:53:49.215 INFO:tasks.workunit.client.0.smithi007.stdout: HOSTCC arch/x86/entry/vdso/vdso2c 2024-04-21T08:53:49.273 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/notify/notification.o 2024-04-21T08:53:49.551 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.551+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/2405570502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c381106f0 msgr2=0x7f0c3810e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:49.551 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.551+0000 7f0c3c9f4640 1 --2- 172.21.15.7:0/2405570502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c381106f0 0x7f0c3810e290 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009860 tx=0x7f0c2c02bb90 comp rx=0 tx=0).stop 2024-04-21T08:53:49.551 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.551+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/2405570502 shutdown_connections 2024-04-21T08:53:49.551 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.551+0000 7f0c3c9f4640 1 --2- 172.21.15.7:0/2405570502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c3811b730 0x7f0c3811db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.551 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.551+0000 7f0c3c9f4640 1 --2- 172.21.15.7:0/2405570502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c3810e7d0 0x7f0c3810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.551 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.551+0000 7f0c3c9f4640 1 --2- 172.21.15.7:0/2405570502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c381106f0 0x7f0c3810e290 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.551 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.551+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/2405570502 >> 172.21.15.7:0/2405570502 conn(0x7f0c3806d030 msgr2=0x7f0c3806d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:49.558 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/sched/fair.o 2024-04-21T08:53:49.562 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.563+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/2405570502 shutdown_connections 2024-04-21T08:53:49.572 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.572+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/2405570502 wait complete. 2024-04-21T08:53:49.572 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.573+0000 7f0c3c9f4640 1 Processor -- start 2024-04-21T08:53:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.573+0000 7f0c3c9f4640 1 -- start start 2024-04-21T08:53:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.573+0000 7f0c3c9f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c3810e7d0 0x7f0c38118d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.573+0000 7f0c3c9f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c381106f0 0x7f0c38111e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.573+0000 7f0c3c9f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c3811b730 0x7f0c38112340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.573+0000 7f0c3c9f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0c38121000 con 0x7f0c3811b730 2024-04-21T08:53:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.573+0000 7f0c3c9f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0c38120e80 con 0x7f0c381106f0 2024-04-21T08:53:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.573+0000 7f0c3c9f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0c38121180 con 0x7f0c3810e7d0 2024-04-21T08:53:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.573+0000 7f0c3659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c3810e7d0 0x7f0c38118d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.573+0000 7f0c3659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c3810e7d0 0x7f0c38118d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:35984/0 (socket says 172.21.15.7:35984) 2024-04-21T08:53:49.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.573+0000 7f0c3659c640 1 -- 172.21.15.7:0/1391791273 learned_addr learned my addr 172.21.15.7:0/1391791273 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:49.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.574+0000 7f0c3659c640 1 -- 172.21.15.7:0/1391791273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c381106f0 msgr2=0x7f0c38111e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:53:49.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.574+0000 7f0c3659c640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c381106f0 0x7f0c38111e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.574+0000 7f0c3659c640 1 -- 172.21.15.7:0/1391791273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c3811b730 msgr2=0x7f0c38112340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:53:49.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.574+0000 7f0c3659c640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c3811b730 0x7f0c38112340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.574+0000 7f0c3659c640 1 -- 172.21.15.7:0/1391791273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c38112980 con 0x7f0c3810e7d0 2024-04-21T08:53:49.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.574+0000 7f0c3659c640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c3810e7d0 0x7f0c38118d80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c037040 tx=0x7f0c2c035be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:49.577 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.577+0000 7f0c277fe640 1 -- 172.21.15.7:0/1391791273 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c041020 con 0x7f0c3810e7d0 2024-04-21T08:53:49.577 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.577+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/1391791273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c38112c10 con 0x7f0c3810e7d0 2024-04-21T08:53:49.577 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.577+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/1391791273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c38116b90 con 0x7f0c3810e7d0 2024-04-21T08:53:49.577 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.577+0000 7f0c277fe640 1 -- 172.21.15.7:0/1391791273 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c2c02c9e0 con 0x7f0c3810e7d0 2024-04-21T08:53:49.577 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.578+0000 7f0c277fe640 1 -- 172.21.15.7:0/1391791273 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c039040 con 0x7f0c3810e7d0 2024-04-21T08:53:49.580 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.581+0000 7f0c277fe640 1 -- 172.21.15.7:0/1391791273 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0c2c044020 con 0x7f0c3810e7d0 2024-04-21T08:53:49.581 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.582+0000 7f0c277fe640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0c200a3f20 0x7f0c200a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.582 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.582+0000 7f0c35d9b640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0c200a3f20 0x7f0c200a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.585 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.583+0000 7f0c277fe640 1 -- 172.21.15.7:0/1391791273 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9624+0+0 (secure 0 0 0) 0x7f0c2c0f3390 con 0x7f0c3810e7d0 2024-04-21T08:53:49.586 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.583+0000 7f0c35d9b640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0c200a3f20 0x7f0c200a63e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0c381135c0 tx=0x7f0c2800b370 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:49.586 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.586+0000 7f0c3f5e7740 1 Processor -- start 2024-04-21T08:53:49.588 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.588+0000 7f0c3f5e7740 1 -- start start 2024-04-21T08:53:49.588 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.588+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635797988d0 0x563579797620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.588+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563579797b60 0x563579797fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.588+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x5635797a34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.588+0000 7f0c3f5e7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5635796fc8d0 con 0x5635797988d0 2024-04-21T08:53:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.588+0000 7f0c3f5e7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563579699e80 con 0x5635797a10e0 2024-04-21T08:53:49.589 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.588+0000 7f0c3f5e7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5635796f8890 con 0x563579797b60 2024-04-21T08:53:49.591 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.589+0000 7f0c36d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x5635797a34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.591 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.589+0000 7f0c36d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x5635797a34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39464/0 (socket says 172.21.15.7:39464) 2024-04-21T08:53:49.591 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.589+0000 7f0c36d9d640 1 -- 172.21.15.7:0/1142457444 learned_addr learned my addr 172.21.15.7:0/1142457444 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:49.591 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.591+0000 7f0c35d9b640 1 --2- 172.21.15.7:0/1142457444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563579797b60 0x563579797fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.595+0000 7f0c3659c640 1 --2- 172.21.15.7:0/1142457444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635797988d0 0x563579797620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.597 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.596+0000 7f0c35d9b640 1 -- 172.21.15.7:0/1142457444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 msgr2=0x5635797a34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:49.597 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.596+0000 7f0c35d9b640 1 --2- 172.21.15.7:0/1142457444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x5635797a34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.597 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.596+0000 7f0c35d9b640 1 -- 172.21.15.7:0/1142457444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635797988d0 msgr2=0x563579797620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:49.597 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.596+0000 7f0c35d9b640 1 --2- 172.21.15.7:0/1142457444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635797988d0 0x563579797620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.597 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.596+0000 7f0c35d9b640 1 -- 172.21.15.7:0/1142457444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635796bb320 con 0x563579797b60 2024-04-21T08:53:49.601 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.600+0000 7f0c35d9b640 1 --2- 172.21.15.7:0/1142457444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563579797b60 0x563579797fe0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800b770 tx=0x7f0c28010000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd49074335d6c56d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.604+0000 7f0c24ff9640 1 -- 172.21.15.7:0/1142457444 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c28010e40 con 0x563579797b60 2024-04-21T08:53:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.604+0000 7f0c24ff9640 1 -- 172.21.15.7:0/1142457444 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c28022070 con 0x563579797b60 2024-04-21T08:53:49.604 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.605+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1142457444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563579797b60 msgr2=0x563579797fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:49.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.605+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/1142457444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563579797b60 0x563579797fe0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800b770 tx=0x7f0c28010000 comp rx=0 tx=0).stop 2024-04-21T08:53:49.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.617+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1142457444 shutdown_connections 2024-04-21T08:53:49.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.617+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/1142457444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x5635797a34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.617 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.617+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/1142457444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563579797b60 0x563579797fe0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.618 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.617+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/1142457444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5635797988d0 0x563579797620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.618 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.617+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1142457444 >> 172.21.15.7:0/1142457444 conn(0x56357975ec90 msgr2=0x563579792850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:49.618 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.618+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1142457444 shutdown_connections 2024-04-21T08:53:49.618 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.619+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1142457444 wait complete. 2024-04-21T08:53:49.619 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.619+0000 7f0c3f5e7740 1 -- start start 2024-04-21T08:53:49.619 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.619+0000 7f0c3f5e7740 1 Processor -- start 2024-04-21T08:53:49.620 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.620+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 0x5635797a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.620 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.620+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797988d0 0x56357979dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.620 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.620+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797a10e0 0x56357979e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.620 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.620+0000 7f0c3f5e7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5635797a3e00 con 0x563579797b60 2024-04-21T08:53:49.620 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.620+0000 7f0c3f5e7740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x563579684eb0 con 0x5635797988d0 2024-04-21T08:53:49.620 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.620+0000 7f0c3f5e7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5635797a3f80 con 0x5635797a10e0 2024-04-21T08:53:49.620 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.620+0000 7f0c3f5e7740 10 client.0 _ll_register_callbacks cb 0x5635796b1b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.621+0000 7f0c36d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797a10e0 0x56357979e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.621+0000 7f0c36d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797a10e0 0x56357979e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:36008/0 (socket says 172.21.15.7:36008) 2024-04-21T08:53:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.621+0000 7f0c36d9d640 1 -- 172.21.15.7:0/4237362440 learned_addr learned my addr 172.21.15.7:0/4237362440 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.621+0000 7f0c36d9d640 1 -- 172.21.15.7:0/4237362440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797988d0 msgr2=0x56357979dd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.621+0000 7f0c36d9d640 1 --2- 172.21.15.7:0/4237362440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797988d0 0x56357979dd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.621+0000 7f0c36d9d640 1 -- 172.21.15.7:0/4237362440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 msgr2=0x5635797a09a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.621+0000 7f0c36d9d640 1 --2- 172.21.15.7:0/4237362440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 0x5635797a09a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.621+0000 7f0c36d9d640 1 -- 172.21.15.7:0/4237362440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56357940f210 con 0x5635797a10e0 2024-04-21T08:53:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.621+0000 7f0c36d9d640 1 --2- 172.21.15.7:0/4237362440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797a10e0 0x56357979e270 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000f9f0 tx=0x7f0c3000feb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80ffb29da25289e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.630+0000 7f0c1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:53:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.630+0000 7f0c1dffb640 1 -- 172.21.15.7:0/4237362440 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3000ec20 con 0x5635797a10e0 2024-04-21T08:53:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.630+0000 7f0c1dffb640 1 -- 172.21.15.7:0/4237362440 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9624+0+0 (secure 0 0 0) 0x7f0c3000edc0 con 0x5635797a10e0 2024-04-21T08:53:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.630+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/4237362440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c28022070 con 0x5635797a10e0 2024-04-21T08:53:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.640+0000 7f0c1dffb640 1 -- 172.21.15.7:0/4237362440 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0c30004b20 con 0x5635797a10e0 2024-04-21T08:53:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.641+0000 7f0c3f5e7740 10 client.26680 fetch_fsmap learned FSMap version 23 2024-04-21T08:53:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.641+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/4237362440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5635794db620 con 0x5635797a10e0 2024-04-21T08:53:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.642+0000 7f0c1dffb640 1 -- 172.21.15.7:0/4237362440 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c28022070 con 0x5635797a10e0 2024-04-21T08:53:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.642+0000 7f0c1dffb640 1 -- 172.21.15.7:0/4237362440 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f0c3000b050 con 0x5635797a10e0 2024-04-21T08:53:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.642+0000 7f0c3f5e7740 10 client.26680 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:53:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.642+0000 7f0c3f5e7740 10 client.26680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.642+0000 7f0c3f5e7740 20 client.26680 populate_metadata read hostname 'smithi007' 2024-04-21T08:53:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.642+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/4237362440 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5635797b15b0 0x5635797b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.642+0000 7f0c3f5e7740 4 client.26680 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.642+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/4237362440 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563579710880 con 0x5635797b15b0 2024-04-21T08:53:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.644+0000 7f0c3659c640 1 --2- 172.21.15.7:0/4237362440 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5635797b15b0 0x5635797b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.645+0000 7f0c3659c640 1 --2- 172.21.15.7:0/4237362440 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5635797b15b0 0x5635797b3a50 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca058e583d758d26 server_cookie=e92429e8e5ea19a2 in_seq=0 out_seq=0 2024-04-21T08:53:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.645+0000 7f0c1dffb640 10 client.26680 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.646+0000 7f0c1dffb640 1 -- 172.21.15.7:0/4237362440 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f0c2c0c7e50 con 0x5635797b15b0 2024-04-21T08:53:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.646+0000 7f0c1dffb640 10 client.26680 handle_command_reply: tid=0 2024-04-21T08:53:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.646+0000 7f0c3f5e7740 1 client.26680 shutdown 2024-04-21T08:53:49.647 INFO:tasks.workunit.client.0.smithi007.stdout: CC init/calibrate.o 2024-04-21T08:53:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.650+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/4237362440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797a10e0 msgr2=0x56357979e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.650+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/4237362440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797a10e0 0x56357979e270 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000f9f0 tx=0x7f0c3000feb0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.650+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/4237362440 shutdown_connections 2024-04-21T08:53:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.650+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/4237362440 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5635797b15b0 0x5635797b3a50 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.650+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/4237362440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797a10e0 0x56357979e270 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.650+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/4237362440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797988d0 0x56357979dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.650+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/4237362440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 0x5635797a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.650+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/4237362440 >> 172.21.15.7:0/4237362440 conn(0x56357975ec90 msgr2=0x5635797945a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.650+0000 7f0c1dffb640 0 client.26680 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:49.651 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.652+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/4237362440 shutdown_connections 2024-04-21T08:53:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.655+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/4237362440 wait complete. 2024-04-21T08:53:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.658+0000 7f0c3f5e7740 20 client.26680 trim_cache size 0 max 16384 2024-04-21T08:53:49.725 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.726+0000 7f0c3f5e7740 1 Processor -- start 2024-04-21T08:53:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.735+0000 7f0c3f5e7740 1 -- start start 2024-04-21T08:53:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.736+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 0x5635797acde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.736+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 0x5635797a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.736+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x5635797a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.736+0000 7f0c35d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 0x5635797a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.736+0000 7f0c35d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 0x5635797a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:36018/0 (socket says 172.21.15.7:36018) 2024-04-21T08:53:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.736+0000 7f0c35d9b640 1 -- 172.21.15.7:0/1001835884 learned_addr learned my addr 172.21.15.7:0/1001835884 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.736+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1001835884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1800e340 con 0x563579797b60 2024-04-21T08:53:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.737+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1001835884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0c18010bc0 con 0x5635797a10e0 2024-04-21T08:53:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.737+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1001835884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1800e9b0 con 0x5635797988d0 2024-04-21T08:53:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.737+0000 7f0c3659c640 1 --2- 172.21.15.7:0/1001835884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 0x5635797acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.738+0000 7f0c36d9d640 1 --2- 172.21.15.7:0/1001835884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x5635797a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.738+0000 7f0c36d9d640 1 -- 172.21.15.7:0/1001835884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 msgr2=0x5635797a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.738+0000 7f0c36d9d640 1 --2- 172.21.15.7:0/1001835884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 0x5635797a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.738+0000 7f0c36d9d640 1 -- 172.21.15.7:0/1001835884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 msgr2=0x5635797acde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.738+0000 7f0c36d9d640 1 --2- 172.21.15.7:0/1001835884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 0x5635797acde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.738+0000 7f0c36d9d640 1 -- 172.21.15.7:0/1001835884 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56357978df30 con 0x5635797a10e0 2024-04-21T08:53:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.738+0000 7f0c36d9d640 1 --2- 172.21.15.7:0/1001835884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x5635797a09a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0c30002230 tx=0x7f0c30005c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2de57f53b47f3b03 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.741+0000 7f0c1ffff640 1 -- 172.21.15.7:0/1001835884 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c30021410 con 0x5635797a10e0 2024-04-21T08:53:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.741+0000 7f0c1ffff640 1 -- 172.21.15.7:0/1001835884 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c30021d10 con 0x5635797a10e0 2024-04-21T08:53:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.741+0000 7f0c1ffff640 1 -- 172.21.15.7:0/1001835884 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c30022a10 con 0x5635797a10e0 2024-04-21T08:53:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.743+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1001835884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 msgr2=0x5635797a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.743+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/1001835884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x5635797a09a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0c30002230 tx=0x7f0c30005c10 comp rx=0 tx=0).stop 2024-04-21T08:53:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.743+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1001835884 shutdown_connections 2024-04-21T08:53:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.743+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/1001835884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x5635797a09a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.743+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/1001835884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 0x5635797a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.743+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/1001835884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 0x5635797acde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.743+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1001835884 >> 172.21.15.7:0/1001835884 conn(0x56357975ec90 msgr2=0x56357979dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.747+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1001835884 shutdown_connections 2024-04-21T08:53:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.747+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/1001835884 wait complete. 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.747+0000 7f0c3f5e7740 1 -- start start 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.747+0000 7f0c3f5e7740 1 Processor -- start 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.748+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 0x5635797a9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.748+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 0x56357979b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.748+0000 7f0c3f5e7740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x56357979b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.748+0000 7f0c35d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 0x56357979b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.748+0000 7f0c35d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 0x56357979b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:36032/0 (socket says 172.21.15.7:36032) 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.749+0000 7f0c36d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x56357979b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.749+0000 7f0c36d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x56357979b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39478/0 (socket says 172.21.15.7:39478) 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.749+0000 7f0c35d9b640 1 -- 172.21.15.7:0/821915020 learned_addr learned my addr 172.21.15.7:0/821915020 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.749+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/821915020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1800f080 con 0x563579797b60 2024-04-21T08:53:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.749+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/821915020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0c180104f0 con 0x5635797a10e0 2024-04-21T08:53:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.749+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/821915020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1800f750 con 0x5635797988d0 2024-04-21T08:53:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.749+0000 7f0c35d9b640 1 -- 172.21.15.7:0/821915020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 msgr2=0x56357979b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.749+0000 7f0c35d9b640 1 --2- 172.21.15.7:0/821915020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x56357979b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.749+0000 7f0c35d9b640 1 -- 172.21.15.7:0/821915020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 msgr2=0x5635797a9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.749+0000 7f0c35d9b640 1 --2- 172.21.15.7:0/821915020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 0x5635797a9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.749+0000 7f0c35d9b640 1 -- 172.21.15.7:0/821915020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56357979a140 con 0x5635797988d0 2024-04-21T08:53:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.750+0000 7f0c35d9b640 1 --2- 172.21.15.7:0/821915020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 0x56357979b2d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0c28010ef0 tx=0x7f0c28010f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e7b26ff148fce57 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.750+0000 7f0c3f5e7740 10 client.0 _ll_register_callbacks cb 0x7f0c30009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.750+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/821915020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c30021d10 con 0x5635797988d0 2024-04-21T08:53:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.753+0000 7f0c1dffb640 10 client.26686 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:53:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.753+0000 7f0c1dffb640 1 -- 172.21.15.7:0/821915020 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2800dce0 con 0x5635797988d0 2024-04-21T08:53:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.753+0000 7f0c1dffb640 1 -- 172.21.15.7:0/821915020 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9624+0+0 (secure 0 0 0) 0x7f0c2800f960 con 0x5635797988d0 2024-04-21T08:53:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.758+0000 7f0c1dffb640 1 -- 172.21.15.7:0/821915020 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0c2800de80 con 0x5635797988d0 2024-04-21T08:53:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.758+0000 7f0c3f5e7740 10 client.26686 fetch_fsmap learned FSMap version 23 2024-04-21T08:53:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.758+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/821915020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=23}) -- 0x563579816ec0 con 0x5635797988d0 2024-04-21T08:53:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.759+0000 7f0c1dffb640 1 -- 172.21.15.7:0/821915020 <== mon.1 v2:172.21.15.77:3300/0 4 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f0c2800fb60 con 0x5635797988d0 2024-04-21T08:53:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.759+0000 7f0c3f5e7740 10 client.26686 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:53:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.759+0000 7f0c3f5e7740 10 client.26686 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.759+0000 7f0c3f5e7740 20 client.26686 populate_metadata read hostname 'smithi007' 2024-04-21T08:53:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.759+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/821915020 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5635798172d0 0x563579819710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.759+0000 7f0c3f5e7740 4 client.26686 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:53:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.759+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/821915020 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f0c30021d10 con 0x5635798172d0 2024-04-21T08:53:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.765+0000 7f0c3659c640 1 --2- 172.21.15.7:0/821915020 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5635798172d0 0x563579819710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.770+0000 7f0c1dffb640 10 client.26686 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:49.770+0000 7f0c3659c640 1 --2- 172.21.15.7:0/821915020 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5635798172d0 0x563579819710 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b77d38561e091443 server_cookie=57af52600fd9642 in_seq=0 out_seq=0 2024-04-21T08:53:50.086 INFO:tasks.workunit.client.0.smithi007.stdout: LDS arch/x86/entry/vdso/vdso32/vdso32.lds 2024-04-21T08:53:50.164 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/entry/vdso/vdso32/note.o 2024-04-21T08:53:50.176 INFO:tasks.workunit.client.0.smithi007.stdout: CC init/init_task.o 2024-04-21T08:53:50.234 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/oom_kill.o 2024-04-21T08:53:50.244 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/entry/vdso/vdso32/system_call.o 2024-04-21T08:53:50.304 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/iomap/direct-io.o 2024-04-21T08:53:50.317 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/notify/group.o 2024-04-21T08:53:50.321 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/entry/vdso/vdso32/sigreturn.o 2024-04-21T08:53:50.456 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/vdso/vdso32/vclock_gettime.o 2024-04-21T08:53:50.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:50 smithi007 ceph-mon[28477]: pgmap v286: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 696 KiB/s wr, 1.67k op/s 2024-04-21T08:53:50.963 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/quota/dquot.o 2024-04-21T08:53:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:50 smithi016 ceph-mon[31259]: pgmap v286: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 696 KiB/s wr, 1.67k op/s 2024-04-21T08:53:51.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:50 smithi077 ceph-mon[31615]: pgmap v286: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 696 KiB/s wr, 1.67k op/s 2024-04-21T08:53:51.114 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/notify/mark.o 2024-04-21T08:53:51.167 INFO:tasks.workunit.client.0.smithi007.stdout: CC init/version.o 2024-04-21T08:53:51.339 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/vdso/vdso32/vgetcpu.o 2024-04-21T08:53:51.494 INFO:tasks.workunit.client.0.smithi007.stdout: AR init/built-in.a 2024-04-21T08:53:51.500 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/iomap/fiemap.o 2024-04-21T08:53:51.743 INFO:tasks.workunit.client.0.smithi007.stdout: VDSO arch/x86/entry/vdso/vdso64.so.dbg 2024-04-21T08:53:51.746 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/quota/quota_v2.o 2024-04-21T08:53:51.833 INFO:tasks.workunit.client.0.smithi007.stdout: CC ipc/compat.o 2024-04-21T08:53:52.043 INFO:tasks.workunit.client.0.smithi007.stdout: VDSO arch/x86/entry/vdso/vdso32.so.dbg 2024-04-21T08:53:52.102 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/fadvise.o 2024-04-21T08:53:52.134 INFO:tasks.workunit.client.0.smithi007.stdout: OBJCOPY arch/x86/entry/vdso/vdso64.so 2024-04-21T08:53:52.165 INFO:tasks.workunit.client.0.smithi007.stdout: OBJCOPY arch/x86/entry/vdso/vdso32.so 2024-04-21T08:53:52.192 INFO:tasks.workunit.client.0.smithi007.stdout: VDSO2C arch/x86/entry/vdso/vdso-image-64.c 2024-04-21T08:53:52.219 INFO:tasks.workunit.client.0.smithi007.stdout: VDSO2C arch/x86/entry/vdso/vdso-image-32.c 2024-04-21T08:53:52.238 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/notify/fdinfo.o 2024-04-21T08:53:52.250 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/vdso/vdso-image-64.o 2024-04-21T08:53:52.444 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/iomap/seek.o 2024-04-21T08:53:52.620 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/vdso/vdso-image-32.o 2024-04-21T08:53:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.764+0000 7f0c1dffb640 1 -- 172.21.15.7:0/821915020 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f0c30021d10 con 0x5635798172d0 2024-04-21T08:53:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.764+0000 7f0c1dffb640 10 client.26686 handle_command_reply: tid=0 2024-04-21T08:53:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.764+0000 7f0c3f5e7740 1 client.26686 shutdown 2024-04-21T08:53:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.787+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/821915020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 msgr2=0x56357979b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.787+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/821915020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 0x56357979b2d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0c28010ef0 tx=0x7f0c28010f20 comp rx=0 tx=0).stop 2024-04-21T08:53:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.787+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/821915020 shutdown_connections 2024-04-21T08:53:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.787+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/821915020 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5635798172d0 0x563579819710 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.787+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/821915020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5635797a10e0 0x56357979b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.787+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/821915020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5635797988d0 0x56357979b2d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.787+0000 7f0c3f5e7740 1 --2- 172.21.15.7:0/821915020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563579797b60 0x5635797a9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.787+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/821915020 >> 172.21.15.7:0/821915020 conn(0x56357975ec90 msgr2=0x56357979c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.788+0000 7f0c1dffb640 0 client.26686 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:53:52.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.793+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/821915020 shutdown_connections 2024-04-21T08:53:52.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.793+0000 7f0c3f5e7740 1 -- 172.21.15.7:0/821915020 wait complete. 2024-04-21T08:53:52.802 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.803+0000 7f0c3f5e7740 20 client.26686 trim_cache size 0 max 16384 2024-04-21T08:53:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:53:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:53:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:53:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:53:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.822+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/1391791273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0c200a3f20 msgr2=0x7f0c200a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.822+0000 7f0c3c9f4640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0c200a3f20 0x7f0c200a63e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0c381135c0 tx=0x7f0c2800b370 comp rx=0 tx=0).stop 2024-04-21T08:53:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.822+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/1391791273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c3810e7d0 msgr2=0x7f0c38118d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.822+0000 7f0c3c9f4640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c3810e7d0 0x7f0c38118d80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c037040 tx=0x7f0c2c035be0 comp rx=0 tx=0).stop 2024-04-21T08:53:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.823+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/1391791273 shutdown_connections 2024-04-21T08:53:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.823+0000 7f0c3c9f4640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c3811b730 0x7f0c38112340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.823+0000 7f0c3c9f4640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c381106f0 0x7f0c38111e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.823+0000 7f0c3c9f4640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f0c200a3f20 0x7f0c200a63e0 secure :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0c381135c0 tx=0x7f0c2800b370 comp rx=0 tx=0).stop 2024-04-21T08:53:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.823+0000 7f0c3c9f4640 1 --2- 172.21.15.7:0/1391791273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c3810e7d0 0x7f0c38118d80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.823+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/1391791273 >> 172.21.15.7:0/1391791273 conn(0x7f0c3806d030 msgr2=0x7f0c3811a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.823+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/1391791273 shutdown_connections 2024-04-21T08:53:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:53:52.823+0000 7f0c3c9f4640 1 -- 172.21.15.7:0/1391791273 wait complete. 2024-04-21T08:53:52.833 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/amd/core.o 2024-04-21T08:53:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:52 smithi007 ceph-mon[28477]: pgmap v287: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 510 KiB/s wr, 1.75k op/s 2024-04-21T08:53:53.010 INFO:tasks.workunit.client.0.smithi007.stdout: CC ipc/util.o 2024-04-21T08:53:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:52 smithi016 ceph-mon[31259]: pgmap v287: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 510 KiB/s wr, 1.75k op/s 2024-04-21T08:53:53.056 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/entry/vdso/built-in.a 2024-04-21T08:53:53.090 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/notify/built-in.a 2024-04-21T08:53:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:52 smithi077 ceph-mon[31615]: pgmap v287: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 510 KiB/s wr, 1.75k op/s 2024-04-21T08:53:53.106 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/maccess.o 2024-04-21T08:53:53.132 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/common.o 2024-04-21T08:53:53.150 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/entry/thunk_64.o 2024-04-21T08:53:53.227 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/entry/entry_64_compat.o 2024-04-21T08:53:53.267 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/iomap/swapfile.o 2024-04-21T08:53:53.316 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/quota/quota_tree.o 2024-04-21T08:53:53.874 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/sched/build_policy.o 2024-04-21T08:53:53.876 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/page-writeback.o 2024-04-21T08:53:53.890 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/entry/syscall_32.o 2024-04-21T08:53:53.905 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/task_mmu.o 2024-04-21T08:53:53.955 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/amd/lbr.o 2024-04-21T08:53:54.050 INFO:tasks.workunit.client.0.smithi007.stdout: CC ipc/msgutil.o 2024-04-21T08:53:54.163 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/iomap/built-in.a 2024-04-21T08:53:54.207 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/quota/quota.o 2024-04-21T08:53:54.272 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/core.o 2024-04-21T08:53:54.625 INFO:tasks.workunit.client.0.smithi007.stdout: CC ipc/msg.o 2024-04-21T08:53:54.700 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/entry/built-in.a 2024-04-21T08:53:54.757 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/amd/ibs.o 2024-04-21T08:53:54.801 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/kernfs/mount.o 2024-04-21T08:53:54.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:54 smithi007 ceph-mon[28477]: pgmap v288: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 955 KiB/s wr, 1.98k op/s 2024-04-21T08:53:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:54 smithi016 ceph-mon[31259]: pgmap v288: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 955 KiB/s wr, 1.98k op/s 2024-04-21T08:53:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:54 smithi077 ceph-mon[31615]: pgmap v288: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 955 KiB/s wr, 1.98k op/s 2024-04-21T08:53:55.438 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/inode.o 2024-04-21T08:53:55.481 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/quota/kqid.o 2024-04-21T08:53:55.601 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/folio-compat.o 2024-04-21T08:53:55.651 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/kernfs/inode.o 2024-04-21T08:53:55.791 INFO:tasks.workunit.client.0.smithi007.stdout: CC ipc/sem.o 2024-04-21T08:53:55.829 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/amd/uncore.o 2024-04-21T08:53:56.078 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/quota/netlink.o 2024-04-21T08:53:56.296 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/root.o 2024-04-21T08:53:56.360 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/kernfs/dir.o 2024-04-21T08:53:56.456 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/readahead.o 2024-04-21T08:53:56.826 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/amd/iommu.o 2024-04-21T08:53:56.857 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/bts.o 2024-04-21T08:53:56.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:56 smithi007 ceph-mon[28477]: pgmap v289: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1020 KiB/s wr, 2.03k op/s 2024-04-21T08:53:57.025 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/quota/built-in.a 2024-04-21T08:53:57.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:56 smithi016 ceph-mon[31259]: pgmap v289: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1020 KiB/s wr, 2.03k op/s 2024-04-21T08:53:57.067 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/sched/build_utility.o 2024-04-21T08:53:57.075 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/base.o 2024-04-21T08:53:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:56 smithi077 ceph-mon[31615]: pgmap v289: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1020 KiB/s wr, 2.03k op/s 2024-04-21T08:53:57.182 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/sysfs/file.o 2024-04-21T08:53:57.444 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/swap.o 2024-04-21T08:53:57.472 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/kernfs/file.o 2024-04-21T08:53:57.558 INFO:tasks.workunit.client.0.smithi007.stdout: CC ipc/shm.o 2024-04-21T08:53:57.778 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/events/amd/built-in.a 2024-04-21T08:53:57.830 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/ds.o 2024-04-21T08:53:57.889 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/devpts/inode.o 2024-04-21T08:53:58.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:57 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:53:58.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:57 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:53:58.126 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/sysfs/dir.o 2024-04-21T08:53:58.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:57 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:53:58.509 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/kernfs/symlink.o 2024-04-21T08:53:58.819 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/sysfs/symlink.o 2024-04-21T08:53:58.832 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/devpts/built-in.a 2024-04-21T08:53:58.936 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/sysfs/mount.o 2024-04-21T08:53:59.037 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/generic.o 2024-04-21T08:53:59.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:58 smithi016 ceph-mon[31259]: pgmap v290: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 800 KiB/s wr, 2.34k op/s 2024-04-21T08:53:59.061 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/truncate.o 2024-04-21T08:53:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:58 smithi077 ceph-mon[31615]: pgmap v290: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 800 KiB/s wr, 2.34k op/s 2024-04-21T08:53:59.110 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/kernfs/built-in.a 2024-04-21T08:53:59.141 INFO:tasks.workunit.client.0.smithi007.stdout: CC ipc/syscall.o 2024-04-21T08:53:59.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:58 smithi007 ceph-mon[28477]: pgmap v290: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 800 KiB/s wr, 2.34k op/s 2024-04-21T08:53:59.201 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/sysfs/group.o 2024-04-21T08:53:59.504 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/array.o 2024-04-21T08:53:59.640 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/knc.o 2024-04-21T08:53:59.825 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/lbr.o 2024-04-21T08:53:59.985 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/fd.o 2024-04-21T08:54:00.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:59 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:54:00.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:59 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:54:00.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:53:59 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:54:00.051 INFO:tasks.workunit.client.0.smithi007.stdout: CC ipc/ipc_sysctl.o 2024-04-21T08:54:00.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:59 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:54:00.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:59 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:54:00.056 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:53:59 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:54:00.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:59 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:54:00.156 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:59 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:54:00.156 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:53:59 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:54:00.163 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/sysfs/built-in.a 2024-04-21T08:54:00.318 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/vmscan.o 2024-04-21T08:54:00.340 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/netfs/buffered_read.o 2024-04-21T08:54:00.366 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:54:00.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:54:00.716 INFO:tasks.workunit.client.0.smithi007.stdout: CC ipc/mqueue.o 2024-04-21T08:54:00.718 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/p4.o 2024-04-21T08:54:00.854 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/balloc.o 2024-04-21T08:54:00.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.896+0000 7f13d3575640 1 -- 172.21.15.7:0/2229837538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc10e1b0 msgr2=0x7f13cc10e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:00.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.896+0000 7f13d3575640 1 --2- 172.21.15.7:0/2229837538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc10e1b0 0x7f13cc10e660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f13bc01a910 tx=0x7f13bc03cca0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.896+0000 7f13d3575640 1 -- 172.21.15.7:0/2229837538 shutdown_connections 2024-04-21T08:54:00.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.896+0000 7f13d3575640 1 --2- 172.21.15.7:0/2229837538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc10e1b0 0x7f13cc10e660 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.896+0000 7f13d3575640 1 --2- 172.21.15.7:0/2229837538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13cc110b10 0x7f13cc109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.896+0000 7f13d3575640 1 --2- 172.21.15.7:0/2229837538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13cc1106a0 0x7f13cc109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.896+0000 7f13d3575640 1 -- 172.21.15.7:0/2229837538 >> 172.21.15.7:0/2229837538 conn(0x7f13cc06e360 msgr2=0x7f13cc06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:00.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.896+0000 7f13d3575640 1 -- 172.21.15.7:0/2229837538 shutdown_connections 2024-04-21T08:54:00.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.896+0000 7f13d3575640 1 -- 172.21.15.7:0/2229837538 wait complete. 2024-04-21T08:54:00.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d3575640 1 Processor -- start 2024-04-21T08:54:00.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d3575640 1 -- start start 2024-04-21T08:54:00.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d3575640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13cc10e1b0 0x7f13cc07fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:00.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d3575640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc1106a0 0x7f13cc0803d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:00.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d3575640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13cc110b10 0x7f13cc07afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:00.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d3575640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc112970 con 0x7f13cc10e1b0 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d3575640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc1127f0 con 0x7f13cc1106a0 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d3575640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc112af0 con 0x7f13cc110b10 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d0b10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc1106a0 0x7f13cc0803d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d1b12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13cc110b10 0x7f13cc07afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d0b10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc1106a0 0x7f13cc0803d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:50260/0 (socket says 172.21.15.7:50260) 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d1b12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13cc110b10 0x7f13cc07afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:33458/0 (socket says 172.21.15.7:33458) 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d0b10640 1 -- 172.21.15.7:0/295865903 learned_addr learned my addr 172.21.15.7:0/295865903 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.897+0000 7f13d1311640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13cc10e1b0 0x7f13cc07fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.908+0000 7f13d0b10640 1 -- 172.21.15.7:0/295865903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13cc110b10 msgr2=0x7f13cc07afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.908+0000 7f13d0b10640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13cc110b10 0x7f13cc07afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.908+0000 7f13d0b10640 1 -- 172.21.15.7:0/295865903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13cc10e1b0 msgr2=0x7f13cc07fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.908+0000 7f13d0b10640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13cc10e1b0 0x7f13cc07fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.908+0000 7f13d0b10640 1 -- 172.21.15.7:0/295865903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13cc07b860 con 0x7f13cc1106a0 2024-04-21T08:54:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.909+0000 7f13d1311640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13cc10e1b0 0x7f13cc07fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.922+0000 7f13d0b10640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc1106a0 0x7f13cc0803d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f13c404f990 tx=0x7f13c404fe50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.922+0000 7f13c27fc640 1 -- 172.21.15.7:0/295865903 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c405d070 con 0x7f13cc1106a0 2024-04-21T08:54:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.922+0000 7f13d3575640 1 -- 172.21.15.7:0/295865903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13cc07bb50 con 0x7f13cc1106a0 2024-04-21T08:54:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.922+0000 7f13d3575640 1 -- 172.21.15.7:0/295865903 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13cc083d70 con 0x7f13cc1106a0 2024-04-21T08:54:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.922+0000 7f13c27fc640 1 -- 172.21.15.7:0/295865903 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13c4058440 con 0x7f13cc1106a0 2024-04-21T08:54:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.922+0000 7f13c27fc640 1 -- 172.21.15.7:0/295865903 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c40565a0 con 0x7f13cc1106a0 2024-04-21T08:54:00.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.926+0000 7f13d6165740 1 Processor -- start 2024-04-21T08:54:00.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.926+0000 7f13d6165740 1 -- start start 2024-04-21T08:54:00.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.927+0000 7f13d6165740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:00.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.927+0000 7f13d6165740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 0x55d0268cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:00.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.927+0000 7f13d6165740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268cfa80 0x55d0268d1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:00.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.927+0000 7f13d6165740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d02682b1e0 con 0x55d0268cfa80 2024-04-21T08:54:00.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.927+0000 7f13d6165740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d0267c8390 con 0x55d0268c6ea0 2024-04-21T08:54:00.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.927+0000 7f13d6165740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d0268271a0 con 0x55d0268c6580 2024-04-21T08:54:00.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.930+0000 7f13d1b12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:00.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.930+0000 7f13d1b12640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268c6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:33462/0 (socket says 172.21.15.7:33462) 2024-04-21T08:54:00.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.930+0000 7f13d1b12640 1 -- 172.21.15.7:0/1460770616 learned_addr learned my addr 172.21.15.7:0/1460770616 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:00.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.930+0000 7f13d1b12640 1 -- 172.21.15.7:0/1460770616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 msgr2=0x55d0268cf540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:00.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.930+0000 7f13d1b12640 1 --2- 172.21.15.7:0/1460770616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 0x55d0268cf540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.930+0000 7f13d1b12640 1 -- 172.21.15.7:0/1460770616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268cfa80 msgr2=0x55d0268d1e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:00.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.930+0000 7f13d1b12640 1 --2- 172.21.15.7:0/1460770616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268cfa80 0x55d0268d1e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.930+0000 7f13d1b12640 1 -- 172.21.15.7:0/1460770616 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0266077d0 con 0x55d0268c6580 2024-04-21T08:54:00.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.930+0000 7f13d1b12640 1 --2- 172.21.15.7:0/1460770616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268c6960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f13bc01a910 tx=0x7f13bc005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f5119b52d99c127 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:00.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.932+0000 7f13c27fc640 1 -- 172.21.15.7:0/295865903 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f13c4060020 con 0x7f13cc1106a0 2024-04-21T08:54:00.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.933+0000 7f13c27fc640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f13b80a3de0 0x7f13b80a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:00.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.933+0000 7f13c27fc640 1 -- 172.21.15.7:0/295865903 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9624+0+0 (secure 0 0 0) 0x7f13c4113e20 con 0x7f13cc1106a0 2024-04-21T08:54:00.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.934+0000 7f13b77fe640 1 -- 172.21.15.7:0/1460770616 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc004b20 con 0x55d0268c6580 2024-04-21T08:54:00.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.934+0000 7f13b77fe640 1 -- 172.21.15.7:0/1460770616 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13bc004cc0 con 0x55d0268c6580 2024-04-21T08:54:00.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.935+0000 7f13d6165740 1 -- 172.21.15.7:0/1460770616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 msgr2=0x55d0268c6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:00.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.935+0000 7f13d6165740 1 --2- 172.21.15.7:0/1460770616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268c6960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f13bc01a910 tx=0x7f13bc005c00 comp rx=0 tx=0).stop 2024-04-21T08:54:00.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.941+0000 7f13d6165740 1 -- 172.21.15.7:0/1460770616 shutdown_connections 2024-04-21T08:54:00.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.941+0000 7f13d6165740 1 --2- 172.21.15.7:0/1460770616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268cfa80 0x55d0268d1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.941+0000 7f13d6165740 1 --2- 172.21.15.7:0/1460770616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 0x55d0268cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.941+0000 7f13d6165740 1 --2- 172.21.15.7:0/1460770616 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268c6960 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.941+0000 7f13d6165740 1 -- 172.21.15.7:0/1460770616 >> 172.21.15.7:0/1460770616 conn(0x55d0268c0cb0 msgr2=0x55d0268c1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:00.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.942+0000 7f13d1311640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f13b80a3de0 0x7f13b80a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:00.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.944+0000 7f13d6165740 1 -- 172.21.15.7:0/1460770616 shutdown_connections 2024-04-21T08:54:00.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.944+0000 7f13d6165740 1 -- 172.21.15.7:0/1460770616 wait complete. 2024-04-21T08:54:00.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.945+0000 7f13d6165740 1 -- start start 2024-04-21T08:54:00.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.945+0000 7f13d6165740 1 Processor -- start 2024-04-21T08:54:00.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.945+0000 7f13d6165740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268c6580 0x55d0268ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:00.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.945+0000 7f13d6165740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6ea0 0x55d0268cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:00.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.945+0000 7f13d6165740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268cfa80 0x55d0268cd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:00.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.945+0000 7f13d6165740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d0268d2860 con 0x55d0268c6580 2024-04-21T08:54:00.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.945+0000 7f13d6165740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55d0267e42f0 con 0x55d0268cfa80 2024-04-21T08:54:00.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.945+0000 7f13d6165740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d0268d29e0 con 0x55d0268c6ea0 2024-04-21T08:54:00.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.945+0000 7f13d6165740 10 client.0 _ll_register_callbacks cb 0x55d0267f1580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:00.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.946+0000 7f13d0b10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6ea0 0x55d0268cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:00.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.946+0000 7f13d0b10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6ea0 0x55d0268cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:33466/0 (socket says 172.21.15.7:33466) 2024-04-21T08:54:00.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.946+0000 7f13d0b10640 1 -- 172.21.15.7:0/3482439654 learned_addr learned my addr 172.21.15.7:0/3482439654 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:00.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.946+0000 7f13d1311640 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268c6580 0x55d0268ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:00.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.946+0000 7f13d0b10640 1 -- 172.21.15.7:0/3482439654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268cfa80 msgr2=0x55d0268cd550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:00.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.946+0000 7f13d0b10640 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268cfa80 0x55d0268cd550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.946+0000 7f13d0b10640 1 -- 172.21.15.7:0/3482439654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268c6580 msgr2=0x55d0268ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:00.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.946+0000 7f13d0b10640 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268c6580 0x55d0268ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.946+0000 7f13d0b10640 1 -- 172.21.15.7:0/3482439654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d02653d3f0 con 0x55d0268c6ea0 2024-04-21T08:54:00.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.946+0000 7f13d0b10640 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6ea0 0x55d0268cd010 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f13c404f960 tx=0x7f13c40e02c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e90bb7065c0efc90 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:00.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.950+0000 7f13b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:54:00.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.950+0000 7f13b4ff9640 1 -- 172.21.15.7:0/3482439654 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c40dfce0 con 0x55d0268c6ea0 2024-04-21T08:54:00.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.950+0000 7f13d6165740 1 -- 172.21.15.7:0/3482439654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13bc004cc0 con 0x55d0268c6ea0 2024-04-21T08:54:00.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.950+0000 7f13b4ff9640 1 -- 172.21.15.7:0/3482439654 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9624+0+0 (secure 0 0 0) 0x7f13c40dfe80 con 0x55d0268c6ea0 2024-04-21T08:54:00.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.950+0000 7f13b4ff9640 1 -- 172.21.15.7:0/3482439654 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f13c40e8260 con 0x55d0268c6ea0 2024-04-21T08:54:00.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.950+0000 7f13d6165740 10 client.26695 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:00.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.950+0000 7f13d6165740 1 -- 172.21.15.7:0/3482439654 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d0264fd0f0 con 0x55d0268c6ea0 2024-04-21T08:54:00.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.953+0000 7f13b4ff9640 1 -- 172.21.15.7:0/3482439654 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13bc004cc0 con 0x55d0268c6ea0 2024-04-21T08:54:00.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.953+0000 7f13b4ff9640 1 -- 172.21.15.7:0/3482439654 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f13c40db7d0 con 0x55d0268c6ea0 2024-04-21T08:54:00.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.953+0000 7f13d6165740 10 client.26695 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:00.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.953+0000 7f13d6165740 10 client.26695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:00.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.953+0000 7f13d6165740 20 client.26695 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:00.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.953+0000 7f13d6165740 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0268e0040 0x55d0268e2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:00.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.953+0000 7f13d6165740 4 client.26695 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:00.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.953+0000 7f13d6165740 1 -- 172.21.15.7:0/3482439654 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d026838c00 con 0x55d0268e0040 2024-04-21T08:54:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.955+0000 7f13d1311640 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0268e0040 0x55d0268e2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.955+0000 7f13d1311640 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0268e0040 0x55d0268e2460 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebe07fe7905c3381 server_cookie=b49940ab2066f8de in_seq=0 out_seq=0 2024-04-21T08:54:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.958+0000 7f13b4ff9640 10 client.26695 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.958+0000 7f13b4ff9640 1 -- 172.21.15.7:0/3482439654 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d026838c00 con 0x55d0268e0040 2024-04-21T08:54:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.958+0000 7f13b4ff9640 10 client.26695 handle_command_reply: tid=0 2024-04-21T08:54:00.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.958+0000 7f13d1311640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f13b80a3de0 0x7f13b80a62a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55d0268c6050 tx=0x7f13c800a0d0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.958+0000 7f13d6165740 1 client.26695 shutdown 2024-04-21T08:54:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.958+0000 7f13d6165740 1 -- 172.21.15.7:0/3482439654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6ea0 msgr2=0x55d0268cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.958+0000 7f13d6165740 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6ea0 0x55d0268cd010 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f13c404f960 tx=0x7f13c40e02c0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.959+0000 7f13d6165740 1 -- 172.21.15.7:0/3482439654 shutdown_connections 2024-04-21T08:54:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.959+0000 7f13d6165740 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d0268e0040 0x55d0268e2460 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.959+0000 7f13d6165740 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268cfa80 0x55d0268cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.959+0000 7f13d6165740 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6ea0 0x55d0268cd010 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.959+0000 7f13d6165740 1 --2- 172.21.15.7:0/3482439654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268c6580 0x55d0268ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:00.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.959+0000 7f13d6165740 1 -- 172.21.15.7:0/3482439654 >> 172.21.15.7:0/3482439654 conn(0x55d0268c0cb0 msgr2=0x55d0268bfd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:00.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.959+0000 7f13b4ff9640 0 client.26695 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:00.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.960+0000 7f13d6165740 1 -- 172.21.15.7:0/3482439654 shutdown_connections 2024-04-21T08:54:00.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.960+0000 7f13d6165740 1 -- 172.21.15.7:0/3482439654 wait complete. 2024-04-21T08:54:00.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:00.968+0000 7f13d6165740 20 client.26695 trim_cache size 0 max 16384 2024-04-21T08:54:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.037+0000 7f13d6165740 1 Processor -- start 2024-04-21T08:54:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.037+0000 7f13d6165740 1 -- start start 2024-04-21T08:54:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.037+0000 7f13d6165740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.038+0000 7f13d6165740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 0x55d0268db800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.038+0000 7f13d6165740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268cfa80 0x55d0268e7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.038+0000 7f13d6165740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f13b000e340 con 0x55d0268cfa80 2024-04-21T08:54:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.038+0000 7f13d6165740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13b0010bc0 con 0x55d0268c6ea0 2024-04-21T08:54:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.038+0000 7f13d6165740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f13b000e9b0 con 0x55d0268c6580 2024-04-21T08:54:01.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.040+0000 7f13d1b12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268cfa80 0x55d0268e7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:01.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.040+0000 7f13d1b12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268cfa80 0x55d0268e7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33840/0 (socket says 172.21.15.7:33840) 2024-04-21T08:54:01.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.040+0000 7f13d1b12640 1 -- 172.21.15.7:0/2591917251 learned_addr learned my addr 172.21.15.7:0/2591917251 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.044+0000 7f13d1b12640 1 -- 172.21.15.7:0/2591917251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 msgr2=0x55d0268cf200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.044+0000 7f13d1b12640 1 --2- 172.21.15.7:0/2591917251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268cf200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.044+0000 7f13d1b12640 1 -- 172.21.15.7:0/2591917251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 msgr2=0x55d0268db800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.044+0000 7f13d1b12640 1 --2- 172.21.15.7:0/2591917251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 0x55d0268db800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.044+0000 7f13d1b12640 1 -- 172.21.15.7:0/2591917251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0268c0ae0 con 0x55d0268cfa80 2024-04-21T08:54:01.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.048+0000 7f13d1b12640 1 --2- 172.21.15.7:0/2591917251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268cfa80 0x55d0268e7d30 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0056d0 tx=0x7f13bc01a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e158788f6b9f90 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:01.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.049+0000 7f13b6ffd640 1 -- 172.21.15.7:0/2591917251 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc004a60 con 0x55d0268cfa80 2024-04-21T08:54:01.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.049+0000 7f13b6ffd640 1 -- 172.21.15.7:0/2591917251 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13bc004c00 con 0x55d0268cfa80 2024-04-21T08:54:01.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.050+0000 7f13d6165740 1 -- 172.21.15.7:0/2591917251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268cfa80 msgr2=0x55d0268e7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:01.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.050+0000 7f13d6165740 1 --2- 172.21.15.7:0/2591917251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268cfa80 0x55d0268e7d30 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0056d0 tx=0x7f13bc01a690 comp rx=0 tx=0).stop 2024-04-21T08:54:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.051+0000 7f13d6165740 1 -- 172.21.15.7:0/2591917251 shutdown_connections 2024-04-21T08:54:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.051+0000 7f13d6165740 1 --2- 172.21.15.7:0/2591917251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268cfa80 0x55d0268e7d30 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.051+0000 7f13d6165740 1 --2- 172.21.15.7:0/2591917251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 0x55d0268db800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.051+0000 7f13d6165740 1 --2- 172.21.15.7:0/2591917251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.051+0000 7f13d6165740 1 -- 172.21.15.7:0/2591917251 >> 172.21.15.7:0/2591917251 conn(0x55d0268c0cb0 msgr2=0x55d0268c7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:01.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.051+0000 7f13d6165740 1 -- 172.21.15.7:0/2591917251 shutdown_connections 2024-04-21T08:54:01.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.053+0000 7f13d6165740 1 -- 172.21.15.7:0/2591917251 wait complete. 2024-04-21T08:54:01.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.053+0000 7f13d6165740 1 -- start start 2024-04-21T08:54:01.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.053+0000 7f13d6165740 1 Processor -- start 2024-04-21T08:54:01.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.054+0000 7f13d6165740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268d7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:01.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.054+0000 7f13d6165740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 0x55d0268d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:01.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.054+0000 7f13d6165740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268d3810 0x55d0268d3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:01.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.054+0000 7f13d6165740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f13b000f080 con 0x55d0268d3810 2024-04-21T08:54:01.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.054+0000 7f13d6165740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f13b00104f0 con 0x55d0268c6ea0 2024-04-21T08:54:01.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.054+0000 7f13d6165740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f13b000f750 con 0x55d0268c6580 2024-04-21T08:54:01.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.054+0000 7f13d6165740 10 client.0 _ll_register_callbacks cb 0x7f13c404d8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:01.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.057+0000 7f13d1b12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268d3810 0x55d0268d3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:01.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.057+0000 7f13d1b12640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268d3810 0x55d0268d3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33850/0 (socket says 172.21.15.7:33850) 2024-04-21T08:54:01.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.057+0000 7f13d1b12640 1 -- 172.21.15.7:0/23811032 learned_addr learned my addr 172.21.15.7:0/23811032 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:01.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.060+0000 7f13d1311640 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268d7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:01.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.060+0000 7f13d1311640 1 -- 172.21.15.7:0/23811032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 msgr2=0x55d0268d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:01.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.060+0000 7f13d1311640 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 0x55d0268d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.060+0000 7f13d1311640 1 -- 172.21.15.7:0/23811032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268d3810 msgr2=0x55d0268d3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:01.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.060+0000 7f13d1311640 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268d3810 0x55d0268d3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.060+0000 7f13d1311640 1 -- 172.21.15.7:0/23811032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0269455a0 con 0x55d0268c6580 2024-04-21T08:54:01.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.061+0000 7f13d1311640 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268d7f00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55d0268e2a40 tx=0x7f13c8008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d38780ce7c1850a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:01.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.061+0000 7f13b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:54:01.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.061+0000 7f13b4ff9640 1 -- 172.21.15.7:0/23811032 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c8027070 con 0x55d0268c6580 2024-04-21T08:54:01.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.061+0000 7f13d6165740 1 -- 172.21.15.7:0/23811032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d026832a70 con 0x55d0268c6580 2024-04-21T08:54:01.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.061+0000 7f13b4ff9640 1 -- 172.21.15.7:0/23811032 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9624+0+0 (secure 0 0 0) 0x7f13c8023250 con 0x55d0268c6580 2024-04-21T08:54:01.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.062+0000 7f13b4ff9640 1 -- 172.21.15.7:0/23811032 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f13c8033050 con 0x55d0268c6580 2024-04-21T08:54:01.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.062+0000 7f13d6165740 10 client.26698 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:01.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.062+0000 7f13d6165740 1 -- 172.21.15.7:0/23811032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d0269468b0 con 0x55d0268c6580 2024-04-21T08:54:01.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.062+0000 7f13b4ff9640 1 -- 172.21.15.7:0/23811032 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d026832a70 con 0x55d0268c6580 2024-04-21T08:54:01.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.064+0000 7f13b4ff9640 1 -- 172.21.15.7:0/23811032 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f13c8015070 con 0x55d0268c6580 2024-04-21T08:54:01.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.064+0000 7f13d6165740 10 client.26698 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:01.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.064+0000 7f13d6165740 10 client.26698 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:01.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.064+0000 7f13d6165740 20 client.26698 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:01.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.064+0000 7f13d6165740 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d026946fe0 0x55d026949400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:01.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.064+0000 7f13d6165740 4 client.26698 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:54:01.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.064+0000 7f13d6165740 1 -- 172.21.15.7:0/23811032 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d0267ae590 con 0x55d026946fe0 2024-04-21T08:54:01.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.064+0000 7f13d0b10640 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d026946fe0 0x55d026949400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:01.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.065+0000 7f13d0b10640 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d026946fe0 0x55d026949400 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3956be15f695fad server_cookie=72ec7fd23dc7687a in_seq=0 out_seq=0 2024-04-21T08:54:01.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13b4ff9640 10 client.26698 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:01.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13b4ff9640 1 -- 172.21.15.7:0/23811032 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55d0267ae590 con 0x55d026946fe0 2024-04-21T08:54:01.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13b4ff9640 10 client.26698 handle_command_reply: tid=0 2024-04-21T08:54:01.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13d6165740 1 client.26698 shutdown 2024-04-21T08:54:01.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13d6165740 1 -- 172.21.15.7:0/23811032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 msgr2=0x55d0268d7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:01.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13d6165740 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268d7f00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55d0268e2a40 tx=0x7f13c8008040 comp rx=0 tx=0).stop 2024-04-21T08:54:01.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13d6165740 1 -- 172.21.15.7:0/23811032 shutdown_connections 2024-04-21T08:54:01.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13d6165740 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55d026946fe0 0x55d026949400 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13d6165740 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d0268d3810 0x55d0268d3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13d6165740 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55d0268c6ea0 0x55d0268d32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13d6165740 1 --2- 172.21.15.7:0/23811032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0268c6580 0x55d0268d7f00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13d6165740 1 -- 172.21.15.7:0/23811032 >> 172.21.15.7:0/23811032 conn(0x55d0268c0cb0 msgr2=0x55d0268bf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:01.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13b4ff9640 0 client.26698 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:01.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.070+0000 7f13d6165740 1 -- 172.21.15.7:0/23811032 shutdown_connections 2024-04-21T08:54:01.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.071+0000 7f13d6165740 1 -- 172.21.15.7:0/23811032 wait complete. 2024-04-21T08:54:01.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.072+0000 7f13d6165740 20 client.26698 trim_cache size 0 max 16384 2024-04-21T08:54:01.072 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:54:01.072 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (4514 inodes in the stack)", 2024-04-21T08:54:01.072 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:54:01.072 INFO:teuthology.orchestra.run.smithi007.stdout: "2f44a19d-b0a7-4681-b27b-fb7529461620": { 2024-04-21T08:54:01.072 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:54:01.072 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "2f44a19d-b0a7-4681-b27b-fb7529461620", 2024-04-21T08:54:01.072 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:54:01.072 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:54:01.073 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:54:01.073 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:54:01.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.077+0000 7f13d3575640 1 -- 172.21.15.7:0/295865903 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f13b80a3de0 msgr2=0x7f13b80a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:01.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.077+0000 7f13d3575640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f13b80a3de0 0x7f13b80a62a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55d0268c6050 tx=0x7f13c800a0d0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.078+0000 7f13d3575640 1 -- 172.21.15.7:0/295865903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc1106a0 msgr2=0x7f13cc0803d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:01.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.078+0000 7f13d3575640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc1106a0 0x7f13cc0803d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f13c404f990 tx=0x7f13c404fe50 comp rx=0 tx=0).stop 2024-04-21T08:54:01.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.078+0000 7f13d3575640 1 -- 172.21.15.7:0/295865903 shutdown_connections 2024-04-21T08:54:01.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.078+0000 7f13d3575640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f13b80a3de0 0x7f13b80a62a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.078+0000 7f13d3575640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13cc110b10 0x7f13cc07afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.078+0000 7f13d3575640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc1106a0 0x7f13cc0803d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.078+0000 7f13d3575640 1 --2- 172.21.15.7:0/295865903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13cc10e1b0 0x7f13cc07fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:01.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.078+0000 7f13d3575640 1 -- 172.21.15.7:0/295865903 >> 172.21.15.7:0/295865903 conn(0x7f13cc06e360 msgr2=0x7f13cc10a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:01.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.079+0000 7f13d3575640 1 -- 172.21.15.7:0/295865903 shutdown_connections 2024-04-21T08:54:01.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:01.079+0000 7f13d3575640 1 -- 172.21.15.7:0/295865903 wait complete. 2024-04-21T08:54:01.086 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/proc_tty.o 2024-04-21T08:54:01.109 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:00 smithi007 ceph-mon[28477]: pgmap v291: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 660 KiB/s wr, 2.39k op/s 2024-04-21T08:54:01.110 INFO:tasks.cephfs.filesystem:scrub status for tag:2f44a19d-b0a7-4681-b27b-fb7529461620 - {'path': '/', 'tag': '2f44a19d-b0a7-4681-b27b-fb7529461620', 'options': 'recursive,force'} 2024-04-21T08:54:01.167 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/sched/built-in.a 2024-04-21T08:54:01.257 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/locking/mutex.o 2024-04-21T08:54:01.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:00 smithi016 ceph-mon[31259]: pgmap v291: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 660 KiB/s wr, 2.39k op/s 2024-04-21T08:54:01.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:00 smithi077 ceph-mon[31615]: pgmap v291: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 660 KiB/s wr, 2.39k op/s 2024-04-21T08:54:01.470 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/netfs/io.o 2024-04-21T08:54:01.560 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/cmdline.o 2024-04-21T08:54:01.767 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/bitmap.o 2024-04-21T08:54:01.980 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/p6.o 2024-04-21T08:54:02.220 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/consoles.o 2024-04-21T08:54:02.438 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/netfs/iterator.o 2024-04-21T08:54:02.448 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/locking/semaphore.o 2024-04-21T08:54:02.470 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/zhaoxin/core.o 2024-04-21T08:54:02.487 INFO:tasks.workunit.client.0.smithi007.stdout: CC ipc/namespace.o 2024-04-21T08:54:02.538 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/block_validity.o 2024-04-21T08:54:02.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:02 smithi016 ceph-mon[31259]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T08:54:02.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:02 smithi016 ceph-mon[31259]: pgmap v293: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.1 MiB/s wr, 2.75k op/s 2024-04-21T08:54:02.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:02 smithi077 ceph-mon[31615]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T08:54:02.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:02 smithi077 ceph-mon[31615]: pgmap v293: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.1 MiB/s wr, 2.75k op/s 2024-04-21T08:54:02.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:02 smithi007 ceph-mon[28477]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T08:54:02.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:02 smithi007 ceph-mon[28477]: pgmap v293: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.1 MiB/s wr, 2.75k op/s 2024-04-21T08:54:02.655 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/pt.o 2024-04-21T08:54:02.747 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/cpuinfo.o 2024-04-21T08:54:03.011 INFO:tasks.workunit.client.0.smithi007.stdout: CC ipc/mq_sysctl.o 2024-04-21T08:54:03.049 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/netfs/main.o 2024-04-21T08:54:03.051 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/locking/rwsem.o 2024-04-21T08:54:03.207 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/events/zhaoxin/built-in.a 2024-04-21T08:54:03.236 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/devices.o 2024-04-21T08:54:03.272 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/core.o 2024-04-21T08:54:03.286 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/dir.o 2024-04-21T08:54:03.300 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/shmem.o 2024-04-21T08:54:03.371 INFO:tasks.workunit.client.0.smithi007.stdout: AR ipc/built-in.a 2024-04-21T08:54:03.406 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/ext4_jbd2.o 2024-04-21T08:54:03.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:03 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:03.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:03 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:03.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:03 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:03.686 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/uncore.o 2024-04-21T08:54:03.758 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/locking/percpu-rwsem.o 2024-04-21T08:54:03.775 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/interrupts.o 2024-04-21T08:54:03.881 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/netfs/objects.o 2024-04-21T08:54:04.118 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/extents.o 2024-04-21T08:54:04.226 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/loadavg.o 2024-04-21T08:54:04.337 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/locking/spinlock.o 2024-04-21T08:54:04.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:04 smithi016 ceph-mon[31259]: pgmap v294: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.0 MiB/s wr, 2.97k op/s 2024-04-21T08:54:04.595 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/extents_status.o 2024-04-21T08:54:04.598 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/netfs/built-in.a 2024-04-21T08:54:04.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:04 smithi077 ceph-mon[31615]: pgmap v294: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.0 MiB/s wr, 2.97k op/s 2024-04-21T08:54:04.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:04 smithi007 ceph-mon[28477]: pgmap v294: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.0 MiB/s wr, 2.97k op/s 2024-04-21T08:54:04.661 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/jbd2/transaction.o 2024-04-21T08:54:04.769 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/locking/osq_lock.o 2024-04-21T08:54:04.771 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/meminfo.o 2024-04-21T08:54:04.809 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/jbd2/commit.o 2024-04-21T08:54:04.945 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/uncore_nhmex.o 2024-04-21T08:54:05.039 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/locking/qspinlock.o 2024-04-21T08:54:05.045 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/util.o 2024-04-21T08:54:05.418 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/stat.o 2024-04-21T08:54:05.473 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/locking/rtmutex_api.o 2024-04-21T08:54:05.838 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/locking/qrwlock.o 2024-04-21T08:54:05.857 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/jbd2/recovery.o 2024-04-21T08:54:05.872 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/uncore_snb.o 2024-04-21T08:54:05.958 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mmzone.o 2024-04-21T08:54:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.003+0000 7f005d63f640 1 -- 172.21.15.7:0/336668711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058110550 msgr2=0x7f0058110930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.003+0000 7f005d63f640 1 --2- 172.21.15.7:0/336668711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058110550 0x7f0058110930 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0050009ca0 tx=0x7f005002c180 comp rx=0 tx=0).stop 2024-04-21T08:54:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.004+0000 7f005d63f640 1 -- 172.21.15.7:0/336668711 shutdown_connections 2024-04-21T08:54:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.004+0000 7f005d63f640 1 --2- 172.21.15.7:0/336668711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0058109b00 0x7f005811d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.004+0000 7f005d63f640 1 --2- 172.21.15.7:0/336668711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0058108f50 0x7f00581093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.004+0000 7f005d63f640 1 --2- 172.21.15.7:0/336668711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058110550 0x7f0058110930 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.004+0000 7f005d63f640 1 -- 172.21.15.7:0/336668711 >> 172.21.15.7:0/336668711 conn(0x7f005806cea0 msgr2=0x7f005806d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.004+0000 7f005d63f640 1 -- 172.21.15.7:0/336668711 shutdown_connections 2024-04-21T08:54:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.004+0000 7f005d63f640 1 -- 172.21.15.7:0/336668711 wait complete. 2024-04-21T08:54:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.005+0000 7f005d63f640 1 Processor -- start 2024-04-21T08:54:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.005+0000 7f005d63f640 1 -- start start 2024-04-21T08:54:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.005+0000 7f005d63f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0058108f50 0x7f00581a7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.005+0000 7f005d63f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0058109b00 0x7f00581a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.005+0000 7f005d63f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058110550 0x7f00581abc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.005+0000 7f005d63f640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0058120eb0 con 0x7f0058110550 2024-04-21T08:54:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.005+0000 7f005d63f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0058120d30 con 0x7f0058109b00 2024-04-21T08:54:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.005+0000 7f005d63f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0058121030 con 0x7f0058108f50 2024-04-21T08:54:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.006+0000 7f00577fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058110550 0x7f00581abc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.006+0000 7f00577fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058110550 0x7f00581abc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34878/0 (socket says 172.21.15.7:34878) 2024-04-21T08:54:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.006+0000 7f00577fe640 1 -- 172.21.15.7:0/627254866 learned_addr learned my addr 172.21.15.7:0/627254866 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.006+0000 7f0056ffd640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0058108f50 0x7f00581a7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.006+0000 7f00567fc640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0058109b00 0x7f00581a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.006+0000 7f0056ffd640 1 -- 172.21.15.7:0/627254866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0058109b00 msgr2=0x7f00581a7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.006+0000 7f0056ffd640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0058109b00 0x7f00581a7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.006+0000 7f0056ffd640 1 -- 172.21.15.7:0/627254866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058110550 msgr2=0x7f00581abc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.006+0000 7f0056ffd640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058110550 0x7f00581abc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.006+0000 7f0056ffd640 1 -- 172.21.15.7:0/627254866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00581ac390 con 0x7f0058108f50 2024-04-21T08:54:06.007 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/uptime.o 2024-04-21T08:54:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.006+0000 7f0056ffd640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0058108f50 0x7f00581a7350 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0050037040 tx=0x7f0050003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.015+0000 7f003ffff640 1 -- 172.21.15.7:0/627254866 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050041020 con 0x7f0058108f50 2024-04-21T08:54:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.016+0000 7f003ffff640 1 -- 172.21.15.7:0/627254866 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00500044f0 con 0x7f0058108f50 2024-04-21T08:54:06.015 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.016+0000 7f005d63f640 1 -- 172.21.15.7:0/627254866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00581ac620 con 0x7f0058108f50 2024-04-21T08:54:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.016+0000 7f005d63f640 1 -- 172.21.15.7:0/627254866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00581b3ec0 con 0x7f0058108f50 2024-04-21T08:54:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.016+0000 7f003ffff640 1 -- 172.21.15.7:0/627254866 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050039040 con 0x7f0058108f50 2024-04-21T08:54:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.019+0000 7f006022e740 1 Processor -- start 2024-04-21T08:54:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.019+0000 7f006022e740 1 -- start start 2024-04-21T08:54:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.019+0000 7f003ffff640 1 -- 172.21.15.7:0/627254866 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0050044020 con 0x7f0058108f50 2024-04-21T08:54:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f006022e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 0x562d3257c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f006022e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 0x562d3257d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f006022e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d325862d0 0x562d325886c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f006022e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562d324e18d0 con 0x562d3257cd50 2024-04-21T08:54:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f006022e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562d3247ee80 con 0x562d325862d0 2024-04-21T08:54:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f006022e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562d324dd890 con 0x562d3257dac0 2024-04-21T08:54:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f00567fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d325862d0 0x562d325886c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f00567fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d325862d0 0x562d325886c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:55908/0 (socket says 172.21.15.7:55908) 2024-04-21T08:54:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f00567fc640 1 -- 172.21.15.7:0/3264226569 learned_addr learned my addr 172.21.15.7:0/3264226569 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f0056ffd640 1 --2- 172.21.15.7:0/3264226569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 0x562d3257d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f00577fe640 1 --2- 172.21.15.7:0/3264226569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 0x562d3257c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f00567fc640 1 -- 172.21.15.7:0/3264226569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 msgr2=0x562d3257c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f00567fc640 1 --2- 172.21.15.7:0/3264226569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 0x562d3257c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.020+0000 7f00567fc640 1 -- 172.21.15.7:0/3264226569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 msgr2=0x562d3257d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.021+0000 7f003ffff640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f00400a3eb0 0x7f00400a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.021+0000 7f00567fc640 1 --2- 172.21.15.7:0/3264226569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 0x562d3257d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.021+0000 7f00567fc640 1 -- 172.21.15.7:0/3264226569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d324a0320 con 0x562d325862d0 2024-04-21T08:54:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.021+0000 7f0056ffd640 1 --2- 172.21.15.7:0/3264226569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 0x562d3257d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.021+0000 7f00577fe640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f00400a3eb0 0x7f00400a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.021+0000 7f003ffff640 1 -- 172.21.15.7:0/627254866 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9596+0+0 (secure 0 0 0) 0x7f00500f2860 con 0x7f0058108f50 2024-04-21T08:54:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.021+0000 7f00567fc640 1 --2- 172.21.15.7:0/3264226569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d325862d0 0x562d325886c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f004800bc90 tx=0x7f004800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=794d917cc1e0c9ce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.022+0000 7f00577fe640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f00400a3eb0 0x7f00400a6370 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f004c00bdb0 tx=0x7f004c00b390 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.033+0000 7f003d7fa640 1 -- 172.21.15.7:0/3264226569 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0048015070 con 0x562d325862d0 2024-04-21T08:54:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.033+0000 7f003d7fa640 1 -- 172.21.15.7:0/3264226569 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00480118f0 con 0x562d325862d0 2024-04-21T08:54:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.033+0000 7f003d7fa640 1 -- 172.21.15.7:0/3264226569 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0048010820 con 0x562d325862d0 2024-04-21T08:54:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.034+0000 7f006022e740 1 -- 172.21.15.7:0/3264226569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d325862d0 msgr2=0x562d325886c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.034+0000 7f006022e740 1 --2- 172.21.15.7:0/3264226569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d325862d0 0x562d325886c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f004800bc90 tx=0x7f004800f730 comp rx=0 tx=0).stop 2024-04-21T08:54:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.034+0000 7f006022e740 1 -- 172.21.15.7:0/3264226569 shutdown_connections 2024-04-21T08:54:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.034+0000 7f006022e740 1 --2- 172.21.15.7:0/3264226569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d325862d0 0x562d325886c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.034+0000 7f006022e740 1 --2- 172.21.15.7:0/3264226569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 0x562d3257d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.034+0000 7f006022e740 1 --2- 172.21.15.7:0/3264226569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 0x562d3257c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.034+0000 7f006022e740 1 -- 172.21.15.7:0/3264226569 >> 172.21.15.7:0/3264226569 conn(0x562d32543c90 msgr2=0x562d32577a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.034+0000 7f006022e740 1 -- 172.21.15.7:0/3264226569 shutdown_connections 2024-04-21T08:54:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.034+0000 7f006022e740 1 -- 172.21.15.7:0/3264226569 wait complete. 2024-04-21T08:54:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.034+0000 7f006022e740 1 -- start start 2024-04-21T08:54:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.034+0000 7f006022e740 1 Processor -- start 2024-04-21T08:54:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f006022e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257cd50 0x562d32585c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f006022e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 0x562d32582f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f006022e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d325862d0 0x562d325834d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f006022e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x562d32589000 con 0x562d325862d0 2024-04-21T08:54:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f006022e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x562d32469eb0 con 0x562d3257cd50 2024-04-21T08:54:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f006022e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562d32589180 con 0x562d3257dac0 2024-04-21T08:54:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f006022e740 10 client.0 _ll_register_callbacks cb 0x562d32496b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f00567fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 0x562d32582f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f00567fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 0x562d32582f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:38298/0 (socket says 172.21.15.7:38298) 2024-04-21T08:54:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f00567fc640 1 -- 172.21.15.7:0/2930051553 learned_addr learned my addr 172.21.15.7:0/2930051553 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f0056ffd640 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257cd50 0x562d32585c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f00577fe640 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d325862d0 0x562d325834d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.035+0000 7f00567fc640 1 -- 172.21.15.7:0/2930051553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257cd50 msgr2=0x562d32585c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.036+0000 7f00567fc640 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257cd50 0x562d32585c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.036+0000 7f00567fc640 1 -- 172.21.15.7:0/2930051553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d325862d0 msgr2=0x562d325834d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.036+0000 7f00567fc640 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d325862d0 0x562d325834d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.036+0000 7f00567fc640 1 -- 172.21.15.7:0/2930051553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d321f4210 con 0x562d3257dac0 2024-04-21T08:54:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.036+0000 7f00577fe640 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d325862d0 0x562d325834d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.036+0000 7f00567fc640 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 0x562d32582f90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f004800bc60 tx=0x7f0048005fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80cb5685480f1f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.050+0000 7f003aff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:54:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.050+0000 7f003aff5640 1 -- 172.21.15.7:0/2930051553 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0048015040 con 0x562d3257dac0 2024-04-21T08:54:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.050+0000 7f003aff5640 1 -- 172.21.15.7:0/2930051553 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9596+0+0 (secure 0 0 0) 0x7f0048011aa0 con 0x562d3257dac0 2024-04-21T08:54:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.050+0000 7f006022e740 1 -- 172.21.15.7:0/2930051553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00480118f0 con 0x562d3257dac0 2024-04-21T08:54:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.050+0000 7f003aff5640 1 -- 172.21.15.7:0/2930051553 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0048030050 con 0x562d3257dac0 2024-04-21T08:54:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.050+0000 7f006022e740 10 client.26710 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.050+0000 7f006022e740 1 -- 172.21.15.7:0/2930051553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562d322c0620 con 0x562d3257dac0 2024-04-21T08:54:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.050+0000 7f003aff5640 1 -- 172.21.15.7:0/2930051553 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00480118f0 con 0x562d3257dac0 2024-04-21T08:54:06.051 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.051+0000 7f003aff5640 1 -- 172.21.15.7:0/2930051553 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f004802b030 con 0x562d3257dac0 2024-04-21T08:54:06.051 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.051+0000 7f006022e740 10 client.26710 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:06.051 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.051+0000 7f006022e740 10 client.26710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:06.051 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.051+0000 7f006022e740 20 client.26710 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:06.051 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.051+0000 7f006022e740 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562d325961d0 0x562d325985c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.051 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.051+0000 7f006022e740 4 client.26710 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:06.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.051+0000 7f006022e740 1 -- 172.21.15.7:0/2930051553 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d324f5880 con 0x562d325961d0 2024-04-21T08:54:06.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.052+0000 7f0056ffd640 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562d325961d0 0x562d325985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.052+0000 7f003aff5640 10 client.26710 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:06.052 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.052+0000 7f0056ffd640 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562d325961d0 0x562d325985c0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48d42ef9a16091ec server_cookie=2874c0bb7060557f in_seq=0 out_seq=0 2024-04-21T08:54:06.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f003aff5640 1 -- 172.21.15.7:0/2930051553 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562d324f5880 con 0x562d325961d0 2024-04-21T08:54:06.053 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f003aff5640 10 client.26710 handle_command_reply: tid=0 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f006022e740 1 client.26710 shutdown 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f006022e740 1 -- 172.21.15.7:0/2930051553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 msgr2=0x562d32582f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f006022e740 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 0x562d32582f90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f004800bc60 tx=0x7f0048005fc0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f006022e740 1 -- 172.21.15.7:0/2930051553 shutdown_connections 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f006022e740 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562d325961d0 0x562d325985c0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f006022e740 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d325862d0 0x562d325834d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f006022e740 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d3257dac0 0x562d32582f90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f003aff5640 0 client.26710 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f006022e740 1 --2- 172.21.15.7:0/2930051553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257cd50 0x562d32585c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.053+0000 7f006022e740 1 -- 172.21.15.7:0/2930051553 >> 172.21.15.7:0/2930051553 conn(0x562d32543c90 msgr2=0x562d32579790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.054+0000 7f006022e740 1 -- 172.21.15.7:0/2930051553 shutdown_connections 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.054+0000 7f006022e740 1 -- 172.21.15.7:0/2930051553 wait complete. 2024-04-21T08:54:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.054+0000 7f006022e740 20 client.26710 trim_cache size 0 max 16384 2024-04-21T08:54:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.110+0000 7f006022e740 1 Processor -- start 2024-04-21T08:54:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.111+0000 7f006022e740 1 -- start start 2024-04-21T08:54:06.111 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.111+0000 7f006022e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 0x562d32591f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.111+0000 7f006022e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257dac0 0x562d325856c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.111+0000 7f006022e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 0x562d32585c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.111+0000 7f006022e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f004400e230 con 0x562d3257cd50 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.111+0000 7f006022e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0044010ab0 con 0x562d3257dac0 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.111+0000 7f006022e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f004400e8a0 con 0x562d325862d0 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.111+0000 7f0056ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 0x562d32591f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.111+0000 7f0056ffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 0x562d32591f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34914/0 (socket says 172.21.15.7:34914) 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.111+0000 7f0056ffd640 1 -- 172.21.15.7:0/3571130706 learned_addr learned my addr 172.21.15.7:0/3571130706 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.111+0000 7f00577fe640 1 --2- 172.21.15.7:0/3571130706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 0x562d32585c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.112+0000 7f00567fc640 1 --2- 172.21.15.7:0/3571130706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257dac0 0x562d325856c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.112+0000 7f00577fe640 1 -- 172.21.15.7:0/3571130706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257dac0 msgr2=0x562d325856c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.112+0000 7f00577fe640 1 --2- 172.21.15.7:0/3571130706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257dac0 0x562d325856c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.112+0000 7f00577fe640 1 -- 172.21.15.7:0/3571130706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 msgr2=0x562d32591f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.112+0000 7f00577fe640 1 --2- 172.21.15.7:0/3571130706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 0x562d32591f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.112+0000 7f00577fe640 1 -- 172.21.15.7:0/3571130706 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d32577be0 con 0x562d325862d0 2024-04-21T08:54:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.112+0000 7f0056ffd640 1 --2- 172.21.15.7:0/3571130706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 0x562d32591f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.112+0000 7f00577fe640 1 --2- 172.21.15.7:0/3571130706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 0x562d32585c00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x562d325862a0 tx=0x7f004c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c5525ac42e48814 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:06.118 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.118+0000 7f003cff9640 1 -- 172.21.15.7:0/3571130706 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c02c070 con 0x562d325862d0 2024-04-21T08:54:06.118 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.118+0000 7f003cff9640 1 -- 172.21.15.7:0/3571130706 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f004c01e0d0 con 0x562d325862d0 2024-04-21T08:54:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.119+0000 7f006022e740 1 -- 172.21.15.7:0/3571130706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 msgr2=0x562d32585c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.119+0000 7f006022e740 1 --2- 172.21.15.7:0/3571130706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 0x562d32585c00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x562d325862a0 tx=0x7f004c016000 comp rx=0 tx=0).stop 2024-04-21T08:54:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.119+0000 7f006022e740 1 -- 172.21.15.7:0/3571130706 shutdown_connections 2024-04-21T08:54:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.119+0000 7f006022e740 1 --2- 172.21.15.7:0/3571130706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 0x562d32585c00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.119+0000 7f006022e740 1 --2- 172.21.15.7:0/3571130706 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257dac0 0x562d325856c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.119+0000 7f006022e740 1 --2- 172.21.15.7:0/3571130706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257cd50 0x562d32591f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.119+0000 7f006022e740 1 -- 172.21.15.7:0/3571130706 >> 172.21.15.7:0/3571130706 conn(0x562d32543c90 msgr2=0x562d32582f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.126+0000 7f006022e740 1 -- 172.21.15.7:0/3571130706 shutdown_connections 2024-04-21T08:54:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.126+0000 7f006022e740 1 -- 172.21.15.7:0/3571130706 wait complete. 2024-04-21T08:54:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.127+0000 7f006022e740 1 -- start start 2024-04-21T08:54:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.127+0000 7f006022e740 1 Processor -- start 2024-04-21T08:54:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.127+0000 7f006022e740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257cd50 0x562d325f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.127+0000 7f006022e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257dac0 0x562d3258e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.127+0000 7f006022e740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 0x562d3258e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.127+0000 7f006022e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f004400ef70 con 0x562d3257dac0 2024-04-21T08:54:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.127+0000 7f006022e740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00440103e0 con 0x562d3257cd50 2024-04-21T08:54:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.127+0000 7f006022e740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f004400f640 con 0x562d325862d0 2024-04-21T08:54:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.127+0000 7f006022e740 10 client.0 _ll_register_callbacks cb 0x7f00500f8420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.128+0000 7f00567fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257dac0 0x562d3258e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.128+0000 7f00577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 0x562d3258e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.128+0000 7f0056ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257cd50 0x562d325f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.128+0000 7f00577fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 0x562d3258e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:38314/0 (socket says 172.21.15.7:38314) 2024-04-21T08:54:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.128+0000 7f00577fe640 1 -- 172.21.15.7:0/3094288815 learned_addr learned my addr 172.21.15.7:0/3094288815 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.128+0000 7f00577fe640 1 -- 172.21.15.7:0/3094288815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257cd50 msgr2=0x562d325f7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.128+0000 7f00577fe640 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257cd50 0x562d325f7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.128+0000 7f00577fe640 1 -- 172.21.15.7:0/3094288815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257dac0 msgr2=0x562d3258e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.128+0000 7f00577fe640 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257dac0 0x562d3258e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.128+0000 7f00577fe640 1 -- 172.21.15.7:0/3094288815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d3257fea0 con 0x562d325862d0 2024-04-21T08:54:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.128+0000 7f0056ffd640 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257cd50 0x562d325f7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.129+0000 7f00567fc640 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257dac0 0x562d3258e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.129+0000 7f00577fe640 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 0x562d3258e9b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f004c01e040 tx=0x7f004c005ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffcb5e7a0225d76e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.129+0000 7f003aff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:54:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.129+0000 7f003aff5640 1 -- 172.21.15.7:0/3094288815 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c02c040 con 0x562d325862d0 2024-04-21T08:54:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.129+0000 7f003aff5640 1 -- 172.21.15.7:0/3094288815 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9596+0+0 (secure 0 0 0) 0x7f004c01e150 con 0x562d325862d0 2024-04-21T08:54:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.130+0000 7f006022e740 1 -- 172.21.15.7:0/3094288815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562d324813d0 con 0x562d325862d0 2024-04-21T08:54:06.131 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.130+0000 7f003aff5640 1 -- 172.21.15.7:0/3094288815 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f004c01c620 con 0x562d325862d0 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.130+0000 7f006022e740 10 client.26716 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.130+0000 7f006022e740 1 -- 172.21.15.7:0/3094288815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562d3260d980 con 0x562d325862d0 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.130+0000 7f003aff5640 1 -- 172.21.15.7:0/3094288815 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562d324813d0 con 0x562d325862d0 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.131+0000 7f003aff5640 1 -- 172.21.15.7:0/3094288815 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f004c03a030 con 0x562d325862d0 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.131+0000 7f006022e740 10 client.26716 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.131+0000 7f006022e740 10 client.26716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.131+0000 7f006022e740 20 client.26716 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.131+0000 7f006022e740 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562d3260dff0 0x562d32610410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.131+0000 7f006022e740 4 client.26716 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.131+0000 7f006022e740 1 -- 172.21.15.7:0/3094288815 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562d324ef2f0 con 0x562d3260dff0 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.131+0000 7f0056ffd640 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562d3260dff0 0x562d32610410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.132+0000 7f0056ffd640 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562d3260dff0 0x562d32610410 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb5be592cf0db4ee server_cookie=ccb2dc48b27028ea in_seq=0 out_seq=0 2024-04-21T08:54:06.133 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:06.132+0000 7f003aff5640 10 client.26716 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:06.150 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/jbd2/checkpoint.o 2024-04-21T08:54:06.288 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/jbd2/revoke.o 2024-04-21T08:54:06.345 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/locking/built-in.a 2024-04-21T08:54:06.503 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/util.o 2024-04-21T08:54:06.507 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/power/qos.o 2024-04-21T08:54:06.558 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/vmstat.o 2024-04-21T08:54:06.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:06 smithi016 ceph-mon[31259]: pgmap v295: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.1 MiB/s wr, 2.94k op/s 2024-04-21T08:54:06.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:06 smithi077 ceph-mon[31615]: pgmap v295: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.1 MiB/s wr, 2.94k op/s 2024-04-21T08:54:06.906 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:06 smithi007 ceph-mon[28477]: pgmap v295: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.1 MiB/s wr, 2.94k op/s 2024-04-21T08:54:07.106 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/uncore_snbep.o 2024-04-21T08:54:07.171 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/jbd2/journal.o 2024-04-21T08:54:07.397 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/file.o 2024-04-21T08:54:07.457 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/version.o 2024-04-21T08:54:07.481 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/realmode/init.o 2024-04-21T08:54:07.739 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/uncore_discovery.o 2024-04-21T08:54:07.960 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/power/main.o 2024-04-21T08:54:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.989+0000 7f003aff5640 1 -- 172.21.15.7:0/3094288815 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562d324ef2f0 con 0x562d3260dff0 2024-04-21T08:54:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.989+0000 7f003aff5640 10 client.26716 handle_command_reply: tid=0 2024-04-21T08:54:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.989+0000 7f006022e740 1 client.26716 shutdown 2024-04-21T08:54:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.993+0000 7f006022e740 1 -- 172.21.15.7:0/3094288815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 msgr2=0x562d3258e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.993+0000 7f006022e740 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 0x562d3258e9b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f004c01e040 tx=0x7f004c005ee0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.994+0000 7f006022e740 1 -- 172.21.15.7:0/3094288815 shutdown_connections 2024-04-21T08:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.994+0000 7f006022e740 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x562d3260dff0 0x562d32610410 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.994+0000 7f006022e740 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562d325862d0 0x562d3258e9b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.994+0000 7f006022e740 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x562d3257dac0 0x562d3258e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.994+0000 7f006022e740 1 --2- 172.21.15.7:0/3094288815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x562d3257cd50 0x562d325f7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.994+0000 7f006022e740 1 -- 172.21.15.7:0/3094288815 >> 172.21.15.7:0/3094288815 conn(0x562d32543c90 msgr2=0x562d3257e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.994+0000 7f003aff5640 0 client.26716 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.998+0000 7f006022e740 1 -- 172.21.15.7:0/3094288815 shutdown_connections 2024-04-21T08:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.998+0000 7f006022e740 1 -- 172.21.15.7:0/3094288815 wait complete. 2024-04-21T08:54:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:07.998+0000 7f006022e740 20 client.26716 trim_cache size 0 max 16384 2024-04-21T08:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:54:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:54:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.021+0000 7f005d63f640 1 -- 172.21.15.7:0/627254866 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f00400a3eb0 msgr2=0x7f00400a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.021+0000 7f005d63f640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f00400a3eb0 0x7f00400a6370 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f004c00bdb0 tx=0x7f004c00b390 comp rx=0 tx=0).stop 2024-04-21T08:54:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.021+0000 7f005d63f640 1 -- 172.21.15.7:0/627254866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0058108f50 msgr2=0x7f00581a7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.021+0000 7f005d63f640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0058108f50 0x7f00581a7350 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0050037040 tx=0x7f0050003c10 comp rx=0 tx=0).stop 2024-04-21T08:54:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.022+0000 7f005d63f640 1 -- 172.21.15.7:0/627254866 shutdown_connections 2024-04-21T08:54:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.022+0000 7f005d63f640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058110550 0x7f00581abc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.022+0000 7f005d63f640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0058109b00 0x7f00581a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.022+0000 7f005d63f640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f00400a3eb0 0x7f00400a6370 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.022+0000 7f005d63f640 1 --2- 172.21.15.7:0/627254866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0058108f50 0x7f00581a7350 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0050037040 tx=0x7f0050003c10 comp rx=0 tx=0).stop 2024-04-21T08:54:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.022+0000 7f005d63f640 1 -- 172.21.15.7:0/627254866 >> 172.21.15.7:0/627254866 conn(0x7f005806cea0 msgr2=0x7f0058070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.024+0000 7f005d63f640 1 -- 172.21.15.7:0/627254866 shutdown_connections 2024-04-21T08:54:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:08.024+0000 7f005d63f640 1 -- 172.21.15.7:0/627254866 wait complete. 2024-04-21T08:54:08.400 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/softirqs.o 2024-04-21T08:54:08.489 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/realmode/rm/header.o 2024-04-21T08:54:08.501 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/backing-dev.o 2024-04-21T08:54:08.636 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/realmode/rm/trampoline_64.o 2024-04-21T08:54:08.776 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/realmode/rm/stack.o 2024-04-21T08:54:08.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:08 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:54:08.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:08 smithi016 ceph-mon[31259]: pgmap v296: 129 pgs: 129 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 2.37k op/s 2024-04-21T08:54:08.806 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:08 smithi016 ceph-mon[31259]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T08:54:08.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:08 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:54:08.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:08 smithi077 ceph-mon[31615]: pgmap v296: 129 pgs: 129 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 2.37k op/s 2024-04-21T08:54:08.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:08 smithi077 ceph-mon[31615]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T08:54:08.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:08 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:54:08.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:08 smithi007 ceph-mon[28477]: pgmap v296: 129 pgs: 129 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 2.37k op/s 2024-04-21T08:54:08.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:08 smithi007 ceph-mon[28477]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T08:54:08.907 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/realmode/rm/reboot.o 2024-04-21T08:54:08.997 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/realmode/rm/wakeup_asm.o 2024-04-21T08:54:09.009 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/namespaces.o 2024-04-21T08:54:09.070 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/realmode/rm/wakemain.o 2024-04-21T08:54:09.113 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/fsmap.o 2024-04-21T08:54:09.279 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/realmode/rm/video-mode.o 2024-04-21T08:54:09.338 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/self.o 2024-04-21T08:54:09.512 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/intel/cstate.o 2024-04-21T08:54:09.624 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/power/console.o 2024-04-21T08:54:09.652 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/realmode/rm/copy.o 2024-04-21T08:54:09.772 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/realmode/rm/bioscall.o 2024-04-21T08:54:09.882 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/realmode/rm/regs.o 2024-04-21T08:54:10.012 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/realmode/rm/video-vga.o 2024-04-21T08:54:10.204 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/realmode/rm/video-vesa.o 2024-04-21T08:54:10.221 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/thread_self.o 2024-04-21T08:54:10.229 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/printk/printk.o 2024-04-21T08:54:10.296 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mm_init.o 2024-04-21T08:54:10.371 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/realmode/rm/video-bios.o 2024-04-21T08:54:10.501 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/events/intel/built-in.a 2024-04-21T08:54:10.539 INFO:tasks.workunit.client.0.smithi007.stdout: PASYMS arch/x86/realmode/rm/pasyms.h 2024-04-21T08:54:10.621 INFO:tasks.workunit.client.0.smithi007.stdout: LDS arch/x86/realmode/rm/realmode.lds 2024-04-21T08:54:10.637 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/power/process.o 2024-04-21T08:54:10.697 INFO:tasks.workunit.client.0.smithi007.stdout: LD arch/x86/realmode/rm/realmode.elf 2024-04-21T08:54:10.742 INFO:tasks.workunit.client.0.smithi007.stdout: RELOCS arch/x86/realmode/rm/realmode.relocs 2024-04-21T08:54:10.758 INFO:tasks.workunit.client.0.smithi007.stdout: OBJCOPY arch/x86/realmode/rm/realmode.bin 2024-04-21T08:54:10.761 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/probe.o 2024-04-21T08:54:10.793 INFO:tasks.workunit.client.0.smithi007.stdout: AS arch/x86/realmode/rmpiggy.o 2024-04-21T08:54:10.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:10 smithi016 ceph-mon[31259]: pgmap v298: 129 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 106 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 918 KiB/s wr, 1.66k op/s 2024-04-21T08:54:10.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:10 smithi077 ceph-mon[31615]: pgmap v298: 129 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 106 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 918 KiB/s wr, 1.66k op/s 2024-04-21T08:54:10.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:10 smithi007 ceph-mon[28477]: pgmap v298: 129 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 106 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 918 KiB/s wr, 1.66k op/s 2024-04-21T08:54:10.992 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/realmode/built-in.a 2024-04-21T08:54:11.096 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/proc_sysctl.o 2024-04-21T08:54:11.097 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/jbd2/built-in.a 2024-04-21T08:54:11.113 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/power/suspend.o 2024-04-21T08:54:11.124 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/fsync.o 2024-04-21T08:54:11.174 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/percpu.o 2024-04-21T08:54:11.273 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/utils.o 2024-04-21T08:54:12.160 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/power/hibernate.o 2024-04-21T08:54:12.264 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/irqdesc.o 2024-04-21T08:54:12.427 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/rapl.o 2024-04-21T08:54:12.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:12 smithi016 ceph-mon[31259]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T08:54:12.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:12 smithi016 ceph-mon[31259]: pgmap v300: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 525 KiB/s wr, 754 op/s 2024-04-21T08:54:12.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:12 smithi077 ceph-mon[31615]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T08:54:12.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:12 smithi077 ceph-mon[31615]: pgmap v300: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 525 KiB/s wr, 754 op/s 2024-04-21T08:54:12.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:12 smithi007 ceph-mon[28477]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T08:54:12.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:12 smithi007 ceph-mon[28477]: pgmap v300: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 1.4 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 525 KiB/s wr, 754 op/s 2024-04-21T08:54:12.875 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/handle.o 2024-04-21T08:54:13.051 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/printk/printk_safe.o 2024-04-21T08:54:13.310 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/hash.o 2024-04-21T08:54:13.337 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/proc_net.o 2024-04-21T08:54:13.486 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/manage.o 2024-04-21T08:54:13.879 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/spurious.o 2024-04-21T08:54:13.942 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/printk/printk_ringbuffer.o 2024-04-21T08:54:13.947 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/events/msr.o 2024-04-21T08:54:14.212 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/power/snapshot.o 2024-04-21T08:54:14.378 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/slab_common.o 2024-04-21T08:54:14.680 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/ialloc.o 2024-04-21T08:54:14.713 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/kcore.o 2024-04-21T08:54:14.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:14 smithi016 ceph-mon[31259]: pgmap v301: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 591 KiB/s wr, 591 op/s 2024-04-21T08:54:14.822 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/printk/sysctl.o 2024-04-21T08:54:14.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:14 smithi077 ceph-mon[31615]: pgmap v301: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 591 KiB/s wr, 591 op/s 2024-04-21T08:54:14.893 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/vmcore.o 2024-04-21T08:54:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:14 smithi007 ceph-mon[28477]: pgmap v301: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 591 KiB/s wr, 591 op/s 2024-04-21T08:54:15.272 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/printk/built-in.a 2024-04-21T08:54:15.456 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/events/built-in.a 2024-04-21T08:54:15.579 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/rcu/update.o 2024-04-21T08:54:15.652 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/fpu/init.o 2024-04-21T08:54:15.873 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/kmsg.o 2024-04-21T08:54:16.041 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/resend.o 2024-04-21T08:54:16.384 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc/page.o 2024-04-21T08:54:16.423 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/fpu/bugs.o 2024-04-21T08:54:16.612 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/power/swap.o 2024-04-21T08:54:16.769 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/fpu/core.o 2024-04-21T08:54:16.774 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/chip.o 2024-04-21T08:54:16.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:16 smithi016 ceph-mon[31259]: pgmap v302: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 353 KiB/s wr, 200 op/s 2024-04-21T08:54:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:16 smithi077 ceph-mon[31615]: pgmap v302: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 353 KiB/s wr, 200 op/s 2024-04-21T08:54:16.867 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/fpu/regset.o 2024-04-21T08:54:16.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:16 smithi007 ceph-mon[28477]: pgmap v302: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 353 KiB/s wr, 200 op/s 2024-04-21T08:54:17.010 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/compaction.o 2024-04-21T08:54:17.409 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/indirect.o 2024-04-21T08:54:17.520 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/proc/built-in.a 2024-04-21T08:54:18.034 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/show_mem.o 2024-04-21T08:54:18.165 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ramfs/inode.o 2024-04-21T08:54:18.211 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/dummychip.o 2024-04-21T08:54:18.230 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/power/user.o 2024-04-21T08:54:18.620 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/fpu/signal.o 2024-04-21T08:54:18.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:18 smithi016 ceph-mon[31259]: pgmap v303: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 744 KiB/s wr, 164 op/s 2024-04-21T08:54:18.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:18 smithi077 ceph-mon[31615]: pgmap v303: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 744 KiB/s wr, 164 op/s 2024-04-21T08:54:18.900 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/rcu/sync.o 2024-04-21T08:54:18.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:18 smithi007 ceph-mon[28477]: pgmap v303: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 744 KiB/s wr, 164 op/s 2024-04-21T08:54:19.083 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/devres.o 2024-04-21T08:54:19.319 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ramfs/file-mmu.o 2024-04-21T08:54:19.394 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/autoprobe.o 2024-04-21T08:54:19.411 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/power/poweroff.o 2024-04-21T08:54:19.432 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/rcu/srcutree.o 2024-04-21T08:54:19.738 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/inline.o 2024-04-21T08:54:19.817 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/irqdomain.o 2024-04-21T08:54:19.831 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/power/built-in.a 2024-04-21T08:54:19.862 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/fpu/xstate.o 2024-04-21T08:54:20.115 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/proc.o 2024-04-21T08:54:20.160 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/livepatch/built-in.a 2024-04-21T08:54:20.246 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/dma/mapping.o 2024-04-21T08:54:20.284 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/ramfs/built-in.a 2024-04-21T08:54:20.539 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/hugetlbfs/inode.o 2024-04-21T08:54:20.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:20 smithi077 ceph-mon[31615]: pgmap v304: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-21T08:54:20.887 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/interval_tree.o 2024-04-21T08:54:20.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:20 smithi007 ceph-mon[28477]: pgmap v304: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-21T08:54:21.019 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/rcu/tree.o 2024-04-21T08:54:21.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:20 smithi016 ceph-mon[31259]: pgmap v304: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-04-21T08:54:21.294 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/migration.o 2024-04-21T08:54:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.445+0000 7fa32ad2e640 1 -- 172.21.15.7:0/1616071912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa324113ea0 msgr2=0x7fa324116290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.445+0000 7fa32ad2e640 1 --2- 172.21.15.7:0/1616071912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa324113ea0 0x7fa324116290 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa314009900 tx=0x7fa31402bc50 comp rx=0 tx=0).stop 2024-04-21T08:54:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.445+0000 7fa32ad2e640 1 -- 172.21.15.7:0/1616071912 shutdown_connections 2024-04-21T08:54:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.445+0000 7fa32ad2e640 1 --2- 172.21.15.7:0/1616071912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa324113ea0 0x7fa324116290 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.445+0000 7fa32ad2e640 1 --2- 172.21.15.7:0/1616071912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa32410b210 0x7fa324113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.445+0000 7fa32ad2e640 1 --2- 172.21.15.7:0/1616071912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa32410a8f0 0x7fa32410acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.445+0000 7fa32ad2e640 1 -- 172.21.15.7:0/1616071912 >> 172.21.15.7:0/1616071912 conn(0x7fa32406d030 msgr2=0x7fa32406d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.445+0000 7fa32ad2e640 1 -- 172.21.15.7:0/1616071912 shutdown_connections 2024-04-21T08:54:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.445+0000 7fa32ad2e640 1 -- 172.21.15.7:0/1616071912 wait complete. 2024-04-21T08:54:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.446+0000 7fa32ad2e640 1 Processor -- start 2024-04-21T08:54:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.446+0000 7fa32ad2e640 1 -- start start 2024-04-21T08:54:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.446+0000 7fa32ad2e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32410a8f0 0x7fa32410fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.446+0000 7fa32ad2e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa32410b210 0x7fa324110060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa32ad2e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa324113ea0 0x7fa3241105a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa32ad2e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa3241188c0 con 0x7fa324113ea0 2024-04-21T08:54:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa32ad2e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa324118740 con 0x7fa32410a8f0 2024-04-21T08:54:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa32ad2e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa324118a40 con 0x7fa32410b210 2024-04-21T08:54:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa328aca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32410a8f0 0x7fa32410fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa328aca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32410a8f0 0x7fa32410fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53340/0 (socket says 172.21.15.7:53340) 2024-04-21T08:54:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa328aca640 1 -- 172.21.15.7:0/3104069165 learned_addr learned my addr 172.21.15.7:0/3104069165 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa323fff640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa32410b210 0x7fa324110060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa328aca640 1 -- 172.21.15.7:0/3104069165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa32410b210 msgr2=0x7fa324110060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa328aca640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa32410b210 0x7fa324110060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa328aca640 1 -- 172.21.15.7:0/3104069165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa324113ea0 msgr2=0x7fa3241105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa328aca640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa324113ea0 0x7fa3241105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa328aca640 1 -- 172.21.15.7:0/3104069165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3241b3830 con 0x7fa32410a8f0 2024-04-21T08:54:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.447+0000 7fa328aca640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32410a8f0 0x7fa32410fb20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa31800b4d0 tx=0x7fa31800b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.449+0000 7fa321ffb640 1 -- 172.21.15.7:0/3104069165 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318015020 con 0x7fa32410a8f0 2024-04-21T08:54:21.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.449+0000 7fa32ad2e640 1 -- 172.21.15.7:0/3104069165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3241b3b20 con 0x7fa32410a8f0 2024-04-21T08:54:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.449+0000 7fa321ffb640 1 -- 172.21.15.7:0/3104069165 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa318011d40 con 0x7fa32410a8f0 2024-04-21T08:54:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.449+0000 7fa321ffb640 1 -- 172.21.15.7:0/3104069165 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318010e00 con 0x7fa32410a8f0 2024-04-21T08:54:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.465+0000 7fa32ad2e640 1 -- 172.21.15.7:0/3104069165 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3241b4070 con 0x7fa32410a8f0 2024-04-21T08:54:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.468+0000 7fa321ffb640 1 -- 172.21.15.7:0/3104069165 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa318018020 con 0x7fa32410a8f0 2024-04-21T08:54:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.469+0000 7fa321ffb640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa30c0a3ea0 0x7fa30c0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.469+0000 7fa321ffb640 1 -- 172.21.15.7:0/3104069165 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fa3180d0520 con 0x7fa32410a8f0 2024-04-21T08:54:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.472+0000 7fa323fff640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa30c0a3ea0 0x7fa30c0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.477+0000 7fa32d9f3740 1 Processor -- start 2024-04-21T08:54:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.482+0000 7fa323fff640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa30c0a3ea0 0x7fa30c0a6360 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa3241112e0 tx=0x7fa31c00d040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa32d9f3740 1 -- start start 2024-04-21T08:54:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b20620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b20fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b2c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e089a858d0 con 0x55e089b218d0 2024-04-21T08:54:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e089a22e80 con 0x55e089b20b60 2024-04-21T08:54:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e089a81890 con 0x55e089b2a0e0 2024-04-21T08:54:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa3292cb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b2c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa3292cb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b2c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:38028/0 (socket says 172.21.15.7:38028) 2024-04-21T08:54:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa3292cb640 1 -- 172.21.15.7:0/364598100 learned_addr learned my addr 172.21.15.7:0/364598100 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa323fff640 1 --2- 172.21.15.7:0/364598100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b20fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.489+0000 7fa328aca640 1 --2- 172.21.15.7:0/364598100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b20620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.490+0000 7fa3292cb640 1 -- 172.21.15.7:0/364598100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 msgr2=0x55e089b20fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.490+0000 7fa3292cb640 1 --2- 172.21.15.7:0/364598100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b20fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.490+0000 7fa3292cb640 1 -- 172.21.15.7:0/364598100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 msgr2=0x55e089b20620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.490+0000 7fa3292cb640 1 --2- 172.21.15.7:0/364598100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b20620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.490+0000 7fa3292cb640 1 -- 172.21.15.7:0/364598100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e089a44320 con 0x55e089b2a0e0 2024-04-21T08:54:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.490+0000 7fa3292cb640 1 --2- 172.21.15.7:0/364598100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b2c4d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa31402c1e0 tx=0x7fa3140036a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f15087d9f6cf9cae server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.500+0000 7fa30affd640 1 -- 172.21.15.7:0/364598100 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa314039040 con 0x55e089b2a0e0 2024-04-21T08:54:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.500+0000 7fa30affd640 1 -- 172.21.15.7:0/364598100 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa314043070 con 0x55e089b2a0e0 2024-04-21T08:54:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.502+0000 7fa32d9f3740 1 -- 172.21.15.7:0/364598100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 msgr2=0x55e089b2c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.502+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/364598100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b2c4d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa31402c1e0 tx=0x7fa3140036a0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.504 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/cpuhotplug.o 2024-04-21T08:54:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.505+0000 7fa32d9f3740 1 -- 172.21.15.7:0/364598100 shutdown_connections 2024-04-21T08:54:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.505+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/364598100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b2c4d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.505+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/364598100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b20fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.505+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/364598100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b20620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.505+0000 7fa32d9f3740 1 -- 172.21.15.7:0/364598100 >> 172.21.15.7:0/364598100 conn(0x55e089ae7c90 msgr2=0x55e089b1b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.505+0000 7fa32d9f3740 1 -- 172.21.15.7:0/364598100 shutdown_connections 2024-04-21T08:54:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.505+0000 7fa32d9f3740 1 -- 172.21.15.7:0/364598100 wait complete. 2024-04-21T08:54:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.505+0000 7fa32d9f3740 1 -- start start 2024-04-21T08:54:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.506+0000 7fa32d9f3740 1 Processor -- start 2024-04-21T08:54:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.506+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b29980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.506+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b26d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.506+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b27250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.506+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e089b2ce00 con 0x55e089b218d0 2024-04-21T08:54:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.506+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e089a0deb0 con 0x55e089b20b60 2024-04-21T08:54:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.506+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e089b2cf80 con 0x55e089b2a0e0 2024-04-21T08:54:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.506+0000 7fa32d9f3740 10 client.0 _ll_register_callbacks cb 0x55e089a3ab40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.506+0000 7fa328aca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b29980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.506+0000 7fa328aca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b29980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53362/0 (socket says 172.21.15.7:53362) 2024-04-21T08:54:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.507+0000 7fa328aca640 1 -- 172.21.15.7:0/375924410 learned_addr learned my addr 172.21.15.7:0/375924410 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.507+0000 7fa3292cb640 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b27250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.507+0000 7fa328aca640 1 -- 172.21.15.7:0/375924410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 msgr2=0x55e089b27250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.507+0000 7fa328aca640 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b27250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.507+0000 7fa328aca640 1 -- 172.21.15.7:0/375924410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 msgr2=0x55e089b26d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.507+0000 7fa328aca640 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b26d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.507+0000 7fa328aca640 1 -- 172.21.15.7:0/375924410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e089798210 con 0x55e089b20b60 2024-04-21T08:54:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.507+0000 7fa328aca640 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b29980 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa31800be90 tx=0x7fa3180989f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee0f32f497c36d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.511+0000 7fa303fff640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:54:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.511+0000 7fa303fff640 1 -- 172.21.15.7:0/375924410 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3180b4020 con 0x55e089b20b60 2024-04-21T08:54:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.511+0000 7fa303fff640 1 -- 172.21.15.7:0/375924410 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fa3180d0120 con 0x55e089b20b60 2024-04-21T08:54:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.511+0000 7fa32d9f3740 1 -- 172.21.15.7:0/375924410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa314043070 con 0x55e089b20b60 2024-04-21T08:54:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.524+0000 7fa303fff640 1 -- 172.21.15.7:0/375924410 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa3180a9050 con 0x55e089b20b60 2024-04-21T08:54:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.524+0000 7fa32d9f3740 10 client.26738 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.524+0000 7fa32d9f3740 1 -- 172.21.15.7:0/375924410 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e089864620 con 0x55e089b20b60 2024-04-21T08:54:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.524+0000 7fa303fff640 1 -- 172.21.15.7:0/375924410 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa314043070 con 0x55e089b20b60 2024-04-21T08:54:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.524+0000 7fa303fff640 1 -- 172.21.15.7:0/375924410 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa31809b5b0 con 0x55e089b20b60 2024-04-21T08:54:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.524+0000 7fa32d9f3740 10 client.26738 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.524+0000 7fa32d9f3740 10 client.26738 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.524+0000 7fa32d9f3740 20 client.26738 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:21.525 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.524+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e089b4b0f0 0x55e089b4d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.525 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.524+0000 7fa32d9f3740 4 client.26738 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:21.525 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.524+0000 7fa32d9f3740 1 -- 172.21.15.7:0/375924410 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e089a99880 con 0x55e089b4b0f0 2024-04-21T08:54:21.525 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.525+0000 7fa323fff640 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e089b4b0f0 0x55e089b4d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.525 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.525+0000 7fa323fff640 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e089b4b0f0 0x55e089b4d520 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d46131842072467e server_cookie=55b3310c77e6f19e in_seq=0 out_seq=0 2024-04-21T08:54:21.526 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.525+0000 7fa303fff640 10 client.26738 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:21.526 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.526+0000 7fa303fff640 1 -- 172.21.15.7:0/375924410 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e089a99880 con 0x55e089b4b0f0 2024-04-21T08:54:21.526 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.526+0000 7fa303fff640 10 client.26738 handle_command_reply: tid=0 2024-04-21T08:54:21.526 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.526+0000 7fa32d9f3740 1 client.26738 shutdown 2024-04-21T08:54:21.526 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.527+0000 7fa32d9f3740 1 -- 172.21.15.7:0/375924410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 msgr2=0x55e089b29980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:21.526 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.527+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b29980 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa31800be90 tx=0x7fa3180989f0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.527+0000 7fa32d9f3740 1 -- 172.21.15.7:0/375924410 shutdown_connections 2024-04-21T08:54:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.527+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e089b4b0f0 0x55e089b4d520 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.527+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b27250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.527+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b26d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.527+0000 7fa303fff640 0 client.26738 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.527+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/375924410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b29980 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.527+0000 7fa32d9f3740 1 -- 172.21.15.7:0/375924410 >> 172.21.15.7:0/375924410 conn(0x55e089ae7c90 msgr2=0x55e089b1d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.527+0000 7fa32d9f3740 1 -- 172.21.15.7:0/375924410 shutdown_connections 2024-04-21T08:54:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.527+0000 7fa32d9f3740 1 -- 172.21.15.7:0/375924410 wait complete. 2024-04-21T08:54:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.528+0000 7fa32d9f3740 20 client.26738 trim_cache size 0 max 16384 2024-04-21T08:54:21.588 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.588+0000 7fa32d9f3740 1 Processor -- start 2024-04-21T08:54:21.588 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.588+0000 7fa32d9f3740 1 -- start start 2024-04-21T08:54:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.589+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b20b60 0x55e089b35db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.589+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b218d0 0x55e089b29440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.589+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b2a0e0 0x55e089b29980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.589+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa324126490 con 0x55e089b2a0e0 2024-04-21T08:54:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.589+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3241b65e0 con 0x55e089b218d0 2024-04-21T08:54:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.589+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa3241b43d0 con 0x55e089b20b60 2024-04-21T08:54:21.591 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.592+0000 7fa3292cb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b2a0e0 0x55e089b29980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.591 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.592+0000 7fa3292cb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b2a0e0 0x55e089b29980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47240/0 (socket says 172.21.15.7:47240) 2024-04-21T08:54:21.592 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.592+0000 7fa3292cb640 1 -- 172.21.15.7:0/2445081664 learned_addr learned my addr 172.21.15.7:0/2445081664 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:21.592 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.593+0000 7fa323fff640 1 --2- 172.21.15.7:0/2445081664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b218d0 0x55e089b29440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.592 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.593+0000 7fa328aca640 1 --2- 172.21.15.7:0/2445081664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b20b60 0x55e089b35db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.593+0000 7fa328aca640 1 -- 172.21.15.7:0/2445081664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b218d0 msgr2=0x55e089b29440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:21.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.593+0000 7fa328aca640 1 --2- 172.21.15.7:0/2445081664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b218d0 0x55e089b29440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.593+0000 7fa328aca640 1 -- 172.21.15.7:0/2445081664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b2a0e0 msgr2=0x55e089b29980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:21.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.593+0000 7fa328aca640 1 --2- 172.21.15.7:0/2445081664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b2a0e0 0x55e089b29980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.593+0000 7fa328aca640 1 -- 172.21.15.7:0/2445081664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e089b4dba0 con 0x55e089b20b60 2024-04-21T08:54:21.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.593+0000 7fa328aca640 1 --2- 172.21.15.7:0/2445081664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b20b60 0x55e089b35db0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa31809c0f0 tx=0x7fa3180108a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ff046f18fc5ff7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:21.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.595+0000 7fa30a7fc640 1 -- 172.21.15.7:0/2445081664 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3180b4040 con 0x55e089b20b60 2024-04-21T08:54:21.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.595+0000 7fa30a7fc640 1 -- 172.21.15.7:0/2445081664 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3180a4dd0 con 0x55e089b20b60 2024-04-21T08:54:21.595 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.596+0000 7fa30a7fc640 1 -- 172.21.15.7:0/2445081664 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318010500 con 0x55e089b20b60 2024-04-21T08:54:21.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.596+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2445081664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b20b60 msgr2=0x55e089b35db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:21.596 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.596+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/2445081664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b20b60 0x55e089b35db0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa31809c0f0 tx=0x7fa3180108a0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.601+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2445081664 shutdown_connections 2024-04-21T08:54:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.601+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/2445081664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b2a0e0 0x55e089b29980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.601+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/2445081664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b218d0 0x55e089b29440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.601+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/2445081664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b20b60 0x55e089b35db0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.601+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2445081664 >> 172.21.15.7:0/2445081664 conn(0x55e089ae7c90 msgr2=0x55e089b26cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.601+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2445081664 shutdown_connections 2024-04-21T08:54:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.601+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2445081664 wait complete. 2024-04-21T08:54:21.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.601+0000 7fa32d9f3740 1 -- start start 2024-04-21T08:54:21.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.602+0000 7fa32d9f3740 1 Processor -- start 2024-04-21T08:54:21.605 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.602+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b23c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.603+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b24170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.603+0000 7fa32d9f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b246b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.603+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa3241b4aa0 con 0x55e089b218d0 2024-04-21T08:54:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.603+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3241b5f10 con 0x55e089b20b60 2024-04-21T08:54:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.603+0000 7fa32d9f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa3241b5170 con 0x55e089b2a0e0 2024-04-21T08:54:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.603+0000 7fa323fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b24170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.603+0000 7fa323fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b24170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47242/0 (socket says 172.21.15.7:47242) 2024-04-21T08:54:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.603+0000 7fa323fff640 1 -- 172.21.15.7:0/2069819383 learned_addr learned my addr 172.21.15.7:0/2069819383 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.606+0000 7fa3292cb640 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b246b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.606 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.606+0000 7fa328aca640 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b23c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.606+0000 7fa3292cb640 1 -- 172.21.15.7:0/2069819383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 msgr2=0x55e089b23c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.606+0000 7fa3292cb640 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b23c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.606+0000 7fa3292cb640 1 -- 172.21.15.7:0/2069819383 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 msgr2=0x55e089b24170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.606+0000 7fa3292cb640 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b24170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.606+0000 7fa3292cb640 1 -- 172.21.15.7:0/2069819383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e089b1bd90 con 0x55e089b2a0e0 2024-04-21T08:54:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.607+0000 7fa32d9f3740 10 client.0 _ll_register_callbacks cb 0x7fa3180cdf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.607+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2069819383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3180a4dd0 con 0x55e089b2a0e0 2024-04-21T08:54:21.607 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.607+0000 7fa3292cb640 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b246b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa314005dc0 tx=0x7fa314009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d14fad33c6ad1439 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.609+0000 7fa303fff640 10 client.26731 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:54:21.608 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.609+0000 7fa303fff640 1 -- 172.21.15.7:0/2069819383 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa314058050 con 0x55e089b2a0e0 2024-04-21T08:54:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.609+0000 7fa303fff640 1 -- 172.21.15.7:0/2069819383 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fa31402c5f0 con 0x55e089b2a0e0 2024-04-21T08:54:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.609+0000 7fa303fff640 1 -- 172.21.15.7:0/2069819383 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa31402c850 con 0x55e089b2a0e0 2024-04-21T08:54:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.609+0000 7fa32d9f3740 10 client.26731 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:21.609 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.609+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2069819383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=23}) -- 0x55e089bb18d0 con 0x55e089b2a0e0 2024-04-21T08:54:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.609+0000 7fa303fff640 1 -- 172.21.15.7:0/2069819383 <== mon.1 v2:172.21.15.77:3300/0 4 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fa31405b020 con 0x55e089b2a0e0 2024-04-21T08:54:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.609+0000 7fa32d9f3740 10 client.26731 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.609+0000 7fa32d9f3740 10 client.26731 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.610+0000 7fa32d9f3740 20 client.26731 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:21.610 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.610+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e089bb2030 0x55e089bb4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.610+0000 7fa32d9f3740 4 client.26731 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.610+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2069819383 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e089a253d0 con 0x55e089bb2030 2024-04-21T08:54:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.610+0000 7fa328aca640 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e089bb2030 0x55e089bb4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.611 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.611+0000 7fa328aca640 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e089bb2030 0x55e089bb4470 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf97ae63801824fa server_cookie=487f426093c1e25b in_seq=0 out_seq=0 2024-04-21T08:54:21.612 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:21.613+0000 7fa303fff640 10 client.26731 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:21.780 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/kernel/fpu/built-in.a 2024-04-21T08:54:21.804 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/dma/direct.o 2024-04-21T08:54:22.181 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/list_lru.o 2024-04-21T08:54:22.264 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/mce/core.o 2024-04-21T08:54:22.322 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/hugetlbfs/built-in.a 2024-04-21T08:54:22.330 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/mce/severity.o 2024-04-21T08:54:22.384 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/pm.o 2024-04-21T08:54:22.795 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/inode.o 2024-04-21T08:54:22.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:22 smithi077 ceph-mon[31615]: pgmap v305: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.2 MiB/s wr, 75 op/s 2024-04-21T08:54:22.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:22 smithi007 ceph-mon[28477]: pgmap v305: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.2 MiB/s wr, 75 op/s 2024-04-21T08:54:23.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:22 smithi016 ceph-mon[31259]: pgmap v305: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 1.2 MiB/s wr, 75 op/s 2024-04-21T08:54:23.065 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/dma/ops_helpers.o 2024-04-21T08:54:23.237 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/dma/dummy.o 2024-04-21T08:54:23.404 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/workingset.o 2024-04-21T08:54:23.440 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/msi.o 2024-04-21T08:54:23.719 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/debug.o 2024-04-21T08:54:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa303fff640 1 -- 172.21.15.7:0/2069819383 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e089a253d0 con 0x55e089bb2030 2024-04-21T08:54:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa303fff640 10 client.26731 handle_command_reply: tid=0 2024-04-21T08:54:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 1 client.26731 shutdown 2024-04-21T08:54:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2069819383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 msgr2=0x55e089b246b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b246b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa314005dc0 tx=0x7fa314009260 comp rx=0 tx=0).stop 2024-04-21T08:54:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2069819383 shutdown_connections 2024-04-21T08:54:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55e089bb2030 0x55e089bb4470 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e089b2a0e0 0x55e089b246b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e089b218d0 0x55e089b24170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 1 --2- 172.21.15.7:0/2069819383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e089b20b60 0x55e089b23c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2069819383 >> 172.21.15.7:0/2069819383 conn(0x55e089ae7c90 msgr2=0x55e089b1a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa303fff640 0 client.26731 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2069819383 shutdown_connections 2024-04-21T08:54:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 1 -- 172.21.15.7:0/2069819383 wait complete. 2024-04-21T08:54:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.861+0000 7fa32d9f3740 20 client.26731 trim_cache size 0 max 16384 2024-04-21T08:54:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:54:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:54:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:54:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:54:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.870+0000 7fa32ad2e640 1 -- 172.21.15.7:0/3104069165 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa30c0a3ea0 msgr2=0x7fa30c0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.870+0000 7fa32ad2e640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa30c0a3ea0 0x7fa30c0a6360 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa3241112e0 tx=0x7fa31c00d040 comp rx=0 tx=0).stop 2024-04-21T08:54:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.870+0000 7fa32ad2e640 1 -- 172.21.15.7:0/3104069165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32410a8f0 msgr2=0x7fa32410fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.870+0000 7fa32ad2e640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32410a8f0 0x7fa32410fb20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa31800b4d0 tx=0x7fa31800b990 comp rx=0 tx=0).stop 2024-04-21T08:54:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.880+0000 7fa32ad2e640 1 -- 172.21.15.7:0/3104069165 shutdown_connections 2024-04-21T08:54:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.880+0000 7fa32ad2e640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa324113ea0 0x7fa3241105a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.880+0000 7fa32ad2e640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa30c0a3ea0 0x7fa30c0a6360 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.880+0000 7fa32ad2e640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa32410b210 0x7fa324110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.880+0000 7fa32ad2e640 1 --2- 172.21.15.7:0/3104069165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa32410a8f0 0x7fa32410fb20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.881+0000 7fa32ad2e640 1 -- 172.21.15.7:0/3104069165 >> 172.21.15.7:0/3104069165 conn(0x7fa32406d030 msgr2=0x7fa324070ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.882+0000 7fa32ad2e640 1 -- 172.21.15.7:0/3104069165 shutdown_connections 2024-04-21T08:54:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:23.882+0000 7fa32ad2e640 1 -- 172.21.15.7:0/3104069165 wait complete. 2024-04-21T08:54:24.193 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/dma/swiotlb.o 2024-04-21T08:54:24.279 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/gup.o 2024-04-21T08:54:24.775 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mmap_lock.o 2024-04-21T08:54:24.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:24 smithi077 ceph-mon[31615]: pgmap v306: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.3 MiB/s wr, 104 op/s 2024-04-21T08:54:24.856 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:24 smithi016 ceph-mon[31259]: pgmap v306: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.3 MiB/s wr, 104 op/s 2024-04-21T08:54:24.884 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/mce/genpool.o 2024-04-21T08:54:24.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:24 smithi007 ceph-mon[28477]: pgmap v306: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.3 MiB/s wr, 104 op/s 2024-04-21T08:54:25.026 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/affinity.o 2024-04-21T08:54:25.113 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq/matrix.o 2024-04-21T08:54:25.726 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/highmem.o 2024-04-21T08:54:25.893 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/mce/intel.o 2024-04-21T08:54:25.926 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/dma/remap.o 2024-04-21T08:54:26.000 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/memory.o 2024-04-21T08:54:26.159 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/rcu/rcu_segcblist.o 2024-04-21T08:54:26.743 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/irq/built-in.a 2024-04-21T08:54:26.889 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fat/cache.o 2024-04-21T08:54:26.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:26 smithi007 ceph-mon[28477]: pgmap v307: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-04-21T08:54:26.968 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/rcu/built-in.a 2024-04-21T08:54:27.028 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/dma/built-in.a 2024-04-21T08:54:27.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:26 smithi016 ceph-mon[31259]: pgmap v307: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-04-21T08:54:27.065 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fat/dir.o 2024-04-21T08:54:27.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:26 smithi077 ceph-mon[31615]: pgmap v307: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.1 MiB/s wr, 96 op/s 2024-04-21T08:54:27.134 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/ioctl.o 2024-04-21T08:54:27.140 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/mce/amd.o 2024-04-21T08:54:27.228 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/entry/common.o 2024-04-21T08:54:27.274 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/module/main.o 2024-04-21T08:54:27.920 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/module/strict_rwx.o 2024-04-21T08:54:28.601 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/entry/syscall_user_dispatch.o 2024-04-21T08:54:28.811 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/time.o 2024-04-21T08:54:29.038 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/futex/core.o 2024-04-21T08:54:29.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:28 smithi016 ceph-mon[31259]: pgmap v308: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-04-21T08:54:29.086 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fat/fatent.o 2024-04-21T08:54:29.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:28 smithi077 ceph-mon[31615]: pgmap v308: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-04-21T08:54:29.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:28 smithi007 ceph-mon[28477]: pgmap v308: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 1.2 MiB/s wr, 145 op/s 2024-04-21T08:54:29.272 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/mce/threshold.o 2024-04-21T08:54:29.557 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/entry/built-in.a 2024-04-21T08:54:29.837 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/mballoc.o 2024-04-21T08:54:29.961 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/futex/syscalls.o 2024-04-21T08:54:30.091 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/module/kmod.o 2024-04-21T08:54:30.473 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/timer.o 2024-04-21T08:54:30.569 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/kernel/cpu/mce/built-in.a 2024-04-21T08:54:30.621 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/futex/pi.o 2024-04-21T08:54:30.841 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fat/file.o 2024-04-21T08:54:30.892 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mincore.o 2024-04-21T08:54:30.972 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/mtrr/mtrr.o 2024-04-21T08:54:31.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:30 smithi016 ceph-mon[31259]: pgmap v309: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 896 KiB/s wr, 132 op/s 2024-04-21T08:54:31.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:30 smithi077 ceph-mon[31615]: pgmap v309: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 896 KiB/s wr, 132 op/s 2024-04-21T08:54:31.111 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:54:31.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:54:31.132 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/futex/requeue.o 2024-04-21T08:54:31.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:30 smithi007 ceph-mon[28477]: pgmap v309: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 896 KiB/s wr, 132 op/s 2024-04-21T08:54:31.191 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/module/tree_lookup.o 2024-04-21T08:54:31.589 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/futex/waitwake.o 2024-04-21T08:54:31.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.804+0000 7fdf9db15640 1 -- 172.21.15.7:0/3494789105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf9810e1b0 msgr2=0x7fdf9810e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:31.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.804+0000 7fdf9db15640 1 --2- 172.21.15.7:0/3494789105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf9810e1b0 0x7fdf9810e660 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c01a910 tx=0x7fdf8c03cc60 comp rx=0 tx=0).stop 2024-04-21T08:54:31.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.804+0000 7fdf9db15640 1 -- 172.21.15.7:0/3494789105 shutdown_connections 2024-04-21T08:54:31.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.804+0000 7fdf9db15640 1 --2- 172.21.15.7:0/3494789105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf9810e1b0 0x7fdf9810e660 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.804+0000 7fdf9db15640 1 --2- 172.21.15.7:0/3494789105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf98110b10 0x7fdf98109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.804+0000 7fdf9db15640 1 --2- 172.21.15.7:0/3494789105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf981106a0 0x7fdf98109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.804+0000 7fdf9db15640 1 -- 172.21.15.7:0/3494789105 >> 172.21.15.7:0/3494789105 conn(0x7fdf9806e360 msgr2=0x7fdf9806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:31.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.810+0000 7fdf9db15640 1 -- 172.21.15.7:0/3494789105 shutdown_connections 2024-04-21T08:54:31.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.810+0000 7fdf9db15640 1 -- 172.21.15.7:0/3494789105 wait complete. 2024-04-21T08:54:31.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.810+0000 7fdf9db15640 1 Processor -- start 2024-04-21T08:54:31.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.811+0000 7fdf9db15640 1 -- start start 2024-04-21T08:54:31.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.811+0000 7fdf9db15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf9810e1b0 0x7fdf9819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.811+0000 7fdf9db15640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf981106a0 0x7fdf9819f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.811+0000 7fdf9db15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf98110b10 0x7fdf981a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.811+0000 7fdf9db15640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdf981128f0 con 0x7fdf981106a0 2024-04-21T08:54:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.811+0000 7fdf9db15640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf98112770 con 0x7fdf9810e1b0 2024-04-21T08:54:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.811+0000 7fdf9db15640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdf98112a70 con 0x7fdf98110b10 2024-04-21T08:54:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.811+0000 7fdf977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf9810e1b0 0x7fdf9819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.811+0000 7fdf977fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf9810e1b0 0x7fdf9819ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:46186/0 (socket says 172.21.15.7:46186) 2024-04-21T08:54:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.811+0000 7fdf977fe640 1 -- 172.21.15.7:0/854654590 learned_addr learned my addr 172.21.15.7:0/854654590 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.812+0000 7fdf97fff640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf98110b10 0x7fdf981a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.812+0000 7fdf977fe640 1 -- 172.21.15.7:0/854654590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf98110b10 msgr2=0x7fdf981a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.812+0000 7fdf977fe640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf98110b10 0x7fdf981a36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.812+0000 7fdf977fe640 1 -- 172.21.15.7:0/854654590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf981106a0 msgr2=0x7fdf9819f320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:54:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.812+0000 7fdf977fe640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf981106a0 0x7fdf9819f320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.812+0000 7fdf977fe640 1 -- 172.21.15.7:0/854654590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf981a3dd0 con 0x7fdf9810e1b0 2024-04-21T08:54:31.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.812+0000 7fdf977fe640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf9810e1b0 0x7fdf9819ede0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800b4a0 tx=0x7fdf8800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:31.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.821+0000 7fdf94ff9640 1 -- 172.21.15.7:0/854654590 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88014070 con 0x7fdf9810e1b0 2024-04-21T08:54:31.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.821+0000 7fdf9db15640 1 -- 172.21.15.7:0/854654590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf981a40c0 con 0x7fdf9810e1b0 2024-04-21T08:54:31.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.821+0000 7fdf9db15640 1 -- 172.21.15.7:0/854654590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf981ab950 con 0x7fdf9810e1b0 2024-04-21T08:54:31.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.821+0000 7fdf94ff9640 1 -- 172.21.15.7:0/854654590 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf88010ac0 con 0x7fdf9810e1b0 2024-04-21T08:54:31.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.821+0000 7fdf94ff9640 1 -- 172.21.15.7:0/854654590 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88017b80 con 0x7fdf9810e1b0 2024-04-21T08:54:31.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.824+0000 7fdf94ff9640 1 -- 172.21.15.7:0/854654590 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fdf88017070 con 0x7fdf9810e1b0 2024-04-21T08:54:31.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.824+0000 7fdfa07e8740 1 Processor -- start 2024-04-21T08:54:31.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.824+0000 7fdfa07e8740 1 -- start start 2024-04-21T08:54:31.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1580 0x5615d45f1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45f1ea0 0x5615d45fa540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45fce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5615d45561e0 con 0x5615d45f1ea0 2024-04-21T08:54:31.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5615d44f3390 con 0x5615d45f1580 2024-04-21T08:54:31.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5615d45521a0 con 0x5615d45faa80 2024-04-21T08:54:31.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdf96ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:31.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdf96ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45fce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:34466/0 (socket says 172.21.15.7:34466) 2024-04-21T08:54:31.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdf96ffd640 1 -- 172.21.15.7:0/3546198860 learned_addr learned my addr 172.21.15.7:0/3546198860 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:31.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdf96ffd640 1 -- 172.21.15.7:0/3546198860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1580 msgr2=0x5615d45f1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:31.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdf96ffd640 1 --2- 172.21.15.7:0/3546198860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1580 0x5615d45f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdf96ffd640 1 -- 172.21.15.7:0/3546198860 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45f1ea0 msgr2=0x5615d45fa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:31.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdf96ffd640 1 --2- 172.21.15.7:0/3546198860 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45f1ea0 0x5615d45fa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.825+0000 7fdf96ffd640 1 -- 172.21.15.7:0/3546198860 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615d43327d0 con 0x5615d45faa80 2024-04-21T08:54:31.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.826+0000 7fdf96ffd640 1 --2- 172.21.15.7:0/3546198860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45fce70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009da0 tx=0x7fdf9000e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4839998e5a62585 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:31.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.827+0000 7fdf94ff9640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdf800a3f20 0x7fdf800a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.827+0000 7fdf94ff9640 1 -- 172.21.15.7:0/854654590 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fdf880cfca0 con 0x7fdf9810e1b0 2024-04-21T08:54:31.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.834+0000 7fdf7dffb640 1 -- 172.21.15.7:0/3546198860 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90004b20 con 0x5615d45faa80 2024-04-21T08:54:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.834+0000 7fdf7dffb640 1 -- 172.21.15.7:0/3546198860 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf90003e20 con 0x5615d45faa80 2024-04-21T08:54:31.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.839+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3546198860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 msgr2=0x5615d45fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:31.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.839+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/3546198860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45fce70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009da0 tx=0x7fdf9000e720 comp rx=0 tx=0).stop 2024-04-21T08:54:31.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.844+0000 7fdf97fff640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdf800a3f20 0x7fdf800a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.844+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3546198860 shutdown_connections 2024-04-21T08:54:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.844+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/3546198860 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45fce70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.844+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/3546198860 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45f1ea0 0x5615d45fa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.844+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/3546198860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1580 0x5615d45f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.844+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3546198860 >> 172.21.15.7:0/3546198860 conn(0x5615d45ebcb0 msgr2=0x5615d45ec090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:31.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.844+0000 7fdf97fff640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdf800a3f20 0x7fdf800a63e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c01a6d0 tx=0x7fdf8c03f040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:31.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.854+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3546198860 shutdown_connections 2024-04-21T08:54:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.856+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3546198860 wait complete. 2024-04-21T08:54:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.856+0000 7fdfa07e8740 1 -- start start 2024-04-21T08:54:31.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.856+0000 7fdfa07e8740 1 Processor -- start 2024-04-21T08:54:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.857+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1580 0x5615d45f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.857+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1ea0 0x5615d45f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.857+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45faa80 0x5615d45f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.857+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5615d45fd8c0 con 0x5615d45faa80 2024-04-21T08:54:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.857+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5615d450f2f0 con 0x5615d45f1ea0 2024-04-21T08:54:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.857+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5615d45fda40 con 0x5615d45f1580 2024-04-21T08:54:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.857+0000 7fdfa07e8740 10 client.0 _ll_register_callbacks cb 0x5615d451c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.857+0000 7fdf977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1580 0x5615d45f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.857+0000 7fdf977fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1580 0x5615d45f7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:34480/0 (socket says 172.21.15.7:34480) 2024-04-21T08:54:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.857+0000 7fdf977fe640 1 -- 172.21.15.7:0/1956665912 learned_addr learned my addr 172.21.15.7:0/1956665912 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.858+0000 7fdf977fe640 1 -- 172.21.15.7:0/1956665912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1ea0 msgr2=0x5615d45f7f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.858+0000 7fdf977fe640 1 --2- 172.21.15.7:0/1956665912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1ea0 0x5615d45f7f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.858+0000 7fdf977fe640 1 -- 172.21.15.7:0/1956665912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45faa80 msgr2=0x5615d45f84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.858+0000 7fdf977fe640 1 --2- 172.21.15.7:0/1956665912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45faa80 0x5615d45f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.858+0000 7fdf977fe640 1 -- 172.21.15.7:0/1956665912 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615d42683f0 con 0x5615d45f1580 2024-04-21T08:54:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.858+0000 7fdf977fe640 1 --2- 172.21.15.7:0/1956665912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1580 0x5615d45f7a40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdf8809b580 tx=0x7fdf8809bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e29c00a0d56e0a79 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:31.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.864+0000 7fdf76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:54:31.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.864+0000 7fdf76ffd640 1 -- 172.21.15.7:0/1956665912 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8801e040 con 0x5615d45f1580 2024-04-21T08:54:31.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.865+0000 7fdfa07e8740 1 -- 172.21.15.7:0/1956665912 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5615d4563c00 con 0x5615d45f1580 2024-04-21T08:54:31.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.865+0000 7fdf76ffd640 1 -- 172.21.15.7:0/1956665912 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fdf880cd1a0 con 0x5615d45f1580 2024-04-21T08:54:31.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.865+0000 7fdf76ffd640 1 -- 172.21.15.7:0/1956665912 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdf880a44f0 con 0x5615d45f1580 2024-04-21T08:54:31.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.868+0000 7fdfa07e8740 10 client.26743 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:31.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.868+0000 7fdfa07e8740 1 -- 172.21.15.7:0/1956665912 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5615d42280f0 con 0x5615d45f1580 2024-04-21T08:54:31.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.868+0000 7fdf76ffd640 1 -- 172.21.15.7:0/1956665912 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5615d4563c00 con 0x5615d45f1580 2024-04-21T08:54:31.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.868+0000 7fdf76ffd640 1 -- 172.21.15.7:0/1956665912 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fdf8809bbf0 con 0x5615d45f1580 2024-04-21T08:54:31.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.868+0000 7fdfa07e8740 10 client.26743 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:31.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.868+0000 7fdfa07e8740 10 client.26743 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:31.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.868+0000 7fdfa07e8740 20 client.26743 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:31.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.868+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/1956665912 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5615d460b060 0x5615d460d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.868+0000 7fdfa07e8740 4 client.26743 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:31.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.868+0000 7fdfa07e8740 1 -- 172.21.15.7:0/1956665912 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615d455da70 con 0x5615d460b060 2024-04-21T08:54:31.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.877+0000 7fdf96ffd640 1 --2- 172.21.15.7:0/1956665912 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5615d460b060 0x5615d460d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.892+0000 7fdf76ffd640 10 client.26743 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:31.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.892+0000 7fdf96ffd640 1 --2- 172.21.15.7:0/1956665912 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5615d460b060 0x5615d460d500 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e3d30cfadaafd35 server_cookie=faa6d1f3d1ab9ae in_seq=0 out_seq=0 2024-04-21T08:54:31.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.896+0000 7fdf76ffd640 1 -- 172.21.15.7:0/1956665912 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5615d455da70 con 0x5615d460b060 2024-04-21T08:54:31.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.896+0000 7fdf76ffd640 10 client.26743 handle_command_reply: tid=0 2024-04-21T08:54:31.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.896+0000 7fdfa07e8740 1 client.26743 shutdown 2024-04-21T08:54:31.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.897+0000 7fdfa07e8740 1 -- 172.21.15.7:0/1956665912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1580 msgr2=0x5615d45f7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.897+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/1956665912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1580 0x5615d45f7a40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdf8809b580 tx=0x7fdf8809bf80 comp rx=0 tx=0).stop 2024-04-21T08:54:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.906+0000 7fdfa07e8740 1 -- 172.21.15.7:0/1956665912 shutdown_connections 2024-04-21T08:54:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.906+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/1956665912 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5615d460b060 0x5615d460d500 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.906+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/1956665912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45faa80 0x5615d45f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.906+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/1956665912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1ea0 0x5615d45f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.906+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/1956665912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1580 0x5615d45f7a40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.906+0000 7fdfa07e8740 1 -- 172.21.15.7:0/1956665912 >> 172.21.15.7:0/1956665912 conn(0x5615d45ebcb0 msgr2=0x5615d45eda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.906+0000 7fdf76ffd640 0 client.26743 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.906+0000 7fdfa07e8740 1 -- 172.21.15.7:0/1956665912 shutdown_connections 2024-04-21T08:54:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.906+0000 7fdfa07e8740 1 -- 172.21.15.7:0/1956665912 wait complete. 2024-04-21T08:54:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.906+0000 7fdfa07e8740 20 client.26743 trim_cache size 0 max 16384 2024-04-21T08:54:31.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.976+0000 7fdfa07e8740 1 Processor -- start 2024-04-21T08:54:31.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.979+0000 7fdfa07e8740 1 -- start start 2024-04-21T08:54:31.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.979+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45f1580 0x5615d45fa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.980+0000 7fdf977fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45f1580 0x5615d45fa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:31.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.980+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1ea0 0x5615d46068a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.980+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45f6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:31.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.980+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8400e2e0 con 0x5615d45f1580 2024-04-21T08:54:31.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.980+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf84010b40 con 0x5615d45f1ea0 2024-04-21T08:54:31.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.980+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8400e930 con 0x5615d45faa80 2024-04-21T08:54:31.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.980+0000 7fdf97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:31.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.980+0000 7fdf97fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45f6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:34494/0 (socket says 172.21.15.7:34494) 2024-04-21T08:54:31.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.980+0000 7fdf97fff640 1 -- 172.21.15.7:0/2762359807 learned_addr learned my addr 172.21.15.7:0/2762359807 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:31.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.981+0000 7fdf96ffd640 1 --2- 172.21.15.7:0/2762359807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1ea0 0x5615d46068a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:31.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.989+0000 7fdf97fff640 1 -- 172.21.15.7:0/2762359807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1ea0 msgr2=0x5615d46068a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:31.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.989+0000 7fdf97fff640 1 --2- 172.21.15.7:0/2762359807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1ea0 0x5615d46068a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.989+0000 7fdf97fff640 1 -- 172.21.15.7:0/2762359807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45f1580 msgr2=0x5615d45fa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:31.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.989+0000 7fdf97fff640 1 --2- 172.21.15.7:0/2762359807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45f1580 0x5615d45fa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:31.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.989+0000 7fdf97fff640 1 -- 172.21.15.7:0/2762359807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615d45ebae0 con 0x5615d45faa80 2024-04-21T08:54:31.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.989+0000 7fdf97fff640 1 --2- 172.21.15.7:0/2762359807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45f6310 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c046000 tx=0x7fdf8c03cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5df9ce31feba9772 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:31.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.990+0000 7fdf7d7fa640 1 -- 172.21.15.7:0/2762359807 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c060020 con 0x5615d45faa80 2024-04-21T08:54:32.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.994+0000 7fdf7d7fa640 1 -- 172.21.15.7:0/2762359807 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf8c004f40 con 0x5615d45faa80 2024-04-21T08:54:32.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.994+0000 7fdf7d7fa640 1 -- 172.21.15.7:0/2762359807 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c003850 con 0x5615d45faa80 2024-04-21T08:54:32.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.995+0000 7fdfa07e8740 1 -- 172.21.15.7:0/2762359807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 msgr2=0x5615d45f6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:32.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:31.995+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/2762359807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45f6310 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c046000 tx=0x7fdf8c03cc60 comp rx=0 tx=0).stop 2024-04-21T08:54:32.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.001+0000 7fdfa07e8740 1 -- 172.21.15.7:0/2762359807 shutdown_connections 2024-04-21T08:54:32.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.001+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/2762359807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45faa80 0x5615d45f6310 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.001+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/2762359807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1ea0 0x5615d46068a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.001+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/2762359807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45f1580 0x5615d45fa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.001+0000 7fdfa07e8740 1 -- 172.21.15.7:0/2762359807 >> 172.21.15.7:0/2762359807 conn(0x5615d45ebcb0 msgr2=0x5615d45ead70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:32.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.003+0000 7fdfa07e8740 1 -- 172.21.15.7:0/2762359807 shutdown_connections 2024-04-21T08:54:32.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.003+0000 7fdfa07e8740 1 -- 172.21.15.7:0/2762359807 wait complete. 2024-04-21T08:54:32.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.004+0000 7fdfa07e8740 1 -- start start 2024-04-21T08:54:32.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.004+0000 7fdfa07e8740 1 Processor -- start 2024-04-21T08:54:32.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1580 0x5615d45fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:32.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1ea0 0x5615d45fee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:32.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdfa07e8740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45faa80 0x5615d45ff360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8400f000 con 0x5615d45faa80 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdf84010470 con 0x5615d45f1580 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdfa07e8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8400f6d0 con 0x5615d45f1ea0 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdfa07e8740 10 client.0 _ll_register_callbacks cb 0x7fdf8809b3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdf96ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1ea0 0x5615d45fee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdf96ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1ea0 0x5615d45fee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:34508/0 (socket says 172.21.15.7:34508) 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdf96ffd640 1 -- 172.21.15.7:0/3976420419 learned_addr learned my addr 172.21.15.7:0/3976420419 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdf96ffd640 1 -- 172.21.15.7:0/3976420419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1580 msgr2=0x5615d45fe8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdf977fe640 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1580 0x5615d45fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdf96ffd640 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1580 0x5615d45fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.005+0000 7fdf96ffd640 1 -- 172.21.15.7:0/3976420419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45faa80 msgr2=0x5615d45ff360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.012+0000 7fdf97fff640 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45faa80 0x5615d45ff360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:32.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.012+0000 7fdf96ffd640 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45faa80 0x5615d45ff360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.012+0000 7fdf96ffd640 1 -- 172.21.15.7:0/3976420419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615d4670710 con 0x5615d45f1ea0 2024-04-21T08:54:32.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.013+0000 7fdf96ffd640 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1ea0 0x5615d45fee20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009da0 tx=0x7fdf900099d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38628c76ba5e3c89 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.015+0000 7fdf76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:54:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.015+0000 7fdf76ffd640 1 -- 172.21.15.7:0/3976420419 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90028020 con 0x5615d45f1ea0 2024-04-21T08:54:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.015+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3976420419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf8c004f40 con 0x5615d45f1ea0 2024-04-21T08:54:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.015+0000 7fdf76ffd640 1 -- 172.21.15.7:0/3976420419 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fdf900245b0 con 0x5615d45f1ea0 2024-04-21T08:54:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.015+0000 7fdf76ffd640 1 -- 172.21.15.7:0/3976420419 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdf90033050 con 0x5615d45f1ea0 2024-04-21T08:54:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.017+0000 7fdfa07e8740 10 client.26749 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.017+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3976420419 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5615d4671a00 con 0x5615d45f1ea0 2024-04-21T08:54:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.018+0000 7fdf76ffd640 1 -- 172.21.15.7:0/3976420419 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf8c004f40 con 0x5615d45f1ea0 2024-04-21T08:54:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.018+0000 7fdf76ffd640 1 -- 172.21.15.7:0/3976420419 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fdf90005420 con 0x5615d45f1ea0 2024-04-21T08:54:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.018+0000 7fdfa07e8740 10 client.26749 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.018+0000 7fdfa07e8740 10 client.26749 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.018+0000 7fdfa07e8740 20 client.26749 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.018+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5615d4671eb0 0x5615d46742d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.018+0000 7fdfa07e8740 4 client.26749 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:54:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.018+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3976420419 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5615d44d9590 con 0x5615d4671eb0 2024-04-21T08:54:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.018+0000 7fdf977fe640 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5615d4671eb0 0x5615d46742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.019+0000 7fdf977fe640 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5615d4671eb0 0x5615d46742d0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e73fdb6485c6d34 server_cookie=66227657a886accd in_seq=0 out_seq=0 2024-04-21T08:54:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.022+0000 7fdf76ffd640 10 client.26749 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.022+0000 7fdf76ffd640 1 -- 172.21.15.7:0/3976420419 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5615d44d9590 con 0x5615d4671eb0 2024-04-21T08:54:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.022+0000 7fdf76ffd640 10 client.26749 handle_command_reply: tid=0 2024-04-21T08:54:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.026+0000 7fdfa07e8740 1 client.26749 shutdown 2024-04-21T08:54:32.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.032+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3976420419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1ea0 msgr2=0x5615d45fee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:32.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.032+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1ea0 0x5615d45fee20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009da0 tx=0x7fdf900099d0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.033+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3976420419 shutdown_connections 2024-04-21T08:54:32.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.033+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5615d4671eb0 0x5615d46742d0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.033+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5615d45faa80 0x5615d45ff360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.033+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615d45f1ea0 0x5615d45fee20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.033+0000 7fdfa07e8740 1 --2- 172.21.15.7:0/3976420419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5615d45f1580 0x5615d45fe8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.033+0000 7fdf76ffd640 0 client.26749 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.033+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3976420419 >> 172.21.15.7:0/3976420419 conn(0x5615d45ebcb0 msgr2=0x5615d45ed0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.034+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3976420419 shutdown_connections 2024-04-21T08:54:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.034+0000 7fdfa07e8740 1 -- 172.21.15.7:0/3976420419 wait complete. 2024-04-21T08:54:32.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.038+0000 7fdfa07e8740 20 client.26749 trim_cache size 0 max 16384 2024-04-21T08:54:32.050 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:54:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:54:32.051 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:54:32.051 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:54:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.069+0000 7fdf9db15640 1 -- 172.21.15.7:0/854654590 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdf800a3f20 msgr2=0x7fdf800a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.069+0000 7fdf9db15640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdf800a3f20 0x7fdf800a63e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c01a6d0 tx=0x7fdf8c03f040 comp rx=0 tx=0).stop 2024-04-21T08:54:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.069+0000 7fdf9db15640 1 -- 172.21.15.7:0/854654590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf9810e1b0 msgr2=0x7fdf9819ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.069+0000 7fdf9db15640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf9810e1b0 0x7fdf9819ede0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800b4a0 tx=0x7fdf8800b960 comp rx=0 tx=0).stop 2024-04-21T08:54:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.069+0000 7fdf9db15640 1 -- 172.21.15.7:0/854654590 shutdown_connections 2024-04-21T08:54:32.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.069+0000 7fdf9db15640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdf800a3f20 0x7fdf800a63e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.069+0000 7fdf9db15640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf98110b10 0x7fdf981a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.069+0000 7fdf9db15640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf981106a0 0x7fdf9819f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.069+0000 7fdf9db15640 1 --2- 172.21.15.7:0/854654590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf9810e1b0 0x7fdf9819ede0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.069+0000 7fdf9db15640 1 -- 172.21.15.7:0/854654590 >> 172.21.15.7:0/854654590 conn(0x7fdf9806e360 msgr2=0x7fdf980729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.073+0000 7fdf9db15640 1 -- 172.21.15.7:0/854654590 shutdown_connections 2024-04-21T08:54:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.073+0000 7fdf9db15640 1 -- 172.21.15.7:0/854654590 wait complete. 2024-04-21T08:54:32.105 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:54:32.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:54:32.188 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/module/kallsyms.o 2024-04-21T08:54:32.201 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mlock.o 2024-04-21T08:54:32.308 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/mtrr/if.o 2024-04-21T08:54:32.316 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/module/procfs.o 2024-04-21T08:54:32.351 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fat/inode.o 2024-04-21T08:54:32.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.780+0000 7f0515470640 1 -- 172.21.15.7:0/678404488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0510109a50 msgr2=0x7f0510109e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:32.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.780+0000 7f0515470640 1 --2- 172.21.15.7:0/678404488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0510109a50 0x7f0510109e30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f050800ad20 tx=0x7f050802d220 comp rx=0 tx=0).stop 2024-04-21T08:54:32.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.780+0000 7f0515470640 1 -- 172.21.15.7:0/678404488 shutdown_connections 2024-04-21T08:54:32.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.780+0000 7f0515470640 1 --2- 172.21.15.7:0/678404488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051010e060 0x7f0510115bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.780+0000 7f0515470640 1 --2- 172.21.15.7:0/678404488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f051010a400 0x7f051010d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.780+0000 7f0515470640 1 --2- 172.21.15.7:0/678404488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0510109a50 0x7f0510109e30 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f050800ad20 tx=0x7f050802d220 comp rx=0 tx=0).stop 2024-04-21T08:54:32.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.780+0000 7f0515470640 1 -- 172.21.15.7:0/678404488 >> 172.21.15.7:0/678404488 conn(0x7f051006cf10 msgr2=0x7f051006d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:32.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.787+0000 7f0515470640 1 -- 172.21.15.7:0/678404488 shutdown_connections 2024-04-21T08:54:32.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.787+0000 7f0515470640 1 -- 172.21.15.7:0/678404488 wait complete. 2024-04-21T08:54:32.787 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/futex/built-in.a 2024-04-21T08:54:32.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.787+0000 7f0515470640 1 Processor -- start 2024-04-21T08:54:32.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.788+0000 7f0515470640 1 -- start start 2024-04-21T08:54:32.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.788+0000 7f0515470640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f051010e060 0x7f05101a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:32.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.789+0000 7f0515470640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05101a1580 0x7f05101a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:32.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.789+0000 7f0515470640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05101a5e40 0x7f05101a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:32.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.789+0000 7f0515470640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0510118680 con 0x7f05101a1580 2024-04-21T08:54:32.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.789+0000 7f0515470640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0510118500 con 0x7f05101a5e40 2024-04-21T08:54:32.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.789+0000 7f0515470640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0510118800 con 0x7f051010e060 2024-04-21T08:54:32.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.789+0000 7f0514c6f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05101a5e40 0x7f05101a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:32.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.789+0000 7f0514c6f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05101a5e40 0x7f05101a62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:46228/0 (socket says 172.21.15.7:46228) 2024-04-21T08:54:32.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.789+0000 7f0514c6f640 1 -- 172.21.15.7:0/1956201628 learned_addr learned my addr 172.21.15.7:0/1956201628 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.789+0000 7f050f7fe640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05101a1580 0x7f05101a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.789+0000 7f050ffff640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f051010e060 0x7f05101a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.790+0000 7f0514c6f640 1 -- 172.21.15.7:0/1956201628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f051010e060 msgr2=0x7f05101a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.790+0000 7f0514c6f640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f051010e060 0x7f05101a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.790+0000 7f0514c6f640 1 -- 172.21.15.7:0/1956201628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05101a1580 msgr2=0x7f05101a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.790+0000 7f0514c6f640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05101a1580 0x7f05101a5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.790+0000 7f0514c6f640 1 -- 172.21.15.7:0/1956201628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051010a900 con 0x7f05101a5e40 2024-04-21T08:54:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.790+0000 7f0514c6f640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05101a5e40 0x7f05101a62f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f050000d750 tx=0x7f050000db50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.791+0000 7f050d7fa640 1 -- 172.21.15.7:0/1956201628 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0500017070 con 0x7f05101a5e40 2024-04-21T08:54:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.791+0000 7f0515470640 1 -- 172.21.15.7:0/1956201628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f051010ab90 con 0x7f05101a5e40 2024-04-21T08:54:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.791+0000 7f0515470640 1 -- 172.21.15.7:0/1956201628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f051010b160 con 0x7f05101a5e40 2024-04-21T08:54:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.791+0000 7f050d7fa640 1 -- 172.21.15.7:0/1956201628 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05000097c0 con 0x7f05101a5e40 2024-04-21T08:54:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.791+0000 7f050d7fa640 1 -- 172.21.15.7:0/1956201628 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0500004c20 con 0x7f05101a5e40 2024-04-21T08:54:32.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.798+0000 7f050d7fa640 1 -- 172.21.15.7:0/1956201628 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0500010460 con 0x7f05101a5e40 2024-04-21T08:54:32.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.799+0000 7f050d7fa640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f04f80a3f50 0x7f04f80a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:32.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.799+0000 7f050d7fa640 1 -- 172.21.15.7:0/1956201628 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7f05000d04d0 con 0x7f05101a5e40 2024-04-21T08:54:32.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.800+0000 7f050ffff640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f04f80a3f50 0x7f04f80a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:32.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.801+0000 7f0515470640 1 -- 172.21.15.7:0/1956201628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04fc0052c0 con 0x7f05101a5e40 2024-04-21T08:54:32.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.807+0000 7f050d7fa640 1 -- 172.21.15.7:0/1956201628 <== mon.2 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f050009d5d0 con 0x7f05101a5e40 2024-04-21T08:54:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:32.852+0000 7f050ffff640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f04f80a3f50 0x7f04f80a6410 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f050802d830 tx=0x7f0508005fd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:33.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:32 smithi016 ceph-mon[31259]: pgmap v310: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 365 KiB/s wr, 108 op/s 2024-04-21T08:54:33.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:32 smithi077 ceph-mon[31615]: pgmap v310: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 365 KiB/s wr, 108 op/s 2024-04-21T08:54:33.153 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/module/sysfs.o 2024-04-21T08:54:33.162 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:32 smithi007 ceph-mon[28477]: pgmap v310: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 365 KiB/s wr, 108 op/s 2024-04-21T08:54:33.236 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cgroup/cgroup.o 2024-04-21T08:54:33.254 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/hrtimer.o 2024-04-21T08:54:33.351 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/mtrr/generic.o 2024-04-21T08:54:33.503 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cgroup/rstat.o 2024-04-21T08:54:33.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.837+0000 7f0515470640 1 -- 172.21.15.7:0/1956201628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04fc005880 con 0x7f05101a5e40 2024-04-21T08:54:33.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.839+0000 7f050d7fa640 1 -- 172.21.15.7:0/1956201628 <== mon.2 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7f05000a1280 con 0x7f05101a5e40 2024-04-21T08:54:33.849 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:54:33.850 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:54:33.850 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:54:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.866+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1956201628 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f04f80a3f50 msgr2=0x7f04f80a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.866+0000 7f04f6ffd640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f04f80a3f50 0x7f04f80a6410 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f050802d830 tx=0x7f0508005fd0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.866+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1956201628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05101a5e40 msgr2=0x7f05101a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.866+0000 7f04f6ffd640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05101a5e40 0x7f05101a62f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f050000d750 tx=0x7f050000db50 comp rx=0 tx=0).stop 2024-04-21T08:54:33.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.872+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1956201628 shutdown_connections 2024-04-21T08:54:33.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.872+0000 7f04f6ffd640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05101a5e40 0x7f05101a62f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.872+0000 7f04f6ffd640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05101a1580 0x7f05101a5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.872+0000 7f04f6ffd640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f04f80a3f50 0x7f04f80a6410 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.872+0000 7f04f6ffd640 1 --2- 172.21.15.7:0/1956201628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f051010e060 0x7f05101a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.872+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1956201628 >> 172.21.15.7:0/1956201628 conn(0x7f051006cf10 msgr2=0x7f051010cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:33.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.897+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1956201628 shutdown_connections 2024-04-21T08:54:33.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:33.904+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1956201628 wait complete. 2024-04-21T08:54:33.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:54:34.224 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mmap.o 2024-04-21T08:54:34.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.429+0000 7fe041bb6640 1 -- 172.21.15.7:0/733071820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03c11b730 msgr2=0x7fe03c11db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:34.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.429+0000 7fe041bb6640 1 --2- 172.21.15.7:0/733071820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03c11b730 0x7fe03c11db20 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fe02c009ca0 tx=0x7fe02c02c140 comp rx=0 tx=0).stop 2024-04-21T08:54:34.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.429+0000 7fe041bb6640 1 -- 172.21.15.7:0/733071820 shutdown_connections 2024-04-21T08:54:34.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.429+0000 7fe041bb6640 1 --2- 172.21.15.7:0/733071820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03c11b730 0x7fe03c11db20 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.429+0000 7fe041bb6640 1 --2- 172.21.15.7:0/733071820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c110b10 0x7fe03c10e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.429+0000 7fe041bb6640 1 --2- 172.21.15.7:0/733071820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe03c1106a0 0x7fe03c10e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.429+0000 7fe041bb6640 1 -- 172.21.15.7:0/733071820 >> 172.21.15.7:0/733071820 conn(0x7fe03c06cff0 msgr2=0x7fe03c06d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:34.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.434+0000 7fe041bb6640 1 -- 172.21.15.7:0/733071820 shutdown_connections 2024-04-21T08:54:34.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.434+0000 7fe041bb6640 1 -- 172.21.15.7:0/733071820 wait complete. 2024-04-21T08:54:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.435+0000 7fe041bb6640 1 Processor -- start 2024-04-21T08:54:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.435+0000 7fe041bb6640 1 -- start start 2024-04-21T08:54:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.435+0000 7fe041bb6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe03c1106a0 0x7fe03c118de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.435+0000 7fe041bb6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03c110b10 0x7fe03c111e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.435+0000 7fe041bb6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c11b730 0x7fe03c1123a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.435+0000 7fe041bb6640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c121090 con 0x7fe03c110b10 2024-04-21T08:54:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.435+0000 7fe041bb6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c120f10 con 0x7fe03c1106a0 2024-04-21T08:54:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.435+0000 7fe041bb6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c121210 con 0x7fe03c11b730 2024-04-21T08:54:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.435+0000 7fe03b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe03c1106a0 0x7fe03c118de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.435+0000 7fe03b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe03c1106a0 0x7fe03c118de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:46238/0 (socket says 172.21.15.7:46238) 2024-04-21T08:54:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.435+0000 7fe03b7fe640 1 -- 172.21.15.7:0/2645675728 learned_addr learned my addr 172.21.15.7:0/2645675728 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.436+0000 7fe03b7fe640 1 -- 172.21.15.7:0/2645675728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c11b730 msgr2=0x7fe03c1123a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.436+0000 7fe03bfff640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c11b730 0x7fe03c1123a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.436+0000 7fe03b7fe640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c11b730 0x7fe03c1123a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.436+0000 7fe03b7fe640 1 -- 172.21.15.7:0/2645675728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03c110b10 msgr2=0x7fe03c111e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:54:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.436+0000 7fe03b7fe640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03c110b10 0x7fe03c111e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.436+0000 7fe03b7fe640 1 -- 172.21.15.7:0/2645675728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03c112a40 con 0x7fe03c1106a0 2024-04-21T08:54:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.436+0000 7fe03bfff640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c11b730 0x7fe03c1123a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.437+0000 7fe03b7fe640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe03c1106a0 0x7fe03c118de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe02400b450 tx=0x7fe02400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.442+0000 7fe038ff9640 1 -- 172.21.15.7:0/2645675728 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe024015020 con 0x7fe03c1106a0 2024-04-21T08:54:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.442+0000 7fe041bb6640 1 -- 172.21.15.7:0/2645675728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe03c116900 con 0x7fe03c1106a0 2024-04-21T08:54:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.442+0000 7fe041bb6640 1 -- 172.21.15.7:0/2645675728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe03c116e50 con 0x7fe03c1106a0 2024-04-21T08:54:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.442+0000 7fe038ff9640 1 -- 172.21.15.7:0/2645675728 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe024011d40 con 0x7fe03c1106a0 2024-04-21T08:54:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.442+0000 7fe038ff9640 1 -- 172.21.15.7:0/2645675728 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02400f340 con 0x7fe03c1106a0 2024-04-21T08:54:34.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.449+0000 7fe038ff9640 1 -- 172.21.15.7:0/2645675728 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fe024018020 con 0x7fe03c1106a0 2024-04-21T08:54:34.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.455+0000 7fe038ff9640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0280a3f20 0x7fe0280a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.455+0000 7fe038ff9640 1 -- 172.21.15.7:0/2645675728 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fe0240d15e0 con 0x7fe03c1106a0 2024-04-21T08:54:34.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.462+0000 7fe03affd640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0280a3f20 0x7fe0280a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.462+0000 7fe0447ab740 1 Processor -- start 2024-04-21T08:54:34.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.462+0000 7fe0447ab740 1 -- start start 2024-04-21T08:54:34.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.462+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 0x561ca9d808b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.462+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d80e80 0x561ca9d89430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.462+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d89a00 0x561ca9d8be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.462+0000 7fe0447ab740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561ca9ce51f0 con 0x561ca9d80e80 2024-04-21T08:54:34.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.462+0000 7fe0447ab740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ca9c824b0 con 0x561ca9d89a00 2024-04-21T08:54:34.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.462+0000 7fe0447ab740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ca9ce11b0 con 0x561ca9d804d0 2024-04-21T08:54:34.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.463+0000 7fe03affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d80e80 0x561ca9d89430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.463+0000 7fe03affd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d80e80 0x561ca9d89430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59050/0 (socket says 172.21.15.7:59050) 2024-04-21T08:54:34.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.463+0000 7fe03affd640 1 -- 172.21.15.7:0/1407793930 learned_addr learned my addr 172.21.15.7:0/1407793930 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:34.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.467+0000 7fe03bfff640 1 --2- 172.21.15.7:0/1407793930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d89a00 0x561ca9d8be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.468+0000 7fe03b7fe640 1 --2- 172.21.15.7:0/1407793930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 0x561ca9d808b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.468+0000 7fe03bfff640 1 -- 172.21.15.7:0/1407793930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 msgr2=0x561ca9d808b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:34.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.468+0000 7fe03bfff640 1 --2- 172.21.15.7:0/1407793930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 0x561ca9d808b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.468+0000 7fe03bfff640 1 -- 172.21.15.7:0/1407793930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d80e80 msgr2=0x561ca9d89430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:34.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.468+0000 7fe03bfff640 1 --2- 172.21.15.7:0/1407793930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d80e80 0x561ca9d89430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.468+0000 7fe03bfff640 1 -- 172.21.15.7:0/1407793930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ca9c741b0 con 0x561ca9d89a00 2024-04-21T08:54:34.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.468+0000 7fe03bfff640 1 --2- 172.21.15.7:0/1407793930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d89a00 0x561ca9d8be50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe03c06b2d0 tx=0x7fe02c009870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80b72ebc71e005ea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:34.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.475+0000 7fe03affd640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0280a3f20 0x7fe0280a63e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x561ca9d7bc10 tx=0x7fe0340090f0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:34.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.476+0000 7fe031ffb640 1 -- 172.21.15.7:0/1407793930 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02c039050 con 0x561ca9d89a00 2024-04-21T08:54:34.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.476+0000 7fe031ffb640 1 -- 172.21.15.7:0/1407793930 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe02c045070 con 0x561ca9d89a00 2024-04-21T08:54:34.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.476+0000 7fe031ffb640 1 -- 172.21.15.7:0/1407793930 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02c004040 con 0x561ca9d89a00 2024-04-21T08:54:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.477+0000 7fe0447ab740 1 -- 172.21.15.7:0/1407793930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d89a00 msgr2=0x561ca9d8be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.477+0000 7fe0447ab740 1 --2- 172.21.15.7:0/1407793930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d89a00 0x561ca9d8be50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe03c06b2d0 tx=0x7fe02c009870 comp rx=0 tx=0).stop 2024-04-21T08:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.482+0000 7fe0447ab740 1 -- 172.21.15.7:0/1407793930 shutdown_connections 2024-04-21T08:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.482+0000 7fe0447ab740 1 --2- 172.21.15.7:0/1407793930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d89a00 0x561ca9d8be50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.482+0000 7fe0447ab740 1 --2- 172.21.15.7:0/1407793930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d80e80 0x561ca9d89430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.482+0000 7fe0447ab740 1 --2- 172.21.15.7:0/1407793930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 0x561ca9d808b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.482+0000 7fe0447ab740 1 -- 172.21.15.7:0/1407793930 >> 172.21.15.7:0/1407793930 conn(0x561ca9d7aba0 msgr2=0x561ca9d7af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.489+0000 7fe0447ab740 1 -- 172.21.15.7:0/1407793930 shutdown_connections 2024-04-21T08:54:34.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.492+0000 7fe0447ab740 1 -- 172.21.15.7:0/1407793930 wait complete. 2024-04-21T08:54:34.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.492+0000 7fe0447ab740 1 -- start start 2024-04-21T08:54:34.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe0447ab740 1 Processor -- start 2024-04-21T08:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d804d0 0x561ca9d89360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 0x561ca9d866f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d89a00 0x561ca9d86d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe0447ab740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561ca9d8c910 con 0x561ca9d89a00 2024-04-21T08:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe0447ab740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ca9c9e500 con 0x561ca9d804d0 2024-04-21T08:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe0447ab740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ca9d8ca90 con 0x561ca9d80e80 2024-04-21T08:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe0447ab740 10 client.0 _ll_register_callbacks cb 0x561ca9c6cb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe03b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d804d0 0x561ca9d89360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe03b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d804d0 0x561ca9d89360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:46252/0 (socket says 172.21.15.7:46252) 2024-04-21T08:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe03b7fe640 1 -- 172.21.15.7:0/3185550505 learned_addr learned my addr 172.21.15.7:0/3185550505 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.493+0000 7fe03affd640 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 0x561ca9d866f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.494+0000 7fe03b7fe640 1 -- 172.21.15.7:0/3185550505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 msgr2=0x561ca9d866f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.494+0000 7fe03b7fe640 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 0x561ca9d866f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.494+0000 7fe03bfff640 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d89a00 0x561ca9d86d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.494+0000 7fe03b7fe640 1 -- 172.21.15.7:0/3185550505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d89a00 msgr2=0x561ca9d86d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.494+0000 7fe03b7fe640 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d89a00 0x561ca9d86d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.494+0000 7fe03b7fe640 1 -- 172.21.15.7:0/3185550505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ca99f73f0 con 0x561ca9d804d0 2024-04-21T08:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.494+0000 7fe03b7fe640 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d804d0 0x561ca9d89360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe02409cec0 tx=0x7fe024010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5298c0e40addc93a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:34.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.509+0000 7fe01affd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:54:34.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.509+0000 7fe01affd640 1 -- 172.21.15.7:0/3185550505 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0240b6070 con 0x561ca9d804d0 2024-04-21T08:54:34.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.509+0000 7fe0447ab740 1 -- 172.21.15.7:0/3185550505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe02c045070 con 0x561ca9d804d0 2024-04-21T08:54:34.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.509+0000 7fe01affd640 1 -- 172.21.15.7:0/3185550505 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fe0240a59a0 con 0x561ca9d804d0 2024-04-21T08:54:34.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.515+0000 7fe01affd640 1 -- 172.21.15.7:0/3185550505 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe0240cece0 con 0x561ca9d804d0 2024-04-21T08:54:34.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.515+0000 7fe0447ab740 10 client.26768 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:34.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.515+0000 7fe0447ab740 1 -- 172.21.15.7:0/3185550505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561ca9ac17d0 con 0x561ca9d804d0 2024-04-21T08:54:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.521+0000 7fe01affd640 1 -- 172.21.15.7:0/3185550505 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe02c045070 con 0x561ca9d804d0 2024-04-21T08:54:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.521+0000 7fe01affd640 1 -- 172.21.15.7:0/3185550505 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fe02409c610 con 0x561ca9d804d0 2024-04-21T08:54:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.521+0000 7fe0447ab740 10 client.26768 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.521+0000 7fe0447ab740 10 client.26768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.521+0000 7fe0447ab740 20 client.26768 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.521+0000 7fe0447ab740 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ca9d9a220 0x561ca9d9c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.521+0000 7fe0447ab740 4 client.26768 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:34.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.521+0000 7fe0447ab740 1 -- 172.21.15.7:0/3185550505 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ca9c84930 con 0x561ca9d9a220 2024-04-21T08:54:34.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.521+0000 7fe03affd640 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ca9d9a220 0x561ca9d9c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.522+0000 7fe01affd640 10 client.26768 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:34.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.522+0000 7fe03affd640 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ca9d9a220 0x561ca9d9c6a0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b348b21806422ce server_cookie=623d323bb666a4b6 in_seq=0 out_seq=0 2024-04-21T08:54:34.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.523+0000 7fe01affd640 1 -- 172.21.15.7:0/3185550505 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ca9c84930 con 0x561ca9d9a220 2024-04-21T08:54:34.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.523+0000 7fe01affd640 10 client.26768 handle_command_reply: tid=0 2024-04-21T08:54:34.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.523+0000 7fe0447ab740 1 client.26768 shutdown 2024-04-21T08:54:34.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe0447ab740 1 -- 172.21.15.7:0/3185550505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d804d0 msgr2=0x561ca9d89360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:34.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe0447ab740 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d804d0 0x561ca9d89360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe02409cec0 tx=0x7fe024010800 comp rx=0 tx=0).stop 2024-04-21T08:54:34.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe0447ab740 1 -- 172.21.15.7:0/3185550505 shutdown_connections 2024-04-21T08:54:34.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe0447ab740 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ca9d9a220 0x561ca9d9c6a0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe0447ab740 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d89a00 0x561ca9d86d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe0447ab740 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 0x561ca9d866f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe0447ab740 1 --2- 172.21.15.7:0/3185550505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d804d0 0x561ca9d89360 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe01affd640 0 client.26768 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:34.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe0447ab740 1 -- 172.21.15.7:0/3185550505 >> 172.21.15.7:0/3185550505 conn(0x561ca9d7aba0 msgr2=0x561ca9d7c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:34.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe0447ab740 1 -- 172.21.15.7:0/3185550505 shutdown_connections 2024-04-21T08:54:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe0447ab740 1 -- 172.21.15.7:0/3185550505 wait complete. 2024-04-21T08:54:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.524+0000 7fe0447ab740 20 client.26768 trim_cache size 0 max 16384 2024-04-21T08:54:34.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.578+0000 7fe0447ab740 1 Processor -- start 2024-04-21T08:54:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.581+0000 7fe0447ab740 1 -- start start 2024-04-21T08:54:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.581+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 0x561ca9d95860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.581+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d80e80 0x561ca9d7c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.581+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d89a00 0x561ca9d88e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.581+0000 7fe03b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 0x561ca9d95860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.581+0000 7fe03b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 0x561ca9d95860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:34548/0 (socket says 172.21.15.7:34548) 2024-04-21T08:54:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.581+0000 7fe03affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d80e80 0x561ca9d7c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.581+0000 7fe03b7fe640 1 -- 172.21.15.7:0/2931306535 learned_addr learned my addr 172.21.15.7:0/2931306535 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.581+0000 7fe0447ab740 1 -- 172.21.15.7:0/2931306535 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe01c00e2a0 con 0x561ca9d89a00 2024-04-21T08:54:34.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.581+0000 7fe0447ab740 1 -- 172.21.15.7:0/2931306535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe01c010b00 con 0x561ca9d80e80 2024-04-21T08:54:34.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.581+0000 7fe0447ab740 1 -- 172.21.15.7:0/2931306535 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe01c00e8f0 con 0x561ca9d804d0 2024-04-21T08:54:34.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.582+0000 7fe03affd640 1 -- 172.21.15.7:0/2931306535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 msgr2=0x561ca9d95860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:34.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.582+0000 7fe03affd640 1 --2- 172.21.15.7:0/2931306535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 0x561ca9d95860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.582+0000 7fe03affd640 1 -- 172.21.15.7:0/2931306535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d89a00 msgr2=0x561ca9d88e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:34.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.582+0000 7fe03affd640 1 --2- 172.21.15.7:0/2931306535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d89a00 0x561ca9d88e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.582+0000 7fe03affd640 1 -- 172.21.15.7:0/2931306535 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ca9d87070 con 0x561ca9d80e80 2024-04-21T08:54:34.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.582+0000 7fe03b7fe640 1 --2- 172.21.15.7:0/2931306535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 0x561ca9d95860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:34.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.582+0000 7fe03affd640 1 --2- 172.21.15.7:0/2931306535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d80e80 0x561ca9d7c700 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x561ca9d85f50 tx=0x7fe034004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d5b739a28988e76 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:34.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.600+0000 7fe0317fa640 1 -- 172.21.15.7:0/2931306535 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03400f040 con 0x561ca9d80e80 2024-04-21T08:54:34.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.600+0000 7fe0317fa640 1 -- 172.21.15.7:0/2931306535 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe034005e20 con 0x561ca9d80e80 2024-04-21T08:54:34.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.604+0000 7fe0317fa640 1 -- 172.21.15.7:0/2931306535 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe034023450 con 0x561ca9d80e80 2024-04-21T08:54:34.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.609+0000 7fe0447ab740 1 -- 172.21.15.7:0/2931306535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d80e80 msgr2=0x561ca9d7c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:34.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.610+0000 7fe0447ab740 1 --2- 172.21.15.7:0/2931306535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d80e80 0x561ca9d7c700 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x561ca9d85f50 tx=0x7fe034004770 comp rx=0 tx=0).stop 2024-04-21T08:54:34.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.611+0000 7fe0447ab740 1 -- 172.21.15.7:0/2931306535 shutdown_connections 2024-04-21T08:54:34.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.611+0000 7fe0447ab740 1 --2- 172.21.15.7:0/2931306535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d89a00 0x561ca9d88e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.611+0000 7fe0447ab740 1 --2- 172.21.15.7:0/2931306535 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d80e80 0x561ca9d7c700 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.611+0000 7fe0447ab740 1 --2- 172.21.15.7:0/2931306535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d804d0 0x561ca9d95860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.611+0000 7fe0447ab740 1 -- 172.21.15.7:0/2931306535 >> 172.21.15.7:0/2931306535 conn(0x561ca9d7aba0 msgr2=0x561ca9d7bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:34.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.611+0000 7fe0447ab740 1 -- 172.21.15.7:0/2931306535 shutdown_connections 2024-04-21T08:54:34.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.611+0000 7fe0447ab740 1 -- 172.21.15.7:0/2931306535 wait complete. 2024-04-21T08:54:34.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.612+0000 7fe0447ab740 1 -- start start 2024-04-21T08:54:34.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.612+0000 7fe0447ab740 1 Processor -- start 2024-04-21T08:54:34.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.612+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d804d0 0x561ca9d91db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.612+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 0x561ca9d922f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.612+0000 7fe0447ab740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d89a00 0x561ca9d8d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.612+0000 7fe0447ab740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe01c00efc0 con 0x561ca9d804d0 2024-04-21T08:54:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.612+0000 7fe0447ab740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe01c010430 con 0x561ca9d89a00 2024-04-21T08:54:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.612+0000 7fe0447ab740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe01c00f690 con 0x561ca9d80e80 2024-04-21T08:54:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.612+0000 7fe0447ab740 10 client.0 _ll_register_callbacks cb 0x7fe02409cbb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.613+0000 7fe03affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 0x561ca9d922f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.613+0000 7fe03affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 0x561ca9d922f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:34550/0 (socket says 172.21.15.7:34550) 2024-04-21T08:54:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.613+0000 7fe03affd640 1 -- 172.21.15.7:0/638767961 learned_addr learned my addr 172.21.15.7:0/638767961 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.613+0000 7fe03bfff640 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d89a00 0x561ca9d8d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.613+0000 7fe03affd640 1 -- 172.21.15.7:0/638767961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d89a00 msgr2=0x561ca9d8d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:34.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.613+0000 7fe03affd640 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d89a00 0x561ca9d8d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.613+0000 7fe03affd640 1 -- 172.21.15.7:0/638767961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d804d0 msgr2=0x561ca9d91db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:34.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.613+0000 7fe03affd640 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d804d0 0x561ca9d91db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.613+0000 7fe03affd640 1 -- 172.21.15.7:0/638767961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ca9d8dcb0 con 0x561ca9d80e80 2024-04-21T08:54:34.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.613+0000 7fe03affd640 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 0x561ca9d922f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe034006c10 tx=0x7fe034024460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fedd0e144a1e66fd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:34.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.623+0000 7fe01affd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:54:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.623+0000 7fe01affd640 1 -- 172.21.15.7:0/638767961 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe034033020 con 0x561ca9d80e80 2024-04-21T08:54:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.623+0000 7fe0447ab740 1 -- 172.21.15.7:0/638767961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe034005e20 con 0x561ca9d80e80 2024-04-21T08:54:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.623+0000 7fe01affd640 1 -- 172.21.15.7:0/638767961 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fe034006410 con 0x561ca9d80e80 2024-04-21T08:54:34.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.627+0000 7fe01affd640 1 -- 172.21.15.7:0/638767961 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe034024930 con 0x561ca9d80e80 2024-04-21T08:54:34.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.627+0000 7fe0447ab740 10 client.26764 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:34.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.627+0000 7fe0447ab740 1 -- 172.21.15.7:0/638767961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561ca9e004f0 con 0x561ca9d80e80 2024-04-21T08:54:34.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.628+0000 7fe01affd640 1 -- 172.21.15.7:0/638767961 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe034005e20 con 0x561ca9d80e80 2024-04-21T08:54:34.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.628+0000 7fe01affd640 1 -- 172.21.15.7:0/638767961 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fe03402fc80 con 0x561ca9d80e80 2024-04-21T08:54:34.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.628+0000 7fe0447ab740 10 client.26764 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:34.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.628+0000 7fe0447ab740 10 client.26764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:34.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.628+0000 7fe0447ab740 20 client.26764 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:34.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.628+0000 7fe0447ab740 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ca9e00bc0 0x561ca9e02fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:34.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.628+0000 7fe0447ab740 4 client.26764 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:54:34.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.628+0000 7fe0447ab740 1 -- 172.21.15.7:0/638767961 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ca9cf2c10 con 0x561ca9e00bc0 2024-04-21T08:54:34.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.628+0000 7fe03b7fe640 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ca9e00bc0 0x561ca9e02fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:34.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.629+0000 7fe01affd640 10 client.26764 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:34.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.629+0000 7fe03b7fe640 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ca9e00bc0 0x561ca9e02fe0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b146635c43491a2 server_cookie=182e1c6315db3812 in_seq=0 out_seq=0 2024-04-21T08:54:34.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.629+0000 7fe01affd640 1 -- 172.21.15.7:0/638767961 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ca9cf2c10 con 0x561ca9e00bc0 2024-04-21T08:54:34.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.629+0000 7fe01affd640 10 client.26764 handle_command_reply: tid=0 2024-04-21T08:54:34.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.630+0000 7fe0447ab740 1 client.26764 shutdown 2024-04-21T08:54:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.642+0000 7fe0447ab740 1 -- 172.21.15.7:0/638767961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 msgr2=0x561ca9d922f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.642+0000 7fe0447ab740 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 0x561ca9d922f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe034006c10 tx=0x7fe034024460 comp rx=0 tx=0).stop 2024-04-21T08:54:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.643+0000 7fe0447ab740 1 -- 172.21.15.7:0/638767961 shutdown_connections 2024-04-21T08:54:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.643+0000 7fe0447ab740 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561ca9e00bc0 0x561ca9e02fe0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.643+0000 7fe0447ab740 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ca9d89a00 0x561ca9d8d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.643+0000 7fe0447ab740 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ca9d80e80 0x561ca9d922f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.643+0000 7fe0447ab740 1 --2- 172.21.15.7:0/638767961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561ca9d804d0 0x561ca9d91db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.643+0000 7fe01affd640 0 client.26764 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.643+0000 7fe0447ab740 1 -- 172.21.15.7:0/638767961 >> 172.21.15.7:0/638767961 conn(0x561ca9d7aba0 msgr2=0x561ca9d7c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.643+0000 7fe0447ab740 1 -- 172.21.15.7:0/638767961 shutdown_connections 2024-04-21T08:54:34.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.643+0000 7fe0447ab740 1 -- 172.21.15.7:0/638767961 wait complete. 2024-04-21T08:54:34.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.643+0000 7fe0447ab740 20 client.26764 trim_cache size 0 max 16384 2024-04-21T08:54:34.645 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:54:34.645 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/module/built-in.a 2024-04-21T08:54:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.663+0000 7fe041bb6640 1 -- 172.21.15.7:0/2645675728 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0280a3f20 msgr2=0x7fe0280a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.663+0000 7fe041bb6640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0280a3f20 0x7fe0280a63e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x561ca9d7bc10 tx=0x7fe0340090f0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.663+0000 7fe041bb6640 1 -- 172.21.15.7:0/2645675728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe03c1106a0 msgr2=0x7fe03c118de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:34.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.663+0000 7fe041bb6640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe03c1106a0 0x7fe03c118de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe02400b450 tx=0x7fe02400b910 comp rx=0 tx=0).stop 2024-04-21T08:54:34.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.668+0000 7fe041bb6640 1 -- 172.21.15.7:0/2645675728 shutdown_connections 2024-04-21T08:54:34.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.668+0000 7fe041bb6640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0280a3f20 0x7fe0280a63e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.668+0000 7fe041bb6640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe03c11b730 0x7fe03c1123a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.668+0000 7fe041bb6640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03c110b10 0x7fe03c111e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.668+0000 7fe041bb6640 1 --2- 172.21.15.7:0/2645675728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe03c1106a0 0x7fe03c118de0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:34.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.668+0000 7fe041bb6640 1 -- 172.21.15.7:0/2645675728 >> 172.21.15.7:0/2645675728 conn(0x7fe03c06cff0 msgr2=0x7fe03c11a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:34.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.674+0000 7fe041bb6640 1 -- 172.21.15.7:0/2645675728 shutdown_connections 2024-04-21T08:54:34.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:34.674+0000 7fe041bb6640 1 -- 172.21.15.7:0/2645675728 wait complete. 2024-04-21T08:54:34.707 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fat/misc.o 2024-04-21T08:54:34.713 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/mtrr/cleanup.o 2024-04-21T08:54:34.786 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mmu_gather.o 2024-04-21T08:54:35.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:34 smithi016 ceph-mon[31259]: pgmap v311: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 398 KiB/s wr, 106 op/s 2024-04-21T08:54:35.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:34 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1956201628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:35.089 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fat/nfs.o 2024-04-21T08:54:35.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:34 smithi077 ceph-mon[31615]: pgmap v311: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 398 KiB/s wr, 106 op/s 2024-04-21T08:54:35.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:34 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1956201628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:35.159 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:34 smithi007 ceph-mon[28477]: pgmap v311: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 398 KiB/s wr, 106 op/s 2024-04-21T08:54:35.160 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:34 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1956201628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:35.494 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/timekeeping.o 2024-04-21T08:54:35.723 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:54:35.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:54:35.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:54:35.823 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fat/namei_vfat.o 2024-04-21T08:54:35.939 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/kernel/cpu/mtrr/built-in.a 2024-04-21T08:54:36.184 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fat/namei_msdos.o 2024-04-21T08:54:36.197 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mprotect.o 2024-04-21T08:54:36.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.265+0000 7fd32a1c3640 1 -- 172.21.15.7:0/2174647008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3241150a0 msgr2=0x7fd324117490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:36.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.265+0000 7fd32a1c3640 1 --2- 172.21.15.7:0/2174647008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3241150a0 0x7fd324117490 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fd318009900 tx=0x7fd31802bc40 comp rx=0 tx=0).stop 2024-04-21T08:54:36.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.265+0000 7fd32a1c3640 1 -- 172.21.15.7:0/2174647008 shutdown_connections 2024-04-21T08:54:36.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.265+0000 7fd32a1c3640 1 --2- 172.21.15.7:0/2174647008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3241150a0 0x7fd324117490 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.265+0000 7fd32a1c3640 1 --2- 172.21.15.7:0/2174647008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd324109a50 0x7fd324114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.265+0000 7fd32a1c3640 1 --2- 172.21.15.7:0/2174647008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd324109130 0x7fd324109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.265+0000 7fd32a1c3640 1 -- 172.21.15.7:0/2174647008 >> 172.21.15.7:0/2174647008 conn(0x7fd32406cff0 msgr2=0x7fd32406d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:36.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.274+0000 7fd32a1c3640 1 -- 172.21.15.7:0/2174647008 shutdown_connections 2024-04-21T08:54:36.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.278+0000 7fd32a1c3640 1 -- 172.21.15.7:0/2174647008 wait complete. 2024-04-21T08:54:36.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.278+0000 7fd32a1c3640 1 Processor -- start 2024-04-21T08:54:36.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.284+0000 7fd32a1c3640 1 -- start start 2024-04-21T08:54:36.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.284+0000 7fd32a1c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd324109130 0x7fd3241a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.284+0000 7fd32a1c3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd324109a50 0x7fd3241a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.284+0000 7fd32a1c3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3241150a0 0x7fd3241a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.284+0000 7fd32a1c3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd32411a9a0 con 0x7fd3241150a0 2024-04-21T08:54:36.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.284+0000 7fd32a1c3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd32411a820 con 0x7fd324109130 2024-04-21T08:54:36.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.284+0000 7fd32a1c3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd32411ab20 con 0x7fd324109a50 2024-04-21T08:54:36.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.288+0000 7fd323fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3241150a0 0x7fd3241a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.288+0000 7fd323fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3241150a0 0x7fd3241a5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51062/0 (socket says 172.21.15.7:51062) 2024-04-21T08:54:36.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.288+0000 7fd323fff640 1 -- 172.21.15.7:0/1302820501 learned_addr learned my addr 172.21.15.7:0/1302820501 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:36.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.296+0000 7fd322ffd640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd324109a50 0x7fd3241a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.296+0000 7fd323fff640 1 -- 172.21.15.7:0/1302820501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd324109a50 msgr2=0x7fd3241a1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:36.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.296+0000 7fd323fff640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd324109a50 0x7fd3241a1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.296+0000 7fd323fff640 1 -- 172.21.15.7:0/1302820501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd324109130 msgr2=0x7fd3241a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:54:36.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.296+0000 7fd323fff640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd324109130 0x7fd3241a0ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.296+0000 7fd323fff640 1 -- 172.21.15.7:0/1302820501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3241a5eb0 con 0x7fd3241150a0 2024-04-21T08:54:36.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.304+0000 7fd323fff640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3241150a0 0x7fd3241a5780 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fd31802c1d0 tx=0x7fd31802c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:36.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.308+0000 7fd320ff9640 1 -- 172.21.15.7:0/1302820501 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd318043020 con 0x7fd3241150a0 2024-04-21T08:54:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.308+0000 7fd320ff9640 1 -- 172.21.15.7:0/1302820501 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3180052b0 con 0x7fd3241150a0 2024-04-21T08:54:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.308+0000 7fd320ff9640 1 -- 172.21.15.7:0/1302820501 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd318003af0 con 0x7fd3241150a0 2024-04-21T08:54:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.308+0000 7fd32a1c3640 1 -- 172.21.15.7:0/1302820501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3241a6140 con 0x7fd3241150a0 2024-04-21T08:54:36.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.308+0000 7fd32a1c3640 1 -- 172.21.15.7:0/1302820501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd32410a8f0 con 0x7fd3241150a0 2024-04-21T08:54:36.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.325+0000 7fd32cdb0740 1 Processor -- start 2024-04-21T08:54:36.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.326+0000 7fd320ff9640 1 -- 172.21.15.7:0/1302820501 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd318046020 con 0x7fd3241150a0 2024-04-21T08:54:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.327+0000 7fd320ff9640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd30c0a3e60 0x7fd30c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.327+0000 7fd320ff9640 1 -- 172.21.15.7:0/1302820501 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fd3180f31f0 con 0x7fd3241150a0 2024-04-21T08:54:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.328+0000 7fd32cdb0740 1 -- start start 2024-04-21T08:54:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.328+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20e40f0 0x55b4c20e44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.328+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 0x55b4c20ed050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.328+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20ed620 0x55b4c20efa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.328+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b4c203ff00 con 0x55b4c20e40f0 2024-04-21T08:54:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.328+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b4c1fdd590 con 0x55b4c20e4aa0 2024-04-21T08:54:36.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.328+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b4c203bec0 con 0x55b4c20ed620 2024-04-21T08:54:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.328+0000 7fd322ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 0x55b4c20ed050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.328+0000 7fd322ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 0x55b4c20ed050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38560/0 (socket says 172.21.15.7:38560) 2024-04-21T08:54:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.328+0000 7fd322ffd640 1 -- 172.21.15.7:0/3759961291 learned_addr learned my addr 172.21.15.7:0/3759961291 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:36.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.329+0000 7fd322ffd640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd30c0a3e60 0x7fd30c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.329+0000 7fd3237fe640 1 --2- 172.21.15.7:0/3759961291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20e40f0 0x55b4c20e44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.329+0000 7fd323fff640 1 --2- 172.21.15.7:0/3759961291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20ed620 0x55b4c20efa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.330+0000 7fd322ffd640 1 -- 172.21.15.7:0/3759961291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20ed620 msgr2=0x55b4c20efa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.330+0000 7fd322ffd640 1 --2- 172.21.15.7:0/3759961291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20ed620 0x55b4c20efa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.330+0000 7fd322ffd640 1 -- 172.21.15.7:0/3759961291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20e40f0 msgr2=0x55b4c20e44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.330+0000 7fd322ffd640 1 --2- 172.21.15.7:0/3759961291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20e40f0 0x55b4c20e44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.330+0000 7fd322ffd640 1 -- 172.21.15.7:0/3759961291 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4c1e1a7e0 con 0x55b4c20e4aa0 2024-04-21T08:54:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.335+0000 7fd322ffd640 1 --2- 172.21.15.7:0/3759961291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 0x55b4c20ed050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd314012a70 tx=0x7fd314012f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abe61a61f00683d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:36.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.339+0000 7fd322ffd640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd30c0a3e60 0x7fd30c0a6320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd31400f000 tx=0x7fd31401f000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.344+0000 7fd309ffb640 1 -- 172.21.15.7:0/3759961291 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3140125f0 con 0x55b4c20e4aa0 2024-04-21T08:54:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.344+0000 7fd309ffb640 1 -- 172.21.15.7:0/3759961291 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd314022070 con 0x55b4c20e4aa0 2024-04-21T08:54:36.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.345+0000 7fd32cdb0740 1 -- 172.21.15.7:0/3759961291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 msgr2=0x55b4c20ed050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:36.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.345+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/3759961291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 0x55b4c20ed050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd314012a70 tx=0x7fd314012f30 comp rx=0 tx=0).stop 2024-04-21T08:54:36.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.346+0000 7fd32cdb0740 1 -- 172.21.15.7:0/3759961291 shutdown_connections 2024-04-21T08:54:36.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.346+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/3759961291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20ed620 0x55b4c20efa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.346+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/3759961291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 0x55b4c20ed050 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.346+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/3759961291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20e40f0 0x55b4c20e44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.346+0000 7fd32cdb0740 1 -- 172.21.15.7:0/3759961291 >> 172.21.15.7:0/3759961291 conn(0x55b4c20de7c0 msgr2=0x55b4c20deba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:36.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.353+0000 7fd32cdb0740 1 -- 172.21.15.7:0/3759961291 shutdown_connections 2024-04-21T08:54:36.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.359+0000 7fd32cdb0740 1 -- 172.21.15.7:0/3759961291 wait complete. 2024-04-21T08:54:36.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.359+0000 7fd32cdb0740 1 -- start start 2024-04-21T08:54:36.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.360+0000 7fd32cdb0740 1 Processor -- start 2024-04-21T08:54:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.360+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c20ea580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.360+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e4aa0 0x55b4c20eaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.360+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 0x55b4c20eb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.360+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b4c20f0580 con 0x55b4c20ed620 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.360+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55b4c1ff9580 con 0x55b4c20e40f0 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.360+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b4c20f0700 con 0x55b4c20e4aa0 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.360+0000 7fd32cdb0740 10 client.0 _ll_register_callbacks cb 0x55b4c1fff650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.361+0000 7fd3237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c20ea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.361+0000 7fd3237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c20ea580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38572/0 (socket says 172.21.15.7:38572) 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.361+0000 7fd3237fe640 1 -- 172.21.15.7:0/2610690388 learned_addr learned my addr 172.21.15.7:0/2610690388 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.361+0000 7fd3237fe640 1 -- 172.21.15.7:0/2610690388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e4aa0 msgr2=0x55b4c20eaac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.361+0000 7fd3237fe640 1 --2- 172.21.15.7:0/2610690388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e4aa0 0x55b4c20eaac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.361+0000 7fd3237fe640 1 -- 172.21.15.7:0/2610690388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 msgr2=0x55b4c20eb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.361+0000 7fd3237fe640 1 --2- 172.21.15.7:0/2610690388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 0x55b4c20eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.361+0000 7fd3237fe640 1 -- 172.21.15.7:0/2610690388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4c1e7af70 con 0x55b4c20e40f0 2024-04-21T08:54:36.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.361+0000 7fd3237fe640 1 --2- 172.21.15.7:0/2610690388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c20ea580 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00f690 tx=0x7fd31c00fb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b917cdf52f48f484 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.367+0000 7fd302ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:54:36.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.367+0000 7fd302ffd640 1 -- 172.21.15.7:0/2610690388 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c00dac0 con 0x55b4c20e40f0 2024-04-21T08:54:36.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.371+0000 7fd32cdb0740 1 -- 172.21.15.7:0/2610690388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd314022070 con 0x55b4c20e40f0 2024-04-21T08:54:36.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.371+0000 7fd302ffd640 1 -- 172.21.15.7:0/2610690388 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fd31c00dc60 con 0x55b4c20e40f0 2024-04-21T08:54:36.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.371+0000 7fd302ffd640 1 -- 172.21.15.7:0/2610690388 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd31c024050 con 0x55b4c20e40f0 2024-04-21T08:54:36.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.377+0000 7fd32cdb0740 10 client.26780 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.377+0000 7fd32cdb0740 1 -- 172.21.15.7:0/2610690388 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b4c1d12170 con 0x55b4c20e40f0 2024-04-21T08:54:36.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.378+0000 7fd302ffd640 1 -- 172.21.15.7:0/2610690388 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd314022070 con 0x55b4c20e40f0 2024-04-21T08:54:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.378+0000 7fd302ffd640 1 -- 172.21.15.7:0/2610690388 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fd31c019420 con 0x55b4c20e40f0 2024-04-21T08:54:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.378+0000 7fd32cdb0740 10 client.26780 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.378+0000 7fd32cdb0740 10 client.26780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.378+0000 7fd32cdb0740 20 client.26780 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.378+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/2610690388 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b4c20d9a80 0x55b4c20dbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.378+0000 7fd32cdb0740 4 client.26780 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.378+0000 7fd32cdb0740 1 -- 172.21.15.7:0/2610690388 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4c204d920 con 0x55b4c20d9a80 2024-04-21T08:54:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.378+0000 7fd322ffd640 1 --2- 172.21.15.7:0/2610690388 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b4c20d9a80 0x55b4c20dbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.379+0000 7fd322ffd640 1 --2- 172.21.15.7:0/2610690388 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b4c20d9a80 0x55b4c20dbf40 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48d354583140babc server_cookie=e78eb22e74710c99 in_seq=0 out_seq=0 2024-04-21T08:54:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.379+0000 7fd302ffd640 10 client.26780 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.381+0000 7fd302ffd640 1 -- 172.21.15.7:0/2610690388 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b4c204d920 con 0x55b4c20d9a80 2024-04-21T08:54:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.381+0000 7fd302ffd640 10 client.26780 handle_command_reply: tid=0 2024-04-21T08:54:36.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.381+0000 7fd32cdb0740 1 client.26780 shutdown 2024-04-21T08:54:36.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.386+0000 7fd32cdb0740 1 -- 172.21.15.7:0/2610690388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 msgr2=0x55b4c20ea580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:36.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.386+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/2610690388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c20ea580 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00f690 tx=0x7fd31c00fb50 comp rx=0 tx=0).stop 2024-04-21T08:54:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.389+0000 7fd32cdb0740 1 -- 172.21.15.7:0/2610690388 shutdown_connections 2024-04-21T08:54:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.389+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/2610690388 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b4c20d9a80 0x55b4c20dbf40 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.389+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/2610690388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 0x55b4c20eb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.389+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/2610690388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e4aa0 0x55b4c20eaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.389+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/2610690388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c20ea580 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.389+0000 7fd32cdb0740 1 -- 172.21.15.7:0/2610690388 >> 172.21.15.7:0/2610690388 conn(0x55b4c20de7c0 msgr2=0x55b4c20e0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.389+0000 7fd302ffd640 0 client.26780 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.389+0000 7fd32cdb0740 1 -- 172.21.15.7:0/2610690388 shutdown_connections 2024-04-21T08:54:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.389+0000 7fd32cdb0740 1 -- 172.21.15.7:0/2610690388 wait complete. 2024-04-21T08:54:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.395+0000 7fd32cdb0740 20 client.26780 trim_cache size 0 max 16384 2024-04-21T08:54:36.418 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/migrate.o 2024-04-21T08:54:36.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.465+0000 7fd32cdb0740 1 Processor -- start 2024-04-21T08:54:36.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.465+0000 7fd32cdb0740 1 -- start start 2024-04-21T08:54:36.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e40f0 0x55b4c20eccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 0x55b4c20f9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 0x55b4c20d4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd31000e330 con 0x55b4c20ed620 2024-04-21T08:54:36.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd310010b90 con 0x55b4c20e4aa0 2024-04-21T08:54:36.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd3237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e40f0 0x55b4c20eccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd31000e980 con 0x55b4c20e40f0 2024-04-21T08:54:36.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd3237fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e40f0 0x55b4c20eccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:41924/0 (socket says 172.21.15.7:41924) 2024-04-21T08:54:36.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd3237fe640 1 -- 172.21.15.7:0/4064809806 learned_addr learned my addr 172.21.15.7:0/4064809806 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd322ffd640 1 --2- 172.21.15.7:0/4064809806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 0x55b4c20f9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd3237fe640 1 -- 172.21.15.7:0/4064809806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 msgr2=0x55b4c20f9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd3237fe640 1 --2- 172.21.15.7:0/4064809806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 0x55b4c20f9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd3237fe640 1 -- 172.21.15.7:0/4064809806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 msgr2=0x55b4c20d4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.468+0000 7fd3237fe640 1 --2- 172.21.15.7:0/4064809806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 0x55b4c20d4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.469+0000 7fd3237fe640 1 -- 172.21.15.7:0/4064809806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4c20e8ad0 con 0x55b4c20e40f0 2024-04-21T08:54:36.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.469+0000 7fd3237fe640 1 --2- 172.21.15.7:0/4064809806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e40f0 0x55b4c20eccb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00f660 tx=0x7fd31c00ad10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=665f8072f657ab7f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:36.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.475+0000 7fd3097fa640 1 -- 172.21.15.7:0/4064809806 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c02f030 con 0x55b4c20e40f0 2024-04-21T08:54:36.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.475+0000 7fd3097fa640 1 -- 172.21.15.7:0/4064809806 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd31c01ae90 con 0x55b4c20e40f0 2024-04-21T08:54:36.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.476+0000 7fd32cdb0740 1 -- 172.21.15.7:0/4064809806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e40f0 msgr2=0x55b4c20eccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:36.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.476+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/4064809806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e40f0 0x55b4c20eccb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00f660 tx=0x7fd31c00ad10 comp rx=0 tx=0).stop 2024-04-21T08:54:36.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.477+0000 7fd32cdb0740 1 -- 172.21.15.7:0/4064809806 shutdown_connections 2024-04-21T08:54:36.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.477+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/4064809806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 0x55b4c20d4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.477+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/4064809806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e4aa0 0x55b4c20f9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.477+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/4064809806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e40f0 0x55b4c20eccb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.477+0000 7fd32cdb0740 1 -- 172.21.15.7:0/4064809806 >> 172.21.15.7:0/4064809806 conn(0x55b4c20de7c0 msgr2=0x55b4c20d4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:36.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.479+0000 7fd32cdb0740 1 -- 172.21.15.7:0/4064809806 shutdown_connections 2024-04-21T08:54:36.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.479+0000 7fd32cdb0740 1 -- 172.21.15.7:0/4064809806 wait complete. 2024-04-21T08:54:36.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.483+0000 7fd32cdb0740 1 -- start start 2024-04-21T08:54:36.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.483+0000 7fd32cdb0740 1 Processor -- start 2024-04-21T08:54:36.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c210f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e4aa0 0x55b4c210f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd32cdb0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 0x55b4c20f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd3237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c210f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd31000f050 con 0x55b4c20ed620 2024-04-21T08:54:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd3100104c0 con 0x55b4c20e40f0 2024-04-21T08:54:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd32cdb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd31000f720 con 0x55b4c20e4aa0 2024-04-21T08:54:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd3237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c210f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38590/0 (socket says 172.21.15.7:38590) 2024-04-21T08:54:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd3237fe640 1 -- 172.21.15.7:0/1316299819 learned_addr learned my addr 172.21.15.7:0/1316299819 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd32cdb0740 10 client.0 _ll_register_callbacks cb 0x7fd31c00a4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd323fff640 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 0x55b4c20f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.484+0000 7fd322ffd640 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e4aa0 0x55b4c210f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.485+0000 7fd3237fe640 1 -- 172.21.15.7:0/1316299819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e4aa0 msgr2=0x55b4c210f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.485+0000 7fd3237fe640 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e4aa0 0x55b4c210f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.485+0000 7fd3237fe640 1 -- 172.21.15.7:0/1316299819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 msgr2=0x55b4c20f3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.485+0000 7fd3237fe640 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 0x55b4c20f3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.485+0000 7fd3237fe640 1 -- 172.21.15.7:0/1316299819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4c20f44a0 con 0x55b4c20e40f0 2024-04-21T08:54:36.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.485+0000 7fd3237fe640 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c210f3e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0199b0 tx=0x7fd31c019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12bca5a3f34cede server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:36.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.491+0000 7fd302ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:54:36.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.491+0000 7fd302ffd640 1 -- 172.21.15.7:0/1316299819 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c039070 con 0x55b4c20e40f0 2024-04-21T08:54:36.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.499+0000 7fd302ffd640 1 -- 172.21.15.7:0/1316299819 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fd31c00dce0 con 0x55b4c20e40f0 2024-04-21T08:54:36.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.499+0000 7fd32cdb0740 1 -- 172.21.15.7:0/1316299819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd31c01ae90 con 0x55b4c20e40f0 2024-04-21T08:54:36.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.504+0000 7fd302ffd640 1 -- 172.21.15.7:0/1316299819 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fd31c045050 con 0x55b4c20e40f0 2024-04-21T08:54:36.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.504+0000 7fd32cdb0740 10 client.26786 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:36.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.504+0000 7fd32cdb0740 1 -- 172.21.15.7:0/1316299819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b4c216c9e0 con 0x55b4c20e40f0 2024-04-21T08:54:36.510 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/microcode/core.o 2024-04-21T08:54:36.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.514+0000 7fd302ffd640 1 -- 172.21.15.7:0/1316299819 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd31c01ae90 con 0x55b4c20e40f0 2024-04-21T08:54:36.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.514+0000 7fd302ffd640 1 -- 172.21.15.7:0/1316299819 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fd31c004510 con 0x55b4c20e40f0 2024-04-21T08:54:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.514+0000 7fd32cdb0740 10 client.26786 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.514+0000 7fd32cdb0740 10 client.26786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.514+0000 7fd32cdb0740 20 client.26786 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.514+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b4c216d040 0x55b4c216f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.514+0000 7fd32cdb0740 4 client.26786 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:54:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.514+0000 7fd32cdb0740 1 -- 172.21.15.7:0/1316299819 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b4c2047790 con 0x55b4c216d040 2024-04-21T08:54:36.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.514+0000 7fd322ffd640 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b4c216d040 0x55b4c216f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:36.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.517+0000 7fd302ffd640 10 client.26786 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:36.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.517+0000 7fd322ffd640 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b4c216d040 0x55b4c216f460 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ab1f24dc66f582d server_cookie=f9d34a67d62a8c75 in_seq=0 out_seq=0 2024-04-21T08:54:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.522+0000 7fd302ffd640 1 -- 172.21.15.7:0/1316299819 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b4c2047790 con 0x55b4c216d040 2024-04-21T08:54:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.522+0000 7fd302ffd640 10 client.26786 handle_command_reply: tid=0 2024-04-21T08:54:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.522+0000 7fd32cdb0740 1 client.26786 shutdown 2024-04-21T08:54:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.525+0000 7fd32cdb0740 1 -- 172.21.15.7:0/1316299819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 msgr2=0x55b4c210f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.525+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c210f3e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0199b0 tx=0x7fd31c019650 comp rx=0 tx=0).stop 2024-04-21T08:54:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.525+0000 7fd32cdb0740 1 -- 172.21.15.7:0/1316299819 shutdown_connections 2024-04-21T08:54:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.525+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55b4c216d040 0x55b4c216f460 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.525+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b4c20ed620 0x55b4c20f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.525+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b4c20e4aa0 0x55b4c210f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.525+0000 7fd302ffd640 0 client.26786 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.525+0000 7fd32cdb0740 1 --2- 172.21.15.7:0/1316299819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55b4c20e40f0 0x55b4c210f3e0 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0199b0 tx=0x7fd31c019650 comp rx=0 tx=0).stop 2024-04-21T08:54:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.525+0000 7fd32cdb0740 1 -- 172.21.15.7:0/1316299819 >> 172.21.15.7:0/1316299819 conn(0x55b4c20de7c0 msgr2=0x55b4c20d5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.539+0000 7fd32cdb0740 1 -- 172.21.15.7:0/1316299819 shutdown_connections 2024-04-21T08:54:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.539+0000 7fd32cdb0740 1 -- 172.21.15.7:0/1316299819 wait complete. 2024-04-21T08:54:36.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.545+0000 7fd32cdb0740 20 client.26786 trim_cache size 0 max 16384 2024-04-21T08:54:36.553 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:54:36.553 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-04-21T08:54:36.553 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "08447483-b5f8-4de7-a923-832d7f72e5ac", 2024-04-21T08:54:36.554 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-04-21T08:54:36.554 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:54:36.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.559+0000 7fd32a1c3640 1 -- 172.21.15.7:0/1302820501 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd30c0a3e60 msgr2=0x7fd30c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:36.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.559+0000 7fd32a1c3640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd30c0a3e60 0x7fd30c0a6320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd31400f000 tx=0x7fd31401f000 comp rx=0 tx=0).stop 2024-04-21T08:54:36.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.565+0000 7fd32a1c3640 1 -- 172.21.15.7:0/1302820501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3241150a0 msgr2=0x7fd3241a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:36.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.565+0000 7fd32a1c3640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3241150a0 0x7fd3241a5780 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fd31802c1d0 tx=0x7fd31802c9c0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.571+0000 7fd32a1c3640 1 -- 172.21.15.7:0/1302820501 shutdown_connections 2024-04-21T08:54:36.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.571+0000 7fd32a1c3640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3241150a0 0x7fd3241a5780 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.571+0000 7fd32a1c3640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd30c0a3e60 0x7fd30c0a6320 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.571+0000 7fd32a1c3640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd324109a50 0x7fd3241a1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.571+0000 7fd32a1c3640 1 --2- 172.21.15.7:0/1302820501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd324109130 0x7fd3241a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:36.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.571+0000 7fd32a1c3640 1 -- 172.21.15.7:0/1302820501 >> 172.21.15.7:0/1302820501 conn(0x7fd32406cff0 msgr2=0x7fd324070bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:36.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.573+0000 7fd32a1c3640 1 -- 172.21.15.7:0/1302820501 shutdown_connections 2024-04-21T08:54:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:36.578+0000 7fd32a1c3640 1 -- 172.21.15.7:0/1302820501 wait complete. 2024-04-21T08:54:36.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:54:36.608 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:54:36.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:36 smithi007 ceph-mon[28477]: pgmap v312: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 197 KiB/s wr, 65 op/s 2024-04-21T08:54:36.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:36 smithi007 ceph-mon[28477]: scrub queued for path: / 2024-04-21T08:54:36.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:36 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:36.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:36 smithi007 ceph-mon[28477]: scrub summary: active paths [/] 2024-04-21T08:54:37.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:36 smithi016 ceph-mon[31259]: pgmap v312: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 197 KiB/s wr, 65 op/s 2024-04-21T08:54:37.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:36 smithi016 ceph-mon[31259]: scrub queued for path: / 2024-04-21T08:54:37.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:36 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:37.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:36 smithi016 ceph-mon[31259]: scrub summary: active paths [/] 2024-04-21T08:54:37.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:36 smithi077 ceph-mon[31615]: pgmap v312: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 197 KiB/s wr, 65 op/s 2024-04-21T08:54:37.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:36 smithi077 ceph-mon[31615]: scrub queued for path: / 2024-04-21T08:54:37.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:36 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:37.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:36 smithi077 ceph-mon[31615]: scrub summary: active paths [/] 2024-04-21T08:54:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.189+0000 7f79086f2640 1 -- 172.21.15.7:0/2081554575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790010a8f0 msgr2=0x7f790010acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.189+0000 7f79086f2640 1 --2- 172.21.15.7:0/2081554575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790010a8f0 0x7f790010acd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009930 tx=0x7f78fc02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.189+0000 7f79086f2640 1 -- 172.21.15.7:0/2081554575 shutdown_connections 2024-04-21T08:54:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.189+0000 7f79086f2640 1 --2- 172.21.15.7:0/2081554575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7900113ea0 0x7f7900116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.189+0000 7f79086f2640 1 --2- 172.21.15.7:0/2081554575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f790010b210 0x7f7900113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.189+0000 7f79086f2640 1 --2- 172.21.15.7:0/2081554575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790010a8f0 0x7f790010acd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.189+0000 7f79086f2640 1 -- 172.21.15.7:0/2081554575 >> 172.21.15.7:0/2081554575 conn(0x7f790006ced0 msgr2=0x7f790006d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.193+0000 7f79086f2640 1 -- 172.21.15.7:0/2081554575 shutdown_connections 2024-04-21T08:54:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.193+0000 7f79086f2640 1 -- 172.21.15.7:0/2081554575 wait complete. 2024-04-21T08:54:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.194+0000 7f79086f2640 1 Processor -- start 2024-04-21T08:54:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.194+0000 7f79086f2640 1 -- start start 2024-04-21T08:54:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f79086f2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790010b210 0x7f79001a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f79086f2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7900113ea0 0x7f79001a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f79086f2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f790019e6e0 0x7f790019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f79086f2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7900118660 con 0x7f790019e6e0 2024-04-21T08:54:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f79086f2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79001184e0 con 0x7f790010b210 2024-04-21T08:54:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f79086f2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f79001187e0 con 0x7f7900113ea0 2024-04-21T08:54:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f790648e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790010b210 0x7f79001a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f7905c8d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7900113ea0 0x7f79001a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f7905c8d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7900113ea0 0x7f79001a4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:41960/0 (socket says 172.21.15.7:41960) 2024-04-21T08:54:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f7905c8d640 1 -- 172.21.15.7:0/974750229 learned_addr learned my addr 172.21.15.7:0/974750229 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f7905c8d640 1 -- 172.21.15.7:0/974750229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790010b210 msgr2=0x7f79001a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f7905c8d640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790010b210 0x7f79001a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f7905c8d640 1 -- 172.21.15.7:0/974750229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f790019e6e0 msgr2=0x7f790019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f7905c8d640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f790019e6e0 0x7f790019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.195+0000 7f7905c8d640 1 -- 172.21.15.7:0/974750229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790019f3d0 con 0x7f7900113ea0 2024-04-21T08:54:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.196+0000 7f7905c8d640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7900113ea0 0x7f79001a4b60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f78f800ba50 tx=0x7f78f800bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.201+0000 7f78f77fe640 1 -- 172.21.15.7:0/974750229 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f8019070 con 0x7f7900113ea0 2024-04-21T08:54:37.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.201+0000 7f79086f2640 1 -- 172.21.15.7:0/974750229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79001ab3c0 con 0x7f7900113ea0 2024-04-21T08:54:37.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.201+0000 7f79086f2640 1 -- 172.21.15.7:0/974750229 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79001ab820 con 0x7f7900113ea0 2024-04-21T08:54:37.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.201+0000 7f78f77fe640 1 -- 172.21.15.7:0/974750229 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78f800cb60 con 0x7f7900113ea0 2024-04-21T08:54:37.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.201+0000 7f78f77fe640 1 -- 172.21.15.7:0/974750229 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f80075d0 con 0x7f7900113ea0 2024-04-21T08:54:37.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.206+0000 7f790b3c2740 1 Processor -- start 2024-04-21T08:54:37.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.206+0000 7f78f77fe640 1 -- 172.21.15.7:0/974750229 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f78f801c070 con 0x7f7900113ea0 2024-04-21T08:54:37.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.206+0000 7f790b3c2740 1 -- start start 2024-04-21T08:54:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.207+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3d94180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.207+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d94750 0x556fe3d808e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.207+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d80f00 0x556fe3d81360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.207+0000 7f790b3c2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556fe3ceb1e0 con 0x556fe3d80f00 2024-04-21T08:54:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.207+0000 7f790b3c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556fe3c88390 con 0x556fe3d94750 2024-04-21T08:54:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.207+0000 7f790b3c2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556fe3ce71a0 con 0x556fe3d93da0 2024-04-21T08:54:37.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.207+0000 7f7906c8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3d94180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.207+0000 7f7906c8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3d94180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:41968/0 (socket says 172.21.15.7:41968) 2024-04-21T08:54:37.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.207+0000 7f7906c8f640 1 -- 172.21.15.7:0/4034881847 learned_addr learned my addr 172.21.15.7:0/4034881847 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:37.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.207+0000 7f790648e640 1 --2- 172.21.15.7:0/4034881847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d94750 0x556fe3d808e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.207+0000 7f7905c8d640 1 --2- 172.21.15.7:0/4034881847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d80f00 0x556fe3d81360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.208+0000 7f7906c8f640 1 -- 172.21.15.7:0/4034881847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d94750 msgr2=0x556fe3d808e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.208+0000 7f7906c8f640 1 --2- 172.21.15.7:0/4034881847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d94750 0x556fe3d808e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.208+0000 7f7906c8f640 1 -- 172.21.15.7:0/4034881847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d80f00 msgr2=0x556fe3d81360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.208+0000 7f7906c8f640 1 --2- 172.21.15.7:0/4034881847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d80f00 0x556fe3d81360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.208+0000 7f7906c8f640 1 -- 172.21.15.7:0/4034881847 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556fe3ac77d0 con 0x556fe3d93da0 2024-04-21T08:54:37.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.208+0000 7f7906c8f640 1 --2- 172.21.15.7:0/4034881847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3d94180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f78f000bbf0 tx=0x7f78f000b380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b69458052fef0a90 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.208+0000 7f78f77fe640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78ec0a3f20 0x7f78ec0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.208+0000 7f78f77fe640 1 -- 172.21.15.7:0/974750229 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7f78f80cfe20 con 0x7f7900113ea0 2024-04-21T08:54:37.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.209+0000 7f790648e640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78ec0a3f20 0x7f78ec0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.218+0000 7f78f4ff9640 1 -- 172.21.15.7:0/4034881847 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f0009bd0 con 0x556fe3d93da0 2024-04-21T08:54:37.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.218+0000 7f78f4ff9640 1 -- 172.21.15.7:0/4034881847 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78f0009d70 con 0x556fe3d93da0 2024-04-21T08:54:37.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.219+0000 7f790b3c2740 1 -- 172.21.15.7:0/4034881847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 msgr2=0x556fe3d94180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.219+0000 7f790b3c2740 1 --2- 172.21.15.7:0/4034881847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3d94180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f78f000bbf0 tx=0x7f78f000b380 comp rx=0 tx=0).stop 2024-04-21T08:54:37.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.219+0000 7f790b3c2740 1 -- 172.21.15.7:0/4034881847 shutdown_connections 2024-04-21T08:54:37.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.219+0000 7f790b3c2740 1 --2- 172.21.15.7:0/4034881847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d80f00 0x556fe3d81360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.219+0000 7f790b3c2740 1 --2- 172.21.15.7:0/4034881847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d94750 0x556fe3d808e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.219+0000 7f790b3c2740 1 --2- 172.21.15.7:0/4034881847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3d94180 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.219+0000 7f790b3c2740 1 -- 172.21.15.7:0/4034881847 >> 172.21.15.7:0/4034881847 conn(0x556fe3d833a0 msgr2=0x556fe3d83780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.219+0000 7f790b3c2740 1 -- 172.21.15.7:0/4034881847 shutdown_connections 2024-04-21T08:54:37.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.219+0000 7f790b3c2740 1 -- 172.21.15.7:0/4034881847 wait complete. 2024-04-21T08:54:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.220+0000 7f790b3c2740 1 -- start start 2024-04-21T08:54:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.220+0000 7f790b3c2740 1 Processor -- start 2024-04-21T08:54:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.220+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 0x556fe3d97e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.220+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d93da0 0x556fe3d98340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.220+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d94750 0x556fe3d98880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.220+0000 7f790b3c2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556fe3d9b230 con 0x556fe3d93da0 2024-04-21T08:54:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.220+0000 7f790b3c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556fe3ca42f0 con 0x556fe3d80f00 2024-04-21T08:54:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.220+0000 7f790b3c2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556fe3d9b3b0 con 0x556fe3d94750 2024-04-21T08:54:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.220+0000 7f790b3c2740 10 client.0 _ll_register_callbacks cb 0x556fe3cb1580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.221+0000 7f7906c8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d94750 0x556fe3d98880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.221+0000 7f7906c8f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d94750 0x556fe3d98880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:41980/0 (socket says 172.21.15.7:41980) 2024-04-21T08:54:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.221+0000 7f7906c8f640 1 -- 172.21.15.7:0/999165178 learned_addr learned my addr 172.21.15.7:0/999165178 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.221+0000 7f790648e640 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 0x556fe3d97e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.221+0000 7f7906c8f640 1 -- 172.21.15.7:0/999165178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 msgr2=0x556fe3d97e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.221+0000 7f7906c8f640 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 0x556fe3d97e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.221+0000 7f7906c8f640 1 -- 172.21.15.7:0/999165178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d93da0 msgr2=0x556fe3d98340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.221+0000 7f7906c8f640 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d93da0 0x556fe3d98340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.221+0000 7f7906c8f640 1 -- 172.21.15.7:0/999165178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556fe39fd3f0 con 0x556fe3d94750 2024-04-21T08:54:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.221+0000 7f7906c8f640 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d94750 0x556fe3d98880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009e10 tx=0x7f78f0009e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=391018b2bb4b7a93 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.230+0000 7f790648e640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78ec0a3f20 0x7f78ec0a63e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009900 tx=0x7f78fc005fd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.238+0000 7f78e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:54:37.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.238+0000 7f78e9ffb640 1 -- 172.21.15.7:0/999165178 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f0023020 con 0x556fe3d94750 2024-04-21T08:54:37.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.238+0000 7f790b3c2740 1 -- 172.21.15.7:0/999165178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556fe3cf8c00 con 0x556fe3d94750 2024-04-21T08:54:37.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.238+0000 7f78e9ffb640 1 -- 172.21.15.7:0/999165178 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7f78f0003800 con 0x556fe3d94750 2024-04-21T08:54:37.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.238+0000 7f78e9ffb640 1 -- 172.21.15.7:0/999165178 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f78f002e050 con 0x556fe3d94750 2024-04-21T08:54:37.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.238+0000 7f790b3c2740 10 client.26791 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:37.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.238+0000 7f790b3c2740 1 -- 172.21.15.7:0/999165178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556fe39bd0f0 con 0x556fe3d94750 2024-04-21T08:54:37.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.238+0000 7f78e9ffb640 1 -- 172.21.15.7:0/999165178 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556fe3cf8c00 con 0x556fe3d94750 2024-04-21T08:54:37.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.245+0000 7f78e9ffb640 1 -- 172.21.15.7:0/999165178 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f78f001f9d0 con 0x556fe3d94750 2024-04-21T08:54:37.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.245+0000 7f790b3c2740 10 client.26791 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:37.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.245+0000 7f790b3c2740 10 client.26791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:37.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.245+0000 7f790b3c2740 20 client.26791 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:37.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.245+0000 7f790b3c2740 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556fe3da8600 0x556fe3daa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.245+0000 7f790b3c2740 4 client.26791 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:37.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.245+0000 7f790b3c2740 1 -- 172.21.15.7:0/999165178 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556fe3cf2a70 con 0x556fe3da8600 2024-04-21T08:54:37.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.245+0000 7f790648e640 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556fe3da8600 0x556fe3daa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.246+0000 7f78e9ffb640 10 client.26791 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:37.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.246+0000 7f790648e640 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556fe3da8600 0x556fe3daa9f0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43e4f3b4c8e51bea server_cookie=18edab20d3490b4 in_seq=0 out_seq=0 2024-04-21T08:54:37.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.246+0000 7f78e9ffb640 1 -- 172.21.15.7:0/999165178 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556fe3cf2a70 con 0x556fe3da8600 2024-04-21T08:54:37.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.246+0000 7f78e9ffb640 10 client.26791 handle_command_reply: tid=0 2024-04-21T08:54:37.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.247+0000 7f790b3c2740 1 client.26791 shutdown 2024-04-21T08:54:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.252+0000 7f790b3c2740 1 -- 172.21.15.7:0/999165178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d94750 msgr2=0x556fe3d98880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.252+0000 7f790b3c2740 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d94750 0x556fe3d98880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009e10 tx=0x7f78f0009e40 comp rx=0 tx=0).stop 2024-04-21T08:54:37.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.261+0000 7f790b3c2740 1 -- 172.21.15.7:0/999165178 shutdown_connections 2024-04-21T08:54:37.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.261+0000 7f790b3c2740 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556fe3da8600 0x556fe3daa9f0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.261+0000 7f790b3c2740 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d94750 0x556fe3d98880 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.261+0000 7f790b3c2740 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d93da0 0x556fe3d98340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.261+0000 7f790b3c2740 1 --2- 172.21.15.7:0/999165178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 0x556fe3d97e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.261+0000 7f790b3c2740 1 -- 172.21.15.7:0/999165178 >> 172.21.15.7:0/999165178 conn(0x556fe3d833a0 msgr2=0x556fe3d82b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.261+0000 7f78e9ffb640 0 client.26791 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:37.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.262+0000 7f790b3c2740 1 -- 172.21.15.7:0/999165178 shutdown_connections 2024-04-21T08:54:37.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.262+0000 7f790b3c2740 1 -- 172.21.15.7:0/999165178 wait complete. 2024-04-21T08:54:37.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.263+0000 7f790b3c2740 20 client.26791 trim_cache size 0 max 16384 2024-04-21T08:54:37.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.339+0000 7f790b3c2740 1 Processor -- start 2024-04-21T08:54:37.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.353+0000 7f790b3c2740 1 -- start start 2024-04-21T08:54:37.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.353+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 0x556fe3d9a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.353+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3da41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.353+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d94750 0x556fe3d986a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.353+0000 7f790b3c2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f78e400e330 con 0x556fe3d94750 2024-04-21T08:54:37.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.353+0000 7f790b3c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78e4010b90 con 0x556fe3d80f00 2024-04-21T08:54:37.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.353+0000 7f790b3c2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f78e400e980 con 0x556fe3d93da0 2024-04-21T08:54:37.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.355+0000 7f790648e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 0x556fe3d9a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.355+0000 7f790648e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 0x556fe3d9a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38658/0 (socket says 172.21.15.7:38658) 2024-04-21T08:54:37.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.355+0000 7f790648e640 1 -- 172.21.15.7:0/2157311408 learned_addr learned my addr 172.21.15.7:0/2157311408 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:37.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.355+0000 7f790648e640 1 -- 172.21.15.7:0/2157311408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 msgr2=0x556fe3da41b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:37.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.355+0000 7f790648e640 1 --2- 172.21.15.7:0/2157311408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3da41b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.355+0000 7f790648e640 1 -- 172.21.15.7:0/2157311408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d94750 msgr2=0x556fe3d986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.355+0000 7f790648e640 1 --2- 172.21.15.7:0/2157311408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d94750 0x556fe3d986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.355+0000 7f790648e640 1 -- 172.21.15.7:0/2157311408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556fe3d9ab80 con 0x556fe3d80f00 2024-04-21T08:54:37.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.355+0000 7f790648e640 1 --2- 172.21.15.7:0/2157311408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 0x556fe3d9a530 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009850 tx=0x7f78fc049000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69130a379dbd0e91 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.361+0000 7f78ebfff640 1 -- 172.21.15.7:0/2157311408 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc02c730 con 0x556fe3d80f00 2024-04-21T08:54:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.361+0000 7f78ebfff640 1 -- 172.21.15.7:0/2157311408 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78fc05c070 con 0x556fe3d80f00 2024-04-21T08:54:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.362+0000 7f790b3c2740 1 -- 172.21.15.7:0/2157311408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 msgr2=0x556fe3d9a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.362+0000 7f790b3c2740 1 --2- 172.21.15.7:0/2157311408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 0x556fe3d9a530 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009850 tx=0x7f78fc049000 comp rx=0 tx=0).stop 2024-04-21T08:54:37.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.362+0000 7f790b3c2740 1 -- 172.21.15.7:0/2157311408 shutdown_connections 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.362+0000 7f790b3c2740 1 --2- 172.21.15.7:0/2157311408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3d94750 0x556fe3d986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.362+0000 7f790b3c2740 1 --2- 172.21.15.7:0/2157311408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3da41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.362+0000 7f790b3c2740 1 --2- 172.21.15.7:0/2157311408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d80f00 0x556fe3d9a530 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009850 tx=0x7f78fc049000 comp rx=0 tx=0).stop 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.362+0000 7f790b3c2740 1 -- 172.21.15.7:0/2157311408 >> 172.21.15.7:0/2157311408 conn(0x556fe3d833a0 msgr2=0x556fe3d92eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.363+0000 7f790b3c2740 1 -- 172.21.15.7:0/2157311408 shutdown_connections 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.363+0000 7f790b3c2740 1 -- 172.21.15.7:0/2157311408 wait complete. 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.363+0000 7f790b3c2740 1 -- start start 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.364+0000 7f790b3c2740 1 Processor -- start 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.364+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3da0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.364+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d94750 0x556fe3daf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.364+0000 7f790b3c2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3daf660 0x556fe3dafae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.364+0000 7f790648e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3da0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.364+0000 7f790b3c2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f78e400f050 con 0x556fe3daf660 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.364+0000 7f790b3c2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78e40104c0 con 0x556fe3d94750 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.364+0000 7f790b3c2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f78e400f720 con 0x556fe3d93da0 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.364+0000 7f790648e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3da0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:42010/0 (socket says 172.21.15.7:42010) 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.364+0000 7f790648e640 1 -- 172.21.15.7:0/2588152112 learned_addr learned my addr 172.21.15.7:0/2588152112 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.364+0000 7f790b3c2740 10 client.0 _ll_register_callbacks cb 0x7f78fc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.365+0000 7f7905c8d640 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d94750 0x556fe3daf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.365+0000 7f790648e640 1 -- 172.21.15.7:0/2588152112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d94750 msgr2=0x556fe3daf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.365+0000 7f790648e640 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d94750 0x556fe3daf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.365+0000 7f790648e640 1 -- 172.21.15.7:0/2588152112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3daf660 msgr2=0x556fe3dafae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.365+0000 7f790648e640 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3daf660 0x556fe3dafae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.365+0000 7f790648e640 1 -- 172.21.15.7:0/2588152112 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556fe3e0d910 con 0x556fe3d93da0 2024-04-21T08:54:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.365+0000 7f790648e640 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3da0990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009930 tx=0x7f78fc005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1576feaf18b98341 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.370+0000 7f78e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:54:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.370+0000 7f78e9ffb640 1 -- 172.21.15.7:0/2588152112 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc0563b0 con 0x556fe3d93da0 2024-04-21T08:54:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.370+0000 7f78e9ffb640 1 -- 172.21.15.7:0/2588152112 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7f78fc056550 con 0x556fe3d93da0 2024-04-21T08:54:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.370+0000 7f790b3c2740 1 -- 172.21.15.7:0/2588152112 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556fe3c6e590 con 0x556fe3d93da0 2024-04-21T08:54:37.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.384+0000 7f78e9ffb640 1 -- 172.21.15.7:0/2588152112 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f78fc056750 con 0x556fe3d93da0 2024-04-21T08:54:37.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.384+0000 7f790b3c2740 10 client.26794 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:37.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.384+0000 7f790b3c2740 1 -- 172.21.15.7:0/2588152112 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556fe3e0ec70 con 0x556fe3d93da0 2024-04-21T08:54:37.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.384+0000 7f78e9ffb640 1 -- 172.21.15.7:0/2588152112 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78fc05c070 con 0x556fe3d93da0 2024-04-21T08:54:37.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.384+0000 7f78e9ffb640 1 -- 172.21.15.7:0/2588152112 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f78fc003680 con 0x556fe3d93da0 2024-04-21T08:54:37.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.384+0000 7f790b3c2740 10 client.26794 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:37.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.384+0000 7f790b3c2740 10 client.26794 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:37.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.384+0000 7f790b3c2740 20 client.26794 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:37.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.384+0000 7f790b3c2740 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556fe3e0f270 0x556fe3e11690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.384+0000 7f790b3c2740 4 client.26794 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:54:37.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.384+0000 7f790b3c2740 1 -- 172.21.15.7:0/2588152112 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556fe3c6e590 con 0x556fe3e0f270 2024-04-21T08:54:37.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.385+0000 7f7905c8d640 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556fe3e0f270 0x556fe3e11690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.385+0000 7f78e9ffb640 10 client.26794 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:37.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.390+0000 7f7905c8d640 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556fe3e0f270 0x556fe3e11690 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53874a08dfff5de1 server_cookie=53ce2fa8bbfe9e6a in_seq=0 out_seq=0 2024-04-21T08:54:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.395+0000 7fe92f59e640 1 -- 172.21.15.7:0/3876883248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93010dfa0 msgr2=0x7fe930115ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.395+0000 7fe92f59e640 1 --2- 172.21.15.7:0/3876883248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93010dfa0 0x7fe930115ae0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fe92400da00 tx=0x7fe92402ff10 comp rx=0 tx=0).stop 2024-04-21T08:54:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f78e9ffb640 1 -- 172.21.15.7:0/2588152112 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x556fe3c6e590 con 0x556fe3e0f270 2024-04-21T08:54:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f78e9ffb640 10 client.26794 handle_command_reply: tid=0 2024-04-21T08:54:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.396+0000 7fe92f59e640 1 -- 172.21.15.7:0/3876883248 shutdown_connections 2024-04-21T08:54:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.396+0000 7fe92f59e640 1 --2- 172.21.15.7:0/3876883248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93010dfa0 0x7fe930115ae0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.396+0000 7fe92f59e640 1 --2- 172.21.15.7:0/3876883248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe93010a400 0x7fe93010d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.397 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.396+0000 7fe92f59e640 1 --2- 172.21.15.7:0/3876883248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe930109a50 0x7fe930109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.398 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.396+0000 7fe92f59e640 1 -- 172.21.15.7:0/3876883248 >> 172.21.15.7:0/3876883248 conn(0x7fe93006ced0 msgr2=0x7fe93006d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f790b3c2740 1 client.26794 shutdown 2024-04-21T08:54:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f790b3c2740 1 -- 172.21.15.7:0/2588152112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 msgr2=0x556fe3da0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f790b3c2740 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3da0990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009930 tx=0x7f78fc005740 comp rx=0 tx=0).stop 2024-04-21T08:54:37.398 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.396+0000 7fe92f59e640 1 -- 172.21.15.7:0/3876883248 shutdown_connections 2024-04-21T08:54:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f790b3c2740 1 -- 172.21.15.7:0/2588152112 shutdown_connections 2024-04-21T08:54:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f790b3c2740 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x556fe3e0f270 0x556fe3e11690 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f790b3c2740 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556fe3daf660 0x556fe3dafae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f790b3c2740 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556fe3d94750 0x556fe3daf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f78e9ffb640 0 client.26794 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f790b3c2740 1 --2- 172.21.15.7:0/2588152112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556fe3d93da0 0x556fe3da0990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f790b3c2740 1 -- 172.21.15.7:0/2588152112 >> 172.21.15.7:0/2588152112 conn(0x556fe3d833a0 msgr2=0x556fe3d7f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.396+0000 7fe92f59e640 1 -- 172.21.15.7:0/3876883248 wait complete. 2024-04-21T08:54:37.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.396+0000 7f790b3c2740 1 -- 172.21.15.7:0/2588152112 shutdown_connections 2024-04-21T08:54:37.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.397+0000 7f790b3c2740 1 -- 172.21.15.7:0/2588152112 wait complete. 2024-04-21T08:54:37.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.397+0000 7f790b3c2740 20 client.26794 trim_cache size 0 max 16384 2024-04-21T08:54:37.401 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:54:37.401 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (3874 inodes in the stack)", 2024-04-21T08:54:37.401 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:54:37.401 INFO:teuthology.orchestra.run.smithi007.stdout: "08447483-b5f8-4de7-a923-832d7f72e5ac": { 2024-04-21T08:54:37.401 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:54:37.401 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "08447483-b5f8-4de7-a923-832d7f72e5ac", 2024-04-21T08:54:37.401 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:54:37.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:54:37.401 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:54:37.401 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:54:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.397+0000 7fe92f59e640 1 Processor -- start 2024-04-21T08:54:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.398+0000 7fe92f59e640 1 -- start start 2024-04-21T08:54:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.398+0000 7fe92f59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe930109a50 0x7fe9301a4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.398+0000 7fe92f59e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93010dfa0 0x7fe9301a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.398+0000 7fe92f59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe93019e9d0 0x7fe93019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.398+0000 7fe92f59e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe9301186f0 con 0x7fe93010dfa0 2024-04-21T08:54:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.398+0000 7fe92f59e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe930118570 con 0x7fe93019e9d0 2024-04-21T08:54:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.398+0000 7fe92f59e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe930118870 con 0x7fe930109a50 2024-04-21T08:54:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.398+0000 7fe92ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe93019e9d0 0x7fe93019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.398+0000 7fe92ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe93019e9d0 0x7fe93019ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38676/0 (socket says 172.21.15.7:38676) 2024-04-21T08:54:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.398+0000 7fe92ed9d640 1 -- 172.21.15.7:0/871681586 learned_addr learned my addr 172.21.15.7:0/871681586 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.398+0000 7fe92e59c640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe930109a50 0x7fe9301a4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.399+0000 7fe92ed9d640 1 -- 172.21.15.7:0/871681586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe930109a50 msgr2=0x7fe9301a4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.399+0000 7fe92ed9d640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe930109a50 0x7fe9301a4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.399+0000 7fe92ed9d640 1 -- 172.21.15.7:0/871681586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93010dfa0 msgr2=0x7fe9301a4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.399+0000 7fe92ed9d640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93010dfa0 0x7fe9301a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.399+0000 7fe92ed9d640 1 -- 172.21.15.7:0/871681586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93019f4f0 con 0x7fe93019e9d0 2024-04-21T08:54:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.399+0000 7fe92e59c640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe930109a50 0x7fe9301a4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.399+0000 7fe92ed9d640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe93019e9d0 0x7fe93019ee80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe9240388d0 tx=0x7fe924004a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.410 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.410+0000 7fe91f7fe640 1 -- 172.21.15.7:0/871681586 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924004250 con 0x7fe93019e9d0 2024-04-21T08:54:37.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.413+0000 7f79086f2640 1 -- 172.21.15.7:0/974750229 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78ec0a3f20 msgr2=0x7f78ec0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.413+0000 7f79086f2640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78ec0a3f20 0x7f78ec0a63e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009900 tx=0x7f78fc005fd0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.413+0000 7f79086f2640 1 -- 172.21.15.7:0/974750229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7900113ea0 msgr2=0x7f79001a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.413+0000 7f79086f2640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7900113ea0 0x7f79001a4b60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f78f800ba50 tx=0x7f78f800bf10 comp rx=0 tx=0).stop 2024-04-21T08:54:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.410+0000 7fe92f59e640 1 -- 172.21.15.7:0/871681586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe93019f780 con 0x7fe93019e9d0 2024-04-21T08:54:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.410+0000 7fe92f59e640 1 -- 172.21.15.7:0/871681586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9301ab9f0 con 0x7fe93019e9d0 2024-04-21T08:54:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.414+0000 7fe9377d2740 1 Processor -- start 2024-04-21T08:54:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.414+0000 7fe91f7fe640 1 -- 172.21.15.7:0/871681586 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe924048070 con 0x7fe93019e9d0 2024-04-21T08:54:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.415+0000 7fe91f7fe640 1 -- 172.21.15.7:0/871681586 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924005290 con 0x7fe93019e9d0 2024-04-21T08:54:37.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.420+0000 7f79086f2640 1 -- 172.21.15.7:0/974750229 shutdown_connections 2024-04-21T08:54:37.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.420+0000 7f79086f2640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f790019e6e0 0x7f790019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.420+0000 7f79086f2640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f78ec0a3f20 0x7f78ec0a63e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.420+0000 7f79086f2640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7900113ea0 0x7f79001a4b60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.420+0000 7f79086f2640 1 --2- 172.21.15.7:0/974750229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f790010b210 0x7f79001a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.420+0000 7f79086f2640 1 -- 172.21.15.7:0/974750229 >> 172.21.15.7:0/974750229 conn(0x7f790006ced0 msgr2=0x7f7900070810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.420+0000 7f79086f2640 1 -- 172.21.15.7:0/974750229 shutdown_connections 2024-04-21T08:54:37.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:54:37.420+0000 7f79086f2640 1 -- 172.21.15.7:0/974750229 wait complete. 2024-04-21T08:54:37.422 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.421+0000 7fe91f7fe640 1 -- 172.21.15.7:0/871681586 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fe924041040 con 0x7fe93019e9d0 2024-04-21T08:54:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.421+0000 7fe9377d2740 1 -- start start 2024-04-21T08:54:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.421+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfc2d0 0x5608bebfb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.421+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bebfb560 0x5608bebfb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.421+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 0x5608bec06ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe9377d2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5608beb578d0 con 0x5608bec04ae0 2024-04-21T08:54:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe9377d2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5608beaf4e80 con 0x5608bebfc2d0 2024-04-21T08:54:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe9377d2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5608beb53890 con 0x5608bebfb560 2024-04-21T08:54:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe92dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfc2d0 0x5608bebfb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe92dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfc2d0 0x5608bebfb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38686/0 (socket says 172.21.15.7:38686) 2024-04-21T08:54:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe92dd9b640 1 -- 172.21.15.7:0/802226729 learned_addr learned my addr 172.21.15.7:0/802226729 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe92dd9b640 1 -- 172.21.15.7:0/802226729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bebfb560 msgr2=0x5608bebfb9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe92dd9b640 1 --2- 172.21.15.7:0/802226729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bebfb560 0x5608bebfb9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe92dd9b640 1 -- 172.21.15.7:0/802226729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 msgr2=0x5608bec06ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe92dd9b640 1 --2- 172.21.15.7:0/802226729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 0x5608bec06ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe92dd9b640 1 -- 172.21.15.7:0/802226729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608beb16320 con 0x5608bebfc2d0 2024-04-21T08:54:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.422+0000 7fe92dd9b640 1 --2- 172.21.15.7:0/802226729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfc2d0 0x5608bebfb020 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe928002200 tx=0x7fe92800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebad5af055a91ee7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.423+0000 7fe91f7fe640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe9180a3d00 0x7fe9180a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.425 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.423+0000 7fe91f7fe640 1 -- 172.21.15.7:0/871681586 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fe9240f2820 con 0x7fe93019e9d0 2024-04-21T08:54:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.442+0000 7fe92e59c640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe9180a3d00 0x7fe9180a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.451+0000 7fe91cff9640 1 -- 172.21.15.7:0/802226729 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928004cd0 con 0x5608bebfc2d0 2024-04-21T08:54:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.451+0000 7fe91cff9640 1 -- 172.21.15.7:0/802226729 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe928007580 con 0x5608bebfc2d0 2024-04-21T08:54:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.452+0000 7fe9377d2740 1 -- 172.21.15.7:0/802226729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfc2d0 msgr2=0x5608bebfb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.452+0000 7fe9377d2740 1 --2- 172.21.15.7:0/802226729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfc2d0 0x5608bebfb020 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe928002200 tx=0x7fe92800e4b0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.452+0000 7fe9377d2740 1 -- 172.21.15.7:0/802226729 shutdown_connections 2024-04-21T08:54:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.452+0000 7fe9377d2740 1 --2- 172.21.15.7:0/802226729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 0x5608bec06ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.452+0000 7fe9377d2740 1 --2- 172.21.15.7:0/802226729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bebfb560 0x5608bebfb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.452+0000 7fe9377d2740 1 --2- 172.21.15.7:0/802226729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfc2d0 0x5608bebfb020 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe928002200 tx=0x7fe92800e4b0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.452+0000 7fe9377d2740 1 -- 172.21.15.7:0/802226729 >> 172.21.15.7:0/802226729 conn(0x5608bebb9c90 msgr2=0x5608bebf6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.453+0000 7fe9377d2740 1 -- 172.21.15.7:0/802226729 shutdown_connections 2024-04-21T08:54:37.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.453+0000 7fe9377d2740 1 -- 172.21.15.7:0/802226729 wait complete. 2024-04-21T08:54:37.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.454+0000 7fe9377d2740 1 -- start start 2024-04-21T08:54:37.454 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.454+0000 7fe9377d2740 1 Processor -- start 2024-04-21T08:54:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.454+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bebfb560 0x5608bec04510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.454+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 0x5608bec01800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.454+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec10df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.454+0000 7fe9377d2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5608bec07860 con 0x5608bec04ae0 2024-04-21T08:54:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.454+0000 7fe9377d2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5608beadfeb0 con 0x5608bec01ea0 2024-04-21T08:54:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.454+0000 7fe9377d2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5608bec079e0 con 0x5608bebfb560 2024-04-21T08:54:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.454+0000 7fe9377d2740 10 client.0 _ll_register_callbacks cb 0x5608beb0cb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.455+0000 7fe92ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec10df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.455+0000 7fe92ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec10df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38702/0 (socket says 172.21.15.7:38702) 2024-04-21T08:54:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.455+0000 7fe92ed9d640 1 -- 172.21.15.7:0/1134519825 learned_addr learned my addr 172.21.15.7:0/1134519825 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.455+0000 7fe92ed9d640 1 -- 172.21.15.7:0/1134519825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bebfb560 msgr2=0x5608bec04510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.455+0000 7fe92ed9d640 1 --2- 172.21.15.7:0/1134519825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bebfb560 0x5608bec04510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.455+0000 7fe92ed9d640 1 -- 172.21.15.7:0/1134519825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 msgr2=0x5608bec01800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.455+0000 7fe92ed9d640 1 --2- 172.21.15.7:0/1134519825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 0x5608bec01800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.455+0000 7fe92ed9d640 1 -- 172.21.15.7:0/1134519825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608be86a210 con 0x5608bec01ea0 2024-04-21T08:54:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.455+0000 7fe92ed9d640 1 --2- 172.21.15.7:0/1134519825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec10df0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe924004aa0 tx=0x7fe9240bdfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6032a7beba7ca37c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.461+0000 7fe915ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:54:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.461+0000 7fe915ffb640 1 -- 172.21.15.7:0/1134519825 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924012060 con 0x5608bec01ea0 2024-04-21T08:54:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.461+0000 7fe9377d2740 1 -- 172.21.15.7:0/1134519825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe928007580 con 0x5608bec01ea0 2024-04-21T08:54:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.461+0000 7fe915ffb640 1 -- 172.21.15.7:0/1134519825 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fe9240bc760 con 0x5608bec01ea0 2024-04-21T08:54:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.462+0000 7fe915ffb640 1 -- 172.21.15.7:0/1134519825 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe9240be460 con 0x5608bec01ea0 2024-04-21T08:54:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.462+0000 7fe9377d2740 10 client.26813 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.462+0000 7fe9377d2740 1 -- 172.21.15.7:0/1134519825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5608be936620 con 0x5608bec01ea0 2024-04-21T08:54:37.469 INFO:tasks.cephfs.filesystem:scrub status for tag:08447483-b5f8-4de7-a923-832d7f72e5ac - {'path': '/', 'tag': '08447483-b5f8-4de7-a923-832d7f72e5ac', 'options': 'recursive,force'} 2024-04-21T08:54:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.467+0000 7fe915ffb640 1 -- 172.21.15.7:0/1134519825 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe928007580 con 0x5608bec01ea0 2024-04-21T08:54:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.467+0000 7fe915ffb640 1 -- 172.21.15.7:0/1134519825 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fe92400b040 con 0x5608bec01ea0 2024-04-21T08:54:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.467+0000 7fe9377d2740 10 client.26813 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.467+0000 7fe9377d2740 10 client.26813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.467+0000 7fe9377d2740 20 client.26813 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.467+0000 7fe9377d2740 1 --2- 172.21.15.7:0/1134519825 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5608bec153a0 0x5608bec17860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.467+0000 7fe9377d2740 4 client.26813 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.467+0000 7fe9377d2740 1 -- 172.21.15.7:0/1134519825 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608beb6b880 con 0x5608bec153a0 2024-04-21T08:54:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.471+0000 7fe92e59c640 1 --2- 172.21.15.7:0/1134519825 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5608bec153a0 0x5608bec17860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.471+0000 7fe92e59c640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe9180a3d00 0x7fe9180a61c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x5608bebfca30 tx=0x7fe92000e5a0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.471+0000 7fe92e59c640 1 --2- 172.21.15.7:0/1134519825 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5608bec153a0 0x5608bec17860 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73485c9c5a50993c server_cookie=c45d31a07358d3ce in_seq=0 out_seq=0 2024-04-21T08:54:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.471+0000 7fe915ffb640 10 client.26813 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.472+0000 7fe915ffb640 1 -- 172.21.15.7:0/1134519825 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5608beb6b880 con 0x5608bec153a0 2024-04-21T08:54:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.472+0000 7fe915ffb640 10 client.26813 handle_command_reply: tid=0 2024-04-21T08:54:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.472+0000 7fe9377d2740 1 client.26813 shutdown 2024-04-21T08:54:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.481+0000 7fe9377d2740 1 -- 172.21.15.7:0/1134519825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 msgr2=0x5608bec10df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.481+0000 7fe9377d2740 1 --2- 172.21.15.7:0/1134519825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec10df0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe924004aa0 tx=0x7fe9240bdfa0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.481+0000 7fe9377d2740 1 -- 172.21.15.7:0/1134519825 shutdown_connections 2024-04-21T08:54:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.481+0000 7fe9377d2740 1 --2- 172.21.15.7:0/1134519825 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5608bec153a0 0x5608bec17860 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.481+0000 7fe9377d2740 1 --2- 172.21.15.7:0/1134519825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec10df0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.481+0000 7fe9377d2740 1 --2- 172.21.15.7:0/1134519825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 0x5608bec01800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.481+0000 7fe9377d2740 1 --2- 172.21.15.7:0/1134519825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bebfb560 0x5608bec04510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.481+0000 7fe915ffb640 0 client.26813 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.481+0000 7fe9377d2740 1 -- 172.21.15.7:0/1134519825 >> 172.21.15.7:0/1134519825 conn(0x5608bebb9c90 msgr2=0x5608bebf7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.483+0000 7fe9377d2740 1 -- 172.21.15.7:0/1134519825 shutdown_connections 2024-04-21T08:54:37.486 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.486+0000 7fe9377d2740 1 -- 172.21.15.7:0/1134519825 wait complete. 2024-04-21T08:54:37.487 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.486+0000 7fe9377d2740 20 client.26813 trim_cache size 0 max 16384 2024-04-21T08:54:37.549 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/ntp.o 2024-04-21T08:54:37.553 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.552+0000 7fe9377d2740 1 Processor -- start 2024-04-21T08:54:37.553 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.552+0000 7fe9377d2740 1 -- start start 2024-04-21T08:54:37.554 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.553+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfb560 0x5608bebe8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.554 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.553+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bec01ea0 0x5608bec04040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.554 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.553+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 0x5608bec04580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.554 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.553+0000 7fe92e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfb560 0x5608bebe8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.554 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.553+0000 7fe92e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfb560 0x5608bebe8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38716/0 (socket says 172.21.15.7:38716) 2024-04-21T08:54:37.554 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.553+0000 7fe92e59c640 1 -- 172.21.15.7:0/847326341 learned_addr learned my addr 172.21.15.7:0/847326341 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.553+0000 7fe92dd9b640 1 --2- 172.21.15.7:0/847326341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bec01ea0 0x5608bec04040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.553+0000 7fe9377d2740 1 -- 172.21.15.7:0/847326341 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe91000e2e0 con 0x5608bec04ae0 2024-04-21T08:54:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.553+0000 7fe9377d2740 1 -- 172.21.15.7:0/847326341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe910010b40 con 0x5608bebfb560 2024-04-21T08:54:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.553+0000 7fe9377d2740 1 -- 172.21.15.7:0/847326341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe91000e930 con 0x5608bec01ea0 2024-04-21T08:54:37.555 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.554+0000 7fe92e59c640 1 -- 172.21.15.7:0/847326341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bec01ea0 msgr2=0x5608bec04040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.554+0000 7fe92e59c640 1 --2- 172.21.15.7:0/847326341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bec01ea0 0x5608bec04040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.554+0000 7fe92e59c640 1 -- 172.21.15.7:0/847326341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 msgr2=0x5608bec04580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.554+0000 7fe92e59c640 1 --2- 172.21.15.7:0/847326341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 0x5608bec04580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.554+0000 7fe92e59c640 1 -- 172.21.15.7:0/847326341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608bec076f0 con 0x5608bebfb560 2024-04-21T08:54:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.554+0000 7fe92dd9b640 1 --2- 172.21.15.7:0/847326341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bec01ea0 0x5608bec04040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:37.556 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.554+0000 7fe92e59c640 1 --2- 172.21.15.7:0/847326341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfb560 0x5608bebe8bd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x5608bec180d0 tx=0x7fe920013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78d35682925d8d67 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.563+0000 7fe917fff640 1 -- 172.21.15.7:0/847326341 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92001c040 con 0x5608bebfb560 2024-04-21T08:54:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.563+0000 7fe917fff640 1 -- 172.21.15.7:0/847326341 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe920015e50 con 0x5608bebfb560 2024-04-21T08:54:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.563+0000 7fe917fff640 1 -- 172.21.15.7:0/847326341 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92002d890 con 0x5608bebfb560 2024-04-21T08:54:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.564+0000 7fe9377d2740 1 -- 172.21.15.7:0/847326341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfb560 msgr2=0x5608bebe8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.564+0000 7fe9377d2740 1 --2- 172.21.15.7:0/847326341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfb560 0x5608bebe8bd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x5608bec180d0 tx=0x7fe920013000 comp rx=0 tx=0).stop 2024-04-21T08:54:37.567 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.567+0000 7fe9377d2740 1 -- 172.21.15.7:0/847326341 shutdown_connections 2024-04-21T08:54:37.568 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.567+0000 7fe9377d2740 1 --2- 172.21.15.7:0/847326341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bec04ae0 0x5608bec04580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.568 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.567+0000 7fe9377d2740 1 --2- 172.21.15.7:0/847326341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bec01ea0 0x5608bec04040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.568 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.567+0000 7fe9377d2740 1 --2- 172.21.15.7:0/847326341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bebfb560 0x5608bebe8bd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.568 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.567+0000 7fe9377d2740 1 -- 172.21.15.7:0/847326341 >> 172.21.15.7:0/847326341 conn(0x5608bebb9c90 msgr2=0x5608bec00110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.568 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.567+0000 7fe9377d2740 1 -- 172.21.15.7:0/847326341 shutdown_connections 2024-04-21T08:54:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.567+0000 7fe9377d2740 1 -- 172.21.15.7:0/847326341 wait complete. 2024-04-21T08:54:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.567+0000 7fe9377d2740 1 -- start start 2024-04-21T08:54:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.567+0000 7fe9377d2740 1 Processor -- start 2024-04-21T08:54:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.568+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bebfb560 0x5608bebff4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.568+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec0cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.568+0000 7fe9377d2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bec04ae0 0x5608bec05a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.568+0000 7fe9377d2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe91000f000 con 0x5608bebfb560 2024-04-21T08:54:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.568+0000 7fe9377d2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe910010470 con 0x5608bec01ea0 2024-04-21T08:54:37.569 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.568+0000 7fe9377d2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe91000f6d0 con 0x5608bec04ae0 2024-04-21T08:54:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.568+0000 7fe9377d2740 10 client.0 _ll_register_callbacks cb 0x7fe924004b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.568+0000 7fe92dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec0cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.568+0000 7fe92dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec0cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38728/0 (socket says 172.21.15.7:38728) 2024-04-21T08:54:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.568+0000 7fe92dd9b640 1 -- 172.21.15.7:0/681115420 learned_addr learned my addr 172.21.15.7:0/681115420 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:37.573 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.573+0000 7fe92ed9d640 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bec04ae0 0x5608bec05a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.573+0000 7fe92e59c640 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bebfb560 0x5608bebff4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.573+0000 7fe92dd9b640 1 -- 172.21.15.7:0/681115420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bec04ae0 msgr2=0x5608bec05a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.573+0000 7fe92dd9b640 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bec04ae0 0x5608bec05a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.573+0000 7fe92dd9b640 1 -- 172.21.15.7:0/681115420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bebfb560 msgr2=0x5608bebff4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.573+0000 7fe92dd9b640 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bebfb560 0x5608bebff4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.575 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.573+0000 7fe92dd9b640 1 -- 172.21.15.7:0/681115420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608bec05fc0 con 0x5608bec01ea0 2024-04-21T08:54:37.575 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.574+0000 7fe92dd9b640 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec0cd50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe92800c040 tx=0x7fe928014d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dac339d3231eab6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.575+0000 7fe915ffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:54:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.575+0000 7fe915ffb640 1 -- 172.21.15.7:0/681115420 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928027060 con 0x5608bec01ea0 2024-04-21T08:54:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.575+0000 7fe9377d2740 1 -- 172.21.15.7:0/681115420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe920015e50 con 0x5608bec01ea0 2024-04-21T08:54:37.576 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.575+0000 7fe915ffb640 1 -- 172.21.15.7:0/681115420 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7fe9280070d0 con 0x5608bec01ea0 2024-04-21T08:54:37.584 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.584+0000 7fe915ffb640 1 -- 172.21.15.7:0/681115420 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe92802e050 con 0x5608bec01ea0 2024-04-21T08:54:37.586 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/microcode/intel.o 2024-04-21T08:54:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.591+0000 7fe9377d2740 10 client.26819 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.591+0000 7fe9377d2740 1 -- 172.21.15.7:0/681115420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5608bec7c030 con 0x5608bec01ea0 2024-04-21T08:54:37.593 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.592+0000 7fe915ffb640 1 -- 172.21.15.7:0/681115420 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe920015e50 con 0x5608bec01ea0 2024-04-21T08:54:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.593+0000 7fe915ffb640 1 -- 172.21.15.7:0/681115420 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fe9280072d0 con 0x5608bec01ea0 2024-04-21T08:54:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.593+0000 7fe9377d2740 10 client.26819 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.593+0000 7fe9377d2740 10 client.26819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.593+0000 7fe9377d2740 20 client.26819 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.593+0000 7fe9377d2740 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5608bec7c450 0x5608bec7e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.593+0000 7fe9377d2740 4 client.26819 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.593+0000 7fe9377d2740 1 -- 172.21.15.7:0/681115420 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5608beaf73d0 con 0x5608bec7c450 2024-04-21T08:54:37.595 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.593+0000 7fe92e59c640 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5608bec7c450 0x5608bec7e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.595 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.594+0000 7fe92e59c640 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5608bec7c450 0x5608bec7e870 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7da494ec09c9dafd server_cookie=f20872e64940f488 in_seq=0 out_seq=0 2024-04-21T08:54:37.595 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:37.594+0000 7fe915ffb640 10 client.26819 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:37.675 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/clocksource.o 2024-04-21T08:54:37.751 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/cacheinfo.o 2024-04-21T08:54:37.827 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/fat/built-in.a 2024-04-21T08:54:38.122 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_clock.o 2024-04-21T08:54:38.161 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mremap.o 2024-04-21T08:54:38.618 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/microcode/amd.o 2024-04-21T08:54:38.788 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/jiffies.o 2024-04-21T08:54:38.808 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/mmp.o 2024-04-21T08:54:38.905 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/ring_buffer.o 2024-04-21T08:54:39.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:38 smithi016 ceph-mon[31259]: pgmap v313: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 597 KiB/s wr, 71 op/s 2024-04-21T08:54:39.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:38 smithi077 ceph-mon[31615]: pgmap v313: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 597 KiB/s wr, 71 op/s 2024-04-21T08:54:39.158 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:38 smithi007 ceph-mon[28477]: pgmap v313: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 597 KiB/s wr, 71 op/s 2024-04-21T08:54:39.244 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/move_extent.o 2024-04-21T08:54:39.287 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cgroup/namespace.o 2024-04-21T08:54:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe915ffb640 1 -- 172.21.15.7:0/681115420 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5608beaf73d0 con 0x5608bec7c450 2024-04-21T08:54:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe915ffb640 10 client.26819 handle_command_reply: tid=0 2024-04-21T08:54:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe9377d2740 1 client.26819 shutdown 2024-04-21T08:54:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe9377d2740 1 -- 172.21.15.7:0/681115420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 msgr2=0x5608bec0cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe9377d2740 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec0cd50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe92800c040 tx=0x7fe928014d10 comp rx=0 tx=0).stop 2024-04-21T08:54:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe9377d2740 1 -- 172.21.15.7:0/681115420 shutdown_connections 2024-04-21T08:54:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe9377d2740 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5608bec7c450 0x5608bec7e870 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe9377d2740 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5608bec04ae0 0x5608bec05a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe9377d2740 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5608bec01ea0 0x5608bec0cd50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe9377d2740 1 --2- 172.21.15.7:0/681115420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608bebfb560 0x5608bebff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe9377d2740 1 -- 172.21.15.7:0/681115420 >> 172.21.15.7:0/681115420 conn(0x5608bebb9c90 msgr2=0x5608bebec250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe915ffb640 0 client.26819 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.292+0000 7fe9377d2740 1 -- 172.21.15.7:0/681115420 shutdown_connections 2024-04-21T08:54:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.293+0000 7fe9377d2740 1 -- 172.21.15.7:0/681115420 wait complete. 2024-04-21T08:54:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.293+0000 7fe9377d2740 20 client.26819 trim_cache size 0 max 16384 2024-04-21T08:54:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:54:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:54:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:54:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:54:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.297+0000 7fe92f59e640 1 -- 172.21.15.7:0/871681586 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe9180a3d00 msgr2=0x7fe9180a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.297+0000 7fe92f59e640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe9180a3d00 0x7fe9180a61c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x5608bebfca30 tx=0x7fe92000e5a0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.297+0000 7fe92f59e640 1 -- 172.21.15.7:0/871681586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe93019e9d0 msgr2=0x7fe93019ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.297+0000 7fe92f59e640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe93019e9d0 0x7fe93019ee80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe9240388d0 tx=0x7fe924004a30 comp rx=0 tx=0).stop 2024-04-21T08:54:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.303+0000 7fe92f59e640 1 -- 172.21.15.7:0/871681586 shutdown_connections 2024-04-21T08:54:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.303+0000 7fe92f59e640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe93019e9d0 0x7fe93019ee80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.303+0000 7fe92f59e640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe93010dfa0 0x7fe9301a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.303+0000 7fe92f59e640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe9180a3d00 0x7fe9180a61c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.303+0000 7fe92f59e640 1 --2- 172.21.15.7:0/871681586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe930109a50 0x7fe9301a4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.303+0000 7fe92f59e640 1 -- 172.21.15.7:0/871681586 >> 172.21.15.7:0/871681586 conn(0x7fe93006ced0 msgr2=0x7fe930071a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:39.309 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.306+0000 7fe92f59e640 1 -- 172.21.15.7:0/871681586 shutdown_connections 2024-04-21T08:54:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:39.312+0000 7fe92f59e640 1 -- 172.21.15.7:0/871681586 wait complete. 2024-04-21T08:54:39.375 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/timer_list.o 2024-04-21T08:54:39.429 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/scattered.o 2024-04-21T08:54:40.036 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/msync.o 2024-04-21T08:54:40.129 INFO:tasks.workunit.client.0.smithi007.stdout: AR arch/x86/kernel/cpu/microcode/built-in.a 2024-04-21T08:54:40.141 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/topology.o 2024-04-21T08:54:40.263 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/namei.o 2024-04-21T08:54:40.302 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/common.o 2024-04-21T08:54:40.315 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cgroup/cgroup-v1.o 2024-04-21T08:54:40.426 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/timeconv.o 2024-04-21T08:54:40.651 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/rdrand.o 2024-04-21T08:54:40.816 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/page-io.o 2024-04-21T08:54:40.978 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/page_vma_mapped.o 2024-04-21T08:54:40.984 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/match.o 2024-04-21T08:54:41.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:40 smithi016 ceph-mon[31259]: pgmap v314: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 978 KiB/s wr, 81 op/s 2024-04-21T08:54:41.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:40 smithi077 ceph-mon[31615]: pgmap v314: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 978 KiB/s wr, 81 op/s 2024-04-21T08:54:41.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:40 smithi007 ceph-mon[28477]: pgmap v314: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 978 KiB/s wr, 81 op/s 2024-04-21T08:54:41.205 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/timecounter.o 2024-04-21T08:54:41.384 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/alarmtimer.o 2024-04-21T08:54:41.660 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace.o 2024-04-21T08:54:41.804 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/readpage.o 2024-04-21T08:54:42.154 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cgroup/freezer.o 2024-04-21T08:54:42.258 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/pagewalk.o 2024-04-21T08:54:42.415 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/bugs.o 2024-04-21T08:54:42.479 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_output.o 2024-04-21T08:54:43.085 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/resize.o 2024-04-21T08:54:43.103 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:42 smithi077 ceph-mon[31615]: pgmap v315: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 960 KiB/s wr, 108 op/s 2024-04-21T08:54:43.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:42 smithi007 ceph-mon[28477]: pgmap v315: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 960 KiB/s wr, 108 op/s 2024-04-21T08:54:43.174 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/posix-timers.o 2024-04-21T08:54:43.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:42 smithi016 ceph-mon[31259]: pgmap v315: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 960 KiB/s wr, 108 op/s 2024-04-21T08:54:43.469 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cgroup/legacy_freezer.o 2024-04-21T08:54:43.711 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/pgtable-generic.o 2024-04-21T08:54:44.213 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/aperfmperf.o 2024-04-21T08:54:44.665 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cgroup/pids.o 2024-04-21T08:54:44.804 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/cpuid-deps.o 2024-04-21T08:54:44.884 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/rmap.o 2024-04-21T08:54:45.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:44 smithi077 ceph-mon[31615]: pgmap v316: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.0 MiB/s wr, 214 op/s 2024-04-21T08:54:45.156 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:44 smithi007 ceph-mon[28477]: pgmap v316: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.0 MiB/s wr, 214 op/s 2024-04-21T08:54:45.212 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/bpf/core.o 2024-04-21T08:54:45.225 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/posix-cpu-timers.o 2024-04-21T08:54:45.246 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/umwait.o 2024-04-21T08:54:45.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:44 smithi016 ceph-mon[31259]: pgmap v316: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 1.0 MiB/s wr, 214 op/s 2024-04-21T08:54:45.396 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/proc.o 2024-04-21T08:54:45.596 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/super.o 2024-04-21T08:54:45.721 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cgroup/rdma.o 2024-04-21T08:54:46.120 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cgroup/cpuset.o 2024-04-21T08:54:46.297 INFO:tasks.workunit.client.0.smithi007.stdout: MKCAP arch/x86/kernel/cpu/capflags.c 2024-04-21T08:54:46.600 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_seq.o 2024-04-21T08:54:46.844 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/posix-clock.o 2024-04-21T08:54:47.095 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cgroup/misc.o 2024-04-21T08:54:47.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:46 smithi007 ceph-mon[28477]: pgmap v317: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 973 KiB/s wr, 238 op/s 2024-04-21T08:54:47.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:46 smithi016 ceph-mon[31259]: pgmap v317: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 973 KiB/s wr, 238 op/s 2024-04-21T08:54:47.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:46 smithi077 ceph-mon[31615]: pgmap v317: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 973 KiB/s wr, 238 op/s 2024-04-21T08:54:47.471 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_stat.o 2024-04-21T08:54:47.542 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/vmalloc.o 2024-04-21T08:54:48.053 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cgroup/debug.o 2024-04-21T08:54:48.218 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/itimer.o 2024-04-21T08:54:48.552 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/bpf/built-in.a 2024-04-21T08:54:48.622 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_printk.o 2024-04-21T08:54:48.651 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/events/core.o 2024-04-21T08:54:49.061 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/events/ring_buffer.o 2024-04-21T08:54:49.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:48 smithi007 ceph-mon[28477]: pgmap v318: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 1.0 MiB/s wr, 302 op/s 2024-04-21T08:54:49.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:48 smithi016 ceph-mon[31259]: pgmap v318: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 1.0 MiB/s wr, 302 op/s 2024-04-21T08:54:49.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:48 smithi077 ceph-mon[31615]: pgmap v318: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 1.0 MiB/s wr, 302 op/s 2024-04-21T08:54:49.379 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/clockevents.o 2024-04-21T08:54:49.407 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/cgroup/built-in.a 2024-04-21T08:54:49.514 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/events/callchain.o 2024-04-21T08:54:49.906 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/pid_list.o 2024-04-21T08:54:50.272 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/tick-common.o 2024-04-21T08:54:50.645 INFO:tasks.workunit.client.0.smithi007.stdout: CC arch/x86/kernel/cpu/powerflags.o 2024-04-21T08:54:50.827 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/tick-broadcast.o 2024-04-21T08:54:50.898 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_sched_switch.o 2024-04-21T08:54:50.991 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/isofs/namei.o 2024-04-21T08:54:51.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:50 smithi016 ceph-mon[31259]: pgmap v319: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 632 KiB/s wr, 377 op/s 2024-04-21T08:54:51.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:50 smithi077 ceph-mon[31615]: pgmap v319: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 632 KiB/s wr, 377 op/s 2024-04-21T08:54:51.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:50 smithi007 ceph-mon[28477]: pgmap v319: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 632 KiB/s wr, 377 op/s 2024-04-21T08:54:51.403 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/process_vm_access.o 2024-04-21T08:54:51.473 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/tick-broadcast-hrtimer.o 2024-04-21T08:54:51.780 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/tick-oneshot.o 2024-04-21T08:54:51.886 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/isofs/inode.o 2024-04-21T08:54:52.185 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_nop.o 2024-04-21T08:54:52.223 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/tick-sched.o 2024-04-21T08:54:52.439 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/page_alloc.o 2024-04-21T08:54:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.645+0000 7f5255ac1640 1 -- 172.21.15.7:0/2351009271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f525010a400 msgr2=0x7f525010da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.645+0000 7f5255ac1640 1 --2- 172.21.15.7:0/2351009271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f525010a400 0x7f525010da10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5240009900 tx=0x7f524002bc50 comp rx=0 tx=0).stop 2024-04-21T08:54:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.645+0000 7f5255ac1640 1 -- 172.21.15.7:0/2351009271 shutdown_connections 2024-04-21T08:54:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.645+0000 7f5255ac1640 1 --2- 172.21.15.7:0/2351009271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f525010e0f0 0x7f5250115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.645+0000 7f5255ac1640 1 --2- 172.21.15.7:0/2351009271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f525010a400 0x7f525010da10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.645+0000 7f5255ac1640 1 --2- 172.21.15.7:0/2351009271 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5250109a50 0x7f5250109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.645+0000 7f5255ac1640 1 -- 172.21.15.7:0/2351009271 >> 172.21.15.7:0/2351009271 conn(0x7f525006d030 msgr2=0x7f525006d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.645+0000 7f5255ac1640 1 -- 172.21.15.7:0/2351009271 shutdown_connections 2024-04-21T08:54:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.645+0000 7f5255ac1640 1 -- 172.21.15.7:0/2351009271 wait complete. 2024-04-21T08:54:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.646+0000 7f5255ac1640 1 Processor -- start 2024-04-21T08:54:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.646+0000 7f5255ac1640 1 -- start start 2024-04-21T08:54:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.646+0000 7f5255ac1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5250109a50 0x7f52501173e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.646+0000 7f5255ac1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f525010e0f0 0x7f5250117920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.646+0000 7f5255ac1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5250119080 0x7f5250117e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.646+0000 7f5255ac1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f525011aa30 con 0x7f5250119080 2024-04-21T08:54:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.646+0000 7f5255ac1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f525011a8b0 con 0x7f5250109a50 2024-04-21T08:54:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.646+0000 7f5255ac1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f525011abb0 con 0x7f525010e0f0 2024-04-21T08:54:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.647+0000 7f524f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5250109a50 0x7f52501173e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.647+0000 7f524f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5250109a50 0x7f52501173e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38374/0 (socket says 172.21.15.7:38374) 2024-04-21T08:54:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.647+0000 7f524f7fe640 1 -- 172.21.15.7:0/965421305 learned_addr learned my addr 172.21.15.7:0/965421305 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.647+0000 7f524ffff640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5250119080 0x7f5250117e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.647+0000 7f524f7fe640 1 -- 172.21.15.7:0/965421305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f525010e0f0 msgr2=0x7f5250117920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.657+0000 7f524effd640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f525010e0f0 0x7f5250117920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.657+0000 7f524f7fe640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f525010e0f0 0x7f5250117920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.657+0000 7f524f7fe640 1 -- 172.21.15.7:0/965421305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5250119080 msgr2=0x7f5250117e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.657+0000 7f524f7fe640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5250119080 0x7f5250117e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.657+0000 7f524f7fe640 1 -- 172.21.15.7:0/965421305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52501a5260 con 0x7f5250109a50 2024-04-21T08:54:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.658+0000 7f524ffff640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5250119080 0x7f5250117e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.658+0000 7f524effd640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f525010e0f0 0x7f5250117920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.658+0000 7f524f7fe640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5250109a50 0x7f52501173e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f524800eab0 tx=0x7f524800ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.658+0000 7f524cff9640 1 -- 172.21.15.7:0/965421305 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524800ce40 con 0x7f5250109a50 2024-04-21T08:54:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.658+0000 7f524cff9640 1 -- 172.21.15.7:0/965421305 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5248011920 con 0x7f5250109a50 2024-04-21T08:54:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.658+0000 7f524cff9640 1 -- 172.21.15.7:0/965421305 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524801aa60 con 0x7f5250109a50 2024-04-21T08:54:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.661+0000 7f5255ac1640 1 -- 172.21.15.7:0/965421305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52501a5550 con 0x7f5250109a50 2024-04-21T08:54:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.661+0000 7f5255ac1640 1 -- 172.21.15.7:0/965421305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52501a5a50 con 0x7f5250109a50 2024-04-21T08:54:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.664+0000 7f524cff9640 1 -- 172.21.15.7:0/965421305 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f524801ac00 con 0x7f5250109a50 2024-04-21T08:54:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.665+0000 7f524cff9640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f52380a3de0 0x7f52380a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.665+0000 7f524cff9640 1 -- 172.21.15.7:0/965421305 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7f52480d1500 con 0x7f5250109a50 2024-04-21T08:54:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.667+0000 7f524effd640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f52380a3de0 0x7f52380a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.668+0000 7f524effd640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f52380a3de0 0x7f52380a62a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5240006010 tx=0x7f5240000bc0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f5258793740 1 Processor -- start 2024-04-21T08:54:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f5258793740 1 -- start start 2024-04-21T08:54:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f5258793740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a612958d0 0x555a61294620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f5258793740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a61294b60 0x555a61294fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f5258793740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 0x555a612a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f5258793740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x555a611f98d0 con 0x555a612958d0 2024-04-21T08:54:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f5258793740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555a61196e80 con 0x555a61294b60 2024-04-21T08:54:52.672 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f5258793740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555a611f5890 con 0x555a6129e0e0 2024-04-21T08:54:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f524ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 0x555a612a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f524effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a61294b60 0x555a61294fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f524ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 0x555a612a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:52082/0 (socket says 172.21.15.7:52082) 2024-04-21T08:54:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f524effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a61294b60 0x555a61294fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38376/0 (socket says 172.21.15.7:38376) 2024-04-21T08:54:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.672+0000 7f524ffff640 1 -- 172.21.15.7:0/1470386056 learned_addr learned my addr 172.21.15.7:0/1470386056 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:52.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.673+0000 7f524ffff640 1 -- 172.21.15.7:0/1470386056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a61294b60 msgr2=0x555a61294fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.673+0000 7f524ffff640 1 --2- 172.21.15.7:0/1470386056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a61294b60 0x555a61294fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.673+0000 7f524ffff640 1 -- 172.21.15.7:0/1470386056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a612958d0 msgr2=0x555a61294620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.673+0000 7f524ffff640 1 --2- 172.21.15.7:0/1470386056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a612958d0 0x555a61294620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.673+0000 7f524ffff640 1 -- 172.21.15.7:0/1470386056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a611b8320 con 0x555a6129e0e0 2024-04-21T08:54:52.674 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.673+0000 7f524ffff640 1 --2- 172.21.15.7:0/1470386056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 0x555a612a04d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f524400af90 tx=0x7f524400f850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12b825e3249e7178 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.677+0000 7f5235ffb640 1 -- 172.21.15.7:0/1470386056 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524401d070 con 0x555a6129e0e0 2024-04-21T08:54:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.677+0000 7f5235ffb640 1 -- 172.21.15.7:0/1470386056 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5244010db0 con 0x555a6129e0e0 2024-04-21T08:54:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.677+0000 7f5235ffb640 1 -- 172.21.15.7:0/1470386056 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5244018e60 con 0x555a6129e0e0 2024-04-21T08:54:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.678+0000 7f5258793740 1 -- 172.21.15.7:0/1470386056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 msgr2=0x555a612a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.679+0000 7f5258793740 1 --2- 172.21.15.7:0/1470386056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 0x555a612a04d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f524400af90 tx=0x7f524400f850 comp rx=0 tx=0).stop 2024-04-21T08:54:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.679+0000 7f5258793740 1 -- 172.21.15.7:0/1470386056 shutdown_connections 2024-04-21T08:54:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.679+0000 7f5258793740 1 --2- 172.21.15.7:0/1470386056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 0x555a612a04d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.679+0000 7f5258793740 1 --2- 172.21.15.7:0/1470386056 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a61294b60 0x555a61294fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.679+0000 7f5258793740 1 --2- 172.21.15.7:0/1470386056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a612958d0 0x555a61294620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.679 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.679+0000 7f5258793740 1 -- 172.21.15.7:0/1470386056 >> 172.21.15.7:0/1470386056 conn(0x555a6125bc90 msgr2=0x555a6128f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:52.681 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.682+0000 7f5258793740 1 -- 172.21.15.7:0/1470386056 shutdown_connections 2024-04-21T08:54:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.682+0000 7f5258793740 1 -- 172.21.15.7:0/1470386056 wait complete. 2024-04-21T08:54:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.682+0000 7f5258793740 1 -- start start 2024-04-21T08:54:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.682+0000 7f5258793740 1 Processor -- start 2024-04-21T08:54:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f5258793740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a61294b60 0x555a6129db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f5258793740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a612958d0 0x555a6129aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f5258793740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 0x555a6129b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f5258793740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x555a612a0e10 con 0x555a61294b60 2024-04-21T08:54:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f5258793740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x555a61181eb0 con 0x555a612958d0 2024-04-21T08:54:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f5258793740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555a612a0f90 con 0x555a6129e0e0 2024-04-21T08:54:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f5258793740 10 client.0 _ll_register_callbacks cb 0x555a611aeb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f524effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a612958d0 0x555a6129aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f524effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a612958d0 0x555a6129aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38384/0 (socket says 172.21.15.7:38384) 2024-04-21T08:54:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f524effd640 1 -- 172.21.15.7:0/656543559 learned_addr learned my addr 172.21.15.7:0/656543559 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f524ffff640 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 0x555a6129b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f524effd640 1 -- 172.21.15.7:0/656543559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 msgr2=0x555a6129b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f524effd640 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 0x555a6129b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f524effd640 1 -- 172.21.15.7:0/656543559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a61294b60 msgr2=0x555a6129db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f524effd640 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a61294b60 0x555a6129db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.683+0000 7f524effd640 1 -- 172.21.15.7:0/656543559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a60f0c210 con 0x555a612958d0 2024-04-21T08:54:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.684+0000 7f524effd640 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a612958d0 0x555a6129aef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5240009900 tx=0x7f5240009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=916fcaf13f2c9f09 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.688+0000 7f52337f6640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:54:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.688+0000 7f52337f6640 1 -- 172.21.15.7:0/656543559 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240050020 con 0x555a612958d0 2024-04-21T08:54:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.688+0000 7f5258793740 1 -- 172.21.15.7:0/656543559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5244010db0 con 0x555a612958d0 2024-04-21T08:54:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.688+0000 7f52337f6640 1 -- 172.21.15.7:0/656543559 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7f5240005590 con 0x555a612958d0 2024-04-21T08:54:52.688 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.689+0000 7f52337f6640 1 -- 172.21.15.7:0/656543559 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f524005b050 con 0x555a612958d0 2024-04-21T08:54:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.691+0000 7f5258793740 10 client.26828 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:52.690 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.691+0000 7f5258793740 1 -- 172.21.15.7:0/656543559 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555a60fd8620 con 0x555a612958d0 2024-04-21T08:54:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.691+0000 7f52337f6640 1 -- 172.21.15.7:0/656543559 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5244010db0 con 0x555a612958d0 2024-04-21T08:54:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.691+0000 7f52337f6640 1 -- 172.21.15.7:0/656543559 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f52400039d0 con 0x555a612958d0 2024-04-21T08:54:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.692+0000 7f5258793740 10 client.26828 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.692+0000 7f5258793740 10 client.26828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.692+0000 7f5258793740 20 client.26828 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.692+0000 7f5258793740 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a612ae300 0x555a612b0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.692+0000 7f5258793740 4 client.26828 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.692+0000 7f5258793740 1 -- 172.21.15.7:0/656543559 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a6120d880 con 0x555a612ae300 2024-04-21T08:54:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.692+0000 7f524f7fe640 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a612ae300 0x555a612b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.693+0000 7f524f7fe640 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a612ae300 0x555a612b0730 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84b51e2b0acd0d19 server_cookie=228f8f1f73bce2b5 in_seq=0 out_seq=0 2024-04-21T08:54:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.693+0000 7f52337f6640 10 client.26828 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.694+0000 7f52337f6640 1 -- 172.21.15.7:0/656543559 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f524807be50 con 0x555a612ae300 2024-04-21T08:54:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.694+0000 7f52337f6640 10 client.26828 handle_command_reply: tid=0 2024-04-21T08:54:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.694+0000 7f5258793740 1 client.26828 shutdown 2024-04-21T08:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.699+0000 7f5258793740 1 -- 172.21.15.7:0/656543559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a612958d0 msgr2=0x555a6129aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.699+0000 7f5258793740 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a612958d0 0x555a6129aef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5240009900 tx=0x7f5240009260 comp rx=0 tx=0).stop 2024-04-21T08:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.701+0000 7f5258793740 1 -- 172.21.15.7:0/656543559 shutdown_connections 2024-04-21T08:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.701+0000 7f5258793740 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a612ae300 0x555a612b0730 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.701+0000 7f5258793740 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a6129e0e0 0x555a6129b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.701+0000 7f5258793740 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a612958d0 0x555a6129aef0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.701+0000 7f5258793740 1 --2- 172.21.15.7:0/656543559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a61294b60 0x555a6129db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.701+0000 7f52337f6640 0 client.26828 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.701+0000 7f5258793740 1 -- 172.21.15.7:0/656543559 >> 172.21.15.7:0/656543559 conn(0x555a6125bc90 msgr2=0x555a612915a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.701+0000 7f5258793740 1 -- 172.21.15.7:0/656543559 shutdown_connections 2024-04-21T08:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.701+0000 7f5258793740 1 -- 172.21.15.7:0/656543559 wait complete. 2024-04-21T08:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.701+0000 7f5258793740 20 client.26828 trim_cache size 0 max 16384 2024-04-21T08:54:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.768+0000 7f5258793740 1 Processor -- start 2024-04-21T08:54:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.768+0000 7f5258793740 1 -- start start 2024-04-21T08:54:52.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.769+0000 7f5258793740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a61294b60 0x555a612a9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.774+0000 7f5258793740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a612958d0 0x555a61299ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.774+0000 7f524f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a61294b60 0x555a612a9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.774+0000 7f524f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a61294b60 0x555a612a9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:52106/0 (socket says 172.21.15.7:52106) 2024-04-21T08:54:52.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.774+0000 7f524f7fe640 1 -- 172.21.15.7:0/1883806134 learned_addr learned my addr 172.21.15.7:0/1883806134 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:52.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.774+0000 7f5258793740 1 --2- 172.21.15.7:0/1883806134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 0x555a6129d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.774+0000 7f5258793740 1 -- 172.21.15.7:0/1883806134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f523c00e2e0 con 0x555a612958d0 2024-04-21T08:54:52.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.774+0000 7f5258793740 1 -- 172.21.15.7:0/1883806134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f523c010b40 con 0x555a6129e0e0 2024-04-21T08:54:52.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.774+0000 7f5258793740 1 -- 172.21.15.7:0/1883806134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f523c00e930 con 0x555a61294b60 2024-04-21T08:54:52.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.775+0000 7f524ffff640 1 --2- 172.21.15.7:0/1883806134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 0x555a6129d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.775+0000 7f524f7fe640 1 -- 172.21.15.7:0/1883806134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 msgr2=0x555a6129d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:52.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.775+0000 7f524f7fe640 1 --2- 172.21.15.7:0/1883806134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 0x555a6129d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.775+0000 7f524f7fe640 1 -- 172.21.15.7:0/1883806134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a612958d0 msgr2=0x555a61299ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:52.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.775+0000 7f524f7fe640 1 --2- 172.21.15.7:0/1883806134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a612958d0 0x555a61299ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.775+0000 7f524f7fe640 1 -- 172.21.15.7:0/1883806134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a6128af30 con 0x555a61294b60 2024-04-21T08:54:52.776 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.776+0000 7f524f7fe640 1 --2- 172.21.15.7:0/1883806134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a61294b60 0x555a612a9da0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f52480cede0 tx=0x7f52480094a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1158e73f020868a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:52.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.792+0000 7f52357fa640 1 -- 172.21.15.7:0/1883806134 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52480ba020 con 0x555a61294b60 2024-04-21T08:54:52.796 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.792+0000 7f52357fa640 1 -- 172.21.15.7:0/1883806134 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555a6120d880 con 0x555a61294b60 2024-04-21T08:54:52.796 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.797+0000 7f52357fa640 1 -- 172.21.15.7:0/1883806134 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52480996a0 con 0x555a61294b60 2024-04-21T08:54:52.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.798+0000 7f5258793740 1 -- 172.21.15.7:0/1883806134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a61294b60 msgr2=0x555a612a9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:52.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.798+0000 7f5258793740 1 --2- 172.21.15.7:0/1883806134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a61294b60 0x555a612a9da0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f52480cede0 tx=0x7f52480094a0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.798+0000 7f5258793740 1 -- 172.21.15.7:0/1883806134 shutdown_connections 2024-04-21T08:54:52.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.798+0000 7f5258793740 1 --2- 172.21.15.7:0/1883806134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 0x555a6129d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.798+0000 7f5258793740 1 --2- 172.21.15.7:0/1883806134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a612958d0 0x555a61299ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.798+0000 7f5258793740 1 --2- 172.21.15.7:0/1883806134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a61294b60 0x555a612a9da0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.798+0000 7f5258793740 1 -- 172.21.15.7:0/1883806134 >> 172.21.15.7:0/1883806134 conn(0x555a6125bc90 msgr2=0x555a612997c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:52.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.798+0000 7f5258793740 1 -- 172.21.15.7:0/1883806134 shutdown_connections 2024-04-21T08:54:52.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.798+0000 7f5258793740 1 -- 172.21.15.7:0/1883806134 wait complete. 2024-04-21T08:54:52.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.798+0000 7f5258793740 1 -- start start 2024-04-21T08:54:52.800 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.798+0000 7f5258793740 1 Processor -- start 2024-04-21T08:54:52.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.799+0000 7f5258793740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a61294b60 0x555a612a20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.806 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.799+0000 7f5258793740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a612958d0 0x555a612a25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.799+0000 7f5258793740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 0x555a61313900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.799+0000 7f5258793740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f523c00f000 con 0x555a61294b60 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.799+0000 7f5258793740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f523c010470 con 0x555a6129e0e0 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.799+0000 7f5258793740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f523c00f6d0 con 0x555a612958d0 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.799+0000 7f5258793740 10 client.0 _ll_register_callbacks cb 0x7f5240009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.799+0000 7f524ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 0x555a61313900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.799+0000 7f524ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 0x555a61313900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:38402/0 (socket says 172.21.15.7:38402) 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.799+0000 7f524ffff640 1 -- 172.21.15.7:0/2744477055 learned_addr learned my addr 172.21.15.7:0/2744477055 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.799+0000 7f524effd640 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a612958d0 0x555a612a25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.800+0000 7f524ffff640 1 -- 172.21.15.7:0/2744477055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a612958d0 msgr2=0x555a612a25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.800+0000 7f524ffff640 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a612958d0 0x555a612a25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.800+0000 7f524ffff640 1 -- 172.21.15.7:0/2744477055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a61294b60 msgr2=0x555a612a20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.800+0000 7f524ffff640 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a61294b60 0x555a612a20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.800+0000 7f524ffff640 1 -- 172.21.15.7:0/2744477055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a61313e40 con 0x555a6129e0e0 2024-04-21T08:54:52.809 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.810+0000 7f524ffff640 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 0x555a61313900 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f524401b040 tx=0x7f5244010700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88105f8835a08c6c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:52.814 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.813+0000 7f52337f6640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:54:52.814 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.813+0000 7f52337f6640 1 -- 172.21.15.7:0/2744477055 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524401d040 con 0x555a6129e0e0 2024-04-21T08:54:52.814 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.813+0000 7f52337f6640 1 -- 172.21.15.7:0/2744477055 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7f5244010c80 con 0x555a6129e0e0 2024-04-21T08:54:52.814 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.813+0000 7f5258793740 1 -- 172.21.15.7:0/2744477055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555a6120d880 con 0x555a6129e0e0 2024-04-21T08:54:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.816+0000 7f52337f6640 1 -- 172.21.15.7:0/2744477055 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5244018440 con 0x555a6129e0e0 2024-04-21T08:54:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.818+0000 7f5258793740 10 client.26834 fetch_fsmap learned FSMap version 23 2024-04-21T08:54:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.818+0000 7f5258793740 1 -- 172.21.15.7:0/2744477055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555a61314ff0 con 0x555a6129e0e0 2024-04-21T08:54:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.838+0000 7f52337f6640 1 -- 172.21.15.7:0/2744477055 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555a6120d880 con 0x555a6129e0e0 2024-04-21T08:54:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.838+0000 7f52337f6640 1 -- 172.21.15.7:0/2744477055 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f5244044610 con 0x555a6129e0e0 2024-04-21T08:54:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.838+0000 7f5258793740 10 client.26834 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:54:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.838+0000 7f5258793740 10 client.26834 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.838+0000 7f5258793740 20 client.26834 populate_metadata read hostname 'smithi007' 2024-04-21T08:54:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.838+0000 7f5258793740 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a61315500 0x555a61317920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.838+0000 7f5258793740 4 client.26834 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.838+0000 7f5258793740 1 -- 172.21.15.7:0/2744477055 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555a611993d0 con 0x555a61315500 2024-04-21T08:54:52.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.838+0000 7f524f7fe640 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a61315500 0x555a61317920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.839+0000 7f52337f6640 10 client.26834 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:52.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:52.839+0000 7f524f7fe640 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a61315500 0x555a61317920 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b3ac0c9efa375f9 server_cookie=e3c03ec335b3a91e in_seq=0 out_seq=0 2024-04-21T08:54:52.958 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/gc.o 2024-04-21T08:54:53.066 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/blktrace.o 2024-04-21T08:54:53.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:52 smithi016 ceph-mon[31259]: pgmap v320: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 230 KiB/s wr, 342 op/s 2024-04-21T08:54:53.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:52 smithi077 ceph-mon[31615]: pgmap v320: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 230 KiB/s wr, 342 op/s 2024-04-21T08:54:53.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:52 smithi007 ceph-mon[28477]: pgmap v320: 129 pgs: 18 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 230 KiB/s wr, 342 op/s 2024-04-21T08:54:53.546 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/isofs/dir.o 2024-04-21T08:54:53.638 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/vsyscall.o 2024-04-21T08:54:53.895 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/key.o 2024-04-21T08:54:54.181 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/timekeeping_debug.o 2024-04-21T08:54:54.242 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.242+0000 7f52337f6640 1 -- 172.21.15.7:0/2744477055 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555a611993d0 con 0x555a61315500 2024-04-21T08:54:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.242+0000 7f52337f6640 10 client.26834 handle_command_reply: tid=0 2024-04-21T08:54:54.243 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.242+0000 7f5258793740 1 client.26834 shutdown 2024-04-21T08:54:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.246+0000 7f5258793740 1 -- 172.21.15.7:0/2744477055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 msgr2=0x555a61313900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.246+0000 7f5258793740 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 0x555a61313900 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f524401b040 tx=0x7f5244010700 comp rx=0 tx=0).stop 2024-04-21T08:54:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.246+0000 7f5258793740 1 -- 172.21.15.7:0/2744477055 shutdown_connections 2024-04-21T08:54:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.246+0000 7f5258793740 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x555a61315500 0x555a61317920 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.246+0000 7f5258793740 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x555a6129e0e0 0x555a61313900 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.246+0000 7f5258793740 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555a612958d0 0x555a612a25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.246+0000 7f5258793740 1 --2- 172.21.15.7:0/2744477055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a61294b60 0x555a612a20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.246+0000 7f5258793740 1 -- 172.21.15.7:0/2744477055 >> 172.21.15.7:0/2744477055 conn(0x555a6125bc90 msgr2=0x555a6128e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.247+0000 7f52337f6640 0 client.26834 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:54:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.247+0000 7f5258793740 1 -- 172.21.15.7:0/2744477055 shutdown_connections 2024-04-21T08:54:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.253+0000 7f5258793740 1 -- 172.21.15.7:0/2744477055 wait complete. 2024-04-21T08:54:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.254+0000 7f5258793740 20 client.26834 trim_cache size 0 max 16384 2024-04-21T08:54:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:54:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:54:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:54:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:54:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.270+0000 7f5255ac1640 1 -- 172.21.15.7:0/965421305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f52380a3de0 msgr2=0x7f52380a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.270+0000 7f5255ac1640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f52380a3de0 0x7f52380a62a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5240006010 tx=0x7f5240000bc0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.270+0000 7f5255ac1640 1 -- 172.21.15.7:0/965421305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5250109a50 msgr2=0x7f52501173e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.270+0000 7f5255ac1640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5250109a50 0x7f52501173e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f524800eab0 tx=0x7f524800ef70 comp rx=0 tx=0).stop 2024-04-21T08:54:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.278+0000 7f5255ac1640 1 -- 172.21.15.7:0/965421305 shutdown_connections 2024-04-21T08:54:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.278+0000 7f5255ac1640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5250119080 0x7f5250117e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.278+0000 7f5255ac1640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f52380a3de0 0x7f52380a62a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.278+0000 7f5255ac1640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f525010e0f0 0x7f5250117920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.278+0000 7f5255ac1640 1 --2- 172.21.15.7:0/965421305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5250109a50 0x7f52501173e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.278+0000 7f5255ac1640 1 -- 172.21.15.7:0/965421305 >> 172.21.15.7:0/965421305 conn(0x7f525006d030 msgr2=0x7f5250071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.285+0000 7f5255ac1640 1 -- 172.21.15.7:0/965421305 shutdown_connections 2024-04-21T08:54:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:54:54.285+0000 7f5255ac1640 1 -- 172.21.15.7:0/965421305 wait complete. 2024-04-21T08:54:54.744 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/isofs/util.o 2024-04-21T08:54:55.221 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/keyring.o 2024-04-21T08:54:55.237 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/time/namespace.o 2024-04-21T08:54:55.304 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_events.o 2024-04-21T08:54:55.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:54 smithi016 ceph-mon[31259]: pgmap v321: 129 pgs: 17 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 70 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 302 KiB/s wr, 448 op/s 2024-04-21T08:54:55.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:54 smithi077 ceph-mon[31615]: pgmap v321: 129 pgs: 17 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 70 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 302 KiB/s wr, 448 op/s 2024-04-21T08:54:55.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:54 smithi007 ceph-mon[28477]: pgmap v321: 129 pgs: 17 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 70 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 302 KiB/s wr, 448 op/s 2024-04-21T08:54:55.638 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/isofs/rock.o 2024-04-21T08:54:56.541 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/events/hw_breakpoint.o 2024-04-21T08:54:56.599 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/isofs/export.o 2024-04-21T08:54:56.649 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/keyctl.o 2024-04-21T08:54:56.712 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/init-mm.o 2024-04-21T08:54:57.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:56 smithi016 ceph-mon[31259]: pgmap v322: 129 pgs: 17 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 73 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 213 KiB/s wr, 376 op/s 2024-04-21T08:54:57.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:56 smithi077 ceph-mon[31615]: pgmap v322: 129 pgs: 17 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 73 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 213 KiB/s wr, 376 op/s 2024-04-21T08:54:57.382 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/time/built-in.a 2024-04-21T08:54:57.397 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/isofs/joliet.o 2024-04-21T08:54:57.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:56 smithi007 ceph-mon[28477]: pgmap v322: 129 pgs: 17 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 73 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 213 KiB/s wr, 376 op/s 2024-04-21T08:54:57.462 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/permission.o 2024-04-21T08:54:57.499 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/memblock.o 2024-04-21T08:54:57.500 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/symlink.o 2024-04-21T08:54:57.748 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_export.o 2024-04-21T08:54:57.918 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/events/uprobes.o 2024-04-21T08:54:58.005 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/process_keys.o 2024-04-21T08:54:58.239 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/request_key.o 2024-04-21T08:54:58.239 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/isofs/compress.o 2024-04-21T08:54:58.502 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/sysfs.o 2024-04-21T08:54:58.677 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_event_perf.o 2024-04-21T08:54:59.222 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/madvise.o 2024-04-21T08:54:59.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:54:58 smithi016 ceph-mon[31259]: pgmap v323: 129 pgs: 17 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 74 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 437 KiB/s wr, 423 op/s 2024-04-21T08:54:59.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:54:58 smithi077 ceph-mon[31615]: pgmap v323: 129 pgs: 17 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 74 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 437 KiB/s wr, 423 op/s 2024-04-21T08:54:59.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:54:58 smithi007 ceph-mon[28477]: pgmap v323: 129 pgs: 17 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 74 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 437 KiB/s wr, 423 op/s 2024-04-21T08:54:59.439 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/request_key_auth.o 2024-04-21T08:54:59.505 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/isofs/built-in.a 2024-04-21T08:54:59.556 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/xattr.o 2024-04-21T08:54:59.621 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/client.o 2024-04-21T08:54:59.716 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/dir.o 2024-04-21T08:54:59.924 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_events_filter.o 2024-04-21T08:55:00.228 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/events/built-in.a 2024-04-21T08:55:00.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:00 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:55:00.342 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/fork.o 2024-04-21T08:55:00.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:00 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:55:00.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:00 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:55:00.635 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/user_defined.o 2024-04-21T08:55:01.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:01 smithi016 ceph-mon[31259]: pgmap v324: 129 pgs: 16 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 80 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 878 KiB/s wr, 435 op/s 2024-04-21T08:55:01.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:01 smithi077 ceph-mon[31615]: pgmap v324: 129 pgs: 16 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 80 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 878 KiB/s wr, 435 op/s 2024-04-21T08:55:01.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:01 smithi007 ceph-mon[28477]: pgmap v324: 129 pgs: 16 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 80 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 878 KiB/s wr, 435 op/s 2024-04-21T08:55:01.429 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/page_io.o 2024-04-21T08:55:01.764 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/compat.o 2024-04-21T08:55:01.963 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_events_trigger.o 2024-04-21T08:55:02.066 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/file.o 2024-04-21T08:55:02.460 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/xattr_hurd.o 2024-04-21T08:55:02.626 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/proc.o 2024-04-21T08:55:02.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: pgmap v325: 129 pgs: 16 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 80 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 968 KiB/s wr, 364 op/s 2024-04-21T08:55:02.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:02 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: pgmap v325: 129 pgs: 16 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 80 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 968 KiB/s wr, 364 op/s 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:02 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: pgmap v325: 129 pgs: 16 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 80 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 968 KiB/s wr, 364 op/s 2024-04-21T08:55:02.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:02.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:02.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:02 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.057 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/swap_state.o 2024-04-21T08:55:03.283 INFO:tasks.workunit.client.0.smithi007.stderr:./arch/x86/kernel/cpu/mkcapflags.sh: line 46: printf: write error: Resource temporarily unavailable 2024-04-21T08:55:03.311 INFO:tasks.workunit.client.0.smithi007.stderr:make[5]: *** [arch/x86/kernel/cpu/Makefile:64: arch/x86/kernel/cpu/capflags.c] Error 1 2024-04-21T08:55:03.311 INFO:tasks.workunit.client.0.smithi007.stderr:make[4]: *** [scripts/Makefile.build:480: arch/x86/kernel/cpu] Error 2 2024-04-21T08:55:03.311 INFO:tasks.workunit.client.0.smithi007.stderr:make[3]: *** [scripts/Makefile.build:480: arch/x86/kernel] Error 2 2024-04-21T08:55:03.311 INFO:tasks.workunit.client.0.smithi007.stderr:make[2]: *** [scripts/Makefile.build:480: arch/x86] Error 2 2024-04-21T08:55:03.311 INFO:tasks.workunit.client.0.smithi007.stderr:make[2]: *** Waiting for unfinished jobs.... 2024-04-21T08:55:03.333 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/xattr_trusted.o 2024-04-21T08:55:03.677 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_eprobe.o 2024-04-21T08:55:03.731 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/sysctl.o 2024-04-21T08:55:03.759 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/xattr_user.o 2024-04-21T08:55:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-04-21T08:55:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-21T08:55:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-04-21T08:55:03.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:03 smithi016 ceph-mon[31259]: Unable to set osd_memory_target on smithi016 to 185893478: error parsing value: Value '185893478' is below minimum 939524096 2024-04-21T08:55:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-04-21T08:55:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-21T08:55:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-04-21T08:55:03.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:03 smithi077 ceph-mon[31615]: Unable to set osd_memory_target on smithi016 to 185893478: error parsing value: Value '185893478' is below minimum 939524096 2024-04-21T08:55:03.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-04-21T08:55:03.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-21T08:55:03.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:03.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi016 to 177.2M 2024-04-21T08:55:03.911 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:03 smithi007 ceph-mon[28477]: Unable to set osd_memory_target on smithi016 to 185893478: error parsing value: Value '185893478' is below minimum 939524096 2024-04-21T08:55:04.034 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/exportfs/expfs.o 2024-04-21T08:55:04.140 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_kprobe.o 2024-04-21T08:55:04.433 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/fast_commit.o 2024-04-21T08:55:04.708 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/keys/keyctl_pkey.o 2024-04-21T08:55:04.742 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/clntlock.o 2024-04-21T08:55:04.748 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/getroot.o 2024-04-21T08:55:04.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:04 smithi016 ceph-mon[31259]: pgmap v326: 129 pgs: 14 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 83 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.0 MiB/s wr, 444 op/s 2024-04-21T08:55:04.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:04 smithi077 ceph-mon[31615]: pgmap v326: 129 pgs: 14 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 83 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.0 MiB/s wr, 444 op/s 2024-04-21T08:55:04.907 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:04 smithi007 ceph-mon[28477]: pgmap v326: 129 pgs: 14 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 83 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.0 MiB/s wr, 444 op/s 2024-04-21T08:55:04.973 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/swapfile.o 2024-04-21T08:55:05.305 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/exportfs/built-in.a 2024-04-21T08:55:05.455 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nls/nls_base.o 2024-04-21T08:55:05.565 INFO:tasks.workunit.client.0.smithi007.stdout: AR security/keys/built-in.a 2024-04-21T08:55:05.711 INFO:tasks.workunit.client.0.smithi007.stdout: GEN security/selinux/flask.h security/selinux/av_permissions.h 2024-04-21T08:55:05.722 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/exec_domain.o 2024-04-21T08:55:05.775 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/avc.o 2024-04-21T08:55:06.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:05 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:06.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:05 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:06.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:05 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:06.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:05 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:06.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:05 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:06.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:05 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:06.263 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nls/nls_cp437.o 2024-04-21T08:55:06.610 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/clntproc.o 2024-04-21T08:55:06.710 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/inode.o 2024-04-21T08:55:06.974 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nls/nls_ascii.o 2024-04-21T08:55:07.003 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/panic.o 2024-04-21T08:55:07.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:06 smithi016 ceph-mon[31259]: pgmap v327: 129 pgs: 13 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 91 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 948 KiB/s wr, 375 op/s 2024-04-21T08:55:07.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:06 smithi077 ceph-mon[31615]: pgmap v327: 129 pgs: 13 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 91 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 948 KiB/s wr, 375 op/s 2024-04-21T08:55:07.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:06 smithi007 ceph-mon[28477]: pgmap v327: 129 pgs: 13 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 91 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 948 KiB/s wr, 375 op/s 2024-04-21T08:55:07.187 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/error_report-traces.o 2024-04-21T08:55:07.470 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:55:07.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:55:07.551 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/orphan.o 2024-04-21T08:55:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.655+0000 7f6ee214a640 1 -- 172.21.15.7:0/1411388577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc109160 msgr2=0x7f6edc109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.655+0000 7f6ee214a640 1 --2- 172.21.15.7:0/1411388577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc109160 0x7f6edc109540 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009860 tx=0x7f6ecc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:55:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.656+0000 7f6ee214a640 1 -- 172.21.15.7:0/1411388577 shutdown_connections 2024-04-21T08:55:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.656+0000 7f6ee214a640 1 --2- 172.21.15.7:0/1411388577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6edc1150a0 0x7f6edc117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.656+0000 7f6ee214a640 1 --2- 172.21.15.7:0/1411388577 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6edc109a80 0x7f6edc114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.656+0000 7f6ee214a640 1 --2- 172.21.15.7:0/1411388577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc109160 0x7f6edc109540 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.656+0000 7f6ee214a640 1 -- 172.21.15.7:0/1411388577 >> 172.21.15.7:0/1411388577 conn(0x7f6edc06d030 msgr2=0x7f6edc06d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.658+0000 7f6ee214a640 1 -- 172.21.15.7:0/1411388577 shutdown_connections 2024-04-21T08:55:07.658 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.658+0000 7f6ee214a640 1 -- 172.21.15.7:0/1411388577 wait complete. 2024-04-21T08:55:07.658 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.658+0000 7f6ee214a640 1 Processor -- start 2024-04-21T08:55:07.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.659+0000 7f6ee214a640 1 -- start start 2024-04-21T08:55:07.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.659+0000 7f6ee214a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6edc109160 0x7f6edc10a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.659+0000 7f6ee214a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc109a80 0x7f6edc10af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.659+0000 7f6ee214a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6edc1150a0 0x7f6edc10b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.659+0000 7f6ee214a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc11ab20 con 0x7f6edc109160 2024-04-21T08:55:07.659 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.659+0000 7f6ee214a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc11a9a0 con 0x7f6edc109a80 2024-04-21T08:55:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.659+0000 7f6ee214a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc11aca0 con 0x7f6edc1150a0 2024-04-21T08:55:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.660+0000 7f6edbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6edc1150a0 0x7f6edc10b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.660+0000 7f6edbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6edc1150a0 0x7f6edc10b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:60298/0 (socket says 172.21.15.7:60298) 2024-04-21T08:55:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.660+0000 7f6edbfff640 1 -- 172.21.15.7:0/2003264081 learned_addr learned my addr 172.21.15.7:0/2003264081 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.660+0000 7f6edbfff640 1 -- 172.21.15.7:0/2003264081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc109a80 msgr2=0x7f6edc10af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.660+0000 7f6edbfff640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc109a80 0x7f6edc10af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.660+0000 7f6edbfff640 1 -- 172.21.15.7:0/2003264081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6edc109160 msgr2=0x7f6edc10a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.660+0000 7f6edbfff640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6edc109160 0x7f6edc10a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.660+0000 7f6edbfff640 1 -- 172.21.15.7:0/2003264081 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6edc1a5260 con 0x7f6edc1150a0 2024-04-21T08:55:07.661 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.660+0000 7f6edbfff640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6edc1150a0 0x7f6edc10b440 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009fa0 tx=0x7f6ed400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:07.666 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.667+0000 7f6ed8ff9640 1 -- 172.21.15.7:0/2003264081 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4004b20 con 0x7f6edc1150a0 2024-04-21T08:55:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.667+0000 7f6ed8ff9640 1 -- 172.21.15.7:0/2003264081 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ed4003e20 con 0x7f6edc1150a0 2024-04-21T08:55:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.667+0000 7f6ed8ff9640 1 -- 172.21.15.7:0/2003264081 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed400b570 con 0x7f6edc1150a0 2024-04-21T08:55:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.667+0000 7f6ee214a640 1 -- 172.21.15.7:0/2003264081 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6edc1a5550 con 0x7f6edc1150a0 2024-04-21T08:55:07.667 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.667+0000 7f6ee214a640 1 -- 172.21.15.7:0/2003264081 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6edc1a5a50 con 0x7f6edc1150a0 2024-04-21T08:55:07.670 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.670+0000 7f6ee4e0f740 1 Processor -- start 2024-04-21T08:55:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.671+0000 7f6ed8ff9640 1 -- 172.21.15.7:0/2003264081 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f6ed401d020 con 0x7f6edc1150a0 2024-04-21T08:55:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.671+0000 7f6ee4e0f740 1 -- start start 2024-04-21T08:55:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.672+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561727678d0 0x556172766620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.672+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556172766b60 0x556172766fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.672+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727700e0 0x5561727724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.672+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5561726cb8d0 con 0x5561727700e0 2024-04-21T08:55:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.672+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556172668e80 con 0x5561727678d0 2024-04-21T08:55:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.672+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5561726c7890 con 0x556172766b60 2024-04-21T08:55:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.672+0000 7f6edb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556172766b60 0x556172766fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.672+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561727678d0 0x556172766620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.672+0000 7f6edb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556172766b60 0x556172766fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:60306/0 (socket says 172.21.15.7:60306) 2024-04-21T08:55:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.672+0000 7f6edb7fe640 1 -- 172.21.15.7:0/1632257228 learned_addr learned my addr 172.21.15.7:0/1632257228 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:07.674 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.673+0000 7f6edaffd640 1 --2- 172.21.15.7:0/1632257228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727700e0 0x5561727724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.674 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.673+0000 7f6edb7fe640 1 -- 172.21.15.7:0/1632257228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561727678d0 msgr2=0x556172766620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:07.674 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.673+0000 7f6edb7fe640 1 --2- 172.21.15.7:0/1632257228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561727678d0 0x556172766620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.674 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.673+0000 7f6edb7fe640 1 -- 172.21.15.7:0/1632257228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727700e0 msgr2=0x5561727724d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:07.675 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.676+0000 7f6edb7fe640 1 --2- 172.21.15.7:0/1632257228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727700e0 0x5561727724d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.676 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.677+0000 7f6edb7fe640 1 -- 172.21.15.7:0/1632257228 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55617268a320 con 0x556172766b60 2024-04-21T08:55:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.677+0000 7f6edb7fe640 1 --2- 172.21.15.7:0/1632257228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556172766b60 0x556172766fe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc02c1c0 tx=0x7f6ecc009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=326bf0af4f5e1840 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.677+0000 7f6ed8ff9640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ec40a3f20 0x7f6ec40a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.678+0000 7f6ed8ff9640 1 -- 172.21.15.7:0/2003264081 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7f6ed40d1bc0 con 0x7f6edc1150a0 2024-04-21T08:55:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.686+0000 7f6ec9ffb640 1 -- 172.21.15.7:0/1632257228 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc039040 con 0x556172766b60 2024-04-21T08:55:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.686+0000 7f6ec9ffb640 1 -- 172.21.15.7:0/1632257228 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ecc034d70 con 0x556172766b60 2024-04-21T08:55:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.686+0000 7f6edaffd640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ec40a3f20 0x7f6ec40a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.686 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.687+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/1632257228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556172766b60 msgr2=0x556172766fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.687+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/1632257228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556172766b60 0x556172766fe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc02c1c0 tx=0x7f6ecc009490 comp rx=0 tx=0).stop 2024-04-21T08:55:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.687+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/1632257228 shutdown_connections 2024-04-21T08:55:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.687+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/1632257228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727700e0 0x5561727724d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.687+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/1632257228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556172766b60 0x556172766fe0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.687+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/1632257228 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561727678d0 0x556172766620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.687+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/1632257228 >> 172.21.15.7:0/1632257228 conn(0x55617272dc90 msgr2=0x556172761850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.687+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/1632257228 shutdown_connections 2024-04-21T08:55:07.688 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.687+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/1632257228 wait complete. 2024-04-21T08:55:07.688 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.688+0000 7f6ee4e0f740 1 -- start start 2024-04-21T08:55:07.688 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.688+0000 7f6ee4e0f740 1 Processor -- start 2024-04-21T08:55:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.688+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727678d0 0x55617276f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.689+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x55617276ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.689+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55617276cfa0 0x55617276d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.689+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556172772e00 con 0x5561727678d0 2024-04-21T08:55:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.689+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x556172653eb0 con 0x55617276cfa0 2024-04-21T08:55:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.689+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556172772f80 con 0x5561727700e0 2024-04-21T08:55:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.689+0000 7f6ee4e0f740 10 client.0 _ll_register_callbacks cb 0x556172680b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.689+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x55617276ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.689+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x55617276ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:60310/0 (socket says 172.21.15.7:60310) 2024-04-21T08:55:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.689+0000 7f6edaffd640 1 -- 172.21.15.7:0/2436276996 learned_addr learned my addr 172.21.15.7:0/2436276996 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.689+0000 7f6edbfff640 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55617276cfa0 0x55617276d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.689+0000 7f6edb7fe640 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727678d0 0x55617276f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.691+0000 7f6edbfff640 1 -- 172.21.15.7:0/2436276996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 msgr2=0x55617276ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:07.690 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.691+0000 7f6edbfff640 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x55617276ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.691+0000 7f6edbfff640 1 -- 172.21.15.7:0/2436276996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727678d0 msgr2=0x55617276f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.691+0000 7f6edbfff640 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727678d0 0x55617276f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.691+0000 7f6edbfff640 1 -- 172.21.15.7:0/2436276996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561723de210 con 0x55617276cfa0 2024-04-21T08:55:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.691+0000 7f6edbfff640 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55617276cfa0 0x55617276d420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40cee40 tx=0x7f6ed409ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d94f2208cf4de0ff server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.691+0000 7f6ebeffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:55:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.691+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2436276996 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed401a050 con 0x55617276cfa0 2024-04-21T08:55:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.692+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2436276996 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7f6ed4099d00 con 0x55617276cfa0 2024-04-21T08:55:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.692+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2436276996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5561726df880 con 0x55617276cfa0 2024-04-21T08:55:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.696+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2436276996 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6ed40cf0c0 con 0x55617276cfa0 2024-04-21T08:55:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.696+0000 7f6ee4e0f740 10 client.26843 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.696+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2436276996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5561724aa620 con 0x55617276cfa0 2024-04-21T08:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.696+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2436276996 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5561726df880 con 0x55617276cfa0 2024-04-21T08:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.697+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2436276996 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6ed400b040 con 0x55617276cfa0 2024-04-21T08:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.697+0000 7f6ee4e0f740 10 client.26843 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.697+0000 7f6ee4e0f740 10 client.26843 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.697+0000 7f6ee4e0f740 20 client.26843 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.697+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5561727803c0 0x5561727827f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.697+0000 7f6ee4e0f740 4 client.26843 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.697+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2436276996 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55617266b3d0 con 0x5561727803c0 2024-04-21T08:55:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.697+0000 7f6edb7fe640 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5561727803c0 0x5561727827f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.698+0000 7f6edb7fe640 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5561727803c0 0x5561727827f0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e77a6a828de750cc server_cookie=758e6d6edd2d2fea in_seq=0 out_seq=0 2024-04-21T08:55:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.698+0000 7f6ebeffd640 10 client.26843 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.698+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2436276996 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55617266b3d0 con 0x5561727803c0 2024-04-21T08:55:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.698+0000 7f6ebeffd640 10 client.26843 handle_command_reply: tid=0 2024-04-21T08:55:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.698+0000 7f6ee4e0f740 1 client.26843 shutdown 2024-04-21T08:55:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.699+0000 7f6edaffd640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ec40a3f20 0x7f6ec40a63e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55617276e030 tx=0x7f6ed00010d0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2436276996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55617276cfa0 msgr2=0x55617276d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55617276cfa0 0x55617276d420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40cee40 tx=0x7f6ed409ded0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2436276996 shutdown_connections 2024-04-21T08:55:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5561727803c0 0x5561727827f0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.705 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55617276cfa0 0x55617276d420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x55617276ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2436276996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727678d0 0x55617276f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2436276996 >> 172.21.15.7:0/2436276996 conn(0x55617272dc90 msgr2=0x556172762d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ebeffd640 0 client.26843 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2436276996 shutdown_connections 2024-04-21T08:55:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2436276996 wait complete. 2024-04-21T08:55:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.705+0000 7f6ee4e0f740 20 client.26843 trim_cache size 0 max 16384 2024-04-21T08:55:07.724 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nls/nls_iso8859-1.o 2024-04-21T08:55:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.751+0000 7f6ee4e0f740 1 Processor -- start 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.751+0000 7f6ee4e0f740 1 -- start start 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.751+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727678d0 0x55617277bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x5561727714d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556172763910 0x55617276f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec000e280 con 0x5561727678d0 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec0010ae0 con 0x556172763910 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec000e8d0 con 0x5561727700e0 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6edbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556172763910 0x55617276f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6edbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556172763910 0x55617276f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:47994/0 (socket says 172.21.15.7:47994) 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6edbfff640 1 -- 172.21.15.7:0/2592590727 learned_addr learned my addr 172.21.15.7:0/2592590727 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6edbfff640 1 -- 172.21.15.7:0/2592590727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 msgr2=0x5561727714d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6edbfff640 1 --2- 172.21.15.7:0/2592590727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x5561727714d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6edbfff640 1 -- 172.21.15.7:0/2592590727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727678d0 msgr2=0x55617277bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6edbfff640 1 --2- 172.21.15.7:0/2592590727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727678d0 0x55617277bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.752+0000 7f6edbfff640 1 -- 172.21.15.7:0/2592590727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556172772940 con 0x556172763910 2024-04-21T08:55:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.753+0000 7f6edbfff640 1 --2- 172.21.15.7:0/2592590727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556172763910 0x55617276f2a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40043b0 tx=0x7f6ed409d010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22c24b4ef8b3d8ca server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:07.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.763+0000 7f6ec97fa640 1 -- 172.21.15.7:0/2592590727 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed401a070 con 0x556172763910 2024-04-21T08:55:07.762 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.763+0000 7f6ec97fa640 1 -- 172.21.15.7:0/2592590727 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ed40a6300 con 0x556172763910 2024-04-21T08:55:07.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.764+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2592590727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556172763910 msgr2=0x55617276f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:07.763 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.764+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2592590727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556172763910 0x55617276f2a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40043b0 tx=0x7f6ed409d010 comp rx=0 tx=0).stop 2024-04-21T08:55:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.766+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2592590727 shutdown_connections 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.766+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2592590727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x556172763910 0x55617276f2a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.766+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2592590727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x5561727714d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.766+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2592590727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5561727678d0 0x55617277bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.766+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2592590727 >> 172.21.15.7:0/2592590727 conn(0x55617272dc90 msgr2=0x55617276cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.763+0000 7f6ec97fa640 1 -- 172.21.15.7:0/2592590727 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed400e0b0 con 0x556172763910 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.767+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2592590727 shutdown_connections 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.767+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2592590727 wait complete. 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.767+0000 7f6ee4e0f740 1 -- start start 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.767+0000 7f6ee4e0f740 1 Processor -- start 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.768+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561727678d0 0x5561727870f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.768+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x556172787630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.774 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.768+0000 7f6ee4e0f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556172787c80 0x5561727772f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.768+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec000efa0 con 0x556172787c80 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.768+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec0010410 con 0x5561727678d0 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.768+0000 7f6ee4e0f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec000f670 con 0x5561727700e0 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.768+0000 7f6ee4e0f740 10 client.0 _ll_register_callbacks cb 0x7f6ed40048c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.768+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x556172787630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.768+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x556172787630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:60324/0 (socket says 172.21.15.7:60324) 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.768+0000 7f6edaffd640 1 -- 172.21.15.7:0/2803439352 learned_addr learned my addr 172.21.15.7:0/2803439352 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.769+0000 7f6edaffd640 1 -- 172.21.15.7:0/2803439352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561727678d0 msgr2=0x5561727870f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.769+0000 7f6edaffd640 1 --2- 172.21.15.7:0/2803439352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561727678d0 0x5561727870f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.769+0000 7f6edaffd640 1 -- 172.21.15.7:0/2803439352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556172787c80 msgr2=0x5561727772f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.769+0000 7f6edaffd640 1 --2- 172.21.15.7:0/2803439352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556172787c80 0x5561727772f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.769+0000 7f6edaffd640 1 -- 172.21.15.7:0/2803439352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556172777b20 con 0x5561727700e0 2024-04-21T08:55:07.775 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.770+0000 7f6edaffd640 1 --2- 172.21.15.7:0/2803439352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x556172787630 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6edc10c210 tx=0x7f6ed0001140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fccf6439b4407c39 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.781+0000 7f6ebeffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:55:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.781+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2803439352 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0029060 con 0x5561727700e0 2024-04-21T08:55:07.780 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.781+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2803439352 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9656+0+0 (secure 0 0 0) 0x7f6ed0004770 con 0x5561727700e0 2024-04-21T08:55:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.781+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2803439352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ed40a6300 con 0x5561727700e0 2024-04-21T08:55:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.781+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2803439352 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6ed0030050 con 0x5561727700e0 2024-04-21T08:55:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.782+0000 7f6ee4e0f740 10 client.26830 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.782+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2803439352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5561727e69d0 con 0x5561727700e0 2024-04-21T08:55:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.782+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2803439352 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ed40a6300 con 0x5561727700e0 2024-04-21T08:55:07.782 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.782+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2803439352 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6ed0015070 con 0x5561727700e0 2024-04-21T08:55:07.787 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.787+0000 7f6ee4e0f740 10 client.26830 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:07.787 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.787+0000 7f6ee4e0f740 10 client.26830 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:07.787 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.787+0000 7f6ee4e0f740 20 client.26830 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:07.787 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.787+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2803439352 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5561727e7130 0x5561727e9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:07.787 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.787+0000 7f6ee4e0f740 4 client.26830 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:07.787 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.787+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2803439352 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5561726d92f0 con 0x5561727e7130 2024-04-21T08:55:07.793 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.794+0000 7f6edb7fe640 1 --2- 172.21.15.7:0/2803439352 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5561727e7130 0x5561727e9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:07.794 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.794+0000 7f6edb7fe640 1 --2- 172.21.15.7:0/2803439352 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5561727e7130 0x5561727e9550 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c77a6c5d4eedc45 server_cookie=7c3909a6986f06f1 in_seq=0 out_seq=0 2024-04-21T08:55:07.794 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:07.794+0000 7f6ebeffd640 10 client.26830 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:08.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:07 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:07 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:07 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:07 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:07 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:07 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:07 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:07 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:07 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:55:08.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:07 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:55:08.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:07 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:08.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.088+0000 7fae35481640 1 -- 172.21.15.7:0/3485768583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae301106a0 msgr2=0x7fae30109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:08.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.088+0000 7fae35481640 1 --2- 172.21.15.7:0/3485768583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae301106a0 0x7fae30109150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fae28009be0 tx=0x7fae2802bf10 comp rx=0 tx=0).stop 2024-04-21T08:55:08.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.091+0000 7fae35481640 1 -- 172.21.15.7:0/3485768583 shutdown_connections 2024-04-21T08:55:08.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.091+0000 7fae35481640 1 --2- 172.21.15.7:0/3485768583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae3010e1b0 0x7fae3010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.091+0000 7fae35481640 1 --2- 172.21.15.7:0/3485768583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae30110b10 0x7fae30109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.091+0000 7fae35481640 1 --2- 172.21.15.7:0/3485768583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae301106a0 0x7fae30109150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.091+0000 7fae35481640 1 -- 172.21.15.7:0/3485768583 >> 172.21.15.7:0/3485768583 conn(0x7fae3006e360 msgr2=0x7fae3006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:08.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.091+0000 7fae35481640 1 -- 172.21.15.7:0/3485768583 shutdown_connections 2024-04-21T08:55:08.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.091+0000 7fae35481640 1 -- 172.21.15.7:0/3485768583 wait complete. 2024-04-21T08:55:08.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.091+0000 7fae35481640 1 Processor -- start 2024-04-21T08:55:08.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.100+0000 7fae35481640 1 -- start start 2024-04-21T08:55:08.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.100+0000 7fae35481640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae3010e1b0 0x7fae301a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:08.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.100+0000 7fae35481640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae301106a0 0x7fae301a7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:08.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.100+0000 7fae35481640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae30110b10 0x7fae301abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:08.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.100+0000 7fae35481640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fae30121180 con 0x7fae3010e1b0 2024-04-21T08:55:08.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.100+0000 7fae35481640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae30121000 con 0x7fae301106a0 2024-04-21T08:55:08.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.100+0000 7fae35481640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fae30121300 con 0x7fae30110b10 2024-04-21T08:55:08.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.100+0000 7fae2e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae301106a0 0x7fae301a7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:08.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.101+0000 7fae2e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae301106a0 0x7fae301a7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:48014/0 (socket says 172.21.15.7:48014) 2024-04-21T08:55:08.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.101+0000 7fae2e7fc640 1 -- 172.21.15.7:0/4221528017 learned_addr learned my addr 172.21.15.7:0/4221528017 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:08.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.101+0000 7fae2e7fc640 1 -- 172.21.15.7:0/4221528017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae30110b10 msgr2=0x7fae301abf00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:55:08.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.101+0000 7fae2e7fc640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae30110b10 0x7fae301abf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.101+0000 7fae2e7fc640 1 -- 172.21.15.7:0/4221528017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae3010e1b0 msgr2=0x7fae301a76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:08.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.101+0000 7fae2e7fc640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae3010e1b0 0x7fae301a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.101+0000 7fae2e7fc640 1 -- 172.21.15.7:0/4221528017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae301ac630 con 0x7fae301106a0 2024-04-21T08:55:08.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.101+0000 7fae2e7fc640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae301106a0 0x7fae301a7c20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae2000d740 tx=0x7fae2000dc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:07 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:07 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:07 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:07 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:07 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:07 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:07 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:07 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:07 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:55:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:07 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:55:08.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:07 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.106+0000 7fae1ffff640 1 -- 172.21.15.7:0/4221528017 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20017070 con 0x7fae301106a0 2024-04-21T08:55:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.106+0000 7fae1ffff640 1 -- 172.21.15.7:0/4221528017 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae2000de50 con 0x7fae301106a0 2024-04-21T08:55:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.106+0000 7fae1ffff640 1 -- 172.21.15.7:0/4221528017 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2000f620 con 0x7fae301106a0 2024-04-21T08:55:08.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.111+0000 7fae35481640 1 -- 172.21.15.7:0/4221528017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae301ac920 con 0x7fae301106a0 2024-04-21T08:55:08.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.111+0000 7fae35481640 1 -- 172.21.15.7:0/4221528017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae301b41b0 con 0x7fae301106a0 2024-04-21T08:55:08.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.114+0000 7fae38073740 1 Processor -- start 2024-04-21T08:55:08.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.115+0000 7fae38073740 1 -- start start 2024-04-21T08:55:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.115+0000 7fae38073740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e32186580 0x561e32186960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.115+0000 7fae38073740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186ea0 0x561e3218f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.115+0000 7fae38073740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e3218fa80 0x561e32191e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.115+0000 7fae38073740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561e320eb1e0 con 0x561e3218fa80 2024-04-21T08:55:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.115+0000 7fae38073740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561e32088390 con 0x561e32186ea0 2024-04-21T08:55:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.115+0000 7fae38073740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561e320e71a0 con 0x561e32186580 2024-04-21T08:55:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.116+0000 7fae2effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186ea0 0x561e3218f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.116+0000 7fae2effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186ea0 0x561e3218f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:48022/0 (socket says 172.21.15.7:48022) 2024-04-21T08:55:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.116+0000 7fae2effd640 1 -- 172.21.15.7:0/784250578 learned_addr learned my addr 172.21.15.7:0/784250578 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.116+0000 7fae2effd640 1 -- 172.21.15.7:0/784250578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e32186580 msgr2=0x561e32186960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.116+0000 7fae2effd640 1 --2- 172.21.15.7:0/784250578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e32186580 0x561e32186960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.116+0000 7fae2effd640 1 -- 172.21.15.7:0/784250578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e3218fa80 msgr2=0x561e32191e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.116+0000 7fae2effd640 1 --2- 172.21.15.7:0/784250578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e3218fa80 0x561e32191e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.116+0000 7fae2effd640 1 -- 172.21.15.7:0/784250578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e31ec77d0 con 0x561e32186ea0 2024-04-21T08:55:08.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.116+0000 7fae2effd640 1 --2- 172.21.15.7:0/784250578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186ea0 0x561e3218f540 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fae28036040 tx=0x7fae280098f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=192755af6637eeaf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:08.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.121+0000 7fae1d7fa640 1 -- 172.21.15.7:0/784250578 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28042020 con 0x561e32186ea0 2024-04-21T08:55:08.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.121+0000 7fae1d7fa640 1 -- 172.21.15.7:0/784250578 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae28033890 con 0x561e32186ea0 2024-04-21T08:55:08.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.122+0000 7fae1ffff640 1 -- 172.21.15.7:0/4221528017 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fae2001a070 con 0x7fae301106a0 2024-04-21T08:55:08.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.122+0000 7fae38073740 1 -- 172.21.15.7:0/784250578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186ea0 msgr2=0x561e3218f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:08.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.122+0000 7fae38073740 1 --2- 172.21.15.7:0/784250578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186ea0 0x561e3218f540 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fae28036040 tx=0x7fae280098f0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.123+0000 7fae1ffff640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fae180a3f20 0x7fae180a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:08.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.123+0000 7fae1ffff640 1 -- 172.21.15.7:0/4221528017 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7fae200d0290 con 0x7fae301106a0 2024-04-21T08:55:08.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.136+0000 7fae2f7fe640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fae180a3f20 0x7fae180a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:08.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.139+0000 7fae38073740 1 -- 172.21.15.7:0/784250578 shutdown_connections 2024-04-21T08:55:08.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.139+0000 7fae38073740 1 --2- 172.21.15.7:0/784250578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e3218fa80 0x561e32191e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.139+0000 7fae38073740 1 --2- 172.21.15.7:0/784250578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186ea0 0x561e3218f540 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.139+0000 7fae38073740 1 --2- 172.21.15.7:0/784250578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e32186580 0x561e32186960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.139+0000 7fae38073740 1 -- 172.21.15.7:0/784250578 >> 172.21.15.7:0/784250578 conn(0x561e32180cb0 msgr2=0x561e32181090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:08.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.147+0000 7fae2f7fe640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fae180a3f20 0x7fae180a63e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fae24009570 tx=0x7fae24006bd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:07 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:07 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:07 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:07 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:07 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:07 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:07 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:07 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:08.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:07 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:55:08.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:07 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:55:08.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:07 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:55:08.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.157+0000 7fae38073740 1 -- 172.21.15.7:0/784250578 shutdown_connections 2024-04-21T08:55:08.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.158+0000 7fae38073740 1 -- 172.21.15.7:0/784250578 wait complete. 2024-04-21T08:55:08.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.158+0000 7fae38073740 1 -- start start 2024-04-21T08:55:08.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.158+0000 7fae38073740 1 Processor -- start 2024-04-21T08:55:08.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.164+0000 7fae38073740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186580 0x561e3218cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:08.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.164+0000 7fae38073740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e3218cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:08.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.164+0000 7fae38073740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e3218d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:08.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.164+0000 7fae38073740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561e32192860 con 0x561e32186ea0 2024-04-21T08:55:08.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.164+0000 7fae38073740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561e320a42f0 con 0x561e32186580 2024-04-21T08:55:08.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.164+0000 7fae38073740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561e321929e0 con 0x561e3218fa80 2024-04-21T08:55:08.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.164+0000 7fae38073740 10 client.0 _ll_register_callbacks cb 0x561e320b1580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:08.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.167+0000 7fae2e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e3218cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:08.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.167+0000 7fae2e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e3218cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42244/0 (socket says 172.21.15.7:42244) 2024-04-21T08:55:08.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.167+0000 7fae2e7fc640 1 -- 172.21.15.7:0/2871725096 learned_addr learned my addr 172.21.15.7:0/2871725096 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:08.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.170+0000 7fae2f7fe640 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e3218d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:08.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.170+0000 7fae2effd640 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186580 0x561e3218cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:08.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.171+0000 7fae2e7fc640 1 -- 172.21.15.7:0/2871725096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 msgr2=0x561e3218d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:08.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.171+0000 7fae2e7fc640 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e3218d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.171+0000 7fae2e7fc640 1 -- 172.21.15.7:0/2871725096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186580 msgr2=0x561e3218cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:08.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.171+0000 7fae2e7fc640 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186580 0x561e3218cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:08.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.171+0000 7fae2e7fc640 1 -- 172.21.15.7:0/2871725096 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e31dfd3f0 con 0x561e32186ea0 2024-04-21T08:55:08.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.177+0000 7fae2e7fc640 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e3218cff0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fae200981f0 tx=0x7fae2009b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1646a7ce9ad13d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:08.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.189+0000 7fae127fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:55:08.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.199+0000 7fae127fc640 1 -- 172.21.15.7:0/2871725096 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae200b4020 con 0x561e32186ea0 2024-04-21T08:55:08.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.199+0000 7fae127fc640 1 -- 172.21.15.7:0/2871725096 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7fae200cd740 con 0x561e32186ea0 2024-04-21T08:55:08.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.199+0000 7fae38073740 1 -- 172.21.15.7:0/2871725096 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae28033890 con 0x561e32186ea0 2024-04-21T08:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.214+0000 7fae127fc640 1 -- 172.21.15.7:0/2871725096 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fae200cd940 con 0x561e32186ea0 2024-04-21T08:55:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.222+0000 7fae38073740 10 client.17226 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.222+0000 7fae38073740 1 -- 172.21.15.7:0/2871725096 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561e31dbd0f0 con 0x561e32186ea0 2024-04-21T08:55:08.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.254+0000 7fae127fc640 1 -- 172.21.15.7:0/2871725096 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae28033890 con 0x561e32186ea0 2024-04-21T08:55:08.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.254+0000 7fae127fc640 1 -- 172.21.15.7:0/2871725096 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fae20097ae0 con 0x561e32186ea0 2024-04-21T08:55:08.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.255+0000 7fae38073740 10 client.17226 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:08.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.255+0000 7fae38073740 10 client.17226 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:08.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.255+0000 7fae38073740 20 client.17226 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:08.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.255+0000 7fae38073740 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561e321b07d0 0x561e321b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:08.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.255+0000 7fae38073740 4 client.17226 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:08.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.255+0000 7fae38073740 1 -- 172.21.15.7:0/2871725096 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e320f8c00 con 0x561e321b07d0 2024-04-21T08:55:08.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.262+0000 7fae2effd640 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561e321b07d0 0x561e321b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.277+0000 7fae2effd640 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561e321b07d0 0x561e321b2bc0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d856dc18fbbfd8a0 server_cookie=77908ed088dfbca8 in_seq=0 out_seq=0 2024-04-21T08:55:08.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:08.277+0000 7fae127fc640 10 client.17226 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:08.360 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/swap_slots.o 2024-04-21T08:55:08.400 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/acl.o 2024-04-21T08:55:08.497 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/hooks.o 2024-04-21T08:55:08.550 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nls/nls_utf8.o 2024-04-21T08:55:08.572 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/power-traces.o 2024-04-21T08:55:08.843 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/clntxdr.o 2024-04-21T08:55:09.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:08 smithi016 ceph-mon[31259]: pgmap v328: 129 pgs: 13 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 91 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 966 KiB/s wr, 431 op/s 2024-04-21T08:55:09.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:08 smithi016 ceph-mon[31259]: Adjusting osd_memory_target on smithi007 to 177.2M 2024-04-21T08:55:09.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:08 smithi016 ceph-mon[31259]: Unable to set osd_memory_target on smithi007 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-21T08:55:09.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:08 smithi016 ceph-mon[31259]: osdmap e108: 12 total, 12 up, 12 in 2024-04-21T08:55:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:08 smithi077 ceph-mon[31615]: pgmap v328: 129 pgs: 13 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 91 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 966 KiB/s wr, 431 op/s 2024-04-21T08:55:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:08 smithi077 ceph-mon[31615]: Adjusting osd_memory_target on smithi007 to 177.2M 2024-04-21T08:55:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:08 smithi077 ceph-mon[31615]: Unable to set osd_memory_target on smithi007 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-21T08:55:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:08 smithi077 ceph-mon[31615]: osdmap e108: 12 total, 12 up, 12 in 2024-04-21T08:55:09.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:08 smithi007 ceph-mon[28477]: pgmap v328: 129 pgs: 13 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 91 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 966 KiB/s wr, 431 op/s 2024-04-21T08:55:09.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:08 smithi007 ceph-mon[28477]: Adjusting osd_memory_target on smithi007 to 177.2M 2024-04-21T08:55:09.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:08 smithi007 ceph-mon[28477]: Unable to set osd_memory_target on smithi007 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-21T08:55:09.154 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:08 smithi007 ceph-mon[28477]: osdmap e108: 12 total, 12 up, 12 in 2024-04-21T08:55:09.200 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/nls/built-in.a 2024-04-21T08:55:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.270+0000 7f6ebeffd640 1 -- 172.21.15.7:0/2803439352 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5561726d92f0 con 0x5561727e7130 2024-04-21T08:55:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.270+0000 7f6ebeffd640 10 client.26830 handle_command_reply: tid=0 2024-04-21T08:55:09.270 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.270+0000 7f6ee4e0f740 1 client.26830 shutdown 2024-04-21T08:55:09.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.272+0000 7fae127fc640 1 -- 172.21.15.7:0/2871725096 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561e320f8c00 con 0x561e321b07d0 2024-04-21T08:55:09.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.272+0000 7fae127fc640 10 client.17226 handle_command_reply: tid=0 2024-04-21T08:55:09.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.272+0000 7fae38073740 1 client.17226 shutdown 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.279+0000 7fae38073740 1 -- 172.21.15.7:0/2871725096 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 msgr2=0x561e3218cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.279+0000 7fae38073740 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e3218cff0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fae200981f0 tx=0x7fae2009b7e0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.279+0000 7fae38073740 1 -- 172.21.15.7:0/2871725096 shutdown_connections 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.279+0000 7fae38073740 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561e321b07d0 0x561e321b2bc0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.279+0000 7fae38073740 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e3218d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.279+0000 7fae38073740 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e3218cff0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.279+0000 7fae38073740 1 --2- 172.21.15.7:0/2871725096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186580 0x561e3218cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.279+0000 7fae38073740 1 -- 172.21.15.7:0/2871725096 >> 172.21.15.7:0/2871725096 conn(0x561e32180cb0 msgr2=0x561e32182a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.280+0000 7fae127fc640 0 client.17226 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.280+0000 7fae38073740 1 -- 172.21.15.7:0/2871725096 shutdown_connections 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.280+0000 7fae38073740 1 -- 172.21.15.7:0/2871725096 wait complete. 2024-04-21T08:55:09.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.280+0000 7fae38073740 20 client.17226 trim_cache size 0 max 16384 2024-04-21T08:55:09.285 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.284+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2803439352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 msgr2=0x556172787630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:09.285 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.284+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2803439352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x556172787630 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6edc10c210 tx=0x7f6ed0001140 comp rx=0 tx=0).stop 2024-04-21T08:55:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.301+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2803439352 shutdown_connections 2024-04-21T08:55:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.301+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2803439352 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5561727e7130 0x5561727e9550 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.301+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2803439352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556172787c80 0x5561727772f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.301+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2803439352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561727700e0 0x556172787630 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.301+0000 7f6ebeffd640 0 client.26830 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.301+0000 7f6ee4e0f740 1 --2- 172.21.15.7:0/2803439352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5561727678d0 0x5561727870f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.301+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2803439352 >> 172.21.15.7:0/2803439352 conn(0x55617272dc90 msgr2=0x556172760040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.301+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2803439352 shutdown_connections 2024-04-21T08:55:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.301+0000 7f6ee4e0f740 1 -- 172.21.15.7:0/2803439352 wait complete. 2024-04-21T08:55:09.302 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/selinuxfs.o 2024-04-21T08:55:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.317+0000 7f6ee4e0f740 20 client.26830 trim_cache size 0 max 16384 2024-04-21T08:55:09.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:55:09.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:55:09.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:55:09.317 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:55:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.329+0000 7f6ee214a640 1 -- 172.21.15.7:0/2003264081 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ec40a3f20 msgr2=0x7f6ec40a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.329+0000 7f6ee214a640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ec40a3f20 0x7f6ec40a63e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55617276e030 tx=0x7f6ed00010d0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.329+0000 7f6ee214a640 1 -- 172.21.15.7:0/2003264081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6edc1150a0 msgr2=0x7f6edc10b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.329+0000 7f6ee214a640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6edc1150a0 0x7f6edc10b440 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009fa0 tx=0x7f6ed400c800 comp rx=0 tx=0).stop 2024-04-21T08:55:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.337+0000 7f6ee214a640 1 -- 172.21.15.7:0/2003264081 shutdown_connections 2024-04-21T08:55:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.337+0000 7f6ee214a640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6ec40a3f20 0x7f6ec40a63e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.337+0000 7f6ee214a640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6edc1150a0 0x7f6edc10b440 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.337+0000 7f6ee214a640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc109a80 0x7f6edc10af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.337+0000 7f6ee214a640 1 --2- 172.21.15.7:0/2003264081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6edc109160 0x7f6edc10a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.337+0000 7f6ee214a640 1 -- 172.21.15.7:0/2003264081 >> 172.21.15.7:0/2003264081 conn(0x7f6edc06d030 msgr2=0x7f6edc070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.337+0000 7f6ee214a640 1 -- 172.21.15.7:0/2003264081 shutdown_connections 2024-04-21T08:55:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:09.337+0000 7f6ee214a640 1 -- 172.21.15.7:0/2003264081 wait complete. 2024-04-21T08:55:09.380 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cpu.o 2024-04-21T08:55:09.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.397+0000 7fae38073740 1 Processor -- start 2024-04-21T08:55:09.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.397+0000 7fae38073740 1 -- start start 2024-04-21T08:55:09.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.397+0000 7fae38073740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186580 0x561e3218f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:09.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.404+0000 7fae38073740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186ea0 0x561e3219b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:09.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.404+0000 7fae38073740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e3218b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:09.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.404+0000 7fae38073740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fae30068e00 con 0x561e32186580 2024-04-21T08:55:09.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.404+0000 7fae38073740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae301b6680 con 0x561e32186ea0 2024-04-21T08:55:09.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.404+0000 7fae38073740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fae301b4470 con 0x561e3218fa80 2024-04-21T08:55:09.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.404+0000 7fae2f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e3218b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:09.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.404+0000 7fae2f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e3218b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:60380/0 (socket says 172.21.15.7:60380) 2024-04-21T08:55:09.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.404+0000 7fae2f7fe640 1 -- 172.21.15.7:0/326045491 learned_addr learned my addr 172.21.15.7:0/326045491 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:09.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.404+0000 7fae2f7fe640 1 -- 172.21.15.7:0/326045491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186ea0 msgr2=0x561e3219b7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:09.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.404+0000 7fae2f7fe640 1 --2- 172.21.15.7:0/326045491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186ea0 0x561e3219b7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.404+0000 7fae2f7fe640 1 -- 172.21.15.7:0/326045491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186580 msgr2=0x561e3218f1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:09.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.414+0000 7fae2f7fe640 1 --2- 172.21.15.7:0/326045491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186580 0x561e3218f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.414+0000 7fae2f7fe640 1 -- 172.21.15.7:0/326045491 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e321b31f0 con 0x561e3218fa80 2024-04-21T08:55:09.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.414+0000 7fae2f7fe640 1 --2- 172.21.15.7:0/326045491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e3218b9f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fae24013a60 tx=0x7fae24012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddc5cc1bd2a2c811 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:09.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.414+0000 7fae1cff9640 1 -- 172.21.15.7:0/326045491 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae24012ca0 con 0x561e3218fa80 2024-04-21T08:55:09.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.415+0000 7fae1cff9640 1 -- 172.21.15.7:0/326045491 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae24024070 con 0x561e3218fa80 2024-04-21T08:55:09.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.415+0000 7fae1cff9640 1 -- 172.21.15.7:0/326045491 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2401e630 con 0x561e3218fa80 2024-04-21T08:55:09.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.418+0000 7fae38073740 1 -- 172.21.15.7:0/326045491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 msgr2=0x561e3218b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:09.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.418+0000 7fae38073740 1 --2- 172.21.15.7:0/326045491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e3218b9f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fae24013a60 tx=0x7fae24012000 comp rx=0 tx=0).stop 2024-04-21T08:55:09.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.422+0000 7fae38073740 1 -- 172.21.15.7:0/326045491 shutdown_connections 2024-04-21T08:55:09.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.422+0000 7fae38073740 1 --2- 172.21.15.7:0/326045491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e3218b9f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.422+0000 7fae38073740 1 --2- 172.21.15.7:0/326045491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186ea0 0x561e3219b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.422+0000 7fae38073740 1 --2- 172.21.15.7:0/326045491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186580 0x561e3218f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.422+0000 7fae38073740 1 -- 172.21.15.7:0/326045491 >> 172.21.15.7:0/326045491 conn(0x561e32180cb0 msgr2=0x561e3217fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:09.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.422+0000 7fae38073740 1 -- 172.21.15.7:0/326045491 shutdown_connections 2024-04-21T08:55:09.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.422+0000 7fae38073740 1 -- 172.21.15.7:0/326045491 wait complete. 2024-04-21T08:55:09.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.423+0000 7fae38073740 1 -- start start 2024-04-21T08:55:09.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.423+0000 7fae38073740 1 Processor -- start 2024-04-21T08:55:09.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.423+0000 7fae38073740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186580 0x561e32193e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:09.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.424+0000 7fae38073740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e321943a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:09.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.424+0000 7fae38073740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e32215c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:09.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.424+0000 7fae2e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e321943a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:09.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.424+0000 7fae2e7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e321943a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42268/0 (socket says 172.21.15.7:42268) 2024-04-21T08:55:09.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.424+0000 7fae2e7fc640 1 -- 172.21.15.7:0/1548850218 learned_addr learned my addr 172.21.15.7:0/1548850218 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:09.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.424+0000 7fae38073740 1 -- 172.21.15.7:0/1548850218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fae301b4b40 con 0x561e32186ea0 2024-04-21T08:55:09.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.424+0000 7fae38073740 1 -- 172.21.15.7:0/1548850218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae301b5fb0 con 0x561e32186580 2024-04-21T08:55:09.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.424+0000 7fae38073740 1 -- 172.21.15.7:0/1548850218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fae301b5210 con 0x561e3218fa80 2024-04-21T08:55:09.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.425+0000 7fae38073740 10 client.0 _ll_register_callbacks cb 0x7fae20010f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:09.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.425+0000 7fae2e7fc640 1 -- 172.21.15.7:0/1548850218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 msgr2=0x561e32215c90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:09.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.425+0000 7fae2e7fc640 1 --2- 172.21.15.7:0/1548850218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e32215c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.425+0000 7fae2e7fc640 1 -- 172.21.15.7:0/1548850218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186580 msgr2=0x561e32193e60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:09.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.425+0000 7fae2e7fc640 1 --2- 172.21.15.7:0/1548850218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186580 0x561e32193e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.425+0000 7fae2e7fc640 1 -- 172.21.15.7:0/1548850218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e32216220 con 0x561e32186ea0 2024-04-21T08:55:09.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.426+0000 7fae2e7fc640 1 --2- 172.21.15.7:0/1548850218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e321943a0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fae2009a040 tx=0x7fae20009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdff89f6af7a167b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:09.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.426+0000 7fae127fc640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:55:09.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.428+0000 7fae127fc640 1 -- 172.21.15.7:0/1548850218 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20098030 con 0x561e32186ea0 2024-04-21T08:55:09.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.428+0000 7fae38073740 1 -- 172.21.15.7:0/1548850218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae24024070 con 0x561e32186ea0 2024-04-21T08:55:09.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.429+0000 7fae127fc640 1 -- 172.21.15.7:0/1548850218 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7fae200cd740 con 0x561e32186ea0 2024-04-21T08:55:09.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.432+0000 7fae127fc640 1 -- 172.21.15.7:0/1548850218 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fae200cd940 con 0x561e32186ea0 2024-04-21T08:55:09.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.432+0000 7fae38073740 10 client.17229 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:09.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.432+0000 7fae38073740 1 -- 172.21.15.7:0/1548850218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561e32217510 con 0x561e32186ea0 2024-04-21T08:55:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.440+0000 7fae127fc640 1 -- 172.21.15.7:0/1548850218 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae200a8070 con 0x561e32186ea0 2024-04-21T08:55:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.441+0000 7fae127fc640 1 -- 172.21.15.7:0/1548850218 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fae200a12d0 con 0x561e32186ea0 2024-04-21T08:55:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.441+0000 7fae38073740 10 client.17229 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.441+0000 7fae38073740 10 client.17229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.441+0000 7fae38073740 20 client.17229 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.441+0000 7fae38073740 1 --2- 172.21.15.7:0/1548850218 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561e32217b60 0x561e32219f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.441+0000 7fae38073740 4 client.17229 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:55:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.441+0000 7fae38073740 1 -- 172.21.15.7:0/1548850218 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561e320f2a70 con 0x561e32217b60 2024-04-21T08:55:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.441+0000 7fae2effd640 1 --2- 172.21.15.7:0/1548850218 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561e32217b60 0x561e32219f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.442+0000 7fae2effd640 1 --2- 172.21.15.7:0/1548850218 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561e32217b60 0x561e32219f80 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a6e52d4584874c1 server_cookie=b4de9b32b1c6bc76 in_seq=0 out_seq=0 2024-04-21T08:55:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.442+0000 7fae127fc640 10 client.17229 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.442+0000 7fae127fc640 1 -- 172.21.15.7:0/1548850218 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x561e320f2a70 con 0x561e32217b60 2024-04-21T08:55:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.442+0000 7fae127fc640 10 client.17229 handle_command_reply: tid=0 2024-04-21T08:55:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.442+0000 7fae38073740 1 client.17229 shutdown 2024-04-21T08:55:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.457+0000 7fae38073740 1 -- 172.21.15.7:0/1548850218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 msgr2=0x561e321943a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.457+0000 7fae38073740 1 --2- 172.21.15.7:0/1548850218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e321943a0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fae2009a040 tx=0x7fae20009ea0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.457+0000 7fae38073740 1 -- 172.21.15.7:0/1548850218 shutdown_connections 2024-04-21T08:55:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.457+0000 7fae38073740 1 --2- 172.21.15.7:0/1548850218 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x561e32217b60 0x561e32219f80 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.457+0000 7fae38073740 1 --2- 172.21.15.7:0/1548850218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3218fa80 0x561e32215c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.457+0000 7fae38073740 1 --2- 172.21.15.7:0/1548850218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561e32186ea0 0x561e321943a0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.457+0000 7fae38073740 1 --2- 172.21.15.7:0/1548850218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561e32186580 0x561e32193e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.457+0000 7fae38073740 1 -- 172.21.15.7:0/1548850218 >> 172.21.15.7:0/1548850218 conn(0x561e32180cb0 msgr2=0x561e3218c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.457+0000 7fae127fc640 0 client.17229 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:09.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.469+0000 7fae38073740 1 -- 172.21.15.7:0/1548850218 shutdown_connections 2024-04-21T08:55:09.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.469+0000 7fae38073740 1 -- 172.21.15.7:0/1548850218 wait complete. 2024-04-21T08:55:09.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.469+0000 7fae38073740 20 client.17229 trim_cache size 0 max 16384 2024-04-21T08:55:09.469 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:55:09.469 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (14740 inodes in the stack)", 2024-04-21T08:55:09.469 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:55:09.469 INFO:teuthology.orchestra.run.smithi007.stdout: "08447483-b5f8-4de7-a923-832d7f72e5ac": { 2024-04-21T08:55:09.469 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:55:09.469 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "08447483-b5f8-4de7-a923-832d7f72e5ac", 2024-04-21T08:55:09.469 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:55:09.470 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:55:09.470 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:55:09.470 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:55:09.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.483+0000 7fae35481640 1 -- 172.21.15.7:0/4221528017 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fae180a3f20 msgr2=0x7fae180a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.483+0000 7fae35481640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fae180a3f20 0x7fae180a63e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fae24009570 tx=0x7fae24006bd0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.483+0000 7fae35481640 1 -- 172.21.15.7:0/4221528017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae301106a0 msgr2=0x7fae301a7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.483+0000 7fae35481640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae301106a0 0x7fae301a7c20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae2000d740 tx=0x7fae2000dc00 comp rx=0 tx=0).stop 2024-04-21T08:55:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.484+0000 7fae35481640 1 -- 172.21.15.7:0/4221528017 shutdown_connections 2024-04-21T08:55:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.484+0000 7fae35481640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fae180a3f20 0x7fae180a63e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.484+0000 7fae35481640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae30110b10 0x7fae301abf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.484+0000 7fae35481640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae301106a0 0x7fae301a7c20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.484+0000 7fae35481640 1 --2- 172.21.15.7:0/4221528017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae3010e1b0 0x7fae301a76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.484+0000 7fae35481640 1 -- 172.21.15.7:0/4221528017 >> 172.21.15.7:0/4221528017 conn(0x7fae3006e360 msgr2=0x7fae300729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:09.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.491+0000 7fae35481640 1 -- 172.21.15.7:0/4221528017 shutdown_connections 2024-04-21T08:55:09.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:09.491+0000 7fae35481640 1 -- 172.21.15.7:0/4221528017 wait complete. 2024-04-21T08:55:09.513 INFO:tasks.cephfs.filesystem:scrub status for tag:08447483-b5f8-4de7-a923-832d7f72e5ac - {'path': '/', 'tag': '08447483-b5f8-4de7-a923-832d7f72e5ac', 'options': 'recursive,force'} 2024-04-21T08:55:09.530 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/dmapool.o 2024-04-21T08:55:09.612 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ext4/xattr_security.o 2024-04-21T08:55:10.250 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/super.o 2024-04-21T08:55:10.537 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/host.o 2024-04-21T08:55:10.688 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/hugetlb.o 2024-04-21T08:55:10.877 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/ext4/built-in.a 2024-04-21T08:55:11.081 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/rpm-traces.o 2024-04-21T08:55:11.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:10 smithi077 ceph-mon[31615]: pgmap v330: 129 pgs: 18 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 398 KiB/s wr, 498 op/s 2024-04-21T08:55:11.111 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/unicode/built-in.a 2024-04-21T08:55:11.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:10 smithi007 ceph-mon[28477]: pgmap v330: 129 pgs: 18 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 398 KiB/s wr, 498 op/s 2024-04-21T08:55:11.228 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/autofs/init.o 2024-04-21T08:55:11.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:10 smithi016 ceph-mon[31259]: pgmap v330: 129 pgs: 18 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 398 KiB/s wr, 498 op/s 2024-04-21T08:55:11.980 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/autofs/inode.o 2024-04-21T08:55:12.036 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/exit.o 2024-04-21T08:55:12.189 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/softirq.o 2024-04-21T08:55:12.328 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/io.o 2024-04-21T08:55:12.576 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/svc.o 2024-04-21T08:55:12.589 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_dynevent.o 2024-04-21T08:55:13.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:12 smithi077 ceph-mon[31615]: pgmap v331: 129 pgs: 18 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 62 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 301 KiB/s wr, 545 op/s 2024-04-21T08:55:13.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:12 smithi007 ceph-mon[28477]: pgmap v331: 129 pgs: 18 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 62 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 301 KiB/s wr, 545 op/s 2024-04-21T08:55:13.173 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/autofs/root.o 2024-04-21T08:55:13.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:12 smithi016 ceph-mon[31259]: pgmap v331: 129 pgs: 18 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 62 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 301 KiB/s wr, 545 op/s 2024-04-21T08:55:13.839 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_probe.o 2024-04-21T08:55:13.870 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/direct.o 2024-04-21T08:55:14.046 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/netlink.o 2024-04-21T08:55:14.212 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/nlmsgtab.o 2024-04-21T08:55:14.441 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/autofs/symlink.o 2024-04-21T08:55:14.493 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/integrity/iint.o 2024-04-21T08:55:14.636 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/svclock.o 2024-04-21T08:55:15.151 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/autofs/waitq.o 2024-04-21T08:55:15.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:14 smithi007 ceph-mon[28477]: pgmap v332: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 275 KiB/s wr, 565 op/s 2024-04-21T08:55:15.162 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/svcshare.o 2024-04-21T08:55:15.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:14 smithi016 ceph-mon[31259]: pgmap v332: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 275 KiB/s wr, 565 op/s 2024-04-21T08:55:15.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:14 smithi077 ceph-mon[31615]: pgmap v332: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 275 KiB/s wr, 565 op/s 2024-04-21T08:55:15.486 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/integrity/integrity_audit.o 2024-04-21T08:55:16.035 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/netif.o 2024-04-21T08:55:16.093 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/hugetlb_vmemmap.o 2024-04-21T08:55:16.124 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/autofs/expire.o 2024-04-21T08:55:16.175 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/trace_uprobe.o 2024-04-21T08:55:16.495 INFO:tasks.workunit.client.0.smithi007.stdout: AR security/integrity/built-in.a 2024-04-21T08:55:16.656 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/trace/rethook.o 2024-04-21T08:55:16.839 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/svcproc.o 2024-04-21T08:55:17.044 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/autofs/dev-ioctl.o 2024-04-21T08:55:17.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:16 smithi007 ceph-mon[28477]: pgmap v333: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 247 KiB/s wr, 563 op/s 2024-04-21T08:55:17.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:16 smithi016 ceph-mon[31259]: pgmap v333: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 247 KiB/s wr, 563 op/s 2024-04-21T08:55:17.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:16 smithi077 ceph-mon[31615]: pgmap v333: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 247 KiB/s wr, 563 op/s 2024-04-21T08:55:17.368 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mempolicy.o 2024-04-21T08:55:17.395 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/netnode.o 2024-04-21T08:55:17.599 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/pagelist.o 2024-04-21T08:55:17.852 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/autofs/built-in.a 2024-04-21T08:55:18.367 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/commoncap.o 2024-04-21T08:55:18.394 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/svcsubs.o 2024-04-21T08:55:18.426 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/trace/built-in.a 2024-04-21T08:55:18.535 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/read.o 2024-04-21T08:55:18.768 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/resource.o 2024-04-21T08:55:18.787 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/9p/vfs_super.o 2024-04-21T08:55:18.873 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/netport.o 2024-04-21T08:55:19.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:18 smithi007 ceph-mon[28477]: pgmap v334: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 440 KiB/s wr, 559 op/s 2024-04-21T08:55:19.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:18 smithi016 ceph-mon[31259]: pgmap v334: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 440 KiB/s wr, 559 op/s 2024-04-21T08:55:19.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:18 smithi077 ceph-mon[31615]: pgmap v334: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 440 KiB/s wr, 559 op/s 2024-04-21T08:55:19.728 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/sparse.o 2024-04-21T08:55:19.925 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/9p/vfs_inode.o 2024-04-21T08:55:19.997 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/9p/vfs_inode_dotl.o 2024-04-21T08:55:20.071 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/mon.o 2024-04-21T08:55:20.495 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/status.o 2024-04-21T08:55:20.516 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/9p/vfs_addr.o 2024-04-21T08:55:20.613 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/sysctl.o 2024-04-21T08:55:20.857 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/symlink.o 2024-04-21T08:55:21.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:20 smithi007 ceph-mon[28477]: pgmap v335: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 891 KiB/s wr, 553 op/s 2024-04-21T08:55:21.231 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/sparse-vmemmap.o 2024-04-21T08:55:21.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:20 smithi016 ceph-mon[31259]: pgmap v335: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 891 KiB/s wr, 553 op/s 2024-04-21T08:55:21.338 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/ss/ebitmap.o 2024-04-21T08:55:21.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:20 smithi077 ceph-mon[31615]: pgmap v335: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 891 KiB/s wr, 553 op/s 2024-04-21T08:55:21.443 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/9p/vfs_file.o 2024-04-21T08:55:21.885 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/trace.o 2024-04-21T08:55:21.937 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/hostfs/built-in.a 2024-04-21T08:55:22.011 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/xdr.o 2024-04-21T08:55:22.308 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/debugfs/inode.o 2024-04-21T08:55:22.678 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/9p/vfs_dir.o 2024-04-21T08:55:22.698 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/mmu_notifier.o 2024-04-21T08:55:22.768 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/unlink.o 2024-04-21T08:55:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.791+0000 7f26ca806640 1 -- 172.21.15.7:0/1066503119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c410dff0 msgr2=0x7f26c4115b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.791+0000 7f26ca806640 1 --2- 172.21.15.7:0/1066503119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c410dff0 0x7f26c4115b50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009900 tx=0x7f26b802bc60 comp rx=0 tx=0).stop 2024-04-21T08:55:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.791+0000 7f26ca806640 1 -- 172.21.15.7:0/1066503119 shutdown_connections 2024-04-21T08:55:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.791+0000 7f26ca806640 1 --2- 172.21.15.7:0/1066503119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c410dff0 0x7f26c4115b50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.791+0000 7f26ca806640 1 --2- 172.21.15.7:0/1066503119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c410a400 0x7f26c410d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.791 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.791+0000 7f26ca806640 1 --2- 172.21.15.7:0/1066503119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26c4109a50 0x7f26c4109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.791+0000 7f26ca806640 1 -- 172.21.15.7:0/1066503119 >> 172.21.15.7:0/1066503119 conn(0x7f26c406cea0 msgr2=0x7f26c406d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.794+0000 7f26ca806640 1 -- 172.21.15.7:0/1066503119 shutdown_connections 2024-04-21T08:55:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.799+0000 7f26ca806640 1 -- 172.21.15.7:0/1066503119 wait complete. 2024-04-21T08:55:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.799+0000 7f26ca806640 1 Processor -- start 2024-04-21T08:55:22.799 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.800+0000 7f26ca806640 1 -- start start 2024-04-21T08:55:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.806+0000 7f26ca806640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26c4109a50 0x7f26c4117270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.806+0000 7f26ca806640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c410dff0 0x7f26c41177b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.806+0000 7f26ca806640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c4118f10 0x7f26c4117cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.806+0000 7f26ca806640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f26c411a910 con 0x7f26c4109a50 2024-04-21T08:55:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.806+0000 7f26ca806640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26c411a790 con 0x7f26c410dff0 2024-04-21T08:55:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.806+0000 7f26ca806640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f26c411aa90 con 0x7f26c4118f10 2024-04-21T08:55:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.806+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c410dff0 0x7f26c41177b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.806+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c410dff0 0x7f26c41177b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:36360/0 (socket says 172.21.15.7:36360) 2024-04-21T08:55:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.806+0000 7f26c37fe640 1 -- 172.21.15.7:0/571837966 learned_addr learned my addr 172.21.15.7:0/571837966 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.807+0000 7f26c37fe640 1 -- 172.21.15.7:0/571837966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c4118f10 msgr2=0x7f26c4117cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:55:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.807+0000 7f26c37fe640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c4118f10 0x7f26c4117cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.807+0000 7f26c37fe640 1 -- 172.21.15.7:0/571837966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26c4109a50 msgr2=0x7f26c4117270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.807+0000 7f26c37fe640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26c4109a50 0x7f26c4117270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.807+0000 7f26c37fe640 1 -- 172.21.15.7:0/571837966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c41b3b60 con 0x7f26c410dff0 2024-04-21T08:55:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.807+0000 7f26c37fe640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c410dff0 0x7f26c41177b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00b9e0 tx=0x7f26bc00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.817+0000 7f26c17fa640 1 -- 172.21.15.7:0/571837966 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26bc019070 con 0x7f26c410dff0 2024-04-21T08:55:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.817+0000 7f26c17fa640 1 -- 172.21.15.7:0/571837966 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26bc00cba0 con 0x7f26c410dff0 2024-04-21T08:55:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.817+0000 7f26c17fa640 1 -- 172.21.15.7:0/571837966 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26bc0125e0 con 0x7f26c410dff0 2024-04-21T08:55:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.821+0000 7f26ca806640 1 -- 172.21.15.7:0/571837966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26c41b3e50 con 0x7f26c410dff0 2024-04-21T08:55:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.821+0000 7f26ca806640 1 -- 172.21.15.7:0/571837966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26c41b4260 con 0x7f26c410dff0 2024-04-21T08:55:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.828+0000 7f26cd3f4740 1 Processor -- start 2024-04-21T08:55:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.830+0000 7f26cd3f4740 1 -- start start 2024-04-21T08:55:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.830+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cdd8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.830+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cdd91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.830+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdf32e0 0x55ab3cdf56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.830+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ab3cd3d8d0 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.830+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ab3ccdae80 con 0x55ab3cdf32e0 2024-04-21T08:55:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.830+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ab3cd39890 con 0x55ab3cdd8d50 2024-04-21T08:55:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.830+0000 7f26c17fa640 1 -- 172.21.15.7:0/571837966 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f26bc014440 con 0x7f26c410dff0 2024-04-21T08:55:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.831+0000 7f26c3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cdd91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.831+0000 7f26c3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cdd91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:40212/0 (socket says 172.21.15.7:40212) 2024-04-21T08:55:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.831+0000 7f26c3fff640 1 -- 172.21.15.7:0/608472021 learned_addr learned my addr 172.21.15.7:0/608472021 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.831+0000 7f26c37fe640 1 --2- 172.21.15.7:0/608472021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdf32e0 0x55ab3cdf56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.832+0000 7f26c17fa640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f26ac0a3f30 0x7f26ac0a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.832+0000 7f26c3fff640 1 -- 172.21.15.7:0/608472021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdf32e0 msgr2=0x55ab3cdf56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.832+0000 7f26c3fff640 1 --2- 172.21.15.7:0/608472021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdf32e0 0x55ab3cdf56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.832+0000 7f26c3fff640 1 -- 172.21.15.7:0/608472021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdd9ac0 msgr2=0x55ab3cdd8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.832+0000 7f26c3fff640 1 --2- 172.21.15.7:0/608472021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cdd8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.832+0000 7f26c3fff640 1 -- 172.21.15.7:0/608472021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab3ccfc320 con 0x55ab3cdd8d50 2024-04-21T08:55:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.832+0000 7f26c3fff640 1 --2- 172.21.15.7:0/608472021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cdd91d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f26b402b8c0 tx=0x7f26b402bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb4023049278cdc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.832+0000 7f26c17fa640 1 -- 172.21.15.7:0/571837966 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7f26bc0d0030 con 0x7f26c410dff0 2024-04-21T08:55:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.837+0000 7f26aa7fc640 1 -- 172.21.15.7:0/608472021 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4037020 con 0x55ab3cdd8d50 2024-04-21T08:55:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.839+0000 7f26aa7fc640 1 -- 172.21.15.7:0/608472021 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26b402dd90 con 0x55ab3cdd8d50 2024-04-21T08:55:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.840+0000 7f26cd3f4740 1 -- 172.21.15.7:0/608472021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 msgr2=0x55ab3cdd91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.840+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/608472021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cdd91d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f26b402b8c0 tx=0x7f26b402bd80 comp rx=0 tx=0).stop 2024-04-21T08:55:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.841+0000 7f26c8da3640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f26ac0a3f30 0x7f26ac0a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.847+0000 7f26c8da3640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f26ac0a3f30 0x7f26ac0a63f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f26b80096c0 tx=0x7f26b80056d0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.849+0000 7f26cd3f4740 1 -- 172.21.15.7:0/608472021 shutdown_connections 2024-04-21T08:55:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.849+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/608472021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdf32e0 0x55ab3cdf56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.849+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/608472021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cdd91d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.849+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/608472021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cdd8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.849+0000 7f26cd3f4740 1 -- 172.21.15.7:0/608472021 >> 172.21.15.7:0/608472021 conn(0x55ab3cd9fc90 msgr2=0x55ab3cdd3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.849+0000 7f26cd3f4740 1 -- 172.21.15.7:0/608472021 shutdown_connections 2024-04-21T08:55:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.849+0000 7f26cd3f4740 1 -- 172.21.15.7:0/608472021 wait complete. 2024-04-21T08:55:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.849+0000 7f26cd3f4740 1 -- start start 2024-04-21T08:55:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.849+0000 7f26cd3f4740 1 Processor -- start 2024-04-21T08:55:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.850+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cde4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.850+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.850+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cddf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.850+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ab3cdda550 con 0x55ab3cdf32e0 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.850+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ab3ccc5eb0 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.850+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ab3cdda6d0 con 0x55ab3cdd8d50 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.850+0000 7f26cd3f4740 10 client.0 _ll_register_callbacks cb 0x55ab3ccf2b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.850+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.850+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:36374/0 (socket says 172.21.15.7:36374) 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.850+0000 7f26c37fe640 1 -- 172.21.15.7:0/3594716107 learned_addr learned my addr 172.21.15.7:0/3594716107 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.850+0000 7f26c3fff640 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cde4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.851+0000 7f26c8da3640 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cddf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.851+0000 7f26c37fe640 1 -- 172.21.15.7:0/3594716107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 msgr2=0x55ab3cde4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.851+0000 7f26c37fe640 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cde4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.851+0000 7f26c37fe640 1 -- 172.21.15.7:0/3594716107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 msgr2=0x55ab3cddf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.851+0000 7f26c37fe640 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cddf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.851+0000 7f26c37fe640 1 -- 172.21.15.7:0/3594716107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab3ca50210 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.851+0000 7f26c8da3640 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cddf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:55:22.853 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.851+0000 7f26c37fe640 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde4780 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f26bc098020 tx=0x7f26bc09b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f16e4e4d3c11364 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.863+0000 7f26a7ff7640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:55:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.863+0000 7f26a7ff7640 1 -- 172.21.15.7:0/3594716107 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26bc09ff70 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.863+0000 7f26a7ff7640 1 -- 172.21.15.7:0/3594716107 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7f26bc0cd4e0 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.866 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.867+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3594716107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26b402dd90 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.867+0000 7f26a7ff7640 1 -- 172.21.15.7:0/3594716107 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f26bc0a4830 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.867+0000 7f26cd3f4740 10 client.26867 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:22.867 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.867+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3594716107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ab3cb1c620 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.870 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.871+0000 7f26a7ff7640 1 -- 172.21.15.7:0/3594716107 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26b402dd90 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.876+0000 7f26a7ff7640 1 -- 172.21.15.7:0/3594716107 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f26bc014730 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.876+0000 7f26cd3f4740 10 client.26867 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.876+0000 7f26cd3f4740 10 client.26867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.876+0000 7f26cd3f4740 20 client.26867 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.876+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ab3cdefc90 0x55ab3cdf20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.876+0000 7f26cd3f4740 4 client.26867 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:22.876 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.876+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3594716107 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab3cd51880 con 0x55ab3cdefc90 2024-04-21T08:55:22.880 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.881+0000 7f26c3fff640 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ab3cdefc90 0x55ab3cdf20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.887+0000 7f26a7ff7640 10 client.26867 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.887+0000 7f26c3fff640 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ab3cdefc90 0x55ab3cdf20d0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d5861137539cc5c server_cookie=abe2a5e42a6a5d68 in_seq=0 out_seq=0 2024-04-21T08:55:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.889+0000 7f26a7ff7640 1 -- 172.21.15.7:0/3594716107 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab3cd51880 con 0x55ab3cdefc90 2024-04-21T08:55:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.889+0000 7f26a7ff7640 10 client.26867 handle_command_reply: tid=0 2024-04-21T08:55:22.889 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.889+0000 7f26cd3f4740 1 client.26867 shutdown 2024-04-21T08:55:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.891+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3594716107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 msgr2=0x55ab3cde4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.890 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.891+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde4780 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f26bc098020 tx=0x7f26bc09b7c0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.898+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3594716107 shutdown_connections 2024-04-21T08:55:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.898+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ab3cdefc90 0x55ab3cdf20d0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.898+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cddf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.898+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde4780 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.898+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/3594716107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cde4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.898+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3594716107 >> 172.21.15.7:0/3594716107 conn(0x55ab3cd9fc90 msgr2=0x55ab3cdd5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.898+0000 7f26a7ff7640 0 client.26867 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:22.902 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.903+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3594716107 shutdown_connections 2024-04-21T08:55:22.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.903+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3594716107 wait complete. 2024-04-21T08:55:22.903 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.903+0000 7f26cd3f4740 20 client.26867 trim_cache size 0 max 16384 2024-04-21T08:55:22.948 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/ss/hashtab.o 2024-04-21T08:55:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.954+0000 7f26cd3f4740 1 Processor -- start 2024-04-21T08:55:22.962 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.963+0000 7f26cd3f4740 1 -- start start 2024-04-21T08:55:22.963 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.963+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cdd3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.963+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cdfae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cdfb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f26c41acef0 con 0x55ab3cdf32e0 2024-04-21T08:55:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26c41b6420 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cdfae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f26c41ad4b0 con 0x55ab3cdd8d50 2024-04-21T08:55:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cdfae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:36384/0 (socket says 172.21.15.7:36384) 2024-04-21T08:55:22.964 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26c37fe640 1 -- 172.21.15.7:0/3545358126 learned_addr learned my addr 172.21.15.7:0/3545358126 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26c37fe640 1 -- 172.21.15.7:0/3545358126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 msgr2=0x55ab3cdd3570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26c37fe640 1 --2- 172.21.15.7:0/3545358126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cdd3570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26c37fe640 1 -- 172.21.15.7:0/3545358126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 msgr2=0x55ab3cdfb340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26c37fe640 1 --2- 172.21.15.7:0/3545358126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cdfb340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.964+0000 7f26c37fe640 1 -- 172.21.15.7:0/3545358126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab3cdd3f80 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.965 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.965+0000 7f26c37fe640 1 --2- 172.21.15.7:0/3545358126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cdfae00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00bfd0 tx=0x7f26bc09a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=508fecdf6fd23057 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.977+0000 7f26a9ffb640 1 -- 172.21.15.7:0/3545358126 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26bc0b4070 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.976 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.977+0000 7f26a9ffb640 1 -- 172.21.15.7:0/3545358126 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26bc09ff70 con 0x55ab3cdd9ac0 2024-04-21T08:55:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.978+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3545358126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 msgr2=0x55ab3cdfae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.977 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.978+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/3545358126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cdfae00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00bfd0 tx=0x7f26bc09a0d0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.999+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3545358126 shutdown_connections 2024-04-21T08:55:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.999+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/3545358126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cdfb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.999+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/3545358126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cdfae00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.999+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/3545358126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cdd3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.999 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:22.999+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3545358126 >> 172.21.15.7:0/3545358126 conn(0x55ab3cd9fc90 msgr2=0x55ab3cdda970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.001+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3545358126 shutdown_connections 2024-04-21T08:55:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.004+0000 7f26cd3f4740 1 -- 172.21.15.7:0/3545358126 wait complete. 2024-04-21T08:55:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.005+0000 7f26cd3f4740 1 -- start start 2024-04-21T08:55:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.005+0000 7f26cd3f4740 1 Processor -- start 2024-04-21T08:55:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.007+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cde60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.007+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.007+0000 7f26cd3f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cde6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.007+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f26c41b48e0 con 0x55ab3cdf32e0 2024-04-21T08:55:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.007+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26c41b5d50 con 0x55ab3cdd9ac0 2024-04-21T08:55:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.007+0000 7f26cd3f4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f26c41b4fb0 con 0x55ab3cdd8d50 2024-04-21T08:55:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.007+0000 7f26cd3f4740 10 client.0 _ll_register_callbacks cb 0x7f26bc09b990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.011+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.011+0000 7f26c3fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cde60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.011+0000 7f26c37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:36386/0 (socket says 172.21.15.7:36386) 2024-04-21T08:55:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.011+0000 7f26c37fe640 1 -- 172.21.15.7:0/1726087128 learned_addr learned my addr 172.21.15.7:0/1726087128 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.017+0000 7f26c8da3640 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cde6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.021+0000 7f26c37fe640 1 -- 172.21.15.7:0/1726087128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 msgr2=0x55ab3cde60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.021+0000 7f26c37fe640 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cde60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.021+0000 7f26c37fe640 1 -- 172.21.15.7:0/1726087128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 msgr2=0x55ab3cde6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.021+0000 7f26c37fe640 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cde6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.021+0000 7f26c37fe640 1 -- 172.21.15.7:0/1726087128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab3ce57660 con 0x55ab3cdd9ac0 2024-04-21T08:55:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.026+0000 7f26c37fe640 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde6600 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f26bc098590 tx=0x7f26bc097ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5940cf2776be7a11 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.026+0000 7f26a7ff7640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:55:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.031+0000 7f26a7ff7640 1 -- 172.21.15.7:0/1726087128 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26bc0b4030 con 0x55ab3cdd9ac0 2024-04-21T08:55:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.031+0000 7f26a7ff7640 1 -- 172.21.15.7:0/1726087128 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7f26bc097910 con 0x55ab3cdd9ac0 2024-04-21T08:55:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.031+0000 7f26cd3f4740 1 -- 172.21.15.7:0/1726087128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26bc09ff70 con 0x55ab3cdd9ac0 2024-04-21T08:55:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.039+0000 7f26a7ff7640 1 -- 172.21.15.7:0/1726087128 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f26bc0d3400 con 0x55ab3cdd9ac0 2024-04-21T08:55:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.039+0000 7f26cd3f4740 10 client.26873 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.039+0000 7f26cd3f4740 1 -- 172.21.15.7:0/1726087128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ab3ce58850 con 0x55ab3cdd9ac0 2024-04-21T08:55:23.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.042+0000 7f26a7ff7640 1 -- 172.21.15.7:0/1726087128 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26bc09ff70 con 0x55ab3cdd9ac0 2024-04-21T08:55:23.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.042+0000 7f26a7ff7640 1 -- 172.21.15.7:0/1726087128 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f26bc0d3840 con 0x55ab3cdd9ac0 2024-04-21T08:55:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.042+0000 7f26cd3f4740 10 client.26873 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.042+0000 7f26cd3f4740 10 client.26873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.042+0000 7f26cd3f4740 20 client.26873 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.042+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ab3ce58d20 0x55ab3ce5b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.042+0000 7f26cd3f4740 4 client.26873 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.043+0000 7f26cd3f4740 1 -- 172.21.15.7:0/1726087128 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab3ccdd3d0 con 0x55ab3ce58d20 2024-04-21T08:55:23.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.049+0000 7f26c3fff640 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ab3ce58d20 0x55ab3ce5b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.059+0000 7f26c3fff640 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ab3ce58d20 0x55ab3ce5b110 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd15b0f201d46f38 server_cookie=d54f2435a86b4341 in_seq=0 out_seq=0 2024-04-21T08:55:23.059 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:23.060+0000 7f26a7ff7640 10 client.26873 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:23.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:22 smithi016 ceph-mon[31259]: pgmap v336: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 940 KiB/s wr, 429 op/s 2024-04-21T08:55:23.352 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/capability.o 2024-04-21T08:55:23.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:22 smithi077 ceph-mon[31615]: pgmap v336: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 940 KiB/s wr, 429 op/s 2024-04-21T08:55:23.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:22 smithi007 ceph-mon[28477]: pgmap v336: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 940 KiB/s wr, 429 op/s 2024-04-21T08:55:23.803 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/debugfs/file.o 2024-04-21T08:55:23.807 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/9p/vfs_dentry.o 2024-04-21T08:55:23.850 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/write.o 2024-04-21T08:55:23.857 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/ss/symtab.o 2024-04-21T08:55:23.947 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/clnt4xdr.o 2024-04-21T08:55:24.268 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/slub.o 2024-04-21T08:55:24.383 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/ss/sidtab.o 2024-04-21T08:55:24.617 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/9p/v9fs.o 2024-04-21T08:55:24.633 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/ptrace.o 2024-04-21T08:55:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.645+0000 7f26a7ff7640 1 -- 172.21.15.7:0/1726087128 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ab3ccdd3d0 con 0x55ab3ce58d20 2024-04-21T08:55:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.645+0000 7f26a7ff7640 10 client.26873 handle_command_reply: tid=0 2024-04-21T08:55:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.645+0000 7f26cd3f4740 1 client.26873 shutdown 2024-04-21T08:55:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.652+0000 7f26cd3f4740 1 -- 172.21.15.7:0/1726087128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 msgr2=0x55ab3cde6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.652+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde6600 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f26bc098590 tx=0x7f26bc097ef0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.661+0000 7f26cd3f4740 1 -- 172.21.15.7:0/1726087128 shutdown_connections 2024-04-21T08:55:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.661+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ab3ce58d20 0x55ab3ce5b110 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.661+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ab3cdf32e0 0x55ab3cde6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.661+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ab3cdd9ac0 0x55ab3cde6600 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.661+0000 7f26cd3f4740 1 --2- 172.21.15.7:0/1726087128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab3cdd8d50 0x55ab3cde60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.661+0000 7f26cd3f4740 1 -- 172.21.15.7:0/1726087128 >> 172.21.15.7:0/1726087128 conn(0x55ab3cd9fc90 msgr2=0x55ab3cddde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.661+0000 7f26a7ff7640 0 client.26873 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.667+0000 7f26cd3f4740 1 -- 172.21.15.7:0/1726087128 shutdown_connections 2024-04-21T08:55:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.674+0000 7f26cd3f4740 1 -- 172.21.15.7:0/1726087128 wait complete. 2024-04-21T08:55:24.677 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.678+0000 7f26cd3f4740 20 client.26873 trim_cache size 0 max 16384 2024-04-21T08:55:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:55:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:55:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:55:24.678 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:55:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.703+0000 7f26ca806640 1 -- 172.21.15.7:0/571837966 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f26ac0a3f30 msgr2=0x7f26ac0a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.703+0000 7f26ca806640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f26ac0a3f30 0x7f26ac0a63f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f26b80096c0 tx=0x7f26b80056d0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.703+0000 7f26ca806640 1 -- 172.21.15.7:0/571837966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c410dff0 msgr2=0x7f26c41177b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.703+0000 7f26ca806640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c410dff0 0x7f26c41177b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00b9e0 tx=0x7f26bc00bea0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.704+0000 7f26ca806640 1 -- 172.21.15.7:0/571837966 shutdown_connections 2024-04-21T08:55:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.704+0000 7f26ca806640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f26ac0a3f30 0x7f26ac0a63f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.704+0000 7f26ca806640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c4118f10 0x7f26c4117cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.704+0000 7f26ca806640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f26c410dff0 0x7f26c41177b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.704+0000 7f26ca806640 1 --2- 172.21.15.7:0/571837966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26c4109a50 0x7f26c4117270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.704+0000 7f26ca806640 1 -- 172.21.15.7:0/571837966 >> 172.21.15.7:0/571837966 conn(0x7f26c406cea0 msgr2=0x7f26c410cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.704+0000 7f26ca806640 1 -- 172.21.15.7:0/571837966 shutdown_connections 2024-04-21T08:55:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:24.704+0000 7f26ca806640 1 -- 172.21.15.7:0/571837966 wait complete. 2024-04-21T08:55:25.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:24 smithi016 ceph-mon[31259]: pgmap v337: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 1.1 MiB/s wr, 448 op/s 2024-04-21T08:55:25.306 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/xdr4.o 2024-04-21T08:55:25.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:24 smithi077 ceph-mon[31615]: pgmap v337: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 1.1 MiB/s wr, 448 op/s 2024-04-21T08:55:25.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:24 smithi007 ceph-mon[28477]: pgmap v337: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 1.1 MiB/s wr, 448 op/s 2024-04-21T08:55:25.618 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/9p/fid.o 2024-04-21T08:55:25.656 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/debugfs/built-in.a 2024-04-21T08:55:25.760 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/ss/avtab.o 2024-04-21T08:55:25.767 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/user.o 2024-04-21T08:55:26.366 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/ss/policydb.o 2024-04-21T08:55:26.536 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/signal.o 2024-04-21T08:55:26.550 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/9p/xattr.o 2024-04-21T08:55:26.594 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/tracefs/inode.o 2024-04-21T08:55:27.240 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/namespace.o 2024-04-21T08:55:27.251 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/svc4proc.o 2024-04-21T08:55:27.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:26 smithi016 ceph-mon[31259]: pgmap v338: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1018 KiB/s wr, 376 op/s 2024-04-21T08:55:27.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:26 smithi007 ceph-mon[28477]: pgmap v338: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1018 KiB/s wr, 376 op/s 2024-04-21T08:55:27.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:26 smithi077 ceph-mon[31615]: pgmap v338: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 1018 KiB/s wr, 376 op/s 2024-04-21T08:55:27.502 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/ss/services.o 2024-04-21T08:55:27.806 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/tracefs/built-in.a 2024-04-21T08:55:27.892 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/9p/built-in.a 2024-04-21T08:55:27.957 INFO:tasks.workunit.client.0.smithi007.stdout: CC [M] fs/efivarfs/inode.o 2024-04-21T08:55:28.026 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/lockd/procfs.o 2024-04-21T08:55:28.357 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/migrate.o 2024-04-21T08:55:29.044 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/memory-tiers.o 2024-04-21T08:55:29.050 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/mount_clnt.o 2024-04-21T08:55:29.085 INFO:tasks.workunit.client.0.smithi007.stdout: CC [M] fs/efivarfs/file.o 2024-04-21T08:55:29.198 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/lockd/built-in.a 2024-04-21T08:55:29.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:28 smithi016 ceph-mon[31259]: pgmap v339: 129 pgs: 19 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 53 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 1.0 MiB/s wr, 393 op/s 2024-04-21T08:55:29.314 INFO:tasks.workunit.client.0.smithi007.stdout: CC [M] fs/efivarfs/super.o 2024-04-21T08:55:29.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:28 smithi077 ceph-mon[31615]: pgmap v339: 129 pgs: 19 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 53 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 1.0 MiB/s wr, 393 op/s 2024-04-21T08:55:29.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:28 smithi007 ceph-mon[28477]: pgmap v339: 129 pgs: 19 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 53 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 1.0 MiB/s wr, 393 op/s 2024-04-21T08:55:29.492 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/ss/conditional.o 2024-04-21T08:55:30.227 INFO:tasks.workunit.client.0.smithi007.stdout: CC [M] fs/efivarfs/vars.o 2024-04-21T08:55:30.326 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/page_counter.o 2024-04-21T08:55:30.509 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/ss/mls.o 2024-04-21T08:55:30.520 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfstrace.o 2024-04-21T08:55:30.633 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/sys.o 2024-04-21T08:55:30.929 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/export.o 2024-04-21T08:55:31.023 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/hugetlb_cgroup.o 2024-04-21T08:55:31.042 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/sysfs.o 2024-04-21T08:55:31.291 INFO:tasks.workunit.client.0.smithi007.stdout: LD [M] fs/efivarfs/efivarfs.o 2024-04-21T08:55:31.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:30 smithi016 ceph-mon[31259]: pgmap v340: 129 pgs: 17 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 60 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 913 KiB/s wr, 414 op/s 2024-04-21T08:55:31.322 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/open.o 2024-04-21T08:55:31.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:30 smithi077 ceph-mon[31615]: pgmap v340: 129 pgs: 17 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 60 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 913 KiB/s wr, 414 op/s 2024-04-21T08:55:31.365 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/read_write.o 2024-04-21T08:55:31.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:30 smithi007 ceph-mon[28477]: pgmap v340: 129 pgs: 17 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 60 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 913 KiB/s wr, 414 op/s 2024-04-21T08:55:32.418 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/ss/context.o 2024-04-21T08:55:32.632 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/early_ioremap.o 2024-04-21T08:55:32.807 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/fs_context.o 2024-04-21T08:55:32.913 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfsroot.o 2024-04-21T08:55:33.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:33 smithi016 ceph-mon[31259]: pgmap v341: 129 pgs: 17 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 60 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 411 KiB/s wr, 381 op/s 2024-04-21T08:55:33.329 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/file_table.o 2024-04-21T08:55:33.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:33 smithi077 ceph-mon[31615]: pgmap v341: 129 pgs: 17 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 60 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 411 KiB/s wr, 381 op/s 2024-04-21T08:55:33.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:33 smithi007 ceph-mon[28477]: pgmap v341: 129 pgs: 17 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 60 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 411 KiB/s wr, 381 op/s 2024-04-21T08:55:33.594 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/super.o 2024-04-21T08:55:33.707 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/umh.o 2024-04-21T08:55:33.859 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/secretmem.o 2024-04-21T08:55:34.146 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/selinux/netlabel.o 2024-04-21T08:55:34.655 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/sysctl.o 2024-04-21T08:55:34.708 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/char_dev.o 2024-04-21T08:55:35.156 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/workqueue.o 2024-04-21T08:55:35.191 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/memfd.o 2024-04-21T08:55:35.269 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/pid.o 2024-04-21T08:55:35.304 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:35 smithi016 ceph-mon[31259]: pgmap v342: 129 pgs: 17 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 60 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 443 KiB/s wr, 456 op/s 2024-04-21T08:55:35.330 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/stat.o 2024-04-21T08:55:35.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:35 smithi077 ceph-mon[31615]: pgmap v342: 129 pgs: 17 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 60 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 443 KiB/s wr, 456 op/s 2024-04-21T08:55:35.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:35 smithi007 ceph-mon[28477]: pgmap v342: 129 pgs: 17 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 60 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 443 KiB/s wr, 456 op/s 2024-04-21T08:55:35.870 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/exec.o 2024-04-21T08:55:35.888 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs2super.o 2024-04-21T08:55:35.944 INFO:tasks.workunit.client.0.smithi007.stdout: AR security/selinux/built-in.a 2024-04-21T08:55:36.020 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/proc.o 2024-04-21T08:55:36.113 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/min_addr.o 2024-04-21T08:55:36.571 INFO:tasks.workunit.client.0.smithi007.stdout: CC mm/ptdump.o 2024-04-21T08:55:36.825 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/security.o 2024-04-21T08:55:37.103 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/lsm_audit.o 2024-04-21T08:55:37.142 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/task_work.o 2024-04-21T08:55:37.167 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs2xdr.o 2024-04-21T08:55:37.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:37 smithi077 ceph-mon[31615]: pgmap v343: 129 pgs: 16 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 422 KiB/s wr, 418 op/s 2024-04-21T08:55:37.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:37 smithi007 ceph-mon[28477]: pgmap v343: 129 pgs: 16 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 422 KiB/s wr, 418 op/s 2024-04-21T08:55:37.531 INFO:tasks.workunit.client.0.smithi007.stdout: AR mm/built-in.a 2024-04-21T08:55:37.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:37 smithi016 ceph-mon[31259]: pgmap v343: 129 pgs: 16 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 422 KiB/s wr, 418 op/s 2024-04-21T08:55:37.660 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs3super.o 2024-04-21T08:55:37.918 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/pipe.o 2024-04-21T08:55:38.003 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/namei.o 2024-04-21T08:55:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.058+0000 7effc5ea5640 1 -- 172.21.15.7:0/1225045531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc0108f50 msgr2=0x7effc01093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.058+0000 7effc5ea5640 1 --2- 172.21.15.7:0/1225045531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc0108f50 0x7effc01093d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7effb401a910 tx=0x7effb403cd40 comp rx=0 tx=0).stop 2024-04-21T08:55:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.059+0000 7effc5ea5640 1 -- 172.21.15.7:0/1225045531 shutdown_connections 2024-04-21T08:55:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.059+0000 7effc5ea5640 1 --2- 172.21.15.7:0/1225045531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc0109b00 0x7effc011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.059+0000 7effc5ea5640 1 --2- 172.21.15.7:0/1225045531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc0108f50 0x7effc01093d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.059+0000 7effc5ea5640 1 --2- 172.21.15.7:0/1225045531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effc0110550 0x7effc0110930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.059+0000 7effc5ea5640 1 -- 172.21.15.7:0/1225045531 >> 172.21.15.7:0/1225045531 conn(0x7effc006cea0 msgr2=0x7effc006d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.059+0000 7effc5ea5640 1 -- 172.21.15.7:0/1225045531 shutdown_connections 2024-04-21T08:55:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.059+0000 7effc5ea5640 1 -- 172.21.15.7:0/1225045531 wait complete. 2024-04-21T08:55:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.060+0000 7effc5ea5640 1 Processor -- start 2024-04-21T08:55:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.060+0000 7effc5ea5640 1 -- start start 2024-04-21T08:55:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effc5ea5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc0108f50 0x7effc011d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effc5ea5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effc0109b00 0x7effc0119400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effc5ea5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc0110550 0x7effc0119940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effc5ea5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7effc0113860 con 0x7effc0108f50 2024-04-21T08:55:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effc5ea5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7effc01136e0 con 0x7effc0109b00 2024-04-21T08:55:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effc5ea5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7effc01139e0 con 0x7effc0110550 2024-04-21T08:55:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effbffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc0110550 0x7effc0119940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effbffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc0110550 0x7effc0119940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:60926/0 (socket says 172.21.15.7:60926) 2024-04-21T08:55:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effbffff640 1 -- 172.21.15.7:0/3529668388 learned_addr learned my addr 172.21.15.7:0/3529668388 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effbffff640 1 -- 172.21.15.7:0/3529668388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effc0109b00 msgr2=0x7effc0119400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effbffff640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effc0109b00 0x7effc0119400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effbffff640 1 -- 172.21.15.7:0/3529668388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc0108f50 msgr2=0x7effc011d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effbffff640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc0108f50 0x7effc011d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effbffff640 1 -- 172.21.15.7:0/3529668388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc011a070 con 0x7effc0110550 2024-04-21T08:55:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.061+0000 7effbffff640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc0110550 0x7effc0119940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7effb0009fd0 tx=0x7effb000b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.073+0000 7effbcff9640 1 -- 172.21.15.7:0/3529668388 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0011830 con 0x7effc0110550 2024-04-21T08:55:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.073+0000 7effbcff9640 1 -- 172.21.15.7:0/3529668388 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effb0016070 con 0x7effc0110550 2024-04-21T08:55:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.073+0000 7effc5ea5640 1 -- 172.21.15.7:0/3529668388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc007a1b0 con 0x7effc0110550 2024-04-21T08:55:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.074+0000 7effc5ea5640 1 -- 172.21.15.7:0/3529668388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc007a780 con 0x7effc0110550 2024-04-21T08:55:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.076+0000 7effbcff9640 1 -- 172.21.15.7:0/3529668388 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0010e50 con 0x7effc0110550 2024-04-21T08:55:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.077+0000 7effc8b72740 1 Processor -- start 2024-04-21T08:55:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.078+0000 7effc8b72740 1 -- start start 2024-04-21T08:55:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effbcff9640 1 -- 172.21.15.7:0/3529668388 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7effb001b4a0 con 0x7effc0110550 2024-04-21T08:55:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effc8b72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f794ac0 0x564a4f793810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effc8b72740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 0x564a4f7941d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effc8b72740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f79d2d0 0x564a4f79f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effc8b72740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564a4f6f88d0 con 0x564a4f793d50 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effc8b72740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564a4f695e80 con 0x564a4f794ac0 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effc8b72740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564a4f6f4890 con 0x564a4f79d2d0 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effbeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f794ac0 0x564a4f793810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effbeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f79d2d0 0x564a4f79f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effbeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f794ac0 0x564a4f793810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53378/0 (socket says 172.21.15.7:53378) 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effbeffd640 1 -- 172.21.15.7:0/3811973335 learned_addr learned my addr 172.21.15.7:0/3811973335 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effbeffd640 1 -- 172.21.15.7:0/3811973335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f79d2d0 msgr2=0x564a4f79f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effbeffd640 1 --2- 172.21.15.7:0/3811973335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f79d2d0 0x564a4f79f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effbeffd640 1 -- 172.21.15.7:0/3811973335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 msgr2=0x564a4f7941d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effbeffd640 1 --2- 172.21.15.7:0/3811973335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 0x564a4f7941d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.079+0000 7effbeffd640 1 -- 172.21.15.7:0/3811973335 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a4f6b7320 con 0x564a4f794ac0 2024-04-21T08:55:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.080+0000 7effbeffd640 1 --2- 172.21.15.7:0/3811973335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f794ac0 0x564a4f793810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7effb4046000 tx=0x7effb4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75c75e7a10fb7e52 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.080+0000 7effbcff9640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7effa80a40d0 0x7effa80a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.081+0000 7effbcff9640 1 -- 172.21.15.7:0/3529668388 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7effb00d0660 con 0x7effc0110550 2024-04-21T08:55:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.083+0000 7effbf7fe640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7effa80a40d0 0x7effa80a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.096+0000 7effa5ffb640 1 -- 172.21.15.7:0/3811973335 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb403d940 con 0x564a4f794ac0 2024-04-21T08:55:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.096+0000 7effa5ffb640 1 -- 172.21.15.7:0/3811973335 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effb40035f0 con 0x564a4f794ac0 2024-04-21T08:55:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.096+0000 7effa5ffb640 1 -- 172.21.15.7:0/3811973335 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb40456a0 con 0x564a4f794ac0 2024-04-21T08:55:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.098+0000 7effbf7fe640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7effa80a40d0 0x7effa80a6590 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x564a4f795220 tx=0x7effb8009250 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.100+0000 7effc8b72740 1 -- 172.21.15.7:0/3811973335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f794ac0 msgr2=0x564a4f793810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.100+0000 7effc8b72740 1 --2- 172.21.15.7:0/3811973335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f794ac0 0x564a4f793810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7effb4046000 tx=0x7effb4000bc0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.100+0000 7effc8b72740 1 -- 172.21.15.7:0/3811973335 shutdown_connections 2024-04-21T08:55:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.100+0000 7effc8b72740 1 --2- 172.21.15.7:0/3811973335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f79d2d0 0x564a4f79f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.100+0000 7effc8b72740 1 --2- 172.21.15.7:0/3811973335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 0x564a4f7941d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.100+0000 7effc8b72740 1 --2- 172.21.15.7:0/3811973335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f794ac0 0x564a4f793810 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.100+0000 7effc8b72740 1 -- 172.21.15.7:0/3811973335 >> 172.21.15.7:0/3811973335 conn(0x564a4f75ac90 msgr2=0x564a4f78ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.100+0000 7effc8b72740 1 -- 172.21.15.7:0/3811973335 shutdown_connections 2024-04-21T08:55:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.100+0000 7effc8b72740 1 -- 172.21.15.7:0/3811973335 wait complete. 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.101+0000 7effc8b72740 1 -- start start 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.101+0000 7effc8b72740 1 Processor -- start 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.102+0000 7effc8b72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f793d50 0x564a4f79cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.102+0000 7effc8b72740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f794ac0 0x564a4f799fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.102+0000 7effc8b72740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f79d2d0 0x564a4f79a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.102+0000 7effc8b72740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564a4f79ff40 con 0x564a4f794ac0 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.102+0000 7effc8b72740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x564a4f680eb0 con 0x564a4f793d50 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.102+0000 7effc8b72740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564a4f7a00c0 con 0x564a4f79d2d0 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.102+0000 7effc8b72740 10 client.0 _ll_register_callbacks cb 0x564a4f6adb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.102+0000 7effbf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f793d50 0x564a4f79cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.102+0000 7effbf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f793d50 0x564a4f79cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53384/0 (socket says 172.21.15.7:53384) 2024-04-21T08:55:38.103 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.102+0000 7effbf7fe640 1 -- 172.21.15.7:0/1460056502 learned_addr learned my addr 172.21.15.7:0/1460056502 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.102+0000 7effbffff640 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f79d2d0 0x564a4f79a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.103+0000 7effbf7fe640 1 -- 172.21.15.7:0/1460056502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f79d2d0 msgr2=0x564a4f79a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.103+0000 7effbf7fe640 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f79d2d0 0x564a4f79a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.103+0000 7effbf7fe640 1 -- 172.21.15.7:0/1460056502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f794ac0 msgr2=0x564a4f799fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.103+0000 7effbf7fe640 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f794ac0 0x564a4f799fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.103+0000 7effbf7fe640 1 -- 172.21.15.7:0/1460056502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a4f40b210 con 0x564a4f793d50 2024-04-21T08:55:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.103+0000 7effbf7fe640 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f793d50 0x564a4f79cc40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7effb8004b10 tx=0x7effb8004bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a347474c2c1934a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.107+0000 7eff9effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:55:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.107+0000 7eff9effd640 1 -- 172.21.15.7:0/1460056502 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb800ce40 con 0x564a4f793d50 2024-04-21T08:55:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.107+0000 7effc8b72740 1 -- 172.21.15.7:0/1460056502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effb40035f0 con 0x564a4f793d50 2024-04-21T08:55:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.107+0000 7eff9effd640 1 -- 172.21.15.7:0/1460056502 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7effb8023480 con 0x564a4f793d50 2024-04-21T08:55:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.107+0000 7eff9effd640 1 -- 172.21.15.7:0/1460056502 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7effb802c050 con 0x564a4f793d50 2024-04-21T08:55:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.107+0000 7effc8b72740 10 client.26879 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.107+0000 7effc8b72740 1 -- 172.21.15.7:0/1460056502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564a4f4d7620 con 0x564a4f793d50 2024-04-21T08:55:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.108+0000 7eff9effd640 1 -- 172.21.15.7:0/1460056502 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effb40035f0 con 0x564a4f793d50 2024-04-21T08:55:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.108+0000 7eff9effd640 1 -- 172.21.15.7:0/1460056502 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7effb801a860 con 0x564a4f793d50 2024-04-21T08:55:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.108+0000 7effc8b72740 10 client.26879 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.108+0000 7effc8b72740 10 client.26879 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.108+0000 7effc8b72740 20 client.26879 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.108+0000 7effc8b72740 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564a4f7ad370 0x564a4f7af800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.108+0000 7effc8b72740 4 client.26879 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.108+0000 7effc8b72740 1 -- 172.21.15.7:0/1460056502 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a4f70c880 con 0x564a4f7ad370 2024-04-21T08:55:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.108+0000 7effbeffd640 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564a4f7ad370 0x564a4f7af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.109+0000 7effbeffd640 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564a4f7ad370 0x564a4f7af800 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d78c676527a6ed3 server_cookie=336bf5f41c11ac93 in_seq=0 out_seq=0 2024-04-21T08:55:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.112+0000 7eff9effd640 10 client.26879 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.112+0000 7eff9effd640 1 -- 172.21.15.7:0/1460056502 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a4f70c880 con 0x564a4f7ad370 2024-04-21T08:55:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.112+0000 7eff9effd640 10 client.26879 handle_command_reply: tid=0 2024-04-21T08:55:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.112+0000 7effc8b72740 1 client.26879 shutdown 2024-04-21T08:55:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.120+0000 7effc8b72740 1 -- 172.21.15.7:0/1460056502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f793d50 msgr2=0x564a4f79cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.120+0000 7effc8b72740 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f793d50 0x564a4f79cc40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7effb8004b10 tx=0x7effb8004bb0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.126+0000 7effc8b72740 1 -- 172.21.15.7:0/1460056502 shutdown_connections 2024-04-21T08:55:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.126+0000 7effc8b72740 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564a4f7ad370 0x564a4f7af800 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.126+0000 7effc8b72740 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f79d2d0 0x564a4f79a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.126+0000 7effc8b72740 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f794ac0 0x564a4f799fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.126+0000 7effc8b72740 1 --2- 172.21.15.7:0/1460056502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f793d50 0x564a4f79cc40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.126+0000 7effc8b72740 1 -- 172.21.15.7:0/1460056502 >> 172.21.15.7:0/1460056502 conn(0x564a4f75ac90 msgr2=0x564a4f790790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.126+0000 7eff9effd640 0 client.26879 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.132+0000 7effc8b72740 1 -- 172.21.15.7:0/1460056502 shutdown_connections 2024-04-21T08:55:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.132+0000 7effc8b72740 1 -- 172.21.15.7:0/1460056502 wait complete. 2024-04-21T08:55:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.138+0000 7effc8b72740 20 client.26879 trim_cache size 0 max 16384 2024-04-21T08:55:38.206 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.206+0000 7effc8b72740 1 Processor -- start 2024-04-21T08:55:38.206 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.206+0000 7effc8b72740 1 -- start start 2024-04-21T08:55:38.207 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.207+0000 7effc8b72740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 0x564a4f7a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.207 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.207+0000 7effc8b72740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f794ac0 0x564a4f79f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.207 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.207+0000 7effc8b72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 0x564a4f79c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.207 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.207+0000 7effc8b72740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7effac00e320 con 0x564a4f793d50 2024-04-21T08:55:38.207 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.207+0000 7effbeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f794ac0 0x564a4f79f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.207 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.207+0000 7effbeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f794ac0 0x564a4f79f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:60954/0 (socket says 172.21.15.7:60954) 2024-04-21T08:55:38.208 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.207+0000 7effbeffd640 1 -- 172.21.15.7:0/2806427544 learned_addr learned my addr 172.21.15.7:0/2806427544 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:38.208 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.207+0000 7effbffff640 1 --2- 172.21.15.7:0/2806427544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 0x564a4f79c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.208 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.208+0000 7effc8b72740 1 -- 172.21.15.7:0/2806427544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7effac010b80 con 0x564a4f79d2d0 2024-04-21T08:55:38.208 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.208+0000 7effc8b72740 1 -- 172.21.15.7:0/2806427544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7effac00e970 con 0x564a4f794ac0 2024-04-21T08:55:38.208 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.208+0000 7effbf7fe640 1 --2- 172.21.15.7:0/2806427544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 0x564a4f7a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.209 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.208+0000 7effbffff640 1 -- 172.21.15.7:0/2806427544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f794ac0 msgr2=0x564a4f79f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:38.209 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.208+0000 7effbffff640 1 --2- 172.21.15.7:0/2806427544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f794ac0 0x564a4f79f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.209 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.208+0000 7effbffff640 1 -- 172.21.15.7:0/2806427544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 msgr2=0x564a4f7a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:38.209 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.208+0000 7effbffff640 1 --2- 172.21.15.7:0/2806427544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 0x564a4f7a8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.209 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.208+0000 7effbffff640 1 -- 172.21.15.7:0/2806427544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a4f793020 con 0x564a4f79d2d0 2024-04-21T08:55:38.209 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.208+0000 7effbffff640 1 --2- 172.21.15.7:0/2806427544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 0x564a4f79c700 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7effb00cd9a0 tx=0x7effb00047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f924d1be5bfcee43 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:38.221 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/extable.o 2024-04-21T08:55:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.225+0000 7effa57fa640 1 -- 172.21.15.7:0/2806427544 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0098860 con 0x564a4f79d2d0 2024-04-21T08:55:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.225+0000 7effa57fa640 1 -- 172.21.15.7:0/2806427544 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effb00cdb10 con 0x564a4f79d2d0 2024-04-21T08:55:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.225+0000 7effa57fa640 1 -- 172.21.15.7:0/2806427544 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb00a4c30 con 0x564a4f79d2d0 2024-04-21T08:55:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.226+0000 7effc8b72740 1 -- 172.21.15.7:0/2806427544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 msgr2=0x564a4f79c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.226+0000 7effc8b72740 1 --2- 172.21.15.7:0/2806427544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 0x564a4f79c700 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7effb00cd9a0 tx=0x7effb00047d0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.226+0000 7effc8b72740 1 -- 172.21.15.7:0/2806427544 shutdown_connections 2024-04-21T08:55:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.226+0000 7effc8b72740 1 --2- 172.21.15.7:0/2806427544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 0x564a4f79c700 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.226+0000 7effc8b72740 1 --2- 172.21.15.7:0/2806427544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f794ac0 0x564a4f79f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.226+0000 7effc8b72740 1 --2- 172.21.15.7:0/2806427544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 0x564a4f7a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.226+0000 7effc8b72740 1 -- 172.21.15.7:0/2806427544 >> 172.21.15.7:0/2806427544 conn(0x564a4f75ac90 msgr2=0x564a4f799860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.226+0000 7effc8b72740 1 -- 172.21.15.7:0/2806427544 shutdown_connections 2024-04-21T08:55:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.226+0000 7effc8b72740 1 -- 172.21.15.7:0/2806427544 wait complete. 2024-04-21T08:55:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.227+0000 7effc8b72740 1 -- start start 2024-04-21T08:55:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.227+0000 7effc8b72740 1 Processor -- start 2024-04-21T08:55:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effc8b72740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 0x564a4f7a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effc8b72740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f794ac0 0x564a4f7a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effc8b72740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 0x564a4f7a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effc8b72740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7effac00f040 con 0x564a4f793d50 2024-04-21T08:55:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effc8b72740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7effac0104b0 con 0x564a4f79d2d0 2024-04-21T08:55:38.228 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effc8b72740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7effac00f710 con 0x564a4f794ac0 2024-04-21T08:55:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effc8b72740 10 client.0 _ll_register_callbacks cb 0x7effb800cb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effbffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 0x564a4f7a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effbffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 0x564a4f7a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53394/0 (socket says 172.21.15.7:53394) 2024-04-21T08:55:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effbffff640 1 -- 172.21.15.7:0/2340596427 learned_addr learned my addr 172.21.15.7:0/2340596427 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effbf7fe640 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 0x564a4f7a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.228+0000 7effbeffd640 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f794ac0 0x564a4f7a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.229+0000 7effbffff640 1 -- 172.21.15.7:0/2340596427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f794ac0 msgr2=0x564a4f7a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.229+0000 7effbffff640 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f794ac0 0x564a4f7a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.229+0000 7effbffff640 1 -- 172.21.15.7:0/2340596427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 msgr2=0x564a4f7a03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.229+0000 7effbffff640 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 0x564a4f7a03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.229+0000 7effbffff640 1 -- 172.21.15.7:0/2340596427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a4f811bc0 con 0x564a4f79d2d0 2024-04-21T08:55:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.229+0000 7effbffff640 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 0x564a4f7a0e20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7effb0098650 tx=0x7effb0010000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=508a2e927dc7c70f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.236+0000 7eff9effd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:55:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.236+0000 7eff9effd640 1 -- 172.21.15.7:0/2340596427 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb00b4070 con 0x564a4f79d2d0 2024-04-21T08:55:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.236+0000 7effc8b72740 1 -- 172.21.15.7:0/2340596427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effb00cdb10 con 0x564a4f79d2d0 2024-04-21T08:55:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.236+0000 7eff9effd640 1 -- 172.21.15.7:0/2340596427 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7effb00a0150 con 0x564a4f79d2d0 2024-04-21T08:55:38.245 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.246+0000 7eff9effd640 1 -- 172.21.15.7:0/2340596427 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7effb00a49d0 con 0x564a4f79d2d0 2024-04-21T08:55:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.247+0000 7effc8b72740 10 client.26885 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.247+0000 7effc8b72740 1 -- 172.21.15.7:0/2340596427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564a4f812e70 con 0x564a4f79d2d0 2024-04-21T08:55:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.247+0000 7eff9effd640 1 -- 172.21.15.7:0/2340596427 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effb00cdb10 con 0x564a4f79d2d0 2024-04-21T08:55:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.248+0000 7eff9effd640 1 -- 172.21.15.7:0/2340596427 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7effb009b640 con 0x564a4f79d2d0 2024-04-21T08:55:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.248+0000 7effc8b72740 10 client.26885 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.248+0000 7effc8b72740 10 client.26885 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.248+0000 7effc8b72740 20 client.26885 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.248+0000 7effc8b72740 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564a4f813420 0x564a4f815840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:38.249 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.248+0000 7effc8b72740 4 client.26885 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:38.249 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.248+0000 7effc8b72740 1 -- 172.21.15.7:0/2340596427 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a4f6983d0 con 0x564a4f813420 2024-04-21T08:55:38.249 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.249+0000 7effbf7fe640 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564a4f813420 0x564a4f815840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:38.249 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.249+0000 7effbf7fe640 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564a4f813420 0x564a4f815840 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40e4f4af6b6590bf server_cookie=78c1daeb9b118eaf in_seq=0 out_seq=0 2024-04-21T08:55:38.249 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:38.249+0000 7eff9effd640 10 client.26885 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:38.630 INFO:tasks.workunit.client.0.smithi007.stdout: CC security/device_cgroup.o 2024-04-21T08:55:38.985 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs3client.o 2024-04-21T08:55:39.123 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fcntl.o 2024-04-21T08:55:39.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:39 smithi077 ceph-mon[31615]: pgmap v344: 129 pgs: 16 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 62 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 825 KiB/s wr, 441 op/s 2024-04-21T08:55:39.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:39 smithi007 ceph-mon[28477]: pgmap v344: 129 pgs: 16 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 62 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 825 KiB/s wr, 441 op/s 2024-04-21T08:55:39.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:55:39.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:55:39.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:39 smithi016 ceph-mon[31259]: pgmap v344: 129 pgs: 16 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 62 active+clean; 1.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 825 KiB/s wr, 441 op/s 2024-04-21T08:55:39.630 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/params.o 2024-04-21T08:55:39.704 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs3proc.o 2024-04-21T08:55:39.756 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/ioctl.o 2024-04-21T08:55:40.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.143+0000 7f14c0552640 1 -- 172.21.15.7:0/1566106295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b810b210 msgr2=0x7f14b8113960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:40.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.143+0000 7f14c0552640 1 --2- 172.21.15.7:0/1566106295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b810b210 0x7f14b8113960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009ca0 tx=0x7f14b002c120 comp rx=0 tx=0).stop 2024-04-21T08:55:40.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.145+0000 7f14be2ee640 1 --2- 172.21.15.7:0/1566106295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14b810a8f0 0x7f14b810acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:55:40.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.148+0000 7f14c0552640 1 -- 172.21.15.7:0/1566106295 shutdown_connections 2024-04-21T08:55:40.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.148+0000 7f14c0552640 1 --2- 172.21.15.7:0/1566106295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b8113ea0 0x7f14b8116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.148+0000 7f14c0552640 1 --2- 172.21.15.7:0/1566106295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b810b210 0x7f14b8113960 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.148+0000 7f14c0552640 1 --2- 172.21.15.7:0/1566106295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14b810a8f0 0x7f14b810acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.148+0000 7f14c0552640 1 -- 172.21.15.7:0/1566106295 >> 172.21.15.7:0/1566106295 conn(0x7f14b806e360 msgr2=0x7f14b806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:40.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.148+0000 7f14c0552640 1 -- 172.21.15.7:0/1566106295 shutdown_connections 2024-04-21T08:55:40.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.148+0000 7f14c0552640 1 -- 172.21.15.7:0/1566106295 wait complete. 2024-04-21T08:55:40.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.149+0000 7f14c0552640 1 Processor -- start 2024-04-21T08:55:40.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.149+0000 7f14c0552640 1 -- start start 2024-04-21T08:55:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.149+0000 7f14c0552640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b810a8f0 0x7f14b8116cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.149+0000 7f14c0552640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14b810b210 0x7f14b8117210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.149+0000 7f14c0552640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b8113ea0 0x7f14b8117750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.149+0000 7f14c0552640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f14b811aa50 con 0x7f14b810b210 2024-04-21T08:55:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.149+0000 7f14c0552640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f14b811a8d0 con 0x7f14b8113ea0 2024-04-21T08:55:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.150+0000 7f14c0552640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f14b811abd0 con 0x7f14b810a8f0 2024-04-21T08:55:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.150+0000 7f14beaef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b8113ea0 0x7f14b8117750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.150+0000 7f14beaef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b8113ea0 0x7f14b8117750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53398/0 (socket says 172.21.15.7:53398) 2024-04-21T08:55:40.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.150+0000 7f14beaef640 1 -- 172.21.15.7:0/196522290 learned_addr learned my addr 172.21.15.7:0/196522290 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:40.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.150+0000 7f14beaef640 1 -- 172.21.15.7:0/196522290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b810a8f0 msgr2=0x7f14b8116cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:55:40.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.150+0000 7f14beaef640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b810a8f0 0x7f14b8116cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.150+0000 7f14beaef640 1 -- 172.21.15.7:0/196522290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14b810b210 msgr2=0x7f14b8117210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:40.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.150+0000 7f14beaef640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14b810b210 0x7f14b8117210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.150+0000 7f14beaef640 1 -- 172.21.15.7:0/196522290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b81b3810 con 0x7f14b8113ea0 2024-04-21T08:55:40.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.151+0000 7f14beaef640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b8113ea0 0x7f14b8117750 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f14b400c970 tx=0x7f14b400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:40.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.161+0000 7f14af7fe640 1 -- 172.21.15.7:0/196522290 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4007bb0 con 0x7f14b8113ea0 2024-04-21T08:55:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.161+0000 7f14c0552640 1 -- 172.21.15.7:0/196522290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b81b3a40 con 0x7f14b8113ea0 2024-04-21T08:55:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.161+0000 7f14c0552640 1 -- 172.21.15.7:0/196522290 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b81b3f90 con 0x7f14b8113ea0 2024-04-21T08:55:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.161+0000 7f14af7fe640 1 -- 172.21.15.7:0/196522290 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b4007d50 con 0x7f14b8113ea0 2024-04-21T08:55:40.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.161+0000 7f14af7fe640 1 -- 172.21.15.7:0/196522290 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4004e60 con 0x7f14b8113ea0 2024-04-21T08:55:40.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.165+0000 7f14c3221740 1 Processor -- start 2024-04-21T08:55:40.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.165+0000 7f14c3221740 1 -- start start 2024-04-21T08:55:40.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.166+0000 7f14c3221740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932580 0x55ef4c932960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14c3221740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c93b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14c3221740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c93de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14c3221740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c8971e0 con 0x55ef4c93ba80 2024-04-21T08:55:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14c3221740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c834390 con 0x55ef4c932ea0 2024-04-21T08:55:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14c3221740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c8931a0 con 0x55ef4c932580 2024-04-21T08:55:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14be2ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c93b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14bdaed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932580 0x55ef4c932960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14be2ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c93b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53410/0 (socket says 172.21.15.7:53410) 2024-04-21T08:55:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14be2ee640 1 -- 172.21.15.7:0/242976913 learned_addr learned my addr 172.21.15.7:0/242976913 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14be2ee640 1 -- 172.21.15.7:0/242976913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932580 msgr2=0x55ef4c932960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14be2ee640 1 --2- 172.21.15.7:0/242976913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932580 0x55ef4c932960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14be2ee640 1 -- 172.21.15.7:0/242976913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 msgr2=0x55ef4c93de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14be2ee640 1 --2- 172.21.15.7:0/242976913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c93de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.167+0000 7f14be2ee640 1 -- 172.21.15.7:0/242976913 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4c6737d0 con 0x55ef4c932ea0 2024-04-21T08:55:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.168+0000 7f14be2ee640 1 --2- 172.21.15.7:0/242976913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c93b540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f14a800aa20 tx=0x7f14a800aee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7d657f2b2f93cf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.168+0000 7f14af7fe640 1 -- 172.21.15.7:0/196522290 <== mon.2 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f14b400f550 con 0x7f14b8113ea0 2024-04-21T08:55:40.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.169+0000 7f14af7fe640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f14a40a3f20 0x7f14a40a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.170+0000 7f14af7fe640 1 -- 172.21.15.7:0/196522290 <== mon.2 v2:172.21.15.16:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7f14b40d18a0 con 0x7f14b8113ea0 2024-04-21T08:55:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.178+0000 7f14bdaed640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f14a40a3f20 0x7f14a40a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.197+0000 7f14acbfe640 1 -- 172.21.15.7:0/242976913 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a800ce40 con 0x55ef4c932ea0 2024-04-21T08:55:40.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.197+0000 7f14acbfe640 1 -- 172.21.15.7:0/242976913 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a800f900 con 0x55ef4c932ea0 2024-04-21T08:55:40.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.201+0000 7f14acbfe640 1 -- 172.21.15.7:0/242976913 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a80189a0 con 0x55ef4c932ea0 2024-04-21T08:55:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.201+0000 7f14c3221740 1 -- 172.21.15.7:0/242976913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 msgr2=0x55ef4c93b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.202+0000 7f14c3221740 1 --2- 172.21.15.7:0/242976913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c93b540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f14a800aa20 tx=0x7f14a800aee0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.202+0000 7f14c3221740 1 -- 172.21.15.7:0/242976913 shutdown_connections 2024-04-21T08:55:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.202+0000 7f14c3221740 1 --2- 172.21.15.7:0/242976913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c93de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.202+0000 7f14c3221740 1 --2- 172.21.15.7:0/242976913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c93b540 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.202+0000 7f14c3221740 1 --2- 172.21.15.7:0/242976913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932580 0x55ef4c932960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.202+0000 7f14c3221740 1 -- 172.21.15.7:0/242976913 >> 172.21.15.7:0/242976913 conn(0x55ef4c92ccb0 msgr2=0x55ef4c92d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:40.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.202+0000 7f14c3221740 1 -- 172.21.15.7:0/242976913 shutdown_connections 2024-04-21T08:55:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.209+0000 7f14c3221740 1 -- 172.21.15.7:0/242976913 wait complete. 2024-04-21T08:55:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.209+0000 7f14c3221740 1 -- start start 2024-04-21T08:55:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.209+0000 7f14c3221740 1 Processor -- start 2024-04-21T08:55:40.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.210+0000 7f14c3221740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c932580 0x55ef4c938ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.210+0000 7f14c3221740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c938ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.210+0000 7f14c3221740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c93ba80 0x55ef4c939530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.210+0000 7f14c3221740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c93e870 con 0x55ef4c932580 2024-04-21T08:55:40.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.210+0000 7f14c3221740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c8502f0 con 0x55ef4c932ea0 2024-04-21T08:55:40.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.210+0000 7f14c3221740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef4c93e9f0 con 0x55ef4c93ba80 2024-04-21T08:55:40.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.210+0000 7f14c3221740 10 client.0 _ll_register_callbacks cb 0x55ef4c85d580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:40.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.210+0000 7f14beaef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c93ba80 0x55ef4c939530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.211+0000 7f14beaef640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c93ba80 0x55ef4c939530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:60998/0 (socket says 172.21.15.7:60998) 2024-04-21T08:55:40.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.211+0000 7f14beaef640 1 -- 172.21.15.7:0/2265262495 learned_addr learned my addr 172.21.15.7:0/2265262495 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.215+0000 7f14beaef640 1 -- 172.21.15.7:0/2265262495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 msgr2=0x55ef4c938ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.215+0000 7f14beaef640 1 --2- 172.21.15.7:0/2265262495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c938ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.215+0000 7f14beaef640 1 -- 172.21.15.7:0/2265262495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c932580 msgr2=0x55ef4c938ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.215+0000 7f14beaef640 1 --2- 172.21.15.7:0/2265262495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c932580 0x55ef4c938ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.215+0000 7f14beaef640 1 -- 172.21.15.7:0/2265262495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef4c5a93f0 con 0x55ef4c93ba80 2024-04-21T08:55:40.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.226+0000 7f14beaef640 1 --2- 172.21.15.7:0/2265262495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c93ba80 0x55ef4c939530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f14b4007590 tx=0x7f14b409d3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3528fc79d70c2318 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:40.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.226+0000 7f14a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:55:40.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.226+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2265262495 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b401d040 con 0x55ef4c93ba80 2024-04-21T08:55:40.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.226+0000 7f14c3221740 1 -- 172.21.15.7:0/2265262495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14a800f900 con 0x55ef4c93ba80 2024-04-21T08:55:40.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.226+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2265262495 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7f14b409c770 con 0x55ef4c93ba80 2024-04-21T08:55:40.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.227+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2265262495 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f14b40cef50 con 0x55ef4c93ba80 2024-04-21T08:55:40.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.227+0000 7f14c3221740 10 client.26875 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:40.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.227+0000 7f14c3221740 1 -- 172.21.15.7:0/2265262495 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ef4c5690f0 con 0x55ef4c93ba80 2024-04-21T08:55:40.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.231+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2265262495 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a800f900 con 0x55ef4c93ba80 2024-04-21T08:55:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.231+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2265262495 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f14b4099180 con 0x55ef4c93ba80 2024-04-21T08:55:40.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.231+0000 7f14c3221740 10 client.26875 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.231+0000 7f14c3221740 10 client.26875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.231+0000 7f14c3221740 20 client.26875 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.231+0000 7f14c3221740 1 --2- 172.21.15.7:0/2265262495 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ef4c95cfc0 0x55ef4c95f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.231+0000 7f14c3221740 4 client.26875 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.231+0000 7f14c3221740 1 -- 172.21.15.7:0/2265262495 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef4c8a4c00 con 0x55ef4c95cfc0 2024-04-21T08:55:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.231+0000 7f14be2ee640 1 --2- 172.21.15.7:0/2265262495 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ef4c95cfc0 0x55ef4c95f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.232+0000 7f14be2ee640 1 --2- 172.21.15.7:0/2265262495 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ef4c95cfc0 0x55ef4c95f480 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57d146fff9d4e6ad server_cookie=af9798c1d5002f62 in_seq=0 out_seq=0 2024-04-21T08:55:40.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.237+0000 7f14a1ffb640 10 client.26875 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:40.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.237+0000 7f14bdaed640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f14a40a3f20 0x7f14a40a63e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f14b0006010 tx=0x7f14b0009820 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:40.281 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs3xdr.o 2024-04-21T08:55:40.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.313+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2265262495 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef4c8a4c00 con 0x55ef4c95cfc0 2024-04-21T08:55:40.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.313+0000 7f14a1ffb640 10 client.26875 handle_command_reply: tid=0 2024-04-21T08:55:40.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.314+0000 7f14c3221740 1 client.26875 shutdown 2024-04-21T08:55:40.318 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:55:40.318 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:55:40.318 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.316+0000 7eff9effd640 1 -- 172.21.15.7:0/2340596427 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a4f6983d0 con 0x564a4f813420 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.316+0000 7eff9effd640 10 client.26885 handle_command_reply: tid=0 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.316+0000 7effc8b72740 1 client.26885 shutdown 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7effc8b72740 1 -- 172.21.15.7:0/2340596427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 msgr2=0x564a4f7a0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7effc8b72740 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 0x564a4f7a0e20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7effb0098650 tx=0x7effb0010000 comp rx=0 tx=0).stop 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7effc8b72740 1 -- 172.21.15.7:0/2340596427 shutdown_connections 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7effc8b72740 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x564a4f813420 0x564a4f815840 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7effc8b72740 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x564a4f79d2d0 0x564a4f7a0e20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7effc8b72740 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564a4f794ac0 0x564a4f7a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7effc8b72740 1 --2- 172.21.15.7:0/2340596427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564a4f793d50 0x564a4f7a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7effc8b72740 1 -- 172.21.15.7:0/2340596427 >> 172.21.15.7:0/2340596427 conn(0x564a4f75ac90 msgr2=0x564a4f795250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7eff9effd640 0 client.26885 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:40.319 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7effc8b72740 1 -- 172.21.15.7:0/2340596427 shutdown_connections 2024-04-21T08:55:40.320 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7effc8b72740 1 -- 172.21.15.7:0/2340596427 wait complete. 2024-04-21T08:55:40.320 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.317+0000 7effc8b72740 20 client.26885 trim_cache size 0 max 16384 2024-04-21T08:55:40.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.320+0000 7f14c3221740 1 -- 172.21.15.7:0/2265262495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c93ba80 msgr2=0x55ef4c939530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:40.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.320+0000 7f14c3221740 1 --2- 172.21.15.7:0/2265262495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c93ba80 0x55ef4c939530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f14b4007590 tx=0x7f14b409d3e0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.327+0000 7f14c3221740 1 -- 172.21.15.7:0/2265262495 shutdown_connections 2024-04-21T08:55:40.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.327+0000 7f14c3221740 1 --2- 172.21.15.7:0/2265262495 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ef4c95cfc0 0x55ef4c95f480 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.327+0000 7f14c3221740 1 --2- 172.21.15.7:0/2265262495 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c93ba80 0x55ef4c939530 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.327+0000 7f14c3221740 1 --2- 172.21.15.7:0/2265262495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c938ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.327+0000 7f14c3221740 1 --2- 172.21.15.7:0/2265262495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c932580 0x55ef4c938ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.327+0000 7f14c3221740 1 -- 172.21.15.7:0/2265262495 >> 172.21.15.7:0/2265262495 conn(0x55ef4c92ccb0 msgr2=0x55ef4c92ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:40.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.327+0000 7f14a1ffb640 0 client.26875 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:40.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.327+0000 7f14c3221740 1 -- 172.21.15.7:0/2265262495 shutdown_connections 2024-04-21T08:55:40.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.327+0000 7f14c3221740 1 -- 172.21.15.7:0/2265262495 wait complete. 2024-04-21T08:55:40.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.327+0000 7f14c3221740 20 client.26875 trim_cache size 0 max 16384 2024-04-21T08:55:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.336+0000 7effc5ea5640 1 -- 172.21.15.7:0/3529668388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7effa80a40d0 msgr2=0x7effa80a6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.336+0000 7effc5ea5640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7effa80a40d0 0x7effa80a6590 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x564a4f795220 tx=0x7effb8009250 comp rx=0 tx=0).stop 2024-04-21T08:55:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.336+0000 7effc5ea5640 1 -- 172.21.15.7:0/3529668388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc0110550 msgr2=0x7effc0119940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.336+0000 7effc5ea5640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc0110550 0x7effc0119940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7effb0009fd0 tx=0x7effb000b8e0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.340+0000 7effc5ea5640 1 -- 172.21.15.7:0/3529668388 shutdown_connections 2024-04-21T08:55:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.340+0000 7effc5ea5640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7effa80a40d0 0x7effa80a6590 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.340+0000 7effc5ea5640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effc0110550 0x7effc0119940 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.340+0000 7effc5ea5640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effc0109b00 0x7effc0119400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.340+0000 7effc5ea5640 1 --2- 172.21.15.7:0/3529668388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effc0108f50 0x7effc011d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.340+0000 7effc5ea5640 1 -- 172.21.15.7:0/3529668388 >> 172.21.15.7:0/3529668388 conn(0x7effc006cea0 msgr2=0x7effc0071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.343+0000 7effc5ea5640 1 -- 172.21.15.7:0/3529668388 shutdown_connections 2024-04-21T08:55:40.344 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:40.343+0000 7effc5ea5640 1 -- 172.21.15.7:0/3529668388 wait complete. 2024-04-21T08:55:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.409+0000 7f14c3221740 1 Processor -- start 2024-04-21T08:55:40.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.410+0000 7f14c3221740 1 -- start start 2024-04-21T08:55:40.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.410+0000 7f14c3221740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932580 0x55ef4c93b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.410+0000 7f14c3221740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c947790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.411+0000 7f14c3221740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c964d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.411+0000 7f14bdaed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c947790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.411+0000 7f14bdaed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c947790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53424/0 (socket says 172.21.15.7:53424) 2024-04-21T08:55:40.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.411+0000 7f14bdaed640 1 -- 172.21.15.7:0/218104007 learned_addr learned my addr 172.21.15.7:0/218104007 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.411+0000 7f14beaef640 1 --2- 172.21.15.7:0/218104007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c964d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.411+0000 7f14c3221740 1 -- 172.21.15.7:0/218104007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f149c00e370 con 0x55ef4c93ba80 2024-04-21T08:55:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.411+0000 7f14c3221740 1 -- 172.21.15.7:0/218104007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f149c010bd0 con 0x55ef4c932ea0 2024-04-21T08:55:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.411+0000 7f14c3221740 1 -- 172.21.15.7:0/218104007 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f149c00e9c0 con 0x55ef4c932580 2024-04-21T08:55:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.412+0000 7f14bdaed640 1 -- 172.21.15.7:0/218104007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932580 msgr2=0x55ef4c93b1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.412+0000 7f14bdaed640 1 --2- 172.21.15.7:0/218104007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932580 0x55ef4c93b1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.412+0000 7f14bdaed640 1 -- 172.21.15.7:0/218104007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 msgr2=0x55ef4c964d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.412+0000 7f14bdaed640 1 --2- 172.21.15.7:0/218104007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c964d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.412+0000 7f14bdaed640 1 -- 172.21.15.7:0/218104007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4c93de70 con 0x55ef4c932ea0 2024-04-21T08:55:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.412+0000 7f14bdaed640 1 --2- 172.21.15.7:0/218104007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c947790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009ca0 tx=0x7f14b003e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94139464529ab5fc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:40.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.418+0000 7f14a3fff640 1 -- 172.21.15.7:0/218104007 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0039040 con 0x55ef4c932ea0 2024-04-21T08:55:40.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.418+0000 7f14a3fff640 1 -- 172.21.15.7:0/218104007 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b0050070 con 0x55ef4c932ea0 2024-04-21T08:55:40.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.418+0000 7f14a3fff640 1 -- 172.21.15.7:0/218104007 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b003f440 con 0x55ef4c932ea0 2024-04-21T08:55:40.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.419+0000 7f14c3221740 1 -- 172.21.15.7:0/218104007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 msgr2=0x55ef4c947790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:40.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.419+0000 7f14c3221740 1 --2- 172.21.15.7:0/218104007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c947790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009ca0 tx=0x7f14b003e000 comp rx=0 tx=0).stop 2024-04-21T08:55:40.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.439+0000 7f14c3221740 1 -- 172.21.15.7:0/218104007 shutdown_connections 2024-04-21T08:55:40.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.439+0000 7f14c3221740 1 --2- 172.21.15.7:0/218104007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c964d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.439+0000 7f14c3221740 1 --2- 172.21.15.7:0/218104007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932ea0 0x55ef4c947790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.439+0000 7f14c3221740 1 --2- 172.21.15.7:0/218104007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932580 0x55ef4c93b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.439+0000 7f14c3221740 1 -- 172.21.15.7:0/218104007 >> 172.21.15.7:0/218104007 conn(0x55ef4c92ccb0 msgr2=0x55ef4c937380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:40.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.441+0000 7f14c3221740 1 -- 172.21.15.7:0/218104007 shutdown_connections 2024-04-21T08:55:40.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.441+0000 7f14c3221740 1 -- 172.21.15.7:0/218104007 wait complete. 2024-04-21T08:55:40.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.441+0000 7f14c3221740 1 -- start start 2024-04-21T08:55:40.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.442+0000 7f14c3221740 1 Processor -- start 2024-04-21T08:55:40.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.442+0000 7f14c3221740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932580 0x55ef4c9439b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.442+0000 7f14c3221740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932ea0 0x55ef4c943ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.442+0000 7f14c3221740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c93ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.442+0000 7f14c3221740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f149c00f090 con 0x55ef4c93ba80 2024-04-21T08:55:40.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.442+0000 7f14c3221740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f149c010500 con 0x55ef4c932580 2024-04-21T08:55:40.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.442+0000 7f14c3221740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f149c00f760 con 0x55ef4c932ea0 2024-04-21T08:55:40.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.442+0000 7f14c3221740 10 client.0 _ll_register_callbacks cb 0x7f14b40995c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:40.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.443+0000 7f14be2ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932580 0x55ef4c9439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.443+0000 7f14be2ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932580 0x55ef4c9439b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:53432/0 (socket says 172.21.15.7:53432) 2024-04-21T08:55:40.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.443+0000 7f14be2ee640 1 -- 172.21.15.7:0/2559855328 learned_addr learned my addr 172.21.15.7:0/2559855328 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:40.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.443+0000 7f14bdaed640 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932ea0 0x55ef4c943ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.443+0000 7f14beaef640 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c93ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.443+0000 7f14beaef640 1 -- 172.21.15.7:0/2559855328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932ea0 msgr2=0x55ef4c943ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:40.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.443+0000 7f14beaef640 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932ea0 0x55ef4c943ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.443+0000 7f14beaef640 1 -- 172.21.15.7:0/2559855328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932580 msgr2=0x55ef4c9439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:40.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.443+0000 7f14beaef640 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932580 0x55ef4c9439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.443+0000 7f14beaef640 1 -- 172.21.15.7:0/2559855328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef4c93f5b0 con 0x55ef4c93ba80 2024-04-21T08:55:40.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.444+0000 7f14beaef640 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c93ee20 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f14b4099780 tx=0x7f14b409b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870651d143b3ef75 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:40.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.444+0000 7f14a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:55:40.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.444+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2559855328 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40b0060 con 0x55ef4c93ba80 2024-04-21T08:55:40.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.445+0000 7f14c3221740 1 -- 172.21.15.7:0/2559855328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14b0050070 con 0x55ef4c93ba80 2024-04-21T08:55:40.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.445+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2559855328 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7f14b4099070 con 0x55ef4c93ba80 2024-04-21T08:55:40.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.445+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2559855328 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f14b40cef50 con 0x55ef4c93ba80 2024-04-21T08:55:40.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.446+0000 7f14c3221740 10 client.17241 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:40.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.446+0000 7f14c3221740 1 -- 172.21.15.7:0/2559855328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ef4c9c3050 con 0x55ef4c93ba80 2024-04-21T08:55:40.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.447+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2559855328 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b0050070 con 0x55ef4c93ba80 2024-04-21T08:55:40.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.448+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2559855328 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f14b40a1170 con 0x55ef4c93ba80 2024-04-21T08:55:40.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.448+0000 7f14c3221740 10 client.17241 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:40.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.448+0000 7f14c3221740 10 client.17241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:40.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.448+0000 7f14c3221740 20 client.17241 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:40.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.448+0000 7f14c3221740 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ef4c9c3610 0x55ef4c9c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:40.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.448+0000 7f14c3221740 4 client.17241 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:55:40.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.448+0000 7f14c3221740 1 -- 172.21.15.7:0/2559855328 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ef4c89ea70 con 0x55ef4c9c3610 2024-04-21T08:55:40.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.449+0000 7f14be2ee640 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ef4c9c3610 0x55ef4c9c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:40.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.449+0000 7f14be2ee640 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ef4c9c3610 0x55ef4c9c5a30 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73da901254ea65e server_cookie=76cb2825cc46cf15 in_seq=0 out_seq=0 2024-04-21T08:55:40.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.449+0000 7f14a1ffb640 10 client.17241 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:40.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.449+0000 7f14a1ffb640 1 -- 172.21.15.7:0/2559855328 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x55ef4c89ea70 con 0x55ef4c9c3610 2024-04-21T08:55:40.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.449+0000 7f14a1ffb640 10 client.17241 handle_command_reply: tid=0 2024-04-21T08:55:40.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.449+0000 7f14c3221740 1 client.17241 shutdown 2024-04-21T08:55:40.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.452+0000 7f14c3221740 1 -- 172.21.15.7:0/2559855328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 msgr2=0x55ef4c93ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:40.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.452+0000 7f14c3221740 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c93ee20 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f14b4099780 tx=0x7f14b409b830 comp rx=0 tx=0).stop 2024-04-21T08:55:40.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.455+0000 7f14c3221740 1 -- 172.21.15.7:0/2559855328 shutdown_connections 2024-04-21T08:55:40.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.455+0000 7f14c3221740 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55ef4c9c3610 0x55ef4c9c5a30 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.455+0000 7f14c3221740 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ef4c93ba80 0x55ef4c93ee20 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.455+0000 7f14c3221740 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4c932ea0 0x55ef4c943ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.455+0000 7f14c3221740 1 --2- 172.21.15.7:0/2559855328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55ef4c932580 0x55ef4c9439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.455+0000 7f14a1ffb640 0 client.17241 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:40.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.455+0000 7f14c3221740 1 -- 172.21.15.7:0/2559855328 >> 172.21.15.7:0/2559855328 conn(0x55ef4c92ccb0 msgr2=0x55ef4c92e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:40.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.455+0000 7f14c3221740 1 -- 172.21.15.7:0/2559855328 shutdown_connections 2024-04-21T08:55:40.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.455+0000 7f14c3221740 1 -- 172.21.15.7:0/2559855328 wait complete. 2024-04-21T08:55:40.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.455+0000 7f14c3221740 20 client.17241 trim_cache size 0 max 16384 2024-04-21T08:55:40.463 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:55:40.463 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (24451 inodes in the stack)", 2024-04-21T08:55:40.464 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:55:40.464 INFO:teuthology.orchestra.run.smithi007.stdout: "08447483-b5f8-4de7-a923-832d7f72e5ac": { 2024-04-21T08:55:40.464 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:55:40.464 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "08447483-b5f8-4de7-a923-832d7f72e5ac", 2024-04-21T08:55:40.464 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:55:40.464 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:55:40.464 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:55:40.464 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:55:40.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.484+0000 7f14c0552640 1 -- 172.21.15.7:0/196522290 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f14a40a3f20 msgr2=0x7f14a40a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:40.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.484+0000 7f14c0552640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f14a40a3f20 0x7f14a40a63e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f14b0006010 tx=0x7f14b0009820 comp rx=0 tx=0).stop 2024-04-21T08:55:40.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.484+0000 7f14c0552640 1 -- 172.21.15.7:0/196522290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b8113ea0 msgr2=0x7f14b8117750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:40.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.484+0000 7f14c0552640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b8113ea0 0x7f14b8117750 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f14b400c970 tx=0x7f14b400ce30 comp rx=0 tx=0).stop 2024-04-21T08:55:40.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.484+0000 7f14c0552640 1 -- 172.21.15.7:0/196522290 shutdown_connections 2024-04-21T08:55:40.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.484+0000 7f14c0552640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b8113ea0 0x7f14b8117750 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.484+0000 7f14c0552640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14b810b210 0x7f14b8117210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.484+0000 7f14c0552640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f14a40a3f20 0x7f14a40a63e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.484+0000 7f14c0552640 1 --2- 172.21.15.7:0/196522290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14b810a8f0 0x7f14b8116cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:40.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.484+0000 7f14c0552640 1 -- 172.21.15.7:0/196522290 >> 172.21.15.7:0/196522290 conn(0x7f14b806e360 msgr2=0x7f14b8070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:40.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.487+0000 7f14c0552640 1 -- 172.21.15.7:0/196522290 shutdown_connections 2024-04-21T08:55:40.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:55:40.487+0000 7f14c0552640 1 -- 172.21.15.7:0/196522290 wait complete. 2024-04-21T08:55:40.515 INFO:tasks.cephfs.filesystem:scrub status for tag:08447483-b5f8-4de7-a923-832d7f72e5ac - {'path': '/', 'tag': '08447483-b5f8-4de7-a923-832d7f72e5ac', 'options': 'recursive,force'} 2024-04-21T08:55:40.606 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs3acl.o 2024-04-21T08:55:40.665 INFO:tasks.workunit.client.0.smithi007.stdout: AR security/built-in.a 2024-04-21T08:55:40.721 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4proc.o 2024-04-21T08:55:40.825 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/readdir.o 2024-04-21T08:55:40.870 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/kthread.o 2024-04-21T08:55:41.217 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4xdr.o 2024-04-21T08:55:41.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:41 smithi077 ceph-mon[31615]: pgmap v345: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 806 KiB/s wr, 496 op/s 2024-04-21T08:55:41.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:41 smithi007 ceph-mon[28477]: pgmap v345: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 806 KiB/s wr, 496 op/s 2024-04-21T08:55:41.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:41 smithi016 ceph-mon[31259]: pgmap v345: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 806 KiB/s wr, 496 op/s 2024-04-21T08:55:41.628 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/select.o 2024-04-21T08:55:42.167 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/dcache.o 2024-04-21T08:55:42.209 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/inode.o 2024-04-21T08:55:42.376 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4state.o 2024-04-21T08:55:42.524 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/sys_ni.o 2024-04-21T08:55:43.094 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/attr.o 2024-04-21T08:55:43.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:43 smithi077 ceph-mon[31615]: pgmap v346: 129 pgs: 12 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 88 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 758 KiB/s wr, 452 op/s 2024-04-21T08:55:43.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:43 smithi007 ceph-mon[28477]: pgmap v346: 129 pgs: 12 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 88 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 758 KiB/s wr, 452 op/s 2024-04-21T08:55:43.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:43 smithi016 ceph-mon[31259]: pgmap v346: 129 pgs: 12 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 88 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 758 KiB/s wr, 452 op/s 2024-04-21T08:55:43.779 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4renewd.o 2024-04-21T08:55:43.781 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/nsproxy.o 2024-04-21T08:55:44.339 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/bad_inode.o 2024-04-21T08:55:44.434 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4super.o 2024-04-21T08:55:44.571 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4file.o 2024-04-21T08:55:44.919 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/delegation.o 2024-04-21T08:55:45.026 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/notifier.o 2024-04-21T08:55:45.116 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/file.o 2024-04-21T08:55:45.315 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/ksysfs.o 2024-04-21T08:55:45.321 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4idmap.o 2024-04-21T08:55:45.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:45 smithi007 ceph-mon[28477]: pgmap v347: 129 pgs: 11 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 89 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 844 KiB/s wr, 569 op/s 2024-04-21T08:55:45.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:45 smithi016 ceph-mon[31259]: pgmap v347: 129 pgs: 11 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 89 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 844 KiB/s wr, 569 op/s 2024-04-21T08:55:45.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:45 smithi077 ceph-mon[31615]: pgmap v347: 129 pgs: 11 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 89 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 844 KiB/s wr, 569 op/s 2024-04-21T08:55:45.773 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/callback.o 2024-04-21T08:55:45.944 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/callback_xdr.o 2024-04-21T08:55:46.197 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/cred.o 2024-04-21T08:55:46.207 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/reboot.o 2024-04-21T08:55:46.609 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/callback_proc.o 2024-04-21T08:55:47.014 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/filesystems.o 2024-04-21T08:55:47.152 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4namespace.o 2024-04-21T08:55:47.359 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4getroot.o 2024-04-21T08:55:47.390 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4client.o 2024-04-21T08:55:47.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:47 smithi007 ceph-mon[28477]: pgmap v348: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 646 KiB/s wr, 569 op/s 2024-04-21T08:55:47.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:47 smithi016 ceph-mon[31259]: pgmap v348: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 646 KiB/s wr, 569 op/s 2024-04-21T08:55:47.588 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/async.o 2024-04-21T08:55:47.591 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4session.o 2024-04-21T08:55:47.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:47 smithi077 ceph-mon[31615]: pgmap v348: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 646 KiB/s wr, 569 op/s 2024-04-21T08:55:47.645 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/range.o 2024-04-21T08:55:47.931 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/smpboot.o 2024-04-21T08:55:48.196 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/namespace.o 2024-04-21T08:55:48.279 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/ucount.o 2024-04-21T08:55:48.537 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/dns_resolve.o 2024-04-21T08:55:48.689 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4trace.o 2024-04-21T08:55:48.843 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/regset.o 2024-04-21T08:55:48.869 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nfs/nfs4sysctl.o 2024-04-21T08:55:49.085 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/ksyms_common.o 2024-04-21T08:55:49.100 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/groups.o 2024-04-21T08:55:49.146 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/kcmp.o 2024-04-21T08:55:49.449 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/freezer.o 2024-04-21T08:55:49.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:49 smithi016 ceph-mon[31259]: pgmap v349: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 558 KiB/s wr, 676 op/s 2024-04-21T08:55:49.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:49 smithi077 ceph-mon[31615]: pgmap v349: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 558 KiB/s wr, 676 op/s 2024-04-21T08:55:49.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:49 smithi007 ceph-mon[28477]: pgmap v349: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 558 KiB/s wr, 676 op/s 2024-04-21T08:55:49.772 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/profile.o 2024-04-21T08:55:49.893 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/stacktrace.o 2024-04-21T08:55:50.100 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/dma.o 2024-04-21T08:55:50.173 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/smp.o 2024-04-21T08:55:50.193 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/uid16.o 2024-04-21T08:55:50.554 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/seq_file.o 2024-04-21T08:55:50.874 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/xattr.o 2024-04-21T08:55:50.953 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/kallsyms.o 2024-04-21T08:55:50.963 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/acct.o 2024-04-21T08:55:51.466 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/libfs.o 2024-04-21T08:55:51.467 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/crash_core.o 2024-04-21T08:55:51.483 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/kexec_core.o 2024-04-21T08:55:51.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:51 smithi016 ceph-mon[31259]: pgmap v350: 129 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 106 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 403 KiB/s wr, 734 op/s 2024-04-21T08:55:51.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:51 smithi077 ceph-mon[31615]: pgmap v350: 129 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 106 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 403 KiB/s wr, 734 op/s 2024-04-21T08:55:51.656 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:51 smithi007 ceph-mon[28477]: pgmap v350: 129 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 106 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 403 KiB/s wr, 734 op/s 2024-04-21T08:55:51.981 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/kexec.o 2024-04-21T08:55:52.233 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/compat.o 2024-04-21T08:55:52.340 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/nfs/built-in.a 2024-04-21T08:55:52.486 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/utsname.o 2024-04-21T08:55:52.514 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/pid_namespace.o 2024-04-21T08:55:52.534 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fs-writeback.o 2024-04-21T08:55:52.535 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/stop_machine.o 2024-04-21T08:55:53.090 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/pnode.o 2024-04-21T08:55:53.113 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/audit.o 2024-04-21T08:55:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/auditfilter.o 2024-04-21T08:55:53.197 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/auditsc.o 2024-04-21T08:55:53.228 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/audit_watch.o 2024-04-21T08:55:53.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:53 smithi016 ceph-mon[31259]: pgmap v351: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 600 KiB/s wr, 690 op/s 2024-04-21T08:55:53.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:53 smithi077 ceph-mon[31615]: pgmap v351: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 600 KiB/s wr, 690 op/s 2024-04-21T08:55:53.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:53 smithi007 ceph-mon[28477]: pgmap v351: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 600 KiB/s wr, 690 op/s 2024-04-21T08:55:53.716 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/audit_fsnotify.o 2024-04-21T08:55:53.800 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/audit_tree.o 2024-04-21T08:55:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.983+0000 7fdeececa640 1 -- 172.21.15.7:0/4162605832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee8108f50 msgr2=0x7fdee81093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.983+0000 7fdeececa640 1 --2- 172.21.15.7:0/4162605832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee8108f50 0x7fdee81093b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fded8009860 tx=0x7fded802bbb0 comp rx=0 tx=0).stop 2024-04-21T08:55:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.983+0000 7fdeececa640 1 -- 172.21.15.7:0/4162605832 shutdown_connections 2024-04-21T08:55:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.983+0000 7fdeececa640 1 --2- 172.21.15.7:0/4162605832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee81099d0 0x7fdee811d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.983+0000 7fdeececa640 1 --2- 172.21.15.7:0/4162605832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee8108f50 0x7fdee81093b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.983+0000 7fdeececa640 1 --2- 172.21.15.7:0/4162605832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee81104a0 0x7fdee8110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.983+0000 7fdeececa640 1 -- 172.21.15.7:0/4162605832 >> 172.21.15.7:0/4162605832 conn(0x7fdee806ced0 msgr2=0x7fdee806d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.983+0000 7fdeececa640 1 -- 172.21.15.7:0/4162605832 shutdown_connections 2024-04-21T08:55:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.983+0000 7fdeececa640 1 -- 172.21.15.7:0/4162605832 wait complete. 2024-04-21T08:55:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.984+0000 7fdeececa640 1 Processor -- start 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.984+0000 7fdeececa640 1 -- start start 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdeececa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee8108f50 0x7fdee81a73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdeececa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee81099d0 0x7fdee81a7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdeececa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee81104a0 0x7fdee81abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdeececa640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdee8120e50 con 0x7fdee81104a0 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdeececa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdee8120cd0 con 0x7fdee8108f50 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdeececa640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdee8120fd0 con 0x7fdee81099d0 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdee77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee8108f50 0x7fdee81a73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdee77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee8108f50 0x7fdee81a73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39328/0 (socket says 172.21.15.7:39328) 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdee77fe640 1 -- 172.21.15.7:0/1224166826 learned_addr learned my addr 172.21.15.7:0/1224166826 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdee6ffd640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee81099d0 0x7fdee81a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdee6ffd640 1 -- 172.21.15.7:0/1224166826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee8108f50 msgr2=0x7fdee81a73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:53.987 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdee6ffd640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee8108f50 0x7fdee81a73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdee6ffd640 1 -- 172.21.15.7:0/1224166826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee81104a0 msgr2=0x7fdee81abc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdee6ffd640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee81104a0 0x7fdee81abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.985+0000 7fdee6ffd640 1 -- 172.21.15.7:0/1224166826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee81ac3c0 con 0x7fdee81099d0 2024-04-21T08:55:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.986+0000 7fdee6ffd640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee81099d0 0x7fdee81a7910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fded802c1c0 tx=0x7fded8004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:53.991 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.990+0000 7fdee4ff9640 1 -- 172.21.15.7:0/1224166826 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded8039040 con 0x7fdee81099d0 2024-04-21T08:55:53.991 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.990+0000 7fdeececa640 1 -- 172.21.15.7:0/1224166826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee81ac650 con 0x7fdee81099d0 2024-04-21T08:55:53.991 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.990+0000 7fdeececa640 1 -- 172.21.15.7:0/1224166826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee81b3f40 con 0x7fdee81099d0 2024-04-21T08:55:53.991 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.990+0000 7fdee4ff9640 1 -- 172.21.15.7:0/1224166826 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fded802ca00 con 0x7fdee81099d0 2024-04-21T08:55:53.991 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.990+0000 7fdee4ff9640 1 -- 172.21.15.7:0/1224166826 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded80046c0 con 0x7fdee81099d0 2024-04-21T08:55:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.993+0000 7fdef0df1740 1 Processor -- start 2024-04-21T08:55:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.995+0000 7fdee4ff9640 1 -- 172.21.15.7:0/1224166826 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fded8044020 con 0x7fdee81099d0 2024-04-21T08:55:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.996+0000 7fdef0df1740 1 -- start start 2024-04-21T08:55:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.996+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc07132810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.996+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc071331d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.996+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 0x55bc0713e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.996+0000 7fdef0df1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bc070978d0 con 0x55bc07133ac0 2024-04-21T08:55:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.996+0000 7fdef0df1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bc07034e80 con 0x55bc07132d50 2024-04-21T08:55:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.996+0000 7fdef0df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bc07093890 con 0x55bc0713c2d0 2024-04-21T08:55:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.996+0000 7fdee77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc071331d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.996+0000 7fdee77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc071331d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39330/0 (socket says 172.21.15.7:39330) 2024-04-21T08:55:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.996+0000 7fdee77fe640 1 -- 172.21.15.7:0/3242056526 learned_addr learned my addr 172.21.15.7:0/3242056526 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.996+0000 7fdee6ffd640 1 --2- 172.21.15.7:0/3242056526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 0x55bc0713e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.997+0000 7fdee77fe640 1 -- 172.21.15.7:0/3242056526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 msgr2=0x55bc0713e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.997+0000 7fdee77fe640 1 --2- 172.21.15.7:0/3242056526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 0x55bc0713e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.997+0000 7fdee77fe640 1 -- 172.21.15.7:0/3242056526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 msgr2=0x55bc07132810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.997+0000 7fdee77fe640 1 --2- 172.21.15.7:0/3242056526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc07132810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.997+0000 7fdee4ff9640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fded00a3eb0 0x7fded00a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.997+0000 7fdee77fe640 1 -- 172.21.15.7:0/3242056526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc07056320 con 0x55bc07132d50 2024-04-21T08:55:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.997+0000 7fdee4ff9640 1 -- 172.21.15.7:0/1224166826 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7fded80f17d0 con 0x7fdee81099d0 2024-04-21T08:55:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.997+0000 7fdee77fe640 1 --2- 172.21.15.7:0/3242056526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc071331d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdedc00b7e0 tx=0x7fdedc00bca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9745e9fa00f2f6fa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:53.999+0000 7fdee6ffd640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fded00a3eb0 0x7fded00a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.000+0000 7fdecdffb640 1 -- 172.21.15.7:0/3242056526 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc003dd0 con 0x55bc07132d50 2024-04-21T08:55:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.008+0000 7fdee6ffd640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fded00a3eb0 0x7fded00a6370 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fded80eec20 tx=0x7fded80b9040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.009 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.009+0000 7fdecdffb640 1 -- 172.21.15.7:0/3242056526 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdedc015070 con 0x55bc07132d50 2024-04-21T08:55:54.010 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.010+0000 7fdef0df1740 1 -- 172.21.15.7:0/3242056526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 msgr2=0x55bc071331d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.010 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.010+0000 7fdef0df1740 1 --2- 172.21.15.7:0/3242056526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc071331d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdedc00b7e0 tx=0x7fdedc00bca0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.017+0000 7fdef0df1740 1 -- 172.21.15.7:0/3242056526 shutdown_connections 2024-04-21T08:55:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.017+0000 7fdef0df1740 1 --2- 172.21.15.7:0/3242056526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 0x55bc0713e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.017+0000 7fdef0df1740 1 --2- 172.21.15.7:0/3242056526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc071331d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.017+0000 7fdef0df1740 1 --2- 172.21.15.7:0/3242056526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc07132810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.017+0000 7fdef0df1740 1 -- 172.21.15.7:0/3242056526 >> 172.21.15.7:0/3242056526 conn(0x55bc070f9c90 msgr2=0x55bc0712da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.017+0000 7fdef0df1740 1 -- 172.21.15.7:0/3242056526 shutdown_connections 2024-04-21T08:55:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.017+0000 7fdef0df1740 1 -- 172.21.15.7:0/3242056526 wait complete. 2024-04-21T08:55:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.017+0000 7fdef0df1740 1 -- start start 2024-04-21T08:55:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.018+0000 7fdef0df1740 1 Processor -- start 2024-04-21T08:55:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.024+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc07132d50 0x55bc0713bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.024+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc07138f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.024+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc0713c2d0 0x55bc071394c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.024+0000 7fdef0df1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bc0713f050 con 0x55bc07133ac0 2024-04-21T08:55:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.024+0000 7fdef0df1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55bc0701feb0 con 0x55bc0713c2d0 2024-04-21T08:55:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.024+0000 7fdef0df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bc0713f1d0 con 0x55bc07132d50 2024-04-21T08:55:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.024+0000 7fdef0df1740 10 client.0 _ll_register_callbacks cb 0x55bc0704cb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.024+0000 7fdee77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc07138f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.024+0000 7fdee77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc07138f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60370/0 (socket says 172.21.15.7:60370) 2024-04-21T08:55:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.024+0000 7fdee77fe640 1 -- 172.21.15.7:0/1365636728 learned_addr learned my addr 172.21.15.7:0/1365636728 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.027+0000 7fdee7fff640 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc07132d50 0x55bc0713bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.027+0000 7fdee7fff640 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc0713c2d0 0x55bc071394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.027+0000 7fdee7fff640 1 -- 172.21.15.7:0/1365636728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc0713c2d0 msgr2=0x55bc071394c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.027+0000 7fdee7fff640 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc0713c2d0 0x55bc071394c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.027+0000 7fdee7fff640 1 -- 172.21.15.7:0/1365636728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 msgr2=0x55bc07138f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.027+0000 7fdee7fff640 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc07138f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.027+0000 7fdee7fff640 1 -- 172.21.15.7:0/1365636728 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc06daa210 con 0x55bc07132d50 2024-04-21T08:55:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.028+0000 7fdee7fff640 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc07132d50 0x55bc0713bbf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdee0013e50 tx=0x7fdee0003d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22fc76624d8f0583 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.028+0000 7fdecb7f6640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:55:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.028+0000 7fdecb7f6640 1 -- 172.21.15.7:0/1365636728 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee0010040 con 0x55bc07132d50 2024-04-21T08:55:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.028+0000 7fdecb7f6640 1 -- 172.21.15.7:0/1365636728 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7fdee00183f0 con 0x55bc07132d50 2024-04-21T08:55:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.028+0000 7fdef0df1740 1 -- 172.21.15.7:0/1365636728 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdedc015070 con 0x55bc07132d50 2024-04-21T08:55:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.028+0000 7fdecb7f6640 1 -- 172.21.15.7:0/1365636728 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdee0023050 con 0x55bc07132d50 2024-04-21T08:55:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.029+0000 7fdef0df1740 10 client.26890 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.029+0000 7fdef0df1740 1 -- 172.21.15.7:0/1365636728 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bc06e76620 con 0x55bc07132d50 2024-04-21T08:55:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.032+0000 7fdecb7f6640 1 -- 172.21.15.7:0/1365636728 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdedc015070 con 0x55bc07132d50 2024-04-21T08:55:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.032+0000 7fdecb7f6640 1 -- 172.21.15.7:0/1365636728 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fdee00174b0 con 0x55bc07132d50 2024-04-21T08:55:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.032+0000 7fdef0df1740 10 client.26890 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.032+0000 7fdef0df1740 10 client.26890 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.032+0000 7fdef0df1740 20 client.26890 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.032+0000 7fdef0df1740 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bc0715d3b0 0x55bc0715f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.032+0000 7fdef0df1740 4 client.26890 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.032+0000 7fdef0df1740 1 -- 172.21.15.7:0/1365636728 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc070ab880 con 0x55bc0715d3b0 2024-04-21T08:55:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.032+0000 7fdee77fe640 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bc0715d3b0 0x55bc0715f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.033+0000 7fdecb7f6640 10 client.26890 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.033+0000 7fdee77fe640 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bc0715d3b0 0x55bc0715f7c0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61fa2d7057931f1e server_cookie=2fe6954fd70fcd8f in_seq=0 out_seq=0 2024-04-21T08:55:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.034+0000 7fdecb7f6640 1 -- 172.21.15.7:0/1365636728 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc070ab880 con 0x55bc0715d3b0 2024-04-21T08:55:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.034+0000 7fdecb7f6640 10 client.26890 handle_command_reply: tid=0 2024-04-21T08:55:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.040+0000 7fdef0df1740 1 client.26890 shutdown 2024-04-21T08:55:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdef0df1740 1 -- 172.21.15.7:0/1365636728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc07132d50 msgr2=0x55bc0713bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdef0df1740 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc07132d50 0x55bc0713bbf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdee0013e50 tx=0x7fdee0003d80 comp rx=0 tx=0).stop 2024-04-21T08:55:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdef0df1740 1 -- 172.21.15.7:0/1365636728 shutdown_connections 2024-04-21T08:55:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdef0df1740 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bc0715d3b0 0x55bc0715f7c0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdef0df1740 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc0713c2d0 0x55bc071394c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdef0df1740 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc07138f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdef0df1740 1 --2- 172.21.15.7:0/1365636728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc07132d50 0x55bc0713bbf0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdecb7f6640 0 client.26890 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdef0df1740 1 -- 172.21.15.7:0/1365636728 >> 172.21.15.7:0/1365636728 conn(0x55bc070f9c90 msgr2=0x55bc0712f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdef0df1740 1 -- 172.21.15.7:0/1365636728 shutdown_connections 2024-04-21T08:55:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdef0df1740 1 -- 172.21.15.7:0/1365636728 wait complete. 2024-04-21T08:55:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.041+0000 7fdef0df1740 20 client.26890 trim_cache size 0 max 16384 2024-04-21T08:55:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.115+0000 7fdef0df1740 1 Processor -- start 2024-04-21T08:55:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.120+0000 7fdef0df1740 1 -- start start 2024-04-21T08:55:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.120+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc07147fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.120+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc0712c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.121+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 0x55bc0713b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.121+0000 7fdef0df1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fded400e3b0 con 0x55bc07133ac0 2024-04-21T08:55:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.121+0000 7fdef0df1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fded4010c10 con 0x55bc07132d50 2024-04-21T08:55:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.121+0000 7fdef0df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fded400ea00 con 0x55bc0713c2d0 2024-04-21T08:55:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.122+0000 7fdee77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc0712c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.122+0000 7fdee77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc0712c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60382/0 (socket says 172.21.15.7:60382) 2024-04-21T08:55:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.122+0000 7fdee77fe640 1 -- 172.21.15.7:0/2804093384 learned_addr learned my addr 172.21.15.7:0/2804093384 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.123+0000 7fdee7fff640 1 --2- 172.21.15.7:0/2804093384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc07147fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.123+0000 7fdee7fff640 1 -- 172.21.15.7:0/2804093384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 msgr2=0x55bc0713b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.123+0000 7fdee7fff640 1 --2- 172.21.15.7:0/2804093384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 0x55bc0713b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.123+0000 7fdee7fff640 1 -- 172.21.15.7:0/2804093384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 msgr2=0x55bc0712c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.123+0000 7fdee7fff640 1 --2- 172.21.15.7:0/2804093384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc0712c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.123+0000 7fdee7fff640 1 -- 172.21.15.7:0/2804093384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc07132020 con 0x55bc07132d50 2024-04-21T08:55:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.123+0000 7fdee7fff640 1 --2- 172.21.15.7:0/2804093384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc07147fd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdee00135c0 tx=0x7fdee0013cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3407e4de9466bc7c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.135+0000 7fdecd7fa640 1 -- 172.21.15.7:0/2804093384 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee0010040 con 0x55bc07132d50 2024-04-21T08:55:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.135+0000 7fdecd7fa640 1 -- 172.21.15.7:0/2804093384 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdee000e030 con 0x55bc07132d50 2024-04-21T08:55:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.135+0000 7fdecd7fa640 1 -- 172.21.15.7:0/2804093384 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee00173a0 con 0x55bc07132d50 2024-04-21T08:55:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.137+0000 7fdef0df1740 1 -- 172.21.15.7:0/2804093384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 msgr2=0x55bc07147fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.137+0000 7fdef0df1740 1 --2- 172.21.15.7:0/2804093384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc07147fd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdee00135c0 tx=0x7fdee0013cc0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.138+0000 7fdef0df1740 1 -- 172.21.15.7:0/2804093384 shutdown_connections 2024-04-21T08:55:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.138+0000 7fdef0df1740 1 --2- 172.21.15.7:0/2804093384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 0x55bc0713b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.138+0000 7fdef0df1740 1 --2- 172.21.15.7:0/2804093384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc0712c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.138+0000 7fdef0df1740 1 --2- 172.21.15.7:0/2804093384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc07147fd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.138+0000 7fdef0df1740 1 -- 172.21.15.7:0/2804093384 >> 172.21.15.7:0/2804093384 conn(0x55bc070f9c90 msgr2=0x55bc0712c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.138+0000 7fdef0df1740 1 -- 172.21.15.7:0/2804093384 shutdown_connections 2024-04-21T08:55:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.147+0000 7fdef0df1740 1 -- 172.21.15.7:0/2804093384 wait complete. 2024-04-21T08:55:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdef0df1740 1 -- start start 2024-04-21T08:55:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdef0df1740 1 Processor -- start 2024-04-21T08:55:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc0713cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc0713d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdef0df1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 0x55bc071434e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdef0df1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fded400f0d0 con 0x55bc07133ac0 2024-04-21T08:55:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdef0df1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fded4010540 con 0x55bc07132d50 2024-04-21T08:55:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdef0df1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fded400f7a0 con 0x55bc0713c2d0 2024-04-21T08:55:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdef0df1740 10 client.0 _ll_register_callbacks cb 0x7fdee00045f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdee7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc0713cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdee7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc0713cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:39372/0 (socket says 172.21.15.7:39372) 2024-04-21T08:55:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.148+0000 7fdee7fff640 1 -- 172.21.15.7:0/3421035471 learned_addr learned my addr 172.21.15.7:0/3421035471 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:55:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.149+0000 7fdee7fff640 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 0x55bc071434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.149+0000 7fdee7fff640 1 -- 172.21.15.7:0/3421035471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 msgr2=0x55bc071434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.149+0000 7fdee7fff640 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 0x55bc071434e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.149+0000 7fdee7fff640 1 -- 172.21.15.7:0/3421035471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 msgr2=0x55bc0713d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.149+0000 7fdee7fff640 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc0713d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.149+0000 7fdee7fff640 1 -- 172.21.15.7:0/3421035471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc07143a50 con 0x55bc07132d50 2024-04-21T08:55:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.149+0000 7fdee7fff640 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc0713cac0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdee0013a10 tx=0x7fdee0015ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=899496f34e3453e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.151 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/splice.o 2024-04-21T08:55:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.156+0000 7fdecb7f6640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:55:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.156+0000 7fdecb7f6640 1 -- 172.21.15.7:0/3421035471 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee00046d0 con 0x55bc07132d50 2024-04-21T08:55:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.156+0000 7fdef0df1740 1 -- 172.21.15.7:0/3421035471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdee000e030 con 0x55bc07132d50 2024-04-21T08:55:54.157 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.156+0000 7fdecb7f6640 1 -- 172.21.15.7:0/3421035471 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9732+0+0 (secure 0 0 0) 0x7fdee0027430 con 0x55bc07132d50 2024-04-21T08:55:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.167+0000 7fdecb7f6640 1 -- 172.21.15.7:0/3421035471 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdee0004870 con 0x55bc07132d50 2024-04-21T08:55:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.167+0000 7fdef0df1740 10 client.26912 fetch_fsmap learned FSMap version 23 2024-04-21T08:55:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.167+0000 7fdef0df1740 1 -- 172.21.15.7:0/3421035471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bc07144d60 con 0x55bc07132d50 2024-04-21T08:55:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.167+0000 7fdecb7f6640 1 -- 172.21.15.7:0/3421035471 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdee000e030 con 0x55bc07132d50 2024-04-21T08:55:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.167+0000 7fdecb7f6640 1 -- 172.21.15.7:0/3421035471 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fdee003d070 con 0x55bc07132d50 2024-04-21T08:55:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.167+0000 7fdef0df1740 10 client.26912 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:55:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.167+0000 7fdef0df1740 10 client.26912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.167+0000 7fdef0df1740 20 client.26912 populate_metadata read hostname 'smithi007' 2024-04-21T08:55:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.167+0000 7fdef0df1740 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bc071c3d10 0x55bc071c6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.167+0000 7fdef0df1740 4 client.26912 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.167+0000 7fdef0df1740 1 -- 172.21.15.7:0/3421035471 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bc070373d0 con 0x55bc071c3d10 2024-04-21T08:55:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.168+0000 7fdee77fe640 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bc071c3d10 0x55bc071c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.168+0000 7fdecb7f6640 10 client.26912 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:54.170 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:54.168+0000 7fdee77fe640 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bc071c3d10 0x55bc071c6130 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a7b2e1b3035d074 server_cookie=d9dad45ff3f1f3b4 in_seq=0 out_seq=0 2024-04-21T08:55:54.655 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/kprobes.o 2024-04-21T08:55:54.939 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/seccomp.o 2024-04-21T08:55:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdecb7f6640 1 -- 172.21.15.7:0/3421035471 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bc070373d0 con 0x55bc071c3d10 2024-04-21T08:55:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdecb7f6640 10 client.26912 handle_command_reply: tid=0 2024-04-21T08:55:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdef0df1740 1 client.26912 shutdown 2024-04-21T08:55:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdef0df1740 1 -- 172.21.15.7:0/3421035471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 msgr2=0x55bc0713cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdef0df1740 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc0713cac0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdee0013a10 tx=0x7fdee0015ec0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdef0df1740 1 -- 172.21.15.7:0/3421035471 shutdown_connections 2024-04-21T08:55:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdef0df1740 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55bc071c3d10 0x55bc071c6130 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdef0df1740 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0713c2d0 0x55bc071434e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdecb7f6640 0 client.26912 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:55:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdef0df1740 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc07133ac0 0x55bc0713d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdef0df1740 1 --2- 172.21.15.7:0/3421035471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55bc07132d50 0x55bc0713cac0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.041+0000 7fdef0df1740 1 -- 172.21.15.7:0/3421035471 >> 172.21.15.7:0/3421035471 conn(0x55bc070f9c90 msgr2=0x55bc0712cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.043+0000 7fdef0df1740 1 -- 172.21.15.7:0/3421035471 shutdown_connections 2024-04-21T08:55:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.044+0000 7fdef0df1740 1 -- 172.21.15.7:0/3421035471 wait complete. 2024-04-21T08:55:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.044+0000 7fdef0df1740 20 client.26912 trim_cache size 0 max 16384 2024-04-21T08:55:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:{ 2024-04-21T08:55:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "message": "", 2024-04-21T08:55:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout: "return_code": 0 2024-04-21T08:55:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stdout:} 2024-04-21T08:55:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.066+0000 7fdeececa640 1 -- 172.21.15.7:0/1224166826 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fded00a3eb0 msgr2=0x7fded00a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.066+0000 7fdeececa640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fded00a3eb0 0x7fded00a6370 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fded80eec20 tx=0x7fded80b9040 comp rx=0 tx=0).stop 2024-04-21T08:55:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.066+0000 7fdeececa640 1 -- 172.21.15.7:0/1224166826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee81099d0 msgr2=0x7fdee81a7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.066+0000 7fdeececa640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee81099d0 0x7fdee81a7910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fded802c1c0 tx=0x7fded8004270 comp rx=0 tx=0).stop 2024-04-21T08:55:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.066+0000 7fdeececa640 1 -- 172.21.15.7:0/1224166826 shutdown_connections 2024-04-21T08:55:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.066+0000 7fdeececa640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee81104a0 0x7fdee81abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.066+0000 7fdeececa640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fded00a3eb0 0x7fded00a6370 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.066+0000 7fdeececa640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdee81099d0 0x7fdee81a7910 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.066+0000 7fdeececa640 1 --2- 172.21.15.7:0/1224166826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdee8108f50 0x7fdee81a73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.066+0000 7fdeececa640 1 -- 172.21.15.7:0/1224166826 >> 172.21.15.7:0/1224166826 conn(0x7fdee806ced0 msgr2=0x7fdee8070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.073 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.073+0000 7fdeececa640 1 -- 172.21.15.7:0/1224166826 shutdown_connections 2024-04-21T08:55:55.073 INFO:teuthology.task.background_exec.ubuntu@smithi007.front.sepia.ceph.com.smithi007.stderr:2024-04-21T08:55:55.073+0000 7fdeececa640 1 -- 172.21.15.7:0/1224166826 wait complete. 2024-04-21T08:55:55.165 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/relay.o 2024-04-21T08:55:55.393 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/utsname_sysctl.o 2024-04-21T08:55:55.402 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/sync.o 2024-04-21T08:55:55.441 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/delayacct.o 2024-04-21T08:55:55.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:55 smithi016 ceph-mon[31259]: pgmap v352: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 780 KiB/s wr, 801 op/s 2024-04-21T08:55:55.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:55 smithi077 ceph-mon[31615]: pgmap v352: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 780 KiB/s wr, 801 op/s 2024-04-21T08:55:55.611 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/taskstats.o 2024-04-21T08:55:55.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:55 smithi007 ceph-mon[28477]: pgmap v352: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 780 KiB/s wr, 801 op/s 2024-04-21T08:55:55.880 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/tsacct.o 2024-04-21T08:55:56.130 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/tracepoint.o 2024-04-21T08:55:56.182 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/utimes.o 2024-04-21T08:55:56.458 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/irq_work.o 2024-04-21T08:55:56.512 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/d_path.o 2024-04-21T08:55:56.632 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/static_call.o 2024-04-21T08:55:56.691 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/static_call_inline.o 2024-04-21T08:55:56.838 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/crash_dump.o 2024-04-21T08:55:56.862 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/jump_label.o 2024-04-21T08:55:57.142 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/context_tracking.o 2024-04-21T08:55:57.230 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/iomem.o 2024-04-21T08:55:57.249 INFO:tasks.workunit.client.0.smithi007.stdout: CC kernel/rseq.o 2024-04-21T08:55:57.453 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/stack.o 2024-04-21T08:55:57.456 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fs_struct.o 2024-04-21T08:55:57.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:57 smithi016 ceph-mon[31259]: pgmap v353: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 693 KiB/s wr, 796 op/s 2024-04-21T08:55:57.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:57 smithi077 ceph-mon[31615]: pgmap v353: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 693 KiB/s wr, 796 op/s 2024-04-21T08:55:57.608 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/statfs.o 2024-04-21T08:55:57.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:57 smithi007 ceph-mon[28477]: pgmap v353: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 693 KiB/s wr, 796 op/s 2024-04-21T08:55:57.669 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fs_pin.o 2024-04-21T08:55:57.932 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/nsfs.o 2024-04-21T08:55:58.034 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fs_types.o 2024-04-21T08:55:58.132 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fs_context.o 2024-04-21T08:55:58.135 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fs_parser.o 2024-04-21T08:55:58.318 INFO:tasks.workunit.client.0.smithi007.stdout: AR kernel/built-in.a 2024-04-21T08:55:58.321 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fsopen.o 2024-04-21T08:55:58.465 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/init.o 2024-04-21T08:55:58.558 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/kernel_read_file.o 2024-04-21T08:55:58.697 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/mnt_idmapping.o 2024-04-21T08:55:58.713 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/remap_range.o 2024-04-21T08:55:58.790 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/buffer.o 2024-04-21T08:55:59.080 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/mpage.o 2024-04-21T08:55:59.130 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/proc_namespace.o 2024-04-21T08:55:59.195 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/direct-io.o 2024-04-21T08:55:59.373 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/eventpoll.o 2024-04-21T08:55:59.397 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/anon_inodes.o 2024-04-21T08:55:59.417 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/signalfd.o 2024-04-21T08:55:59.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:55:59 smithi016 ceph-mon[31259]: pgmap v354: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 881 KiB/s wr, 995 op/s 2024-04-21T08:55:59.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:55:59 smithi077 ceph-mon[31615]: pgmap v354: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 881 KiB/s wr, 995 op/s 2024-04-21T08:55:59.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:55:59 smithi007 ceph-mon[28477]: pgmap v354: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 881 KiB/s wr, 995 op/s 2024-04-21T08:55:59.885 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/timerfd.o 2024-04-21T08:55:59.905 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/eventfd.o 2024-04-21T08:56:00.266 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/aio.o 2024-04-21T08:56:00.385 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/locks.o 2024-04-21T08:56:00.448 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/binfmt_misc.o 2024-04-21T08:56:00.448 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/binfmt_script.o 2024-04-21T08:56:00.929 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/binfmt_elf.o 2024-04-21T08:56:01.173 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/compat_binfmt_elf.o 2024-04-21T08:56:01.276 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/mbcache.o 2024-04-21T08:56:01.279 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/posix_acl.o 2024-04-21T08:56:01.361 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/coredump.o 2024-04-21T08:56:01.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:01 smithi016 ceph-mon[31259]: pgmap v355: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 842 KiB/s wr, 1.17k op/s 2024-04-21T08:56:01.597 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/drop_caches.o 2024-04-21T08:56:01.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:01 smithi077 ceph-mon[31615]: pgmap v355: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 842 KiB/s wr, 1.17k op/s 2024-04-21T08:56:01.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:01 smithi007 ceph-mon[28477]: pgmap v355: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 842 KiB/s wr, 1.17k op/s 2024-04-21T08:56:02.088 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/sysctls.o 2024-04-21T08:56:02.126 INFO:tasks.workunit.client.0.smithi007.stdout: CC fs/fhandle.o 2024-04-21T08:56:02.784 INFO:tasks.workunit.client.0.smithi007.stdout: AR fs/built-in.a 2024-04-21T08:56:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-21T08:56:02.902 INFO:tasks.workunit.client.0.smithi007.stderr:make[1]: *** [/home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/Makefile:2034: .] Error 2 2024-04-21T08:56:02.902 INFO:tasks.workunit.client.0.smithi007.stderr:make: *** [Makefile:234: __sub-make] Error 2 2024-04-21T08:56:02.902 INFO:tasks.workunit:Stopping ['kernel_untar_build.sh'] on client.0... 2024-04-21T08:56:02.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-21T08:56:03.374 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:03 smithi007 ceph-mon[28477]: pgmap v356: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 736 KiB/s wr, 1.43k op/s 2024-04-21T08:56:03.376 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test kernel_untar_build.sh) on smithi007 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2024-04-21T08:56:03.645 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0bf063190c9c4617b0ca798cd5eef841 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test kernel_untar_build.sh) on smithi007 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2024-04-21T08:56:03.648 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-21T08:56:03.661 INFO:tasks.quiescer:joining Quiescers 2024-04-21T08:56:03.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:56:03.666 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:03 smithi077 ceph-mon[31615]: pgmap v356: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 736 KiB/s wr, 1.43k op/s 2024-04-21T08:56:03.667 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:03 smithi016 ceph-mon[31259]: pgmap v356: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 736 KiB/s wr, 1.43k op/s 2024-04-21T08:56:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.910+0000 7f54bbfff640 1 -- 172.21.15.7:0/2090955307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b40957e0 msgr2=0x7f54b40a1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.910+0000 7f54bbfff640 1 --2- 172.21.15.7:0/2090955307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b40957e0 0x7f54b40a1ab0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f54ac01a940 tx=0x7f54ac03ee90 comp rx=0 tx=0).stop 2024-04-21T08:56:03.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.911+0000 7f54bbfff640 1 -- 172.21.15.7:0/2090955307 shutdown_connections 2024-04-21T08:56:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.911+0000 7f54bbfff640 1 --2- 172.21.15.7:0/2090955307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b40957e0 0x7f54b40a1ab0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.911+0000 7f54bbfff640 1 --2- 172.21.15.7:0/2090955307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54b4094e40 0x7f54b40952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.911+0000 7f54bbfff640 1 --2- 172.21.15.7:0/2090955307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54b4099bb0 0x7f54b4099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.911+0000 7f54bbfff640 1 -- 172.21.15.7:0/2090955307 >> 172.21.15.7:0/2090955307 conn(0x7f54b400a730 msgr2=0x7f54b400ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:03.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.917+0000 7f54bbfff640 1 -- 172.21.15.7:0/2090955307 shutdown_connections 2024-04-21T08:56:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.919+0000 7f54bbfff640 1 -- 172.21.15.7:0/2090955307 wait complete. 2024-04-21T08:56:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.920+0000 7f54bbfff640 1 Processor -- start 2024-04-21T08:56:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.920+0000 7f54bbfff640 1 -- start start 2024-04-21T08:56:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.920+0000 7f54bbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4094e40 0x7f54b412a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.921+0000 7f54bbfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54b40957e0 0x7f54b412a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.921+0000 7f54bbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54b4099bb0 0x7f54b412ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.921+0000 7f54bbfff640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f54b40a3e20 con 0x7f54b40957e0 2024-04-21T08:56:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.921+0000 7f54bbfff640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54b40a3ca0 con 0x7f54b4094e40 2024-04-21T08:56:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.921+0000 7f54bbfff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f54b40a3fa0 con 0x7f54b4099bb0 2024-04-21T08:56:03.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.921+0000 7f54bb7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54b4099bb0 0x7f54b412ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:03.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.921+0000 7f54ba7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54b40957e0 0x7f54b412a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:03.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.921+0000 7f54ba7fc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54b40957e0 0x7f54b412a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35694/0 (socket says 172.21.15.7:35694) 2024-04-21T08:56:03.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.921+0000 7f54ba7fc640 1 -- 172.21.15.7:0/1852580690 learned_addr learned my addr 172.21.15.7:0/1852580690 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:03.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.922+0000 7f54baffd640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4094e40 0x7f54b412a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:03.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.922+0000 7f54ba7fc640 1 -- 172.21.15.7:0/1852580690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54b4099bb0 msgr2=0x7f54b412ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:03.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.922+0000 7f54ba7fc640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54b4099bb0 0x7f54b412ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:03.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.922+0000 7f54ba7fc640 1 -- 172.21.15.7:0/1852580690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4094e40 msgr2=0x7f54b412a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:03.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.922+0000 7f54ba7fc640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4094e40 0x7f54b412a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:03.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.922+0000 7f54ba7fc640 1 -- 172.21.15.7:0/1852580690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b412f3b0 con 0x7f54b40957e0 2024-04-21T08:56:03.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.922+0000 7f54ba7fc640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54b40957e0 0x7f54b412a900 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f54bc065470 tx=0x7f54bc070820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:03.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.924+0000 7f54a3fff640 1 -- 172.21.15.7:0/1852580690 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc075590 con 0x7f54b40957e0 2024-04-21T08:56:03.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.924+0000 7f54a3fff640 1 -- 172.21.15.7:0/1852580690 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54bc075be0 con 0x7f54b40957e0 2024-04-21T08:56:03.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.924+0000 7f54a3fff640 1 -- 172.21.15.7:0/1852580690 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc076d40 con 0x7f54b40957e0 2024-04-21T08:56:03.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.924+0000 7f54bbfff640 1 -- 172.21.15.7:0/1852580690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b412f6a0 con 0x7f54b40957e0 2024-04-21T08:56:03.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.925+0000 7f54bbfff640 1 -- 172.21.15.7:0/1852580690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b4158f40 con 0x7f54b40957e0 2024-04-21T08:56:03.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.928+0000 7f54bbfff640 1 -- 172.21.15.7:0/1852580690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54b40061b0 con 0x7f54b40957e0 2024-04-21T08:56:03.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.931+0000 7f54a3fff640 1 -- 172.21.15.7:0/1852580690 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f54bc076420 con 0x7f54b40957e0 2024-04-21T08:56:03.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.932+0000 7f54a3fff640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f54a40a3f20 0x7f54a40a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:03.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.932+0000 7f54a3fff640 1 -- 172.21.15.7:0/1852580690 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9764+0+0 (secure 0 0 0) 0x7f54bc133260 con 0x7f54b40957e0 2024-04-21T08:56:03.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.940+0000 7f54a3fff640 1 -- 172.21.15.7:0/1852580690 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f54bc0ff4d0 con 0x7f54b40957e0 2024-04-21T08:56:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.940+0000 7f54baffd640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f54a40a3f20 0x7f54a40a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:03.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:03.958+0000 7f54baffd640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f54a40a3f20 0x7f54a40a63e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54b000a7d0 tx=0x7f54b0008040 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:04.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.414+0000 7f54bbfff640 1 -- 172.21.15.7:0/1852580690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54b40022f0 con 0x7f54b40957e0 2024-04-21T08:56:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.415+0000 7f54a3fff640 1 -- 172.21.15.7:0/1852580690 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7f54bc0fedd0 con 0x7f54b40957e0 2024-04-21T08:56:04.416 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:56:04.416 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:56:04.416 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:56:04.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 -- 172.21.15.7:0/1852580690 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f54a40a3f20 msgr2=0x7f54a40a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:04.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f54a40a3f20 0x7f54a40a63e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54b000a7d0 tx=0x7f54b0008040 comp rx=0 tx=0).stop 2024-04-21T08:56:04.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 -- 172.21.15.7:0/1852580690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54b40957e0 msgr2=0x7f54b412a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:04.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54b40957e0 0x7f54b412a900 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f54bc065470 tx=0x7f54bc070820 comp rx=0 tx=0).stop 2024-04-21T08:56:04.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 -- 172.21.15.7:0/1852580690 shutdown_connections 2024-04-21T08:56:04.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f54a40a3f20 0x7f54a40a63e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54b4099bb0 0x7f54b412ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54b40957e0 0x7f54b412a900 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 --2- 172.21.15.7:0/1852580690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4094e40 0x7f54b412a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 -- 172.21.15.7:0/1852580690 >> 172.21.15.7:0/1852580690 conn(0x7f54b400a730 msgr2=0x7f54b4092d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:04.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 -- 172.21.15.7:0/1852580690 shutdown_connections 2024-04-21T08:56:04.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.419+0000 7f54bbfff640 1 -- 172.21.15.7:0/1852580690 wait complete. 2024-04-21T08:56:04.440 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --cancel --all' 2024-04-21T08:56:04.441 DEBUG:teuthology.orchestra.run.smithi007:> timeout 45 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24533 quiesce db --cancel --all 2024-04-21T08:56:04.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:04 smithi016 ceph-mon[31259]: osdmap e109: 12 total, 12 up, 12 in 2024-04-21T08:56:04.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:04 smithi077 ceph-mon[31615]: osdmap e109: 12 total, 12 up, 12 in 2024-04-21T08:56:04.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.645+0000 7f6fb5df4640 1 -- 172.21.15.7:0/1652899857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb0110b10 msgr2=0x7f6fb010e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:04.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.645+0000 7f6fb5df4640 1 --2- 172.21.15.7:0/1652899857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb0110b10 0x7f6fb010e7a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6fa001a910 tx=0x7f6fa003cca0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.647+0000 7f6fb5df4640 1 -- 172.21.15.7:0/1652899857 shutdown_connections 2024-04-21T08:56:04.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.647+0000 7f6fb5df4640 1 --2- 172.21.15.7:0/1652899857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fb011b730 0x7f6fb011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.647+0000 7f6fb5df4640 1 --2- 172.21.15.7:0/1652899857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb0110b10 0x7f6fb010e7a0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.647+0000 7f6fb5df4640 1 --2- 172.21.15.7:0/1652899857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb01106a0 0x7f6fb010e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.647+0000 7f6fb5df4640 1 -- 172.21.15.7:0/1652899857 >> 172.21.15.7:0/1652899857 conn(0x7f6fb006cff0 msgr2=0x7f6fb006d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:04.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:04 smithi007 ceph-mon[28477]: osdmap e109: 12 total, 12 up, 12 in 2024-04-21T08:56:04.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.653+0000 7f6fb5df4640 1 -- 172.21.15.7:0/1652899857 shutdown_connections 2024-04-21T08:56:04.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.654+0000 7f6fb5df4640 1 -- 172.21.15.7:0/1652899857 wait complete. 2024-04-21T08:56:04.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.655+0000 7f6fb5df4640 1 Processor -- start 2024-04-21T08:56:04.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.655+0000 7f6fb5df4640 1 -- start start 2024-04-21T08:56:04.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.655+0000 7f6fb5df4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb01106a0 0x7f6fb011af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.656+0000 7f6fb5df4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb0110b10 0x7f6fb0116fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.656+0000 7f6faf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb01106a0 0x7f6fb011af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.656+0000 7f6faf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb01106a0 0x7f6fb011af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:41920/0 (socket says 172.21.15.7:41920) 2024-04-21T08:56:04.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.656+0000 7f6faf7fe640 1 -- 172.21.15.7:0/1003572969 learned_addr learned my addr 172.21.15.7:0/1003572969 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:04.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.656+0000 7f6fb5df4640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fb011b730 0x7f6fb01174f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.656+0000 7f6fb5df4640 1 -- 172.21.15.7:0/1003572969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb0113880 con 0x7f6fb0110b10 2024-04-21T08:56:04.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.656+0000 7f6fb5df4640 1 -- 172.21.15.7:0/1003572969 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb0113700 con 0x7f6fb011b730 2024-04-21T08:56:04.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.656+0000 7f6faffff640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fb011b730 0x7f6fb01174f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.657+0000 7f6fb5df4640 1 -- 172.21.15.7:0/1003572969 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb0113a00 con 0x7f6fb01106a0 2024-04-21T08:56:04.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.659+0000 7f6faeffd640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb0110b10 0x7f6fb0116fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.660+0000 7f6faeffd640 1 -- 172.21.15.7:0/1003572969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb01106a0 msgr2=0x7f6fb011af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:04.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.660+0000 7f6faeffd640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb01106a0 0x7f6fb011af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.660+0000 7f6faeffd640 1 -- 172.21.15.7:0/1003572969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fb011b730 msgr2=0x7f6fb01174f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:04.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.660+0000 7f6faeffd640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fb011b730 0x7f6fb01174f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.660+0000 7f6faeffd640 1 -- 172.21.15.7:0/1003572969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fb0117c70 con 0x7f6fb0110b10 2024-04-21T08:56:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.661+0000 7f6faeffd640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb0110b10 0x7f6fb0116fb0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f6fa003d2b0 tx=0x7f6fa001a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.664+0000 7f6facff9640 1 -- 172.21.15.7:0/1003572969 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0052020 con 0x7f6fb0110b10 2024-04-21T08:56:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.664+0000 7f6facff9640 1 -- 172.21.15.7:0/1003572969 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fa003dd00 con 0x7f6fb0110b10 2024-04-21T08:56:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.664+0000 7f6facff9640 1 -- 172.21.15.7:0/1003572969 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0003d00 con 0x7f6fb0110b10 2024-04-21T08:56:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.664+0000 7f6faffff640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fb011b730 0x7f6fb01174f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:56:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.664+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb01106a0 0x7f6fb011af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:56:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.664+0000 7f6fb5df4640 1 -- 172.21.15.7:0/1003572969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fb01ab4d0 con 0x7f6fb0110b10 2024-04-21T08:56:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.664+0000 7f6fb5df4640 1 -- 172.21.15.7:0/1003572969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fb01ab940 con 0x7f6fb0110b10 2024-04-21T08:56:04.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.668+0000 7f6fb89dd740 1 Processor -- start 2024-04-21T08:56:04.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.668+0000 7f6fb89dd740 1 -- start start 2024-04-21T08:56:04.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.668+0000 7f6fb89dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630bc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.668+0000 7f6fb89dd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630bc9f0 0x55a8630c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.668+0000 7f6fb89dd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a8630c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.669+0000 7f6fb89dd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a863021000 con 0x55a8630bc9f0 2024-04-21T08:56:04.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.669+0000 7f6fb89dd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a862fbdb20 con 0x55a8630c5570 2024-04-21T08:56:04.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.669+0000 7f6fb89dd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a86301cfc0 con 0x55a8630bc040 2024-04-21T08:56:04.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.670+0000 7f6faf7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630bc9f0 0x55a8630c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.670+0000 7f6faf7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630bc9f0 0x55a8630c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42532/0 (socket says 172.21.15.7:42532) 2024-04-21T08:56:04.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.670+0000 7f6faf7fe640 1 -- 172.21.15.7:0/464724036 learned_addr learned my addr 172.21.15.7:0/464724036 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:04.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.670+0000 7f6faeffd640 1 --2- 172.21.15.7:0/464724036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a8630c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.672+0000 7f6faf7fe640 1 -- 172.21.15.7:0/464724036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 msgr2=0x55a8630bc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:04.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.672+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/464724036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630bc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.672+0000 7f6faf7fe640 1 -- 172.21.15.7:0/464724036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 msgr2=0x55a8630c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:04.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.672+0000 7f6facff9640 1 -- 172.21.15.7:0/1003572969 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f6fa0055020 con 0x7f6fb0110b10 2024-04-21T08:56:04.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.672+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/464724036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a8630c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.672+0000 7f6faf7fe640 1 -- 172.21.15.7:0/464724036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a862fd7ea0 con 0x55a8630bc9f0 2024-04-21T08:56:04.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.673+0000 7f6facff9640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f980a3ef0 0x7f6f980a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.674+0000 7f6faeffd640 1 --2- 172.21.15.7:0/464724036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a8630c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:56:04.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.674+0000 7f6facff9640 1 -- 172.21.15.7:0/1003572969 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9764+0+0 (secure 0 0 0) 0x7f6fa01033c0 con 0x7f6fb0110b10 2024-04-21T08:56:04.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.674+0000 7f6faffff640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f980a3ef0 0x7f6f980a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.678+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/464724036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630bc9f0 0x55a8630c4fa0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400aa70 tx=0x7f6fa400af30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5cf04f15d6e8c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:04.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.681+0000 7f6faffff640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f980a3ef0 0x7f6f980a63b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6fb006b2d0 tx=0x7f6fa8008330 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:04.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.681+0000 7f6f957fa640 1 -- 172.21.15.7:0/464724036 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4015070 con 0x55a8630bc9f0 2024-04-21T08:56:04.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.681+0000 7f6f957fa640 1 -- 172.21.15.7:0/464724036 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fa400ccc0 con 0x55a8630bc9f0 2024-04-21T08:56:04.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.682+0000 7f6fb89dd740 1 -- 172.21.15.7:0/464724036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630bc9f0 msgr2=0x55a8630c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:04.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.682+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/464724036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630bc9f0 0x55a8630c4fa0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400aa70 tx=0x7f6fa400af30 comp rx=0 tx=0).stop 2024-04-21T08:56:04.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.683+0000 7f6fb89dd740 1 -- 172.21.15.7:0/464724036 shutdown_connections 2024-04-21T08:56:04.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.683+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/464724036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a8630c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.683+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/464724036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630bc9f0 0x55a8630c4fa0 secure :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400aa70 tx=0x7f6fa400af30 comp rx=0 tx=0).stop 2024-04-21T08:56:04.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.683+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/464724036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630bc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.683+0000 7f6fb89dd740 1 -- 172.21.15.7:0/464724036 >> 172.21.15.7:0/464724036 conn(0x55a8630b6710 msgr2=0x55a8630b6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:04.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.687+0000 7f6fb89dd740 1 -- 172.21.15.7:0/464724036 shutdown_connections 2024-04-21T08:56:04.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.692+0000 7f6fb89dd740 1 -- 172.21.15.7:0/464724036 wait complete. 2024-04-21T08:56:04.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.692+0000 7f6fb89dd740 1 -- start start 2024-04-21T08:56:04.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.692+0000 7f6fb89dd740 1 Processor -- start 2024-04-21T08:56:04.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.696+0000 7f6fb89dd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630bc040 0x55a8630c2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.696+0000 7f6fb89dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630c5570 0x55a8630c2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.696+0000 7f6fb89dd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c2e90 0x55a8630d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.696+0000 7f6fb89dd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a8630c8450 con 0x55a8630bc040 2024-04-21T08:56:04.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.696+0000 7f6fb89dd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55a862fbb510 con 0x55a8630c2e90 2024-04-21T08:56:04.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.696+0000 7f6fb89dd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a8630c85d0 con 0x55a8630c5570 2024-04-21T08:56:04.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.696+0000 7f6fb89dd740 10 client.0 _ll_register_callbacks cb 0x55a862fbaf00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:04.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.696+0000 7f6faeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630c5570 0x55a8630c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.696+0000 7f6faeffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630c5570 0x55a8630c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:41944/0 (socket says 172.21.15.7:41944) 2024-04-21T08:56:04.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.696+0000 7f6faeffd640 1 -- 172.21.15.7:0/1474763467 learned_addr learned my addr 172.21.15.7:0/1474763467 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:04.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.697+0000 7f6faffff640 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c2e90 0x55a8630d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.697+0000 7f6faeffd640 1 -- 172.21.15.7:0/1474763467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c2e90 msgr2=0x55a8630d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:04.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.697+0000 7f6faeffd640 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c2e90 0x55a8630d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.697+0000 7f6faeffd640 1 -- 172.21.15.7:0/1474763467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630bc040 msgr2=0x55a8630c2410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:56:04.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.697+0000 7f6faeffd640 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630bc040 0x55a8630c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.697+0000 7f6faeffd640 1 -- 172.21.15.7:0/1474763467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a862dffa20 con 0x55a8630c5570 2024-04-21T08:56:04.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.701+0000 7f6faeffd640 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630c5570 0x55a8630c2950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0000c00 tx=0x7f6fa00cb290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4662d4606b401431 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:04.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.701+0000 7f6f937f6640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:56:04.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.701+0000 7f6f937f6640 1 -- 172.21.15.7:0/1474763467 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0004ce0 con 0x55a8630c5570 2024-04-21T08:56:04.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.701+0000 7f6f937f6640 1 -- 172.21.15.7:0/1474763467 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9764+0+0 (secure 0 0 0) 0x7f6fa00cd230 con 0x55a8630c5570 2024-04-21T08:56:04.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.702+0000 7f6fb89dd740 1 -- 172.21.15.7:0/1474763467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a86302ea20 con 0x55a8630c5570 2024-04-21T08:56:04.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.702+0000 7f6f937f6640 1 -- 172.21.15.7:0/1474763467 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6fa00035f0 con 0x55a8630c5570 2024-04-21T08:56:04.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.703+0000 7f6fb89dd740 10 client.26899 fetch_fsmap learned FSMap version 23 2024-04-21T08:56:04.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.703+0000 7f6fb89dd740 1 -- 172.21.15.7:0/1474763467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a862e601b0 con 0x55a8630c5570 2024-04-21T08:56:04.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.707+0000 7f6f937f6640 1 -- 172.21.15.7:0/1474763467 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a86302ea20 con 0x55a8630c5570 2024-04-21T08:56:04.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.707+0000 7f6f937f6640 1 -- 172.21.15.7:0/1474763467 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6fa00d2030 con 0x55a8630c5570 2024-04-21T08:56:04.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.707+0000 7f6fb89dd740 10 client.26899 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:56:04.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.707+0000 7f6fb89dd740 10 client.26899 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-21T08:56:04.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.707+0000 7f6fb89dd740 20 client.26899 populate_metadata read hostname 'smithi007' 2024-04-21T08:56:04.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.707+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8630d5d80 0x55a8630d81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.707+0000 7f6fb89dd740 4 client.26899 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:56:04.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.707+0000 7f6fb89dd740 1 -- 172.21.15.7:0/1474763467 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a863028890 con 0x55a8630d5d80 2024-04-21T08:56:04.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.708+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8630d5d80 0x55a8630d81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.711+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8630d5d80 0x55a8630d81f0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d9dad053f1f3ab7 server_cookie=8a01a1f0ba593a1f in_seq=0 out_seq=0 2024-04-21T08:56:04.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.711+0000 7f6f937f6640 10 client.26899 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:04.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.712+0000 7f6f937f6640 1 -- 172.21.15.7:0/1474763467 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a863028890 con 0x55a8630d5d80 2024-04-21T08:56:04.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.712+0000 7f6f937f6640 10 client.26899 handle_command_reply: tid=0 2024-04-21T08:56:04.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.715+0000 7f6fb89dd740 1 client.26899 shutdown 2024-04-21T08:56:04.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.721+0000 7f6fb89dd740 1 -- 172.21.15.7:0/1474763467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630c5570 msgr2=0x55a8630c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:04.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.721+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630c5570 0x55a8630c2950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0000c00 tx=0x7f6fa00cb290 comp rx=0 tx=0).stop 2024-04-21T08:56:04.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.723+0000 7f6fb89dd740 1 -- 172.21.15.7:0/1474763467 shutdown_connections 2024-04-21T08:56:04.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.723+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a8630d5d80 0x55a8630d81f0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.723+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c2e90 0x55a8630d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.723+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630c5570 0x55a8630c2950 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.723+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/1474763467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630bc040 0x55a8630c2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.723+0000 7f6fb89dd740 1 -- 172.21.15.7:0/1474763467 >> 172.21.15.7:0/1474763467 conn(0x55a8630b6710 msgr2=0x55a8630b8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:04.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.723+0000 7f6f937f6640 0 client.26899 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:04.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.724+0000 7f6fb89dd740 1 -- 172.21.15.7:0/1474763467 shutdown_connections 2024-04-21T08:56:04.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.724+0000 7f6fb89dd740 1 -- 172.21.15.7:0/1474763467 wait complete. 2024-04-21T08:56:04.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.725+0000 7f6fb89dd740 20 client.26899 trim_cache size 0 max 16384 2024-04-21T08:56:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.764+0000 7f6fb89dd740 1 Processor -- start 2024-04-21T08:56:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.764+0000 7f6fb89dd740 1 -- start start 2024-04-21T08:56:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.764+0000 7f6fb89dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630ddb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.765+0000 7f6fb89dd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630c2e90 0x55a8630de040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.765+0000 7f6faf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630ddb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.765+0000 7f6faf7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630ddb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:41946/0 (socket says 172.21.15.7:41946) 2024-04-21T08:56:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.765+0000 7f6faf7fe640 1 -- 172.21.15.7:0/2328835024 learned_addr learned my addr 172.21.15.7:0/2328835024 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.765+0000 7f6faeffd640 1 --2- 172.21.15.7:0/2328835024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630c2e90 0x55a8630de040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.765+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/2328835024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a8630cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.765+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2328835024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb01b3f20 con 0x55a8630c2e90 2024-04-21T08:56:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.765+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2328835024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb01b6780 con 0x55a8630c5570 2024-04-21T08:56:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.765+0000 7f6faffff640 1 --2- 172.21.15.7:0/2328835024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a8630cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.766+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2328835024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb01b4570 con 0x55a8630bc040 2024-04-21T08:56:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.766+0000 7f6faf7fe640 1 -- 172.21.15.7:0/2328835024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 msgr2=0x55a8630cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.766+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/2328835024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a8630cd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.766+0000 7f6faf7fe640 1 -- 172.21.15.7:0/2328835024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630c2e90 msgr2=0x55a8630de040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.766+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/2328835024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630c2e90 0x55a8630de040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.766+0000 7f6faf7fe640 1 -- 172.21.15.7:0/2328835024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a862cf3310 con 0x55a8630bc040 2024-04-21T08:56:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.766+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/2328835024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630ddb00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4008010 tx=0x7f6fa400a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d3412199651efb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.769+0000 7f6f94ff9640 1 -- 172.21.15.7:0/2328835024 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4015040 con 0x55a8630bc040 2024-04-21T08:56:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.770+0000 7f6f94ff9640 1 -- 172.21.15.7:0/2328835024 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a863028890 con 0x55a8630bc040 2024-04-21T08:56:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.770+0000 7f6f94ff9640 1 -- 172.21.15.7:0/2328835024 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4004450 con 0x55a8630bc040 2024-04-21T08:56:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.772+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2328835024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 msgr2=0x55a8630ddb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.772+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/2328835024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630ddb00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4008010 tx=0x7f6fa400a490 comp rx=0 tx=0).stop 2024-04-21T08:56:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.772+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2328835024 shutdown_connections 2024-04-21T08:56:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.772+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/2328835024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a8630cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.772+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/2328835024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630c2e90 0x55a8630de040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.772+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/2328835024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630ddb00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.772+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2328835024 >> 172.21.15.7:0/2328835024 conn(0x55a8630b6710 msgr2=0x55a8630b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.775+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2328835024 shutdown_connections 2024-04-21T08:56:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.776+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2328835024 wait complete. 2024-04-21T08:56:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.777+0000 7f6fb89dd740 1 -- start start 2024-04-21T08:56:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.777+0000 7f6fb89dd740 1 Processor -- start 2024-04-21T08:56:04.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.778+0000 7f6fb89dd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630bf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.778+0000 7f6fb89dd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630c2e90 0x55a8630bfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.778+0000 7f6fb89dd740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a86313b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.778+0000 7f6fb89dd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb01b4c40 con 0x55a8630c2e90 2024-04-21T08:56:04.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.778+0000 7f6fb89dd740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb01b60b0 con 0x55a8630c5570 2024-04-21T08:56:04.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.778+0000 7f6fb89dd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb01b5310 con 0x55a8630bc040 2024-04-21T08:56:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.778+0000 7f6fb89dd740 10 client.0 _ll_register_callbacks cb 0x7f6fa0004b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.779+0000 7f6faeffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630c2e90 0x55a8630bfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.779+0000 7f6faeffd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630c2e90 0x55a8630bfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42548/0 (socket says 172.21.15.7:42548) 2024-04-21T08:56:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.779+0000 7f6faeffd640 1 -- 172.21.15.7:0/2394371273 learned_addr learned my addr 172.21.15.7:0/2394371273 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.779+0000 7f6faffff640 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a86313b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.780+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630bf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.783+0000 7f6faffff640 1 -- 172.21.15.7:0/2394371273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 msgr2=0x55a8630bf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:04.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.783+0000 7f6faffff640 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630bf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.783+0000 7f6faffff640 1 -- 172.21.15.7:0/2394371273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630c2e90 msgr2=0x55a8630bfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:04.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.783+0000 7f6faffff640 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630c2e90 0x55a8630bfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.783+0000 7f6faffff640 1 -- 172.21.15.7:0/2394371273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86313b8b0 con 0x55a8630c5570 2024-04-21T08:56:04.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.787+0000 7f6faffff640 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a86313b370 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8004b90 tx=0x7f6fa8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fab84b4447cda29d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:04.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.788+0000 7f6f92ff5640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:56:04.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.789+0000 7f6f92ff5640 1 -- 172.21.15.7:0/2394371273 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8028670 con 0x55a8630c5570 2024-04-21T08:56:04.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.789+0000 7f6f92ff5640 1 -- 172.21.15.7:0/2394371273 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9764+0+0 (secure 0 0 0) 0x7f6fa801f400 con 0x55a8630c5570 2024-04-21T08:56:04.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.790+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2394371273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a863028890 con 0x55a8630c5570 2024-04-21T08:56:04.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.797+0000 7f6f92ff5640 1 -- 172.21.15.7:0/2394371273 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6fa8010e20 con 0x55a8630c5570 2024-04-21T08:56:04.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.802+0000 7f6fb89dd740 10 client.26924 fetch_fsmap learned FSMap version 23 2024-04-21T08:56:04.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.802+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2394371273 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a86313cb30 con 0x55a8630c5570 2024-04-21T08:56:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.806+0000 7f6f92ff5640 1 -- 172.21.15.7:0/2394371273 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a863028890 con 0x55a8630c5570 2024-04-21T08:56:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.806+0000 7f6f92ff5640 1 -- 172.21.15.7:0/2394371273 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6fa801fb80 con 0x55a8630c5570 2024-04-21T08:56:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.806+0000 7f6fb89dd740 10 client.26924 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:56:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.806+0000 7f6fb89dd740 10 client.26924 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-21T08:56:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.806+0000 7f6fb89dd740 20 client.26924 populate_metadata read hostname 'smithi007' 2024-04-21T08:56:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.806+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a86313d2c0 0x55a86313f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.806+0000 7f6fb89dd740 4 client.26924 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-21T08:56:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.806+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2394371273 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55a862e9ab80 con 0x55a86313d2c0 2024-04-21T08:56:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.808+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a86313d2c0 0x55a86313f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:04.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.816+0000 7f6faf7fe640 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a86313d2c0 0x55a86313f6e0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=689eb65ef5350805 server_cookie=18d4226bbaef9257 in_seq=0 out_seq=0 2024-04-21T08:56:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.818+0000 7f6f92ff5640 10 client.26924 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.818+0000 7f6f92ff5640 1 -- 172.21.15.7:0/2394371273 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+79 (crc 0 0 0) 0x55a862e9ab80 con 0x55a86313d2c0 2024-04-21T08:56:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.818+0000 7f6f92ff5640 10 client.26924 handle_command_reply: tid=0 2024-04-21T08:56:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.818+0000 7f6fb89dd740 1 client.26924 shutdown 2024-04-21T08:56:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.821+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2394371273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 msgr2=0x55a86313b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.821+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a86313b370 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8004b90 tx=0x7f6fa8004a40 comp rx=0 tx=0).stop 2024-04-21T08:56:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.823+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2394371273 shutdown_connections 2024-04-21T08:56:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.823+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55a86313d2c0 0x55a86313f6e0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.823+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55a8630c5570 0x55a86313b370 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.823+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a8630c2e90 0x55a8630bfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.823+0000 7f6fb89dd740 1 --2- 172.21.15.7:0/2394371273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a8630bc040 0x55a8630bf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.823+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2394371273 >> 172.21.15.7:0/2394371273 conn(0x55a8630b6710 msgr2=0x55a8630b7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.823+0000 7f6f92ff5640 0 client.26924 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.826+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2394371273 shutdown_connections 2024-04-21T08:56:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.830+0000 7f6fb89dd740 1 -- 172.21.15.7:0/2394371273 wait complete. 2024-04-21T08:56:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.834+0000 7f6fb89dd740 20 client.26924 trim_cache size 0 max 16384 2024-04-21T08:56:04.834 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:56:04.834 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 23, 2024-04-21T08:56:04.834 INFO:teuthology.orchestra.run.smithi007.stdout: "leader": 24533, 2024-04-21T08:56:04.834 INFO:teuthology.orchestra.run.smithi007.stdout: "set_version": 2, 2024-04-21T08:56:04.834 INFO:teuthology.orchestra.run.smithi007.stdout: "sets": {} 2024-04-21T08:56:04.834 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:56:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.835+0000 7f6f967fc640 1 -- 172.21.15.7:0/1003572969 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f980a3ef0 msgr2=0x7f6f980a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.835+0000 7f6f967fc640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f980a3ef0 0x7f6f980a63b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6fb006b2d0 tx=0x7f6fa8008330 comp rx=0 tx=0).stop 2024-04-21T08:56:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.835+0000 7f6f967fc640 1 -- 172.21.15.7:0/1003572969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb0110b10 msgr2=0x7f6fb0116fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.835+0000 7f6f967fc640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb0110b10 0x7f6fb0116fb0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f6fa003d2b0 tx=0x7f6fa001a480 comp rx=0 tx=0).stop 2024-04-21T08:56:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.835+0000 7f6f967fc640 1 -- 172.21.15.7:0/1003572969 shutdown_connections 2024-04-21T08:56:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.835+0000 7f6f967fc640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fb011b730 0x7f6fb01174f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.835+0000 7f6f967fc640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fb0110b10 0x7f6fb0116fb0 secure :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f6fa003d2b0 tx=0x7f6fa001a480 comp rx=0 tx=0).stop 2024-04-21T08:56:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.835+0000 7f6f967fc640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6f980a3ef0 0x7f6f980a63b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.835+0000 7f6f967fc640 1 --2- 172.21.15.7:0/1003572969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb01106a0 0x7f6fb011af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.835+0000 7f6f967fc640 1 -- 172.21.15.7:0/1003572969 >> 172.21.15.7:0/1003572969 conn(0x7f6fb006cff0 msgr2=0x7f6fb011c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.836+0000 7f6f967fc640 1 -- 172.21.15.7:0/1003572969 shutdown_connections 2024-04-21T08:56:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:04.836+0000 7f6f967fc640 1 -- 172.21.15.7:0/1003572969 wait complete. 2024-04-21T08:56:04.858 ERROR:teuthology.run_tasks:Manager failed: quiescer Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 310, in task stop_all_quiescers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 263, in stop_all_quiescers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: Error quiescing root: -110 (ETIMEDOUT) 2024-04-21T08:56:04.862 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-21T08:56:04.872 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi007.front.sepia.ceph.com 2024-04-21T08:56:04.883 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-21T08:56:04.913 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-21T08:56:04.913 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:56:04.913 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-21T08:56:04.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-21T08:56:04.985 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-21T08:56:05.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T08:56:05.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.220+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1348525597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a481072b0 msgr2=0x7f5a4810ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:05.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.220+0000 7f5a4fac1640 1 --2- 172.21.15.7:0/1348525597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a481072b0 0x7f5a4810ee80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009ca0 tx=0x7f5a4002c140 comp rx=0 tx=0).stop 2024-04-21T08:56:05.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.220+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1348525597 shutdown_connections 2024-04-21T08:56:05.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.220+0000 7f5a4fac1640 1 --2- 172.21.15.7:0/1348525597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a481072b0 0x7f5a4810ee80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:05.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.220+0000 7f5a4fac1640 1 --2- 172.21.15.7:0/1348525597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a48102b90 0x7f5a48106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:05.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.220+0000 7f5a4fac1640 1 --2- 172.21.15.7:0/1348525597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a481021e0 0x7f5a481025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:05.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.220+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1348525597 >> 172.21.15.7:0/1348525597 conn(0x7f5a480fda60 msgr2=0x7f5a480ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:05.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.228+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1348525597 shutdown_connections 2024-04-21T08:56:05.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.230+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1348525597 wait complete. 2024-04-21T08:56:05.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.230+0000 7f5a4fac1640 1 Processor -- start 2024-04-21T08:56:05.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.230+0000 7f5a4fac1640 1 -- start start 2024-04-21T08:56:05.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.232+0000 7f5a4fac1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a481021e0 0x7f5a48073840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:05.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.232+0000 7f5a4fac1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a481072b0 0x7f5a4806c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:05.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.232+0000 7f5a4fac1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4806ce50 0x7f5a4806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:05.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.232+0000 7f5a4fac1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5a48112310 con 0x7f5a4806ce50 2024-04-21T08:56:05.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.232+0000 7f5a4fac1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5a48112190 con 0x7f5a481072b0 2024-04-21T08:56:05.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.232+0000 7f5a4fac1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5a48112490 con 0x7f5a481021e0 2024-04-21T08:56:05.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.232+0000 7f5a4e05e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4806ce50 0x7f5a4806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:05.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.232+0000 7f5a4e05e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4806ce50 0x7f5a4806d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42558/0 (socket says 172.21.15.7:42558) 2024-04-21T08:56:05.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.232+0000 7f5a4e05e640 1 -- 172.21.15.7:0/1499424390 learned_addr learned my addr 172.21.15.7:0/1499424390 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:05.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.233+0000 7f5a4e05e640 1 -- 172.21.15.7:0/1499424390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a481021e0 msgr2=0x7f5a48073840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:56:05.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.233+0000 7f5a4e05e640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a481021e0 0x7f5a48073840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:05.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.233+0000 7f5a4e05e640 1 -- 172.21.15.7:0/1499424390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a481072b0 msgr2=0x7f5a4806c910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:56:05.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.233+0000 7f5a4e05e640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a481072b0 0x7f5a4806c910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:05.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.233+0000 7f5a4e05e640 1 -- 172.21.15.7:0/1499424390 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a48071360 con 0x7f5a4806ce50 2024-04-21T08:56:05.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.233+0000 7f5a4e05e640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4806ce50 0x7f5a4806d300 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f5a40037040 tx=0x7f5a40035d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:05.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.235+0000 7f5a3effd640 1 -- 172.21.15.7:0/1499424390 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40039040 con 0x7f5a4806ce50 2024-04-21T08:56:05.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.235+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1499424390 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a480715f0 con 0x7f5a4806ce50 2024-04-21T08:56:05.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.235+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1499424390 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a48071ae0 con 0x7f5a4806ce50 2024-04-21T08:56:05.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.236+0000 7f5a3effd640 1 -- 172.21.15.7:0/1499424390 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a40044070 con 0x7f5a4806ce50 2024-04-21T08:56:05.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.236+0000 7f5a3effd640 1 -- 172.21.15.7:0/1499424390 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a400044d0 con 0x7f5a4806ce50 2024-04-21T08:56:05.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.238+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1499424390 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a481036b0 con 0x7f5a4806ce50 2024-04-21T08:56:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.240+0000 7f5a3effd640 1 -- 172.21.15.7:0/1499424390 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f5a40004670 con 0x7f5a4806ce50 2024-04-21T08:56:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.241+0000 7f5a3effd640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a340ac730 0x7f5a340aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:05.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.242+0000 7f5a3effd640 1 -- 172.21.15.7:0/1499424390 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9764+0+0 (secure 0 0 0) 0x7f5a400f2e20 con 0x7f5a4806ce50 2024-04-21T08:56:05.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.250+0000 7f5a4d85d640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a340ac730 0x7f5a340aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:05.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.250+0000 7f5a3effd640 1 -- 172.21.15.7:0/1499424390 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a4004d020 con 0x7f5a4806ce50 2024-04-21T08:56:05.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.264+0000 7f5a4d85d640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a340ac730 0x7f5a340aebf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5a44009670 tx=0x7f5a44006bd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:05.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:05 smithi016 ceph-mon[31259]: pgmap v358: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 418 KiB/s wr, 1.95k op/s 2024-04-21T08:56:05.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:05 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1852580690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:05.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:05 smithi077 ceph-mon[31615]: pgmap v358: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 418 KiB/s wr, 1.95k op/s 2024-04-21T08:56:05.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:05 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1852580690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:05.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:05 smithi007 ceph-mon[28477]: pgmap v358: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 418 KiB/s wr, 1.95k op/s 2024-04-21T08:56:05.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:05 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1852580690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:05.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.661+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1499424390 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f5a48102b90 con 0x7f5a340ac730 2024-04-21T08:56:05.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.775+0000 7f5a3effd640 1 -- 172.21.15.7:0/1499424390 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f5a48102b90 con 0x7f5a340ac730 2024-04-21T08:56:05.775 INFO:teuthology.orchestra.run.smithi007.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T08:44:47", "first": "2024-04-21T08:45:00", "last": "2024-04-21T08:56:00", "last_pruned": "2024-04-21T08:56:01", "created_count": 12, "pruned_count": 6, "active": true} 2024-04-21T08:56:05.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.778+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1499424390 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a340ac730 msgr2=0x7f5a340aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:05.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.778+0000 7f5a4fac1640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a340ac730 0x7f5a340aebf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5a44009670 tx=0x7f5a44006bd0 comp rx=0 tx=0).stop 2024-04-21T08:56:05.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.778+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1499424390 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4806ce50 msgr2=0x7f5a4806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:05.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.778+0000 7f5a4fac1640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4806ce50 0x7f5a4806d300 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f5a40037040 tx=0x7f5a40035d50 comp rx=0 tx=0).stop 2024-04-21T08:56:05.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.778+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1499424390 shutdown_connections 2024-04-21T08:56:05.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.778+0000 7f5a4fac1640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4806ce50 0x7f5a4806d300 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:05.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.778+0000 7f5a4fac1640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5a481072b0 0x7f5a4806c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:05.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.778+0000 7f5a4fac1640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f5a340ac730 0x7f5a340aebf0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:05.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.778+0000 7f5a4fac1640 1 --2- 172.21.15.7:0/1499424390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a481021e0 0x7f5a48073840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:05.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.778+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1499424390 >> 172.21.15.7:0/1499424390 conn(0x7f5a480fda60 msgr2=0x7f5a480fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:05.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.779+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1499424390 shutdown_connections 2024-04-21T08:56:05.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:05.779+0000 7f5a4fac1640 1 -- 172.21.15.7:0/1499424390 wait complete. 2024-04-21T08:56:05.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T08:56:06.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.007+0000 7f08f8031640 1 -- 172.21.15.7:0/1056536194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08f0102b80 msgr2=0x7f08f0106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:06.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.007+0000 7f08f8031640 1 --2- 172.21.15.7:0/1056536194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08f0102b80 0x7f08f0106bc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f08e8009ca0 tx=0x7f08e802c140 comp rx=0 tx=0).stop 2024-04-21T08:56:06.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.013+0000 7f08f8031640 1 -- 172.21.15.7:0/1056536194 shutdown_connections 2024-04-21T08:56:06.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.013+0000 7f08f8031640 1 --2- 172.21.15.7:0/1056536194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08f01072a0 0x7f08f010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:06.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.013+0000 7f08f8031640 1 --2- 172.21.15.7:0/1056536194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08f0102b80 0x7f08f0106bc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:06.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.013+0000 7f08f8031640 1 --2- 172.21.15.7:0/1056536194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f01021d0 0x7f08f01025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:06.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.013+0000 7f08f8031640 1 -- 172.21.15.7:0/1056536194 >> 172.21.15.7:0/1056536194 conn(0x7f08f00fda10 msgr2=0x7f08f00ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:06.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.015+0000 7f08f8031640 1 -- 172.21.15.7:0/1056536194 shutdown_connections 2024-04-21T08:56:06.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.015+0000 7f08f8031640 1 -- 172.21.15.7:0/1056536194 wait complete. 2024-04-21T08:56:06.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.016+0000 7f08f8031640 1 Processor -- start 2024-04-21T08:56:06.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.017+0000 7f08f8031640 1 -- start start 2024-04-21T08:56:06.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.017+0000 7f08f8031640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08f01021d0 0x7f08f01966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:06.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.017+0000 7f08f8031640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f01072a0 0x7f08f0196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:06.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.017+0000 7f08f8031640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08f019af20 0x7f08f019b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:06.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.017+0000 7f08f8031640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f08f0110150 con 0x7f08f01072a0 2024-04-21T08:56:06.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.017+0000 7f08f8031640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08f010ffd0 con 0x7f08f01021d0 2024-04-21T08:56:06.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.017+0000 7f08f8031640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f08f01102d0 con 0x7f08f019af20 2024-04-21T08:56:06.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.017+0000 7f08f55cc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f01072a0 0x7f08f0196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:06.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.017+0000 7f08f55cc640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f01072a0 0x7f08f0196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42574/0 (socket says 172.21.15.7:42574) 2024-04-21T08:56:06.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.017+0000 7f08f55cc640 1 -- 172.21.15.7:0/3649883878 learned_addr learned my addr 172.21.15.7:0/3649883878 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.018+0000 7f08f55cc640 1 -- 172.21.15.7:0/3649883878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08f019af20 msgr2=0x7f08f019b3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:56:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.018+0000 7f08f55cc640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08f019af20 0x7f08f019b3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.018+0000 7f08f55cc640 1 -- 172.21.15.7:0/3649883878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08f01021d0 msgr2=0x7f08f01966b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:56:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.018+0000 7f08f55cc640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08f01021d0 0x7f08f01966b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.018+0000 7f08f55cc640 1 -- 172.21.15.7:0/3649883878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f019ba10 con 0x7f08f01072a0 2024-04-21T08:56:06.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.018+0000 7f08f55cc640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f01072a0 0x7f08f0196bf0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f08e8037040 tx=0x7f08e8035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:06.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.019+0000 7f08e2ffd640 1 -- 172.21.15.7:0/3649883878 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e8041020 con 0x7f08f01072a0 2024-04-21T08:56:06.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.019+0000 7f08f8031640 1 -- 172.21.15.7:0/3649883878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08f019bca0 con 0x7f08f01072a0 2024-04-21T08:56:06.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.019+0000 7f08f8031640 1 -- 172.21.15.7:0/3649883878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08f006bc60 con 0x7f08f01072a0 2024-04-21T08:56:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.021+0000 7f08f8031640 1 -- 172.21.15.7:0/3649883878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08f0050530 con 0x7f08f01072a0 2024-04-21T08:56:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.021+0000 7f08e2ffd640 1 -- 172.21.15.7:0/3649883878 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08e80326f0 con 0x7f08f01072a0 2024-04-21T08:56:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.021+0000 7f08e2ffd640 1 -- 172.21.15.7:0/3649883878 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e8039040 con 0x7f08f01072a0 2024-04-21T08:56:06.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.027+0000 7f08e2ffd640 1 -- 172.21.15.7:0/3649883878 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f08e8044020 con 0x7f08f01072a0 2024-04-21T08:56:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.028+0000 7f08e2ffd640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f08dc0a8260 0x7f08dc0aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.029+0000 7f08e2ffd640 1 -- 172.21.15.7:0/3649883878 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9764+0+0 (secure 0 0 0) 0x7f08e80f2eb0 con 0x7f08f01072a0 2024-04-21T08:56:06.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.030+0000 7f08f5dcd640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f08dc0a8260 0x7f08dc0aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:06.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.031+0000 7f08f5dcd640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f08dc0a8260 0x7f08dc0aa720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f08e40096b0 tx=0x7f08e4006c20 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:06.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.037+0000 7f08e2ffd640 1 -- 172.21.15.7:0/3649883878 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f08e803b2f0 con 0x7f08f01072a0 2024-04-21T08:56:06.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.452+0000 7f08f8031640 1 -- 172.21.15.7:0/3649883878 --> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f08f0102b80 con 0x7f08dc0a8260 2024-04-21T08:56:06.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.562+0000 7f08e2ffd640 1 -- 172.21.15.7:0/3649883878 <== mgr.15408 v2:172.21.15.7:6800/3443135673 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f08f0102b80 con 0x7f08dc0a8260 2024-04-21T08:56:06.563 INFO:teuthology.orchestra.run.smithi007.stdout:/ 1m 6m3h 2024-04-21T08:56:06.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.567+0000 7f08f8031640 1 -- 172.21.15.7:0/3649883878 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f08dc0a8260 msgr2=0x7f08dc0aa720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:06.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.567+0000 7f08f8031640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f08dc0a8260 0x7f08dc0aa720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f08e40096b0 tx=0x7f08e4006c20 comp rx=0 tx=0).stop 2024-04-21T08:56:06.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.567+0000 7f08f8031640 1 -- 172.21.15.7:0/3649883878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f01072a0 msgr2=0x7f08f0196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:06.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.567+0000 7f08f8031640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f01072a0 0x7f08f0196bf0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f08e8037040 tx=0x7f08e8035ce0 comp rx=0 tx=0).stop 2024-04-21T08:56:06.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.567+0000 7f08f8031640 1 -- 172.21.15.7:0/3649883878 shutdown_connections 2024-04-21T08:56:06.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.567+0000 7f08f8031640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f08dc0a8260 0x7f08dc0aa720 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:06.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.567+0000 7f08f8031640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08f019af20 0x7f08f019b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:06.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.567+0000 7f08f8031640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f01072a0 0x7f08f0196bf0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:06.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.567+0000 7f08f8031640 1 --2- 172.21.15.7:0/3649883878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08f01021d0 0x7f08f01966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:06.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.567+0000 7f08f8031640 1 -- 172.21.15.7:0/3649883878 >> 172.21.15.7:0/3649883878 conn(0x7f08f00fda10 msgr2=0x7f08f00fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:06.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.568+0000 7f08f8031640 1 -- 172.21.15.7:0/3649883878 shutdown_connections 2024-04-21T08:56:06.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:06.568+0000 7f08f8031640 1 -- 172.21.15.7:0/3649883878 wait complete. 2024-04-21T08:56:06.586 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-21T08:56:06.597 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-21T08:56:07.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:07 smithi016 ceph-mon[31259]: pgmap v359: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 637 KiB/s wr, 2.17k op/s 2024-04-21T08:56:07.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:07 smithi016 ceph-mon[31259]: from='client.17271 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:07.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:07 smithi077 ceph-mon[31615]: pgmap v359: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 637 KiB/s wr, 2.17k op/s 2024-04-21T08:56:07.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:07 smithi077 ceph-mon[31615]: from='client.17271 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:07.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:07 smithi007 ceph-mon[28477]: pgmap v359: 129 pgs: 13 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 77 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 637 KiB/s wr, 2.17k op/s 2024-04-21T08:56:07.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:07 smithi007 ceph-mon[28477]: from='client.17271 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:08.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:08 smithi016 ceph-mon[31259]: from='client.17274 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:08.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:08 smithi077 ceph-mon[31615]: from='client.17274 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:08.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:08 smithi007 ceph-mon[28477]: from='client.17274 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:09.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:09 smithi016 ceph-mon[31259]: pgmap v360: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 985 KiB/s wr, 2.19k op/s 2024-04-21T08:56:09.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:09 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:56:09.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:09 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:09.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:09 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:09.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:09 smithi077 ceph-mon[31615]: pgmap v360: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 985 KiB/s wr, 2.19k op/s 2024-04-21T08:56:09.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:09 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:56:09.605 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:09 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:09.605 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:09 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:09.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:09 smithi007 ceph-mon[28477]: pgmap v360: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 985 KiB/s wr, 2.19k op/s 2024-04-21T08:56:09.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:09 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:56:09.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:09 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:09.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:09 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:56:10.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:56:10.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:10 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:10 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:10 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:10 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:10 smithi007 ceph-mon[28477]: pgmap v361: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 1.94k op/s 2024-04-21T08:56:10.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.750+0000 7f893cafd640 1 -- 172.21.15.7:0/209542104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89381106a0 msgr2=0x7f8938109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:10.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.750+0000 7f893cafd640 1 --2- 172.21.15.7:0/209542104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89381106a0 0x7f8938109150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8928009900 tx=0x7f892802bc80 comp rx=0 tx=0).stop 2024-04-21T08:56:10.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.753+0000 7f893cafd640 1 -- 172.21.15.7:0/209542104 shutdown_connections 2024-04-21T08:56:10.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.753+0000 7f893cafd640 1 --2- 172.21.15.7:0/209542104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f893810e1b0 0x7f893810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.753+0000 7f893cafd640 1 --2- 172.21.15.7:0/209542104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8938110b10 0x7f8938109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.753+0000 7f893cafd640 1 --2- 172.21.15.7:0/209542104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89381106a0 0x7f8938109150 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.753+0000 7f893cafd640 1 -- 172.21.15.7:0/209542104 >> 172.21.15.7:0/209542104 conn(0x7f893806e360 msgr2=0x7f893806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:10.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.759+0000 7f893cafd640 1 -- 172.21.15.7:0/209542104 shutdown_connections 2024-04-21T08:56:10.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.761+0000 7f893cafd640 1 -- 172.21.15.7:0/209542104 wait complete. 2024-04-21T08:56:10.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.762+0000 7f893cafd640 1 Processor -- start 2024-04-21T08:56:10.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.764+0000 7f893cafd640 1 -- start start 2024-04-21T08:56:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.764+0000 7f893cafd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893810e1b0 0x7f893811ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.764+0000 7f893cafd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89381106a0 0x7f8938118e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.764+0000 7f893cafd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8938110b10 0x7f8938119380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.764+0000 7f893cafd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f89381117d0 con 0x7f8938110b10 2024-04-21T08:56:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.764+0000 7f893cafd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8938111650 con 0x7f89381106a0 2024-04-21T08:56:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.764+0000 7f893cafd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8938111950 con 0x7f893810e1b0 2024-04-21T08:56:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.765+0000 7f893659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893810e1b0 0x7f893811ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.765+0000 7f8936d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8938110b10 0x7f8938119380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.765+0000 7f8936d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8938110b10 0x7f8938119380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42590/0 (socket says 172.21.15.7:42590) 2024-04-21T08:56:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.765+0000 7f8936d9d640 1 -- 172.21.15.7:0/394011656 learned_addr learned my addr 172.21.15.7:0/394011656 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.769+0000 7f893659c640 1 -- 172.21.15.7:0/394011656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89381106a0 msgr2=0x7f8938118e40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:56:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.769+0000 7f893659c640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89381106a0 0x7f8938118e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.769+0000 7f893659c640 1 -- 172.21.15.7:0/394011656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8938110b10 msgr2=0x7f8938119380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.769+0000 7f893659c640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8938110b10 0x7f8938119380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.769+0000 7f893659c640 1 -- 172.21.15.7:0/394011656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89381199c0 con 0x7f893810e1b0 2024-04-21T08:56:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.771+0000 7f893659c640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893810e1b0 0x7f893811ed80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f892802c290 tx=0x7f892802eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.772+0000 7f892f7fe640 1 -- 172.21.15.7:0/394011656 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928041020 con 0x7f893810e1b0 2024-04-21T08:56:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.773+0000 7f892f7fe640 1 -- 172.21.15.7:0/394011656 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f892802c620 con 0x7f893810e1b0 2024-04-21T08:56:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.773+0000 7f893cafd640 1 -- 172.21.15.7:0/394011656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8938119c50 con 0x7f893810e1b0 2024-04-21T08:56:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.773+0000 7f893cafd640 1 -- 172.21.15.7:0/394011656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89381bc7a0 con 0x7f893810e1b0 2024-04-21T08:56:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.773+0000 7f892f7fe640 1 -- 172.21.15.7:0/394011656 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89280346a0 con 0x7f893810e1b0 2024-04-21T08:56:10.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.785+0000 7f892f7fe640 1 -- 172.21.15.7:0/394011656 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f8928044020 con 0x7f893810e1b0 2024-04-21T08:56:10.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.786+0000 7f892f7fe640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89240a3f20 0x7f89240a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.786+0000 7f892f7fe640 1 -- 172.21.15.7:0/394011656 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9764+0+0 (secure 0 0 0) 0x7f89280f2220 con 0x7f893810e1b0 2024-04-21T08:56:10.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.787+0000 7f893f7c9740 1 Processor -- start 2024-04-21T08:56:10.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.791+0000 7f893f7c9740 1 -- start start 2024-04-21T08:56:10.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.791+0000 7f8935d9b640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89240a3f20 0x7f89240a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.791+0000 7f893f7c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d10b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.791+0000 7f893f7c9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 0x55fa9d114540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.791+0000 7f893f7c9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d116e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.791+0000 7f893f7c9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55fa9d0701e0 con 0x55fa9d10bea0 2024-04-21T08:56:10.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.791+0000 7f893f7c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fa9d00d390 con 0x55fa9d10b580 2024-04-21T08:56:10.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.791+0000 7f893f7c9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fa9d06c1a0 con 0x55fa9d114a80 2024-04-21T08:56:10.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.792+0000 7f8936d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d116e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.792+0000 7f8936d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d116e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.7:41994/0 (socket says 172.21.15.7:41994) 2024-04-21T08:56:10.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.792+0000 7f8936d9d640 1 -- 172.21.15.7:0/1686803246 learned_addr learned my addr 172.21.15.7:0/1686803246 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:10.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.794+0000 7f8936d9d640 1 -- 172.21.15.7:0/1686803246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 msgr2=0x55fa9d10b960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:56:10.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.794+0000 7f8936d9d640 1 --2- 172.21.15.7:0/1686803246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d10b960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.794+0000 7f8936d9d640 1 -- 172.21.15.7:0/1686803246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 msgr2=0x55fa9d114540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:10.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.794+0000 7f8936d9d640 1 --2- 172.21.15.7:0/1686803246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 0x55fa9d114540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.794+0000 7f8936d9d640 1 -- 172.21.15.7:0/1686803246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa9ce4c7d0 con 0x55fa9d114a80 2024-04-21T08:56:10.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.796+0000 7f8936d9d640 1 --2- 172.21.15.7:0/1686803246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d116e70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f89300104d0 tx=0x7f8930010990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a11bbbccc320a10f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:10.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.796+0000 7f892cff9640 1 -- 172.21.15.7:0/1686803246 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8930011760 con 0x55fa9d114a80 2024-04-21T08:56:10.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.796+0000 7f892cff9640 1 -- 172.21.15.7:0/1686803246 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8930011db0 con 0x55fa9d114a80 2024-04-21T08:56:10.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.796+0000 7f892cff9640 1 -- 172.21.15.7:0/1686803246 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89300193d0 con 0x55fa9d114a80 2024-04-21T08:56:10.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.798+0000 7f893f7c9740 1 -- 172.21.15.7:0/1686803246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 msgr2=0x55fa9d116e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:10.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.798+0000 7f893f7c9740 1 --2- 172.21.15.7:0/1686803246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d116e70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f89300104d0 tx=0x7f8930010990 comp rx=0 tx=0).stop 2024-04-21T08:56:10.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.803+0000 7f8935d9b640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89240a3f20 0x7f89240a63e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55fa9d10b050 tx=0x7f892000a2d0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:10.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.803+0000 7f893f7c9740 1 -- 172.21.15.7:0/1686803246 shutdown_connections 2024-04-21T08:56:10.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.803+0000 7f893f7c9740 1 --2- 172.21.15.7:0/1686803246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d116e70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.803+0000 7f893f7c9740 1 --2- 172.21.15.7:0/1686803246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 0x55fa9d114540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.803+0000 7f893f7c9740 1 --2- 172.21.15.7:0/1686803246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d10b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.803+0000 7f893f7c9740 1 -- 172.21.15.7:0/1686803246 >> 172.21.15.7:0/1686803246 conn(0x55fa9d105cb0 msgr2=0x55fa9d106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:10.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:10 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:10 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:10 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:10 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:10 smithi016 ceph-mon[31259]: pgmap v361: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 1.94k op/s 2024-04-21T08:56:10.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.811+0000 7f893f7c9740 1 -- 172.21.15.7:0/1686803246 shutdown_connections 2024-04-21T08:56:10.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.812+0000 7f893f7c9740 1 -- 172.21.15.7:0/1686803246 wait complete. 2024-04-21T08:56:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.812+0000 7f893f7c9740 1 -- start start 2024-04-21T08:56:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.812+0000 7f893f7c9740 1 Processor -- start 2024-04-21T08:56:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.812+0000 7f893f7c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d111a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.812+0000 7f893f7c9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 0x55fa9d111fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.812+0000 7f893f7c9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d1124f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.812+0000 7f893f7c9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55fa9d117870 con 0x55fa9d10bea0 2024-04-21T08:56:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.812+0000 7f893f7c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55fa9d0292f0 con 0x55fa9d10b580 2024-04-21T08:56:10.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.812+0000 7f893f7c9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fa9d1179f0 con 0x55fa9d114a80 2024-04-21T08:56:10.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.813+0000 7f893f7c9740 10 client.0 _ll_register_callbacks cb 0x55fa9d036580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:10.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.814+0000 7f893659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d111a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.814+0000 7f893659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d111a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:47928/0 (socket says 172.21.15.7:47928) 2024-04-21T08:56:10.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.814+0000 7f893659c640 1 -- 172.21.15.7:0/801032866 learned_addr learned my addr 172.21.15.7:0/801032866 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:10.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.814+0000 7f8936d9d640 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d1124f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.814+0000 7f8935d9b640 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 0x55fa9d111fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.814+0000 7f8936d9d640 1 -- 172.21.15.7:0/801032866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 msgr2=0x55fa9d111a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:10.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.814+0000 7f8936d9d640 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d111a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.814+0000 7f8936d9d640 1 -- 172.21.15.7:0/801032866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 msgr2=0x55fa9d111fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:10.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.814+0000 7f8936d9d640 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 0x55fa9d111fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.814+0000 7f8936d9d640 1 -- 172.21.15.7:0/801032866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa9cd823f0 con 0x55fa9d114a80 2024-04-21T08:56:10.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.821+0000 7f8936d9d640 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d1124f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f89300104a0 tx=0x7f893000ede0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53c3f9a2ff65eea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:10.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.822+0000 7f891dffb640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-04-21T08:56:10.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.824+0000 7f891dffb640 1 -- 172.21.15.7:0/801032866 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8930004a40 con 0x55fa9d114a80 2024-04-21T08:56:10.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.824+0000 7f891dffb640 1 -- 172.21.15.7:0/801032866 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9764+0+0 (secure 0 0 0) 0x7f8930004be0 con 0x55fa9d114a80 2024-04-21T08:56:10.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.827+0000 7f893f7c9740 1 -- 172.21.15.7:0/801032866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8930011db0 con 0x55fa9d114a80 2024-04-21T08:56:10.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.831+0000 7f891dffb640 1 -- 172.21.15.7:0/801032866 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8930004de0 con 0x55fa9d114a80 2024-04-21T08:56:10.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.835+0000 7f893f7c9740 10 client.26920 fetch_fsmap learned FSMap version 23 2024-04-21T08:56:10.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.835+0000 7f893f7c9740 1 -- 172.21.15.7:0/801032866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fa9cd420f0 con 0x55fa9d114a80 2024-04-21T08:56:10.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.839+0000 7f891dffb640 1 -- 172.21.15.7:0/801032866 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8930011db0 con 0x55fa9d114a80 2024-04-21T08:56:10.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.839+0000 7f891dffb640 1 -- 172.21.15.7:0/801032866 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f893002e480 con 0x55fa9d114a80 2024-04-21T08:56:10.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.841+0000 7f893f7c9740 10 client.26920 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:56:10.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.841+0000 7f893f7c9740 10 client.26920 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:10.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.841+0000 7f893f7c9740 20 client.26920 populate_metadata read hostname 'smithi007' 2024-04-21T08:56:10.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.841+0000 7f893f7c9740 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fa9d1469b0 0x55fa9d148da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.841+0000 7f893f7c9740 4 client.26920 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:56:10.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.841+0000 7f893f7c9740 1 -- 172.21.15.7:0/801032866 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa9d07dc00 con 0x55fa9d1469b0 2024-04-21T08:56:10.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.846+0000 7f893659c640 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fa9d1469b0 0x55fa9d148da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.853+0000 7f893659c640 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fa9d1469b0 0x55fa9d148da0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7615bb74c136d031 server_cookie=31444e9931f7b90f in_seq=0 out_seq=0 2024-04-21T08:56:10.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.853+0000 7f891dffb640 10 client.26920 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:10.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:10 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:10 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:10 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:10 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:10.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:10 smithi077 ceph-mon[31615]: pgmap v361: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 1.94k op/s 2024-04-21T08:56:10.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.861+0000 7f891dffb640 1 -- 172.21.15.7:0/801032866 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa9d07dc00 con 0x55fa9d1469b0 2024-04-21T08:56:10.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.861+0000 7f891dffb640 10 client.26920 handle_command_reply: tid=0 2024-04-21T08:56:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.862+0000 7f893f7c9740 1 client.26920 shutdown 2024-04-21T08:56:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.866+0000 7f893f7c9740 1 -- 172.21.15.7:0/801032866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 msgr2=0x55fa9d1124f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.866+0000 7f893f7c9740 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d1124f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f89300104a0 tx=0x7f893000ede0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.870+0000 7f893f7c9740 1 -- 172.21.15.7:0/801032866 shutdown_connections 2024-04-21T08:56:10.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.870+0000 7f893f7c9740 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fa9d1469b0 0x55fa9d148da0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.870+0000 7f893f7c9740 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d1124f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.870+0000 7f893f7c9740 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 0x55fa9d111fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.870+0000 7f893f7c9740 1 --2- 172.21.15.7:0/801032866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d111a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.870+0000 7f891dffb640 0 client.26920 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:10.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.870+0000 7f893f7c9740 1 -- 172.21.15.7:0/801032866 >> 172.21.15.7:0/801032866 conn(0x55fa9d105cb0 msgr2=0x55fa9d107a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:10.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.874+0000 7f893f7c9740 1 -- 172.21.15.7:0/801032866 shutdown_connections 2024-04-21T08:56:10.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.877+0000 7f893f7c9740 1 -- 172.21.15.7:0/801032866 wait complete. 2024-04-21T08:56:10.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.879+0000 7f893f7c9740 20 client.26920 trim_cache size 0 max 16384 2024-04-21T08:56:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.924+0000 7f893f7c9740 1 Processor -- start 2024-04-21T08:56:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.924+0000 7f893f7c9740 1 -- start start 2024-04-21T08:56:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.925+0000 7f893f7c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d1109b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.925+0000 7f893f7c9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 0x55fa9d1207c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.925+0000 7f893659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d1109b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.925+0000 7f893659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d1109b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:47944/0 (socket says 172.21.15.7:47944) 2024-04-21T08:56:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.925+0000 7f893659c640 1 -- 172.21.15.7:0/577907061 learned_addr learned my addr 172.21.15.7:0/577907061 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.925+0000 7f893f7c9740 1 --2- 172.21.15.7:0/577907061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d112310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.926+0000 7f893f7c9740 1 -- 172.21.15.7:0/577907061 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f891800e340 con 0x55fa9d10bea0 2024-04-21T08:56:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.926+0000 7f893f7c9740 1 -- 172.21.15.7:0/577907061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8918010ba0 con 0x55fa9d10b580 2024-04-21T08:56:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.926+0000 7f8936d9d640 1 --2- 172.21.15.7:0/577907061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d112310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.926+0000 7f893f7c9740 1 -- 172.21.15.7:0/577907061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f891800e990 con 0x55fa9d114a80 2024-04-21T08:56:10.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.926+0000 7f8935d9b640 1 --2- 172.21.15.7:0/577907061 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 0x55fa9d1207c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.926+0000 7f893659c640 1 -- 172.21.15.7:0/577907061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 msgr2=0x55fa9d112310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.926+0000 7f893659c640 1 --2- 172.21.15.7:0/577907061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d112310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.926+0000 7f893659c640 1 -- 172.21.15.7:0/577907061 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 msgr2=0x55fa9d1207c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.926+0000 7f893659c640 1 --2- 172.21.15.7:0/577907061 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 0x55fa9d1207c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.926+0000 7f893659c640 1 -- 172.21.15.7:0/577907061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa9d10ffc0 con 0x55fa9d10b580 2024-04-21T08:56:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.927+0000 7f893659c640 1 --2- 172.21.15.7:0/577907061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d1109b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f89280bdd20 tx=0x7f89280f2040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93464c340b5305a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.927+0000 7f891ffff640 1 -- 172.21.15.7:0/577907061 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89280dc070 con 0x55fa9d10b580 2024-04-21T08:56:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.927+0000 7f8936d9d640 1 --2- 172.21.15.7:0/577907061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d112310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:56:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.928+0000 7f891ffff640 1 -- 172.21.15.7:0/577907061 <== mon.2 v2:172.21.15.16:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89280bdd90 con 0x55fa9d10b580 2024-04-21T08:56:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.929+0000 7f893f7c9740 1 -- 172.21.15.7:0/577907061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 msgr2=0x55fa9d1109b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.929+0000 7f893f7c9740 1 --2- 172.21.15.7:0/577907061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d1109b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f89280bdd20 tx=0x7f89280f2040 comp rx=0 tx=0).stop 2024-04-21T08:56:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.929+0000 7f893f7c9740 1 -- 172.21.15.7:0/577907061 shutdown_connections 2024-04-21T08:56:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.929+0000 7f893f7c9740 1 --2- 172.21.15.7:0/577907061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d114a80 0x55fa9d112310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.929+0000 7f893f7c9740 1 --2- 172.21.15.7:0/577907061 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d10bea0 0x55fa9d1207c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.929+0000 7f893f7c9740 1 --2- 172.21.15.7:0/577907061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10b580 0x55fa9d1109b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.929+0000 7f893f7c9740 1 -- 172.21.15.7:0/577907061 >> 172.21.15.7:0/577907061 conn(0x55fa9d105cb0 msgr2=0x55fa9d104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.929+0000 7f893f7c9740 1 -- 172.21.15.7:0/577907061 shutdown_connections 2024-04-21T08:56:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.930+0000 7f893f7c9740 1 -- 172.21.15.7:0/577907061 wait complete. 2024-04-21T08:56:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.930+0000 7f893f7c9740 1 -- start start 2024-04-21T08:56:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.930+0000 7f893f7c9740 1 Processor -- start 2024-04-21T08:56:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.930+0000 7f893f7c9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d10b580 0x55fa9d11d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.930+0000 7f893f7c9740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10bea0 0x55fa9d10e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.930+0000 7f893f7c9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d114a80 0x55fa9d10e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f893f7c9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f891800f060 con 0x55fa9d114a80 2024-04-21T08:56:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f893f7c9740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89180104d0 con 0x55fa9d10bea0 2024-04-21T08:56:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f893f7c9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f891800f730 con 0x55fa9d10b580 2024-04-21T08:56:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f893f7c9740 10 client.0 _ll_register_callbacks cb 0x7f8930009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f8935d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10bea0 0x55fa9d10e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f8935d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10bea0 0x55fa9d10e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:47956/0 (socket says 172.21.15.7:47956) 2024-04-21T08:56:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f8935d9b640 1 -- 172.21.15.7:0/2679336141 learned_addr learned my addr 172.21.15.7:0/2679336141 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f893659c640 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d10b580 0x55fa9d11d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f8935d9b640 1 -- 172.21.15.7:0/2679336141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d10b580 msgr2=0x55fa9d11d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:10.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f8935d9b640 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d10b580 0x55fa9d11d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f8935d9b640 1 -- 172.21.15.7:0/2679336141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d114a80 msgr2=0x55fa9d10e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:10.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f8935d9b640 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d114a80 0x55fa9d10e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.931+0000 7f8935d9b640 1 -- 172.21.15.7:0/2679336141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa9d10f190 con 0x55fa9d10bea0 2024-04-21T08:56:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.932+0000 7f8935d9b640 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10bea0 0x55fa9d10e420 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8938070990 tx=0x7f8920004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50df4fde266d09b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.932+0000 7f891dffb640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-21T08:56:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.936+0000 7f891dffb640 1 -- 172.21.15.7:0/2679336141 <== mon.2 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892001ece0 con 0x55fa9d10bea0 2024-04-21T08:56:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.936+0000 7f893f7c9740 1 -- 172.21.15.7:0/2679336141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89280bdd90 con 0x55fa9d10bea0 2024-04-21T08:56:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.936+0000 7f891dffb640 1 -- 172.21.15.7:0/2679336141 <== mon.2 v2:172.21.15.16:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9764+0+0 (secure 0 0 0) 0x7f8920005020 con 0x55fa9d10bea0 2024-04-21T08:56:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.936+0000 7f891dffb640 1 -- 172.21.15.7:0/2679336141 <== mon.2 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8920032050 con 0x55fa9d10bea0 2024-04-21T08:56:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.937+0000 7f893f7c9740 10 client.26936 fetch_fsmap learned FSMap version 23 2024-04-21T08:56:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.937+0000 7f893f7c9740 1 -- 172.21.15.7:0/2679336141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fa9d1acc60 con 0x55fa9d10bea0 2024-04-21T08:56:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.944+0000 7f891dffb640 1 -- 172.21.15.7:0/2679336141 <== mon.2 v2:172.21.15.16:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89280bdd90 con 0x55fa9d10bea0 2024-04-21T08:56:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.944+0000 7f891dffb640 1 -- 172.21.15.7:0/2679336141 <== mon.2 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f892001c050 con 0x55fa9d10bea0 2024-04-21T08:56:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.944+0000 7f893f7c9740 10 client.26936 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:56:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.944+0000 7f893f7c9740 10 client.26936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.944+0000 7f893f7c9740 20 client.26936 populate_metadata read hostname 'smithi007' 2024-04-21T08:56:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.944+0000 7f893f7c9740 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fa9d1ad3f0 0x55fa9d1af810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.944+0000 7f893f7c9740 4 client.26936 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:56:10.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.944+0000 7f893f7c9740 1 -- 172.21.15.7:0/2679336141 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fa9d077a70 con 0x55fa9d1ad3f0 2024-04-21T08:56:10.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.948+0000 7f893659c640 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fa9d1ad3f0 0x55fa9d1af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:10.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.955+0000 7f893659c640 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fa9d1ad3f0 0x55fa9d1af810 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cc07394aac7bbc0 server_cookie=66996cbaaafe8ee5 in_seq=0 out_seq=0 2024-04-21T08:56:10.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.955+0000 7f891dffb640 10 client.26936 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:10.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.955+0000 7f891dffb640 1 -- 172.21.15.7:0/2679336141 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x55fa9d077a70 con 0x55fa9d1ad3f0 2024-04-21T08:56:10.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.955+0000 7f891dffb640 10 client.26936 handle_command_reply: tid=0 2024-04-21T08:56:10.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.956+0000 7f893f7c9740 1 client.26936 shutdown 2024-04-21T08:56:10.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.960+0000 7f893f7c9740 1 -- 172.21.15.7:0/2679336141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10bea0 msgr2=0x55fa9d10e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:10.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.960+0000 7f893f7c9740 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10bea0 0x55fa9d10e420 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8938070990 tx=0x7f8920004730 comp rx=0 tx=0).stop 2024-04-21T08:56:10.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.962+0000 7f893f7c9740 1 -- 172.21.15.7:0/2679336141 shutdown_connections 2024-04-21T08:56:10.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.962+0000 7f893f7c9740 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x55fa9d1ad3f0 0x55fa9d1af810 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.962+0000 7f893f7c9740 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fa9d114a80 0x55fa9d10e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.962+0000 7f893f7c9740 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55fa9d10bea0 0x55fa9d10e420 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.962+0000 7f893f7c9740 1 --2- 172.21.15.7:0/2679336141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa9d10b580 0x55fa9d11d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.962+0000 7f893f7c9740 1 -- 172.21.15.7:0/2679336141 >> 172.21.15.7:0/2679336141 conn(0x55fa9d105cb0 msgr2=0x55fa9d106f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:10.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.962+0000 7f891dffb640 0 client.26936 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:10.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.964+0000 7f893f7c9740 1 -- 172.21.15.7:0/2679336141 shutdown_connections 2024-04-21T08:56:10.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.965+0000 7f893f7c9740 1 -- 172.21.15.7:0/2679336141 wait complete. 2024-04-21T08:56:10.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.968+0000 7f893f7c9740 20 client.26936 trim_cache size 0 max 16384 2024-04-21T08:56:10.973 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:56:10.973 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (17870 inodes in the stack)", 2024-04-21T08:56:10.973 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-04-21T08:56:10.973 INFO:teuthology.orchestra.run.smithi007.stdout: "08447483-b5f8-4de7-a923-832d7f72e5ac": { 2024-04-21T08:56:10.973 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-04-21T08:56:10.973 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "08447483-b5f8-4de7-a923-832d7f72e5ac", 2024-04-21T08:56:10.974 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-04-21T08:56:10.974 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:10.974 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:10.974 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:56:10.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.977+0000 7f893cafd640 1 -- 172.21.15.7:0/394011656 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89240a3f20 msgr2=0x7f89240a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:10.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.977+0000 7f893cafd640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89240a3f20 0x7f89240a63e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55fa9d10b050 tx=0x7f892000a2d0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.977+0000 7f893cafd640 1 -- 172.21.15.7:0/394011656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893810e1b0 msgr2=0x7f893811ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:10.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.977+0000 7f893cafd640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893810e1b0 0x7f893811ed80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f892802c290 tx=0x7f892802eb90 comp rx=0 tx=0).stop 2024-04-21T08:56:10.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.978+0000 7f893cafd640 1 -- 172.21.15.7:0/394011656 shutdown_connections 2024-04-21T08:56:10.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.978+0000 7f893cafd640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8938110b10 0x7f8938119380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.978+0000 7f893cafd640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89381106a0 0x7f8938118e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.978+0000 7f893cafd640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f89240a3f20 0x7f89240a63e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.978+0000 7f893cafd640 1 --2- 172.21.15.7:0/394011656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f893810e1b0 0x7f893811ed80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:10.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.978+0000 7f893cafd640 1 -- 172.21.15.7:0/394011656 >> 172.21.15.7:0/394011656 conn(0x7f893806e360 msgr2=0x7f893810a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:10.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.980+0000 7f893cafd640 1 -- 172.21.15.7:0/394011656 shutdown_connections 2024-04-21T08:56:10.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:10.980+0000 7f893cafd640 1 -- 172.21.15.7:0/394011656 wait complete. 2024-04-21T08:56:10.999 INFO:tasks.cephfs.filesystem:scrub status for tag:08447483-b5f8-4de7-a923-832d7f72e5ac - {'path': '/', 'tag': '08447483-b5f8-4de7-a923-832d7f72e5ac', 'options': 'recursive,force'} 2024-04-21T08:56:11.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:11 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:56:11.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:11 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:56:11.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:11 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:11.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:11 smithi077 ceph-mon[31615]: osdmap e110: 12 total, 12 up, 12 in 2024-04-21T08:56:11.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:11 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:56:11.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:11 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:56:11.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:11 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:11.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:11 smithi007 ceph-mon[28477]: osdmap e110: 12 total, 12 up, 12 in 2024-04-21T08:56:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:11 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:56:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:11 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:56:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:11 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:56:12.055 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:11 smithi016 ceph-mon[31259]: osdmap e110: 12 total, 12 up, 12 in 2024-04-21T08:56:12.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:12 smithi077 ceph-mon[31615]: pgmap v363: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 1.39k op/s 2024-04-21T08:56:12.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:12 smithi007 ceph-mon[28477]: pgmap v363: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 1.39k op/s 2024-04-21T08:56:13.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:12 smithi016 ceph-mon[31259]: pgmap v363: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 1.39k op/s 2024-04-21T08:56:14.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:14 smithi007 ceph-mon[28477]: pgmap v364: 129 pgs: 18 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 62 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 1.14k op/s 2024-04-21T08:56:15.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:14 smithi016 ceph-mon[31259]: pgmap v364: 129 pgs: 18 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 62 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 1.14k op/s 2024-04-21T08:56:15.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:14 smithi077 ceph-mon[31615]: pgmap v364: 129 pgs: 18 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 62 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 1.14k op/s 2024-04-21T08:56:17.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:17 smithi016 ceph-mon[31259]: pgmap v365: 129 pgs: 14 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 90 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1020 KiB/s wr, 869 op/s 2024-04-21T08:56:17.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:17 smithi077 ceph-mon[31615]: pgmap v365: 129 pgs: 14 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 90 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1020 KiB/s wr, 869 op/s 2024-04-21T08:56:17.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:17 smithi007 ceph-mon[28477]: pgmap v365: 129 pgs: 14 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 90 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1020 KiB/s wr, 869 op/s 2024-04-21T08:56:19.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:19 smithi016 ceph-mon[31259]: pgmap v366: 129 pgs: 13 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 96 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 458 KiB/s wr, 759 op/s 2024-04-21T08:56:19.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:19 smithi077 ceph-mon[31615]: pgmap v366: 129 pgs: 13 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 96 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 458 KiB/s wr, 759 op/s 2024-04-21T08:56:19.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:19 smithi007 ceph-mon[28477]: pgmap v366: 129 pgs: 13 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 96 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 458 KiB/s wr, 759 op/s 2024-04-21T08:56:21.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:21 smithi016 ceph-mon[31259]: pgmap v367: 129 pgs: 8 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 106 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 254 KiB/s wr, 1.00k op/s 2024-04-21T08:56:21.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:21 smithi016 ceph-mon[31259]: osdmap e111: 12 total, 12 up, 12 in 2024-04-21T08:56:21.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:21 smithi077 ceph-mon[31615]: pgmap v367: 129 pgs: 8 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 106 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 254 KiB/s wr, 1.00k op/s 2024-04-21T08:56:21.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:21 smithi077 ceph-mon[31615]: osdmap e111: 12 total, 12 up, 12 in 2024-04-21T08:56:21.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:21 smithi007 ceph-mon[28477]: pgmap v367: 129 pgs: 8 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 106 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 254 KiB/s wr, 1.00k op/s 2024-04-21T08:56:21.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:21 smithi007 ceph-mon[28477]: osdmap e111: 12 total, 12 up, 12 in 2024-04-21T08:56:23.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:23 smithi016 ceph-mon[31259]: pgmap v369: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 252 KiB/s wr, 1.34k op/s 2024-04-21T08:56:23.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:23 smithi016 ceph-mon[31259]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:23.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:23 smithi077 ceph-mon[31615]: pgmap v369: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 252 KiB/s wr, 1.34k op/s 2024-04-21T08:56:23.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:23 smithi077 ceph-mon[31615]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:23.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:23 smithi007 ceph-mon[28477]: pgmap v369: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 252 KiB/s wr, 1.34k op/s 2024-04-21T08:56:23.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:23 smithi007 ceph-mon[28477]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:25 smithi016 ceph-mon[31259]: pgmap v370: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 27 KiB/s wr, 2.16k op/s 2024-04-21T08:56:25.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:25 smithi077 ceph-mon[31615]: pgmap v370: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 27 KiB/s wr, 2.16k op/s 2024-04-21T08:56:25.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:25 smithi007 ceph-mon[28477]: pgmap v370: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 27 KiB/s wr, 2.16k op/s 2024-04-21T08:56:27.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:27 smithi016 ceph-mon[31259]: pgmap v371: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 27 KiB/s wr, 2.52k op/s 2024-04-21T08:56:27.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:27 smithi016 ceph-mon[31259]: osdmap e112: 12 total, 12 up, 12 in 2024-04-21T08:56:27.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:27 smithi077 ceph-mon[31615]: pgmap v371: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 27 KiB/s wr, 2.52k op/s 2024-04-21T08:56:27.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:27 smithi077 ceph-mon[31615]: osdmap e112: 12 total, 12 up, 12 in 2024-04-21T08:56:27.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:27 smithi007 ceph-mon[28477]: pgmap v371: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 27 KiB/s wr, 2.52k op/s 2024-04-21T08:56:27.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:27 smithi007 ceph-mon[28477]: osdmap e112: 12 total, 12 up, 12 in 2024-04-21T08:56:28.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:28 smithi016 ceph-mon[31259]: scrub summary: idle 2024-04-21T08:56:28.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:28 smithi077 ceph-mon[31615]: scrub summary: idle 2024-04-21T08:56:28.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:28 smithi007 ceph-mon[28477]: scrub summary: idle 2024-04-21T08:56:29.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:29 smithi016 ceph-mon[31259]: pgmap v373: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 67 KiB/s wr, 2.70k op/s 2024-04-21T08:56:29.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:29 smithi077 ceph-mon[31615]: pgmap v373: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 67 KiB/s wr, 2.70k op/s 2024-04-21T08:56:29.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:29 smithi007 ceph-mon[28477]: pgmap v373: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 67 KiB/s wr, 2.70k op/s 2024-04-21T08:56:31.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:31 smithi016 ceph-mon[31259]: pgmap v374: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 64 KiB/s wr, 2.10k op/s 2024-04-21T08:56:31.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:31 smithi016 ceph-mon[31259]: osdmap e113: 12 total, 12 up, 12 in 2024-04-21T08:56:31.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:31 smithi077 ceph-mon[31615]: pgmap v374: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 64 KiB/s wr, 2.10k op/s 2024-04-21T08:56:31.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:31 smithi077 ceph-mon[31615]: osdmap e113: 12 total, 12 up, 12 in 2024-04-21T08:56:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:31 smithi007 ceph-mon[28477]: pgmap v374: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 64 KiB/s wr, 2.10k op/s 2024-04-21T08:56:31.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:31 smithi007 ceph-mon[28477]: osdmap e113: 12 total, 12 up, 12 in 2024-04-21T08:56:33.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:33 smithi016 ceph-mon[31259]: pgmap v376: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 48 KiB/s wr, 957 op/s 2024-04-21T08:56:33.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:33 smithi077 ceph-mon[31615]: pgmap v376: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 48 KiB/s wr, 957 op/s 2024-04-21T08:56:33.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:33 smithi007 ceph-mon[28477]: pgmap v376: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 48 KiB/s wr, 957 op/s 2024-04-21T08:56:35.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:35 smithi016 ceph-mon[31259]: pgmap v377: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 56 KiB/s wr, 352 op/s 2024-04-21T08:56:35.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:35 smithi077 ceph-mon[31615]: pgmap v377: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 56 KiB/s wr, 352 op/s 2024-04-21T08:56:35.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:35 smithi007 ceph-mon[28477]: pgmap v377: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 56 KiB/s wr, 352 op/s 2024-04-21T08:56:37.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:37 smithi016 ceph-mon[31259]: pgmap v378: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 48 KiB/s wr, 186 op/s 2024-04-21T08:56:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:37 smithi077 ceph-mon[31615]: pgmap v378: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 48 KiB/s wr, 186 op/s 2024-04-21T08:56:37.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:37 smithi007 ceph-mon[28477]: pgmap v378: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 48 KiB/s wr, 186 op/s 2024-04-21T08:56:39.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:39 smithi016 ceph-mon[31259]: pgmap v379: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.8 KiB/s wr, 2 op/s 2024-04-21T08:56:39.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:39 smithi077 ceph-mon[31615]: pgmap v379: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.8 KiB/s wr, 2 op/s 2024-04-21T08:56:39.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:39 smithi007 ceph-mon[28477]: pgmap v379: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.8 KiB/s wr, 2 op/s 2024-04-21T08:56:41.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:56:41.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:56:41.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.119+0000 7f831ba70640 1 -- 172.21.15.7:0/162551857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8314101240 msgr2=0x7f831410d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.119+0000 7f831ba70640 1 --2- 172.21.15.7:0/162551857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8314101240 0x7f831410d430 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f830c009ca0 tx=0x7f830c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 -- 172.21.15.7:0/162551857 shutdown_connections 2024-04-21T08:56:41.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 --2- 172.21.15.7:0/162551857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8314101240 0x7f831410d430 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 --2- 172.21.15.7:0/162551857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8314100880 0x7f8314100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 --2- 172.21.15.7:0/162551857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8314106670 0x7f8314106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 -- 172.21.15.7:0/162551857 >> 172.21.15.7:0/162551857 conn(0x7f83140760d0 msgr2=0x7f83140764e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 -- 172.21.15.7:0/162551857 shutdown_connections 2024-04-21T08:56:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 -- 172.21.15.7:0/162551857 wait complete. 2024-04-21T08:56:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 Processor -- start 2024-04-21T08:56:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 -- start start 2024-04-21T08:56:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8314100880 0x7f83141986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8314101240 0x7f8314198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8314106670 0x7f831419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8314112220 con 0x7f8314100880 2024-04-21T08:56:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83141120a0 con 0x7f8314101240 2024-04-21T08:56:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.120+0000 7f831ba70640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f83141123a0 con 0x7f8314106670 2024-04-21T08:56:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831980c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8314100880 0x7f83141986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831980c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8314100880 0x7f83141986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56574/0 (socket says 172.21.15.7:56574) 2024-04-21T08:56:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831980c640 1 -- 172.21.15.7:0/552635048 learned_addr learned my addr 172.21.15.7:0/552635048 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831900b640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8314101240 0x7f8314198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831a00d640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8314106670 0x7f831419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831980c640 1 -- 172.21.15.7:0/552635048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8314106670 msgr2=0x7f831419cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831980c640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8314106670 0x7f831419cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831980c640 1 -- 172.21.15.7:0/552635048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8314101240 msgr2=0x7f8314198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831980c640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8314101240 0x7f8314198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831980c640 1 -- 172.21.15.7:0/552635048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f831419d6b0 con 0x7f8314100880 2024-04-21T08:56:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831980c640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8314100880 0x7f83141986c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f831000e9f0 tx=0x7f831000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f830affd640 1 -- 172.21.15.7:0/552635048 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831000cd80 con 0x7f8314100880 2024-04-21T08:56:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f830affd640 1 -- 172.21.15.7:0/552635048 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83100108e0 con 0x7f8314100880 2024-04-21T08:56:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831ba70640 1 -- 172.21.15.7:0/552635048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f831419d9a0 con 0x7f8314100880 2024-04-21T08:56:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f830affd640 1 -- 172.21.15.7:0/552635048 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310018980 con 0x7f8314100880 2024-04-21T08:56:41.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.121+0000 7f831ba70640 1 -- 172.21.15.7:0/552635048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83141a5230 con 0x7f8314100880 2024-04-21T08:56:41.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.124+0000 7f830affd640 1 -- 172.21.15.7:0/552635048 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f8310010430 con 0x7f8314100880 2024-04-21T08:56:41.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.124+0000 7f831e65a740 1 Processor -- start 2024-04-21T08:56:41.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.124+0000 7f831e65a740 1 -- start start 2024-04-21T08:56:41.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831e65a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e14c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831e65a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14cea0 0x56552e155540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831e65a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e155a80 0x56552e157e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831e65a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56552e0b11e0 con 0x56552e14c580 2024-04-21T08:56:41.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831e65a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56552e04e390 con 0x56552e14cea0 2024-04-21T08:56:41.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831e65a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56552e0ad1a0 con 0x56552e155a80 2024-04-21T08:56:41.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f830affd640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f83000a3d20 0x7f83000a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831a00d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831a00d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e14c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56588/0 (socket says 172.21.15.7:56588) 2024-04-21T08:56:41.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831a00d640 1 -- 172.21.15.7:0/3532732681 learned_addr learned my addr 172.21.15.7:0/3532732681 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831980c640 1 --2- 172.21.15.7:0/3532732681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14cea0 0x56552e155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831900b640 1 --2- 172.21.15.7:0/3532732681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e155a80 0x56552e157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831a00d640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f83000a3d20 0x7f83000a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f830affd640 1 -- 172.21.15.7:0/552635048 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7f83100d0e60 con 0x7f8314100880 2024-04-21T08:56:41.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831a00d640 1 -- 172.21.15.7:0/3532732681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e155a80 msgr2=0x56552e157e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831a00d640 1 --2- 172.21.15.7:0/3532732681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e155a80 0x56552e157e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831a00d640 1 -- 172.21.15.7:0/3532732681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14cea0 msgr2=0x56552e155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831a00d640 1 --2- 172.21.15.7:0/3532732681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14cea0 0x56552e155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.125+0000 7f831a00d640 1 -- 172.21.15.7:0/3532732681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56552de8d7d0 con 0x56552e14c580 2024-04-21T08:56:41.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.126+0000 7f831a00d640 1 --2- 172.21.15.7:0/3532732681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e14c960 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f830c009430 tx=0x7f830c032bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d855d06e763eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.126+0000 7f82fffff640 1 -- 172.21.15.7:0/3532732681 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830c050070 con 0x56552e14c580 2024-04-21T08:56:41.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.126+0000 7f82fffff640 1 -- 172.21.15.7:0/3532732681 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f830c003b70 con 0x56552e14c580 2024-04-21T08:56:41.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.126+0000 7f82fffff640 1 -- 172.21.15.7:0/3532732681 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830c039040 con 0x56552e14c580 2024-04-21T08:56:41.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.126+0000 7f831e65a740 1 -- 172.21.15.7:0/3532732681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 msgr2=0x56552e14c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.126+0000 7f831a00d640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f83000a3d20 0x7f83000a61e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f830c009a60 tx=0x7f830c048000 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.126+0000 7f831e65a740 1 --2- 172.21.15.7:0/3532732681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e14c960 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f830c009430 tx=0x7f830c032bc0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.127+0000 7f831e65a740 1 -- 172.21.15.7:0/3532732681 shutdown_connections 2024-04-21T08:56:41.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.127+0000 7f831e65a740 1 --2- 172.21.15.7:0/3532732681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e155a80 0x56552e157e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.127+0000 7f831e65a740 1 --2- 172.21.15.7:0/3532732681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14cea0 0x56552e155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.127+0000 7f831e65a740 1 --2- 172.21.15.7:0/3532732681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e14c960 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.127+0000 7f831e65a740 1 -- 172.21.15.7:0/3532732681 >> 172.21.15.7:0/3532732681 conn(0x56552e146cb0 msgr2=0x56552e147090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.127+0000 7f831e65a740 1 -- 172.21.15.7:0/3532732681 shutdown_connections 2024-04-21T08:56:41.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.127+0000 7f831e65a740 1 -- 172.21.15.7:0/3532732681 wait complete. 2024-04-21T08:56:41.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.127+0000 7f831e65a740 1 -- start start 2024-04-21T08:56:41.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.127+0000 7f831e65a740 1 Processor -- start 2024-04-21T08:56:41.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831e65a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e152a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831e65a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831e65a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e155a80 0x56552e153500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831e65a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56552e158870 con 0x56552e14c580 2024-04-21T08:56:41.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831e65a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56552e06a2f0 con 0x56552e155a80 2024-04-21T08:56:41.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831e65a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56552e1589f0 con 0x56552e14cea0 2024-04-21T08:56:41.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831e65a740 10 client.0 _ll_register_callbacks cb 0x56552e077580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:41.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831980c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831980c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e152a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56596/0 (socket says 172.21.15.7:56596) 2024-04-21T08:56:41.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831980c640 1 -- 172.21.15.7:0/239694526 learned_addr learned my addr 172.21.15.7:0/239694526 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831900b640 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831a00d640 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e155a80 0x56552e153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831980c640 1 -- 172.21.15.7:0/239694526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 msgr2=0x56552e152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831980c640 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831980c640 1 -- 172.21.15.7:0/239694526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e155a80 msgr2=0x56552e153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831980c640 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e155a80 0x56552e153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831980c640 1 -- 172.21.15.7:0/239694526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56552ddc33f0 con 0x56552e14c580 2024-04-21T08:56:41.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831980c640 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e152a80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8310099480 tx=0x7f831006b520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02c951bbd32ee59 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f82fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:56:41.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f82fd7fa640 1 -- 172.21.15.7:0/239694526 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310026040 con 0x56552e14c580 2024-04-21T08:56:41.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f82fd7fa640 1 -- 172.21.15.7:0/239694526 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7f83100bbbc0 con 0x56552e14c580 2024-04-21T08:56:41.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.128+0000 7f831e65a740 1 -- 172.21.15.7:0/239694526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f830c003b70 con 0x56552e14c580 2024-04-21T08:56:41.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f82fd7fa640 1 -- 172.21.15.7:0/239694526 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f83100ce630 con 0x56552e14c580 2024-04-21T08:56:41.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f831e65a740 10 client.17298 fetch_fsmap learned FSMap version 23 2024-04-21T08:56:41.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f831e65a740 1 -- 172.21.15.7:0/239694526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56552dd830f0 con 0x56552e14c580 2024-04-21T08:56:41.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f82fd7fa640 1 -- 172.21.15.7:0/239694526 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f830c003b70 con 0x56552e14c580 2024-04-21T08:56:41.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f82fd7fa640 1 -- 172.21.15.7:0/239694526 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f831006b160 con 0x56552e14c580 2024-04-21T08:56:41.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f831e65a740 10 client.17298 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:56:41.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f831e65a740 10 client.17298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:41.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f831e65a740 20 client.17298 populate_metadata read hostname 'smithi007' 2024-04-21T08:56:41.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f831e65a740 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56552e1658c0 0x56552e167cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f831e65a740 4 client.17298 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:56:41.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f831e65a740 1 -- 172.21.15.7:0/239694526 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56552e0bec00 con 0x56552e1658c0 2024-04-21T08:56:41.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.129+0000 7f831900b640 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56552e1658c0 0x56552e167cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.130+0000 7f82fd7fa640 10 client.17298 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:41.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.130+0000 7f831900b640 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56552e1658c0 0x56552e167cf0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24cc975c54987da3 server_cookie=b5bd73a37af2df5f in_seq=0 out_seq=0 2024-04-21T08:56:41.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f82fd7fa640 1 -- 172.21.15.7:0/239694526 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56552e0bec00 con 0x56552e1658c0 2024-04-21T08:56:41.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f82fd7fa640 10 client.17298 handle_command_reply: tid=0 2024-04-21T08:56:41.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 1 client.17298 shutdown 2024-04-21T08:56:41.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 1 -- 172.21.15.7:0/239694526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 msgr2=0x56552e152a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e152a80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8310099480 tx=0x7f831006b520 comp rx=0 tx=0).stop 2024-04-21T08:56:41.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 1 -- 172.21.15.7:0/239694526 shutdown_connections 2024-04-21T08:56:41.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56552e1658c0 0x56552e167cf0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e155a80 0x56552e153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 1 --2- 172.21.15.7:0/239694526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e14c580 0x56552e152a80 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f82fd7fa640 0 client.17298 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:41.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 1 -- 172.21.15.7:0/239694526 >> 172.21.15.7:0/239694526 conn(0x56552e146cb0 msgr2=0x56552e148a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 1 -- 172.21.15.7:0/239694526 shutdown_connections 2024-04-21T08:56:41.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 1 -- 172.21.15.7:0/239694526 wait complete. 2024-04-21T08:56:41.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.131+0000 7f831e65a740 20 client.17298 trim_cache size 0 max 16384 2024-04-21T08:56:41.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831e65a740 1 Processor -- start 2024-04-21T08:56:41.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831e65a740 1 -- start start 2024-04-21T08:56:41.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831e65a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14c580 0x56552e1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831e65a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e161730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831e65a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e148970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831e65a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f83141ad560 con 0x56552e155a80 2024-04-21T08:56:41.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831e65a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83141afe00 con 0x56552e14c580 2024-04-21T08:56:41.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831e65a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f83141adbf0 con 0x56552e14cea0 2024-04-21T08:56:41.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831a00d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831a00d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e148970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56604/0 (socket says 172.21.15.7:56604) 2024-04-21T08:56:41.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831a00d640 1 -- 172.21.15.7:0/3846165650 learned_addr learned my addr 172.21.15.7:0/3846165650 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831980c640 1 --2- 172.21.15.7:0/3846165650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14c580 0x56552e1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831900b640 1 --2- 172.21.15.7:0/3846165650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e161730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831a00d640 1 -- 172.21.15.7:0/3846165650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 msgr2=0x56552e161730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.149+0000 7f831a00d640 1 --2- 172.21.15.7:0/3846165650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e161730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.150+0000 7f831a00d640 1 -- 172.21.15.7:0/3846165650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14c580 msgr2=0x56552e1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.150+0000 7f831a00d640 1 --2- 172.21.15.7:0/3846165650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14c580 0x56552e1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.150+0000 7f831a00d640 1 -- 172.21.15.7:0/3846165650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56552e14ccd0 con 0x56552e155a80 2024-04-21T08:56:41.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.150+0000 7f831a00d640 1 --2- 172.21.15.7:0/3846165650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e148970 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f830c009ca0 tx=0x7f830c02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd52ae97494171ce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.150+0000 7f82ff7fe640 1 -- 172.21.15.7:0/3846165650 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830c039040 con 0x56552e155a80 2024-04-21T08:56:41.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.150+0000 7f82ff7fe640 1 -- 172.21.15.7:0/3846165650 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f830c0044f0 con 0x56552e155a80 2024-04-21T08:56:41.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.150+0000 7f82ff7fe640 1 -- 172.21.15.7:0/3846165650 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830c050040 con 0x56552e155a80 2024-04-21T08:56:41.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.150+0000 7f831e65a740 1 -- 172.21.15.7:0/3846165650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 msgr2=0x56552e148970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.150+0000 7f831e65a740 1 --2- 172.21.15.7:0/3846165650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e148970 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f830c009ca0 tx=0x7f830c02e040 comp rx=0 tx=0).stop 2024-04-21T08:56:41.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 -- 172.21.15.7:0/3846165650 shutdown_connections 2024-04-21T08:56:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 --2- 172.21.15.7:0/3846165650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e148970 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 --2- 172.21.15.7:0/3846165650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e161730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 --2- 172.21.15.7:0/3846165650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14c580 0x56552e1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 -- 172.21.15.7:0/3846165650 >> 172.21.15.7:0/3846165650 conn(0x56552e146cb0 msgr2=0x56552e1519c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 -- 172.21.15.7:0/3846165650 shutdown_connections 2024-04-21T08:56:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 -- 172.21.15.7:0/3846165650 wait complete. 2024-04-21T08:56:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 -- start start 2024-04-21T08:56:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 Processor -- start 2024-04-21T08:56:41.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14c580 0x56552e159e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e15a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e1cae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f83141ae2c0 con 0x56552e155a80 2024-04-21T08:56:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83141af730 con 0x56552e14c580 2024-04-21T08:56:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f83141ae990 con 0x56552e14cea0 2024-04-21T08:56:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.151+0000 7f831e65a740 10 client.0 _ll_register_callbacks cb 0x7f8310098e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831a00d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e1cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831a00d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e1cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56620/0 (socket says 172.21.15.7:56620) 2024-04-21T08:56:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831a00d640 1 -- 172.21.15.7:0/346385063 learned_addr learned my addr 172.21.15.7:0/346385063 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831980c640 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14c580 0x56552e159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831900b640 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e15a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831a00d640 1 -- 172.21.15.7:0/346385063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 msgr2=0x56552e15a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831a00d640 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e15a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831a00d640 1 -- 172.21.15.7:0/346385063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14c580 msgr2=0x56552e159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831a00d640 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14c580 0x56552e159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831a00d640 1 -- 172.21.15.7:0/346385063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56552e1cb3c0 con 0x56552e155a80 2024-04-21T08:56:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831900b640 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e15a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:56:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831a00d640 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e1cae30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f830c02cf50 tx=0x7f830c009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76a9f9a1481c070 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f82fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:56:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f82fd7fa640 1 -- 172.21.15.7:0/346385063 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830c064060 con 0x56552e155a80 2024-04-21T08:56:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f82fd7fa640 1 -- 172.21.15.7:0/346385063 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7f830c004240 con 0x56552e155a80 2024-04-21T08:56:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831e65a740 1 -- 172.21.15.7:0/346385063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f830c0044f0 con 0x56552e155a80 2024-04-21T08:56:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f82fd7fa640 1 -- 172.21.15.7:0/346385063 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f830c050070 con 0x56552e155a80 2024-04-21T08:56:41.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831e65a740 10 client.17304 fetch_fsmap learned FSMap version 23 2024-04-21T08:56:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.152+0000 7f831e65a740 1 -- 172.21.15.7:0/346385063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56552e1cc6d0 con 0x56552e155a80 2024-04-21T08:56:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.153+0000 7f82fd7fa640 1 -- 172.21.15.7:0/346385063 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f830c0044f0 con 0x56552e155a80 2024-04-21T08:56:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.153+0000 7f82fd7fa640 1 -- 172.21.15.7:0/346385063 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f830c03f070 con 0x56552e155a80 2024-04-21T08:56:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.153+0000 7f831e65a740 10 client.17304 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:56:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.153+0000 7f831e65a740 10 client.17304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.153+0000 7f831e65a740 20 client.17304 populate_metadata read hostname 'smithi007' 2024-04-21T08:56:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.153+0000 7f831e65a740 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56552e1cc940 0x56552e1ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.153+0000 7f831e65a740 4 client.17304 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:56:41.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.153+0000 7f831e65a740 1 -- 172.21.15.7:0/346385063 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56552e0b8a70 con 0x56552e1cc940 2024-04-21T08:56:41.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.153+0000 7f831980c640 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56552e1cc940 0x56552e1ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.153+0000 7f82fd7fa640 10 client.17304 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:41.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.153+0000 7f831980c640 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56552e1cc940 0x56552e1ced30 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b121836bd9f3041a server_cookie=1d5af3ff8726a61e in_seq=0 out_seq=0 2024-04-21T08:56:41.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f82fd7fa640 1 -- 172.21.15.7:0/346385063 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56552e0b8a70 con 0x56552e1cc940 2024-04-21T08:56:41.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f82fd7fa640 10 client.17304 handle_command_reply: tid=0 2024-04-21T08:56:41.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 1 client.17304 shutdown 2024-04-21T08:56:41.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 1 -- 172.21.15.7:0/346385063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 msgr2=0x56552e1cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e1cae30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f830c02cf50 tx=0x7f830c009800 comp rx=0 tx=0).stop 2024-04-21T08:56:41.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 1 -- 172.21.15.7:0/346385063 shutdown_connections 2024-04-21T08:56:41.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x56552e1cc940 0x56552e1ced30 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56552e155a80 0x56552e1cae30 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56552e14cea0 0x56552e15a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 1 --2- 172.21.15.7:0/346385063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56552e14c580 0x56552e159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f82fd7fa640 0 client.17304 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 1 -- 172.21.15.7:0/346385063 >> 172.21.15.7:0/346385063 conn(0x56552e146cb0 msgr2=0x56552e1532d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 1 -- 172.21.15.7:0/346385063 shutdown_connections 2024-04-21T08:56:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 1 -- 172.21.15.7:0/346385063 wait complete. 2024-04-21T08:56:41.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831e65a740 20 client.17304 trim_cache size 0 max 16384 2024-04-21T08:56:41.165 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:56:41.166 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-04-21T08:56:41.166 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-04-21T08:56:41.166 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:56:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831ba70640 1 -- 172.21.15.7:0/552635048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f83000a3d20 msgr2=0x7f83000a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831ba70640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f83000a3d20 0x7f83000a61e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f830c009a60 tx=0x7f830c048000 comp rx=0 tx=0).stop 2024-04-21T08:56:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831ba70640 1 -- 172.21.15.7:0/552635048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8314100880 msgr2=0x7f83141986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831ba70640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8314100880 0x7f83141986c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f831000e9f0 tx=0x7f831000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831ba70640 1 -- 172.21.15.7:0/552635048 shutdown_connections 2024-04-21T08:56:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831ba70640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f83000a3d20 0x7f83000a61e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.154+0000 7f831ba70640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8314106670 0x7f831419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.155+0000 7f831ba70640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8314101240 0x7f8314198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.155+0000 7f831ba70640 1 --2- 172.21.15.7:0/552635048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8314100880 0x7f83141986c0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.155+0000 7f831ba70640 1 -- 172.21.15.7:0/552635048 >> 172.21.15.7:0/552635048 conn(0x7f83140760d0 msgr2=0x7f83140ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.155+0000 7f831ba70640 1 -- 172.21.15.7:0/552635048 shutdown_connections 2024-04-21T08:56:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.155+0000 7f831ba70640 1 -- 172.21.15.7:0/552635048 wait complete. 2024-04-21T08:56:41.169 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:56:41.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:56:41.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.274+0000 7f55749bd640 1 -- 172.21.15.7:0/580419568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55701006c0 msgr2=0x7f5570100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.274+0000 7f55749bd640 1 --2- 172.21.15.7:0/580419568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55701006c0 0x7f5570100aa0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f5560009ca0 tx=0x7f556002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.275+0000 7f55749bd640 1 -- 172.21.15.7:0/580419568 shutdown_connections 2024-04-21T08:56:41.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.275+0000 7f55749bd640 1 --2- 172.21.15.7:0/580419568 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5570107280 0x7f557010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.275+0000 7f55749bd640 1 --2- 172.21.15.7:0/580419568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5570101070 0x7f5570106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.275+0000 7f55749bd640 1 --2- 172.21.15.7:0/580419568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55701006c0 0x7f5570100aa0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.275+0000 7f55749bd640 1 -- 172.21.15.7:0/580419568 >> 172.21.15.7:0/580419568 conn(0x7f55700773a0 msgr2=0x7f55700fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.275+0000 7f55749bd640 1 -- 172.21.15.7:0/580419568 shutdown_connections 2024-04-21T08:56:41.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.275+0000 7f55749bd640 1 -- 172.21.15.7:0/580419568 wait complete. 2024-04-21T08:56:41.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.275+0000 7f55749bd640 1 Processor -- start 2024-04-21T08:56:41.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.275+0000 7f55749bd640 1 -- start start 2024-04-21T08:56:41.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f55749bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55701006c0 0x7f5570198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f55749bd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5570107280 0x7f5570198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f55749bd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f557019d000 0x7f557019d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f55749bd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f55701121c0 con 0x7f557019d000 2024-04-21T08:56:41.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f55749bd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5570112040 con 0x7f55701006c0 2024-04-21T08:56:41.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f55749bd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5570112340 con 0x7f5570107280 2024-04-21T08:56:41.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f556ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f557019d000 0x7f557019d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f556ed9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f557019d000 0x7f557019d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56652/0 (socket says 172.21.15.7:56652) 2024-04-21T08:56:41.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f556ed9d640 1 -- 172.21.15.7:0/115865410 learned_addr learned my addr 172.21.15.7:0/115865410 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f556e59c640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55701006c0 0x7f5570198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f556dd9b640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5570107280 0x7f5570198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f556ed9d640 1 -- 172.21.15.7:0/115865410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5570107280 msgr2=0x7f5570198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f556ed9d640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5570107280 0x7f5570198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f556ed9d640 1 -- 172.21.15.7:0/115865410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55701006c0 msgr2=0x7f5570198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f556ed9d640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55701006c0 0x7f5570198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f556ed9d640 1 -- 172.21.15.7:0/115865410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557019daf0 con 0x7f557019d000 2024-04-21T08:56:41.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f556ed9d640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f557019d000 0x7f557019d4b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f556800e970 tx=0x7f556800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f555f7fe640 1 -- 172.21.15.7:0/115865410 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556800ccc0 con 0x7f557019d000 2024-04-21T08:56:41.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f555f7fe640 1 -- 172.21.15.7:0/115865410 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f556800ce60 con 0x7f557019d000 2024-04-21T08:56:41.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.276+0000 7f55749bd640 1 -- 172.21.15.7:0/115865410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557019dde0 con 0x7f557019d000 2024-04-21T08:56:41.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.277+0000 7f555f7fe640 1 -- 172.21.15.7:0/115865410 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5568010430 con 0x7f557019d000 2024-04-21T08:56:41.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.277+0000 7f55749bd640 1 -- 172.21.15.7:0/115865410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55701a5580 con 0x7f557019d000 2024-04-21T08:56:41.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.277+0000 7f55749bd640 1 -- 172.21.15.7:0/115865410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5570069250 con 0x7f557019d000 2024-04-21T08:56:41.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.279+0000 7f555f7fe640 1 -- 172.21.15.7:0/115865410 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f5568004ff0 con 0x7f557019d000 2024-04-21T08:56:41.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.279+0000 7f555f7fe640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f55580a3d20 0x7f55580a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.279+0000 7f556e59c640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f55580a3d20 0x7f55580a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.279+0000 7f555f7fe640 1 -- 172.21.15.7:0/115865410 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7f55680d0f90 con 0x7f557019d000 2024-04-21T08:56:41.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.280+0000 7f556e59c640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f55580a3d20 0x7f55580a61e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5560000f80 tx=0x7f5560009360 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.284+0000 7f555f7fe640 1 -- 172.21.15.7:0/115865410 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f556809d230 con 0x7f557019d000 2024-04-21T08:56:41.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:41 smithi007 ceph-mon[28477]: pgmap v380: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.7 KiB/s wr, 2 op/s 2024-04-21T08:56:41.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.514+0000 7f55749bd640 1 -- 172.21.15.7:0/115865410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5570101070 con 0x7f557019d000 2024-04-21T08:56:41.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.515+0000 7f555f7fe640 1 -- 172.21.15.7:0/115865410 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7f556809cb30 con 0x7f557019d000 2024-04-21T08:56:41.515 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:56:41.515 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:56:41.515 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:56:41.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 -- 172.21.15.7:0/115865410 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f55580a3d20 msgr2=0x7f55580a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f55580a3d20 0x7f55580a61e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5560000f80 tx=0x7f5560009360 comp rx=0 tx=0).stop 2024-04-21T08:56:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 -- 172.21.15.7:0/115865410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f557019d000 msgr2=0x7f557019d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f557019d000 0x7f557019d4b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f556800e970 tx=0x7f556800ed70 comp rx=0 tx=0).stop 2024-04-21T08:56:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 -- 172.21.15.7:0/115865410 shutdown_connections 2024-04-21T08:56:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f557019d000 0x7f557019d4b0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f55580a3d20 0x7f55580a61e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5570107280 0x7f5570198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 --2- 172.21.15.7:0/115865410 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55701006c0 0x7f5570198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 -- 172.21.15.7:0/115865410 >> 172.21.15.7:0/115865410 conn(0x7f55700773a0 msgr2=0x7f55701055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 -- 172.21.15.7:0/115865410 shutdown_connections 2024-04-21T08:56:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.517+0000 7f55749bd640 1 -- 172.21.15.7:0/115865410 wait complete. 2024-04-21T08:56:41.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:56:41.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:41 smithi016 ceph-mon[31259]: pgmap v380: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.7 KiB/s wr, 2 op/s 2024-04-21T08:56:41.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:41 smithi077 ceph-mon[31615]: pgmap v380: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.7 KiB/s wr, 2 op/s 2024-04-21T08:56:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.632+0000 7fc316d0e640 1 -- 172.21.15.7:0/3790833097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc310103b70 msgr2=0x7fc31010bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.632+0000 7fc316d0e640 1 --2- 172.21.15.7:0/3790833097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc310103b70 0x7fc31010bfc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fc304009900 tx=0x7fc30402bc30 comp rx=0 tx=0).stop 2024-04-21T08:56:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.632+0000 7fc316d0e640 1 -- 172.21.15.7:0/3790833097 shutdown_connections 2024-04-21T08:56:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.632+0000 7fc316d0e640 1 --2- 172.21.15.7:0/3790833097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc31010c6d0 0x7fc31010eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.632+0000 7fc316d0e640 1 --2- 172.21.15.7:0/3790833097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc310103b70 0x7fc31010bfc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.632+0000 7fc316d0e640 1 --2- 172.21.15.7:0/3790833097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3101031c0 0x7fc3101035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.632+0000 7fc316d0e640 1 -- 172.21.15.7:0/3790833097 >> 172.21.15.7:0/3790833097 conn(0x7fc310077f70 msgr2=0x7fc310078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.632+0000 7fc316d0e640 1 -- 172.21.15.7:0/3790833097 shutdown_connections 2024-04-21T08:56:41.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.632+0000 7fc316d0e640 1 -- 172.21.15.7:0/3790833097 wait complete. 2024-04-21T08:56:41.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc316d0e640 1 Processor -- start 2024-04-21T08:56:41.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc316d0e640 1 -- start start 2024-04-21T08:56:41.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc316d0e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3101031c0 0x7fc31019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc316d0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc31010c6d0 0x7fc31019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc316d0e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3101987e0 0x7fc310198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc316d0e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc310112350 con 0x7fc3101031c0 2024-04-21T08:56:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc316d0e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101121d0 con 0x7fc3101987e0 2024-04-21T08:56:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc316d0e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101124d0 con 0x7fc31010c6d0 2024-04-21T08:56:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc314aaa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3101031c0 0x7fc31019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc314aaa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3101031c0 0x7fc31019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56668/0 (socket says 172.21.15.7:56668) 2024-04-21T08:56:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc314aaa640 1 -- 172.21.15.7:0/86764813 learned_addr learned my addr 172.21.15.7:0/86764813 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc30ffff640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc31010c6d0 0x7fc31019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc3152ab640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3101987e0 0x7fc310198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc314aaa640 1 -- 172.21.15.7:0/86764813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc31010c6d0 msgr2=0x7fc31019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc314aaa640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc31010c6d0 0x7fc31019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.633+0000 7fc314aaa640 1 -- 172.21.15.7:0/86764813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3101987e0 msgr2=0x7fc310198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.634+0000 7fc314aaa640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3101987e0 0x7fc310198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.634+0000 7fc314aaa640 1 -- 172.21.15.7:0/86764813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc310199370 con 0x7fc3101031c0 2024-04-21T08:56:41.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.634+0000 7fc3152ab640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3101987e0 0x7fc310198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:56:41.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.634+0000 7fc314aaa640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3101031c0 0x7fc31019e670 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc30000e9f0 tx=0x7fc30000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.634+0000 7fc30dffb640 1 -- 172.21.15.7:0/86764813 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30000cd80 con 0x7fc3101031c0 2024-04-21T08:56:41.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.634+0000 7fc316d0e640 1 -- 172.21.15.7:0/86764813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc310199660 con 0x7fc3101031c0 2024-04-21T08:56:41.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.634+0000 7fc30dffb640 1 -- 172.21.15.7:0/86764813 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3000054d0 con 0x7fc3101031c0 2024-04-21T08:56:41.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.634+0000 7fc30dffb640 1 -- 172.21.15.7:0/86764813 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300010690 con 0x7fc3101031c0 2024-04-21T08:56:41.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.634+0000 7fc316d0e640 1 -- 172.21.15.7:0/86764813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3101a57c0 con 0x7fc3101031c0 2024-04-21T08:56:41.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.636+0000 7fc30dffb640 1 -- 172.21.15.7:0/86764813 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fc300004e90 con 0x7fc3101031c0 2024-04-21T08:56:41.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.636+0000 7fc3199dc740 1 Processor -- start 2024-04-21T08:56:41.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.636+0000 7fc3199dc740 1 -- start start 2024-04-21T08:56:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587a4d0 0x5646c587a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587ae80 0x5646c5883430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5885e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc3199dc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5646c57df1f0 con 0x5646c5883a00 2024-04-21T08:56:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc3199dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5646c577c4b0 con 0x5646c587ae80 2024-04-21T08:56:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc3199dc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5646c57db1b0 con 0x5646c587a4d0 2024-04-21T08:56:41.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc30dffb640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc2f80a3d20 0x7fc2f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc30ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5885e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc3152ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587a4d0 0x5646c587a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc30dffb640 1 -- 172.21.15.7:0/86764813 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7fc3000d1160 con 0x7fc3101031c0 2024-04-21T08:56:41.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc30ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5885e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56678/0 (socket says 172.21.15.7:56678) 2024-04-21T08:56:41.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc30ffff640 1 -- 172.21.15.7:0/456055252 learned_addr learned my addr 172.21.15.7:0/456055252 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc314aaa640 1 --2- 172.21.15.7:0/456055252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587ae80 0x5646c5883430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc3152ab640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc2f80a3d20 0x7fc2f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc30ffff640 1 -- 172.21.15.7:0/456055252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587a4d0 msgr2=0x5646c587a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc30ffff640 1 --2- 172.21.15.7:0/456055252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587a4d0 0x5646c587a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc30ffff640 1 -- 172.21.15.7:0/456055252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587ae80 msgr2=0x5646c5883430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc30ffff640 1 --2- 172.21.15.7:0/456055252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587ae80 0x5646c5883430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc30ffff640 1 -- 172.21.15.7:0/456055252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646c576e1b0 con 0x5646c5883a00 2024-04-21T08:56:41.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc30ffff640 1 --2- 172.21.15.7:0/456055252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5885e50 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc304000c00 tx=0x7fc304005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feae8f0f71ae205b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.637+0000 7fc3152ab640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc2f80a3d20 0x7fc2f80a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc308005cc0 tx=0x7fc308005c30 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc2f6ffd640 1 -- 172.21.15.7:0/456055252 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc304044020 con 0x5646c5883a00 2024-04-21T08:56:41.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc2f6ffd640 1 -- 172.21.15.7:0/456055252 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30402cc90 con 0x5646c5883a00 2024-04-21T08:56:41.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc2f6ffd640 1 -- 172.21.15.7:0/456055252 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3040046c0 con 0x5646c5883a00 2024-04-21T08:56:41.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc3199dc740 1 -- 172.21.15.7:0/456055252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 msgr2=0x5646c5885e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc3199dc740 1 --2- 172.21.15.7:0/456055252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5885e50 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc304000c00 tx=0x7fc304005e60 comp rx=0 tx=0).stop 2024-04-21T08:56:41.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc3199dc740 1 -- 172.21.15.7:0/456055252 shutdown_connections 2024-04-21T08:56:41.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc3199dc740 1 --2- 172.21.15.7:0/456055252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5885e50 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc3199dc740 1 --2- 172.21.15.7:0/456055252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587ae80 0x5646c5883430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc3199dc740 1 --2- 172.21.15.7:0/456055252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587a4d0 0x5646c587a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc3199dc740 1 -- 172.21.15.7:0/456055252 >> 172.21.15.7:0/456055252 conn(0x5646c5874ba0 msgr2=0x5646c5874f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc3199dc740 1 -- 172.21.15.7:0/456055252 shutdown_connections 2024-04-21T08:56:41.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.638+0000 7fc3199dc740 1 -- 172.21.15.7:0/456055252 wait complete. 2024-04-21T08:56:41.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3199dc740 1 -- start start 2024-04-21T08:56:41.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3199dc740 1 Processor -- start 2024-04-21T08:56:41.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587a4d0 0x5646c58832f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587ae80 0x5646c5880680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5880c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3199dc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5646c5886910 con 0x5646c5883a00 2024-04-21T08:56:41.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3199dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5646c5798500 con 0x5646c587a4d0 2024-04-21T08:56:41.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3199dc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5646c5886a90 con 0x5646c587ae80 2024-04-21T08:56:41.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3199dc740 10 client.0 _ll_register_callbacks cb 0x5646c5766b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:41.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3152ab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5880c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3152ab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5880c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56690/0 (socket says 172.21.15.7:56690) 2024-04-21T08:56:41.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc3152ab640 1 -- 172.21.15.7:0/1869581560 learned_addr learned my addr 172.21.15.7:0/1869581560 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc30ffff640 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587ae80 0x5646c5880680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.639+0000 7fc314aaa640 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587a4d0 0x5646c58832f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc3152ab640 1 -- 172.21.15.7:0/1869581560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587ae80 msgr2=0x5646c5880680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc3152ab640 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587ae80 0x5646c5880680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc3152ab640 1 -- 172.21.15.7:0/1869581560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587a4d0 msgr2=0x5646c58832f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc3152ab640 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587a4d0 0x5646c58832f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc3152ab640 1 -- 172.21.15.7:0/1869581560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646c54f13f0 con 0x5646c5883a00 2024-04-21T08:56:41.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc314aaa640 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587a4d0 0x5646c58832f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:56:41.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc3152ab640 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5880c00 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc30800fd80 tx=0x7fc308004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eade0c2f847748c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc2effff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:56:41.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc2effff640 1 -- 172.21.15.7:0/1869581560 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30801d620 con 0x5646c5883a00 2024-04-21T08:56:41.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc2effff640 1 -- 172.21.15.7:0/1869581560 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7fc30801d7c0 con 0x5646c5883a00 2024-04-21T08:56:41.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc3199dc740 1 -- 172.21.15.7:0/1869581560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc30402cc90 con 0x5646c5883a00 2024-04-21T08:56:41.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc2effff640 1 -- 172.21.15.7:0/1869581560 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc30800de10 con 0x5646c5883a00 2024-04-21T08:56:41.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc3199dc740 10 client.17322 fetch_fsmap learned FSMap version 23 2024-04-21T08:56:41.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc3199dc740 1 -- 172.21.15.7:0/1869581560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5646c55bb7d0 con 0x5646c5883a00 2024-04-21T08:56:41.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.640+0000 7fc2effff640 1 -- 172.21.15.7:0/1869581560 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30402cc90 con 0x5646c5883a00 2024-04-21T08:56:41.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.641+0000 7fc2effff640 1 -- 172.21.15.7:0/1869581560 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fc308027980 con 0x5646c5883a00 2024-04-21T08:56:41.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.641+0000 7fc3199dc740 10 client.17322 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:56:41.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.641+0000 7fc3199dc740 10 client.17322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:41.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.641+0000 7fc3199dc740 20 client.17322 populate_metadata read hostname 'smithi007' 2024-04-21T08:56:41.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.641+0000 7fc3199dc740 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5646c5893c90 0x5646c58960f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.641+0000 7fc3199dc740 4 client.17322 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:56:41.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.641+0000 7fc3199dc740 1 -- 172.21.15.7:0/1869581560 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646c577e930 con 0x5646c5893c90 2024-04-21T08:56:41.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.641+0000 7fc314aaa640 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5646c5893c90 0x5646c58960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.641+0000 7fc2effff640 10 client.17322 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:41.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.641+0000 7fc314aaa640 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5646c5893c90 0x5646c58960f0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea2907cdaede2c3a server_cookie=b03fa0820189ad8e in_seq=0 out_seq=0 2024-04-21T08:56:41.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc2effff640 1 -- 172.21.15.7:0/1869581560 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646c577e930 con 0x5646c5893c90 2024-04-21T08:56:41.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc2effff640 10 client.17322 handle_command_reply: tid=0 2024-04-21T08:56:41.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 1 client.17322 shutdown 2024-04-21T08:56:41.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 1 -- 172.21.15.7:0/1869581560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 msgr2=0x5646c5880c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5880c00 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc30800fd80 tx=0x7fc308004ab0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 1 -- 172.21.15.7:0/1869581560 shutdown_connections 2024-04-21T08:56:41.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5646c5893c90 0x5646c58960f0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c5883a00 0x5646c5880c00 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587ae80 0x5646c5880680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 1 --2- 172.21.15.7:0/1869581560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587a4d0 0x5646c58832f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 1 -- 172.21.15.7:0/1869581560 >> 172.21.15.7:0/1869581560 conn(0x5646c5874ba0 msgr2=0x5646c58768f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc2effff640 0 client.17322 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:41.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 1 -- 172.21.15.7:0/1869581560 shutdown_connections 2024-04-21T08:56:41.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 1 -- 172.21.15.7:0/1869581560 wait complete. 2024-04-21T08:56:41.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.642+0000 7fc3199dc740 20 client.17322 trim_cache size 0 max 16384 2024-04-21T08:56:41.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.656+0000 7fc3199dc740 1 Processor -- start 2024-04-21T08:56:41.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.656+0000 7fc3199dc740 1 -- start start 2024-04-21T08:56:41.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.656+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.656+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587ae80 0x5646c5882db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.656+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c5883a00 0x5646c58832f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.656+0000 7fc3199dc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101ad4a0 con 0x5646c587a4d0 2024-04-21T08:56:41.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.656+0000 7fc3199dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101afd40 con 0x5646c5883a00 2024-04-21T08:56:41.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.656+0000 7fc3199dc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101adb30 con 0x5646c587ae80 2024-04-21T08:56:41.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc314aaa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc314aaa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56696/0 (socket says 172.21.15.7:56696) 2024-04-21T08:56:41.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc314aaa640 1 -- 172.21.15.7:0/1459457832 learned_addr learned my addr 172.21.15.7:0/1459457832 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc3152ab640 1 --2- 172.21.15.7:0/1459457832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c5883a00 0x5646c58832f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc30ffff640 1 --2- 172.21.15.7:0/1459457832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587ae80 0x5646c5882db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc314aaa640 1 -- 172.21.15.7:0/1459457832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587ae80 msgr2=0x5646c5882db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc314aaa640 1 --2- 172.21.15.7:0/1459457832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587ae80 0x5646c5882db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc314aaa640 1 -- 172.21.15.7:0/1459457832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c5883a00 msgr2=0x5646c58832f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc314aaa640 1 --2- 172.21.15.7:0/1459457832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c5883a00 0x5646c58832f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc314aaa640 1 -- 172.21.15.7:0/1459457832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646c587c270 con 0x5646c587a4d0 2024-04-21T08:56:41.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc3152ab640 1 --2- 172.21.15.7:0/1459457832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c5883a00 0x5646c58832f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:56:41.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc314aaa640 1 --2- 172.21.15.7:0/1459457832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588f7d0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc3000097d0 tx=0x7fc30001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ef4eb616743bc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc2f67fc640 1 -- 172.21.15.7:0/1459457832 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300005180 con 0x5646c587a4d0 2024-04-21T08:56:41.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc2f67fc640 1 -- 172.21.15.7:0/1459457832 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3000b5070 con 0x5646c587a4d0 2024-04-21T08:56:41.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.657+0000 7fc2f67fc640 1 -- 172.21.15.7:0/1459457832 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30009c430 con 0x5646c587a4d0 2024-04-21T08:56:41.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 -- 172.21.15.7:0/1459457832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 msgr2=0x5646c588f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 --2- 172.21.15.7:0/1459457832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588f7d0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fc3000097d0 tx=0x7fc30001a040 comp rx=0 tx=0).stop 2024-04-21T08:56:41.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 -- 172.21.15.7:0/1459457832 shutdown_connections 2024-04-21T08:56:41.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 --2- 172.21.15.7:0/1459457832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c5883a00 0x5646c58832f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 --2- 172.21.15.7:0/1459457832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c587ae80 0x5646c5882db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 --2- 172.21.15.7:0/1459457832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588f7d0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 -- 172.21.15.7:0/1459457832 >> 172.21.15.7:0/1459457832 conn(0x5646c5874ba0 msgr2=0x5646c5880640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 -- 172.21.15.7:0/1459457832 shutdown_connections 2024-04-21T08:56:41.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 -- 172.21.15.7:0/1459457832 wait complete. 2024-04-21T08:56:41.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 -- start start 2024-04-21T08:56:41.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 Processor -- start 2024-04-21T08:56:41.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587ae80 0x5646c587cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c5883a00 0x5646c587d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.658+0000 7fc3199dc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101ae200 con 0x5646c587a4d0 2024-04-21T08:56:41.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc3199dc740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101af670 con 0x5646c587ae80 2024-04-21T08:56:41.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc3199dc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101ae8d0 con 0x5646c5883a00 2024-04-21T08:56:41.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc3199dc740 10 client.0 _ll_register_callbacks cb 0x7fc3080126c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:41.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc314aaa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc314aaa640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56708/0 (socket says 172.21.15.7:56708) 2024-04-21T08:56:41.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc314aaa640 1 -- 172.21.15.7:0/4009811603 learned_addr learned my addr 172.21.15.7:0/4009811603 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc30ffff640 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587ae80 0x5646c587cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc3152ab640 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c5883a00 0x5646c587d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc314aaa640 1 -- 172.21.15.7:0/4009811603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c5883a00 msgr2=0x5646c587d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc314aaa640 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c5883a00 0x5646c587d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc314aaa640 1 -- 172.21.15.7:0/4009811603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587ae80 msgr2=0x5646c587cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc314aaa640 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587ae80 0x5646c587cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc314aaa640 1 -- 172.21.15.7:0/4009811603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646c587d780 con 0x5646c587a4d0 2024-04-21T08:56:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc30ffff640 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587ae80 0x5646c587cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:56:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc314aaa640 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588bed0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc3000996d0 tx=0x7fc30009cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47497b34b54aca65 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc2effff640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-04-21T08:56:41.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc2effff640 1 -- 172.21.15.7:0/4009811603 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300065020 con 0x5646c587a4d0 2024-04-21T08:56:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc2effff640 1 -- 172.21.15.7:0/4009811603 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7fc300098ce0 con 0x5646c587a4d0 2024-04-21T08:56:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc3199dc740 1 -- 172.21.15.7:0/4009811603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3000b5070 con 0x5646c587a4d0 2024-04-21T08:56:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc2effff640 1 -- 172.21.15.7:0/4009811603 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc30009d400 con 0x5646c587a4d0 2024-04-21T08:56:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc3199dc740 10 client.17328 fetch_fsmap learned FSMap version 23 2024-04-21T08:56:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc3199dc740 1 -- 172.21.15.7:0/4009811603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5646c58f9e70 con 0x5646c587a4d0 2024-04-21T08:56:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.659+0000 7fc2effff640 1 -- 172.21.15.7:0/4009811603 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3000b5070 con 0x5646c587a4d0 2024-04-21T08:56:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.660+0000 7fc2effff640 1 -- 172.21.15.7:0/4009811603 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fc300099370 con 0x5646c587a4d0 2024-04-21T08:56:41.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.660+0000 7fc3199dc740 10 client.17328 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T08:56:41.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.660+0000 7fc3199dc740 10 client.17328 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:41.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.660+0000 7fc3199dc740 20 client.17328 populate_metadata read hostname 'smithi007' 2024-04-21T08:56:41.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.660+0000 7fc3199dc740 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5646c58fa480 0x5646c58fc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.660+0000 7fc3199dc740 4 client.17328 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:56:41.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.660+0000 7fc3199dc740 1 -- 172.21.15.7:0/4009811603 --> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5646c57ecc10 con 0x5646c58fa480 2024-04-21T08:56:41.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.660+0000 7fc30ffff640 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5646c58fa480 0x5646c58fc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.660+0000 7fc2effff640 10 client.17328 ms_handle_connect on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.660+0000 7fc30ffff640 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5646c58fa480 0x5646c58fc870 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc797d2d262f462c server_cookie=6b9b905a35d77ea3 in_seq=0 out_seq=0 2024-04-21T08:56:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.660+0000 7fc2effff640 1 -- 172.21.15.7:0/4009811603 <== mds.0 v2:172.21.15.16:6838/1703654929 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5646c57ecc10 con 0x5646c58fa480 2024-04-21T08:56:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc2effff640 10 client.17328 handle_command_reply: tid=0 2024-04-21T08:56:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 1 client.17328 shutdown 2024-04-21T08:56:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 1 -- 172.21.15.7:0/4009811603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 msgr2=0x5646c588bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588bed0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc3000996d0 tx=0x7fc30009cd70 comp rx=0 tx=0).stop 2024-04-21T08:56:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 1 -- 172.21.15.7:0/4009811603 shutdown_connections 2024-04-21T08:56:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.16:6838/1703654929,v1:172.21.15.16:6839/1703654929] conn(0x5646c58fa480 0x5646c58fc870 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5646c5883a00 0x5646c587d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5646c587ae80 0x5646c587cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc2effff640 0 client.17328 ms_handle_reset on v2:172.21.15.16:6838/1703654929 2024-04-21T08:56:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 1 --2- 172.21.15.7:0/4009811603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5646c587a4d0 0x5646c588bed0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 1 -- 172.21.15.7:0/4009811603 >> 172.21.15.7:0/4009811603 conn(0x5646c5874ba0 msgr2=0x5646c587b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 1 -- 172.21.15.7:0/4009811603 shutdown_connections 2024-04-21T08:56:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 1 -- 172.21.15.7:0/4009811603 wait complete. 2024-04-21T08:56:41.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc3199dc740 20 client.17328 trim_cache size 0 max 16384 2024-04-21T08:56:41.674 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-04-21T08:56:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc316d0e640 1 -- 172.21.15.7:0/86764813 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc2f80a3d20 msgr2=0x7fc2f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc316d0e640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc2f80a3d20 0x7fc2f80a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc308005cc0 tx=0x7fc308005c30 comp rx=0 tx=0).stop 2024-04-21T08:56:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc316d0e640 1 -- 172.21.15.7:0/86764813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3101031c0 msgr2=0x7fc31019e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc316d0e640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3101031c0 0x7fc31019e670 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc30000e9f0 tx=0x7fc30000edf0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc316d0e640 1 -- 172.21.15.7:0/86764813 shutdown_connections 2024-04-21T08:56:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc316d0e640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3101987e0 0x7fc310198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc316d0e640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fc2f80a3d20 0x7fc2f80a61e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc316d0e640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc31010c6d0 0x7fc31019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc316d0e640 1 --2- 172.21.15.7:0/86764813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3101031c0 0x7fc31019e670 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.661+0000 7fc316d0e640 1 -- 172.21.15.7:0/86764813 >> 172.21.15.7:0/86764813 conn(0x7fc310077f70 msgr2=0x7fc3100ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.662+0000 7fc316d0e640 1 -- 172.21.15.7:0/86764813 shutdown_connections 2024-04-21T08:56:41.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.662+0000 7fc316d0e640 1 -- 172.21.15.7:0/86764813 wait complete. 2024-04-21T08:56:41.677 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/thrasher.py", line 43, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-04-21T08:56:41.682 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 152, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 310, in task stop_all_quiescers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 263, in stop_all_quiescers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: Error quiescing root: -110 (ETIMEDOUT) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 155, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 90, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: 2024-04-21T08:56:41.689 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-21T08:56:41.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:56:41.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.811+0000 7fe6b3031640 1 -- 172.21.15.7:0/3780768221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac105dd0 msgr2=0x7fe6ac1061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.811+0000 7fe6b3031640 1 --2- 172.21.15.7:0/3780768221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac105dd0 0x7fe6ac1061b0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009900 tx=0x7fe6a002bc30 comp rx=0 tx=0).stop 2024-04-21T08:56:41.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.811+0000 7fe6b3031640 1 -- 172.21.15.7:0/3780768221 shutdown_connections 2024-04-21T08:56:41.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.811+0000 7fe6b3031640 1 --2- 172.21.15.7:0/3780768221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6ac101a40 0x7fe6ac10dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.811+0000 7fe6b3031640 1 --2- 172.21.15.7:0/3780768221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ac1010a0 0x7fe6ac101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.811+0000 7fe6b3031640 1 --2- 172.21.15.7:0/3780768221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac105dd0 0x7fe6ac1061b0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.811+0000 7fe6b3031640 1 -- 172.21.15.7:0/3780768221 >> 172.21.15.7:0/3780768221 conn(0x7fe6ac077c40 msgr2=0x7fe6ac078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:41.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.811+0000 7fe6b3031640 1 -- 172.21.15.7:0/3780768221 shutdown_connections 2024-04-21T08:56:41.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.811+0000 7fe6b3031640 1 -- 172.21.15.7:0/3780768221 wait complete. 2024-04-21T08:56:41.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6b3031640 1 Processor -- start 2024-04-21T08:56:41.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6b3031640 1 -- start start 2024-04-21T08:56:41.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6b3031640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ac1010a0 0x7fe6ac19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6b3031640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac101a40 0x7fe6ac19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6b3031640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6ac105dd0 0x7fe6ac1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6b3031640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac1111a0 con 0x7fe6ac101a40 2024-04-21T08:56:41.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6b3031640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac111020 con 0x7fe6ac105dd0 2024-04-21T08:56:41.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6b3031640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac111320 con 0x7fe6ac1010a0 2024-04-21T08:56:41.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6abfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac101a40 0x7fe6ac19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6abfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac101a40 0x7fe6ac19e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56726/0 (socket says 172.21.15.7:56726) 2024-04-21T08:56:41.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6abfff640 1 -- 172.21.15.7:0/3020184734 learned_addr learned my addr 172.21.15.7:0/3020184734 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:41.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6b0dcd640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ac1010a0 0x7fe6ac19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6b15ce640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6ac105dd0 0x7fe6ac1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6abfff640 1 -- 172.21.15.7:0/3020184734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ac1010a0 msgr2=0x7fe6ac19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6abfff640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ac1010a0 0x7fe6ac19e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6abfff640 1 -- 172.21.15.7:0/3020184734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6ac105dd0 msgr2=0x7fe6ac1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:41.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6abfff640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6ac105dd0 0x7fe6ac1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:41.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.812+0000 7fe6abfff640 1 -- 172.21.15.7:0/3020184734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ac198d70 con 0x7fe6ac101a40 2024-04-21T08:56:41.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.813+0000 7fe6abfff640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac101a40 0x7fe6ac19e930 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fe69800d950 tx=0x7fe69800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.813+0000 7fe6a9ffb640 1 -- 172.21.15.7:0/3020184734 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698017030 con 0x7fe6ac101a40 2024-04-21T08:56:41.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.813+0000 7fe6a9ffb640 1 -- 172.21.15.7:0/3020184734 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe698004020 con 0x7fe6ac101a40 2024-04-21T08:56:41.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.813+0000 7fe6a9ffb640 1 -- 172.21.15.7:0/3020184734 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6980047a0 con 0x7fe6ac101a40 2024-04-21T08:56:41.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.813+0000 7fe6b3031640 1 -- 172.21.15.7:0/3020184734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ac199060 con 0x7fe6ac101a40 2024-04-21T08:56:41.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.813+0000 7fe6b3031640 1 -- 172.21.15.7:0/3020184734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ac1a52b0 con 0x7fe6ac101a40 2024-04-21T08:56:41.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.814+0000 7fe6b3031640 1 -- 172.21.15.7:0/3020184734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6ac102540 con 0x7fe6ac101a40 2024-04-21T08:56:41.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.815+0000 7fe6a9ffb640 1 -- 172.21.15.7:0/3020184734 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fe698009a30 con 0x7fe6ac101a40 2024-04-21T08:56:41.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.816+0000 7fe6a9ffb640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe69c0a3da0 0x7fe69c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:41.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.816+0000 7fe6b0dcd640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe69c0a3da0 0x7fe69c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:41.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.816+0000 7fe6a9ffb640 1 -- 172.21.15.7:0/3020184734 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7fe6980d0690 con 0x7fe6ac101a40 2024-04-21T08:56:41.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.816+0000 7fe6b0dcd640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe69c0a3da0 0x7fe69c0a6260 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0005ab0 tx=0x7fe6a0005a20 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:41.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:41.820+0000 7fe6a9ffb640 1 -- 172.21.15.7:0/3020184734 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe69809c8f0 con 0x7fe6ac101a40 2024-04-21T08:56:42.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.011+0000 7fe6b3031640 1 -- 172.21.15.7:0/3020184734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe6ac10a370 con 0x7fe6ac101a40 2024-04-21T08:56:42.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.013+0000 7fe6a9ffb640 1 -- 172.21.15.7:0/3020184734 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+25180 (secure 0 0 0) 0x7fe69809c1f0 con 0x7fe6ac101a40 2024-04-21T08:56:42.013 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:56:42.013 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":113,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:56:31.247782+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3451243741":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/117789107":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/4167672076":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/3630897301":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":19,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:56:42.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 -- 172.21.15.7:0/3020184734 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe69c0a3da0 msgr2=0x7fe69c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe69c0a3da0 0x7fe69c0a6260 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0005ab0 tx=0x7fe6a0005a20 comp rx=0 tx=0).stop 2024-04-21T08:56:42.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 -- 172.21.15.7:0/3020184734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac101a40 msgr2=0x7fe6ac19e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac101a40 0x7fe6ac19e930 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fe69800d950 tx=0x7fe69800de10 comp rx=0 tx=0).stop 2024-04-21T08:56:42.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 -- 172.21.15.7:0/3020184734 shutdown_connections 2024-04-21T08:56:42.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6ac105dd0 0x7fe6ac1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac101a40 0x7fe6ac19e930 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe69c0a3da0 0x7fe69c0a6260 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 --2- 172.21.15.7:0/3020184734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6ac1010a0 0x7fe6ac19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 -- 172.21.15.7:0/3020184734 >> 172.21.15.7:0/3020184734 conn(0x7fe6ac077c40 msgr2=0x7fe6ac06a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:42.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 -- 172.21.15.7:0/3020184734 shutdown_connections 2024-04-21T08:56:42.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.015+0000 7fe6b3031640 1 -- 172.21.15.7:0/3020184734 wait complete. 2024-04-21T08:56:42.026 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 109, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 109, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:56:42.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:56:42.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.132+0000 7f11e04eb640 1 -- 172.21.15.7:0/3537725536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8102a60 msgr2=0x7f11d8102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.132+0000 7f11e04eb640 1 --2- 172.21.15.7:0/3537725536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8102a60 0x7f11d8102ec0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009930 tx=0x7f11c802bc30 comp rx=0 tx=0).stop 2024-04-21T08:56:42.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.132+0000 7f11e04eb640 1 -- 172.21.15.7:0/3537725536 shutdown_connections 2024-04-21T08:56:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.132+0000 7f11e04eb640 1 --2- 172.21.15.7:0/3537725536 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d8103400 0x7f11d810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.132+0000 7f11e04eb640 1 --2- 172.21.15.7:0/3537725536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8102a60 0x7f11d8102ec0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.132+0000 7f11e04eb640 1 --2- 172.21.15.7:0/3537725536 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d8101180 0x7f11d8101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.132+0000 7f11e04eb640 1 -- 172.21.15.7:0/3537725536 >> 172.21.15.7:0/3537725536 conn(0x7f11d8077f70 msgr2=0x7f11d8078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.132+0000 7f11e04eb640 1 -- 172.21.15.7:0/3537725536 shutdown_connections 2024-04-21T08:56:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.132+0000 7f11e04eb640 1 -- 172.21.15.7:0/3537725536 wait complete. 2024-04-21T08:56:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11e04eb640 1 Processor -- start 2024-04-21T08:56:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11e04eb640 1 -- start start 2024-04-21T08:56:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11e04eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d8101180 0x7f11d819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11e04eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d8102a60 0x7f11d819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11e04eb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8103400 0x7f11d8198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11e04eb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f11d81111a0 con 0x7f11d8103400 2024-04-21T08:56:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11e04eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11d8111020 con 0x7f11d8102a60 2024-04-21T08:56:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11e04eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f11d8111320 con 0x7f11d8101180 2024-04-21T08:56:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11dea88640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8103400 0x7f11d8198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11dea88640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8103400 0x7f11d8198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56740/0 (socket says 172.21.15.7:56740) 2024-04-21T08:56:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11dea88640 1 -- 172.21.15.7:0/3053794859 learned_addr learned my addr 172.21.15.7:0/3053794859 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11de287640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d8101180 0x7f11d819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11dda86640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d8102a60 0x7f11d819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11dea88640 1 -- 172.21.15.7:0/3053794859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d8101180 msgr2=0x7f11d819e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11dea88640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d8101180 0x7f11d819e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11dea88640 1 -- 172.21.15.7:0/3053794859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d8102a60 msgr2=0x7f11d819eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11dea88640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d8102a60 0x7f11d819eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.133+0000 7f11dea88640 1 -- 172.21.15.7:0/3053794859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d8198f50 con 0x7f11d8103400 2024-04-21T08:56:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.134+0000 7f11dea88640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8103400 0x7f11d8198690 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f11d000eaa0 tx=0x7f11d000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.134+0000 7f11cf7fe640 1 -- 172.21.15.7:0/3053794859 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d000cdb0 con 0x7f11d8103400 2024-04-21T08:56:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.134+0000 7f11cf7fe640 1 -- 172.21.15.7:0/3053794859 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11d0005020 con 0x7f11d8103400 2024-04-21T08:56:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.134+0000 7f11e04eb640 1 -- 172.21.15.7:0/3053794859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d8199240 con 0x7f11d8103400 2024-04-21T08:56:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.134+0000 7f11cf7fe640 1 -- 172.21.15.7:0/3053794859 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d00116d0 con 0x7f11d8103400 2024-04-21T08:56:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.134+0000 7f11e04eb640 1 -- 172.21.15.7:0/3053794859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d81a5550 con 0x7f11d8103400 2024-04-21T08:56:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.135+0000 7f11e04eb640 1 -- 172.21.15.7:0/3053794859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d8102570 con 0x7f11d8103400 2024-04-21T08:56:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.136+0000 7f11cf7fe640 1 -- 172.21.15.7:0/3053794859 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f11d00051d0 con 0x7f11d8103400 2024-04-21T08:56:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.137+0000 7f11cf7fe640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f11c40a3d20 0x7f11c40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.137+0000 7f11de287640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f11c40a3d20 0x7f11c40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.137+0000 7f11cf7fe640 1 -- 172.21.15.7:0/3053794859 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7f11d00d0da0 con 0x7f11d8103400 2024-04-21T08:56:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.137+0000 7f11de287640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f11c40a3d20 0x7f11c40a61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f11d4005fd0 tx=0x7f11d400b390 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:42.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.141+0000 7f11cf7fe640 1 -- 172.21.15.7:0/3053794859 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f11d0016030 con 0x7f11d8103400 2024-04-21T08:56:42.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.329+0000 7f11e04eb640 1 -- 172.21.15.7:0/3053794859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f11d8199540 con 0x7f11d8103400 2024-04-21T08:56:42.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.329+0000 7f11cf7fe640 1 -- 172.21.15.7:0/3053794859 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) ==== 93+0+10 (secure 0 0 0) 0x7f11d009d000 con 0x7f11d8103400 2024-04-21T08:56:42.330 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:56:42.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 -- 172.21.15.7:0/3053794859 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f11c40a3d20 msgr2=0x7f11c40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f11c40a3d20 0x7f11c40a61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f11d4005fd0 tx=0x7f11d400b390 comp rx=0 tx=0).stop 2024-04-21T08:56:42.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 -- 172.21.15.7:0/3053794859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8103400 msgr2=0x7f11d8198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8103400 0x7f11d8198690 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f11d000eaa0 tx=0x7f11d000ef60 comp rx=0 tx=0).stop 2024-04-21T08:56:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 -- 172.21.15.7:0/3053794859 shutdown_connections 2024-04-21T08:56:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8103400 0x7f11d8198690 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11d8102a60 0x7f11d819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f11c40a3d20 0x7f11c40a61e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 --2- 172.21.15.7:0/3053794859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11d8101180 0x7f11d819e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 -- 172.21.15.7:0/3053794859 >> 172.21.15.7:0/3053794859 conn(0x7f11d8077f70 msgr2=0x7f11d8100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 -- 172.21.15.7:0/3053794859 shutdown_connections 2024-04-21T08:56:42.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.331+0000 7f11e04eb640 1 -- 172.21.15.7:0/3053794859 wait complete. 2024-04-21T08:56:42.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:56:42.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.447+0000 7fdabe10d640 1 -- 172.21.15.7:0/1389062218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab8101180 msgr2=0x7fdab8101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.447+0000 7fdabe10d640 1 --2- 172.21.15.7:0/1389062218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab8101180 0x7fdab8101560 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009930 tx=0x7fdaa802bc30 comp rx=0 tx=0).stop 2024-04-21T08:56:42.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.447+0000 7fdabe10d640 1 -- 172.21.15.7:0/1389062218 shutdown_connections 2024-04-21T08:56:42.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.447+0000 7fdabe10d640 1 --2- 172.21.15.7:0/1389062218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab8103400 0x7fdab810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.447+0000 7fdabe10d640 1 --2- 172.21.15.7:0/1389062218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab8102a60 0x7fdab8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.447+0000 7fdabe10d640 1 --2- 172.21.15.7:0/1389062218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab8101180 0x7fdab8101560 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.447+0000 7fdabe10d640 1 -- 172.21.15.7:0/1389062218 >> 172.21.15.7:0/1389062218 conn(0x7fdab8077f70 msgr2=0x7fdab8078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:42.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.447+0000 7fdabe10d640 1 -- 172.21.15.7:0/1389062218 shutdown_connections 2024-04-21T08:56:42.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.447+0000 7fdabe10d640 1 -- 172.21.15.7:0/1389062218 wait complete. 2024-04-21T08:56:42.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.448+0000 7fdabe10d640 1 Processor -- start 2024-04-21T08:56:42.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.448+0000 7fdabe10d640 1 -- start start 2024-04-21T08:56:42.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.448+0000 7fdabe10d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab8101180 0x7fdab8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.448+0000 7fdabe10d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab8102a60 0x7fdab8198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.448+0000 7fdabe10d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab8103400 0x7fdab819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.448+0000 7fdabe10d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdab8112300 con 0x7fdab8103400 2024-04-21T08:56:42.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.448+0000 7fdabe10d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdab8112180 con 0x7fdab8101180 2024-04-21T08:56:42.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.448+0000 7fdabe10d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdab8112480 con 0x7fdab8102a60 2024-04-21T08:56:42.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.448+0000 7fdab7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab8103400 0x7fdab819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:42.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.448+0000 7fdab7fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab8103400 0x7fdab819d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56756/0 (socket says 172.21.15.7:56756) 2024-04-21T08:56:42.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.448+0000 7fdab7fff640 1 -- 172.21.15.7:0/2187725423 learned_addr learned my addr 172.21.15.7:0/2187725423 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:42.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdab6ffd640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab8102a60 0x7fdab8198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:42.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdab7fff640 1 -- 172.21.15.7:0/2187725423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab8102a60 msgr2=0x7fdab8198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdab7fff640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab8102a60 0x7fdab8198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdab7fff640 1 -- 172.21.15.7:0/2187725423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab8101180 msgr2=0x7fdab8198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdab7fff640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab8101180 0x7fdab8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdab7fff640 1 -- 172.21.15.7:0/2187725423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab819d730 con 0x7fdab8103400 2024-04-21T08:56:42.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdab7fff640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab8103400 0x7fdab819d000 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fdab000cce0 tx=0x7fdab000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:42.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdab4ff9640 1 -- 172.21.15.7:0/2187725423 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab000f450 con 0x7fdab8103400 2024-04-21T08:56:42.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdab4ff9640 1 -- 172.21.15.7:0/2187725423 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdab000faa0 con 0x7fdab8103400 2024-04-21T08:56:42.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdab4ff9640 1 -- 172.21.15.7:0/2187725423 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab0018b70 con 0x7fdab8103400 2024-04-21T08:56:42.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdabe10d640 1 -- 172.21.15.7:0/2187725423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab819da20 con 0x7fdab8103400 2024-04-21T08:56:42.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.449+0000 7fdabe10d640 1 -- 172.21.15.7:0/2187725423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab81a52b0 con 0x7fdab8103400 2024-04-21T08:56:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.450+0000 7fdabe10d640 1 -- 172.21.15.7:0/2187725423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab8069250 con 0x7fdab8103400 2024-04-21T08:56:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.451+0000 7fdab4ff9640 1 -- 172.21.15.7:0/2187725423 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fdab0018d10 con 0x7fdab8103400 2024-04-21T08:56:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.452+0000 7fdab4ff9640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdaa00a3d20 0x7fdaa00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.452+0000 7fdab77fe640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdaa00a3d20 0x7fdaa00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:42.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.452+0000 7fdab4ff9640 1 -- 172.21.15.7:0/2187725423 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7fdab00d11b0 con 0x7fdab8103400 2024-04-21T08:56:42.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.452+0000 7fdab77fe640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdaa00a3d20 0x7fdaa00a61e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8005770 tx=0x7fdaa8000bc0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:42.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.456+0000 7fdab4ff9640 1 -- 172.21.15.7:0/2187725423 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdab0014030 con 0x7fdab8103400 2024-04-21T08:56:42.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:42 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/115865410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:42.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:42 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3020184734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:56:42.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:42 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/115865410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:42.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:42 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3020184734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:56:42.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.642+0000 7fdabe10d640 1 -- 172.21.15.7:0/2187725423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdab8109480 con 0x7fdab8103400 2024-04-21T08:56:42.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.643+0000 7fdab4ff9640 1 -- 172.21.15.7:0/2187725423 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) ==== 104+0+11 (secure 0 0 0) 0x7fdab009d4e0 con 0x7fdab8103400 2024-04-21T08:56:42.646 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:56:42.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 -- 172.21.15.7:0/2187725423 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdaa00a3d20 msgr2=0x7fdaa00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdaa00a3d20 0x7fdaa00a61e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8005770 tx=0x7fdaa8000bc0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 -- 172.21.15.7:0/2187725423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab8103400 msgr2=0x7fdab819d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab8103400 0x7fdab819d000 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fdab000cce0 tx=0x7fdab000a620 comp rx=0 tx=0).stop 2024-04-21T08:56:42.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 -- 172.21.15.7:0/2187725423 shutdown_connections 2024-04-21T08:56:42.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab8103400 0x7fdab819d000 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fdaa00a3d20 0x7fdaa00a61e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdab8102a60 0x7fdab8198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 --2- 172.21.15.7:0/2187725423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdab8101180 0x7fdab8198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 -- 172.21.15.7:0/2187725423 >> 172.21.15.7:0/2187725423 conn(0x7fdab8077f70 msgr2=0x7fdab8104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:42.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 -- 172.21.15.7:0/2187725423 shutdown_connections 2024-04-21T08:56:42.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.645+0000 7fdabe10d640 1 -- 172.21.15.7:0/2187725423 wait complete. 2024-04-21T08:56:42.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:42 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/115865410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:42.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:42 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3020184734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:56:42.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:56:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.761+0000 7fbeaa89a640 1 -- 172.21.15.7:0/3416437867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbea4106160 msgr2=0x7fbea410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.761+0000 7fbeaa89a640 1 --2- 172.21.15.7:0/3416437867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbea4106160 0x7fbea410dd30 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c009ca0 tx=0x7fbe9c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.761+0000 7fbeaa89a640 1 -- 172.21.15.7:0/3416437867 shutdown_connections 2024-04-21T08:56:42.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.761+0000 7fbeaa89a640 1 --2- 172.21.15.7:0/3416437867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbea4106160 0x7fbea410dd30 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.761+0000 7fbeaa89a640 1 --2- 172.21.15.7:0/3416437867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea4101a40 0x7fbea4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.761+0000 7fbeaa89a640 1 --2- 172.21.15.7:0/3416437867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea4101090 0x7fbea4101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.761+0000 7fbeaa89a640 1 -- 172.21.15.7:0/3416437867 >> 172.21.15.7:0/3416437867 conn(0x7fbea4078000 msgr2=0x7fbea4078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:42.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.761+0000 7fbeaa89a640 1 -- 172.21.15.7:0/3416437867 shutdown_connections 2024-04-21T08:56:42.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.761+0000 7fbeaa89a640 1 -- 172.21.15.7:0/3416437867 wait complete. 2024-04-21T08:56:42.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.762+0000 7fbeaa89a640 1 Processor -- start 2024-04-21T08:56:42.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.762+0000 7fbeaa89a640 1 -- start start 2024-04-21T08:56:42.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.762+0000 7fbeaa89a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea4101090 0x7fbea4198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.762+0000 7fbeaa89a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea4106160 0x7fbea4198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.762+0000 7fbeaa89a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbea419cf20 0x7fbea419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.762+0000 7fbeaa89a640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbea41121e0 con 0x7fbea419cf20 2024-04-21T08:56:42.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.762+0000 7fbeaa89a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbea4112060 con 0x7fbea4101090 2024-04-21T08:56:42.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.762+0000 7fbeaa89a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbea4112360 con 0x7fbea4106160 2024-04-21T08:56:42.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.762+0000 7fbea8e37640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbea419cf20 0x7fbea419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.762+0000 7fbea8e37640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbea419cf20 0x7fbea419d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56776/0 (socket says 172.21.15.7:56776) 2024-04-21T08:56:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.762+0000 7fbea8e37640 1 -- 172.21.15.7:0/4170818117 learned_addr learned my addr 172.21.15.7:0/4170818117 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea3fff640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea4101090 0x7fbea4198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea37fe640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea4106160 0x7fbea4198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea8e37640 1 -- 172.21.15.7:0/4170818117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea4106160 msgr2=0x7fbea4198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea8e37640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea4106160 0x7fbea4198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea8e37640 1 -- 172.21.15.7:0/4170818117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea4101090 msgr2=0x7fbea4198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea8e37640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea4101090 0x7fbea4198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea8e37640 1 -- 172.21.15.7:0/4170818117 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea419da10 con 0x7fbea419cf20 2024-04-21T08:56:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea3fff640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea4101090 0x7fbea4198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:56:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea37fe640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea4106160 0x7fbea4198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:56:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea8e37640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbea419cf20 0x7fbea419d3d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c005e30 tx=0x7fbe9c032420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea17fa640 1 -- 172.21.15.7:0/4170818117 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe9c041020 con 0x7fbea419cf20 2024-04-21T08:56:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea17fa640 1 -- 172.21.15.7:0/4170818117 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe9c032970 con 0x7fbea419cf20 2024-04-21T08:56:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbea17fa640 1 -- 172.21.15.7:0/4170818117 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe9c039040 con 0x7fbea419cf20 2024-04-21T08:56:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbeaa89a640 1 -- 172.21.15.7:0/4170818117 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea419dca0 con 0x7fbea419cf20 2024-04-21T08:56:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.763+0000 7fbeaa89a640 1 -- 172.21.15.7:0/4170818117 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea41a54c0 con 0x7fbea419cf20 2024-04-21T08:56:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.764+0000 7fbeaa89a640 1 -- 172.21.15.7:0/4170818117 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea4068e00 con 0x7fbea419cf20 2024-04-21T08:56:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.765+0000 7fbea17fa640 1 -- 172.21.15.7:0/4170818117 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fbe9c044020 con 0x7fbea419cf20 2024-04-21T08:56:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.766+0000 7fbea17fa640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbe900a3da0 0x7fbe900a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.766+0000 7fbea17fa640 1 -- 172.21.15.7:0/4170818117 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7fbe9c0f28b0 con 0x7fbea419cf20 2024-04-21T08:56:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.766+0000 7fbea3fff640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbe900a3da0 0x7fbe900a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.766+0000 7fbea3fff640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbe900a3da0 0x7fbe900a6260 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c005fd0 tx=0x7fbe8c00b5f0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.770+0000 7fbea17fa640 1 -- 172.21.15.7:0/4170818117 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe9c04b020 con 0x7fbea419cf20 2024-04-21T08:56:42.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.958+0000 7fbeaa89a640 1 -- 172.21.15.7:0/4170818117 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbea4101a40 con 0x7fbea419cf20 2024-04-21T08:56:42.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.958+0000 7fbea17fa640 1 -- 172.21.15.7:0/4170818117 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) ==== 100+0+11 (secure 0 0 0) 0x7fbe9c0bec60 con 0x7fbea419cf20 2024-04-21T08:56:42.958 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:56:42.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 -- 172.21.15.7:0/4170818117 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbe900a3da0 msgr2=0x7fbe900a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbe900a3da0 0x7fbe900a6260 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c005fd0 tx=0x7fbe8c00b5f0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 -- 172.21.15.7:0/4170818117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbea419cf20 msgr2=0x7fbea419d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:42.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbea419cf20 0x7fbea419d3d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c005e30 tx=0x7fbe9c032420 comp rx=0 tx=0).stop 2024-04-21T08:56:42.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 -- 172.21.15.7:0/4170818117 shutdown_connections 2024-04-21T08:56:42.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbea419cf20 0x7fbea419d3d0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbe900a3da0 0x7fbe900a6260 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbea4106160 0x7fbea4198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 --2- 172.21.15.7:0/4170818117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea4101090 0x7fbea4198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:42.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 -- 172.21.15.7:0/4170818117 >> 172.21.15.7:0/4170818117 conn(0x7fbea4078000 msgr2=0x7fbea40fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:42.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 -- 172.21.15.7:0/4170818117 shutdown_connections 2024-04-21T08:56:42.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:42.960+0000 7fbeaa89a640 1 -- 172.21.15.7:0/4170818117 wait complete. 2024-04-21T08:56:42.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:56:43.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.076+0000 7f6c10843640 1 -- 172.21.15.7:0/70205867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c0c106160 msgr2=0x7f6c0c10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:43.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.076+0000 7f6c10843640 1 --2- 172.21.15.7:0/70205867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c0c106160 0x7f6c0c10dd30 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009ca0 tx=0x7f6c0402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.076+0000 7f6c10843640 1 -- 172.21.15.7:0/70205867 shutdown_connections 2024-04-21T08:56:43.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.076+0000 7f6c10843640 1 --2- 172.21.15.7:0/70205867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c0c106160 0x7f6c0c10dd30 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.076+0000 7f6c10843640 1 --2- 172.21.15.7:0/70205867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c101a40 0x7f6c0c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.076+0000 7f6c10843640 1 --2- 172.21.15.7:0/70205867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c101090 0x7f6c0c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.076+0000 7f6c10843640 1 -- 172.21.15.7:0/70205867 >> 172.21.15.7:0/70205867 conn(0x7f6c0c078000 msgr2=0x7f6c0c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:43.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.076+0000 7f6c10843640 1 -- 172.21.15.7:0/70205867 shutdown_connections 2024-04-21T08:56:43.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.076+0000 7f6c10843640 1 -- 172.21.15.7:0/70205867 wait complete. 2024-04-21T08:56:43.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.077+0000 7f6c10843640 1 Processor -- start 2024-04-21T08:56:43.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.077+0000 7f6c10843640 1 -- start start 2024-04-21T08:56:43.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.077+0000 7f6c10843640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c101090 0x7f6c0c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:43.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.077+0000 7f6c10843640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c106160 0x7f6c0c198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:43.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.077+0000 7f6c10843640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c0c19cfc0 0x7f6c0c19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:43.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.077+0000 7f6c10843640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c1110d0 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.077+0000 7f6c10843640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c110f50 con 0x7f6c0c101090 2024-04-21T08:56:43.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.077+0000 7f6c10843640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c111250 con 0x7f6c0c106160 2024-04-21T08:56:43.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.077+0000 7f6c0ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c0c19cfc0 0x7f6c0c19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:43.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.077+0000 7f6c0ad9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c0c19cfc0 0x7f6c0c19d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56800/0 (socket says 172.21.15.7:56800) 2024-04-21T08:56:43.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.077+0000 7f6c0ad9d640 1 -- 172.21.15.7:0/2465053508 learned_addr learned my addr 172.21.15.7:0/2465053508 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:43.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6c09d9b640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c106160 0x7f6c0c198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:43.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6c0a59c640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c101090 0x7f6c0c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:43.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6c0ad9d640 1 -- 172.21.15.7:0/2465053508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c106160 msgr2=0x7f6c0c198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:43.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6c0ad9d640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c106160 0x7f6c0c198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6c0ad9d640 1 -- 172.21.15.7:0/2465053508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c101090 msgr2=0x7f6c0c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:43.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6c0ad9d640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c101090 0x7f6c0c198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6c0ad9d640 1 -- 172.21.15.7:0/2465053508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c19dab0 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6c0a59c640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c101090 0x7f6c0c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:56:43.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6c0ad9d640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c0c19cfc0 0x7f6c0c19d470 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6c04005e30 tx=0x7f6c040346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:43.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6bfb7fe640 1 -- 172.21.15.7:0/2465053508 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04041020 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6bfb7fe640 1 -- 172.21.15.7:0/2465053508 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c04032970 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6c10843640 1 -- 172.21.15.7:0/2465053508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c0c19dd40 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6bfb7fe640 1 -- 172.21.15.7:0/2465053508 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04039040 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.078+0000 7f6c10843640 1 -- 172.21.15.7:0/2465053508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c0c1a5500 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.079+0000 7f6c10843640 1 -- 172.21.15.7:0/2465053508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c0c069250 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.080+0000 7f6bfb7fe640 1 -- 172.21.15.7:0/2465053508 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f6c04044020 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.081+0000 7f6bfb7fe640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bf40a3d20 0x7f6bf40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:43.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.081+0000 7f6c0a59c640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bf40a3d20 0x7f6bf40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:43.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.081+0000 7f6bfb7fe640 1 -- 172.21.15.7:0/2465053508 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7f6c040f27c0 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.081+0000 7f6c0a59c640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bf40a3d20 0x7f6bf40a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc009b50 tx=0x7f6bfc005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:43.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.085+0000 7f6bfb7fe640 1 -- 172.21.15.7:0/2465053508 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c0404b020 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.269+0000 7f6c10843640 1 -- 172.21.15.7:0/2465053508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6c0c101a40 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.270+0000 7f6bfb7fe640 1 -- 172.21.15.7:0/2465053508 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v113) ==== 103+0+11 (secure 0 0 0) 0x7f6c040beb30 con 0x7f6c0c19cfc0 2024-04-21T08:56:43.270 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:56:43.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 -- 172.21.15.7:0/2465053508 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bf40a3d20 msgr2=0x7f6bf40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:43.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bf40a3d20 0x7f6bf40a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc009b50 tx=0x7f6bfc005f90 comp rx=0 tx=0).stop 2024-04-21T08:56:43.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 -- 172.21.15.7:0/2465053508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c0c19cfc0 msgr2=0x7f6c0c19d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c0c19cfc0 0x7f6c0c19d470 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6c04005e30 tx=0x7f6c040346d0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 -- 172.21.15.7:0/2465053508 shutdown_connections 2024-04-21T08:56:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c0c19cfc0 0x7f6c0c19d470 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f6bf40a3d20 0x7f6bf40a61e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c0c106160 0x7f6c0c198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 --2- 172.21.15.7:0/2465053508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c101090 0x7f6c0c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 -- 172.21.15.7:0/2465053508 >> 172.21.15.7:0/2465053508 conn(0x7f6c0c078000 msgr2=0x7f6c0c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 -- 172.21.15.7:0/2465053508 shutdown_connections 2024-04-21T08:56:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.272+0000 7f6c10843640 1 -- 172.21.15.7:0/2465053508 wait complete. 2024-04-21T08:56:43.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-21T08:56:43.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.385+0000 7f710e2f4640 1 -- 172.21.15.7:0/4058006882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7108106f50 msgr2=0x7f7108107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:43.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.385+0000 7f710e2f4640 1 --2- 172.21.15.7:0/4058006882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7108106f50 0x7f7108107330 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009900 tx=0x7f70fc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:56:43.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.385+0000 7f710e2f4640 1 -- 172.21.15.7:0/4058006882 shutdown_connections 2024-04-21T08:56:43.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.385+0000 7f710e2f4640 1 --2- 172.21.15.7:0/4058006882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7108069c10 0x7f710810ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.385+0000 7f710e2f4640 1 --2- 172.21.15.7:0/4058006882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7108069250 0x7f71080696d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.385+0000 7f710e2f4640 1 --2- 172.21.15.7:0/4058006882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7108106f50 0x7f7108107330 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.385+0000 7f710e2f4640 1 -- 172.21.15.7:0/4058006882 >> 172.21.15.7:0/4058006882 conn(0x7f7108077c40 msgr2=0x7f7108078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:43.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.385+0000 7f710e2f4640 1 -- 172.21.15.7:0/4058006882 shutdown_connections 2024-04-21T08:56:43.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.385+0000 7f710e2f4640 1 -- 172.21.15.7:0/4058006882 wait complete. 2024-04-21T08:56:43.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f710e2f4640 1 Processor -- start 2024-04-21T08:56:43.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f710e2f4640 1 -- start start 2024-04-21T08:56:43.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f710e2f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7108069250 0x7f71081986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:43.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f710e2f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7108069c10 0x7f7108198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:43.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f710e2f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7108106f50 0x7f710819cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:43.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f710e2f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f71081121d0 con 0x7f7108069c10 2024-04-21T08:56:43.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f710e2f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7108112050 con 0x7f7108106f50 2024-04-21T08:56:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f710e2f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7108112350 con 0x7f7108069250 2024-04-21T08:56:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f71077fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7108069c10 0x7f7108198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f71077fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7108069c10 0x7f7108198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56814/0 (socket says 172.21.15.7:56814) 2024-04-21T08:56:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f71077fe640 1 -- 172.21.15.7:0/974070730 learned_addr learned my addr 172.21.15.7:0/974070730 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:56:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f7107fff640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7108069250 0x7f71081986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f710c891640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7108106f50 0x7f710819cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.386+0000 7f71077fe640 1 -- 172.21.15.7:0/974070730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7108069250 msgr2=0x7f71081986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.387+0000 7f71077fe640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7108069250 0x7f71081986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.387+0000 7f71077fe640 1 -- 172.21.15.7:0/974070730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7108106f50 msgr2=0x7f710819cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.387+0000 7f71077fe640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7108106f50 0x7f710819cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.387+0000 7f71077fe640 1 -- 172.21.15.7:0/974070730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710819d670 con 0x7f7108069c10 2024-04-21T08:56:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.387+0000 7f71077fe640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7108069c10 0x7f7108198c10 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f70f800d950 tx=0x7f70f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.387+0000 7f71057fa640 1 -- 172.21.15.7:0/974070730 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f8017030 con 0x7f7108069c10 2024-04-21T08:56:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.387+0000 7f71057fa640 1 -- 172.21.15.7:0/974070730 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70f8004020 con 0x7f7108069c10 2024-04-21T08:56:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.387+0000 7f710e2f4640 1 -- 172.21.15.7:0/974070730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f710819d960 con 0x7f7108069c10 2024-04-21T08:56:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.387+0000 7f71057fa640 1 -- 172.21.15.7:0/974070730 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f80047a0 con 0x7f7108069c10 2024-04-21T08:56:43.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.387+0000 7f710e2f4640 1 -- 172.21.15.7:0/974070730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71081a51f0 con 0x7f7108069c10 2024-04-21T08:56:43.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.388+0000 7f710e2f4640 1 -- 172.21.15.7:0/974070730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7108068e00 con 0x7f7108069c10 2024-04-21T08:56:43.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.389+0000 7f71057fa640 1 -- 172.21.15.7:0/974070730 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f70f8009a30 con 0x7f7108069c10 2024-04-21T08:56:43.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.390+0000 7f71057fa640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f70f00a3d20 0x7f70f00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:43.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.390+0000 7f7107fff640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f70f00a3d20 0x7f70f00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:43.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.390+0000 7f71057fa640 1 -- 172.21.15.7:0/974070730 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9552+0+0 (secure 0 0 0) 0x7f70f80d0d60 con 0x7f7108069c10 2024-04-21T08:56:43.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.390+0000 7f7107fff640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f70f00a3d20 0x7f70f00a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f70fc005ab0 tx=0x7f70fc005a20 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:43.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.394+0000 7f71057fa640 1 -- 172.21.15.7:0/974070730 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f70f809d000 con 0x7f7108069c10 2024-04-21T08:56:43.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:43 smithi016 ceph-mon[31259]: pgmap v381: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.5 KiB/s wr, 2 op/s 2024-04-21T08:56:43.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:43 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3053794859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:56:43.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:43 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2187725423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:56:43.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:43 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4170818117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:56:43.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:43 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2465053508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:56:43.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:43 smithi077 ceph-mon[31615]: pgmap v381: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.5 KiB/s wr, 2 op/s 2024-04-21T08:56:43.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:43 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3053794859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:56:43.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:43 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2187725423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:56:43.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:43 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4170818117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:56:43.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:43 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2465053508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:56:43.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.633+0000 7f710e2f4640 1 -- 172.21.15.7:0/974070730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f71081999d0 con 0x7f7108069c10 2024-04-21T08:56:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.641+0000 7f71057fa640 1 -- 172.21.15.7:0/974070730 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746052 (secure 0 0 0) 0x7f71081999d0 con 0x7f7108069c10 2024-04-21T08:56:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:56:43.652 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:56:43.652 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 26, 2024-04-21T08:56:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "active_gid": 15408, 2024-04-21T08:56:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "active_name": "x", 2024-04-21T08:56:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "active_addrs": { 2024-04-21T08:56:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-04-21T08:56:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-04-21T08:56:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6800", 2024-04-21T08:56:43.653 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3443135673 2024-04-21T08:56:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2024-04-21T08:56:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-04-21T08:56:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 3443135673 2024-04-21T08:56:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:56:43.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "active_addr": "172.21.15.7:6801/3443135673", 2024-04-21T08:56:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "active_change": "2024-04-21T08:44:41.244954+0000", 2024-04-21T08:56:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "active_mgr_features": 4540701547738038271, 2024-04-21T08:56:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "available": true, 2024-04-21T08:56:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "standbys": [ 2024-04-21T08:56:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "gid": 25058, 2024-04-21T08:56:43.655 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "y", 2024-04-21T08:56:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_features": 4540701547738038271, 2024-04-21T08:56:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: "available_modules": [ 2024-04-21T08:56:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "alerts", 2024-04-21T08:56:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:43.656 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.657 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T08:56:43.658 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.658 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.658 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.658 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:43.658 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:43.658 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.658 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.659 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.659 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.659 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.659 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.659 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.659 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.659 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.659 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.659 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.660 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.660 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.660 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.660 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.660 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.660 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.660 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.661 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:43.661 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:43.661 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.661 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.661 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.661 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.661 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.661 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.662 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.662 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.662 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.662 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.662 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.662 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:43.663 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:43.663 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.663 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.663 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.663 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:43.663 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.663 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.663 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.663 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.664 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.664 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.664 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.664 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.664 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.664 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.664 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.664 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.664 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.665 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.665 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:43.665 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:43.665 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.665 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.665 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.665 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.666 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.666 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.666 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.666 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.666 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.666 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.666 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.667 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_destination": { 2024-04-21T08:56:43.667 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_destination", 2024-04-21T08:56:43.667 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.667 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.667 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.667 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.667 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.667 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.668 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.668 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email address to send alerts to", 2024-04-21T08:56:43.668 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.668 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.668 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.668 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_from_name": { 2024-04-21T08:56:43.668 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_from_name", 2024-04-21T08:56:43.668 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.669 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.669 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.669 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "Ceph", 2024-04-21T08:56:43.669 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.669 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.669 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.669 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email From: name", 2024-04-21T08:56:43.669 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.670 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.670 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.670 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_host": { 2024-04-21T08:56:43.670 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_host", 2024-04-21T08:56:43.670 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.670 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.670 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.671 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.671 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.671 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.671 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.671 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP server", 2024-04-21T08:56:43.671 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.671 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.671 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.672 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_password": { 2024-04-21T08:56:43.672 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_password", 2024-04-21T08:56:43.672 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.672 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.672 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.672 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.672 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.672 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.673 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.673 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Password to authenticate with", 2024-04-21T08:56:43.673 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.673 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.673 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.673 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_port": { 2024-04-21T08:56:43.673 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_port", 2024-04-21T08:56:43.673 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.674 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.674 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.674 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "465", 2024-04-21T08:56:43.674 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.674 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.674 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.674 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP port", 2024-04-21T08:56:43.674 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.675 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.675 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.675 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_sender": { 2024-04-21T08:56:43.675 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_sender", 2024-04-21T08:56:43.675 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.675 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.675 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.676 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.676 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.676 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.676 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.676 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP envelope sender", 2024-04-21T08:56:43.676 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.676 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.676 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.677 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_ssl": { 2024-04-21T08:56:43.677 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_ssl", 2024-04-21T08:56:43.677 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.677 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.677 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.677 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.677 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.678 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.678 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.678 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T08:56:43.678 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.678 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.678 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.678 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_user": { 2024-04-21T08:56:43.679 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_user", 2024-04-21T08:56:43.679 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.679 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.679 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.679 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.679 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.679 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.679 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.679 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "User to authenticate as", 2024-04-21T08:56:43.680 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.680 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.680 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.680 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.680 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:43.680 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "balancer", 2024-04-21T08:56:43.681 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:43.681 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:43.681 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:43.681 INFO:teuthology.orchestra.run.smithi007.stdout: "active": { 2024-04-21T08:56:43.681 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "active", 2024-04-21T08:56:43.681 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.681 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.681 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.682 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.682 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.682 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.682 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.682 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T08:56:43.682 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.682 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.682 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.683 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_time": { 2024-04-21T08:56:43.683 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_time", 2024-04-21T08:56:43.683 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.683 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.683 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.683 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0000", 2024-04-21T08:56:43.683 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.683 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.684 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.684 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T08:56:43.684 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:43.684 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.684 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.684 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_weekday": { 2024-04-21T08:56:43.684 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_weekday", 2024-04-21T08:56:43.685 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:43.685 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.685 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.685 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:43.685 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-04-21T08:56:43.685 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-04-21T08:56:43.685 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.685 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T08:56:43.685 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:43.686 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.686 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.686 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_max_iterations": { 2024-04-21T08:56:43.686 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_max_iterations", 2024-04-21T08:56:43.686 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:43.686 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.687 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.687 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "25", 2024-04-21T08:56:43.687 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:43.687 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "250", 2024-04-21T08:56:43.687 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.687 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T08:56:43.687 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.687 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.687 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.688 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_metrics": { 2024-04-21T08:56:43.688 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_metrics", 2024-04-21T08:56:43.688 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.688 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.688 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.688 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T08:56:43.688 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.688 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.689 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.689 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T08:56:43.689 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T08:56:43.689 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.689 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.689 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_step": { 2024-04-21T08:56:43.689 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_step", 2024-04-21T08:56:43.690 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:43.690 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.690 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.690 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.5", 2024-04-21T08:56:43.690 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0.001", 2024-04-21T08:56:43.690 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "0.999", 2024-04-21T08:56:43.690 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.691 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "aggressiveness of optimization", 2024-04-21T08:56:43.691 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T08:56:43.691 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.691 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.691 INFO:teuthology.orchestra.run.smithi007.stdout: "end_time": { 2024-04-21T08:56:43.691 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_time", 2024-04-21T08:56:43.691 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2359", 2024-04-21T08:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T08:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:43.692 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "end_weekday": { 2024-04-21T08:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_weekday", 2024-04-21T08:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.693 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-04-21T08:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-04-21T08:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T08:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.695 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.696 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.697 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.697 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.697 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.697 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.697 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.697 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:43.697 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:43.698 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.698 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.698 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.698 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.698 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.698 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.698 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.699 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.699 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.699 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.699 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.699 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:43.699 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:43.699 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.699 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.700 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.700 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:43.700 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.700 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.700 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.700 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.700 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.701 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.701 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.701 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.701 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.701 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.701 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.701 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.701 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.701 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.702 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:43.702 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:43.702 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.702 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.702 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.702 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.702 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.702 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.703 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.703 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.703 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.703 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.703 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.703 INFO:teuthology.orchestra.run.smithi007.stdout: "min_score": { 2024-04-21T08:56:43.703 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_score", 2024-04-21T08:56:43.703 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:43.704 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.704 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.704 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:43.704 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.704 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.704 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.704 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T08:56:43.704 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.705 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.705 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.705 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-04-21T08:56:43.705 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-04-21T08:56:43.705 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.705 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.705 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.705 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "upmap", 2024-04-21T08:56:43.706 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.706 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.706 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.706 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-compat", 2024-04-21T08:56:43.706 INFO:teuthology.orchestra.run.smithi007.stdout: "none", 2024-04-21T08:56:43.706 INFO:teuthology.orchestra.run.smithi007.stdout: "read", 2024-04-21T08:56:43.706 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap", 2024-04-21T08:56:43.706 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap-read" 2024-04-21T08:56:43.707 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.707 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Balancer mode", 2024-04-21T08:56:43.707 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.707 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.707 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.707 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_ids": { 2024-04-21T08:56:43.707 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_ids", 2024-04-21T08:56:43.707 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.708 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.708 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.708 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.708 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.708 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.708 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.708 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T08:56:43.708 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.708 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.708 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.709 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:43.709 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:43.709 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.709 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.709 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.709 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:43.709 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.709 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.710 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.710 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T08:56:43.710 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.710 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.710 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.710 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_deviation": { 2024-04-21T08:56:43.710 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_deviation", 2024-04-21T08:56:43.710 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.711 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.711 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.711 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:43.711 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:43.711 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.711 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.711 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T08:56:43.711 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T08:56:43.712 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.712 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.712 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.712 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_optimizations": { 2024-04-21T08:56:43.712 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_optimizations", 2024-04-21T08:56:43.712 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:43.712 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.712 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.713 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:43.713 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.713 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.713 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.713 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T08:56:43.713 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.713 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.713 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.713 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.713 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.714 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:43.714 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm", 2024-04-21T08:56:43.714 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:43.714 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:43.714 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:43.714 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_down_multiplier": { 2024-04-21T08:56:43.714 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_down_multiplier", 2024-04-21T08:56:43.715 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:43.715 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.715 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.715 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-04-21T08:56:43.715 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.715 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.715 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.715 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T08:56:43.715 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.715 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.716 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.716 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_refresh_rate": { 2024-04-21T08:56:43.716 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_refresh_rate", 2024-04-21T08:56:43.716 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.716 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.716 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.716 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "20", 2024-04-21T08:56:43.717 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.717 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.717 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.717 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T08:56:43.717 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.717 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.717 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.717 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_starting_port": { 2024-04-21T08:56:43.717 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_starting_port", 2024-04-21T08:56:43.718 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.718 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.718 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.718 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4721", 2024-04-21T08:56:43.718 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.718 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.718 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.719 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T08:56:43.719 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.719 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.719 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.719 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_ptrace": { 2024-04-21T08:56:43.719 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_ptrace", 2024-04-21T08:56:43.720 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.720 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.720 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.720 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.720 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.720 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.720 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.720 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T08:56:43.721 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T08:56:43.721 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.721 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.721 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_interval": { 2024-04-21T08:56:43.721 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_interval", 2024-04-21T08:56:43.721 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.721 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.722 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.722 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:43.722 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.722 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.722 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.722 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T08:56:43.722 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.723 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.723 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.723 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_memory_target_ratio": { 2024-04-21T08:56:43.723 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T08:56:43.723 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:43.723 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.723 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.724 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.7", 2024-04-21T08:56:43.724 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.724 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.724 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.724 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T08:56:43.724 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.724 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.724 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.725 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm_log_destination": { 2024-04-21T08:56:43.725 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm_log_destination", 2024-04-21T08:56:43.725 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.725 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.725 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.725 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.725 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.726 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.726 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.726 INFO:teuthology.orchestra.run.smithi007.stdout: "file", 2024-04-21T08:56:43.726 INFO:teuthology.orchestra.run.smithi007.stdout: "file,syslog", 2024-04-21T08:56:43.726 INFO:teuthology.orchestra.run.smithi007.stdout: "syslog" 2024-04-21T08:56:43.726 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.726 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T08:56:43.726 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.727 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.727 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.727 INFO:teuthology.orchestra.run.smithi007.stdout: "cgroups_split": { 2024-04-21T08:56:43.727 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cgroups_split", 2024-04-21T08:56:43.727 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.727 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.727 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.728 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.728 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.728 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.728 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.728 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T08:56:43.728 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.728 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.728 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.729 INFO:teuthology.orchestra.run.smithi007.stdout: "config_checks_enabled": { 2024-04-21T08:56:43.729 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_checks_enabled", 2024-04-21T08:56:43.729 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.729 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.729 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.729 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.729 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.729 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.729 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.730 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T08:56:43.730 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.730 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.730 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.730 INFO:teuthology.orchestra.run.smithi007.stdout: "config_dashboard": { 2024-04-21T08:56:43.730 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_dashboard", 2024-04-21T08:56:43.730 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.730 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.731 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.731 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.731 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.731 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.731 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.731 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T08:56:43.731 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.731 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.732 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.732 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_alertmanager": { 2024-04-21T08:56:43.732 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_alertmanager", 2024-04-21T08:56:43.732 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.732 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.732 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.732 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T08:56:43.733 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.733 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.733 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.733 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:43.733 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.733 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.733 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.733 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_base": { 2024-04-21T08:56:43.734 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_base", 2024-04-21T08:56:43.734 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.734 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.734 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.734 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T08:56:43.734 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.734 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.734 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.734 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Container image name, without the tag", 2024-04-21T08:56:43.735 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.735 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.735 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.735 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_elasticsearch": { 2024-04-21T08:56:43.735 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_elasticsearch", 2024-04-21T08:56:43.735 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.735 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.735 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.736 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T08:56:43.736 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.736 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.736 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.736 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "elasticsearch container image", 2024-04-21T08:56:43.736 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.736 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.736 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.737 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_grafana": { 2024-04-21T08:56:43.737 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_grafana", 2024-04-21T08:56:43.737 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.737 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.737 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.737 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T08:56:43.737 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.737 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.738 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.738 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:43.738 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.738 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.738 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.738 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_haproxy": { 2024-04-21T08:56:43.738 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_haproxy", 2024-04-21T08:56:43.738 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.739 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.739 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.739 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T08:56:43.739 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.739 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.739 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.739 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "HAproxy container image", 2024-04-21T08:56:43.739 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.740 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.740 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.740 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_agent": { 2024-04-21T08:56:43.740 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_agent", 2024-04-21T08:56:43.740 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.740 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.740 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.740 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T08:56:43.741 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.741 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.741 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.741 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger agent container image", 2024-04-21T08:56:43.741 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.741 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.741 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.741 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_collector": { 2024-04-21T08:56:43.742 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_collector", 2024-04-21T08:56:43.742 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.742 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.742 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.742 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T08:56:43.742 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.742 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.742 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.743 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger collector container image", 2024-04-21T08:56:43.743 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.743 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.743 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.743 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_query": { 2024-04-21T08:56:43.743 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_query", 2024-04-21T08:56:43.743 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.743 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.744 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.744 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T08:56:43.744 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.744 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.744 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.744 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger query container image", 2024-04-21T08:56:43.744 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.744 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.745 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.745 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_keepalived": { 2024-04-21T08:56:43.745 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_keepalived", 2024-04-21T08:56:43.745 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.745 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.745 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.745 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T08:56:43.745 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.746 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.746 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.746 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Keepalived container image", 2024-04-21T08:56:43.746 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.746 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.746 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.747 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_loki": { 2024-04-21T08:56:43.747 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_loki", 2024-04-21T08:56:43.747 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.747 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.747 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.747 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T08:56:43.747 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.748 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.748 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.748 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Loki container image", 2024-04-21T08:56:43.748 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.748 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.748 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.748 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_node_exporter": { 2024-04-21T08:56:43.748 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_node_exporter", 2024-04-21T08:56:43.749 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.749 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.749 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.749 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T08:56:43.749 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.749 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.749 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.749 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:43.749 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.749 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.750 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.750 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_nvmeof": { 2024-04-21T08:56:43.750 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_nvmeof", 2024-04-21T08:56:43.750 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.750 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.750 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.750 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T08:56:43.750 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.751 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.751 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.751 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Nvme-of container image", 2024-04-21T08:56:43.751 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.751 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.751 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.751 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.751 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_prometheus": { 2024-04-21T08:56:43.751 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_prometheus", 2024-04-21T08:56:43.752 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.752 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.752 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.752 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T08:56:43.752 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.752 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.752 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.752 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:43.752 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.753 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.753 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.753 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_promtail": { 2024-04-21T08:56:43.753 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_promtail", 2024-04-21T08:56:43.753 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.753 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.753 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.753 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T08:56:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Promtail container image", 2024-04-21T08:56:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.754 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_samba": { 2024-04-21T08:56:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_samba", 2024-04-21T08:56:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T08:56:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.755 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Samba/SMB container image", 2024-04-21T08:56:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_snmp_gateway": { 2024-04-21T08:56:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_snmp_gateway", 2024-04-21T08:56:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.756 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T08:56:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SNMP Gateway container image", 2024-04-21T08:56:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "container_init": { 2024-04-21T08:56:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_init", 2024-04-21T08:56:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.758 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T08:56:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.759 INFO:teuthology.orchestra.run.smithi007.stdout: "daemon_cache_timeout": { 2024-04-21T08:56:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "daemon_cache_timeout", 2024-04-21T08:56:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.760 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T08:56:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "default_cephadm_command_timeout": { 2024-04-21T08:56:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T08:56:43.761 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "900", 2024-04-21T08:56:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T08:56:43.762 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: "default_registry": { 2024-04-21T08:56:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_registry", 2024-04-21T08:56:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.763 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io", 2024-04-21T08:56:43.764 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.764 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.764 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.764 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T08:56:43.764 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.764 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.764 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.764 INFO:teuthology.orchestra.run.smithi007.stdout: "device_cache_timeout": { 2024-04-21T08:56:43.765 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_cache_timeout", 2024-04-21T08:56:43.765 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.765 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.765 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.765 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1800", 2024-04-21T08:56:43.765 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.765 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.765 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.765 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache device inventory", 2024-04-21T08:56:43.765 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.766 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.766 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.766 INFO:teuthology.orchestra.run.smithi007.stdout: "device_enhanced_scan": { 2024-04-21T08:56:43.766 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_enhanced_scan", 2024-04-21T08:56:43.766 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.766 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.766 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.767 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.767 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.767 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.767 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.767 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T08:56:43.767 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.767 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.767 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.767 INFO:teuthology.orchestra.run.smithi007.stdout: "facts_cache_timeout": { 2024-04-21T08:56:43.768 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "facts_cache_timeout", 2024-04-21T08:56:43.768 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.768 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.768 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.768 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:43.768 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.768 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.768 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.768 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache host facts data", 2024-04-21T08:56:43.769 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.769 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.769 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.769 INFO:teuthology.orchestra.run.smithi007.stdout: "grafana_dashboards_path": { 2024-04-21T08:56:43.769 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "grafana_dashboards_path", 2024-04-21T08:56:43.769 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.769 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.769 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.770 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T08:56:43.770 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.770 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.770 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.770 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T08:56:43.770 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.770 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.770 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.771 INFO:teuthology.orchestra.run.smithi007.stdout: "host_check_interval": { 2024-04-21T08:56:43.771 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "host_check_interval", 2024-04-21T08:56:43.771 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.771 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.771 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.771 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:43.771 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.771 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.771 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.771 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to perform a host check", 2024-04-21T08:56:43.772 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.772 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.772 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.772 INFO:teuthology.orchestra.run.smithi007.stdout: "hw_monitoring": { 2024-04-21T08:56:43.772 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hw_monitoring", 2024-04-21T08:56:43.772 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.772 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.772 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.773 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.773 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.773 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.773 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.773 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T08:56:43.773 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.773 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.773 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.774 INFO:teuthology.orchestra.run.smithi007.stdout: "inventory_list_all": { 2024-04-21T08:56:43.774 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "inventory_list_all", 2024-04-21T08:56:43.774 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.774 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.774 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.774 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.774 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.774 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.774 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.775 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T08:56:43.775 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.775 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.775 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.775 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:43.775 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:43.775 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.775 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.776 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.776 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.776 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.776 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.776 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.776 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.776 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.776 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.777 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.777 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.777 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.777 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.777 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.777 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.777 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.777 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.778 INFO:teuthology.orchestra.run.smithi007.stdout: "log_refresh_metadata": { 2024-04-21T08:56:43.778 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_refresh_metadata", 2024-04-21T08:56:43.778 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.778 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.778 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.778 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.778 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.778 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.779 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.779 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T08:56:43.779 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.779 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.779 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.779 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:43.779 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:43.779 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.780 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.780 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.780 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.780 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.780 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.780 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.780 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T08:56:43.780 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.780 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.781 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.781 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:43.781 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:43.781 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.781 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.781 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.781 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:43.781 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.782 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.782 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.782 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.782 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.782 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.782 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.782 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.782 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.782 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.783 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.783 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.783 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.783 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.783 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:43.783 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:43.783 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.783 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.784 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.784 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.784 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.784 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.784 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.784 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.784 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.784 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.784 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.785 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T08:56:43.785 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T08:56:43.785 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.785 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.785 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.785 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.785 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.785 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.786 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.786 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T08:56:43.786 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.786 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.786 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.786 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T08:56:43.786 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T08:56:43.786 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.787 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.787 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.787 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "*", 2024-04-21T08:56:43.787 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.787 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.787 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.787 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T08:56:43.787 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.788 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.788 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.788 INFO:teuthology.orchestra.run.smithi007.stdout: "max_count_per_host": { 2024-04-21T08:56:43.788 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_count_per_host", 2024-04-21T08:56:43.788 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.788 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.788 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.789 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:43.789 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.789 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.789 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.789 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of daemons per service per host", 2024-04-21T08:56:43.789 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.789 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.789 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.790 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd_draining_count": { 2024-04-21T08:56:43.790 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_osd_draining_count", 2024-04-21T08:56:43.790 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.790 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.790 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.790 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:43.790 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.790 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.791 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.791 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T08:56:43.791 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.791 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.791 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.791 INFO:teuthology.orchestra.run.smithi007.stdout: "migration_current": { 2024-04-21T08:56:43.791 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "migration_current", 2024-04-21T08:56:43.791 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.792 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.792 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.792 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.792 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.792 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.792 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.792 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "internal - do not modify", 2024-04-21T08:56:43.792 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.793 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.793 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.793 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-04-21T08:56:43.793 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-04-21T08:56:43.793 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.793 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.793 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.793 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "root", 2024-04-21T08:56:43.793 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.794 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.794 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.794 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm-package", 2024-04-21T08:56:43.794 INFO:teuthology.orchestra.run.smithi007.stdout: "root" 2024-04-21T08:56:43.794 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.794 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T08:56:43.794 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.794 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.794 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.795 INFO:teuthology.orchestra.run.smithi007.stdout: "oob_default_addr": { 2024-04-21T08:56:43.795 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "oob_default_addr", 2024-04-21T08:56:43.795 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.795 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.795 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.795 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "169.254.1.1", 2024-04-21T08:56:43.795 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.796 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.796 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.796 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T08:56:43.796 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.796 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.796 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.796 INFO:teuthology.orchestra.run.smithi007.stdout: "prometheus_alerts_path": { 2024-04-21T08:56:43.797 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus_alerts_path", 2024-04-21T08:56:43.797 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.797 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.797 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.797 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T08:56:43.797 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.797 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.797 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.798 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T08:56:43.798 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.798 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.798 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.798 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_insecure": { 2024-04-21T08:56:43.798 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_insecure", 2024-04-21T08:56:43.798 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.799 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.799 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.799 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.799 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.799 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.799 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.799 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T08:56:43.799 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.799 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.800 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.800 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_password": { 2024-04-21T08:56:43.800 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_password", 2024-04-21T08:56:43.800 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.800 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.800 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.800 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.800 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.801 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.801 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.801 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T08:56:43.801 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.801 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.801 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.801 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_url": { 2024-04-21T08:56:43.801 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_url", 2024-04-21T08:56:43.802 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.802 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.802 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.802 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.802 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.802 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.802 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.802 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T08:56:43.803 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.803 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.803 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.803 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_username": { 2024-04-21T08:56:43.803 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_username", 2024-04-21T08:56:43.803 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.803 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.804 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.804 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.804 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.804 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.804 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.804 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T08:56:43.804 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.804 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.805 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.805 INFO:teuthology.orchestra.run.smithi007.stdout: "secure_monitoring_stack": { 2024-04-21T08:56:43.805 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "secure_monitoring_stack", 2024-04-21T08:56:43.805 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.805 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.805 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.805 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.806 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.806 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.806 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.806 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T08:56:43.806 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.806 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.806 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.806 INFO:teuthology.orchestra.run.smithi007.stdout: "service_discovery_port": { 2024-04-21T08:56:43.807 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "service_discovery_port", 2024-04-21T08:56:43.807 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.807 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.807 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.807 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8765", 2024-04-21T08:56:43.807 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.807 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.807 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.808 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "cephadm service discovery port", 2024-04-21T08:56:43.808 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.808 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.808 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.808 INFO:teuthology.orchestra.run.smithi007.stdout: "ssh_config_file": { 2024-04-21T08:56:43.808 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssh_config_file", 2024-04-21T08:56:43.808 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.808 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.809 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.809 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.809 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.809 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.809 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.809 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T08:56:43.809 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.809 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.809 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.810 INFO:teuthology.orchestra.run.smithi007.stdout: "use_agent": { 2024-04-21T08:56:43.810 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_agent", 2024-04-21T08:56:43.810 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.810 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.810 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.810 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.810 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.811 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.811 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T08:56:43.811 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.811 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.811 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.811 INFO:teuthology.orchestra.run.smithi007.stdout: "use_repo_digest": { 2024-04-21T08:56:43.812 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_repo_digest", 2024-04-21T08:56:43.812 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.812 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.812 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.812 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.812 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.812 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.812 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.812 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T08:56:43.813 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.813 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.813 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.813 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_failed_host_check": { 2024-04-21T08:56:43.813 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_failed_host_check", 2024-04-21T08:56:43.813 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.813 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.814 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.814 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.814 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.814 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.814 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.814 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T08:56:43.814 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.814 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.815 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.815 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_daemons": { 2024-04-21T08:56:43.815 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_daemons", 2024-04-21T08:56:43.815 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.815 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.815 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.815 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.816 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.816 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.816 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.816 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T08:56:43.816 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.816 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.816 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_hosts": { 2024-04-21T08:56:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_hosts", 2024-04-21T08:56:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.817 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T08:56:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.818 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crash", 2024-04-21T08:56:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.819 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.820 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.822 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:43.822 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:43.822 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.822 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.822 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.822 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.822 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.822 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.822 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.823 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.823 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.823 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.823 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.823 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:43.823 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:43.823 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.824 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.824 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.824 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:43.824 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.824 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.824 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.824 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.824 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.825 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.825 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.825 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.825 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.825 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.825 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.825 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.825 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.826 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.826 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:43.826 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:43.826 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.826 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.826 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.826 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.827 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.827 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.827 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.827 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.827 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.827 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.827 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.827 INFO:teuthology.orchestra.run.smithi007.stdout: "retain_interval": { 2024-04-21T08:56:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retain_interval", 2024-04-21T08:56:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "31536000", 2024-04-21T08:56:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T08:56:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_recent_interval": { 2024-04-21T08:56:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_recent_interval", 2024-04-21T08:56:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1209600", 2024-04-21T08:56:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T08:56:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dashboard", 2024-04-21T08:56:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T08:56:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T08:56:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T08:56:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T08:56:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T08:56:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T08:56:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_ENABLED": { 2024-04-21T08:56:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T08:56:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T08:56:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T08:56:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T08:56:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T08:56:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T08:56:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T08:56:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T08:56:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T08:56:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T08:56:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T08:56:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T08:56:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T08:56:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T08:56:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T08:56:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T08:56:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T08:56:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.851 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.851 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.851 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.851 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T08:56:43.851 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T08:56:43.851 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.851 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.851 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.852 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.852 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.852 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.852 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.852 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.852 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.852 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.852 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.853 INFO:teuthology.orchestra.run.smithi007.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T08:56:43.853 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T08:56:43.853 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.853 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.853 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.853 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.853 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.853 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.853 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.854 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.854 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.854 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.854 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.854 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T08:56:43.854 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T08:56:43.854 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.855 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.855 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.855 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-04-21T08:56:43.855 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.855 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.855 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.855 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.855 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.855 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.856 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.856 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T08:56:43.856 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T08:56:43.856 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.856 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.856 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.857 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.857 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.857 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.857 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.857 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.857 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.857 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.857 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.858 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_URL": { 2024-04-21T08:56:43.858 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_URL", 2024-04-21T08:56:43.858 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.858 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.858 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.858 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.858 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T08:56:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T08:56:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-04-21T08:56:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.860 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.861 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T08:56:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T08:56:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.862 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.863 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T08:56:43.864 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T08:56:43.864 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.864 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.864 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.864 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.864 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.864 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.864 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.865 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.865 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.865 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.865 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.865 INFO:teuthology.orchestra.run.smithi007.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T08:56:43.865 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T08:56:43.865 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.866 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.866 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.866 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.866 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.866 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.866 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.866 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.866 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.866 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.867 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.867 INFO:teuthology.orchestra.run.smithi007.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T08:56:43.867 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T08:56:43.867 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.867 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.867 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.867 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.868 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.868 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.868 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.868 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.868 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.868 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.868 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.869 INFO:teuthology.orchestra.run.smithi007.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T08:56:43.869 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T08:56:43.869 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.869 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.869 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.869 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "[]", 2024-04-21T08:56:43.869 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.869 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.869 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.870 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.870 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.870 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.870 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.870 INFO:teuthology.orchestra.run.smithi007.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T08:56:43.870 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T08:56:43.870 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.871 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.871 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.871 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "{}", 2024-04-21T08:56:43.871 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.871 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.871 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.871 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.871 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.872 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.872 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.872 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T08:56:43.872 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T08:56:43.872 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.872 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.872 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.872 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.873 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.873 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.873 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.873 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.873 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.873 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.873 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.874 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T08:56:43.874 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T08:56:43.874 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.874 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.874 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.874 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.874 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.874 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.875 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.875 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.875 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.875 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.875 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.875 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T08:56:43.875 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T08:56:43.876 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.876 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.876 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.876 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.876 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.876 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.876 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.876 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.877 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.877 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.877 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.877 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T08:56:43.877 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T08:56:43.877 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.877 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.878 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.878 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.878 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.878 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.878 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.878 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.878 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.878 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.879 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.879 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T08:56:43.879 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T08:56:43.879 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.879 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.879 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.879 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.879 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.880 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.880 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.880 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.880 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.880 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.880 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.880 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T08:56:43.880 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T08:56:43.881 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.881 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.881 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.881 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.881 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.881 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.881 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.881 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.882 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.882 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.882 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.882 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T08:56:43.882 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T08:56:43.882 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.882 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.883 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.883 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.883 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.883 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.883 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.883 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.883 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.883 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.883 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.884 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T08:56:43.884 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T08:56:43.884 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.884 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.884 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.884 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.884 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.884 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.885 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.885 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.885 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.885 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.885 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.885 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T08:56:43.885 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T08:56:43.886 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.886 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.886 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.886 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.886 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.886 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.886 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.887 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.887 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.887 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.887 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.887 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T08:56:43.887 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T08:56:43.887 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.888 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.888 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.888 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.888 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.888 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.888 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.888 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.888 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T08:56:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T08:56:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.889 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T08:56:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.890 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T08:56:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T08:56:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:43.891 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.892 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T08:56:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T08:56:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8", 2024-04-21T08:56:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.893 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T08:56:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T08:56:43.894 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.895 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.895 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.895 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "45", 2024-04-21T08:56:43.895 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.895 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.895 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.895 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.895 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.895 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.895 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.896 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T08:56:43.896 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T08:56:43.896 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.896 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.896 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.896 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.896 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.896 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.897 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.897 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.897 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.897 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.897 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.897 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T08:56:43.898 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T08:56:43.898 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.898 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.898 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.898 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-04-21T08:56:43.898 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.898 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.898 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.899 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.899 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.899 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.899 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.899 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T08:56:43.899 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T08:56:43.899 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.899 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.899 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.900 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.900 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.900 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.900 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.900 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.900 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.900 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.900 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.901 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T08:56:43.901 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T08:56:43.901 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.901 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.901 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.901 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.901 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.902 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.902 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.902 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.902 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.902 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.902 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.902 INFO:teuthology.orchestra.run.smithi007.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T08:56:43.903 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T08:56:43.903 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.903 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.903 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.903 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.903 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.903 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.903 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.903 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.904 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.904 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.904 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.904 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T08:56:43.904 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T08:56:43.904 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.904 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.904 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.905 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:43.905 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.905 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.905 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.905 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.905 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.905 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.905 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.906 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T08:56:43.906 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T08:56:43.906 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.906 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.906 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.906 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:43.906 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.907 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.907 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.907 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.907 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.907 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.907 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.907 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T08:56:43.907 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T08:56:43.908 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.908 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.908 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.908 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:43.908 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.908 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.908 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.908 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.908 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.909 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.909 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.909 INFO:teuthology.orchestra.run.smithi007.stdout: "cross_origin_url": { 2024-04-21T08:56:43.909 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cross_origin_url", 2024-04-21T08:56:43.909 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.909 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.909 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.910 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.910 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.910 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.910 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.910 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.910 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.910 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.910 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.911 INFO:teuthology.orchestra.run.smithi007.stdout: "crt_file": { 2024-04-21T08:56:43.911 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crt_file", 2024-04-21T08:56:43.911 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.911 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.911 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.911 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.912 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.912 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.912 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.912 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.912 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.912 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.912 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.913 INFO:teuthology.orchestra.run.smithi007.stdout: "debug": { 2024-04-21T08:56:43.913 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "debug", 2024-04-21T08:56:43.913 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.913 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.913 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.913 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.913 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.913 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.913 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.914 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable/disable debug options", 2024-04-21T08:56:43.914 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.914 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.914 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.914 INFO:teuthology.orchestra.run.smithi007.stdout: "jwt_token_ttl": { 2024-04-21T08:56:43.914 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "jwt_token_ttl", 2024-04-21T08:56:43.914 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.915 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.915 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.915 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "28800", 2024-04-21T08:56:43.915 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.915 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.915 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.915 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.915 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.915 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.916 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.916 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-04-21T08:56:43.916 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-04-21T08:56:43.916 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.916 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.916 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.916 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.917 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.917 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.917 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.917 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.917 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.917 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.917 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.918 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:43.918 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:43.918 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.918 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.918 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.918 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.918 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.918 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.919 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.919 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.919 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.919 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.919 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.919 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.919 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.920 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.920 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.920 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.920 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.920 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.920 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:43.920 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.922 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.922 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.922 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:43.922 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:43.922 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.922 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.922 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.923 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:43.923 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.923 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.923 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.923 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.923 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.923 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.923 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.924 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.924 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.924 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.924 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.924 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.924 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.924 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.925 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:43.925 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:43.925 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.925 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.925 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.925 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.925 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.926 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.926 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.926 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.926 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.926 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.926 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.926 INFO:teuthology.orchestra.run.smithi007.stdout: "motd": { 2024-04-21T08:56:43.927 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "motd", 2024-04-21T08:56:43.927 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.927 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.927 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.927 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.927 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.927 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.927 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.928 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "The message of the day", 2024-04-21T08:56:43.928 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.928 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.928 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.928 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_resolve_ip_addr": { 2024-04-21T08:56:43.928 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T08:56:43.928 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.928 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.929 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.929 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.929 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.929 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.929 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.929 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.929 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.929 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.929 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.930 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-04-21T08:56:43.930 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-04-21T08:56:43.930 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.930 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.930 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.930 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-04-21T08:56:43.930 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.931 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.931 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.931 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.931 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.931 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.931 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.931 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-04-21T08:56:43.931 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-04-21T08:56:43.932 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.932 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.932 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.932 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8080", 2024-04-21T08:56:43.932 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.932 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.932 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.932 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.933 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.933 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.933 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.933 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-04-21T08:56:43.933 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-04-21T08:56:43.933 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.933 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.933 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.934 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.934 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.934 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.934 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.934 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.934 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.934 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.934 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.935 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl_server_port": { 2024-04-21T08:56:43.935 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl_server_port", 2024-04-21T08:56:43.935 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.935 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.935 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.935 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8443", 2024-04-21T08:56:43.935 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.935 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.936 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.936 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.936 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.936 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.936 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.936 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-04-21T08:56:43.936 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-04-21T08:56:43.937 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.937 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.937 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.937 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "redirect", 2024-04-21T08:56:43.937 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.937 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.937 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.937 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.937 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect" 2024-04-21T08:56:43.938 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.938 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.938 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.938 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.938 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.938 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-04-21T08:56:43.938 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:43.939 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.939 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.939 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.939 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-04-21T08:56:43.939 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-04-21T08:56:43.939 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-04-21T08:56:43.939 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.939 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.939 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.940 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.940 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.940 INFO:teuthology.orchestra.run.smithi007.stdout: "url_prefix": { 2024-04-21T08:56:43.940 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url_prefix", 2024-04-21T08:56:43.940 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.940 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.940 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.940 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.941 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.941 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.941 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.941 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.941 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.941 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.941 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.941 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.941 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.942 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:43.942 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "devicehealth", 2024-04-21T08:56:43.942 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:43.942 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:43.942 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:43.942 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_monitoring": { 2024-04-21T08:56:43.942 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_monitoring", 2024-04-21T08:56:43.943 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.943 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.943 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.943 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.943 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.943 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.943 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.943 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "monitor device health metrics", 2024-04-21T08:56:43.943 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.944 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.944 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.944 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:43.944 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:43.944 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.944 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.944 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.945 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.945 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.945 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.945 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.945 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.945 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.945 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.945 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.945 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.946 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.946 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.946 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.946 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.946 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.946 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.947 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:43.947 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:43.947 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.947 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.947 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.947 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.947 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.947 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.948 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.948 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.948 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.948 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.948 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.948 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:43.948 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:43.949 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.949 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.949 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.949 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:43.949 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.949 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.949 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.950 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.950 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.950 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.950 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.950 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.950 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.950 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.950 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.951 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.951 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.951 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.951 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:43.951 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:43.951 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.951 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.952 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.952 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.952 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.952 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.952 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.952 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.952 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.953 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.953 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.953 INFO:teuthology.orchestra.run.smithi007.stdout: "mark_out_threshold": { 2024-04-21T08:56:43.953 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mark_out_threshold", 2024-04-21T08:56:43.953 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.953 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.953 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.954 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2419200", 2024-04-21T08:56:43.954 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.954 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.954 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.954 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T08:56:43.954 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.954 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.955 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.955 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": { 2024-04-21T08:56:43.955 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_name", 2024-04-21T08:56:43.955 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.955 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.955 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.955 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "device_health_metrics", 2024-04-21T08:56:43.955 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.956 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.956 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.956 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T08:56:43.956 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.956 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.956 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.956 INFO:teuthology.orchestra.run.smithi007.stdout: "retention_period": { 2024-04-21T08:56:43.956 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retention_period", 2024-04-21T08:56:43.957 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.957 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.957 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.957 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15552000", 2024-04-21T08:56:43.957 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.957 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.957 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.957 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain device health metrics", 2024-04-21T08:56:43.958 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.958 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.958 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.958 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_frequency": { 2024-04-21T08:56:43.958 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_frequency", 2024-04-21T08:56:43.958 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.958 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.959 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.959 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-04-21T08:56:43.959 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.959 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.959 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.959 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T08:56:43.959 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.960 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.960 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.960 INFO:teuthology.orchestra.run.smithi007.stdout: "self_heal": { 2024-04-21T08:56:43.960 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "self_heal", 2024-04-21T08:56:43.960 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.960 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.960 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.961 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:43.961 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.961 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.961 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.961 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T08:56:43.961 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.961 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.961 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.962 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:43.962 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:43.962 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.962 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.962 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.962 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:43.963 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.963 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.963 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.963 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T08:56:43.963 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.963 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.963 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.964 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_threshold": { 2024-04-21T08:56:43.964 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_threshold", 2024-04-21T08:56:43.964 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.964 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.964 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.964 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7257600", 2024-04-21T08:56:43.964 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.964 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.964 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.965 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T08:56:43.965 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.965 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.965 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.965 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.965 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.965 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:43.966 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "diskprediction_local", 2024-04-21T08:56:43.966 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:43.966 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:43.966 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:43.966 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:43.966 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:43.966 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.966 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.967 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.967 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.967 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.967 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.967 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.967 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.967 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.968 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.968 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.968 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.968 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.968 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.968 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.968 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.968 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.968 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.969 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:43.969 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:43.969 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.969 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.969 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.969 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.969 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.969 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.970 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.970 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.970 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.970 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.970 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.970 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:43.970 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:43.971 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.971 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.971 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.971 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:43.971 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.971 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.971 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.971 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.971 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.972 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.972 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.972 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.972 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.972 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.972 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.972 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.972 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.972 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.973 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:43.973 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:43.973 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.973 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.973 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.973 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.973 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.973 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.974 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.974 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.974 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.974 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.974 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.974 INFO:teuthology.orchestra.run.smithi007.stdout: "predict_interval": { 2024-04-21T08:56:43.974 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predict_interval", 2024-04-21T08:56:43.974 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.975 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.975 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.975 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-04-21T08:56:43.975 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.975 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.975 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.975 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.975 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.975 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.976 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.976 INFO:teuthology.orchestra.run.smithi007.stdout: "predictor_model": { 2024-04-21T08:56:43.976 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predictor_model", 2024-04-21T08:56:43.976 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.976 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.976 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.977 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "prophetstor", 2024-04-21T08:56:43.977 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.977 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.977 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.977 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.977 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.977 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.977 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.978 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:43.978 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:43.978 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.978 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.978 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.978 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:43.978 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.979 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.979 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.979 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.979 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.979 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.979 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.979 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.980 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:43.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.980 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:43.980 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "influx", 2024-04-21T08:56:43.980 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": false, 2024-04-21T08:56:43.980 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "influxdb python module not found", 2024-04-21T08:56:43.980 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:43.980 INFO:teuthology.orchestra.run.smithi007.stdout: "batch_size": { 2024-04-21T08:56:43.981 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "batch_size", 2024-04-21T08:56:43.981 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.981 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.981 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.981 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5000", 2024-04-21T08:56:43.981 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.981 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.981 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.982 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T08:56:43.982 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.982 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.982 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.982 INFO:teuthology.orchestra.run.smithi007.stdout: "database": { 2024-04-21T08:56:43.982 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "database", 2024-04-21T08:56:43.982 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.983 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.983 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.983 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "ceph", 2024-04-21T08:56:43.983 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.983 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.983 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.983 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T08:56:43.983 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.983 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.984 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.984 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.984 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": { 2024-04-21T08:56:43.984 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hostname", 2024-04-21T08:56:43.984 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.984 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.984 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.985 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.985 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.985 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.985 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.985 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server hostname", 2024-04-21T08:56:43.985 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.985 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.985 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.986 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:43.986 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:43.986 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:43.986 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.986 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.986 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "30", 2024-04-21T08:56:43.986 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "5", 2024-04-21T08:56:43.987 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.987 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.987 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T08:56:43.987 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.987 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.987 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.987 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:43.988 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:43.988 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.988 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.988 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.988 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.988 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.988 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.988 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.989 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.989 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.989 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.989 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.989 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.989 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.989 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.989 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.990 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.990 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.990 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.990 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:43.990 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:43.990 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.990 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.990 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.991 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.991 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.991 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.991 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.991 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.991 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.991 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.991 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.992 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:43.992 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:43.992 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.992 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.992 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.992 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:43.992 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.992 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.992 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:43.993 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:43.993 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:43.993 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:43.993 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:43.993 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:43.993 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:43.993 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:43.993 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.994 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.994 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.994 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.994 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:43.994 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:43.994 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:43.994 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.995 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:43.995 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:43.995 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.995 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.995 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.995 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:43.995 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.995 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.995 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.996 INFO:teuthology.orchestra.run.smithi007.stdout: "password": { 2024-04-21T08:56:43.996 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "password", 2024-04-21T08:56:43.996 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.996 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.996 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.996 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:43.996 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.997 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.997 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.997 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "password of InfluxDB server user", 2024-04-21T08:56:43.997 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.997 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.997 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.997 INFO:teuthology.orchestra.run.smithi007.stdout: "port": { 2024-04-21T08:56:43.997 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "port", 2024-04-21T08:56:43.998 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:43.998 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:43.998 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:43.998 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8086", 2024-04-21T08:56:43.998 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:43.998 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:43.998 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:43.998 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server port", 2024-04-21T08:56:43.999 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:43.999 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:43.999 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:43.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:43.999 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-04-21T08:56:43.999 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-04-21T08:56:43.999 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:43.999 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.000 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.000 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "false", 2024-04-21T08:56:44.000 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.000 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.000 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.000 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:44.000 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.000 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.001 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.001 INFO:teuthology.orchestra.run.smithi007.stdout: "threads": { 2024-04-21T08:56:44.001 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threads", 2024-04-21T08:56:44.001 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.001 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.001 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.001 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:44.001 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:44.002 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "32", 2024-04-21T08:56:44.002 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.002 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T08:56:44.002 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.002 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.002 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.002 INFO:teuthology.orchestra.run.smithi007.stdout: "username": { 2024-04-21T08:56:44.003 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "username", 2024-04-21T08:56:44.003 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.003 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.003 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.003 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.003 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.003 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.003 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.003 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "username of InfluxDB server user", 2024-04-21T08:56:44.004 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.004 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.004 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.004 INFO:teuthology.orchestra.run.smithi007.stdout: "verify_ssl": { 2024-04-21T08:56:44.004 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "verify_ssl", 2024-04-21T08:56:44.004 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.004 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.004 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.005 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "true", 2024-04-21T08:56:44.005 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.005 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.005 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.005 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:44.005 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.005 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.005 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.005 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.006 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.006 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.006 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "insights", 2024-04-21T08:56:44.006 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.006 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.006 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.006 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.006 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.007 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.007 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.007 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.007 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.007 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.007 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.007 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.007 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.008 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.008 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.008 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.008 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.008 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.008 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.008 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.008 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.008 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.008 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.009 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.009 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.009 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.009 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.009 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.009 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.009 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.009 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.010 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.010 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.010 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.010 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.010 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.010 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.010 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.011 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.011 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.011 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.011 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.011 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.011 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.011 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.011 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.011 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.012 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.012 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.012 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.012 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.012 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.012 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.012 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.012 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.012 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.013 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.013 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.013 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.013 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.013 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.013 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.013 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.013 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.014 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.014 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.014 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.014 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.014 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.014 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.014 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.014 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.015 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.015 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "iostat", 2024-04-21T08:56:44.015 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.015 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.015 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.015 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.015 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.015 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.015 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.016 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.016 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.016 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.016 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.016 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.016 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.016 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.016 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.016 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.017 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.017 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.017 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.017 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.017 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.017 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.017 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.017 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.018 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.018 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.018 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.018 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.018 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.018 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.018 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.018 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.018 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.019 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.019 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.019 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.019 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.019 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.019 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.019 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.019 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.020 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.020 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.020 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.020 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.020 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.020 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.020 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.020 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.020 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.021 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.021 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.021 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.021 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.021 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.021 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.021 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.021 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.022 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.022 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.022 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.022 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.022 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.022 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.022 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.022 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.022 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.023 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.023 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.023 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.023 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.023 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.023 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "k8sevents", 2024-04-21T08:56:44.023 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.023 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.024 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.024 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_event_retention_days": { 2024-04-21T08:56:44.024 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ceph_event_retention_days", 2024-04-21T08:56:44.024 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.024 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.024 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.024 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7", 2024-04-21T08:56:44.024 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.024 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.025 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.025 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T08:56:44.025 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.025 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.025 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.025 INFO:teuthology.orchestra.run.smithi007.stdout: "config_check_secs": { 2024-04-21T08:56:44.025 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_check_secs", 2024-04-21T08:56:44.026 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.026 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.026 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.026 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:44.026 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "10", 2024-04-21T08:56:44.026 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.026 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.026 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T08:56:44.027 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.027 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.027 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.027 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.027 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.027 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.027 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.027 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.028 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.028 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.028 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.028 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.028 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.028 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.028 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.028 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.029 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.029 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.029 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.029 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.029 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.029 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.029 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.029 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.030 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.030 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.030 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.030 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.030 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.030 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.030 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.030 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.031 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.031 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.031 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.031 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.031 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.031 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.031 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.031 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.032 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.032 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.032 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.032 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.032 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.032 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.032 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.032 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.033 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.033 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.033 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.033 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.033 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.033 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.033 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.033 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.034 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.034 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.034 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.034 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.034 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.034 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.034 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.034 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.035 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.035 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.035 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.035 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.035 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.036 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.036 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "localpool", 2024-04-21T08:56:44.036 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.036 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.036 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.036 INFO:teuthology.orchestra.run.smithi007.stdout: "failure_domain": { 2024-04-21T08:56:44.036 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "failure_domain", 2024-04-21T08:56:44.036 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.037 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.037 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.037 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "host", 2024-04-21T08:56:44.037 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.037 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.037 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.037 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "failure domain for any created local pool", 2024-04-21T08:56:44.037 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T08:56:44.037 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.038 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.038 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.038 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.038 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.038 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.038 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.038 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.039 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.039 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.039 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.039 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.039 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.039 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.039 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.039 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.040 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.040 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.040 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.040 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.040 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.040 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.040 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.040 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.041 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.041 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.041 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.041 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.041 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.041 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.041 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.041 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.041 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.042 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.042 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.042 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.042 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.042 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.042 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.042 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.043 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.043 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.043 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.043 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.043 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.043 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.043 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.043 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.044 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.044 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.044 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.044 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.044 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.044 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.044 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.045 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.045 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.045 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.045 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.045 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.045 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.045 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.046 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.046 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.046 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.046 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.046 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.046 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.047 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": { 2024-04-21T08:56:44.047 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_size", 2024-04-21T08:56:44.047 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.047 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.047 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.047 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.047 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.047 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.048 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.048 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default min_size for any created local pool", 2024-04-21T08:56:44.048 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T08:56:44.048 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.048 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.048 INFO:teuthology.orchestra.run.smithi007.stdout: "num_rep": { 2024-04-21T08:56:44.049 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "num_rep", 2024-04-21T08:56:44.049 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.049 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.049 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.049 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3", 2024-04-21T08:56:44.049 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.049 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.049 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.050 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default replica count for any created local pool", 2024-04-21T08:56:44.050 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.050 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.050 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.050 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": { 2024-04-21T08:56:44.050 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_num", 2024-04-21T08:56:44.051 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.051 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.051 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.051 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "128", 2024-04-21T08:56:44.051 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.051 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.051 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.051 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T08:56:44.052 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.052 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.052 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.052 INFO:teuthology.orchestra.run.smithi007.stdout: "prefix": { 2024-04-21T08:56:44.052 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prefix", 2024-04-21T08:56:44.052 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.052 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.053 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.053 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.053 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.053 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.053 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.053 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name prefix for any created local pool", 2024-04-21T08:56:44.053 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.053 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.053 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.054 INFO:teuthology.orchestra.run.smithi007.stdout: "subtree": { 2024-04-21T08:56:44.054 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "subtree", 2024-04-21T08:56:44.054 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.054 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.054 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.054 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "rack", 2024-04-21T08:56:44.054 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.055 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.055 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.055 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T08:56:44.055 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T08:56:44.055 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.055 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.055 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.055 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.056 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.056 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mds_autoscaler", 2024-04-21T08:56:44.056 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.056 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.056 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.056 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.056 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.057 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.057 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.057 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.057 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.057 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.057 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.057 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.057 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.058 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.058 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.058 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.058 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.058 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.058 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.058 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.058 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.058 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.059 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.059 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.059 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.059 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.059 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.059 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.059 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.060 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.060 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.060 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.060 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.060 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.060 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.060 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.060 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.061 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.061 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.061 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.061 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.061 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.061 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.061 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.061 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.062 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.062 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.062 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.062 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.062 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.062 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.062 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.063 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.063 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.063 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.063 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.063 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.063 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.064 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.064 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.064 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.064 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.064 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.064 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.064 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.064 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.065 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.065 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.065 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.065 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.065 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.065 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.066 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirroring", 2024-04-21T08:56:44.066 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.066 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.066 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.066 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.066 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.066 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.066 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.067 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.067 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.067 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.067 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.067 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.067 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.067 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.067 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.068 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.068 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.068 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.068 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.068 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.068 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.068 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.068 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.069 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.069 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.069 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.069 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.069 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.069 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.069 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.069 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.070 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.070 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.070 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.070 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.070 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.070 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.074 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.075 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.075 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.075 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.075 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.075 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.075 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.075 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.075 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.076 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.076 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.076 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.076 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.076 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.076 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.076 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.076 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.076 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.077 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.077 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.077 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.077 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.077 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.077 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.077 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.078 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.078 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.078 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.078 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.078 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.078 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.078 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.078 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.079 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.079 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.079 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.079 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "nfs", 2024-04-21T08:56:44.079 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.079 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.079 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.079 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.079 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.080 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.080 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.080 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.080 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.080 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.080 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.080 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.080 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.081 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.081 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.081 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.081 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.081 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.081 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.081 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.081 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.082 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.082 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.082 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.082 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.082 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.082 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.082 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.083 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.083 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.083 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.083 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.083 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.083 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.083 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.083 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.084 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.084 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.084 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.084 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.084 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.084 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.084 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.085 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.085 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.085 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.085 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.085 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.085 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.085 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.085 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.085 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.086 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.086 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.086 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.086 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.086 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.086 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.086 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.086 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.087 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.087 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.087 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.087 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.087 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.087 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.087 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.087 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.088 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.088 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.088 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.088 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.088 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.088 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-04-21T08:56:44.088 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.089 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.089 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.089 INFO:teuthology.orchestra.run.smithi007.stdout: "fail_fs": { 2024-04-21T08:56:44.089 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "fail_fs", 2024-04-21T08:56:44.089 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.089 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.089 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.089 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.090 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.090 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.090 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.090 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T08:56:44.090 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.090 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.090 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.091 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.091 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.091 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.091 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.091 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.091 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.091 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.091 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.092 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.092 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.092 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.092 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.092 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.092 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.092 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.092 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.092 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.093 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.093 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.093 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.093 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.093 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.093 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.093 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.094 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.094 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.094 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.094 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.094 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.094 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.094 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.094 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.095 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.095 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.095 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.095 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.095 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.095 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.095 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.096 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.096 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.096 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.096 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.096 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.096 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.096 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.097 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.097 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.097 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.097 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.097 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.097 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.097 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.098 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.098 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.098 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.098 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.098 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.098 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.098 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.098 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.099 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.099 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.099 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.099 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.099 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.099 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator": { 2024-04-21T08:56:44.099 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-04-21T08:56:44.100 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.100 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.100 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.100 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.100 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.100 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.100 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.100 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm", 2024-04-21T08:56:44.101 INFO:teuthology.orchestra.run.smithi007.stdout: "rook", 2024-04-21T08:56:44.101 INFO:teuthology.orchestra.run.smithi007.stdout: "test_orchestrator" 2024-04-21T08:56:44.101 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.101 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Orchestrator backend", 2024-04-21T08:56:44.101 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.101 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.101 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.101 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.101 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.102 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.102 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_perf_query", 2024-04-21T08:56:44.102 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.102 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.102 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.102 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.102 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.103 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.104 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.104 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.104 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.104 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.104 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.104 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.104 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.104 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.105 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.105 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.105 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.105 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.105 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.105 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.105 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.106 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.106 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.106 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.106 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.106 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.106 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.106 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.106 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.107 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.107 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.107 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.107 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.107 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.107 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.107 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.108 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.108 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.108 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.108 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.108 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.108 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.108 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.108 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.109 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.109 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.109 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.109 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.109 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.109 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.109 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.110 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.110 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.110 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.110 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.110 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.110 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.110 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.110 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.111 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.111 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.111 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.111 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.111 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.111 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_support", 2024-04-21T08:56:44.111 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.112 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.112 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.112 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.112 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.112 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.112 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.112 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.113 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.113 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.113 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.113 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.113 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.113 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.113 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.113 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.113 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.114 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.114 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.114 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.114 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.114 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.114 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.115 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.116 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.117 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.118 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.119 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.120 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.121 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.121 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_autoscaler", 2024-04-21T08:56:44.121 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.121 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.121 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.121 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.121 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.122 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.122 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.122 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.122 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.122 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.122 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.122 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.122 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.123 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.123 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.123 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.123 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.123 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.123 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.123 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.123 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.123 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.123 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.124 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.124 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.124 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.124 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.124 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.124 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.124 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.125 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.125 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.125 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.125 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.125 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.125 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.125 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.125 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.125 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.125 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.126 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.126 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.126 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.126 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.126 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.126 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.126 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.126 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.127 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.127 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.127 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.127 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.127 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.127 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.127 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.127 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.128 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.128 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.128 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.128 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.128 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.128 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.128 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.128 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.129 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.129 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.129 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.129 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.129 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.129 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.129 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:44.130 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:44.130 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.130 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.130 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.130 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:44.130 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.130 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.130 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.130 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.131 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.131 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.131 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.131 INFO:teuthology.orchestra.run.smithi007.stdout: "threshold": { 2024-04-21T08:56:44.131 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threshold", 2024-04-21T08:56:44.131 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:44.131 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.132 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.132 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-04-21T08:56:44.132 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1.0", 2024-04-21T08:56:44.132 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.132 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.132 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "scaling threshold", 2024-04-21T08:56:44.132 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T08:56:44.132 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.133 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.133 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.133 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.133 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "progress", 2024-04-21T08:56:44.133 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.133 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.134 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.134 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_pg_recovery_event": { 2024-04-21T08:56:44.134 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_pg_recovery_event", 2024-04-21T08:56:44.134 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.134 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.134 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.134 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.134 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.134 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.135 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.135 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T08:56:44.135 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.135 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.135 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.135 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-04-21T08:56:44.135 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-04-21T08:56:44.136 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.136 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.136 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.136 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.136 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.136 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.136 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.136 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.137 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.137 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.137 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.137 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.137 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.137 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.137 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.138 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.138 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.138 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.138 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.138 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.138 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.138 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.138 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.139 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.139 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.139 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.139 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.139 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.139 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.139 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.139 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.140 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.140 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.140 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.140 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.140 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.140 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.140 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.140 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.141 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.141 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.141 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.141 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.141 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.141 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.141 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.142 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.142 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.142 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.142 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.142 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.142 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.142 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.142 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.143 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.144 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.145 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "max_completed_events": { 2024-04-21T08:56:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_completed_events", 2024-04-21T08:56:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.146 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "50", 2024-04-21T08:56:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "number of past completed events to remember", 2024-04-21T08:56:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.147 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.148 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.148 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:44.148 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:44.148 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.148 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.148 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.148 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:44.149 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.149 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.149 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.149 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long the module is going to sleep", 2024-04-21T08:56:44.149 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.149 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.149 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.149 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.149 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.150 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.150 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus", 2024-04-21T08:56:44.150 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.150 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.150 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.150 INFO:teuthology.orchestra.run.smithi007.stdout: "cache": { 2024-04-21T08:56:44.150 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cache", 2024-04-21T08:56:44.151 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.151 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.151 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.151 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.151 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.151 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.151 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.151 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.151 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.151 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.152 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.152 INFO:teuthology.orchestra.run.smithi007.stdout: "exclude_perf_counters": { 2024-04-21T08:56:44.152 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "exclude_perf_counters", 2024-04-21T08:56:44.152 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.152 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.152 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.152 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.152 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.153 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.153 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.153 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T08:56:44.153 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T08:56:44.153 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.153 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.153 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.153 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.154 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.154 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.154 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.154 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.154 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.154 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.154 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.154 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.155 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.155 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.155 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.155 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.155 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.155 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.155 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.155 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.156 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.156 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.156 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.156 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.156 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.156 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.156 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.157 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.157 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.157 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.157 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.157 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.157 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.157 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.157 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.158 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.158 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.158 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.158 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.158 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.158 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.158 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.158 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.159 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.159 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.159 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.159 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.159 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.159 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.159 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.159 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.159 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.160 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.160 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.160 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.160 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.160 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.160 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.160 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.161 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.161 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.161 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.161 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.161 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.161 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.161 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.161 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.161 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.162 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools": { 2024-04-21T08:56:44.162 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools", 2024-04-21T08:56:44.162 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.162 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.162 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.162 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.162 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.163 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.163 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.163 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.163 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.163 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.163 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.163 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T08:56:44.163 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T08:56:44.164 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.164 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.164 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.164 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "300", 2024-04-21T08:56:44.164 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.164 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.164 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.164 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.165 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.165 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.165 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.165 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_interval": { 2024-04-21T08:56:44.165 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_interval", 2024-04-21T08:56:44.165 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:44.165 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.166 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.166 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15.0", 2024-04-21T08:56:44.166 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.166 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.166 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.166 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-04-21T08:56:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-04-21T08:56:44.167 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-04-21T08:56:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T08:56:44.168 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-04-21T08:56:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-04-21T08:56:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.169 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "9283", 2024-04-21T08:56:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T08:56:44.170 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "stale_cache_strategy": { 2024-04-21T08:56:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stale_cache_strategy", 2024-04-21T08:56:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.171 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "log", 2024-04-21T08:56:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.172 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-04-21T08:56:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-04-21T08:56:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.173 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "default", 2024-04-21T08:56:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "default", 2024-04-21T08:56:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "error" 2024-04-21T08:56:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.174 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-04-21T08:56:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.175 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-04-21T08:56:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-04-21T08:56:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-04-21T08:56:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.176 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_support", 2024-04-21T08:56:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.177 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.178 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.179 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.180 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.180 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.180 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.180 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.180 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.180 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.180 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.181 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.181 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.181 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.181 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.181 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.181 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.181 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.181 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.182 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.182 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.182 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.182 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.182 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.182 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.182 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.183 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.183 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.183 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.183 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.183 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.183 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.183 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.183 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.183 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.184 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.184 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.184 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.184 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.184 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.184 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.184 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.184 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.185 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.185 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.185 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.185 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.185 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.185 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.185 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.185 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.185 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.186 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.186 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.186 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_snap_create": { 2024-04-21T08:56:44.186 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_snap_create", 2024-04-21T08:56:44.186 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.186 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.186 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.187 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:44.187 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.187 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.187 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.187 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.187 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.187 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.187 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.188 INFO:teuthology.orchestra.run.smithi007.stdout: "mirror_snapshot_schedule": { 2024-04-21T08:56:44.188 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T08:56:44.188 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.188 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.188 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.188 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.188 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.189 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.189 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.189 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.189 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.189 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.189 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.189 INFO:teuthology.orchestra.run.smithi007.stdout: "trash_purge_schedule": { 2024-04-21T08:56:44.190 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "trash_purge_schedule", 2024-04-21T08:56:44.190 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.190 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.190 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.190 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.190 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.191 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.191 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.191 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.191 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.191 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.191 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.191 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.192 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.192 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "restful", 2024-04-21T08:56:44.192 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.192 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.192 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.192 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_auth": { 2024-04-21T08:56:44.192 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_auth", 2024-04-21T08:56:44.192 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.193 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.193 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.193 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.193 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.193 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.193 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.193 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.193 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.193 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.194 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.194 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-04-21T08:56:44.194 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-04-21T08:56:44.194 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.194 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.194 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.194 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.195 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.195 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.195 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.195 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.195 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.195 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.195 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.195 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.196 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.196 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.196 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.196 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.196 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.196 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.196 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.196 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.196 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.197 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.197 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.197 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.197 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.197 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.197 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.197 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.197 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.197 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.198 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.198 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.198 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.198 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.198 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.198 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.198 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.198 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.199 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.199 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.199 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.199 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.199 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.199 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.199 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.200 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.200 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.200 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.200 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.200 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.200 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.200 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.200 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.200 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.201 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.201 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.201 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.201 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.201 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.201 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.201 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.201 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.201 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.202 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.202 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.202 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.202 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.202 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.202 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.202 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.203 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.203 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.203 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.203 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.203 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.203 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.203 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-04-21T08:56:44.203 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-04-21T08:56:44.204 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.204 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.204 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.204 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.204 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.204 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.204 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.204 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.204 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.205 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.205 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.205 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-04-21T08:56:44.205 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-04-21T08:56:44.205 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.205 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.205 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.206 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.206 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.206 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.206 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.206 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.206 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.206 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.206 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.207 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.207 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.207 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.207 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rgw", 2024-04-21T08:56:44.207 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.207 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.207 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.208 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.208 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.208 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.208 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.208 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.208 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.208 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.208 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.209 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.209 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.209 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.209 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.209 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.209 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.209 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.209 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.210 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.210 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.210 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.210 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.210 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.210 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rook", 2024-04-21T08:56:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.217 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.217 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.217 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.217 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.217 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.217 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.217 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.217 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.217 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.218 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.218 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.218 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.218 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.218 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.218 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.218 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.218 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.219 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.219 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.219 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.219 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.219 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.219 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.219 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.219 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.219 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.220 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.220 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.220 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.220 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.220 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.220 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.220 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.220 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.220 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.221 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.221 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.221 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.221 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.221 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.221 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.221 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.221 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.222 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.222 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.222 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.222 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.222 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.222 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.222 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.222 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.223 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.223 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.223 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.223 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.223 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.223 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.223 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.224 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.224 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.224 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.224 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.224 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.224 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.224 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.224 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.224 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.225 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.225 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_class": { 2024-04-21T08:56:44.225 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "storage_class", 2024-04-21T08:56:44.225 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.225 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.225 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.225 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "local", 2024-04-21T08:56:44.226 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.226 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.226 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.226 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T08:56:44.226 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.226 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.226 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.226 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.227 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.227 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.227 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "selftest", 2024-04-21T08:56:44.227 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.227 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.227 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.227 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.227 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.227 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.228 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.228 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.228 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.228 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.228 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.228 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.228 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.228 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.228 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.229 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.229 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.229 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.229 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.229 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.229 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.229 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.229 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.230 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.230 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.230 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.230 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.230 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.230 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.230 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.231 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.231 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.231 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.231 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.231 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.231 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.231 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.232 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.232 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.232 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.232 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.232 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.232 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.232 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.232 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.233 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.233 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.233 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.233 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.233 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.233 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.233 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.233 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.234 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.234 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.234 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.234 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.234 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.234 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.235 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.235 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.235 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.235 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.235 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.235 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.235 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.235 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.236 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.236 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.236 INFO:teuthology.orchestra.run.smithi007.stdout: "roption1": { 2024-04-21T08:56:44.236 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption1", 2024-04-21T08:56:44.236 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.236 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.236 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.237 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.237 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.237 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.237 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.237 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.237 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.237 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.237 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.238 INFO:teuthology.orchestra.run.smithi007.stdout: "roption2": { 2024-04-21T08:56:44.238 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption2", 2024-04-21T08:56:44.238 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.238 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.238 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.238 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "xyz", 2024-04-21T08:56:44.238 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.238 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.239 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.239 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.239 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.239 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.239 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.239 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption1": { 2024-04-21T08:56:44.239 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption1", 2024-04-21T08:56:44.240 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.240 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.240 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.240 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.240 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.240 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.240 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.240 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.240 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.241 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.241 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.241 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption2": { 2024-04-21T08:56:44.241 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption2", 2024-04-21T08:56:44.241 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.241 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.241 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.241 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.242 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.242 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.242 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.242 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.242 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.242 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.242 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.242 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption3": { 2024-04-21T08:56:44.243 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption3", 2024-04-21T08:56:44.243 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:44.243 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.243 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.243 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.243 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.243 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.243 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.243 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.244 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.244 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.244 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.244 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption4": { 2024-04-21T08:56:44.244 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption4", 2024-04-21T08:56:44.244 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.244 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.244 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.245 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.245 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.245 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.245 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.245 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.245 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.245 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.245 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.246 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption5": { 2024-04-21T08:56:44.246 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption5", 2024-04-21T08:56:44.246 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.246 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.246 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.246 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.246 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.247 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.247 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.247 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.247 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.247 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.247 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.247 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.247 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption6": { 2024-04-21T08:56:44.248 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption6", 2024-04-21T08:56:44.248 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.248 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.248 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.248 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.248 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.248 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.248 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.248 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.249 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.249 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.249 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.249 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption7": { 2024-04-21T08:56:44.249 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption7", 2024-04-21T08:56:44.249 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.249 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.250 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.250 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.250 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:44.250 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "42", 2024-04-21T08:56:44.250 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.250 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.250 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.250 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.250 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.251 INFO:teuthology.orchestra.run.smithi007.stdout: "testkey": { 2024-04-21T08:56:44.251 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testkey", 2024-04-21T08:56:44.251 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.251 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.251 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.251 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.251 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.251 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.252 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.252 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.252 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.252 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.252 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.252 INFO:teuthology.orchestra.run.smithi007.stdout: "testlkey": { 2024-04-21T08:56:44.252 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testlkey", 2024-04-21T08:56:44.252 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.253 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.253 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.253 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.253 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.253 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.253 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.253 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.253 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.253 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.254 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.254 INFO:teuthology.orchestra.run.smithi007.stdout: "testnewline": { 2024-04-21T08:56:44.254 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testnewline", 2024-04-21T08:56:44.254 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.254 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.254 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.254 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.255 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.255 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.255 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.255 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.255 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.255 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.255 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.255 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.255 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.256 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.256 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snap_schedule", 2024-04-21T08:56:44.256 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.256 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.256 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.256 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_m_granularity": { 2024-04-21T08:56:44.256 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_m_granularity", 2024-04-21T08:56:44.257 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.257 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.257 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.257 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.257 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.257 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.257 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.257 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T08:56:44.257 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.258 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.258 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.258 INFO:teuthology.orchestra.run.smithi007.stdout: "dump_on_update": { 2024-04-21T08:56:44.258 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dump_on_update", 2024-04-21T08:56:44.258 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.258 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.258 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.259 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.259 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.259 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.259 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.259 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "dump database to debug log on update", 2024-04-21T08:56:44.259 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.259 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.259 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.260 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.260 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.260 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.260 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.260 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.260 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.260 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.261 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.261 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.261 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.261 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.261 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.261 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.261 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.261 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.261 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.262 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.262 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.262 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.262 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.262 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.262 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.262 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.262 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.263 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.263 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.263 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.263 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.263 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.263 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.263 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.263 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.263 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.264 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.264 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.264 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.264 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.264 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.264 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.264 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.264 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.265 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.265 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.265 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.265 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.265 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.265 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.265 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.265 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.265 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.266 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.266 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.266 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.266 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.266 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.266 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.266 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.267 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.267 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.267 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.267 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.267 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.267 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.267 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.267 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.267 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.268 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.268 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.268 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stats", 2024-04-21T08:56:44.268 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.268 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.268 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.268 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.269 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.269 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.269 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.269 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.269 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.269 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.269 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.269 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.269 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.269 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.270 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.270 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.270 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.270 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.270 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.270 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.270 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.270 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.271 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.271 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.271 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.271 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.271 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.271 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.271 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.275 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.275 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.275 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.275 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.275 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.275 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.275 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.275 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.277 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.277 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.277 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.277 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.277 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.278 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.278 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.278 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.278 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.278 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.278 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "status", 2024-04-21T08:56:44.278 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.279 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.279 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.279 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.279 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.279 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.279 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.279 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.279 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.280 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.282 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.283 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.284 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.285 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.286 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.286 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.286 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.286 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.286 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.286 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.286 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.286 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.287 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.287 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.287 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.287 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.287 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telegraf", 2024-04-21T08:56:44.287 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.287 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.287 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.288 INFO:teuthology.orchestra.run.smithi007.stdout: "address": { 2024-04-21T08:56:44.288 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "address", 2024-04-21T08:56:44.288 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.288 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.288 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.288 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T08:56:44.288 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.288 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.289 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.289 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.289 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.289 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.289 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.289 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:44.290 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:44.290 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.290 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.290 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.290 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15", 2024-04-21T08:56:44.290 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.290 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.290 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.291 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.291 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.291 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.291 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.291 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.291 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.291 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.291 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.292 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.293 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.294 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.294 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.294 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.294 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.294 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.294 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.294 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.294 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.295 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.295 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.295 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.295 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.295 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.295 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.295 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.295 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.295 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.296 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.296 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.296 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.296 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.296 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.296 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.296 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.296 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.297 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.297 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.297 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.297 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.297 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.297 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.297 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.297 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.298 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.298 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.298 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.298 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.298 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.298 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.298 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.298 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.298 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.299 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.299 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.299 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.299 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.299 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.299 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.299 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.300 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telemetry", 2024-04-21T08:56:44.300 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.300 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.300 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.300 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_basic": { 2024-04-21T08:56:44.300 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_basic", 2024-04-21T08:56:44.300 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.300 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.300 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.301 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.301 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.301 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.301 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.301 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T08:56:44.301 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.301 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.301 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.302 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_crash": { 2024-04-21T08:56:44.302 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_crash", 2024-04-21T08:56:44.302 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.302 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.302 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.302 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.302 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.302 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.302 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.303 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T08:56:44.303 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.303 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.303 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.303 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_device": { 2024-04-21T08:56:44.303 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_device", 2024-04-21T08:56:44.303 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.304 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.304 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.304 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.304 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.304 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.304 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.304 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T08:56:44.304 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.305 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.305 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.305 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_ident": { 2024-04-21T08:56:44.305 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_ident", 2024-04-21T08:56:44.305 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.305 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.305 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.305 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.306 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.306 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.306 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.306 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T08:56:44.306 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.306 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.306 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.307 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.307 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_perf": { 2024-04-21T08:56:44.307 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_perf", 2024-04-21T08:56:44.307 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.307 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.307 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.307 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.307 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.308 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.308 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.308 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T08:56:44.308 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.308 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.308 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.308 INFO:teuthology.orchestra.run.smithi007.stdout: "contact": { 2024-04-21T08:56:44.309 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "contact", 2024-04-21T08:56:44.309 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.309 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.309 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.309 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.309 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.309 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.309 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.309 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.310 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.310 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.310 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.310 INFO:teuthology.orchestra.run.smithi007.stdout: "description": { 2024-04-21T08:56:44.310 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "description", 2024-04-21T08:56:44.310 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.310 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.311 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.311 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.311 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.311 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.311 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.311 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.311 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.312 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.312 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.312 INFO:teuthology.orchestra.run.smithi007.stdout: "device_url": { 2024-04-21T08:56:44.312 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_url", 2024-04-21T08:56:44.312 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.312 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.312 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.312 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T08:56:44.313 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.313 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.313 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.313 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.313 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.313 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.313 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.314 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-04-21T08:56:44.314 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-04-21T08:56:44.314 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.314 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.314 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.314 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.314 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.314 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.314 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.315 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.315 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.315 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.315 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.315 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.315 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:44.315 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:44.315 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "24", 2024-04-21T08:56:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "8", 2024-04-21T08:56:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.316 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "last_opt_revision": { 2024-04-21T08:56:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "last_opt_revision", 2024-04-21T08:56:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.317 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1", 2024-04-21T08:56:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.318 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard": { 2024-04-21T08:56:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard", 2024-04-21T08:56:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.319 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard_description": { 2024-04-21T08:56:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard_description", 2024-04-21T08:56:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.320 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.321 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.322 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.322 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.322 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.322 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.322 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.322 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.322 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.323 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.323 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.323 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.323 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.323 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.323 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.323 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.323 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.324 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.324 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.324 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.324 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.324 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.324 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.324 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.325 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.325 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.325 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.325 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.325 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.325 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.325 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.325 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.325 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.326 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.327 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.329 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.329 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.329 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.329 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.329 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.329 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.329 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.329 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.329 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.330 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.330 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.330 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.330 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.330 INFO:teuthology.orchestra.run.smithi007.stdout: "organization": { 2024-04-21T08:56:44.330 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "organization", 2024-04-21T08:56:44.330 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.331 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.331 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.331 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.331 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.331 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.331 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.331 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.331 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.332 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.332 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.332 INFO:teuthology.orchestra.run.smithi007.stdout: "proxy": { 2024-04-21T08:56:44.332 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "proxy", 2024-04-21T08:56:44.332 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.332 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.332 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.332 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.333 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.333 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.333 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.333 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.333 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.333 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.333 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.334 INFO:teuthology.orchestra.run.smithi007.stdout: "url": { 2024-04-21T08:56:44.334 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url", 2024-04-21T08:56:44.334 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.334 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.334 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.334 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T08:56:44.334 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.334 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.335 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.335 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.335 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.335 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.335 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.336 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.336 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test_orchestrator", 2024-04-21T08:56:44.336 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.336 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.336 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.336 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.336 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.336 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.336 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.337 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.337 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.337 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.337 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.337 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.337 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.337 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.337 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.338 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.338 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.338 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.338 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.338 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.338 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.338 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.338 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.340 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.340 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.340 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.340 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.340 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.340 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.341 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.341 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.341 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.341 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.341 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.341 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.341 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.341 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.342 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.342 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.342 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.342 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.342 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.342 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.342 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.343 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.343 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.343 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.343 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.343 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.343 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.343 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.343 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.344 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.344 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.344 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.344 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.344 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.344 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.344 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.345 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.345 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.345 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.345 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.345 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.345 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.346 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.346 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "volumes", 2024-04-21T08:56:44.346 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.346 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.346 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.346 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.346 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.346 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.347 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.347 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.347 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.347 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.347 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.347 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.347 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.347 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.348 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.348 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.348 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.348 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.348 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.348 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.348 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.349 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.349 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.349 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.349 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.349 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.349 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.349 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.350 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.350 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.350 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.350 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.350 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.350 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.350 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.350 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.351 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.351 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.351 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.351 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.351 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.351 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.351 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.351 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.352 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.352 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.352 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.352 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.352 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.352 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.352 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.352 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.353 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.353 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.353 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.353 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.353 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.353 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.353 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.353 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.354 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.354 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.354 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.354 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.354 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.354 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.354 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.354 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.355 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.355 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.355 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.355 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.355 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_clones": { 2024-04-21T08:56:44.355 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_clones", 2024-04-21T08:56:44.355 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.355 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.356 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.356 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4", 2024-04-21T08:56:44.356 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.356 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.356 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.356 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T08:56:44.356 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.356 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.357 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.357 INFO:teuthology.orchestra.run.smithi007.stdout: "periodic_async_work": { 2024-04-21T08:56:44.357 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "periodic_async_work", 2024-04-21T08:56:44.357 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.357 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.357 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.358 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.358 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.358 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.358 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.358 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Periodically check for async work", 2024-04-21T08:56:44.358 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.358 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.358 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.359 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_delay": { 2024-04-21T08:56:44.359 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_delay", 2024-04-21T08:56:44.359 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.359 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.359 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.359 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:44.360 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.360 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.360 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.360 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T08:56:44.360 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.360 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.360 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.361 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_no_wait": { 2024-04-21T08:56:44.361 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T08:56:44.361 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.361 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.361 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.361 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.361 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.361 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.362 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.362 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T08:56:44.362 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.362 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.362 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.362 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.362 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.363 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.363 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix", 2024-04-21T08:56:44.363 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.363 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.363 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.363 INFO:teuthology.orchestra.run.smithi007.stdout: "discovery_interval": { 2024-04-21T08:56:44.363 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "discovery_interval", 2024-04-21T08:56:44.363 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:44.364 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.364 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.364 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "100", 2024-04-21T08:56:44.364 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.364 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.364 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.364 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.364 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.365 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.365 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.365 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.365 INFO:teuthology.orchestra.run.smithi007.stdout: "identifier": { 2024-04-21T08:56:44.365 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "identifier", 2024-04-21T08:56:44.365 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.365 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.365 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.366 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.366 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.366 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.366 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.366 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.366 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.366 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.367 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.367 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:44.367 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:44.367 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.367 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.367 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.367 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:44.368 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.368 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.368 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.368 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.368 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.368 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.368 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.368 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.369 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.369 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.369 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.369 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.369 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.369 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.369 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.369 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.370 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.370 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.370 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.370 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.370 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.370 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.371 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.371 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.371 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.371 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.371 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.371 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.372 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.372 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.372 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.372 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.372 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.372 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.372 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.372 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.373 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.373 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.373 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.373 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.373 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.373 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.374 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.374 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.374 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.374 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.374 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.374 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.374 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.374 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.375 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.375 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.375 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.375 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.375 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.375 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.375 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.375 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.375 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.376 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.376 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.376 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.376 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.376 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.376 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.376 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.376 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.377 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.377 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.377 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.377 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.377 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.377 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.377 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.378 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_host": { 2024-04-21T08:56:44.378 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_host", 2024-04-21T08:56:44.378 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.378 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.378 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.378 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.378 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.378 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.379 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.379 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.379 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.379 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.379 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.379 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_port": { 2024-04-21T08:56:44.379 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_port", 2024-04-21T08:56:44.379 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.380 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.380 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.380 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10051", 2024-04-21T08:56:44.380 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.380 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.380 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.380 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.381 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.381 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.381 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.381 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_sender": { 2024-04-21T08:56:44.381 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_sender", 2024-04-21T08:56:44.381 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.381 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.382 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.382 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T08:56:44.382 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.382 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.382 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.382 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.382 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.382 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.383 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.383 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.383 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.383 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.383 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:56:44.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.383 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.383 INFO:teuthology.orchestra.run.smithi007.stdout: "gid": 25084, 2024-04-21T08:56:44.384 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "z", 2024-04-21T08:56:44.384 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_features": 4540701547738038271, 2024-04-21T08:56:44.384 INFO:teuthology.orchestra.run.smithi007.stdout: "available_modules": [ 2024-04-21T08:56:44.384 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.384 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "alerts", 2024-04-21T08:56:44.384 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.384 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.385 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.385 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:44.385 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:44.385 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.385 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.385 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.385 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:44.386 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.386 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.386 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.386 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T08:56:44.386 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.386 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.386 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.387 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.387 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.387 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.387 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.387 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.387 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.388 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.388 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.388 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.388 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.388 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.388 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.388 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.388 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.389 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.389 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.389 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.389 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.389 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.389 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.390 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.390 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.390 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.390 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.390 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.390 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.390 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.390 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.391 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.391 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.391 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.391 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.391 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.391 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.392 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.392 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.392 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.392 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.392 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.392 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.392 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_destination": { 2024-04-21T08:56:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_destination", 2024-04-21T08:56:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email address to send alerts to", 2024-04-21T08:56:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.398 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.398 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_from_name": { 2024-04-21T08:56:44.398 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_from_name", 2024-04-21T08:56:44.398 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.398 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.398 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.398 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "Ceph", 2024-04-21T08:56:44.399 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.399 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.399 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.399 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email From: name", 2024-04-21T08:56:44.399 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.399 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.399 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.399 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_host": { 2024-04-21T08:56:44.400 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_host", 2024-04-21T08:56:44.400 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.400 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.400 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.400 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.400 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.400 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.400 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.400 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP server", 2024-04-21T08:56:44.401 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.401 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.401 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.401 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_password": { 2024-04-21T08:56:44.401 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_password", 2024-04-21T08:56:44.401 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.401 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.401 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.402 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.402 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.402 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.402 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.402 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Password to authenticate with", 2024-04-21T08:56:44.402 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.402 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.402 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.403 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_port": { 2024-04-21T08:56:44.403 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_port", 2024-04-21T08:56:44.403 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.403 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.403 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.403 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "465", 2024-04-21T08:56:44.404 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.404 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.404 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.404 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP port", 2024-04-21T08:56:44.404 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.404 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.404 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.405 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_sender": { 2024-04-21T08:56:44.405 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_sender", 2024-04-21T08:56:44.405 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.405 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.405 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.405 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.405 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.405 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.406 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.406 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP envelope sender", 2024-04-21T08:56:44.406 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.406 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.406 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.406 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_ssl": { 2024-04-21T08:56:44.406 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_ssl", 2024-04-21T08:56:44.406 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.407 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.407 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.407 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.407 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.407 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.407 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.407 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T08:56:44.407 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.408 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.408 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.408 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_user": { 2024-04-21T08:56:44.408 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_user", 2024-04-21T08:56:44.408 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.408 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.408 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.409 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.409 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.409 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.409 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.409 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "User to authenticate as", 2024-04-21T08:56:44.409 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.409 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.410 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.410 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.410 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.410 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.410 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "balancer", 2024-04-21T08:56:44.410 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.411 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.411 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.411 INFO:teuthology.orchestra.run.smithi007.stdout: "active": { 2024-04-21T08:56:44.411 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "active", 2024-04-21T08:56:44.411 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.411 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.411 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.411 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T08:56:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.413 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_time": { 2024-04-21T08:56:44.413 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_time", 2024-04-21T08:56:44.413 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.413 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.413 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.413 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0000", 2024-04-21T08:56:44.413 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.413 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.414 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.414 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T08:56:44.414 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:44.414 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.414 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.414 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_weekday": { 2024-04-21T08:56:44.414 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_weekday", 2024-04-21T08:56:44.415 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:44.415 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.415 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.415 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:44.415 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-04-21T08:56:44.415 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-04-21T08:56:44.415 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.416 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T08:56:44.416 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:44.416 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.416 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.416 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_max_iterations": { 2024-04-21T08:56:44.416 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_max_iterations", 2024-04-21T08:56:44.417 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:44.417 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.417 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.417 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "25", 2024-04-21T08:56:44.417 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:44.417 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "250", 2024-04-21T08:56:44.417 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.417 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T08:56:44.417 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.418 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.418 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.418 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_metrics": { 2024-04-21T08:56:44.418 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_metrics", 2024-04-21T08:56:44.418 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.418 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.418 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.418 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T08:56:44.419 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.419 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.419 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.419 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T08:56:44.419 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T08:56:44.419 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.419 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.419 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.419 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_step": { 2024-04-21T08:56:44.420 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_step", 2024-04-21T08:56:44.420 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:44.420 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.420 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.420 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.5", 2024-04-21T08:56:44.420 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0.001", 2024-04-21T08:56:44.420 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "0.999", 2024-04-21T08:56:44.420 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.420 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "aggressiveness of optimization", 2024-04-21T08:56:44.421 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T08:56:44.421 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.421 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.421 INFO:teuthology.orchestra.run.smithi007.stdout: "end_time": { 2024-04-21T08:56:44.421 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_time", 2024-04-21T08:56:44.421 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.421 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.421 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.422 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2359", 2024-04-21T08:56:44.422 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.422 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.422 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.422 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T08:56:44.422 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:44.422 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.422 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.423 INFO:teuthology.orchestra.run.smithi007.stdout: "end_weekday": { 2024-04-21T08:56:44.423 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_weekday", 2024-04-21T08:56:44.423 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:44.423 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.423 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.423 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:44.423 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-04-21T08:56:44.423 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-04-21T08:56:44.423 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.424 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T08:56:44.424 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:44.424 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.424 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.424 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.424 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.424 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.424 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.425 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.425 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.425 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.425 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.425 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.425 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.425 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.425 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.426 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.426 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.426 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.426 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.426 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.426 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.426 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.426 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.427 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.427 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.427 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.427 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.427 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.427 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.427 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.427 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.428 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.429 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.430 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.431 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.432 INFO:teuthology.orchestra.run.smithi007.stdout: "min_score": { 2024-04-21T08:56:44.433 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_score", 2024-04-21T08:56:44.433 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:44.433 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.433 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.433 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:44.433 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.433 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.433 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.434 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T08:56:44.434 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.434 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.434 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.434 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-04-21T08:56:44.434 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-04-21T08:56:44.434 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.434 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.435 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.435 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "upmap", 2024-04-21T08:56:44.435 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.435 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.435 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.435 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-compat", 2024-04-21T08:56:44.435 INFO:teuthology.orchestra.run.smithi007.stdout: "none", 2024-04-21T08:56:44.435 INFO:teuthology.orchestra.run.smithi007.stdout: "read", 2024-04-21T08:56:44.435 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap", 2024-04-21T08:56:44.436 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap-read" 2024-04-21T08:56:44.436 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.436 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Balancer mode", 2024-04-21T08:56:44.436 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.436 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.436 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.436 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_ids": { 2024-04-21T08:56:44.436 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_ids", 2024-04-21T08:56:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T08:56:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.439 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:44.439 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.439 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.439 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.439 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T08:56:44.439 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.439 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.439 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.440 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_deviation": { 2024-04-21T08:56:44.440 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_deviation", 2024-04-21T08:56:44.440 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.440 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.440 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.440 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:44.440 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:44.440 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.440 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.441 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T08:56:44.441 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T08:56:44.441 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.441 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.441 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_optimizations": { 2024-04-21T08:56:44.441 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_optimizations", 2024-04-21T08:56:44.441 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:44.441 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.442 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.442 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:44.442 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.442 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.442 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.442 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T08:56:44.442 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.442 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.443 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.443 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.443 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.443 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.443 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm", 2024-04-21T08:56:44.443 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.443 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.444 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.444 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_down_multiplier": { 2024-04-21T08:56:44.444 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_down_multiplier", 2024-04-21T08:56:44.444 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:44.444 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.444 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.444 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-04-21T08:56:44.444 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.445 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.445 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.445 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T08:56:44.445 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.445 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.445 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.445 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_refresh_rate": { 2024-04-21T08:56:44.446 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_refresh_rate", 2024-04-21T08:56:44.446 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.446 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.446 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.446 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "20", 2024-04-21T08:56:44.446 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.446 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.446 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.447 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T08:56:44.447 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.447 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.447 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.447 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_starting_port": { 2024-04-21T08:56:44.447 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_starting_port", 2024-04-21T08:56:44.447 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.447 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.448 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.448 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4721", 2024-04-21T08:56:44.448 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.448 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.448 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.448 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T08:56:44.448 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.448 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.448 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.449 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_ptrace": { 2024-04-21T08:56:44.449 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_ptrace", 2024-04-21T08:56:44.449 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.449 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.449 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.450 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.450 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.450 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.450 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.450 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T08:56:44.450 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T08:56:44.450 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.450 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.451 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_interval": { 2024-04-21T08:56:44.451 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_interval", 2024-04-21T08:56:44.451 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.451 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.451 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.451 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:44.451 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.452 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.452 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.452 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T08:56:44.452 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.452 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.452 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.452 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_memory_target_ratio": { 2024-04-21T08:56:44.452 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T08:56:44.453 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:44.453 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.453 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.453 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.7", 2024-04-21T08:56:44.453 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.453 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.453 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.453 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T08:56:44.453 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.454 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.454 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.454 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm_log_destination": { 2024-04-21T08:56:44.454 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm_log_destination", 2024-04-21T08:56:44.454 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.454 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.454 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.454 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.455 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.455 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.455 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.455 INFO:teuthology.orchestra.run.smithi007.stdout: "file", 2024-04-21T08:56:44.455 INFO:teuthology.orchestra.run.smithi007.stdout: "file,syslog", 2024-04-21T08:56:44.455 INFO:teuthology.orchestra.run.smithi007.stdout: "syslog" 2024-04-21T08:56:44.455 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.455 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T08:56:44.455 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.456 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.456 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.456 INFO:teuthology.orchestra.run.smithi007.stdout: "cgroups_split": { 2024-04-21T08:56:44.456 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cgroups_split", 2024-04-21T08:56:44.456 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.456 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.456 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.456 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.457 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.457 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.457 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.457 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T08:56:44.457 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.457 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.457 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.458 INFO:teuthology.orchestra.run.smithi007.stdout: "config_checks_enabled": { 2024-04-21T08:56:44.458 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_checks_enabled", 2024-04-21T08:56:44.458 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.458 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.458 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.458 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.458 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.458 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.458 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.459 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T08:56:44.459 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.459 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.459 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.459 INFO:teuthology.orchestra.run.smithi007.stdout: "config_dashboard": { 2024-04-21T08:56:44.459 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_dashboard", 2024-04-21T08:56:44.459 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.460 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.460 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.460 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.460 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.460 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.460 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.460 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T08:56:44.460 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.460 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.461 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.461 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_alertmanager": { 2024-04-21T08:56:44.461 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_alertmanager", 2024-04-21T08:56:44.461 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.461 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.461 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.462 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T08:56:44.462 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.462 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.462 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.462 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:44.462 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.462 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.462 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.463 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_base": { 2024-04-21T08:56:44.463 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_base", 2024-04-21T08:56:44.463 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.463 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.463 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.463 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T08:56:44.463 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.463 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.463 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.464 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Container image name, without the tag", 2024-04-21T08:56:44.464 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.464 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.464 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.464 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.464 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_elasticsearch": { 2024-04-21T08:56:44.464 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_elasticsearch", 2024-04-21T08:56:44.464 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.465 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.465 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.465 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T08:56:44.465 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.465 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.465 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.465 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "elasticsearch container image", 2024-04-21T08:56:44.465 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.465 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.466 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.466 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_grafana": { 2024-04-21T08:56:44.466 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_grafana", 2024-04-21T08:56:44.466 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.466 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.466 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.466 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T08:56:44.466 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.467 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.467 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.467 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:44.467 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.467 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.467 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.467 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_haproxy": { 2024-04-21T08:56:44.467 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_haproxy", 2024-04-21T08:56:44.468 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.468 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.468 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.468 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T08:56:44.468 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.468 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.468 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.468 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "HAproxy container image", 2024-04-21T08:56:44.468 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.469 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.469 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.469 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_agent": { 2024-04-21T08:56:44.469 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_agent", 2024-04-21T08:56:44.469 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.469 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.469 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.469 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T08:56:44.470 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.470 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.470 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.470 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger agent container image", 2024-04-21T08:56:44.470 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.470 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.470 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.470 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_collector": { 2024-04-21T08:56:44.471 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_collector", 2024-04-21T08:56:44.471 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.471 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.471 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.471 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T08:56:44.471 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.471 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.472 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.472 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger collector container image", 2024-04-21T08:56:44.472 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.472 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.472 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.472 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.472 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_query": { 2024-04-21T08:56:44.473 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_query", 2024-04-21T08:56:44.473 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.473 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.473 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.473 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T08:56:44.473 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.473 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.473 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.474 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger query container image", 2024-04-21T08:56:44.474 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.474 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.474 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.474 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_keepalived": { 2024-04-21T08:56:44.474 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_keepalived", 2024-04-21T08:56:44.475 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.475 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.475 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.475 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T08:56:44.475 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.475 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.475 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.475 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Keepalived container image", 2024-04-21T08:56:44.476 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.476 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.476 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.476 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_loki": { 2024-04-21T08:56:44.476 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_loki", 2024-04-21T08:56:44.476 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.476 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.477 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.477 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T08:56:44.477 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.477 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.477 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.477 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Loki container image", 2024-04-21T08:56:44.477 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.477 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.477 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.478 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.478 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_node_exporter": { 2024-04-21T08:56:44.478 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_node_exporter", 2024-04-21T08:56:44.478 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.478 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.478 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.478 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T08:56:44.478 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.479 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.479 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.479 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:44.479 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.479 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.479 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.479 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.479 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_nvmeof": { 2024-04-21T08:56:44.479 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_nvmeof", 2024-04-21T08:56:44.480 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.480 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.480 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.480 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T08:56:44.480 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.480 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.480 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.481 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Nvme-of container image", 2024-04-21T08:56:44.481 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.481 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.481 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.481 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_prometheus": { 2024-04-21T08:56:44.481 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_prometheus", 2024-04-21T08:56:44.481 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.482 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.482 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.482 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T08:56:44.482 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.482 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.482 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.482 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:44.482 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.483 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.483 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.483 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_promtail": { 2024-04-21T08:56:44.483 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_promtail", 2024-04-21T08:56:44.483 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.483 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.483 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.484 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T08:56:44.484 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.484 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.484 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.484 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Promtail container image", 2024-04-21T08:56:44.484 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.484 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.484 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.484 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_samba": { 2024-04-21T08:56:44.485 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_samba", 2024-04-21T08:56:44.485 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.485 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.485 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.485 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T08:56:44.485 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.485 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.486 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.486 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Samba/SMB container image", 2024-04-21T08:56:44.486 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.486 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.486 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.486 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_snmp_gateway": { 2024-04-21T08:56:44.486 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_snmp_gateway", 2024-04-21T08:56:44.487 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.487 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.487 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.487 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T08:56:44.487 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.487 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.487 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.487 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SNMP Gateway container image", 2024-04-21T08:56:44.487 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.488 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.488 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.488 INFO:teuthology.orchestra.run.smithi007.stdout: "container_init": { 2024-04-21T08:56:44.488 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_init", 2024-04-21T08:56:44.488 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.488 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.488 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.488 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.489 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.489 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.489 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.489 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T08:56:44.489 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.489 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.489 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.489 INFO:teuthology.orchestra.run.smithi007.stdout: "daemon_cache_timeout": { 2024-04-21T08:56:44.490 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "daemon_cache_timeout", 2024-04-21T08:56:44.490 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.490 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.490 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.490 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:44.490 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.490 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.490 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.491 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T08:56:44.491 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.491 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.491 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.491 INFO:teuthology.orchestra.run.smithi007.stdout: "default_cephadm_command_timeout": { 2024-04-21T08:56:44.491 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T08:56:44.491 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.492 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.492 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.492 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "900", 2024-04-21T08:56:44.492 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.492 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.492 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.492 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T08:56:44.492 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.492 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.493 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.493 INFO:teuthology.orchestra.run.smithi007.stdout: "default_registry": { 2024-04-21T08:56:44.493 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_registry", 2024-04-21T08:56:44.493 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.493 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.493 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.493 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io", 2024-04-21T08:56:44.493 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.494 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.494 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.494 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T08:56:44.494 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.494 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.494 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.494 INFO:teuthology.orchestra.run.smithi007.stdout: "device_cache_timeout": { 2024-04-21T08:56:44.494 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_cache_timeout", 2024-04-21T08:56:44.495 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.495 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.495 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.495 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1800", 2024-04-21T08:56:44.495 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.495 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.495 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.495 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache device inventory", 2024-04-21T08:56:44.496 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.496 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.496 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.496 INFO:teuthology.orchestra.run.smithi007.stdout: "device_enhanced_scan": { 2024-04-21T08:56:44.496 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_enhanced_scan", 2024-04-21T08:56:44.496 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.496 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.497 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.497 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.497 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.497 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.497 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.497 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T08:56:44.497 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.497 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.497 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.498 INFO:teuthology.orchestra.run.smithi007.stdout: "facts_cache_timeout": { 2024-04-21T08:56:44.498 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "facts_cache_timeout", 2024-04-21T08:56:44.498 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.498 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.498 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.498 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:44.498 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.499 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.499 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.499 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache host facts data", 2024-04-21T08:56:44.499 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.499 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.499 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.499 INFO:teuthology.orchestra.run.smithi007.stdout: "grafana_dashboards_path": { 2024-04-21T08:56:44.499 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "grafana_dashboards_path", 2024-04-21T08:56:44.500 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.500 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.500 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.500 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T08:56:44.500 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.500 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.500 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.501 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T08:56:44.501 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.501 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.501 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.501 INFO:teuthology.orchestra.run.smithi007.stdout: "host_check_interval": { 2024-04-21T08:56:44.501 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "host_check_interval", 2024-04-21T08:56:44.501 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.501 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.502 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.502 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:44.502 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.502 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.502 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.502 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to perform a host check", 2024-04-21T08:56:44.502 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.502 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.502 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.503 INFO:teuthology.orchestra.run.smithi007.stdout: "hw_monitoring": { 2024-04-21T08:56:44.503 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hw_monitoring", 2024-04-21T08:56:44.503 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.503 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.503 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.503 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.503 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.504 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.504 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.504 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T08:56:44.504 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.504 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.504 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.504 INFO:teuthology.orchestra.run.smithi007.stdout: "inventory_list_all": { 2024-04-21T08:56:44.505 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "inventory_list_all", 2024-04-21T08:56:44.505 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.505 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.505 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.505 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.505 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.505 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.505 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.506 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T08:56:44.506 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.506 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.506 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.506 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.506 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.506 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.507 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.507 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.507 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.507 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.507 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.507 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.507 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.507 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.507 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.508 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.508 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.508 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.508 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.508 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.508 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.508 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.508 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.509 INFO:teuthology.orchestra.run.smithi007.stdout: "log_refresh_metadata": { 2024-04-21T08:56:44.509 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_refresh_metadata", 2024-04-21T08:56:44.509 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.509 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.509 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.509 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.509 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.509 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.509 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.510 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T08:56:44.510 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.510 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.510 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.510 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.510 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.511 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.511 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.511 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.511 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.511 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.511 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.511 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.511 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T08:56:44.512 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.512 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.512 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.512 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.512 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.512 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.512 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.512 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.513 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.513 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.513 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.513 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.513 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.513 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.513 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.514 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.514 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.514 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.514 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.514 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.514 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.514 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.515 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.515 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.515 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.515 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.515 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.515 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.515 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.515 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.516 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.516 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.516 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.516 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.516 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.516 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T08:56:44.516 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T08:56:44.517 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.517 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.517 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.517 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.517 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.517 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.517 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.517 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T08:56:44.517 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.518 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.518 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.518 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T08:56:44.518 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T08:56:44.518 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.518 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.518 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.518 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "*", 2024-04-21T08:56:44.519 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.519 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.519 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.519 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T08:56:44.519 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.519 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.519 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.520 INFO:teuthology.orchestra.run.smithi007.stdout: "max_count_per_host": { 2024-04-21T08:56:44.520 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_count_per_host", 2024-04-21T08:56:44.520 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.520 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.520 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.520 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:44.521 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.521 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.521 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.521 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of daemons per service per host", 2024-04-21T08:56:44.521 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.521 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.521 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.521 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd_draining_count": { 2024-04-21T08:56:44.522 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_osd_draining_count", 2024-04-21T08:56:44.522 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.522 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.522 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.522 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:44.522 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.522 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.523 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.523 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T08:56:44.523 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.523 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.523 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.523 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.523 INFO:teuthology.orchestra.run.smithi007.stdout: "migration_current": { 2024-04-21T08:56:44.523 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "migration_current", 2024-04-21T08:56:44.524 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.524 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.524 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.524 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.524 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.524 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.524 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.524 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "internal - do not modify", 2024-04-21T08:56:44.524 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.525 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.525 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.525 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-04-21T08:56:44.525 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-04-21T08:56:44.525 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.525 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.525 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.526 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "root", 2024-04-21T08:56:44.526 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.526 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.526 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.526 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm-package", 2024-04-21T08:56:44.526 INFO:teuthology.orchestra.run.smithi007.stdout: "root" 2024-04-21T08:56:44.527 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.527 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T08:56:44.527 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.527 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.527 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.527 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.527 INFO:teuthology.orchestra.run.smithi007.stdout: "oob_default_addr": { 2024-04-21T08:56:44.527 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "oob_default_addr", 2024-04-21T08:56:44.528 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.528 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.528 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.528 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "169.254.1.1", 2024-04-21T08:56:44.528 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.528 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.528 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.529 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T08:56:44.529 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.529 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.529 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.529 INFO:teuthology.orchestra.run.smithi007.stdout: "prometheus_alerts_path": { 2024-04-21T08:56:44.530 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus_alerts_path", 2024-04-21T08:56:44.530 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.530 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.530 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.530 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T08:56:44.530 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.530 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.531 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.531 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T08:56:44.531 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.531 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.531 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.531 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_insecure": { 2024-04-21T08:56:44.532 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_insecure", 2024-04-21T08:56:44.532 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.532 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.532 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.532 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.532 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.532 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.533 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.533 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T08:56:44.533 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.533 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.533 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.533 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_password": { 2024-04-21T08:56:44.533 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_password", 2024-04-21T08:56:44.534 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.534 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.534 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.534 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.534 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.534 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.534 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.535 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T08:56:44.535 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.535 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.535 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.535 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_url": { 2024-04-21T08:56:44.535 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_url", 2024-04-21T08:56:44.535 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.536 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.536 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.536 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.536 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.536 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.536 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.536 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T08:56:44.536 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.537 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.537 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.537 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.537 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_username": { 2024-04-21T08:56:44.537 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_username", 2024-04-21T08:56:44.537 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.537 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.537 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.538 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.538 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.538 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.538 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.538 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T08:56:44.538 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.538 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.538 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.539 INFO:teuthology.orchestra.run.smithi007.stdout: "secure_monitoring_stack": { 2024-04-21T08:56:44.539 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "secure_monitoring_stack", 2024-04-21T08:56:44.539 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.539 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.539 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.539 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.539 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.539 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.540 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.540 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T08:56:44.540 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.540 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.540 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.540 INFO:teuthology.orchestra.run.smithi007.stdout: "service_discovery_port": { 2024-04-21T08:56:44.540 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "service_discovery_port", 2024-04-21T08:56:44.540 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.541 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.541 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.541 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8765", 2024-04-21T08:56:44.541 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.541 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.541 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.541 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "cephadm service discovery port", 2024-04-21T08:56:44.541 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.542 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.542 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.542 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.542 INFO:teuthology.orchestra.run.smithi007.stdout: "ssh_config_file": { 2024-04-21T08:56:44.542 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssh_config_file", 2024-04-21T08:56:44.542 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.542 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.543 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.543 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.543 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.543 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.543 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.543 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T08:56:44.543 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.543 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.544 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.544 INFO:teuthology.orchestra.run.smithi007.stdout: "use_agent": { 2024-04-21T08:56:44.544 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_agent", 2024-04-21T08:56:44.544 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.544 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.544 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.544 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.544 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.545 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.545 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.545 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T08:56:44.545 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.545 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.545 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.545 INFO:teuthology.orchestra.run.smithi007.stdout: "use_repo_digest": { 2024-04-21T08:56:44.546 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_repo_digest", 2024-04-21T08:56:44.546 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.546 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.546 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.546 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.546 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.546 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.546 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.547 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T08:56:44.547 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.547 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.547 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.547 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_failed_host_check": { 2024-04-21T08:56:44.547 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_failed_host_check", 2024-04-21T08:56:44.547 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.548 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.548 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.548 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.548 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.548 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.548 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.548 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T08:56:44.548 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.548 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.549 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.549 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_daemons": { 2024-04-21T08:56:44.549 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_daemons", 2024-04-21T08:56:44.549 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.549 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.549 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.550 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.550 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.550 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.550 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.550 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T08:56:44.550 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.550 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.550 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.551 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.551 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_hosts": { 2024-04-21T08:56:44.551 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_hosts", 2024-04-21T08:56:44.551 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.551 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.551 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.551 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.552 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.552 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.552 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.552 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T08:56:44.552 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.552 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.552 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.553 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.553 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.553 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crash", 2024-04-21T08:56:44.553 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.553 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.554 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.554 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.554 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.554 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.554 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.554 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.554 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.555 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.555 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.555 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.555 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.555 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.555 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.555 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.555 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.556 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.556 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.556 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.556 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.556 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.556 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.557 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.557 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.557 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.557 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.557 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.557 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.558 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.558 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.558 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.558 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.558 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.558 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.558 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.559 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.559 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.559 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.559 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.559 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.559 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.559 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.559 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.560 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.560 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.560 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.560 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.560 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.560 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.560 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.560 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.560 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.561 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.561 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.561 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.561 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.561 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.561 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.561 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.562 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.562 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.562 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.562 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.562 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.562 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.562 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.562 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.563 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.563 INFO:teuthology.orchestra.run.smithi007.stdout: "retain_interval": { 2024-04-21T08:56:44.563 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retain_interval", 2024-04-21T08:56:44.563 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.563 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.563 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.563 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "31536000", 2024-04-21T08:56:44.564 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.564 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.564 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.564 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T08:56:44.564 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.564 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.564 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.564 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.565 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_recent_interval": { 2024-04-21T08:56:44.565 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_recent_interval", 2024-04-21T08:56:44.565 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.565 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.565 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.565 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1209600", 2024-04-21T08:56:44.565 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.565 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.565 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.565 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T08:56:44.566 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.566 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.566 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.566 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.566 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.566 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.566 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.566 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dashboard", 2024-04-21T08:56:44.567 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.567 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.567 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.567 INFO:teuthology.orchestra.run.smithi007.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T08:56:44.567 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T08:56:44.567 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.567 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.568 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.568 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:44.568 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.568 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.568 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.568 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.568 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.568 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.569 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.569 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.569 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T08:56:44.569 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T08:56:44.569 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.569 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.569 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.570 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.570 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.570 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.570 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.570 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.570 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.570 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.571 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.571 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T08:56:44.571 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T08:56:44.571 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.571 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.571 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.572 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.572 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.572 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.572 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.572 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.572 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.572 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.572 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.573 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_ENABLED": { 2024-04-21T08:56:44.573 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T08:56:44.573 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.573 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.573 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.574 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.574 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.574 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.574 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.574 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.574 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.574 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.574 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.575 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T08:56:44.575 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T08:56:44.575 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.575 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.575 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.575 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.576 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.576 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.576 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.576 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.576 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.576 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.576 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.577 INFO:teuthology.orchestra.run.smithi007.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T08:56:44.577 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T08:56:44.577 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.577 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.577 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.577 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.577 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.577 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.577 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.578 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.578 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.578 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.578 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.578 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T08:56:44.578 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T08:56:44.578 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.579 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.579 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.579 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.579 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.579 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.579 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.579 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.579 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.580 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.580 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.580 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T08:56:44.580 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T08:56:44.580 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.580 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.580 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.580 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.581 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.581 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.581 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.581 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.581 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.581 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.581 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.582 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T08:56:44.582 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T08:56:44.582 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.582 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.582 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.582 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.582 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.582 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.583 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.583 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.583 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.583 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.583 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.583 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.583 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T08:56:44.583 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T08:56:44.584 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.584 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.584 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.584 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.584 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.584 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.584 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.584 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.585 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.585 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.585 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.585 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T08:56:44.585 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T08:56:44.585 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.585 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.585 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.586 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.586 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.586 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.586 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.586 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.586 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.586 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.586 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T08:56:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T08:56:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.587 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.588 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.588 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.588 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.588 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.588 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.588 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.588 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T08:56:44.589 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T08:56:44.589 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.589 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.589 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.589 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.589 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.589 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.589 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.589 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.590 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.590 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.590 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.590 INFO:teuthology.orchestra.run.smithi007.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T08:56:44.590 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T08:56:44.590 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.591 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.591 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.591 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.591 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.591 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.591 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.591 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.591 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.592 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.592 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.592 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T08:56:44.592 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T08:56:44.592 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.592 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.592 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.592 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-04-21T08:56:44.593 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.593 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.593 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.593 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.593 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.593 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.593 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.594 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T08:56:44.594 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T08:56:44.594 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.594 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.594 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.594 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.594 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.594 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.595 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.595 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.595 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.595 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.595 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.595 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_URL": { 2024-04-21T08:56:44.595 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_URL", 2024-04-21T08:56:44.595 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.596 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.596 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.596 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.596 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.596 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.596 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.596 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.596 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.597 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.597 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.597 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T08:56:44.597 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T08:56:44.597 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.597 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.598 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.598 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-04-21T08:56:44.598 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.598 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.598 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.598 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.598 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.598 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.599 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.599 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T08:56:44.599 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T08:56:44.599 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.599 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.599 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.599 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.600 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.600 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.600 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.600 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.600 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.600 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.600 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.600 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T08:56:44.601 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T08:56:44.601 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.601 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.601 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.601 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.601 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.601 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.602 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.602 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.602 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.602 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.602 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.602 INFO:teuthology.orchestra.run.smithi007.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T08:56:44.602 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T08:56:44.603 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.603 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.603 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.603 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.603 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.603 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.603 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.603 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.604 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.604 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.604 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.604 INFO:teuthology.orchestra.run.smithi007.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T08:56:44.604 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T08:56:44.604 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.604 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.605 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.605 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.605 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.605 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.605 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.605 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.605 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.605 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.605 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.606 INFO:teuthology.orchestra.run.smithi007.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T08:56:44.606 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T08:56:44.606 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.606 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.606 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.606 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "[]", 2024-04-21T08:56:44.606 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.607 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.607 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.607 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.607 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.607 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.607 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.607 INFO:teuthology.orchestra.run.smithi007.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T08:56:44.608 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T08:56:44.608 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.608 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.608 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.608 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "{}", 2024-04-21T08:56:44.608 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.608 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.608 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.609 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.609 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.609 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.609 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.609 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T08:56:44.609 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T08:56:44.609 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.609 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.610 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.610 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.610 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.610 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.610 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.610 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.610 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.610 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.611 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.611 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T08:56:44.611 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T08:56:44.611 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.611 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.611 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.612 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.612 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.612 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.612 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.612 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.612 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.612 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.612 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.613 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T08:56:44.613 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T08:56:44.613 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.613 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.613 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.613 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.613 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.614 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.614 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.614 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.614 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.614 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.614 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.614 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T08:56:44.615 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T08:56:44.615 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.615 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.615 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.615 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.615 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.615 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.615 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.615 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.616 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.616 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.616 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.616 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T08:56:44.616 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T08:56:44.616 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.616 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.617 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.617 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.617 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.617 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.617 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.617 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.617 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.618 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.618 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.618 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T08:56:44.618 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T08:56:44.618 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.618 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.618 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.619 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.619 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.619 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.619 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.619 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.619 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.619 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.619 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.620 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T08:56:44.620 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T08:56:44.620 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.620 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.620 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.620 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.621 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.621 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.621 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.621 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.621 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.621 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.621 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.621 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T08:56:44.622 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T08:56:44.622 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.622 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.622 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.622 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.622 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.622 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.622 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.623 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.623 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.623 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.623 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.623 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T08:56:44.623 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T08:56:44.623 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.624 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.624 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.624 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.624 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.624 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.624 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.624 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.624 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.625 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.625 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.625 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T08:56:44.625 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T08:56:44.625 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.625 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.625 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.626 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.626 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.626 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.626 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.626 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.626 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.626 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.626 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.627 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T08:56:44.627 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T08:56:44.627 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.627 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.627 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.627 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T08:56:44.627 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.628 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.628 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.628 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.628 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.628 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.628 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.628 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T08:56:44.629 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T08:56:44.629 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.629 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.629 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.629 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:44.629 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.629 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.629 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.630 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.630 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.630 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.630 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.630 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T08:56:44.630 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T08:56:44.630 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.630 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.631 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.631 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8", 2024-04-21T08:56:44.631 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.631 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.631 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.631 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.631 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.631 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.632 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.632 INFO:teuthology.orchestra.run.smithi007.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T08:56:44.632 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T08:56:44.632 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.632 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.632 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.632 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "45", 2024-04-21T08:56:44.632 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.633 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.633 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.633 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.633 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.633 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.633 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.633 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T08:56:44.634 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T08:56:44.634 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.634 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.634 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.634 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.634 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.634 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.634 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.634 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.635 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.635 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.635 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.635 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T08:56:44.635 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T08:56:44.635 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.636 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.636 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.636 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-04-21T08:56:44.636 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.636 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.636 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.636 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.636 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.637 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.637 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.637 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T08:56:44.637 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T08:56:44.637 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.637 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.637 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.638 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.638 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.638 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.638 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.638 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.638 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.638 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.639 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.639 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T08:56:44.639 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T08:56:44.639 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.639 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.639 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.639 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.640 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.640 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.640 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.640 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.640 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.640 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.640 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.641 INFO:teuthology.orchestra.run.smithi007.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T08:56:44.641 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T08:56:44.641 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.641 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.641 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.641 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.641 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.641 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.641 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.642 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.642 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.642 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.642 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.642 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T08:56:44.642 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T08:56:44.643 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.643 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.643 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.643 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:44.643 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.643 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.643 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.643 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.644 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.644 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.644 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.644 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T08:56:44.644 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T08:56:44.644 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.644 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.645 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.645 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:44.645 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.645 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.645 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.645 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.645 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.645 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.646 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.646 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T08:56:44.646 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T08:56:44.646 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.646 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.646 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.646 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:44.647 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.647 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.647 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.647 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.647 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.647 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.647 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.647 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.648 INFO:teuthology.orchestra.run.smithi007.stdout: "cross_origin_url": { 2024-04-21T08:56:44.648 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cross_origin_url", 2024-04-21T08:56:44.648 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.648 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.648 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.648 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.648 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.648 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.649 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.649 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.649 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.649 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.649 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.649 INFO:teuthology.orchestra.run.smithi007.stdout: "crt_file": { 2024-04-21T08:56:44.649 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crt_file", 2024-04-21T08:56:44.649 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.650 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.650 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.650 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.650 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.650 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.650 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.650 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.650 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.651 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.651 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.651 INFO:teuthology.orchestra.run.smithi007.stdout: "debug": { 2024-04-21T08:56:44.651 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "debug", 2024-04-21T08:56:44.651 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.651 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.651 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.652 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.652 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.652 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.652 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.652 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable/disable debug options", 2024-04-21T08:56:44.652 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.652 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.652 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.653 INFO:teuthology.orchestra.run.smithi007.stdout: "jwt_token_ttl": { 2024-04-21T08:56:44.653 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "jwt_token_ttl", 2024-04-21T08:56:44.653 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.653 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.653 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.653 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "28800", 2024-04-21T08:56:44.653 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.653 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.654 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.654 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.654 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.654 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.654 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.654 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-04-21T08:56:44.654 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-04-21T08:56:44.655 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.655 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.655 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.655 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.655 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.655 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.655 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.655 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.655 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.656 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.656 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.656 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.656 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.656 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.656 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.656 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.657 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.657 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.657 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.657 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.657 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.657 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.657 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.657 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.658 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.658 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.658 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.658 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.658 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.658 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.658 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: "motd": { 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "motd", 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "The message of the day", 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_resolve_ip_addr": { 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8080", 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl_server_port": { 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl_server_port", 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8443", 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "redirect", 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect" 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.678 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-04-21T08:56:44.678 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-04-21T08:56:44.678 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-04-21T08:56:44.678 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.678 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.678 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.678 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.678 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.679 INFO:teuthology.orchestra.run.smithi007.stdout: "url_prefix": { 2024-04-21T08:56:44.679 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url_prefix", 2024-04-21T08:56:44.679 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.679 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.679 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.679 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.679 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.680 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.680 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.680 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.680 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.680 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.680 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.680 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.681 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "devicehealth", 2024-04-21T08:56:44.681 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.681 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.681 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.681 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_monitoring": { 2024-04-21T08:56:44.682 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_monitoring", 2024-04-21T08:56:44.682 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.682 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.682 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.682 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.682 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.682 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.682 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.683 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "monitor device health metrics", 2024-04-21T08:56:44.683 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.683 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.683 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.683 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.683 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.683 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.683 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.684 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.684 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.684 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.684 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.684 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.684 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.684 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.684 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.684 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.685 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.685 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.685 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.685 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.685 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.685 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.685 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.685 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.686 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.686 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.686 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.686 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.686 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.686 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.686 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.686 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.686 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.687 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.687 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.687 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.687 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.687 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.687 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.687 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.688 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.688 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.688 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.688 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.688 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.688 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.688 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.688 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.688 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.689 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.689 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.689 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.689 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.689 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.689 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.689 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.689 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.690 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.690 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.690 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.690 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.690 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.690 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.690 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.690 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.691 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.691 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.691 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.691 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.691 INFO:teuthology.orchestra.run.smithi007.stdout: "mark_out_threshold": { 2024-04-21T08:56:44.691 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mark_out_threshold", 2024-04-21T08:56:44.691 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.692 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.692 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.692 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2419200", 2024-04-21T08:56:44.692 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.692 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.692 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.692 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T08:56:44.692 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.692 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.693 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.693 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": { 2024-04-21T08:56:44.693 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_name", 2024-04-21T08:56:44.693 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.693 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.693 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.693 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "device_health_metrics", 2024-04-21T08:56:44.693 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.694 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.694 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.694 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T08:56:44.694 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.694 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.694 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.694 INFO:teuthology.orchestra.run.smithi007.stdout: "retention_period": { 2024-04-21T08:56:44.694 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retention_period", 2024-04-21T08:56:44.695 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.695 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.695 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.695 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15552000", 2024-04-21T08:56:44.695 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.695 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.695 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.695 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain device health metrics", 2024-04-21T08:56:44.695 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.695 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.696 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.696 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_frequency": { 2024-04-21T08:56:44.696 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_frequency", 2024-04-21T08:56:44.696 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.696 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.696 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.696 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-04-21T08:56:44.696 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T08:56:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "self_heal": { 2024-04-21T08:56:44.697 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "self_heal", 2024-04-21T08:56:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.698 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T08:56:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.699 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T08:56:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.700 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_threshold": { 2024-04-21T08:56:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_threshold", 2024-04-21T08:56:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7257600", 2024-04-21T08:56:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.701 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.702 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.702 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T08:56:44.702 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.702 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.702 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.702 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.702 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.703 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "diskprediction_local", 2024-04-21T08:56:44.703 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.703 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.703 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.703 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.703 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.703 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.704 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.704 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.704 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.704 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.704 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.704 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.704 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.704 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.704 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.705 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.705 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.705 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.705 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.705 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.705 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.705 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.705 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.706 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.706 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.706 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.706 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.706 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.706 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.706 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.706 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.707 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.707 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.707 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.707 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.707 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.707 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.707 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.707 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.708 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.708 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.708 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.708 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.708 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.708 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.708 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.708 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.708 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.709 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.709 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.709 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.709 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.709 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.709 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.709 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.709 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.709 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.710 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.710 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.710 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.710 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.710 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.710 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.710 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.710 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.711 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.711 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.711 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.711 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.711 INFO:teuthology.orchestra.run.smithi007.stdout: "predict_interval": { 2024-04-21T08:56:44.711 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predict_interval", 2024-04-21T08:56:44.711 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.711 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.712 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.712 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-04-21T08:56:44.712 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.712 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.712 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.712 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.712 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.712 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.712 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.713 INFO:teuthology.orchestra.run.smithi007.stdout: "predictor_model": { 2024-04-21T08:56:44.713 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predictor_model", 2024-04-21T08:56:44.713 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.713 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.713 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.713 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "prophetstor", 2024-04-21T08:56:44.713 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.713 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.714 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.714 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.714 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.714 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.714 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.714 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:44.714 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:44.715 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.715 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.715 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.715 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:44.715 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.715 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.715 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.715 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.715 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.715 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.716 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.716 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.716 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.716 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "influx", 2024-04-21T08:56:44.716 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": false, 2024-04-21T08:56:44.716 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "influxdb python module not found", 2024-04-21T08:56:44.717 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.717 INFO:teuthology.orchestra.run.smithi007.stdout: "batch_size": { 2024-04-21T08:56:44.717 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "batch_size", 2024-04-21T08:56:44.717 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.717 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.717 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.717 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5000", 2024-04-21T08:56:44.717 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.717 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.718 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.718 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T08:56:44.718 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.718 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.718 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.718 INFO:teuthology.orchestra.run.smithi007.stdout: "database": { 2024-04-21T08:56:44.718 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "database", 2024-04-21T08:56:44.718 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.719 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.719 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.719 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "ceph", 2024-04-21T08:56:44.719 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.719 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.719 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.719 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T08:56:44.719 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.719 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.720 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.720 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": { 2024-04-21T08:56:44.720 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hostname", 2024-04-21T08:56:44.720 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.720 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.720 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.720 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.720 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.721 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.721 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.721 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server hostname", 2024-04-21T08:56:44.721 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.721 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.721 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.721 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:44.721 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "30", 2024-04-21T08:56:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "5", 2024-04-21T08:56:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.722 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T08:56:44.723 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.723 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.723 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.723 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.723 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.723 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.723 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.723 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.723 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.724 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.724 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.724 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.724 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.724 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.724 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.724 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.724 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.725 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.725 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.725 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.725 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.725 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.725 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.725 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.726 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.726 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.726 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.726 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.726 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.726 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.726 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.726 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.727 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.727 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.727 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.727 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.727 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.727 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.727 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.728 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.728 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.728 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.728 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.728 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.728 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.728 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.728 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.728 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.728 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.729 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.729 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.729 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.729 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.729 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.729 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.729 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.729 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.730 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.730 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.730 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.730 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.730 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.730 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.730 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.730 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.730 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.731 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.731 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.731 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.731 INFO:teuthology.orchestra.run.smithi007.stdout: "password": { 2024-04-21T08:56:44.731 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "password", 2024-04-21T08:56:44.731 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.731 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.732 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.732 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.732 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.732 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.732 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.732 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "password of InfluxDB server user", 2024-04-21T08:56:44.732 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.732 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.732 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.733 INFO:teuthology.orchestra.run.smithi007.stdout: "port": { 2024-04-21T08:56:44.733 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "port", 2024-04-21T08:56:44.733 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.733 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.733 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.733 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8086", 2024-04-21T08:56:44.733 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.734 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.734 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.734 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server port", 2024-04-21T08:56:44.734 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.734 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.734 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.734 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-04-21T08:56:44.735 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-04-21T08:56:44.735 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.735 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.735 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.735 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "false", 2024-04-21T08:56:44.735 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.735 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.735 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.735 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: "threads": { 2024-04-21T08:56:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threads", 2024-04-21T08:56:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.736 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "32", 2024-04-21T08:56:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T08:56:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.737 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "username": { 2024-04-21T08:56:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "username", 2024-04-21T08:56:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.738 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "username of InfluxDB server user", 2024-04-21T08:56:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "verify_ssl": { 2024-04-21T08:56:44.739 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "verify_ssl", 2024-04-21T08:56:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "true", 2024-04-21T08:56:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.740 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "insights", 2024-04-21T08:56:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.741 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.742 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.743 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.744 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.745 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.746 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.748 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.749 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "iostat", 2024-04-21T08:56:44.750 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.751 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.752 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.753 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.754 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.755 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.756 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.757 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.758 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.759 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "k8sevents", 2024-04-21T08:56:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.760 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_event_retention_days": { 2024-04-21T08:56:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ceph_event_retention_days", 2024-04-21T08:56:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7", 2024-04-21T08:56:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.761 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T08:56:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "config_check_secs": { 2024-04-21T08:56:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_check_secs", 2024-04-21T08:56:44.762 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "10", 2024-04-21T08:56:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T08:56:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.763 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.764 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.765 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.766 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.767 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.768 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.769 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.771 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.772 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "localpool", 2024-04-21T08:56:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.773 INFO:teuthology.orchestra.run.smithi007.stdout: "failure_domain": { 2024-04-21T08:56:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "failure_domain", 2024-04-21T08:56:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "host", 2024-04-21T08:56:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.774 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "failure domain for any created local pool", 2024-04-21T08:56:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T08:56:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.775 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.776 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.777 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.778 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.779 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.780 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.781 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.782 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": { 2024-04-21T08:56:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_size", 2024-04-21T08:56:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.784 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default min_size for any created local pool", 2024-04-21T08:56:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T08:56:44.785 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "num_rep": { 2024-04-21T08:56:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "num_rep", 2024-04-21T08:56:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.786 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3", 2024-04-21T08:56:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default replica count for any created local pool", 2024-04-21T08:56:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.787 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": { 2024-04-21T08:56:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_num", 2024-04-21T08:56:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "128", 2024-04-21T08:56:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.788 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T08:56:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "prefix": { 2024-04-21T08:56:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prefix", 2024-04-21T08:56:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.789 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name prefix for any created local pool", 2024-04-21T08:56:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.790 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "subtree": { 2024-04-21T08:56:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "subtree", 2024-04-21T08:56:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "rack", 2024-04-21T08:56:44.791 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T08:56:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T08:56:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mds_autoscaler", 2024-04-21T08:56:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.793 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.794 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.795 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.796 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.797 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.798 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.799 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.802 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirroring", 2024-04-21T08:56:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.803 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.804 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.805 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.806 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.808 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.809 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.810 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.811 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "nfs", 2024-04-21T08:56:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.813 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.814 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.815 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.816 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.817 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.818 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.819 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.820 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.821 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-04-21T08:56:44.822 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "fail_fs": { 2024-04-21T08:56:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "fail_fs", 2024-04-21T08:56:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.823 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T08:56:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.824 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.825 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.826 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.827 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.828 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.828 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.828 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.828 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.828 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.828 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.828 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.829 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.829 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.829 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.829 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.829 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.829 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.829 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.829 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.830 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.830 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.830 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.830 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.830 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.830 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.830 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.830 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.830 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.831 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.831 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.831 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.831 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.831 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.831 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.831 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.831 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.832 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.832 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.832 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.832 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.832 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.832 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.832 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.833 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator": { 2024-04-21T08:56:44.833 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-04-21T08:56:44.833 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.833 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.833 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.833 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.833 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.833 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.833 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.834 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm", 2024-04-21T08:56:44.834 INFO:teuthology.orchestra.run.smithi007.stdout: "rook", 2024-04-21T08:56:44.834 INFO:teuthology.orchestra.run.smithi007.stdout: "test_orchestrator" 2024-04-21T08:56:44.834 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.834 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Orchestrator backend", 2024-04-21T08:56:44.834 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.834 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.835 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.835 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_perf_query", 2024-04-21T08:56:44.835 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.835 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.835 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.836 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.836 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.836 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.836 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.836 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.836 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.836 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.837 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.837 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.837 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.837 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.837 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.837 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.837 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.838 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.838 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.838 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.838 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.838 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.838 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.839 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.839 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.839 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.839 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.839 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.839 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.839 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.840 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.840 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.840 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.840 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.840 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.840 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.840 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.840 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.841 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.841 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.841 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.841 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.841 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.841 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.841 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.841 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.842 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.842 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.842 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.842 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.842 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.842 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.842 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.843 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.843 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.843 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.843 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.843 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.843 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.843 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.843 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.844 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.844 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.844 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.844 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.844 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.844 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.844 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.844 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.845 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.845 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.845 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.845 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.845 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_support", 2024-04-21T08:56:44.845 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.845 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.845 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.845 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.846 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.846 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.846 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.846 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.846 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.846 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.846 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.846 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.847 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.847 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.847 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.847 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.847 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.847 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.847 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.848 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.848 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.848 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.848 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.848 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.848 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.848 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.848 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.849 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.849 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.849 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.849 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.849 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.849 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.849 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.849 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.849 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.850 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.850 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.850 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.850 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.850 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.850 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.850 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.851 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.851 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.851 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.851 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.851 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.851 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.851 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.851 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.851 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.852 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.852 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.852 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.852 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.852 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.852 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.852 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.853 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.853 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.853 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.853 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.853 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.853 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.853 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.853 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.853 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.854 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.854 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.854 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.854 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.854 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_autoscaler", 2024-04-21T08:56:44.854 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.854 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.855 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.855 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.855 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.855 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.855 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.855 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.855 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.855 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.855 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.856 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.856 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.856 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.856 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.856 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.856 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.856 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.856 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.856 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.857 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.857 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.857 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.857 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.857 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.857 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.857 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.857 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.858 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.858 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.858 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.858 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.858 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.858 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.858 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.858 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.859 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.859 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.859 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.859 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.859 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.859 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.859 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.859 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.860 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.860 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.860 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.860 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.860 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.860 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.860 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.860 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.861 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.861 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.861 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.861 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.861 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.861 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.861 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.861 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.862 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.862 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.862 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.862 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.862 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.862 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.862 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.862 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.863 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.863 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:44.863 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:44.863 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.863 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.863 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.863 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:44.863 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: "threshold": { 2024-04-21T08:56:44.864 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threshold", 2024-04-21T08:56:44.865 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:44.865 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.865 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.865 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-04-21T08:56:44.865 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1.0", 2024-04-21T08:56:44.865 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.865 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.865 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "scaling threshold", 2024-04-21T08:56:44.865 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T08:56:44.865 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.866 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.866 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.866 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.866 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.866 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "progress", 2024-04-21T08:56:44.866 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.866 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.867 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.867 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_pg_recovery_event": { 2024-04-21T08:56:44.867 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_pg_recovery_event", 2024-04-21T08:56:44.867 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.867 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.867 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.867 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.867 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.868 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.868 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.868 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T08:56:44.868 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.868 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.868 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.868 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-04-21T08:56:44.868 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-04-21T08:56:44.868 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.869 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.869 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.869 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.869 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.869 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.869 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.869 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.869 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.869 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.870 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.870 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.870 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.870 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.870 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.870 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.870 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.870 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.871 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.871 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.871 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.871 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.871 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.871 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.871 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.871 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.871 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.872 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.872 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.872 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.872 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.872 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.872 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.872 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.872 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.873 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.873 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.873 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.873 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.873 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.873 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.873 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.873 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.873 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.874 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.874 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.874 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.874 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.874 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.874 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.874 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.874 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.874 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.875 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.876 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.876 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.876 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.876 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.876 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.876 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.876 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.876 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.877 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.877 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.877 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.877 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.877 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.877 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.877 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.878 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.878 INFO:teuthology.orchestra.run.smithi007.stdout: "max_completed_events": { 2024-04-21T08:56:44.878 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_completed_events", 2024-04-21T08:56:44.878 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.878 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.878 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.878 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "50", 2024-04-21T08:56:44.879 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.879 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.879 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.879 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "number of past completed events to remember", 2024-04-21T08:56:44.879 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.879 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.879 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.880 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:44.880 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:44.880 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:44.880 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.880 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.880 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:44.880 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.881 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.881 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.881 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long the module is going to sleep", 2024-04-21T08:56:44.881 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.881 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.881 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.881 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.882 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.882 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.882 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus", 2024-04-21T08:56:44.882 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.882 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.882 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.882 INFO:teuthology.orchestra.run.smithi007.stdout: "cache": { 2024-04-21T08:56:44.882 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cache", 2024-04-21T08:56:44.883 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.883 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.883 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.883 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.883 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.883 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.883 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.883 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.883 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.884 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.884 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.884 INFO:teuthology.orchestra.run.smithi007.stdout: "exclude_perf_counters": { 2024-04-21T08:56:44.884 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "exclude_perf_counters", 2024-04-21T08:56:44.884 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.884 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.884 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.884 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.884 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.885 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.885 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.885 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T08:56:44.885 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T08:56:44.885 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.885 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.885 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.885 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.886 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.886 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.886 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.886 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.886 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.886 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.886 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.887 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.887 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.887 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.887 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.887 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.887 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.887 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.887 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.887 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.888 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.888 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.888 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.888 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.888 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.888 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.888 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.888 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.889 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.889 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.889 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.889 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.889 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.889 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.889 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.889 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.890 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.890 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.890 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.890 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.890 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.890 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.890 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.890 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.890 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.891 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.891 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.891 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.891 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.891 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.891 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.891 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.891 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.891 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.892 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.892 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.892 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.892 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.892 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.892 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.892 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.892 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.893 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.893 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.893 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.893 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.893 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.893 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.893 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools": { 2024-04-21T08:56:44.894 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools", 2024-04-21T08:56:44.894 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.894 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.894 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.894 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.894 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.894 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.895 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.895 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.895 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.895 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.895 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.895 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T08:56:44.895 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T08:56:44.896 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.896 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.896 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.896 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "300", 2024-04-21T08:56:44.896 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.896 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.896 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.896 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.897 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.897 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.897 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.897 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_interval": { 2024-04-21T08:56:44.897 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_interval", 2024-04-21T08:56:44.897 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15.0", 2024-04-21T08:56:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.898 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-04-21T08:56:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-04-21T08:56:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.899 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-04-21T08:56:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T08:56:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.900 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-04-21T08:56:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-04-21T08:56:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "9283", 2024-04-21T08:56:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.901 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T08:56:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: "stale_cache_strategy": { 2024-04-21T08:56:44.902 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stale_cache_strategy", 2024-04-21T08:56:44.903 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.903 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.903 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.903 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "log", 2024-04-21T08:56:44.903 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.903 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.903 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.903 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.904 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.904 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.904 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.904 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-04-21T08:56:44.904 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-04-21T08:56:44.904 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.904 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.905 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.905 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "default", 2024-04-21T08:56:44.905 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.905 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.905 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.905 INFO:teuthology.orchestra.run.smithi007.stdout: "default", 2024-04-21T08:56:44.905 INFO:teuthology.orchestra.run.smithi007.stdout: "error" 2024-04-21T08:56:44.905 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.906 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.906 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.906 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.906 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.906 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.906 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-04-21T08:56:44.906 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:44.907 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.907 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.907 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.907 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-04-21T08:56:44.907 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-04-21T08:56:44.907 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-04-21T08:56:44.907 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.907 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.908 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.908 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.908 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.908 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.908 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.908 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.908 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_support", 2024-04-21T08:56:44.909 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.909 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.909 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.909 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.909 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.909 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.909 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.909 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.910 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.910 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.910 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.910 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.910 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.910 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.910 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.910 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.910 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.911 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.911 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.911 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.911 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.911 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.911 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.911 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.911 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.912 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.912 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.912 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.912 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.912 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.912 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.912 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.912 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.912 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.913 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.913 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.913 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.913 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.913 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.913 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.913 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.913 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.914 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.914 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.914 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.914 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.914 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.914 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.914 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.914 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.914 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.915 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.915 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.915 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.915 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.915 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.915 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.915 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.915 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.916 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.916 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.916 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.916 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.916 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.916 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.916 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.916 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.916 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.917 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.917 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_snap_create": { 2024-04-21T08:56:44.917 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_snap_create", 2024-04-21T08:56:44.917 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.917 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.917 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.917 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:44.918 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.918 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.918 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.918 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.918 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.918 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.918 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.918 INFO:teuthology.orchestra.run.smithi007.stdout: "mirror_snapshot_schedule": { 2024-04-21T08:56:44.919 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T08:56:44.919 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.919 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.919 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.919 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.919 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.919 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.919 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.919 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.920 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.920 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.920 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.920 INFO:teuthology.orchestra.run.smithi007.stdout: "trash_purge_schedule": { 2024-04-21T08:56:44.920 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "trash_purge_schedule", 2024-04-21T08:56:44.920 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.920 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.920 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.921 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.921 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.921 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.921 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.921 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.921 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.921 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.921 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.921 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.922 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.922 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.922 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "restful", 2024-04-21T08:56:44.922 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.922 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.922 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.922 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_auth": { 2024-04-21T08:56:44.923 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_auth", 2024-04-21T08:56:44.923 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.923 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.923 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.923 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.923 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.923 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.923 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.924 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.924 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.924 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.924 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.924 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-04-21T08:56:44.924 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-04-21T08:56:44.924 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.924 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.925 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.925 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.925 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.925 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.925 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.925 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.925 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.925 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.925 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.926 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.926 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.926 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.926 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.926 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.926 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.927 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.928 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.928 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.928 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.928 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.928 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.928 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.928 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.929 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.929 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.929 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.929 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.929 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.929 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.929 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.929 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.929 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.930 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.930 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.930 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.930 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.930 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.930 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.930 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.931 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.931 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.931 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.931 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.931 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.931 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.931 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.931 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.931 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.932 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.932 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.932 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.932 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.932 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.932 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.932 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.932 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.933 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.933 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.933 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.933 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.933 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.933 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.933 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.933 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.933 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.933 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.934 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.934 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-04-21T08:56:44.934 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-04-21T08:56:44.934 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.934 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.934 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.934 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.935 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.935 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.935 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.935 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.935 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.935 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.935 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.936 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-04-21T08:56:44.936 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-04-21T08:56:44.936 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.936 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.936 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.936 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.936 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.936 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.936 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.937 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.937 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.937 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.937 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.937 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.937 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.938 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rgw", 2024-04-21T08:56:44.938 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.938 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.938 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.938 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.938 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.938 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.938 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.938 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.939 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.939 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.939 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.939 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.939 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.939 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.939 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.939 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.939 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.940 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.940 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.940 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.940 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.940 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.940 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.940 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.941 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.941 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.941 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.941 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.941 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.941 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.941 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.941 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.941 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.942 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.943 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.944 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.945 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.946 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rook", 2024-04-21T08:56:44.947 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.947 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.947 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.947 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.947 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.947 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.947 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.947 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.947 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.948 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.948 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.948 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.948 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.948 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.948 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.948 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.948 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.949 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.949 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.949 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.949 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.949 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.949 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.949 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.951 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.952 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.953 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.953 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.953 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.953 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.953 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.953 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.953 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.953 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.954 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.954 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.954 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.954 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.954 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.954 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.954 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.954 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.955 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.955 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.955 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.955 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.955 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.955 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.955 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_class": { 2024-04-21T08:56:44.956 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "storage_class", 2024-04-21T08:56:44.956 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.956 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.956 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.956 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "local", 2024-04-21T08:56:44.956 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.956 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.956 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.956 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T08:56:44.957 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.957 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.957 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.957 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.957 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.957 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.957 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "selftest", 2024-04-21T08:56:44.957 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.958 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.958 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.958 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.958 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.958 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.958 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.958 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.958 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.959 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.959 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.959 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.959 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.959 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.959 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.959 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.959 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.959 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.960 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.960 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.960 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.960 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.960 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.960 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.960 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.960 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.960 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.961 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.961 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.961 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.961 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.961 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.961 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.961 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.961 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.961 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.962 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.962 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.962 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.962 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.962 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.962 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.962 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.963 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.963 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.963 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.963 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.963 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.963 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.963 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.963 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.963 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.964 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.964 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.964 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.964 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.964 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.964 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.964 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.965 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.965 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.965 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.965 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.965 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.965 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.965 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.965 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.966 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.966 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.966 INFO:teuthology.orchestra.run.smithi007.stdout: "roption1": { 2024-04-21T08:56:44.966 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption1", 2024-04-21T08:56:44.966 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.966 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.966 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.967 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.967 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.967 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.967 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.967 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.967 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.967 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.967 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.968 INFO:teuthology.orchestra.run.smithi007.stdout: "roption2": { 2024-04-21T08:56:44.968 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption2", 2024-04-21T08:56:44.968 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.968 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.968 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.968 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "xyz", 2024-04-21T08:56:44.968 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.968 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.969 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.969 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.969 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.969 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.969 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.969 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption1": { 2024-04-21T08:56:44.969 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption1", 2024-04-21T08:56:44.969 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.970 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.970 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.970 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.970 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.970 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.970 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.970 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.970 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.970 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.971 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.971 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption2": { 2024-04-21T08:56:44.971 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption2", 2024-04-21T08:56:44.971 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.971 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.971 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.971 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.971 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.972 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.972 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.972 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.972 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.972 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.972 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.972 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.972 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption3": { 2024-04-21T08:56:44.972 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption3", 2024-04-21T08:56:44.973 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:44.973 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.973 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.973 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.973 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.973 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.973 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.973 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.974 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.974 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.974 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.974 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption4": { 2024-04-21T08:56:44.974 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption4", 2024-04-21T08:56:44.974 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.974 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.975 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.975 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.975 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.975 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.975 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.975 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.975 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.976 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.976 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.976 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption5": { 2024-04-21T08:56:44.976 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption5", 2024-04-21T08:56:44.976 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.976 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.977 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.977 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.977 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.977 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.977 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.977 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.977 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.977 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.978 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.978 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption6": { 2024-04-21T08:56:44.978 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption6", 2024-04-21T08:56:44.978 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.978 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.978 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.978 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:44.979 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.979 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.979 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.979 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.979 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.979 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.979 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.980 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption7": { 2024-04-21T08:56:44.980 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption7", 2024-04-21T08:56:44.980 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:44.980 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.980 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.980 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.980 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:44.981 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "42", 2024-04-21T08:56:44.981 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.981 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.981 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.981 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.981 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.981 INFO:teuthology.orchestra.run.smithi007.stdout: "testkey": { 2024-04-21T08:56:44.982 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testkey", 2024-04-21T08:56:44.982 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.982 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.982 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.982 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.982 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.982 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.982 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.983 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.983 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.983 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.983 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.983 INFO:teuthology.orchestra.run.smithi007.stdout: "testlkey": { 2024-04-21T08:56:44.983 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testlkey", 2024-04-21T08:56:44.983 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.984 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "testnewline": { 2024-04-21T08:56:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testnewline", 2024-04-21T08:56:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.985 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.986 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snap_schedule", 2024-04-21T08:56:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_m_granularity": { 2024-04-21T08:56:44.987 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_m_granularity", 2024-04-21T08:56:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T08:56:44.988 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.989 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.989 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.989 INFO:teuthology.orchestra.run.smithi007.stdout: "dump_on_update": { 2024-04-21T08:56:44.989 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dump_on_update", 2024-04-21T08:56:44.989 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.989 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.989 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.989 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.990 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.990 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.990 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.990 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "dump database to debug log on update", 2024-04-21T08:56:44.990 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.990 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.990 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.991 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:44.991 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:44.991 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.991 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.991 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.991 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:44.991 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.991 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.993 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.993 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.993 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.993 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:44.993 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:44.993 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.993 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.993 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.994 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.994 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.994 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.994 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.994 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.994 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.994 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.994 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.995 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:44.995 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:44.995 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:44.995 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.995 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.995 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:44.996 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.996 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.996 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:44.996 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:44.996 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:44.996 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:44.996 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:44.996 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:44.997 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:44.997 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:44.997 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.997 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.997 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.997 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.997 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:44.998 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:44.998 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:44.998 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:44.998 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:44.998 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:44.998 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:44.998 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:44.998 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:44.999 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:44.999 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:44.999 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:44.999 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:44.999 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.999 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:44.999 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:44.999 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:44.999 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stats", 2024-04-21T08:56:45.000 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.000 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.000 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.000 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.000 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.000 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.000 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.000 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.000 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.001 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.001 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.001 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.001 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.001 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.001 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.001 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.001 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.001 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.002 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.002 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.002 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.002 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.002 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.002 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.002 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.002 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.003 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.003 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.003 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.003 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.003 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.003 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.003 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.004 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.004 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.004 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.004 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.004 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.004 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.004 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.005 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.005 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.005 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.005 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.005 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.005 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.005 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.005 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.005 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.006 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.006 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.006 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.006 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.006 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.006 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.006 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.007 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.007 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.007 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.007 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.007 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.007 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.007 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.008 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.008 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.008 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.008 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.008 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.008 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.008 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.008 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.009 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.009 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "status", 2024-04-21T08:56:45.009 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.009 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.009 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.009 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.009 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.010 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.010 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.010 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.010 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.010 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.010 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.010 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.011 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.011 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.011 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.011 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.011 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.011 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.011 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.011 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.012 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.012 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.012 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.012 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.012 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.012 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.012 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.013 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.013 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.013 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.013 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.013 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.013 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.013 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.013 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.014 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.015 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.015 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.015 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.015 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.015 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.015 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.015 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.015 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.016 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.016 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.016 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.016 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.016 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.016 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.016 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.016 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.017 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.017 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.017 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.017 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.017 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.017 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.017 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.018 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.018 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.018 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.018 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.018 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.018 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.018 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.019 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.019 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telegraf", 2024-04-21T08:56:45.019 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.019 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.019 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.019 INFO:teuthology.orchestra.run.smithi007.stdout: "address": { 2024-04-21T08:56:45.019 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "address", 2024-04-21T08:56:45.019 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.020 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.020 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.020 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T08:56:45.020 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.020 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.020 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.020 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.020 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.021 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.021 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.021 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:45.021 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:45.021 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.021 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.021 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.022 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15", 2024-04-21T08:56:45.022 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.022 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.022 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.022 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.022 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.022 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.023 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.023 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.023 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.023 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.023 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.023 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.023 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.023 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.024 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.024 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.024 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.024 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.024 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.024 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.024 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.024 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.024 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.025 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.025 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.025 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.025 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.025 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.025 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.025 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.025 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.026 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.026 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.026 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.026 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.026 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.026 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.026 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.026 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.027 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.027 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.027 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.027 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.027 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.027 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.027 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.028 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.028 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.028 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.028 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.028 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.028 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.028 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.028 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.028 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.029 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.029 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.029 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.029 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.029 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.029 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.029 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.029 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.030 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.030 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.030 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.030 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.030 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.030 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.030 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.031 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.031 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.031 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.031 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.031 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.031 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.031 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telemetry", 2024-04-21T08:56:45.032 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.032 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.032 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.032 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_basic": { 2024-04-21T08:56:45.032 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_basic", 2024-04-21T08:56:45.032 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.032 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.032 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.033 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.033 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.033 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.033 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.033 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T08:56:45.033 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.033 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.033 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.034 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_crash": { 2024-04-21T08:56:45.034 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_crash", 2024-04-21T08:56:45.034 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.034 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.034 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.034 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.034 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.034 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.035 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.035 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T08:56:45.035 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.035 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.035 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.035 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_device": { 2024-04-21T08:56:45.035 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_device", 2024-04-21T08:56:45.036 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.036 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.036 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.036 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.036 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.036 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.036 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.036 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T08:56:45.037 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.037 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.037 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.037 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_ident": { 2024-04-21T08:56:45.037 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_ident", 2024-04-21T08:56:45.037 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T08:56:45.038 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_perf": { 2024-04-21T08:56:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_perf", 2024-04-21T08:56:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.039 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T08:56:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "contact": { 2024-04-21T08:56:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "contact", 2024-04-21T08:56:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.041 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "description": { 2024-04-21T08:56:45.042 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "description", 2024-04-21T08:56:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.043 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "device_url": { 2024-04-21T08:56:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_url", 2024-04-21T08:56:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T08:56:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.045 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.046 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.046 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-04-21T08:56:45.046 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-04-21T08:56:45.046 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.046 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.046 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.047 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.047 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.047 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.047 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.047 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.047 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.047 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.048 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.048 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:45.048 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:45.048 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.048 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.048 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.049 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "24", 2024-04-21T08:56:45.049 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "8", 2024-04-21T08:56:45.049 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.049 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.049 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.049 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.049 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.050 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.050 INFO:teuthology.orchestra.run.smithi007.stdout: "last_opt_revision": { 2024-04-21T08:56:45.050 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "last_opt_revision", 2024-04-21T08:56:45.050 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.050 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.050 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.051 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1", 2024-04-21T08:56:45.051 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.051 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.051 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.051 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.051 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.051 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.051 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard": { 2024-04-21T08:56:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard", 2024-04-21T08:56:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.052 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard_description": { 2024-04-21T08:56:45.053 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard_description", 2024-04-21T08:56:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.054 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.055 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.056 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.057 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.058 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.058 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.058 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.058 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.058 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.058 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.058 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.058 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.059 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.059 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.059 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.059 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.059 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.059 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.059 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.059 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.060 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.060 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.060 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.060 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.060 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.060 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.060 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.060 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.060 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.061 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.061 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.061 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.061 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.061 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.061 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.061 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.062 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.062 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.062 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.062 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.062 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.062 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.062 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.062 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.063 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.063 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.063 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.063 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.063 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.063 INFO:teuthology.orchestra.run.smithi007.stdout: "organization": { 2024-04-21T08:56:45.063 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "organization", 2024-04-21T08:56:45.064 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.064 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.064 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.064 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.064 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.064 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.064 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.064 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.065 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.065 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.065 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.065 INFO:teuthology.orchestra.run.smithi007.stdout: "proxy": { 2024-04-21T08:56:45.065 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "proxy", 2024-04-21T08:56:45.065 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.065 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.066 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.066 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.066 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.066 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.066 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.066 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.066 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.066 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.067 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.067 INFO:teuthology.orchestra.run.smithi007.stdout: "url": { 2024-04-21T08:56:45.067 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url", 2024-04-21T08:56:45.067 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.067 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.067 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.067 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T08:56:45.068 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.068 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.068 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.068 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.068 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.068 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.068 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.069 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.069 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test_orchestrator", 2024-04-21T08:56:45.069 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.069 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.069 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.069 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.070 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.070 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.070 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.070 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.070 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.070 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.070 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.071 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.072 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.073 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.074 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.075 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.076 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.077 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.078 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.078 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.078 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.078 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.078 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.078 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.078 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "volumes", 2024-04-21T08:56:45.079 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.079 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.079 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.079 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.079 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.079 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.079 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.079 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.079 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.080 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.080 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.080 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.080 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.080 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.080 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.080 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.080 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.080 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.081 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.081 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.081 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.081 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.081 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.081 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.081 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.081 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.082 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.082 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.082 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.082 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.082 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.082 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.082 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.083 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.083 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.083 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.083 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.083 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.083 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.083 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.083 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.084 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.084 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.084 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.084 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.084 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.084 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.084 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.084 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.084 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.085 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.085 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.085 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.085 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.085 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.085 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.085 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.086 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.086 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.086 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.086 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.086 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.086 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.086 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.086 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.087 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.087 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.087 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.087 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.087 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_clones": { 2024-04-21T08:56:45.087 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_clones", 2024-04-21T08:56:45.087 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.087 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.088 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.088 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4", 2024-04-21T08:56:45.088 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.088 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.088 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.088 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T08:56:45.088 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.088 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.088 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.089 INFO:teuthology.orchestra.run.smithi007.stdout: "periodic_async_work": { 2024-04-21T08:56:45.089 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "periodic_async_work", 2024-04-21T08:56:45.089 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.089 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.089 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.089 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.089 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.090 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.090 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.090 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Periodically check for async work", 2024-04-21T08:56:45.090 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.090 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.090 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.090 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_delay": { 2024-04-21T08:56:45.091 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_delay", 2024-04-21T08:56:45.091 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.091 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.091 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.091 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:45.091 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.091 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.091 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.091 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T08:56:45.092 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.092 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.092 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.092 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_no_wait": { 2024-04-21T08:56:45.092 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T08:56:45.092 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.092 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T08:56:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix", 2024-04-21T08:56:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: "discovery_interval": { 2024-04-21T08:56:45.095 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "discovery_interval", 2024-04-21T08:56:45.095 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:45.095 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.095 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.095 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "100", 2024-04-21T08:56:45.095 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.095 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.095 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.095 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.095 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.096 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.096 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.096 INFO:teuthology.orchestra.run.smithi007.stdout: "identifier": { 2024-04-21T08:56:45.096 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "identifier", 2024-04-21T08:56:45.096 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.096 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.096 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.096 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.097 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.097 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.097 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.097 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.097 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.097 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.097 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.098 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:45.098 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:45.098 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.098 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.098 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.098 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:45.098 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.098 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.098 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.099 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.099 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.099 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.099 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.099 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.099 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.100 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.100 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.100 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.100 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.100 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.100 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.100 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.100 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.101 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.101 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.101 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.101 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.101 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.101 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.101 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.101 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.102 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.102 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.102 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.102 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.102 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.102 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.102 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.103 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.103 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.103 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.103 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.103 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.103 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.103 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.103 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.104 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.104 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.104 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.104 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.104 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.104 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.105 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.105 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.105 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.105 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.105 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.105 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.105 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.105 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.105 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.106 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.106 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.106 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.106 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.106 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.106 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.106 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.107 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.107 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.107 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.107 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.107 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.107 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.107 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.107 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.108 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.108 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.108 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.108 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.108 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_host": { 2024-04-21T08:56:45.108 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_host", 2024-04-21T08:56:45.109 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.109 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.109 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.109 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.109 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.109 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.109 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.109 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.110 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.110 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.110 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.110 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.110 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_port": { 2024-04-21T08:56:45.110 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_port", 2024-04-21T08:56:45.110 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.110 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.111 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.111 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10051", 2024-04-21T08:56:45.111 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.111 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.111 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.111 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.111 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.111 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.111 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.112 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_sender": { 2024-04-21T08:56:45.112 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_sender", 2024-04-21T08:56:45.112 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.112 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.112 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.112 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T08:56:45.112 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.113 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.113 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.113 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.113 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.113 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.113 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.113 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.113 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.114 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.114 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:56:45.114 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.114 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.114 INFO:teuthology.orchestra.run.smithi007.stdout: "modules": [ 2024-04-21T08:56:45.114 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm", 2024-04-21T08:56:45.114 INFO:teuthology.orchestra.run.smithi007.stdout: "dashboard", 2024-04-21T08:56:45.114 INFO:teuthology.orchestra.run.smithi007.stdout: "iostat", 2024-04-21T08:56:45.115 INFO:teuthology.orchestra.run.smithi007.stdout: "nfs", 2024-04-21T08:56:45.115 INFO:teuthology.orchestra.run.smithi007.stdout: "restful", 2024-04-21T08:56:45.115 INFO:teuthology.orchestra.run.smithi007.stdout: "snap_schedule" 2024-04-21T08:56:45.115 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.115 INFO:teuthology.orchestra.run.smithi007.stdout: "available_modules": [ 2024-04-21T08:56:45.115 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.115 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "alerts", 2024-04-21T08:56:45.115 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.116 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.116 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.116 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:45.116 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:45.116 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.116 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.116 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.116 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:45.117 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.117 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.117 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.117 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T08:56:45.117 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.117 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.117 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.118 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.118 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.118 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.118 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.118 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.118 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.118 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.118 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.119 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.119 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.119 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.119 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.119 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.119 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.119 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.119 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.119 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.120 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.120 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.120 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.120 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.120 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.120 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.120 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.120 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.121 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.121 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.121 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.121 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.121 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.121 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.121 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.121 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.122 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.122 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.122 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.122 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.122 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.122 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.122 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.122 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.123 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.123 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.123 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.123 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.123 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.123 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.123 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.123 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.124 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.124 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.124 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.124 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.124 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.124 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.124 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.124 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.125 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.125 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.125 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.125 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.125 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.125 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.125 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.125 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.125 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.126 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.126 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_destination": { 2024-04-21T08:56:45.126 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_destination", 2024-04-21T08:56:45.126 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.126 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.126 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.127 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.127 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.127 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.127 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.127 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email address to send alerts to", 2024-04-21T08:56:45.127 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.127 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.127 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.128 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_from_name": { 2024-04-21T08:56:45.128 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_from_name", 2024-04-21T08:56:45.128 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.128 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.128 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.128 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "Ceph", 2024-04-21T08:56:45.128 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.128 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.128 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.129 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email From: name", 2024-04-21T08:56:45.129 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.129 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.129 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.129 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_host": { 2024-04-21T08:56:45.129 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_host", 2024-04-21T08:56:45.129 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.129 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.130 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.130 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.130 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.130 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.130 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.130 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP server", 2024-04-21T08:56:45.130 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.131 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.131 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.131 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_password": { 2024-04-21T08:56:45.131 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_password", 2024-04-21T08:56:45.131 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.131 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.131 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.131 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.132 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.132 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.132 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.132 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Password to authenticate with", 2024-04-21T08:56:45.132 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.132 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.132 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_port": { 2024-04-21T08:56:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_port", 2024-04-21T08:56:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "465", 2024-04-21T08:56:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.133 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP port", 2024-04-21T08:56:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_sender": { 2024-04-21T08:56:45.134 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_sender", 2024-04-21T08:56:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP envelope sender", 2024-04-21T08:56:45.135 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_ssl": { 2024-04-21T08:56:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_ssl", 2024-04-21T08:56:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.136 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T08:56:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.137 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_user": { 2024-04-21T08:56:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_user", 2024-04-21T08:56:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.138 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.139 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.139 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.139 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "User to authenticate as", 2024-04-21T08:56:45.139 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.139 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.139 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.139 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.139 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.140 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.140 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "balancer", 2024-04-21T08:56:45.140 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.140 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.140 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.140 INFO:teuthology.orchestra.run.smithi007.stdout: "active": { 2024-04-21T08:56:45.140 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "active", 2024-04-21T08:56:45.140 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.141 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.141 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.141 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.141 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.141 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.141 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.141 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T08:56:45.141 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.142 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.142 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.142 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_time": { 2024-04-21T08:56:45.142 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_time", 2024-04-21T08:56:45.142 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.142 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.142 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.143 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0000", 2024-04-21T08:56:45.143 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.143 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.143 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.143 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T08:56:45.143 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:45.143 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.143 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.144 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_weekday": { 2024-04-21T08:56:45.144 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_weekday", 2024-04-21T08:56:45.144 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:45.144 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.144 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.144 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:45.144 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-04-21T08:56:45.145 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-04-21T08:56:45.145 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.145 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T08:56:45.145 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:45.145 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.145 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.145 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_max_iterations": { 2024-04-21T08:56:45.145 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_max_iterations", 2024-04-21T08:56:45.146 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:45.146 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.146 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.146 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "25", 2024-04-21T08:56:45.146 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:45.146 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "250", 2024-04-21T08:56:45.146 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.146 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T08:56:45.147 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.147 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.147 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.147 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_metrics": { 2024-04-21T08:56:45.147 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_metrics", 2024-04-21T08:56:45.147 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.147 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.148 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.148 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T08:56:45.148 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.148 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.148 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.148 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T08:56:45.148 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T08:56:45.148 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.148 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.149 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_step": { 2024-04-21T08:56:45.149 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_step", 2024-04-21T08:56:45.149 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:45.149 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.149 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.149 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.5", 2024-04-21T08:56:45.149 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0.001", 2024-04-21T08:56:45.149 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "0.999", 2024-04-21T08:56:45.150 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.150 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "aggressiveness of optimization", 2024-04-21T08:56:45.150 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T08:56:45.150 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.150 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.150 INFO:teuthology.orchestra.run.smithi007.stdout: "end_time": { 2024-04-21T08:56:45.150 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_time", 2024-04-21T08:56:45.150 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.151 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.151 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.151 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2359", 2024-04-21T08:56:45.151 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.151 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.151 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.151 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T08:56:45.151 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:45.152 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.152 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.152 INFO:teuthology.orchestra.run.smithi007.stdout: "end_weekday": { 2024-04-21T08:56:45.152 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_weekday", 2024-04-21T08:56:45.152 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:45.152 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.152 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.152 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:45.153 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-04-21T08:56:45.153 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-04-21T08:56:45.153 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.153 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T08:56:45.153 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:45.153 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.153 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.153 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.154 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.154 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.154 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.154 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.154 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.154 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.154 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.154 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.154 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.155 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.155 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.155 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.155 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.155 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.155 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.155 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.155 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.156 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.156 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.156 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.156 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.156 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.156 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.156 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.157 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.157 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.157 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.157 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.157 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.157 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.157 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.157 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.158 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.158 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.158 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.158 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.158 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.158 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.158 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.158 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.159 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.159 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.159 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.159 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.159 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.159 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.159 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.159 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.159 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.160 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.160 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.160 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.160 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.160 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.160 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.160 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.160 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.161 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.161 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.161 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.161 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.161 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.161 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.161 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.161 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.162 INFO:teuthology.orchestra.run.smithi007.stdout: "min_score": { 2024-04-21T08:56:45.162 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_score", 2024-04-21T08:56:45.162 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:45.162 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.162 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.162 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:45.162 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.162 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.162 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.163 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T08:56:45.163 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.163 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.163 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.163 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-04-21T08:56:45.163 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-04-21T08:56:45.163 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.163 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.164 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.164 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "upmap", 2024-04-21T08:56:45.164 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.164 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.164 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.164 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-compat", 2024-04-21T08:56:45.164 INFO:teuthology.orchestra.run.smithi007.stdout: "none", 2024-04-21T08:56:45.164 INFO:teuthology.orchestra.run.smithi007.stdout: "read", 2024-04-21T08:56:45.164 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap", 2024-04-21T08:56:45.165 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap-read" 2024-04-21T08:56:45.165 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.165 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Balancer mode", 2024-04-21T08:56:45.165 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.165 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.165 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.165 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_ids": { 2024-04-21T08:56:45.165 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_ids", 2024-04-21T08:56:45.166 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.166 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.166 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.166 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.166 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.166 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.166 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.166 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T08:56:45.166 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.167 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.167 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.167 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:45.167 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:45.167 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.167 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.167 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.168 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:45.168 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.168 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.168 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.168 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T08:56:45.168 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.168 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.168 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.168 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_deviation": { 2024-04-21T08:56:45.169 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_deviation", 2024-04-21T08:56:45.169 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.169 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.169 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.169 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:45.169 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:45.169 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.169 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.170 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T08:56:45.170 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T08:56:45.170 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.170 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.170 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_optimizations": { 2024-04-21T08:56:45.170 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_optimizations", 2024-04-21T08:56:45.170 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:45.171 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.171 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.171 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:45.171 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.171 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.171 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.171 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T08:56:45.171 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.172 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.172 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.172 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.172 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm", 2024-04-21T08:56:45.172 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.172 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.173 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.173 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_down_multiplier": { 2024-04-21T08:56:45.173 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_down_multiplier", 2024-04-21T08:56:45.173 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:45.173 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.173 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.173 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-04-21T08:56:45.173 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.173 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.173 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.174 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T08:56:45.174 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.174 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.174 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.174 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_refresh_rate": { 2024-04-21T08:56:45.174 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_refresh_rate", 2024-04-21T08:56:45.174 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.175 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.175 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.175 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "20", 2024-04-21T08:56:45.175 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.175 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.175 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.175 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T08:56:45.175 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.176 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.176 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.176 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_starting_port": { 2024-04-21T08:56:45.176 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_starting_port", 2024-04-21T08:56:45.176 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.176 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.176 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.176 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4721", 2024-04-21T08:56:45.177 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.177 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.177 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.177 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T08:56:45.177 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.177 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.177 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.178 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_ptrace": { 2024-04-21T08:56:45.178 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_ptrace", 2024-04-21T08:56:45.178 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.178 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.178 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.178 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.178 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.178 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.179 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.179 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T08:56:45.179 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T08:56:45.179 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.179 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.179 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_interval": { 2024-04-21T08:56:45.179 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_interval", 2024-04-21T08:56:45.180 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.180 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.180 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.180 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:45.180 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.180 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.180 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.180 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T08:56:45.180 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.181 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.181 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.181 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_memory_target_ratio": { 2024-04-21T08:56:45.181 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T08:56:45.181 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:45.181 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.181 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.182 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.7", 2024-04-21T08:56:45.182 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.182 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.182 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.182 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T08:56:45.182 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.182 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.182 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.183 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm_log_destination": { 2024-04-21T08:56:45.183 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm_log_destination", 2024-04-21T08:56:45.183 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.183 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.183 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.183 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.183 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.184 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.184 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.184 INFO:teuthology.orchestra.run.smithi007.stdout: "file", 2024-04-21T08:56:45.184 INFO:teuthology.orchestra.run.smithi007.stdout: "file,syslog", 2024-04-21T08:56:45.184 INFO:teuthology.orchestra.run.smithi007.stdout: "syslog" 2024-04-21T08:56:45.184 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.184 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T08:56:45.184 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.184 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.185 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.185 INFO:teuthology.orchestra.run.smithi007.stdout: "cgroups_split": { 2024-04-21T08:56:45.185 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cgroups_split", 2024-04-21T08:56:45.185 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.185 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.185 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.185 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.185 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.186 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.186 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.186 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T08:56:45.186 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.186 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.186 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.186 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.187 INFO:teuthology.orchestra.run.smithi007.stdout: "config_checks_enabled": { 2024-04-21T08:56:45.187 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_checks_enabled", 2024-04-21T08:56:45.187 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.187 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.187 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.187 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.187 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.187 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.187 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.188 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T08:56:45.188 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.188 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.188 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.188 INFO:teuthology.orchestra.run.smithi007.stdout: "config_dashboard": { 2024-04-21T08:56:45.188 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_dashboard", 2024-04-21T08:56:45.188 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.188 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.189 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.189 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.189 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.189 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.189 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.189 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T08:56:45.189 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.189 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.190 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.190 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_alertmanager": { 2024-04-21T08:56:45.190 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_alertmanager", 2024-04-21T08:56:45.190 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.190 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.190 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.191 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T08:56:45.191 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.191 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.191 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.191 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:45.191 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.191 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.191 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.192 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_base": { 2024-04-21T08:56:45.192 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_base", 2024-04-21T08:56:45.192 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.192 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.192 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.192 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T08:56:45.192 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.192 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.193 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.193 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Container image name, without the tag", 2024-04-21T08:56:45.193 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.193 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.193 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.193 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_elasticsearch": { 2024-04-21T08:56:45.193 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_elasticsearch", 2024-04-21T08:56:45.193 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.193 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.194 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.194 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T08:56:45.194 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.194 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.194 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.194 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "elasticsearch container image", 2024-04-21T08:56:45.194 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.195 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.195 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.195 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_grafana": { 2024-04-21T08:56:45.195 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_grafana", 2024-04-21T08:56:45.195 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.195 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.195 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.195 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T08:56:45.196 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.196 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.196 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.196 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:45.196 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.196 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.196 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.197 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_haproxy": { 2024-04-21T08:56:45.197 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_haproxy", 2024-04-21T08:56:45.197 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.197 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.197 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.197 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T08:56:45.197 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.197 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.197 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.197 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "HAproxy container image", 2024-04-21T08:56:45.198 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.198 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.198 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.198 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_agent": { 2024-04-21T08:56:45.198 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_agent", 2024-04-21T08:56:45.198 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.198 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.199 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.199 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T08:56:45.199 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.199 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.199 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.199 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger agent container image", 2024-04-21T08:56:45.199 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.199 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.200 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.200 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_collector": { 2024-04-21T08:56:45.200 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_collector", 2024-04-21T08:56:45.200 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.200 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.200 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.200 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T08:56:45.200 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.201 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.201 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.201 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger collector container image", 2024-04-21T08:56:45.201 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.201 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.201 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.201 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_query": { 2024-04-21T08:56:45.202 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_query", 2024-04-21T08:56:45.202 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.202 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.202 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.202 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T08:56:45.202 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.202 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.202 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.203 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger query container image", 2024-04-21T08:56:45.203 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.203 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.203 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.203 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_keepalived": { 2024-04-21T08:56:45.203 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_keepalived", 2024-04-21T08:56:45.203 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.203 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.204 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.204 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T08:56:45.204 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.204 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.204 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.204 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Keepalived container image", 2024-04-21T08:56:45.204 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.204 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.204 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.205 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_loki": { 2024-04-21T08:56:45.205 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_loki", 2024-04-21T08:56:45.205 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.205 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.205 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.205 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T08:56:45.205 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.206 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.206 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.206 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Loki container image", 2024-04-21T08:56:45.206 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.206 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.206 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.206 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_node_exporter": { 2024-04-21T08:56:45.206 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_node_exporter", 2024-04-21T08:56:45.207 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.207 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.207 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.207 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T08:56:45.207 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.207 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.207 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.208 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:45.208 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.208 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.208 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.208 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_nvmeof": { 2024-04-21T08:56:45.208 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_nvmeof", 2024-04-21T08:56:45.208 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.209 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.209 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.209 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T08:56:45.209 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.209 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.209 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.209 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Nvme-of container image", 2024-04-21T08:56:45.209 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.210 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.210 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.210 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.210 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_prometheus": { 2024-04-21T08:56:45.210 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_prometheus", 2024-04-21T08:56:45.210 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.210 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.210 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.211 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T08:56:45.211 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.211 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.211 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.211 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:45.211 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.211 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.211 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.212 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_promtail": { 2024-04-21T08:56:45.212 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_promtail", 2024-04-21T08:56:45.212 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.212 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.212 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.212 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T08:56:45.212 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.212 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.213 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.213 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Promtail container image", 2024-04-21T08:56:45.213 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.213 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.213 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.213 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_samba": { 2024-04-21T08:56:45.213 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_samba", 2024-04-21T08:56:45.214 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.214 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.214 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.214 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T08:56:45.214 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.214 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.214 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.214 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Samba/SMB container image", 2024-04-21T08:56:45.215 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.215 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.215 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.215 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_snmp_gateway": { 2024-04-21T08:56:45.215 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_snmp_gateway", 2024-04-21T08:56:45.215 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.215 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.216 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.216 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T08:56:45.216 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.216 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.216 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.216 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SNMP Gateway container image", 2024-04-21T08:56:45.216 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.216 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.217 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.217 INFO:teuthology.orchestra.run.smithi007.stdout: "container_init": { 2024-04-21T08:56:45.217 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_init", 2024-04-21T08:56:45.217 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.217 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.217 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.217 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.218 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.218 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.218 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.218 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T08:56:45.218 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.218 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.218 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.219 INFO:teuthology.orchestra.run.smithi007.stdout: "daemon_cache_timeout": { 2024-04-21T08:56:45.219 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "daemon_cache_timeout", 2024-04-21T08:56:45.219 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.219 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.219 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.219 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:45.219 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.219 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.219 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.220 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T08:56:45.220 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.220 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.220 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.220 INFO:teuthology.orchestra.run.smithi007.stdout: "default_cephadm_command_timeout": { 2024-04-21T08:56:45.220 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T08:56:45.220 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.221 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.221 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.221 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "900", 2024-04-21T08:56:45.221 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.221 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.221 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.221 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T08:56:45.222 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.222 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.222 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.222 INFO:teuthology.orchestra.run.smithi007.stdout: "default_registry": { 2024-04-21T08:56:45.222 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_registry", 2024-04-21T08:56:45.222 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.222 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.223 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.223 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io", 2024-04-21T08:56:45.223 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.223 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.223 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.223 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T08:56:45.223 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.223 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.224 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.224 INFO:teuthology.orchestra.run.smithi007.stdout: "device_cache_timeout": { 2024-04-21T08:56:45.224 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_cache_timeout", 2024-04-21T08:56:45.224 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.224 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.224 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.224 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1800", 2024-04-21T08:56:45.224 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.225 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.225 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.225 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache device inventory", 2024-04-21T08:56:45.225 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.225 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.225 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.225 INFO:teuthology.orchestra.run.smithi007.stdout: "device_enhanced_scan": { 2024-04-21T08:56:45.225 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_enhanced_scan", 2024-04-21T08:56:45.226 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.226 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.226 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.226 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.226 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.226 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.226 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.226 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T08:56:45.227 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.227 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.227 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.227 INFO:teuthology.orchestra.run.smithi007.stdout: "facts_cache_timeout": { 2024-04-21T08:56:45.227 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "facts_cache_timeout", 2024-04-21T08:56:45.227 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.227 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.227 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.228 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:45.228 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.228 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.228 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.228 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache host facts data", 2024-04-21T08:56:45.228 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.228 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.228 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.229 INFO:teuthology.orchestra.run.smithi007.stdout: "grafana_dashboards_path": { 2024-04-21T08:56:45.229 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "grafana_dashboards_path", 2024-04-21T08:56:45.229 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.229 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.229 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.229 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T08:56:45.229 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.229 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.229 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.230 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T08:56:45.230 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.230 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.230 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.230 INFO:teuthology.orchestra.run.smithi007.stdout: "host_check_interval": { 2024-04-21T08:56:45.230 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "host_check_interval", 2024-04-21T08:56:45.230 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.231 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.231 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.231 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:45.231 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.231 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.231 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.231 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to perform a host check", 2024-04-21T08:56:45.231 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.232 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.232 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.232 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.232 INFO:teuthology.orchestra.run.smithi007.stdout: "hw_monitoring": { 2024-04-21T08:56:45.232 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hw_monitoring", 2024-04-21T08:56:45.232 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.232 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.232 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.233 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.233 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.233 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.233 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.233 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T08:56:45.233 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.233 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.233 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.234 INFO:teuthology.orchestra.run.smithi007.stdout: "inventory_list_all": { 2024-04-21T08:56:45.234 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "inventory_list_all", 2024-04-21T08:56:45.234 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.234 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.234 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.234 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.234 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.235 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.235 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.235 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T08:56:45.235 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.235 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.235 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.235 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.235 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.236 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.236 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.236 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.236 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.236 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.236 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.236 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.237 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.237 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.237 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.237 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.237 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.237 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.237 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.237 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.238 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.238 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.238 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.238 INFO:teuthology.orchestra.run.smithi007.stdout: "log_refresh_metadata": { 2024-04-21T08:56:45.238 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_refresh_metadata", 2024-04-21T08:56:45.238 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.238 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.239 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.239 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.239 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.239 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.239 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.239 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T08:56:45.239 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.239 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.240 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.240 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.240 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.240 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.240 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.240 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.240 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.240 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.241 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.241 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.241 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.241 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T08:56:45.241 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.241 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.241 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.241 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.242 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.242 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.242 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.242 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.242 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.242 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.242 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.243 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.243 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.243 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.243 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.243 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.243 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.243 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.243 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.243 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.244 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.244 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.244 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.244 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.244 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.244 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.244 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.244 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.245 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.245 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.245 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.245 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.245 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.245 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.245 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.245 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.245 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.246 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T08:56:45.246 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T08:56:45.246 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.246 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.246 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.246 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.246 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.246 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T08:56:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T08:56:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T08:56:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.248 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.248 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.248 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "*", 2024-04-21T08:56:45.248 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.248 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.248 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.248 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T08:56:45.248 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.248 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.249 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.249 INFO:teuthology.orchestra.run.smithi007.stdout: "max_count_per_host": { 2024-04-21T08:56:45.249 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_count_per_host", 2024-04-21T08:56:45.249 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.249 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.249 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.249 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:45.249 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.250 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.250 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.250 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of daemons per service per host", 2024-04-21T08:56:45.250 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.250 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.250 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.250 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.250 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd_draining_count": { 2024-04-21T08:56:45.250 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_osd_draining_count", 2024-04-21T08:56:45.251 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.251 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.251 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.251 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:45.251 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.251 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.251 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.251 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T08:56:45.252 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.252 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.252 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.252 INFO:teuthology.orchestra.run.smithi007.stdout: "migration_current": { 2024-04-21T08:56:45.252 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "migration_current", 2024-04-21T08:56:45.252 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.252 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.253 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.253 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.253 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.253 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.253 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.253 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "internal - do not modify", 2024-04-21T08:56:45.253 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.253 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.253 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.254 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-04-21T08:56:45.254 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-04-21T08:56:45.254 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.254 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.254 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.254 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "root", 2024-04-21T08:56:45.254 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.255 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.255 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.255 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm-package", 2024-04-21T08:56:45.255 INFO:teuthology.orchestra.run.smithi007.stdout: "root" 2024-04-21T08:56:45.255 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.255 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T08:56:45.255 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.255 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.255 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.256 INFO:teuthology.orchestra.run.smithi007.stdout: "oob_default_addr": { 2024-04-21T08:56:45.256 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "oob_default_addr", 2024-04-21T08:56:45.256 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.256 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.256 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.256 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "169.254.1.1", 2024-04-21T08:56:45.256 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.257 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.257 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.257 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T08:56:45.257 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.257 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.257 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.257 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.257 INFO:teuthology.orchestra.run.smithi007.stdout: "prometheus_alerts_path": { 2024-04-21T08:56:45.257 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus_alerts_path", 2024-04-21T08:56:45.258 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.258 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.258 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.258 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T08:56:45.258 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.258 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.258 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.258 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T08:56:45.259 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.259 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.259 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.259 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_insecure": { 2024-04-21T08:56:45.259 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_insecure", 2024-04-21T08:56:45.259 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.259 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.260 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.260 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.260 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.260 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.260 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.260 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T08:56:45.260 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.261 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.261 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.261 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.261 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_password": { 2024-04-21T08:56:45.261 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_password", 2024-04-21T08:56:45.261 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.261 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.261 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.261 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.262 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.262 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.262 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.262 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T08:56:45.262 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.262 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.262 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.263 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_url": { 2024-04-21T08:56:45.263 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_url", 2024-04-21T08:56:45.263 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.263 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.263 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.263 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.263 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.263 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.263 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.264 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T08:56:45.264 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.264 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.264 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.264 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_username": { 2024-04-21T08:56:45.264 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_username", 2024-04-21T08:56:45.264 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.264 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.265 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.265 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.265 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.265 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.265 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.265 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T08:56:45.265 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.265 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.265 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.266 INFO:teuthology.orchestra.run.smithi007.stdout: "secure_monitoring_stack": { 2024-04-21T08:56:45.266 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "secure_monitoring_stack", 2024-04-21T08:56:45.266 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.266 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.266 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.266 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.266 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.267 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.267 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.267 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T08:56:45.267 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.267 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.267 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.267 INFO:teuthology.orchestra.run.smithi007.stdout: "service_discovery_port": { 2024-04-21T08:56:45.268 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "service_discovery_port", 2024-04-21T08:56:45.268 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.268 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.268 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.268 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8765", 2024-04-21T08:56:45.268 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.268 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.268 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.269 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "cephadm service discovery port", 2024-04-21T08:56:45.269 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.269 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.269 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.269 INFO:teuthology.orchestra.run.smithi007.stdout: "ssh_config_file": { 2024-04-21T08:56:45.269 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssh_config_file", 2024-04-21T08:56:45.269 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.269 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.270 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.270 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.270 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.270 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.270 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.270 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T08:56:45.270 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.270 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "use_agent": { 2024-04-21T08:56:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_agent", 2024-04-21T08:56:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.271 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T08:56:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "use_repo_digest": { 2024-04-21T08:56:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_repo_digest", 2024-04-21T08:56:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.273 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T08:56:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_failed_host_check": { 2024-04-21T08:56:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_failed_host_check", 2024-04-21T08:56:45.274 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T08:56:45.275 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.276 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.276 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.276 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_daemons": { 2024-04-21T08:56:45.276 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_daemons", 2024-04-21T08:56:45.276 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.276 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.276 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.276 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.277 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.277 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.277 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.277 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T08:56:45.277 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.277 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.277 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.278 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_hosts": { 2024-04-21T08:56:45.278 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_hosts", 2024-04-21T08:56:45.278 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.278 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.278 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.278 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.278 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.278 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.278 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.279 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T08:56:45.279 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.279 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.279 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.279 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.279 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.279 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.279 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crash", 2024-04-21T08:56:45.280 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.280 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.280 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.280 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.280 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.280 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.280 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.280 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.280 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.281 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.281 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.281 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.281 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.281 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.281 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.281 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.281 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.281 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.282 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.282 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.282 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.282 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.282 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.282 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.282 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.282 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.283 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.283 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.283 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.283 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.283 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.283 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.283 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.283 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.284 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.284 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.284 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.284 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.284 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.284 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.284 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.284 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.285 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.285 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.285 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.285 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.285 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.285 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.285 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.285 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.286 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.286 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.286 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.286 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.286 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.286 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.286 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.287 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.287 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.287 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.287 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.287 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.287 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.287 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.288 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.288 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.288 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.288 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.288 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.288 INFO:teuthology.orchestra.run.smithi007.stdout: "retain_interval": { 2024-04-21T08:56:45.288 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retain_interval", 2024-04-21T08:56:45.289 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.289 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.289 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.289 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "31536000", 2024-04-21T08:56:45.289 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.289 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.289 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.289 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T08:56:45.290 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.290 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.290 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.290 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.290 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_recent_interval": { 2024-04-21T08:56:45.290 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_recent_interval", 2024-04-21T08:56:45.290 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.290 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.291 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.291 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1209600", 2024-04-21T08:56:45.291 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.291 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.291 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.291 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T08:56:45.291 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.291 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.292 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.292 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.292 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.292 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.292 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dashboard", 2024-04-21T08:56:45.292 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.292 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.293 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.293 INFO:teuthology.orchestra.run.smithi007.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T08:56:45.293 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T08:56:45.293 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.293 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.293 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.293 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:45.293 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.294 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.294 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.294 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.294 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.294 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.294 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.294 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T08:56:45.295 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T08:56:45.295 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.295 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.295 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.295 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.295 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.295 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.295 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.296 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.296 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.296 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.296 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.296 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T08:56:45.296 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T08:56:45.296 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.297 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.297 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.297 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.297 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.297 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.297 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.297 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.297 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.297 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.298 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.298 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_ENABLED": { 2024-04-21T08:56:45.298 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T08:56:45.298 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.298 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.298 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.299 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.299 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.299 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.299 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.299 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.299 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.299 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.299 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.300 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T08:56:45.300 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T08:56:45.300 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.300 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.300 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.300 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.300 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.300 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.300 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.301 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.301 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.301 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.301 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.301 INFO:teuthology.orchestra.run.smithi007.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T08:56:45.301 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T08:56:45.301 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.301 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.302 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.302 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.302 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.302 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.302 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.302 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.302 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.302 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.302 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.303 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T08:56:45.303 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T08:56:45.303 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.303 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.303 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.303 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.303 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.304 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.304 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.304 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.304 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.304 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.304 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.304 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T08:56:45.304 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T08:56:45.305 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.305 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.305 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.305 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.305 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.305 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.305 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.305 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.305 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.306 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.306 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.306 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T08:56:45.306 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T08:56:45.306 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.306 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.306 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.307 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T08:56:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T08:56:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.308 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T08:56:45.309 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T08:56:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.310 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T08:56:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T08:56:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.311 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.312 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T08:56:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T08:56:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.313 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.314 INFO:teuthology.orchestra.run.smithi007.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T08:56:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T08:56:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.315 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T08:56:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T08:56:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.316 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-04-21T08:56:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.317 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T08:56:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T08:56:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.318 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.319 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_URL": { 2024-04-21T08:56:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_URL", 2024-04-21T08:56:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.320 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T08:56:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T08:56:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.321 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-04-21T08:56:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T08:56:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T08:56:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.323 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T08:56:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T08:56:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.324 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.325 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T08:56:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T08:56:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.326 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.327 INFO:teuthology.orchestra.run.smithi007.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T08:56:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T08:56:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.328 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T08:56:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T08:56:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.329 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "[]", 2024-04-21T08:56:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.330 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T08:56:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T08:56:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "{}", 2024-04-21T08:56:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T08:56:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T08:56:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.333 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T08:56:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T08:56:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.334 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T08:56:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T08:56:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.336 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T08:56:45.337 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T08:56:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.338 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T08:56:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T08:56:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.340 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T08:56:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T08:56:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.341 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.342 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.342 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.342 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T08:56:45.342 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T08:56:45.342 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.342 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.342 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.343 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.343 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.343 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.343 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.343 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.343 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.343 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.343 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T08:56:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T08:56:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.344 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T08:56:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T08:56:45.345 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.346 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T08:56:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T08:56:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.347 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.348 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T08:56:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T08:56:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T08:56:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.349 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T08:56:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T08:56:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.350 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.351 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T08:56:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T08:56:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8", 2024-04-21T08:56:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.352 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T08:56:45.353 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T08:56:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "45", 2024-04-21T08:56:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.354 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T08:56:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T08:56:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.355 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.356 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T08:56:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T08:56:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-04-21T08:56:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.357 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T08:56:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T08:56:45.358 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.359 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T08:56:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T08:56:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.360 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T08:56:45.361 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T08:56:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.362 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T08:56:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T08:56:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.363 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.364 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T08:56:45.365 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T08:56:45.365 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.365 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.365 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.365 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:45.365 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.365 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.365 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.366 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.366 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.366 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.366 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.366 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.366 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T08:56:45.366 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T08:56:45.366 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.366 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "cross_origin_url": { 2024-04-21T08:56:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cross_origin_url", 2024-04-21T08:56:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.368 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "crt_file": { 2024-04-21T08:56:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crt_file", 2024-04-21T08:56:45.369 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.370 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "debug": { 2024-04-21T08:56:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "debug", 2024-04-21T08:56:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable/disable debug options", 2024-04-21T08:56:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "jwt_token_ttl": { 2024-04-21T08:56:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "jwt_token_ttl", 2024-04-21T08:56:45.373 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.373 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.373 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.373 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "28800", 2024-04-21T08:56:45.373 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.373 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.373 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.373 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.373 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.374 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.374 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.374 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-04-21T08:56:45.374 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-04-21T08:56:45.374 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.374 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.374 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.374 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.375 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.375 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.375 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.375 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.375 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.375 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.375 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.376 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.376 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.376 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.376 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.376 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.376 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.376 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.376 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.377 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.377 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.377 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.377 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.377 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.377 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.377 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.377 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.378 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.378 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.378 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.378 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.378 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.378 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.378 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.378 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.381 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.381 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.381 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.381 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.381 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.381 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.381 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.381 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.381 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.381 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.382 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.382 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.382 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.382 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.382 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.382 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.382 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.383 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.383 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.383 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.383 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.383 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.383 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.383 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.383 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.383 INFO:teuthology.orchestra.run.smithi007.stdout: "motd": { 2024-04-21T08:56:45.384 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "motd", 2024-04-21T08:56:45.384 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.384 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.384 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.384 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.384 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.384 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.384 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.385 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "The message of the day", 2024-04-21T08:56:45.385 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.385 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.385 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.385 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_resolve_ip_addr": { 2024-04-21T08:56:45.385 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T08:56:45.385 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.385 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.386 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.386 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.386 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.386 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.386 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.386 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.386 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.387 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.387 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.387 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-04-21T08:56:45.387 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-04-21T08:56:45.387 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.387 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.387 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.387 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-04-21T08:56:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.388 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-04-21T08:56:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-04-21T08:56:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8080", 2024-04-21T08:56:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.389 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-04-21T08:56:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-04-21T08:56:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.390 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.391 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.391 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.391 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.391 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.391 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.391 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.391 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.391 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.391 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.392 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl_server_port": { 2024-04-21T08:56:45.392 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl_server_port", 2024-04-21T08:56:45.392 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.392 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.392 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.392 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8443", 2024-04-21T08:56:45.392 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.392 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.393 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.393 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.393 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.393 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.393 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.393 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-04-21T08:56:45.393 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-04-21T08:56:45.393 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.394 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.394 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.394 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "redirect", 2024-04-21T08:56:45.394 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.394 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.394 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.394 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.394 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect" 2024-04-21T08:56:45.394 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.395 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.395 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.395 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.395 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.395 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-04-21T08:56:45.395 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:45.395 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.395 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.396 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.396 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-04-21T08:56:45.396 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-04-21T08:56:45.396 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-04-21T08:56:45.396 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.396 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.396 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.396 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.397 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.397 INFO:teuthology.orchestra.run.smithi007.stdout: "url_prefix": { 2024-04-21T08:56:45.397 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url_prefix", 2024-04-21T08:56:45.397 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.397 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.397 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.397 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.397 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.398 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.398 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.398 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.398 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.398 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.398 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.398 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.398 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.399 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.399 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "devicehealth", 2024-04-21T08:56:45.399 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.399 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.399 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.399 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_monitoring": { 2024-04-21T08:56:45.399 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_monitoring", 2024-04-21T08:56:45.399 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.399 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.399 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.400 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.400 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.400 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.400 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.400 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "monitor device health metrics", 2024-04-21T08:56:45.400 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.400 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.400 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.401 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.401 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.401 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.401 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.401 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.401 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.401 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.401 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.402 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.402 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.402 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.402 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.402 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.402 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.402 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.402 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.402 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.402 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.403 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.403 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.403 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.403 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.403 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.403 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.403 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.403 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.404 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.404 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.404 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.404 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.404 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.404 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.404 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.405 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.405 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.405 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.405 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.405 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.405 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.405 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.405 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.405 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.406 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.406 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.406 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.406 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.406 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.406 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.406 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.407 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.407 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.407 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.407 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.407 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.407 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.407 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.408 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.408 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.408 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.408 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.408 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.408 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.408 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.408 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.408 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.409 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.409 INFO:teuthology.orchestra.run.smithi007.stdout: "mark_out_threshold": { 2024-04-21T08:56:45.409 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mark_out_threshold", 2024-04-21T08:56:45.409 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.409 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.409 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.409 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2419200", 2024-04-21T08:56:45.410 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.410 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.410 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.410 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T08:56:45.410 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.410 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.410 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.411 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": { 2024-04-21T08:56:45.411 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_name", 2024-04-21T08:56:45.411 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.411 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.411 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.411 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "device_health_metrics", 2024-04-21T08:56:45.411 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.411 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.411 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.412 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T08:56:45.412 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.412 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.412 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.412 INFO:teuthology.orchestra.run.smithi007.stdout: "retention_period": { 2024-04-21T08:56:45.412 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retention_period", 2024-04-21T08:56:45.412 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.412 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.413 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.413 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15552000", 2024-04-21T08:56:45.413 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.413 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.413 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.413 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain device health metrics", 2024-04-21T08:56:45.414 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.414 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.414 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.414 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_frequency": { 2024-04-21T08:56:45.414 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_frequency", 2024-04-21T08:56:45.415 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.415 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.415 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.415 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-04-21T08:56:45.415 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.415 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.415 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.415 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T08:56:45.415 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.416 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.416 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.416 INFO:teuthology.orchestra.run.smithi007.stdout: "self_heal": { 2024-04-21T08:56:45.416 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "self_heal", 2024-04-21T08:56:45.416 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.416 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.416 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.417 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.417 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.417 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.417 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.417 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T08:56:45.417 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.417 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.418 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.418 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:45.418 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:45.418 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.418 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.418 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.418 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:45.419 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.419 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.419 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.419 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T08:56:45.419 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.419 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.419 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.420 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_threshold": { 2024-04-21T08:56:45.420 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_threshold", 2024-04-21T08:56:45.420 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.420 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.420 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.420 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7257600", 2024-04-21T08:56:45.420 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.420 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.421 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.421 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T08:56:45.421 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.421 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.421 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.421 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.421 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.422 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.422 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "diskprediction_local", 2024-04-21T08:56:45.422 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.422 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.422 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.422 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.422 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.423 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.423 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.423 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.423 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.423 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.423 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.423 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.423 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.423 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.424 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.424 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.424 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.424 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.424 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.424 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.424 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.424 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.425 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.425 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.425 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.425 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.425 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.425 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.425 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.426 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.426 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.426 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.426 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.426 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.426 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.426 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.427 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.427 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.427 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.427 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.427 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.427 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.427 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.427 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.428 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.428 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.428 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.428 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.428 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.428 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.428 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.428 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.428 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.429 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.429 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.429 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.429 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.429 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.429 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.429 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.430 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.430 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.430 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.430 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.430 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.430 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.430 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.430 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.431 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.431 INFO:teuthology.orchestra.run.smithi007.stdout: "predict_interval": { 2024-04-21T08:56:45.431 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predict_interval", 2024-04-21T08:56:45.431 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.431 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.431 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.431 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-04-21T08:56:45.431 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.431 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.432 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.432 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.432 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.432 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.432 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.432 INFO:teuthology.orchestra.run.smithi007.stdout: "predictor_model": { 2024-04-21T08:56:45.432 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predictor_model", 2024-04-21T08:56:45.432 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.433 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.433 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.433 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "prophetstor", 2024-04-21T08:56:45.433 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.433 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.433 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.433 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.433 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.433 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.434 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.434 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:45.434 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:45.434 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.434 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.434 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.434 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-04-21T08:56:45.435 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.435 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.435 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.435 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.435 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.435 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.435 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.435 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.435 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.436 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.436 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "influx", 2024-04-21T08:56:45.436 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": false, 2024-04-21T08:56:45.436 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "influxdb python module not found", 2024-04-21T08:56:45.436 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.436 INFO:teuthology.orchestra.run.smithi007.stdout: "batch_size": { 2024-04-21T08:56:45.436 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "batch_size", 2024-04-21T08:56:45.436 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.437 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.437 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.437 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5000", 2024-04-21T08:56:45.437 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.437 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.437 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.437 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T08:56:45.438 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.438 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.438 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.438 INFO:teuthology.orchestra.run.smithi007.stdout: "database": { 2024-04-21T08:56:45.438 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "database", 2024-04-21T08:56:45.438 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.438 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.438 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.439 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "ceph", 2024-04-21T08:56:45.439 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.439 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.439 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.439 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T08:56:45.439 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.439 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.439 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.440 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": { 2024-04-21T08:56:45.440 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hostname", 2024-04-21T08:56:45.440 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.440 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.440 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.440 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.440 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.440 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.440 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.441 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server hostname", 2024-04-21T08:56:45.441 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.441 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.441 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.441 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:45.441 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:45.441 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.441 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.441 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.442 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "30", 2024-04-21T08:56:45.442 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "5", 2024-04-21T08:56:45.442 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.442 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.442 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T08:56:45.442 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.442 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.442 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.443 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.443 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.443 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.443 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.443 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.443 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.443 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.443 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.444 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.444 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.444 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.444 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.444 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.444 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.444 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.444 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.444 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.445 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.445 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.445 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.445 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.445 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.445 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.445 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.445 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.446 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.446 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.446 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.446 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.446 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.446 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.446 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.446 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.447 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.447 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.447 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.447 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.447 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.447 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.447 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.447 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.448 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.448 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.448 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.448 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.448 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.448 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.448 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.448 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.448 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.449 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.449 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.449 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.449 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.449 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.449 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.449 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.449 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.450 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.450 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.450 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.450 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.450 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.450 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.450 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.450 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.451 INFO:teuthology.orchestra.run.smithi007.stdout: "password": { 2024-04-21T08:56:45.451 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "password", 2024-04-21T08:56:45.451 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.451 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.451 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.451 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.451 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.451 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.451 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.452 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "password of InfluxDB server user", 2024-04-21T08:56:45.452 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.452 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.452 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.452 INFO:teuthology.orchestra.run.smithi007.stdout: "port": { 2024-04-21T08:56:45.452 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "port", 2024-04-21T08:56:45.452 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.452 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.453 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.453 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8086", 2024-04-21T08:56:45.453 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.453 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.453 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.453 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server port", 2024-04-21T08:56:45.453 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.453 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.453 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.454 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-04-21T08:56:45.454 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-04-21T08:56:45.454 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.454 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.454 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.454 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "false", 2024-04-21T08:56:45.454 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.454 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.455 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.455 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:45.455 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.455 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.455 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.455 INFO:teuthology.orchestra.run.smithi007.stdout: "threads": { 2024-04-21T08:56:45.455 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threads", 2024-04-21T08:56:45.456 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.456 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.456 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.456 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:45.456 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:45.456 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "32", 2024-04-21T08:56:45.456 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.456 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T08:56:45.457 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.457 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.457 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.457 INFO:teuthology.orchestra.run.smithi007.stdout: "username": { 2024-04-21T08:56:45.457 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "username", 2024-04-21T08:56:45.457 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.457 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.457 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.458 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.458 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.458 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.458 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.458 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "username of InfluxDB server user", 2024-04-21T08:56:45.458 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.458 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.458 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.459 INFO:teuthology.orchestra.run.smithi007.stdout: "verify_ssl": { 2024-04-21T08:56:45.459 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "verify_ssl", 2024-04-21T08:56:45.459 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.459 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.459 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.459 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "true", 2024-04-21T08:56:45.459 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.459 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.460 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.460 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:45.460 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.460 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.460 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.460 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.460 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.461 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.461 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "insights", 2024-04-21T08:56:45.461 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.461 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.461 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.461 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.461 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.461 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.462 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.462 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.462 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.462 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.462 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.462 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.462 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.462 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.463 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.463 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.463 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.463 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.463 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.463 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.463 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.463 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.464 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.464 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.464 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.464 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.464 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.464 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.464 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.465 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.465 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.465 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.465 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.465 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.465 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.465 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.465 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.465 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.466 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.466 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.466 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.466 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.466 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.466 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.466 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.466 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.467 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.467 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.467 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.467 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.467 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.467 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.467 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.467 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.467 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.468 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.468 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.468 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.468 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.468 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.468 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.468 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.469 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.469 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.469 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.469 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.469 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.469 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.469 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.469 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.469 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.470 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.470 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.470 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "iostat", 2024-04-21T08:56:45.470 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.470 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.470 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.470 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.471 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.471 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.471 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.471 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.471 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.471 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.471 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.471 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.471 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.472 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.472 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.472 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.472 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.472 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.472 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.472 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.472 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.472 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.473 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.473 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.473 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.473 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.473 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.473 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.473 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.473 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.474 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.474 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.474 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.474 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.474 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.474 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.474 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.475 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.475 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.475 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.475 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.475 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.475 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.475 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.475 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.476 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.476 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.476 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.476 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.476 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.476 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.476 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.476 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.476 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.477 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.477 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.477 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.477 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.477 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.477 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.477 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.478 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.478 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.478 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.478 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.478 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.478 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.478 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.478 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.479 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.479 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.479 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.479 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.479 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "k8sevents", 2024-04-21T08:56:45.479 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.479 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.479 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.479 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_event_retention_days": { 2024-04-21T08:56:45.480 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ceph_event_retention_days", 2024-04-21T08:56:45.480 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.480 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.480 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.480 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7", 2024-04-21T08:56:45.480 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.480 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.480 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.480 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T08:56:45.481 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.481 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.481 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.481 INFO:teuthology.orchestra.run.smithi007.stdout: "config_check_secs": { 2024-04-21T08:56:45.481 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_check_secs", 2024-04-21T08:56:45.481 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.481 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.481 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.482 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:45.482 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "10", 2024-04-21T08:56:45.482 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.482 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.482 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T08:56:45.482 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.482 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.483 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.483 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.483 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.483 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.483 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.483 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.483 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.483 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.484 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.484 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.484 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.484 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.484 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.484 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.484 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.484 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.485 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.485 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.485 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.485 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.485 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.485 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.485 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.485 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.486 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.486 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.486 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.486 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.486 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.486 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.486 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.486 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.487 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.487 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.487 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.487 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.487 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.487 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.487 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.488 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.488 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.488 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.488 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.488 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.488 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.488 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.488 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.488 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.489 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.489 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.489 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.489 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.489 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.489 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.489 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.489 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.490 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.490 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.490 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.490 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.490 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.490 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.490 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.490 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.491 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.491 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.491 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.491 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.491 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.491 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.491 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "localpool", 2024-04-21T08:56:45.492 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.492 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.492 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.492 INFO:teuthology.orchestra.run.smithi007.stdout: "failure_domain": { 2024-04-21T08:56:45.492 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "failure_domain", 2024-04-21T08:56:45.492 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.492 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.492 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "host", 2024-04-21T08:56:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "failure domain for any created local pool", 2024-04-21T08:56:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T08:56:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.493 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.494 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.495 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.496 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.497 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.498 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.498 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.498 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.498 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.498 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.498 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.498 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.499 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.499 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.499 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.499 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.499 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.501 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.501 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.502 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.502 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.502 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.502 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.502 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.502 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.502 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.502 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.503 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.503 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.503 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.503 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.503 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.503 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.503 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.504 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.504 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.504 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.504 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.504 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.504 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.504 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": { 2024-04-21T08:56:45.505 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_size", 2024-04-21T08:56:45.505 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.505 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.505 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.505 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.505 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.505 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.505 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.505 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default min_size for any created local pool", 2024-04-21T08:56:45.506 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T08:56:45.506 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.506 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.506 INFO:teuthology.orchestra.run.smithi007.stdout: "num_rep": { 2024-04-21T08:56:45.506 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "num_rep", 2024-04-21T08:56:45.506 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.506 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.507 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.507 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3", 2024-04-21T08:56:45.507 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.507 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.507 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.507 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default replica count for any created local pool", 2024-04-21T08:56:45.507 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.507 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.508 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.508 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": { 2024-04-21T08:56:45.508 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_num", 2024-04-21T08:56:45.508 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.508 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.508 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.508 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "128", 2024-04-21T08:56:45.509 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.509 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.509 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.509 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T08:56:45.509 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.509 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.509 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.510 INFO:teuthology.orchestra.run.smithi007.stdout: "prefix": { 2024-04-21T08:56:45.510 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prefix", 2024-04-21T08:56:45.510 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.510 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.510 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.510 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.510 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.510 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.511 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.511 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name prefix for any created local pool", 2024-04-21T08:56:45.511 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.511 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.511 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.511 INFO:teuthology.orchestra.run.smithi007.stdout: "subtree": { 2024-04-21T08:56:45.512 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "subtree", 2024-04-21T08:56:45.512 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.512 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.512 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.512 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "rack", 2024-04-21T08:56:45.512 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.512 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.512 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.513 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T08:56:45.513 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T08:56:45.513 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.513 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.513 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.513 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.513 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.513 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mds_autoscaler", 2024-04-21T08:56:45.514 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.514 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.514 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.514 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.514 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.514 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.514 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.515 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.515 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.515 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.515 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.515 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.515 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.515 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.515 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.516 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.516 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.516 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.516 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.516 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.516 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.516 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.516 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.517 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.517 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.517 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.517 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.517 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.517 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.517 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.517 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.518 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.518 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.518 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.518 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.518 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.518 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.518 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.518 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.519 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.519 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.519 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.519 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.519 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.519 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.519 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.519 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.519 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.520 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.520 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.520 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.520 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.520 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.520 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.520 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.520 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.520 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.521 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.521 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.521 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.521 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.521 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.521 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.521 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.521 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.521 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.522 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.522 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.522 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.522 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.522 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.522 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.522 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.522 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirroring", 2024-04-21T08:56:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.524 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.524 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.524 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.524 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.524 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.524 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.524 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.524 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.524 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.525 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.525 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.525 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.525 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.525 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.525 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.525 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.525 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.526 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.526 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.526 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.526 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.526 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.526 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.526 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.526 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.527 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.527 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.527 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.527 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.527 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.527 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.527 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.527 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.527 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.528 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.528 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.528 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.528 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.528 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.528 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.528 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.528 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.528 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.528 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.529 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.529 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.529 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.529 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.529 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.529 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.529 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.529 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.530 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.530 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.530 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.530 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.530 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.530 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.530 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.530 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.531 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.531 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.531 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.531 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.531 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "nfs", 2024-04-21T08:56:45.531 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.531 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.531 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.532 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.532 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.532 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.532 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.532 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.532 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.532 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.532 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.532 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.533 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.533 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.533 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.533 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.533 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.533 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.533 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.533 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.533 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.533 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.534 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.534 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.534 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.534 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.534 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.534 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.534 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.534 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.535 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.535 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.535 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.535 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.535 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.535 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.535 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.535 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.536 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.536 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.536 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.536 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.536 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.536 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.536 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.536 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.536 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.537 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.537 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.537 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.537 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.537 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.537 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.537 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.537 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.538 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.538 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.538 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.538 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.538 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.538 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.538 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.539 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.539 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.539 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.539 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.539 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.539 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.539 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.539 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.539 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.540 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.540 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-04-21T08:56:45.540 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.540 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.540 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.540 INFO:teuthology.orchestra.run.smithi007.stdout: "fail_fs": { 2024-04-21T08:56:45.540 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "fail_fs", 2024-04-21T08:56:45.540 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.541 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.541 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.541 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.541 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.541 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.541 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.541 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T08:56:45.541 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.541 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.542 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.542 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.542 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.542 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.542 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.542 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.542 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.542 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.542 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.543 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.543 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.543 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.543 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.543 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.543 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.543 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.543 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.543 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.543 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.544 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.544 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.544 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.544 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.544 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.544 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.544 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.544 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.545 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.545 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.545 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.545 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.545 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.545 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.545 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.545 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.546 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.546 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.546 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.546 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.546 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.546 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.546 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.546 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.547 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.547 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.547 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.547 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.547 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.547 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.547 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.547 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.547 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.548 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.548 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.548 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.548 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.548 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.548 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.548 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.549 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.549 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.549 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.549 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.549 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.549 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.549 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.549 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.549 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.550 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator": { 2024-04-21T08:56:45.550 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-04-21T08:56:45.550 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.550 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.550 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.550 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.550 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.550 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.551 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.551 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm", 2024-04-21T08:56:45.551 INFO:teuthology.orchestra.run.smithi007.stdout: "rook", 2024-04-21T08:56:45.551 INFO:teuthology.orchestra.run.smithi007.stdout: "test_orchestrator" 2024-04-21T08:56:45.551 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.551 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Orchestrator backend", 2024-04-21T08:56:45.551 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.551 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.551 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.552 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.552 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.552 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.552 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_perf_query", 2024-04-21T08:56:45.552 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.552 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.552 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.552 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.553 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.553 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.553 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.553 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.553 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.553 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.553 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.553 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.554 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.554 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.554 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.554 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.554 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.554 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.554 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.554 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.555 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.555 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.555 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.555 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.555 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.555 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.555 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.555 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.555 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.556 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.556 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.556 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.556 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.556 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.556 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.556 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.557 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.557 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.557 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.557 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.557 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.557 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.557 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.557 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.558 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.558 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.558 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.558 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.558 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.558 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.558 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.558 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.558 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.559 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.559 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.559 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.559 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.559 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.559 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.559 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.559 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.560 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.560 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.560 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.560 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.560 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.560 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.560 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.560 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.560 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.561 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.561 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_support", 2024-04-21T08:56:45.561 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.561 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.561 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.561 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.561 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.561 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.562 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.562 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.562 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.562 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.562 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.562 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.562 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.562 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.562 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.563 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.563 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.563 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.563 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.563 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.563 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.563 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.563 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.564 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.564 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.564 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.564 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.564 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.564 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.564 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.564 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.565 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.565 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.565 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.565 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.565 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.565 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.565 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.565 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.566 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.566 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.566 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.566 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.566 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.566 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.566 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.566 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.566 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.567 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.567 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.567 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.567 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.567 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.567 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.567 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.567 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.568 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.568 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.568 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.568 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.568 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.568 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.568 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.568 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.568 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.569 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.569 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.569 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.569 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.569 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.569 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.569 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.569 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.569 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.569 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_autoscaler", 2024-04-21T08:56:45.570 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.570 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.570 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.570 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.570 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.570 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.570 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.570 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.570 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.571 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.571 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.571 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.571 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.571 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.571 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.571 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.571 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.571 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.572 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.572 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.572 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.572 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.572 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.572 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.572 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.573 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.573 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.573 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.573 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.573 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.573 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.573 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.573 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.573 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.574 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.574 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.574 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.574 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.574 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.574 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.574 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.575 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.575 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.575 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.575 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.575 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.575 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.575 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.575 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.575 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.576 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.576 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.576 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.576 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.576 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.576 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.576 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.577 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.577 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.577 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.577 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.577 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.577 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.577 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.577 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.578 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.578 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.578 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.578 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.578 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:45.578 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:45.578 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.578 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.579 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.579 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:45.579 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.579 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.579 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.579 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.579 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.579 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.579 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.580 INFO:teuthology.orchestra.run.smithi007.stdout: "threshold": { 2024-04-21T08:56:45.580 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threshold", 2024-04-21T08:56:45.580 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:45.580 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.580 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.580 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-04-21T08:56:45.580 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1.0", 2024-04-21T08:56:45.580 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.581 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.581 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "scaling threshold", 2024-04-21T08:56:45.581 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T08:56:45.581 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.581 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.581 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.581 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.582 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "progress", 2024-04-21T08:56:45.582 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.582 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.582 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.582 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_pg_recovery_event": { 2024-04-21T08:56:45.582 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_pg_recovery_event", 2024-04-21T08:56:45.582 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.582 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.583 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.583 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.583 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.583 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.583 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.583 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T08:56:45.583 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.583 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.583 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.584 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-04-21T08:56:45.584 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-04-21T08:56:45.584 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.584 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.584 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.584 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.584 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.585 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.585 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.585 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.585 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.585 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.585 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.585 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.587 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.587 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.587 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.587 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.587 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.587 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.587 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.587 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.588 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.588 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.588 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.588 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.588 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.588 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.588 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.589 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.590 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.591 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.592 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.593 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "max_completed_events": { 2024-04-21T08:56:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_completed_events", 2024-04-21T08:56:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.594 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "50", 2024-04-21T08:56:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "number of past completed events to remember", 2024-04-21T08:56:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-04-21T08:56:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-04-21T08:56:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-04-21T08:56:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.596 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.597 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long the module is going to sleep", 2024-04-21T08:56:45.597 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.597 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.597 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.598 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus", 2024-04-21T08:56:45.598 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.598 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.598 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.598 INFO:teuthology.orchestra.run.smithi007.stdout: "cache": { 2024-04-21T08:56:45.598 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cache", 2024-04-21T08:56:45.598 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.598 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.599 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.599 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.599 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.599 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.599 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.599 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.599 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.599 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.599 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.600 INFO:teuthology.orchestra.run.smithi007.stdout: "exclude_perf_counters": { 2024-04-21T08:56:45.600 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "exclude_perf_counters", 2024-04-21T08:56:45.600 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.600 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.600 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.600 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.600 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.601 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.601 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.601 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T08:56:45.601 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T08:56:45.601 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.601 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.601 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.601 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.602 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.603 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.606 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.607 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.608 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.609 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.610 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools": { 2024-04-21T08:56:45.610 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools", 2024-04-21T08:56:45.610 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.610 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.610 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.610 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.610 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.611 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.611 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.611 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.611 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.611 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.611 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.611 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T08:56:45.612 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T08:56:45.612 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.612 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.612 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.612 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "300", 2024-04-21T08:56:45.612 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.612 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.612 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.612 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.613 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.613 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.613 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.613 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_interval": { 2024-04-21T08:56:45.613 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_interval", 2024-04-21T08:56:45.613 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:45.613 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.614 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.614 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15.0", 2024-04-21T08:56:45.614 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.614 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.614 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.614 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.614 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.614 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.615 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.615 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-04-21T08:56:45.615 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-04-21T08:56:45.615 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.615 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.615 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.615 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-04-21T08:56:45.616 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.616 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.616 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.616 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T08:56:45.616 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.616 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.616 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.617 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-04-21T08:56:45.617 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-04-21T08:56:45.617 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.617 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.617 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.617 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "9283", 2024-04-21T08:56:45.617 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.617 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.618 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.618 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T08:56:45.618 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.618 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.618 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.618 INFO:teuthology.orchestra.run.smithi007.stdout: "stale_cache_strategy": { 2024-04-21T08:56:45.618 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stale_cache_strategy", 2024-04-21T08:56:45.619 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.619 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.619 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.619 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "log", 2024-04-21T08:56:45.619 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.619 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.619 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.619 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-04-21T08:56:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-04-21T08:56:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.621 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "default", 2024-04-21T08:56:45.621 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.621 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.621 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.621 INFO:teuthology.orchestra.run.smithi007.stdout: "default", 2024-04-21T08:56:45.621 INFO:teuthology.orchestra.run.smithi007.stdout: "error" 2024-04-21T08:56:45.621 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.622 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.622 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.622 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.622 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.622 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-04-21T08:56:45.622 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:45.622 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.622 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.623 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.623 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-04-21T08:56:45.623 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-04-21T08:56:45.623 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-04-21T08:56:45.623 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.623 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.623 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.623 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.624 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.624 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.624 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.624 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.624 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_support", 2024-04-21T08:56:45.624 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.624 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.624 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.625 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.625 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.625 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.625 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.625 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.625 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.625 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.625 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.625 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.626 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.626 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.626 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.626 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.626 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.626 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.626 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.626 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.627 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.627 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.627 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.627 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.627 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.627 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.627 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.627 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.628 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.628 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.628 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.628 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.628 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.628 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.628 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.628 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.628 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.629 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.629 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.629 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.629 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.629 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.629 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.629 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.629 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.630 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.630 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.630 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.630 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.630 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.630 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.630 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.630 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.630 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.631 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.631 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.631 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.631 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.631 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.631 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.631 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.631 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.632 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.632 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.632 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.632 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.632 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.632 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.632 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.632 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_snap_create": { 2024-04-21T08:56:45.632 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_snap_create", 2024-04-21T08:56:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-04-21T08:56:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.633 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "mirror_snapshot_schedule": { 2024-04-21T08:56:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T08:56:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.634 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.635 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "trash_purge_schedule": { 2024-04-21T08:56:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "trash_purge_schedule", 2024-04-21T08:56:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.636 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.638 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "restful", 2024-04-21T08:56:45.638 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.638 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.638 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.638 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_auth": { 2024-04-21T08:56:45.638 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_auth", 2024-04-21T08:56:45.638 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.638 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.639 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.639 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.639 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.639 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.639 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.639 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.639 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.639 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.639 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.640 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-04-21T08:56:45.640 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-04-21T08:56:45.640 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.640 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.640 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.640 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.640 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.641 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.641 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.641 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.641 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.641 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.641 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.641 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.641 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.642 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.642 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.642 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.642 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.642 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.642 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.642 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.642 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.643 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.643 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.643 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.643 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.643 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.643 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.643 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.643 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.644 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.644 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.644 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.644 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.644 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.644 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.644 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.645 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.645 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.645 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.645 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.645 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.645 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.645 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.645 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.646 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.646 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.646 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.646 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.646 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.646 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.646 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.647 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.647 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.647 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.647 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.647 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.647 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.647 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.647 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.648 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.648 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.648 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.648 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.648 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.648 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.649 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.649 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.649 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.649 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.649 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.649 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.649 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.649 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.649 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.650 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.650 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.650 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.650 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.650 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-04-21T08:56:45.650 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-04-21T08:56:45.650 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.650 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.651 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.651 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.651 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.651 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.651 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.651 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.651 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.651 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.652 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.652 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.652 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-04-21T08:56:45.652 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-04-21T08:56:45.652 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.652 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.652 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.652 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.652 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.653 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.653 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.653 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.653 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.653 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.653 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.653 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.653 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.654 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.654 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rgw", 2024-04-21T08:56:45.654 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.654 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.654 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.654 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.654 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.654 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.654 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.655 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.655 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.655 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.655 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.655 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.655 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.655 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.655 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.655 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.656 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.656 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.656 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.656 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.656 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.656 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.656 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.656 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.657 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.657 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.657 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.657 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.657 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.657 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.657 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.657 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.658 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.658 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.658 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.658 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.658 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.658 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.658 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.658 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.659 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.659 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.659 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.659 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.659 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.659 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.659 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.659 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.660 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.660 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.660 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.660 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.660 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.660 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.660 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.660 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.661 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.661 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.661 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.661 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.661 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.661 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.661 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.661 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.661 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.662 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.662 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.662 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.662 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.662 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.662 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.662 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.662 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.663 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rook", 2024-04-21T08:56:45.663 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.663 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.663 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.663 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.663 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.663 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.663 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.663 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.664 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.664 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.664 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.664 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.664 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.664 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.664 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.664 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.665 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.665 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.665 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.665 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.665 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.665 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.665 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.665 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.666 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.666 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.666 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.666 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.666 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.666 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.666 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.666 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.667 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.667 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.667 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.667 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.667 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.667 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.667 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.667 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.668 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.668 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.668 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.668 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.668 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.668 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.668 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.668 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.669 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.669 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.669 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.669 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.669 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.669 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.669 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.669 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.670 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.670 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.670 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.670 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.670 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.670 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.670 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.670 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.671 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.671 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.671 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.671 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.671 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.671 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_class": { 2024-04-21T08:56:45.671 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "storage_class", 2024-04-21T08:56:45.672 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.672 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.672 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.672 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "local", 2024-04-21T08:56:45.672 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.672 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.672 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.672 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T08:56:45.672 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.673 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.673 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.673 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.673 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.673 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.673 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "selftest", 2024-04-21T08:56:45.673 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.673 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.674 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.674 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.674 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.674 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.674 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.674 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.674 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.674 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.675 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.675 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.675 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.675 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.675 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.675 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.675 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.675 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.676 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.676 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.676 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.676 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.676 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.676 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.676 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.676 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.677 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.677 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.677 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.677 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.677 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.677 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.677 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.677 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.678 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.678 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.678 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.678 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.678 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.678 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.678 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.679 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.681 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.682 INFO:teuthology.orchestra.run.smithi007.stdout: "roption1": { 2024-04-21T08:56:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption1", 2024-04-21T08:56:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.683 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.684 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.684 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.684 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.684 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.684 INFO:teuthology.orchestra.run.smithi007.stdout: "roption2": { 2024-04-21T08:56:45.684 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption2", 2024-04-21T08:56:45.684 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.685 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.685 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.685 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "xyz", 2024-04-21T08:56:45.685 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.685 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.685 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.685 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.685 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.686 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.686 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.686 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption1": { 2024-04-21T08:56:45.686 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption1", 2024-04-21T08:56:45.686 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.686 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.686 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption2": { 2024-04-21T08:56:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption2", 2024-04-21T08:56:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.689 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.689 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.689 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.689 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.689 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.689 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption3": { 2024-04-21T08:56:45.689 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption3", 2024-04-21T08:56:45.689 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-04-21T08:56:45.690 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.690 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.690 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.690 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.690 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.690 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.690 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.690 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.691 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.691 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.691 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.691 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption4": { 2024-04-21T08:56:45.691 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption4", 2024-04-21T08:56:45.691 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.691 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.691 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.691 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.692 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.692 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.692 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.692 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.692 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.692 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.692 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.693 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption5": { 2024-04-21T08:56:45.693 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption5", 2024-04-21T08:56:45.693 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.693 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.693 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.693 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.693 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.693 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.694 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.694 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.694 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.694 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.694 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.694 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption6": { 2024-04-21T08:56:45.694 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption6", 2024-04-21T08:56:45.695 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.695 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.695 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.695 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.695 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.695 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.695 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.695 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.695 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.696 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.696 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.696 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption7": { 2024-04-21T08:56:45.696 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption7", 2024-04-21T08:56:45.696 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.696 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.696 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.697 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.697 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-04-21T08:56:45.697 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "42", 2024-04-21T08:56:45.697 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.697 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.697 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.697 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.697 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.698 INFO:teuthology.orchestra.run.smithi007.stdout: "testkey": { 2024-04-21T08:56:45.698 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testkey", 2024-04-21T08:56:45.698 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.698 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.698 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.698 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.698 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.699 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.699 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.699 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.699 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.699 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.699 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.699 INFO:teuthology.orchestra.run.smithi007.stdout: "testlkey": { 2024-04-21T08:56:45.699 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testlkey", 2024-04-21T08:56:45.700 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.700 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.700 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.700 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.700 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.700 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.700 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.700 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.700 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.701 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.701 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.701 INFO:teuthology.orchestra.run.smithi007.stdout: "testnewline": { 2024-04-21T08:56:45.701 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testnewline", 2024-04-21T08:56:45.701 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.701 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.701 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.702 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.702 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.702 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.702 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.702 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.702 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.702 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.702 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.703 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.703 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.703 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.703 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snap_schedule", 2024-04-21T08:56:45.703 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.703 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.703 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.703 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_m_granularity": { 2024-04-21T08:56:45.704 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_m_granularity", 2024-04-21T08:56:45.704 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.704 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.704 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.704 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.704 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.704 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.704 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.705 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T08:56:45.705 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.705 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.705 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.705 INFO:teuthology.orchestra.run.smithi007.stdout: "dump_on_update": { 2024-04-21T08:56:45.705 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dump_on_update", 2024-04-21T08:56:45.705 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.705 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.706 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.706 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.706 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.706 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.706 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.706 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "dump database to debug log on update", 2024-04-21T08:56:45.706 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.706 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.707 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.707 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.707 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.707 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.707 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.707 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.707 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.708 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.708 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.708 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.708 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.708 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.708 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.708 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.708 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.709 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.709 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.709 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.709 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.709 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.709 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.709 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.710 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.710 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.710 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.710 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.710 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.710 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.710 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.710 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.710 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.711 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.711 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.711 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.711 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.711 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.711 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.711 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.711 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.712 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.712 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.712 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.712 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.712 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.712 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.712 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.712 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.713 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.713 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.713 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.713 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.713 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.713 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.713 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.713 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.714 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.714 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.714 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.714 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.714 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.714 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.714 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.714 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.715 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.715 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.715 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.715 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.715 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.715 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.715 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.715 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.716 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.716 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.716 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stats", 2024-04-21T08:56:45.716 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.716 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.716 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.716 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.717 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.717 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.717 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.717 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.717 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.717 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.717 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.717 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.717 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.718 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.718 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.718 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.718 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.718 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.718 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.718 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.718 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.719 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.719 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.719 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.719 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.719 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.719 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.719 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.720 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.720 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.720 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.720 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.720 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.720 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.720 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.720 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.721 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.721 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.721 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.721 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.721 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.721 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.721 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.722 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.722 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.722 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.722 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.722 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.722 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.722 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.722 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.723 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.724 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.725 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "status", 2024-04-21T08:56:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.726 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.727 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.728 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.729 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.729 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.729 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.729 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.729 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.729 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.729 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.729 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.730 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.730 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.730 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.730 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.730 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.730 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.730 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.731 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.731 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.731 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.731 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.731 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.731 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.731 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.731 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.732 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.732 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.732 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.732 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.732 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.732 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.732 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.732 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.733 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.733 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.733 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.733 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.733 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.733 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.733 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.733 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.734 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.735 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.735 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.735 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.735 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telegraf", 2024-04-21T08:56:45.735 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.735 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.735 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.735 INFO:teuthology.orchestra.run.smithi007.stdout: "address": { 2024-04-21T08:56:45.736 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "address", 2024-04-21T08:56:45.736 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.736 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.736 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.736 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T08:56:45.736 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.736 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.736 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.737 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.737 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.737 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.737 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.737 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:45.737 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:45.737 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.738 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.738 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.738 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15", 2024-04-21T08:56:45.738 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.738 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.738 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.738 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.738 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.739 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.739 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.739 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.739 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.739 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.739 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.739 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.740 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.740 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.740 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.740 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.740 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.740 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.740 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.740 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.740 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.741 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.741 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.741 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.741 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.741 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.741 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.741 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.741 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.742 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.742 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.742 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.742 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.742 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.742 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.742 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.743 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.743 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.743 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.743 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.743 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.743 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.743 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.743 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.744 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.744 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.744 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.744 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.744 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.744 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.744 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.744 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.745 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.745 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.745 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.745 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.745 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.745 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.745 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.745 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.746 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.746 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.746 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.746 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.746 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.746 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.746 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.747 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.747 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.747 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.747 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.747 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.747 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.747 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.747 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.748 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.748 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telemetry", 2024-04-21T08:56:45.748 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.748 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.748 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.748 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_basic": { 2024-04-21T08:56:45.748 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_basic", 2024-04-21T08:56:45.749 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.749 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.749 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.749 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.749 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.749 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.749 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.749 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T08:56:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_crash": { 2024-04-21T08:56:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_crash", 2024-04-21T08:56:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.750 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.751 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.751 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.751 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.751 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.751 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.751 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T08:56:45.751 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.751 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.752 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.752 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_device": { 2024-04-21T08:56:45.752 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_device", 2024-04-21T08:56:45.752 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.752 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.752 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.752 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.753 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.753 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.753 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.753 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T08:56:45.753 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.753 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.753 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.753 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_ident": { 2024-04-21T08:56:45.754 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_ident", 2024-04-21T08:56:45.754 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.754 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.754 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.754 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.754 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.754 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.754 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.755 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T08:56:45.755 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.755 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.755 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.755 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_perf": { 2024-04-21T08:56:45.755 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_perf", 2024-04-21T08:56:45.755 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.756 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.756 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.756 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.756 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.756 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.756 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.756 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T08:56:45.756 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.756 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.757 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.757 INFO:teuthology.orchestra.run.smithi007.stdout: "contact": { 2024-04-21T08:56:45.757 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "contact", 2024-04-21T08:56:45.757 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.757 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.757 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.757 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.758 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.758 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.758 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.758 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.758 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.758 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.758 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.759 INFO:teuthology.orchestra.run.smithi007.stdout: "description": { 2024-04-21T08:56:45.759 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "description", 2024-04-21T08:56:45.759 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.759 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.759 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.759 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.759 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.759 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.760 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.760 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.760 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.760 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.760 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.760 INFO:teuthology.orchestra.run.smithi007.stdout: "device_url": { 2024-04-21T08:56:45.760 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_url", 2024-04-21T08:56:45.760 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.761 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.761 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.761 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T08:56:45.761 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.761 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.761 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.761 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.761 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.762 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.762 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.762 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-04-21T08:56:45.762 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-04-21T08:56:45.762 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.762 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.762 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.763 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.763 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.763 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.763 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.763 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.763 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.763 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.763 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.764 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.764 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:45.764 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:45.764 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.764 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.764 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.764 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "24", 2024-04-21T08:56:45.764 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "8", 2024-04-21T08:56:45.765 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.765 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.765 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.765 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.765 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.765 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.765 INFO:teuthology.orchestra.run.smithi007.stdout: "last_opt_revision": { 2024-04-21T08:56:45.765 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "last_opt_revision", 2024-04-21T08:56:45.766 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.766 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.766 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.766 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1", 2024-04-21T08:56:45.766 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.766 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.766 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.766 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.767 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.767 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.767 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.767 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard": { 2024-04-21T08:56:45.767 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard", 2024-04-21T08:56:45.767 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.767 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.768 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.768 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.768 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.768 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.768 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.768 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.768 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.768 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.769 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.769 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard_description": { 2024-04-21T08:56:45.769 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard_description", 2024-04-21T08:56:45.769 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.769 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.769 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.769 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.769 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.770 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.770 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.770 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.770 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.770 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.770 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.770 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.770 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.771 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.771 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.771 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.771 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.771 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.771 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.771 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.771 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.772 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.772 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.772 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.772 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.772 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.772 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.772 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.772 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.772 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.773 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.773 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.773 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.773 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.773 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.773 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.773 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.773 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.773 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.774 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.774 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.774 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.774 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.774 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.774 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.774 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.775 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.775 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.775 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.775 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.775 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.775 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.775 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.775 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.775 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.776 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.776 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.776 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.776 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.776 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.776 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.776 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.776 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.777 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.777 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.777 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.777 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.777 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.777 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.777 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.778 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "organization": { 2024-04-21T08:56:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "organization", 2024-04-21T08:56:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.779 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.780 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.780 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.780 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.780 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.780 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.780 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.780 INFO:teuthology.orchestra.run.smithi007.stdout: "proxy": { 2024-04-21T08:56:45.781 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "proxy", 2024-04-21T08:56:45.781 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.781 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.781 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.781 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.781 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.781 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.781 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.781 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.782 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.782 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.782 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.782 INFO:teuthology.orchestra.run.smithi007.stdout: "url": { 2024-04-21T08:56:45.782 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url", 2024-04-21T08:56:45.782 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.782 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.783 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.783 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T08:56:45.783 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.783 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.783 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.783 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.783 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.783 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.784 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.784 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.784 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.784 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.784 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.784 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test_orchestrator", 2024-04-21T08:56:45.784 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.784 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.784 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.785 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.785 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.785 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.785 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.785 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.785 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.785 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.785 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.786 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.786 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.786 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.786 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.786 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.786 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.786 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.786 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.787 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.787 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.787 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.787 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.787 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.787 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.787 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.788 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.788 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.788 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.788 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.788 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.788 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.788 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.788 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.788 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.789 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.789 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.789 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.789 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.789 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.789 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.789 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.790 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.790 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.790 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.790 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.790 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.790 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.790 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.790 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.791 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.791 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.791 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.791 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.791 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.791 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.791 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.792 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.792 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.792 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.792 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.792 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.792 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.792 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.792 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.792 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.793 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.793 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.793 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.793 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.793 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "volumes", 2024-04-21T08:56:45.794 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.794 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.794 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.794 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.794 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.794 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.794 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.795 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.795 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.795 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.795 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.795 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.795 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.795 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.795 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.795 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.796 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.796 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.796 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.796 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.796 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.796 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.796 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.796 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.797 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.797 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.797 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.797 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.797 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.797 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.797 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.797 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.798 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.798 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.798 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.798 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.798 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.798 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.798 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.799 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.799 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.799 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.799 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.799 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.799 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.799 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.799 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.799 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.800 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.800 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.800 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.800 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.800 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.800 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.800 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.800 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.801 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.801 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.801 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.801 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.801 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.801 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.801 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.801 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.802 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.802 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.802 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.802 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.802 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.802 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_clones": { 2024-04-21T08:56:45.802 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_clones", 2024-04-21T08:56:45.803 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.803 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.803 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.803 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4", 2024-04-21T08:56:45.803 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.803 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.803 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.803 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T08:56:45.804 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.804 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.804 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.804 INFO:teuthology.orchestra.run.smithi007.stdout: "periodic_async_work": { 2024-04-21T08:56:45.804 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "periodic_async_work", 2024-04-21T08:56:45.804 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.804 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.804 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.805 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.805 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.805 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.805 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.805 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Periodically check for async work", 2024-04-21T08:56:45.805 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.805 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.805 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.805 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.806 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_delay": { 2024-04-21T08:56:45.806 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_delay", 2024-04-21T08:56:45.806 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.806 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.806 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.806 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-04-21T08:56:45.806 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.807 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.807 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.807 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T08:56:45.807 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.807 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.807 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.807 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_no_wait": { 2024-04-21T08:56:45.808 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T08:56:45.808 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.808 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.808 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.808 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-04-21T08:56:45.808 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.808 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.808 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.809 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T08:56:45.809 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.809 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.809 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.809 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.809 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.809 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.810 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix", 2024-04-21T08:56:45.810 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-04-21T08:56:45.810 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-04-21T08:56:45.810 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-04-21T08:56:45.810 INFO:teuthology.orchestra.run.smithi007.stdout: "discovery_interval": { 2024-04-21T08:56:45.810 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "discovery_interval", 2024-04-21T08:56:45.810 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-04-21T08:56:45.810 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.811 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.811 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "100", 2024-04-21T08:56:45.811 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.811 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.811 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.811 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.811 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.812 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.812 INFO:teuthology.orchestra.run.smithi007.stdout: "identifier": { 2024-04-21T08:56:45.812 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "identifier", 2024-04-21T08:56:45.812 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.812 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.812 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.812 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.813 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.813 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.813 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.813 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.813 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.813 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.813 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.813 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-04-21T08:56:45.814 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-04-21T08:56:45.814 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-04-21T08:56:45.814 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.814 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.814 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-04-21T08:56:45.814 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.814 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.815 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.815 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.815 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.815 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.815 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.815 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-04-21T08:56:45.815 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-04-21T08:56:45.815 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.816 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.816 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.816 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.816 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.816 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.816 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.816 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.817 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.817 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.817 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.817 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.817 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.817 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.817 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.817 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.817 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.818 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.818 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-04-21T08:56:45.818 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-04-21T08:56:45.818 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.818 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.818 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.818 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.819 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.819 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.819 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.819 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.819 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.819 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.819 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.820 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-04-21T08:56:45.820 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:45.820 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.820 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.820 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.820 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-04-21T08:56:45.820 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.820 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.821 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-04-21T08:56:45.821 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-04-21T08:56:45.821 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-04-21T08:56:45.821 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-04-21T08:56:45.821 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-04-21T08:56:45.821 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-04-21T08:56:45.821 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-04-21T08:56:45.821 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.821 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.822 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.822 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.822 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.822 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-04-21T08:56:45.822 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-04-21T08:56:45.822 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-04-21T08:56:45.822 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.823 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-04-21T08:56:45.823 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-04-21T08:56:45.823 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.823 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.823 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.823 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.823 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.823 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.824 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.824 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_host": { 2024-04-21T08:56:45.824 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_host", 2024-04-21T08:56:45.824 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.824 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.824 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.824 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-04-21T08:56:45.824 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.825 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.825 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.825 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.825 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.825 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.825 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.825 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.825 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_port": { 2024-04-21T08:56:45.826 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_port", 2024-04-21T08:56:45.826 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-04-21T08:56:45.826 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.826 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.826 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10051", 2024-04-21T08:56:45.826 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.826 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.826 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.827 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.827 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.827 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.827 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.827 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_sender": { 2024-04-21T08:56:45.827 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_sender", 2024-04-21T08:56:45.827 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-04-21T08:56:45.827 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-04-21T08:56:45.828 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-04-21T08:56:45.828 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T08:56:45.828 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-04-21T08:56:45.828 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-04-21T08:56:45.828 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-04-21T08:56:45.828 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-04-21T08:56:45.828 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-04-21T08:56:45.828 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-21T08:56:45.829 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-04-21T08:56:45.829 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.829 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.829 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.829 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.829 INFO:teuthology.orchestra.run.smithi007.stdout: "services": { 2024-04-21T08:56:45.829 INFO:teuthology.orchestra.run.smithi007.stdout: "dashboard": "https://172.21.15.7:8443/" 2024-04-21T08:56:45.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.829 INFO:teuthology.orchestra.run.smithi007.stdout: "always_on_modules": { 2024-04-21T08:56:45.830 INFO:teuthology.orchestra.run.smithi007.stdout: "octopus": [ 2024-04-21T08:56:45.830 INFO:teuthology.orchestra.run.smithi007.stdout: "balancer", 2024-04-21T08:56:45.830 INFO:teuthology.orchestra.run.smithi007.stdout: "crash", 2024-04-21T08:56:45.830 INFO:teuthology.orchestra.run.smithi007.stdout: "devicehealth", 2024-04-21T08:56:45.830 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator", 2024-04-21T08:56:45.830 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscaler", 2024-04-21T08:56:45.830 INFO:teuthology.orchestra.run.smithi007.stdout: "progress", 2024-04-21T08:56:45.831 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_support", 2024-04-21T08:56:45.831 INFO:teuthology.orchestra.run.smithi007.stdout: "status", 2024-04-21T08:56:45.831 INFO:teuthology.orchestra.run.smithi007.stdout: "telemetry", 2024-04-21T08:56:45.831 INFO:teuthology.orchestra.run.smithi007.stdout: "volumes" 2024-04-21T08:56:45.831 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.831 INFO:teuthology.orchestra.run.smithi007.stdout: "pacific": [ 2024-04-21T08:56:45.831 INFO:teuthology.orchestra.run.smithi007.stdout: "balancer", 2024-04-21T08:56:45.831 INFO:teuthology.orchestra.run.smithi007.stdout: "crash", 2024-04-21T08:56:45.831 INFO:teuthology.orchestra.run.smithi007.stdout: "devicehealth", 2024-04-21T08:56:45.832 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator", 2024-04-21T08:56:45.832 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscaler", 2024-04-21T08:56:45.832 INFO:teuthology.orchestra.run.smithi007.stdout: "progress", 2024-04-21T08:56:45.832 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_support", 2024-04-21T08:56:45.832 INFO:teuthology.orchestra.run.smithi007.stdout: "status", 2024-04-21T08:56:45.832 INFO:teuthology.orchestra.run.smithi007.stdout: "telemetry", 2024-04-21T08:56:45.832 INFO:teuthology.orchestra.run.smithi007.stdout: "volumes" 2024-04-21T08:56:45.833 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.833 INFO:teuthology.orchestra.run.smithi007.stdout: "quincy": [ 2024-04-21T08:56:45.833 INFO:teuthology.orchestra.run.smithi007.stdout: "balancer", 2024-04-21T08:56:45.833 INFO:teuthology.orchestra.run.smithi007.stdout: "crash", 2024-04-21T08:56:45.833 INFO:teuthology.orchestra.run.smithi007.stdout: "devicehealth", 2024-04-21T08:56:45.833 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator", 2024-04-21T08:56:45.833 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscaler", 2024-04-21T08:56:45.833 INFO:teuthology.orchestra.run.smithi007.stdout: "progress", 2024-04-21T08:56:45.834 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_support", 2024-04-21T08:56:45.834 INFO:teuthology.orchestra.run.smithi007.stdout: "status", 2024-04-21T08:56:45.834 INFO:teuthology.orchestra.run.smithi007.stdout: "telemetry", 2024-04-21T08:56:45.834 INFO:teuthology.orchestra.run.smithi007.stdout: "volumes" 2024-04-21T08:56:45.834 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.834 INFO:teuthology.orchestra.run.smithi007.stdout: "reef": [ 2024-04-21T08:56:45.834 INFO:teuthology.orchestra.run.smithi007.stdout: "balancer", 2024-04-21T08:56:45.834 INFO:teuthology.orchestra.run.smithi007.stdout: "crash", 2024-04-21T08:56:45.835 INFO:teuthology.orchestra.run.smithi007.stdout: "devicehealth", 2024-04-21T08:56:45.835 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator", 2024-04-21T08:56:45.835 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscaler", 2024-04-21T08:56:45.835 INFO:teuthology.orchestra.run.smithi007.stdout: "progress", 2024-04-21T08:56:45.835 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_support", 2024-04-21T08:56:45.835 INFO:teuthology.orchestra.run.smithi007.stdout: "status", 2024-04-21T08:56:45.835 INFO:teuthology.orchestra.run.smithi007.stdout: "telemetry", 2024-04-21T08:56:45.835 INFO:teuthology.orchestra.run.smithi007.stdout: "volumes" 2024-04-21T08:56:45.835 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-21T08:56:45.836 INFO:teuthology.orchestra.run.smithi007.stdout: "squid": [ 2024-04-21T08:56:45.836 INFO:teuthology.orchestra.run.smithi007.stdout: "balancer", 2024-04-21T08:56:45.836 INFO:teuthology.orchestra.run.smithi007.stdout: "crash", 2024-04-21T08:56:45.836 INFO:teuthology.orchestra.run.smithi007.stdout: "devicehealth", 2024-04-21T08:56:45.836 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator", 2024-04-21T08:56:45.836 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscaler", 2024-04-21T08:56:45.836 INFO:teuthology.orchestra.run.smithi007.stdout: "progress", 2024-04-21T08:56:45.837 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_support", 2024-04-21T08:56:45.837 INFO:teuthology.orchestra.run.smithi007.stdout: "status", 2024-04-21T08:56:45.837 INFO:teuthology.orchestra.run.smithi007.stdout: "telemetry", 2024-04-21T08:56:45.837 INFO:teuthology.orchestra.run.smithi007.stdout: "volumes" 2024-04-21T08:56:45.837 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:56:45.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.837 INFO:teuthology.orchestra.run.smithi007.stdout: "last_failure_osd_epoch": 95, 2024-04-21T08:56:45.837 INFO:teuthology.orchestra.run.smithi007.stdout: "active_clients": [ 2024-04-21T08:56:45.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.838 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "devicehealth", 2024-04-21T08:56:45.838 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-04-21T08:56:45.838 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.838 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-04-21T08:56:45.838 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2024-04-21T08:56:45.838 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2959679628 2024-04-21T08:56:45.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.839 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:56:45.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.839 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "libcephsqlite", 2024-04-21T08:56:45.839 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-04-21T08:56:45.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.839 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-04-21T08:56:45.839 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2024-04-21T08:56:45.840 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1813737758 2024-04-21T08:56:45.840 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.840 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:56:45.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.840 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.840 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_support", 2024-04-21T08:56:45.840 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-04-21T08:56:45.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.841 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-04-21T08:56:45.841 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2024-04-21T08:56:45.841 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 368294612 2024-04-21T08:56:45.841 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.841 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:56:45.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.841 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snap_schedule", 2024-04-21T08:56:45.842 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-04-21T08:56:45.842 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.842 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-04-21T08:56:45.842 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2024-04-21T08:56:45.842 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2137608117 2024-04-21T08:56:45.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.842 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:56:45.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.843 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snap_schedule", 2024-04-21T08:56:45.843 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-04-21T08:56:45.843 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.843 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-04-21T08:56:45.843 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2024-04-21T08:56:45.843 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1554146701 2024-04-21T08:56:45.844 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.844 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:56:45.844 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:45.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.844 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "volumes", 2024-04-21T08:56:45.844 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-04-21T08:56:45.844 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-21T08:56:45.844 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-04-21T08:56:45.845 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2024-04-21T08:56:45.845 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1178373638 2024-04-21T08:56:45.845 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.845 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:56:45.845 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:45.845 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-21T08:56:45.845 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:56:45.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 -- 172.21.15.7:0/974070730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f70f00a3d20 msgr2=0x7f70f00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:45.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f70f00a3d20 0x7f70f00a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f70fc005ab0 tx=0x7f70fc005a20 comp rx=0 tx=0).stop 2024-04-21T08:56:45.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 -- 172.21.15.7:0/974070730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7108069c10 msgr2=0x7f7108198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:45.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7108069c10 0x7f7108198c10 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f70f800d950 tx=0x7f70f800de10 comp rx=0 tx=0).stop 2024-04-21T08:56:45.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 -- 172.21.15.7:0/974070730 shutdown_connections 2024-04-21T08:56:45.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7108106f50 0x7f710819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:45.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7108069c10 0x7f7108198c10 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:45.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f70f00a3d20 0x7f70f00a61e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:45.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 --2- 172.21.15.7:0/974070730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7108069250 0x7f71081986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:45.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 -- 172.21.15.7:0/974070730 >> 172.21.15.7:0/974070730 conn(0x7f7108077c40 msgr2=0x7f71080fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:45.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 -- 172.21.15.7:0/974070730 shutdown_connections 2024-04-21T08:56:45.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:56:43.647+0000 7f710e2f4640 1 -- 172.21.15.7:0/974070730 wait complete. 2024-04-21T08:56:45.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:43 smithi007 ceph-mon[28477]: pgmap v381: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.5 KiB/s wr, 2 op/s 2024-04-21T08:56:45.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:43 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3053794859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:56:45.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:43 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2187725423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:56:45.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:43 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4170818117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:56:45.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:43 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2465053508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:56:45.850 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:44 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/974070730' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T08:56:45.850 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:45 smithi077 ceph-mon[31615]: pgmap v382: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.1 KiB/s wr, 2 op/s 2024-04-21T08:56:45.850 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:44 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/974070730' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T08:56:45.850 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:45 smithi016 ceph-mon[31259]: pgmap v382: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.1 KiB/s wr, 2 op/s 2024-04-21T08:56:45.858 DEBUG:tasks.check_counter:Getting stats from a 2024-04-21T08:56:45.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-21T08:56:45.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:44 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/974070730' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T08:56:45.859 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:45 smithi007 ceph-mon[28477]: pgmap v382: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.1 KiB/s wr, 2 op/s 2024-04-21T08:56:46.134 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:56:46.527 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:46 smithi007 ceph-mon[28477]: pgmap v383: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-04-21T08:56:46.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:46 smithi016 ceph-mon[31259]: pgmap v383: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-04-21T08:56:46.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:46 smithi077 ceph-mon[31615]: pgmap v383: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-04-21T08:56:48.303 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:56:48.303 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:48.303 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 0, 2024-04-21T08:56:48.303 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 2, 2024-04-21T08:56:48.303 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:56:48.303 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 0, 2024-04-21T08:56:48.304 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:48.304 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:48.304 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.001811761, 2024-04-21T08:56:48.304 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:56:48.304 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.000266067, 2024-04-21T08:56:48.304 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:48.304 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:48.304 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:48.304 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:48.304 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:48.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:48.305 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:48.305 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:48.305 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:48.305 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:48.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:48.305 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:56:48.305 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:56:48.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:48.306 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:48.306 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 6, 2024-04-21T08:56:48.306 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 182, 2024-04-21T08:56:48.306 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 3264, 2024-04-21T08:56:48.306 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 188112, 2024-04-21T08:56:48.306 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:48.306 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:48.306 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.013783178, 2024-04-21T08:56:48.307 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.009390382, 2024-04-21T08:56:48.307 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.002434739, 2024-04-21T08:56:48.307 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:48.307 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:48.307 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 180, 2024-04-21T08:56:48.307 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.012231225, 2024-04-21T08:56:48.307 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000067951 2024-04-21T08:56:48.307 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:48.308 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:48.308 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:48.308 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:48.308 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:48.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:48.308 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-04-21T08:56:48.308 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 188112 2024-04-21T08:56:48.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:48.308 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:48.308 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 246, 2024-04-21T08:56:48.309 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 235, 2024-04-21T08:56:48.309 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 1596240, 2024-04-21T08:56:48.309 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 144704, 2024-04-21T08:56:48.309 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:48.309 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:48.309 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.048656687, 2024-04-21T08:56:48.309 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.013402212, 2024-04-21T08:56:48.309 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.040312323, 2024-04-21T08:56:48.310 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.005709514, 2024-04-21T08:56:48.310 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:48.310 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 234, 2024-04-21T08:56:48.310 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.026178146, 2024-04-21T08:56:48.310 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000111872 2024-04-21T08:56:48.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:48.310 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:48.310 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:48.311 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:48.311 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:48.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:48.311 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 1596240, 2024-04-21T08:56:48.311 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 144704 2024-04-21T08:56:48.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:48.311 INFO:teuthology.orchestra.run.smithi007.stdout: "cct": { 2024-04-21T08:56:48.311 INFO:teuthology.orchestra.run.smithi007.stdout: "total_workers": 0, 2024-04-21T08:56:48.312 INFO:teuthology.orchestra.run.smithi007.stdout: "unhealthy_workers": 0 2024-04-21T08:56:48.312 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:48.312 INFO:teuthology.orchestra.run.smithi007.stdout: "mempool": { 2024-04-21T08:56:48.312 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:48.312 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:48.312 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:48.312 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:48.312 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:48.312 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:48.312 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:48.313 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:48.313 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:48.313 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:48.313 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:48.313 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:48.313 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:48.313 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:48.313 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:48.313 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:48.314 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:48.314 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:48.314 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:48.314 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:48.314 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:48.314 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:48.315 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:48.315 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:48.315 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:48.315 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:48.315 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:48.315 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:48.315 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:48.315 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:48.316 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:48.316 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_items": 0, 2024-04-21T08:56:48.316 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:48.316 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:48.316 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:48.316 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:48.316 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_bytes": 148977, 2024-04-21T08:56:48.316 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_items": 30, 2024-04-21T08:56:48.316 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:48.317 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:48.317 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bytes": 0, 2024-04-21T08:56:48.317 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_items": 0, 2024-04-21T08:56:48.317 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:48.317 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:48.317 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:48.317 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:48.317 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:48.318 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_items": 0, 2024-04-21T08:56:48.318 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:48.318 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:48.318 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:48.318 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_items": 0, 2024-04-21T08:56:48.318 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:48.318 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_items": 2, 2024-04-21T08:56:48.318 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:48.318 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_items": 0, 2024-04-21T08:56:48.319 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:48.319 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_items": 0 2024-04-21T08:56:48.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:48.319 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:48.319 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-04-21T08:56:48.319 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 104857600, 2024-04-21T08:56:48.319 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-04-21T08:56:48.319 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 248, 2024-04-21T08:56:48.319 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 1561535, 2024-04-21T08:56:48.320 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:48.320 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 248, 2024-04-21T08:56:48.320 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-04-21T08:56:48.320 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-04-21T08:56:48.320 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 248, 2024-04-21T08:56:48.320 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 1561535, 2024-04-21T08:56:48.320 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-04-21T08:56:48.320 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:48.321 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:48.321 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:48.321 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:48.321 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:48.321 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:56:48.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:48 smithi007 ceph-mon[28477]: pgmap v384: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:56:48.955 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-21T08:56:48.955 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-21T08:56:48.955 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-21T08:56:48.955 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-04-21T08:56:48.956 DEBUG:tasks.check_counter:Getting stats from d 2024-04-21T08:56:48.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-21T08:56:49.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:48 smithi016 ceph-mon[31259]: pgmap v384: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:56:49.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:48 smithi077 ceph-mon[31615]: pgmap v384: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:56:49.209 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:56:50.863 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:56:50.863 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:50.864 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 4, 2024-04-21T08:56:50.864 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 4, 2024-04-21T08:56:50.864 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:56:50.864 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 448, 2024-04-21T08:56:50.864 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:50.864 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:50.864 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.001632554, 2024-04-21T08:56:50.864 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.000033764, 2024-04-21T08:56:50.865 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.000565854, 2024-04-21T08:56:50.865 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:50.865 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:50.865 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 3, 2024-04-21T08:56:50.865 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000689397, 2024-04-21T08:56:50.865 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000229799 2024-04-21T08:56:50.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:50.865 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:50.866 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:50.866 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:50.866 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:50.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:50.866 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:56:50.866 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:56:50.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:50.867 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:50.867 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 2, 2024-04-21T08:56:50.867 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 184, 2024-04-21T08:56:50.867 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:56:50.867 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 188368, 2024-04-21T08:56:50.867 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:50.867 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:50.867 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.014684445, 2024-04-21T08:56:50.868 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.010145570, 2024-04-21T08:56:50.868 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.001766461, 2024-04-21T08:56:50.868 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:50.868 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:50.868 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 181, 2024-04-21T08:56:50.868 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.023875579, 2024-04-21T08:56:50.868 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000131909 2024-04-21T08:56:50.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:50.869 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:50.869 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:50.869 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:50.869 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:50.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:50.869 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:56:50.869 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 188368 2024-04-21T08:56:50.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:50.870 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:50.870 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 253, 2024-04-21T08:56:50.870 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 242, 2024-04-21T08:56:50.870 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 1598768, 2024-04-21T08:56:50.870 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 148880, 2024-04-21T08:56:50.870 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:50.870 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:50.871 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.045891621, 2024-04-21T08:56:50.871 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.014806039, 2024-04-21T08:56:50.871 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.035669208, 2024-04-21T08:56:50.871 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.005253790, 2024-04-21T08:56:50.871 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:50.871 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 241, 2024-04-21T08:56:50.871 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.034417053, 2024-04-21T08:56:50.872 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000142809 2024-04-21T08:56:50.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:50.872 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:50.872 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:50.872 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:50.872 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:50.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:50.872 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 1598768, 2024-04-21T08:56:50.873 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 148880 2024-04-21T08:56:50.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:50.873 INFO:teuthology.orchestra.run.smithi007.stdout: "cct": { 2024-04-21T08:56:50.873 INFO:teuthology.orchestra.run.smithi007.stdout: "total_workers": 0, 2024-04-21T08:56:50.873 INFO:teuthology.orchestra.run.smithi007.stdout: "unhealthy_workers": 0 2024-04-21T08:56:50.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:50.873 INFO:teuthology.orchestra.run.smithi007.stdout: "mempool": { 2024-04-21T08:56:50.873 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:50.873 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:50.874 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:50.874 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:50.874 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:50.874 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:50.874 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:50.874 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:50.874 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:50.875 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:50.875 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:50.875 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:50.875 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:50.876 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:50.876 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:50.876 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:50.876 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:50.876 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:50.876 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:50.876 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:50.876 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:50.877 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:50.877 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:50.877 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:50.877 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:50.877 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:50.877 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:50.877 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:50.877 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:50.878 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:50.878 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:50.878 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_items": 0, 2024-04-21T08:56:50.878 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:50.878 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:50.878 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:50.878 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:50.879 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_bytes": 148977, 2024-04-21T08:56:50.879 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_items": 30, 2024-04-21T08:56:50.879 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:50.879 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:50.879 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bytes": 0, 2024-04-21T08:56:50.879 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_items": 0, 2024-04-21T08:56:50.879 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:50.879 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:50.880 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:50.880 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:50.880 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:50.880 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_items": 0, 2024-04-21T08:56:50.880 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:50.880 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:50.880 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:50.880 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_items": 0, 2024-04-21T08:56:50.881 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:50.881 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_items": 2, 2024-04-21T08:56:50.881 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:50.881 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_items": 0, 2024-04-21T08:56:50.881 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:50.881 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_items": 0 2024-04-21T08:56:50.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:50.881 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:50.881 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-04-21T08:56:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 104857600, 2024-04-21T08:56:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-04-21T08:56:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 255, 2024-04-21T08:56:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 1563082, 2024-04-21T08:56:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 255, 2024-04-21T08:56:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-04-21T08:56:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-04-21T08:56:50.883 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 255, 2024-04-21T08:56:50.883 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 1563082, 2024-04-21T08:56:50.883 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-04-21T08:56:50.883 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:50.883 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:50.883 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:50.883 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:50.883 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:50.883 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:56:50.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:50 smithi007 ceph-mon[28477]: pgmap v385: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:56:51.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:50 smithi016 ceph-mon[31259]: pgmap v385: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:56:51.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:50 smithi077 ceph-mon[31615]: pgmap v385: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:56:51.599 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-21T08:56:51.599 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-21T08:56:51.599 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-21T08:56:51.599 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-04-21T08:56:51.599 DEBUG:tasks.check_counter:Getting stats from g 2024-04-21T08:56:51.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-21T08:56:51.847 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:56:52.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:52 smithi007 ceph-mon[28477]: pgmap v386: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-04-21T08:56:53.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:52 smithi016 ceph-mon[31259]: pgmap v386: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-04-21T08:56:53.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:52 smithi077 ceph-mon[31615]: pgmap v386: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.5 KiB/s wr, 1 op/s 2024-04-21T08:56:53.599 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:56:53.599 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:53.599 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 248, 2024-04-21T08:56:53.599 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 237, 2024-04-21T08:56:53.599 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 1596784, 2024-04-21T08:56:53.599 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 145984, 2024-04-21T08:56:53.600 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:53.600 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:53.600 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.047179319, 2024-04-21T08:56:53.600 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.013838657, 2024-04-21T08:56:53.600 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.036344038, 2024-04-21T08:56:53.600 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.005353801, 2024-04-21T08:56:53.600 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:53.600 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 236, 2024-04-21T08:56:53.601 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.022563287, 2024-04-21T08:56:53.601 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000095607 2024-04-21T08:56:53.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:53.601 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:53.601 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:53.601 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:53.601 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:53.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:53.601 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 1596784, 2024-04-21T08:56:53.602 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 145984 2024-04-21T08:56:53.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:53.602 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:53.602 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 2, 2024-04-21T08:56:53.602 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 182, 2024-04-21T08:56:53.602 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:56:53.602 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 188016, 2024-04-21T08:56:53.602 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:53.603 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:53.603 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.014417916, 2024-04-21T08:56:53.603 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.009758919, 2024-04-21T08:56:53.603 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.002097916, 2024-04-21T08:56:53.603 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:53.603 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:53.603 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 179, 2024-04-21T08:56:53.603 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.029258400, 2024-04-21T08:56:53.603 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000163454 2024-04-21T08:56:53.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:53.604 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:53.604 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:53.604 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:53.604 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:53.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:53.604 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:56:53.604 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 188016 2024-04-21T08:56:53.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:53.605 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:53.605 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 4, 2024-04-21T08:56:53.605 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 4, 2024-04-21T08:56:53.605 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:56:53.605 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 448, 2024-04-21T08:56:53.605 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:53.605 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:53.605 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.000636864, 2024-04-21T08:56:53.606 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.000038070, 2024-04-21T08:56:53.606 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.000567444, 2024-04-21T08:56:53.606 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:53.606 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:53.606 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 3, 2024-04-21T08:56:53.606 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000788190, 2024-04-21T08:56:53.606 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000262730 2024-04-21T08:56:53.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:53.607 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:53.607 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:53.607 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:53.607 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:53.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:53.607 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:56:53.607 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:56:53.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:53.607 INFO:teuthology.orchestra.run.smithi007.stdout: "cct": { 2024-04-21T08:56:53.608 INFO:teuthology.orchestra.run.smithi007.stdout: "total_workers": 0, 2024-04-21T08:56:53.608 INFO:teuthology.orchestra.run.smithi007.stdout: "unhealthy_workers": 0 2024-04-21T08:56:53.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:53.608 INFO:teuthology.orchestra.run.smithi007.stdout: "mempool": { 2024-04-21T08:56:53.608 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:53.608 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:53.608 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:53.608 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:53.609 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:53.609 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:53.609 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:53.609 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:53.609 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:53.609 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:53.609 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:53.609 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:53.609 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:53.610 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:53.611 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_items": 0, 2024-04-21T08:56:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_bytes": 148994, 2024-04-21T08:56:53.612 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_items": 31, 2024-04-21T08:56:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bytes": 0, 2024-04-21T08:56:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_items": 0, 2024-04-21T08:56:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:53.613 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_items": 0, 2024-04-21T08:56:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_items": 0, 2024-04-21T08:56:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_items": 2, 2024-04-21T08:56:53.614 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:53.615 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_items": 0, 2024-04-21T08:56:53.615 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:53.615 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_items": 0 2024-04-21T08:56:53.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:53.615 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:53.615 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-04-21T08:56:53.615 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 104857600, 2024-04-21T08:56:53.615 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-04-21T08:56:53.615 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 250, 2024-04-21T08:56:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 1561795, 2024-04-21T08:56:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 250, 2024-04-21T08:56:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-04-21T08:56:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-04-21T08:56:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 250, 2024-04-21T08:56:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 1561795, 2024-04-21T08:56:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-04-21T08:56:53.617 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:53.617 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:53.617 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:53.617 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:53.617 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:53.617 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:56:54.292 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-21T08:56:54.292 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-21T08:56:54.292 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-21T08:56:54.293 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-04-21T08:56:54.293 DEBUG:tasks.check_counter:Getting stats from j 2024-04-21T08:56:54.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-21T08:56:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:56:54.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:54 smithi007 ceph-mon[28477]: pgmap v387: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:56:55.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:54 smithi016 ceph-mon[31259]: pgmap v387: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:56:55.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:54 smithi077 ceph-mon[31615]: pgmap v387: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 0, 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 2, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 0, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.001325328, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.000287343, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:56.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:56:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:56:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 2, 2024-04-21T08:56:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 186, 2024-04-21T08:56:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:56:56.202 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 188720, 2024-04-21T08:56:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.013478869, 2024-04-21T08:56:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.008939940, 2024-04-21T08:56:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.002006047, 2024-04-21T08:56:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 183, 2024-04-21T08:56:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.013802261, 2024-04-21T08:56:56.203 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000075422 2024-04-21T08:56:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:56:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 188720 2024-04-21T08:56:56.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 259, 2024-04-21T08:56:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 246, 2024-04-21T08:56:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 1602256, 2024-04-21T08:56:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 150608, 2024-04-21T08:56:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.051188871, 2024-04-21T08:56:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.014177696, 2024-04-21T08:56:56.205 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.043356027, 2024-04-21T08:56:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.005406634, 2024-04-21T08:56:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 246, 2024-04-21T08:56:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.020003516, 2024-04-21T08:56:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000081315 2024-04-21T08:56:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:56.206 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 1602256, 2024-04-21T08:56:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 150608 2024-04-21T08:56:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:56.207 INFO:teuthology.orchestra.run.smithi007.stdout: "cct": { 2024-04-21T08:56:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "total_workers": 0, 2024-04-21T08:56:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "unhealthy_workers": 0 2024-04-21T08:56:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mempool": { 2024-04-21T08:56:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:56.208 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:56.209 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:56.210 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_items": 0, 2024-04-21T08:56:56.211 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_bytes": 149017, 2024-04-21T08:56:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_items": 31, 2024-04-21T08:56:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bytes": 0, 2024-04-21T08:56:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_items": 0, 2024-04-21T08:56:56.212 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_items": 0, 2024-04-21T08:56:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:56.213 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_items": 0, 2024-04-21T08:56:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_items": 2, 2024-04-21T08:56:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_items": 0, 2024-04-21T08:56:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_items": 0 2024-04-21T08:56:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-21T08:56:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:56.214 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-04-21T08:56:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 104857600, 2024-04-21T08:56:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-04-21T08:56:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 257, 2024-04-21T08:56:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 1563342, 2024-04-21T08:56:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 257, 2024-04-21T08:56:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-04-21T08:56:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-04-21T08:56:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 257, 2024-04-21T08:56:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 1563342, 2024-04-21T08:56:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-04-21T08:56:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-04-21T08:56:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-04-21T08:56:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-04-21T08:56:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:56.216 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-21T08:56:56.216 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-21T08:56:56.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:56 smithi007 ceph-mon[28477]: pgmap v388: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:56:56.881 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-21T08:56:56.881 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-21T08:56:56.881 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-21T08:56:56.881 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-04-21T08:56:56.881 DEBUG:tasks.check_counter:Getting stats from b 2024-04-21T08:56:56.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-21T08:56:56.953 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:56 smithi016 ceph-mon[31259]: pgmap v388: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:56:57.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:56 smithi077 ceph-mon[31615]: pgmap v388: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:56:57.221 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:56:58.763 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:56:58 smithi016 ceph-mon[31259]: pgmap v389: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T08:56:58.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:56:58 smithi007 ceph-mon[28477]: pgmap v389: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T08:56:59.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:56:58 smithi077 ceph-mon[31615]: pgmap v389: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T08:56:59.461 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-21T08:56:59.461 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:59.461 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 262178, 2024-04-21T08:56:59.462 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 261496, 2024-04-21T08:56:59.462 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 154491551, 2024-04-21T08:56:59.462 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 461385218, 2024-04-21T08:56:59.462 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 18, 2024-04-21T08:56:59.462 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 5, 2024-04-21T08:56:59.462 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 15.751717567, 2024-04-21T08:56:59.462 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 4.878331223, 2024-04-21T08:56:59.462 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 3.260349032, 2024-04-21T08:56:59.463 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 6.446871968, 2024-04-21T08:56:59.463 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:59.463 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 261495, 2024-04-21T08:56:59.463 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 136.979393925, 2024-04-21T08:56:59.463 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000523831 2024-04-21T08:56:59.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.463 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:59.463 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1044, 2024-04-21T08:56:59.463 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000170306, 2024-04-21T08:56:59.464 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000163 2024-04-21T08:56:59.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.464 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 382128, 2024-04-21T08:56:59.464 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 111776 2024-04-21T08:56:59.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.464 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:59.464 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 194808, 2024-04-21T08:56:59.464 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 194706, 2024-04-21T08:56:59.464 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 111969042, 2024-04-21T08:56:59.465 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 408075886, 2024-04-21T08:56:59.465 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 193, 2024-04-21T08:56:59.465 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 8, 2024-04-21T08:56:59.465 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 12.658059268, 2024-04-21T08:56:59.465 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 3.761583469, 2024-04-21T08:56:59.465 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 2.623249622, 2024-04-21T08:56:59.465 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 5.421137643, 2024-04-21T08:56:59.465 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:59.465 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 194705, 2024-04-21T08:56:59.466 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 104.163767899, 2024-04-21T08:56:59.466 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000534982 2024-04-21T08:56:59.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.466 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:59.466 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 291, 2024-04-21T08:56:59.466 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000038942, 2024-04-21T08:56:59.466 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000133 2024-04-21T08:56:59.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.467 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1697248, 2024-04-21T08:56:59.467 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 179568 2024-04-21T08:56:59.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.467 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:59.467 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 480794, 2024-04-21T08:56:59.467 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 665770, 2024-04-21T08:56:59.467 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 225449081, 2024-04-21T08:56:59.467 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 435000548, 2024-04-21T08:56:59.467 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 24, 2024-04-21T08:56:59.468 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 5, 2024-04-21T08:56:59.468 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 28.497688438, 2024-04-21T08:56:59.468 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 10.308348896, 2024-04-21T08:56:59.468 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 133.972551219, 2024-04-21T08:56:59.468 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 5.539300114, 2024-04-21T08:56:59.468 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:59.468 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 665767, 2024-04-21T08:56:59.468 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 142.901947326, 2024-04-21T08:56:59.468 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000214642 2024-04-21T08:56:59.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.469 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:59.469 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 633249, 2024-04-21T08:56:59.469 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.115009000, 2024-04-21T08:56:59.469 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000181 2024-04-21T08:56:59.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.469 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 100128, 2024-04-21T08:56:59.469 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 530368 2024-04-21T08:56:59.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.470 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-21T08:56:59.470 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 1, 2024-04-21T08:56:59.470 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-21T08:56:59.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.470 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-MDSRank": { 2024-04-21T08:56:59.470 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-04-21T08:56:59.470 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-04-21T08:56:59.470 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 338647, 2024-04-21T08:56:59.471 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 314.043933179, 2024-04-21T08:56:59.471 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000927348 2024-04-21T08:56:59.471 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:56:59.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.471 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-PurgeQueue": { 2024-04-21T08:56:59.471 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-04-21T08:56:59.471 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-04-21T08:56:59.471 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1811, 2024-04-21T08:56:59.471 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.198571767, 2024-04-21T08:56:59.472 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000109647 2024-04-21T08:56:59.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:56:59.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.472 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-21T08:56:59.472 INFO:teuthology.orchestra.run.smithi016.stdout: "request": 100151, 2024-04-21T08:56:59.472 INFO:teuthology.orchestra.run.smithi016.stdout: "reply": 100151, 2024-04-21T08:56:59.472 INFO:teuthology.orchestra.run.smithi016.stdout: "reply_latency": { 2024-04-21T08:56:59.472 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 100151, 2024-04-21T08:56:59.472 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 459.475422544, 2024-04-21T08:56:59.473 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.004587826 2024-04-21T08:56:59.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.473 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_reply": 0, 2024-04-21T08:56:59.473 INFO:teuthology.orchestra.run.smithi016.stdout: "forward": 0, 2024-04-21T08:56:59.473 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_complete": 0, 2024-04-21T08:56:59.473 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_keys": 0, 2024-04-21T08:56:59.473 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_commit": 19685, 2024-04-21T08:56:59.473 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_split": 206, 2024-04-21T08:56:59.473 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_merge": 0, 2024-04-21T08:56:59.474 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes": 107573, 2024-04-21T08:56:59.474 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_top": 13533, 2024-04-21T08:56:59.474 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_bottom": 94040, 2024-04-21T08:56:59.474 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pin_tail": 0, 2024-04-21T08:56:59.474 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pinned": 87328, 2024-04-21T08:56:59.474 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_expired": 0, 2024-04-21T08:56:59.474 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_with_caps": 86961, 2024-04-21T08:56:59.474 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": 86962, 2024-04-21T08:56:59.475 INFO:teuthology.orchestra.run.smithi016.stdout: "subtrees": 2, 2024-04-21T08:56:59.475 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse": 100674, 2024-04-21T08:56:59.475 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_hit": 99607, 2024-04-21T08:56:59.475 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_forward": 0, 2024-04-21T08:56:59.475 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_discover": 0, 2024-04-21T08:56:59.475 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_dir_fetch": 0, 2024-04-21T08:56:59.475 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_remote_ino": 0, 2024-04-21T08:56:59.475 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_lock": 0, 2024-04-21T08:56:59.476 INFO:teuthology.orchestra.run.smithi016.stdout: "load_cent": 0, 2024-04-21T08:56:59.476 INFO:teuthology.orchestra.run.smithi016.stdout: "q": 0, 2024-04-21T08:56:59.476 INFO:teuthology.orchestra.run.smithi016.stdout: "exported": 0, 2024-04-21T08:56:59.476 INFO:teuthology.orchestra.run.smithi016.stdout: "exported_inodes": 0, 2024-04-21T08:56:59.476 INFO:teuthology.orchestra.run.smithi016.stdout: "imported": 0, 2024-04-21T08:56:59.476 INFO:teuthology.orchestra.run.smithi016.stdout: "imported_inodes": 0, 2024-04-21T08:56:59.476 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_dir_fetch": 0, 2024-04-21T08:56:59.476 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_backtrace_fetch": 470600, 2024-04-21T08:56:59.476 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_peer_discover": 0, 2024-04-21T08:56:59.477 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rfiles": 84661, 2024-04-21T08:56:59.477 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rbytes": 1548381344, 2024-04-21T08:56:59.477 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rsnaps": 6, 2024-04-21T08:56:59.477 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_fetch": 470600, 2024-04-21T08:56:59.477 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_set_tag": 0, 2024-04-21T08:56:59.477 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T08:56:59.477 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_inotable_repaired": 0, 2024-04-21T08:56:59.477 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_inodes": 27162, 2024-04-21T08:56:59.477 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_base_inodes": 12, 2024-04-21T08:56:59.478 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dirfrag_rstats": 27150, 2024-04-21T08:56:59.478 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_file_inodes": 443438, 2024-04-21T08:56:59.478 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_inode_file_caps": 0, 2024-04-21T08:56:59.478 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_revoke": 25065, 2024-04-21T08:56:59.478 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_grant": 95601, 2024-04-21T08:56:59.478 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_trunc": 65, 2024-04-21T08:56:59.478 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T08:56:59.478 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T08:56:59.479 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps": 151656, 2024-04-21T08:56:59.479 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps_dirty": 125412, 2024-04-21T08:56:59.479 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_cap_release": 985, 2024-04-21T08:56:59.479 INFO:teuthology.orchestra.run.smithi016.stdout: "process_request_cap_release": 13424 2024-04-21T08:56:59.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.479 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache": { 2024-04-21T08:56:59.479 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update": 0, 2024-04-21T08:56:59.479 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update_receipt": 0, 2024-04-21T08:56:59.479 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_try_discover": 0, 2024-04-21T08:56:59.479 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_send_discover": 0, 2024-04-21T08:56:59.480 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_handle_discover": 0, 2024-04-21T08:56:59.480 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays": 0, 2024-04-21T08:56:59.480 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_delayed": 0, 2024-04-21T08:56:59.480 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_enqueuing": 0, 2024-04-21T08:56:59.480 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_created": 1056, 2024-04-21T08:56:59.480 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_enqueued": 1056, 2024-04-21T08:56:59.480 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_reintegrated": 0, 2024-04-21T08:56:59.480 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_migrated": 0, 2024-04-21T08:56:59.481 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_processing": 0, 2024-04-21T08:56:59.481 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_enqueued": 0, 2024-04-21T08:56:59.481 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_prioritized": 0, 2024-04-21T08:56:59.481 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_started": 0, 2024-04-21T08:56:59.481 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_completed": 0, 2024-04-21T08:56:59.481 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_path": 1, 2024-04-21T08:56:59.481 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_inode": 19450, 2024-04-21T08:56:59.481 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_enqueue_scrub": 12, 2024-04-21T08:56:59.481 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_exportdir": 0, 2024-04-21T08:56:59.482 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_flush": 0, 2024-04-21T08:56:59.482 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragmentdir": 209, 2024-04-21T08:56:59.482 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragstats": 0, 2024-04-21T08:56:59.482 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_inodestats": 0 2024-04-21T08:56:59.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.482 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log": { 2024-04-21T08:56:59.482 INFO:teuthology.orchestra.run.smithi016.stdout: "evlrg": 0, 2024-04-21T08:56:59.482 INFO:teuthology.orchestra.run.smithi016.stdout: "evadd": 233562, 2024-04-21T08:56:59.482 INFO:teuthology.orchestra.run.smithi016.stdout: "evex": 232376, 2024-04-21T08:56:59.483 INFO:teuthology.orchestra.run.smithi016.stdout: "evtrm": 232376, 2024-04-21T08:56:59.483 INFO:teuthology.orchestra.run.smithi016.stdout: "ev": 1186, 2024-04-21T08:56:59.483 INFO:teuthology.orchestra.run.smithi016.stdout: "evexg": 0, 2024-04-21T08:56:59.483 INFO:teuthology.orchestra.run.smithi016.stdout: "evexd": 737, 2024-04-21T08:56:59.483 INFO:teuthology.orchestra.run.smithi016.stdout: "segadd": 312, 2024-04-21T08:56:59.483 INFO:teuthology.orchestra.run.smithi016.stdout: "segex": 310, 2024-04-21T08:56:59.483 INFO:teuthology.orchestra.run.smithi016.stdout: "segtrm": 310, 2024-04-21T08:56:59.483 INFO:teuthology.orchestra.run.smithi016.stdout: "seg": 2, 2024-04-21T08:56:59.483 INFO:teuthology.orchestra.run.smithi016.stdout: "segmjr": 1, 2024-04-21T08:56:59.484 INFO:teuthology.orchestra.run.smithi016.stdout: "segexg": 0, 2024-04-21T08:56:59.484 INFO:teuthology.orchestra.run.smithi016.stdout: "segexd": 2, 2024-04-21T08:56:59.484 INFO:teuthology.orchestra.run.smithi016.stdout: "expos": 614683639, 2024-04-21T08:56:59.484 INFO:teuthology.orchestra.run.smithi016.stdout: "wrpos": 620413578, 2024-04-21T08:56:59.484 INFO:teuthology.orchestra.run.smithi016.stdout: "rdpos": 0, 2024-04-21T08:56:59.484 INFO:teuthology.orchestra.run.smithi016.stdout: "jlat": { 2024-04-21T08:56:59.484 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 16031, 2024-04-21T08:56:59.484 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 3224.101023519, 2024-04-21T08:56:59.484 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.201116650 2024-04-21T08:56:59.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.485 INFO:teuthology.orchestra.run.smithi016.stdout: "replayed": 0 2024-04-21T08:56:59.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.485 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mem": { 2024-04-21T08:56:59.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 106337, 2024-04-21T08:56:59.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ino+": 107405, 2024-04-21T08:56:59.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ino-": 1068, 2024-04-21T08:56:59.485 INFO:teuthology.orchestra.run.smithi016.stdout: "dir": 6753, 2024-04-21T08:56:59.485 INFO:teuthology.orchestra.run.smithi016.stdout: "dir+": 7065, 2024-04-21T08:56:59.486 INFO:teuthology.orchestra.run.smithi016.stdout: "dir-": 312, 2024-04-21T08:56:59.486 INFO:teuthology.orchestra.run.smithi016.stdout: "dn": 107573, 2024-04-21T08:56:59.486 INFO:teuthology.orchestra.run.smithi016.stdout: "dn+": 109127, 2024-04-21T08:56:59.486 INFO:teuthology.orchestra.run.smithi016.stdout: "dn-": 1554, 2024-04-21T08:56:59.486 INFO:teuthology.orchestra.run.smithi016.stdout: "cap": 86962, 2024-04-21T08:56:59.486 INFO:teuthology.orchestra.run.smithi016.stdout: "cap+": 110452, 2024-04-21T08:56:59.486 INFO:teuthology.orchestra.run.smithi016.stdout: "cap-": 23490, 2024-04-21T08:56:59.486 INFO:teuthology.orchestra.run.smithi016.stdout: "rss": 594940, 2024-04-21T08:56:59.487 INFO:teuthology.orchestra.run.smithi016.stdout: "heap": 231800 2024-04-21T08:56:59.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.487 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_server": { 2024-04-21T08:56:59.487 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_client_request": 105660, 2024-04-21T08:56:59.487 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_server_request": 0, 2024-04-21T08:56:59.487 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_request": 100151, 2024-04-21T08:56:59.487 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_session": 38, 2024-04-21T08:56:59.487 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_peer_request": 0, 2024-04-21T08:56:59.487 INFO:teuthology.orchestra.run.smithi016.stdout: "req_create_latency": { 2024-04-21T08:56:59.488 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 85549, 2024-04-21T08:56:59.488 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 343.183411157, 2024-04-21T08:56:59.488 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.004011542 2024-04-21T08:56:59.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.488 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getattr_latency": { 2024-04-21T08:56:59.488 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 262, 2024-04-21T08:56:59.488 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.269755292, 2024-04-21T08:56:59.488 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.001029600 2024-04-21T08:56:59.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.489 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getfilelock_latency": { 2024-04-21T08:56:59.489 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.489 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.489 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.489 INFO:teuthology.orchestra.run.smithi016.stdout: "req_link_latency": { 2024-04-21T08:56:59.489 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.489 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.489 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.490 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookup_latency": { 2024-04-21T08:56:59.490 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 91, 2024-04-21T08:56:59.490 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 5.887247432, 2024-04-21T08:56:59.490 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.064695026 2024-04-21T08:56:59.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.490 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookuphash_latency": { 2024-04-21T08:56:59.490 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.490 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.491 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.491 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupino_latency": { 2024-04-21T08:56:59.491 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2, 2024-04-21T08:56:59.491 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000409859, 2024-04-21T08:56:59.491 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000204929 2024-04-21T08:56:59.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.491 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupname_latency": { 2024-04-21T08:56:59.491 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.492 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.492 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.492 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupparent_latency": { 2024-04-21T08:56:59.492 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.492 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.492 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.492 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupsnap_latency": { 2024-04-21T08:56:59.493 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.493 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.493 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.494 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lssnap_latency": { 2024-04-21T08:56:59.494 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 12, 2024-04-21T08:56:59.494 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.016259341, 2024-04-21T08:56:59.494 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.001354945 2024-04-21T08:56:59.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.494 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mkdir_latency": { 2024-04-21T08:56:59.494 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 5405, 2024-04-21T08:56:59.494 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 25.536798118, 2024-04-21T08:56:59.495 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.004724662 2024-04-21T08:56:59.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.495 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mknod_latency": { 2024-04-21T08:56:59.495 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.495 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.495 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.495 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mksnap_latency": { 2024-04-21T08:56:59.495 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 12, 2024-04-21T08:56:59.496 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 4.850861540, 2024-04-21T08:56:59.496 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.404238461 2024-04-21T08:56:59.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.496 INFO:teuthology.orchestra.run.smithi016.stdout: "req_open_latency": { 2024-04-21T08:56:59.496 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 703, 2024-04-21T08:56:59.496 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.388406578, 2024-04-21T08:56:59.496 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000552498 2024-04-21T08:56:59.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.496 INFO:teuthology.orchestra.run.smithi016.stdout: "req_readdir_latency": { 2024-04-21T08:56:59.497 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 6824, 2024-04-21T08:56:59.497 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 62.901566496, 2024-04-21T08:56:59.497 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.009217697 2024-04-21T08:56:59.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.497 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rename_latency": { 2024-04-21T08:56:59.497 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 30, 2024-04-21T08:56:59.497 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.044012047, 2024-04-21T08:56:59.497 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.001467068 2024-04-21T08:56:59.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.498 INFO:teuthology.orchestra.run.smithi016.stdout: "req_renamesnap_latency": { 2024-04-21T08:56:59.498 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.498 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.498 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.498 INFO:teuthology.orchestra.run.smithi016.stdout: "req_snapdiff_latency": { 2024-04-21T08:56:59.498 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.498 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.499 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.499 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmdir_latency": { 2024-04-21T08:56:59.499 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 106, 2024-04-21T08:56:59.499 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 2.692525070, 2024-04-21T08:56:59.499 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.025401179 2024-04-21T08:56:59.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.499 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmsnap_latency": { 2024-04-21T08:56:59.499 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 6, 2024-04-21T08:56:59.500 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 4.538342945, 2024-04-21T08:56:59.500 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.756390490 2024-04-21T08:56:59.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.500 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmxattr_latency": { 2024-04-21T08:56:59.500 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.500 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.500 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.500 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setattr_latency": { 2024-04-21T08:56:59.501 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 129, 2024-04-21T08:56:59.501 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 6.645085586, 2024-04-21T08:56:59.501 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.051512291 2024-04-21T08:56:59.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.501 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setdirlayout_latency": { 2024-04-21T08:56:59.501 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.501 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.501 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.502 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setfilelock_latency": { 2024-04-21T08:56:59.502 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.502 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.502 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.502 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setlayout_latency": { 2024-04-21T08:56:59.502 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.502 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.502 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.503 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setxattr_latency": { 2024-04-21T08:56:59.503 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 8, 2024-04-21T08:56:59.503 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.022232845, 2024-04-21T08:56:59.503 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.002779105 2024-04-21T08:56:59.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.503 INFO:teuthology.orchestra.run.smithi016.stdout: "req_symlink_latency": { 2024-04-21T08:56:59.503 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 62, 2024-04-21T08:56:59.503 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.134420407, 2024-04-21T08:56:59.504 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.002168071 2024-04-21T08:56:59.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.504 INFO:teuthology.orchestra.run.smithi016.stdout: "req_unlink_latency": { 2024-04-21T08:56:59.504 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 950, 2024-04-21T08:56:59.504 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 2.364087831, 2024-04-21T08:56:59.504 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.002488513 2024-04-21T08:56:59.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.504 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_revoke_eviction": 0, 2024-04-21T08:56:59.505 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_acquisition_throttle": 0, 2024-04-21T08:56:59.505 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getvxattr_latency": { 2024-04-21T08:56:59.505 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.505 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.505 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:56:59.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.505 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessions": { 2024-04-21T08:56:59.506 INFO:teuthology.orchestra.run.smithi016.stdout: "session_count": 2, 2024-04-21T08:56:59.506 INFO:teuthology.orchestra.run.smithi016.stdout: "session_add": 6, 2024-04-21T08:56:59.506 INFO:teuthology.orchestra.run.smithi016.stdout: "session_remove": 4, 2024-04-21T08:56:59.506 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_open": 2, 2024-04-21T08:56:59.506 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_stale": 0, 2024-04-21T08:56:59.506 INFO:teuthology.orchestra.run.smithi016.stdout: "total_load": 4336, 2024-04-21T08:56:59.506 INFO:teuthology.orchestra.run.smithi016.stdout: "average_load": 2168, 2024-04-21T08:56:59.506 INFO:teuthology.orchestra.run.smithi016.stdout: "avg_session_uptime": 727, 2024-04-21T08:56:59.506 INFO:teuthology.orchestra.run.smithi016.stdout: "mdthresh_evicted": 0 2024-04-21T08:56:59.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.507 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-21T08:56:59.507 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:59.507 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:59.507 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:59.507 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:59.507 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:59.507 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:59.507 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:59.508 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:59.508 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:59.508 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:59.508 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:59.508 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:59.508 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:59.508 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:59.508 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:59.508 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:59.509 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:59.509 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:59.509 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:59.509 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:59.509 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:59.509 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:59.509 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:59.509 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:59.509 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:59.510 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:59.510 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:59.510 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:59.510 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:59.510 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:59.510 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:59.510 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-21T08:56:59.510 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:59.510 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:59.511 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:59.511 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:59.511 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 3624236, 2024-04-21T08:56:59.511 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 90085, 2024-04-21T08:56:59.511 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:59.511 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:59.511 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-21T08:56:59.511 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-21T08:56:59.511 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:59.512 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:59.512 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:59.512 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:59.512 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 9760, 2024-04-21T08:56:59.512 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 186, 2024-04-21T08:56:59.512 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:59.512 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:59.512 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:59.512 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-21T08:56:59.513 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 383662058, 2024-04-21T08:56:59.513 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 4823621, 2024-04-21T08:56:59.513 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:59.513 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-21T08:56:59.513 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:59.513 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-21T08:56:59.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.513 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter": { 2024-04-21T08:56:59.513 INFO:teuthology.orchestra.run.smithi016.stdout: "op_active": 0, 2024-04-21T08:56:59.514 INFO:teuthology.orchestra.run.smithi016.stdout: "op_laggy": 0, 2024-04-21T08:56:59.514 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send": 685668, 2024-04-21T08:56:59.514 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send_bytes": 865463048, 2024-04-21T08:56:59.514 INFO:teuthology.orchestra.run.smithi016.stdout: "op_resend": 37, 2024-04-21T08:56:59.514 INFO:teuthology.orchestra.run.smithi016.stdout: "op_reply": 685668, 2024-04-21T08:56:59.514 INFO:teuthology.orchestra.run.smithi016.stdout: "op_latency": { 2024-04-21T08:56:59.514 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 685668, 2024-04-21T08:56:59.514 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 211461.900255880, 2024-04-21T08:56:59.514 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.308402755 2024-04-21T08:56:59.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.515 INFO:teuthology.orchestra.run.smithi016.stdout: "op_inflight": 0, 2024-04-21T08:56:59.515 INFO:teuthology.orchestra.run.smithi016.stdout: "oplen_avg": { 2024-04-21T08:56:59.515 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 685668, 2024-04-21T08:56:59.515 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 983029 2024-04-21T08:56:59.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.515 INFO:teuthology.orchestra.run.smithi016.stdout: "op": 685668, 2024-04-21T08:56:59.515 INFO:teuthology.orchestra.run.smithi016.stdout: "op_r": 470699, 2024-04-21T08:56:59.515 INFO:teuthology.orchestra.run.smithi016.stdout: "op_w": 214969, 2024-04-21T08:56:59.516 INFO:teuthology.orchestra.run.smithi016.stdout: "op_rmw": 0, 2024-04-21T08:56:59.516 INFO:teuthology.orchestra.run.smithi016.stdout: "op_pg": 0, 2024-04-21T08:56:59.516 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_stat": 12725, 2024-04-21T08:56:59.516 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_create": 175284, 2024-04-21T08:56:59.516 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_read": 87, 2024-04-21T08:56:59.516 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_write": 16106, 2024-04-21T08:56:59.516 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writefull": 252, 2024-04-21T08:56:59.516 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writesame": 0, 2024-04-21T08:56:59.516 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_append": 0, 2024-04-21T08:56:59.517 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_zero": 0, 2024-04-21T08:56:59.517 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_truncate": 169, 2024-04-21T08:56:59.517 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_delete": 2846, 2024-04-21T08:56:59.517 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_mapext": 0, 2024-04-21T08:56:59.517 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_sparse_read": 0, 2024-04-21T08:56:59.517 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_clonerange": 0, 2024-04-21T08:56:59.517 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_getxattr": 470612, 2024-04-21T08:56:59.517 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_setxattr": 265702, 2024-04-21T08:56:59.517 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_cmpxattr": 0, 2024-04-21T08:56:59.518 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_rmxattr": 0, 2024-04-21T08:56:59.518 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_resetxattrs": 0, 2024-04-21T08:56:59.518 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_call": 0, 2024-04-21T08:56:59.518 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_watch": 0, 2024-04-21T08:56:59.518 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_notify": 0, 2024-04-21T08:56:59.518 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T08:56:59.518 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls": 0, 2024-04-21T08:56:59.518 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls_filter": 0, 2024-04-21T08:56:59.519 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_other": 65, 2024-04-21T08:56:59.519 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_active": 0, 2024-04-21T08:56:59.519 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_send": 0, 2024-04-21T08:56:59.519 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_resend": 0, 2024-04-21T08:56:59.519 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_ping": 0, 2024-04-21T08:56:59.519 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_active": 0, 2024-04-21T08:56:59.519 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_send": 0, 2024-04-21T08:56:59.519 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_resend": 0, 2024-04-21T08:56:59.519 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_active": 0, 2024-04-21T08:56:59.520 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_send": 0, 2024-04-21T08:56:59.520 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_resend": 0, 2024-04-21T08:56:59.520 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_active": 0, 2024-04-21T08:56:59.520 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_send": 0, 2024-04-21T08:56:59.520 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_resend": 0, 2024-04-21T08:56:59.520 INFO:teuthology.orchestra.run.smithi016.stdout: "command_active": 0, 2024-04-21T08:56:59.520 INFO:teuthology.orchestra.run.smithi016.stdout: "command_send": 0, 2024-04-21T08:56:59.520 INFO:teuthology.orchestra.run.smithi016.stdout: "command_resend": 0, 2024-04-21T08:56:59.520 INFO:teuthology.orchestra.run.smithi016.stdout: "map_epoch": 113, 2024-04-21T08:56:59.520 INFO:teuthology.orchestra.run.smithi016.stdout: "map_full": 0, 2024-04-21T08:56:59.521 INFO:teuthology.orchestra.run.smithi016.stdout: "map_inc": 28, 2024-04-21T08:56:59.521 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_sessions": 12, 2024-04-21T08:56:59.521 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_open": 12, 2024-04-21T08:56:59.521 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_close": 0, 2024-04-21T08:56:59.521 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_laggy": 0, 2024-04-21T08:56:59.521 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_wr": 38594, 2024-04-21T08:56:59.521 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_rd": 0, 2024-04-21T08:56:59.521 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_del": 587 2024-04-21T08:56:59.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.522 INFO:teuthology.orchestra.run.smithi016.stdout: "oft": { 2024-04-21T08:56:59.522 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_objs": 1, 2024-04-21T08:56:59.522 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_kv_pairs": 0, 2024-04-21T08:56:59.522 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_updates": 0, 2024-04-21T08:56:59.522 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_removes": 0 2024-04-21T08:56:59.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.522 INFO:teuthology.orchestra.run.smithi016.stdout: "osd-slow-ops": { 2024-04-21T08:56:59.522 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_ops_count": 19454 2024-04-21T08:56:59.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.523 INFO:teuthology.orchestra.run.smithi016.stdout: "purge_queue": { 2024-04-21T08:56:59.523 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops": 0, 2024-04-21T08:56:59.523 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops_high_water": 49, 2024-04-21T08:56:59.523 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing": 0, 2024-04-21T08:56:59.523 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_high_water": 38, 2024-04-21T08:56:59.523 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed_ops": 2006, 2024-04-21T08:56:59.523 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed": 1056, 2024-04-21T08:56:59.523 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_item_in_journal": 0 2024-04-21T08:56:59.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.524 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:59.524 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-21T08:56:59.524 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-21T08:56:59.524 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-21T08:56:59.524 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 937776, 2024-04-21T08:56:59.524 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 407434037, 2024-04-21T08:56:59.524 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:59.524 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 937776, 2024-04-21T08:56:59.525 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-21T08:56:59.525 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-21T08:56:59.525 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 937776, 2024-04-21T08:56:59.525 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 407434037, 2024-04-21T08:56:59.525 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-21T08:56:59.525 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.525 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.525 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.525 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:56:59.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.526 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_bytes": { 2024-04-21T08:56:59.526 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-21T08:56:59.526 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-21T08:56:59.526 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-21T08:56:59.526 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-21T08:56:59.526 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-21T08:56:59.526 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:59.526 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-21T08:56:59.527 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 685668, 2024-04-21T08:56:59.527 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 865577624, 2024-04-21T08:56:59.527 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 685668, 2024-04-21T08:56:59.527 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 865577624, 2024-04-21T08:56:59.527 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-21T08:56:59.527 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.527 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.528 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.528 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:56:59.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.528 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_ops": { 2024-04-21T08:56:59.528 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-21T08:56:59.528 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 1024, 2024-04-21T08:56:59.528 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-21T08:56:59.528 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-21T08:56:59.528 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-21T08:56:59.529 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:59.529 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-21T08:56:59.529 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 685668, 2024-04-21T08:56:59.529 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 685668, 2024-04-21T08:56:59.529 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 685668, 2024-04-21T08:56:59.529 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 685668, 2024-04-21T08:56:59.529 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-21T08:56:59.529 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.529 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.530 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.530 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:56:59.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.530 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle": { 2024-04-21T08:56:59.530 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-21T08:56:59.530 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-04-21T08:56:59.530 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-21T08:56:59.530 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 1056, 2024-04-21T08:56:59.530 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 114576, 2024-04-21T08:56:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 1056, 2024-04-21T08:56:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-21T08:56:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-21T08:56:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 75, 2024-04-21T08:56:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 114576, 2024-04-21T08:56:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-21T08:56:59.531 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:56:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:56:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle-0x559a3246a460": { 2024-04-21T08:56:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-21T08:56:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-04-21T08:56:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-21T08:56:59.532 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 233562, 2024-04-21T08:56:59.533 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 616219274, 2024-04-21T08:56:59.533 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:59.533 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 233562, 2024-04-21T08:56:59.533 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-21T08:56:59.533 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-21T08:56:59.533 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 16031, 2024-04-21T08:56:59.533 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 616219274, 2024-04-21T08:56:59.533 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-21T08:56:59.534 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:56:59.534 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.534 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.534 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:56:59.534 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:56:59.534 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-21T08:57:00.401 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-21T08:57:00.401 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=12 2024-04-21T08:57:00.402 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=6 2024-04-21T08:57:00.402 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=206 2024-04-21T08:57:00.402 DEBUG:tasks.check_counter:Getting stats from e 2024-04-21T08:57:00.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-21T08:57:00.656 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:57:00.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:00 smithi007 ceph-mon[28477]: pgmap v390: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:57:00.953 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:00 smithi016 ceph-mon[31259]: pgmap v390: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:57:01.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:00 smithi077 ceph-mon[31615]: pgmap v390: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:57:02.737 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-21T08:57:02.737 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:57:02.737 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 4, 2024-04-21T08:57:02.737 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 4, 2024-04-21T08:57:02.737 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:57:02.738 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 448, 2024-04-21T08:57:02.738 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-04-21T08:57:02.738 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-04-21T08:57:02.738 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001702974, 2024-04-21T08:57:02.738 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000039418, 2024-04-21T08:57:02.738 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000582744, 2024-04-21T08:57:02.738 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:02.738 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:02.739 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3, 2024-04-21T08:57:02.739 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000803566, 2024-04-21T08:57:02.739 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000267855 2024-04-21T08:57:02.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:02.739 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:02.739 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:02.739 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:02.739 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:02.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:02.740 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:57:02.740 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:57:02.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:02.740 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:57:02.740 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2, 2024-04-21T08:57:02.740 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 187, 2024-04-21T08:57:02.740 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:57:02.740 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 188896, 2024-04-21T08:57:02.741 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-04-21T08:57:02.741 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-04-21T08:57:02.741 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.010786024, 2024-04-21T08:57:02.741 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.006887822, 2024-04-21T08:57:02.741 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001739147, 2024-04-21T08:57:02.742 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:02.742 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:02.742 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 184, 2024-04-21T08:57:02.743 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.010996924, 2024-04-21T08:57:02.743 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000059765 2024-04-21T08:57:02.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:02.743 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:02.743 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:02.743 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:02.743 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:02.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:02.743 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:57:02.744 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 188896 2024-04-21T08:57:02.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:02.744 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:57:02.744 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 255, 2024-04-21T08:57:02.744 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 244, 2024-04-21T08:57:02.744 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1599312, 2024-04-21T08:57:02.744 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 150160, 2024-04-21T08:57:02.744 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-04-21T08:57:02.744 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-04-21T08:57:02.745 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.044110559, 2024-04-21T08:57:02.745 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.012853885, 2024-04-21T08:57:02.745 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.034442592, 2024-04-21T08:57:02.745 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.005865746, 2024-04-21T08:57:02.745 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:02.745 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 243, 2024-04-21T08:57:02.745 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.012498052, 2024-04-21T08:57:02.745 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000051432 2024-04-21T08:57:02.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:02.746 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:02.746 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:02.746 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:02.746 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:02.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:02.746 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1599312, 2024-04-21T08:57:02.746 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 150160 2024-04-21T08:57:02.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:02.746 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-21T08:57:02.747 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-04-21T08:57:02.747 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-21T08:57:02.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:02.747 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-21T08:57:02.747 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:57:02.747 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-21T08:57:02.747 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:57:02.747 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:57:02.747 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:57:02.748 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:57:02.748 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:57:02.748 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:57:02.748 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:57:02.748 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:57:02.748 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:57:02.748 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:57:02.748 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:57:02.748 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:57:02.749 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:57:02.749 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-21T08:57:02.749 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:57:02.749 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-21T08:57:02.749 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:57:02.749 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:57:02.749 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:57:02.749 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:57:02.750 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:57:02.750 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:57:02.750 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:57:02.750 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-21T08:57:02.750 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:57:02.750 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:57:02.750 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:57:02.750 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-21T08:57:02.751 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-21T08:57:02.751 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-21T08:57:02.751 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:57:02.751 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:57:02.751 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:57:02.751 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:57:02.751 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148977, 2024-04-21T08:57:02.751 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 30, 2024-04-21T08:57:02.751 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:57:02.752 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-21T08:57:02.752 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-21T08:57:02.752 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-21T08:57:02.752 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:57:02.752 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-21T08:57:02.752 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:57:02.752 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-21T08:57:02.752 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-04-21T08:57:02.752 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-04-21T08:57:02.753 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:57:02.753 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:57:02.753 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-21T08:57:02.753 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-21T08:57:02.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-04-21T08:57:02.753 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-04-21T08:57:02.753 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-21T08:57:02.753 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-21T08:57:02.754 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-21T08:57:02.754 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-21T08:57:02.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:02.754 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:57:02.754 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-21T08:57:02.754 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-21T08:57:02.754 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-21T08:57:02.754 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 257, 2024-04-21T08:57:02.755 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1563342, 2024-04-21T08:57:02.755 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:02.755 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 257, 2024-04-21T08:57:02.755 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-21T08:57:02.755 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-21T08:57:02.755 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 257, 2024-04-21T08:57:02.755 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1563342, 2024-04-21T08:57:02.755 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-21T08:57:02.755 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:02.756 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:02.756 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:02.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:57:02.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:57:02.756 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-21T08:57:02.906 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:02 smithi077 ceph-mon[31615]: pgmap v391: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 22 KiB/s wr, 3 op/s 2024-04-21T08:57:03.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:02 smithi016 ceph-mon[31259]: pgmap v391: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 22 KiB/s wr, 3 op/s 2024-04-21T08:57:03.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:02 smithi007 ceph-mon[28477]: pgmap v391: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 22 KiB/s wr, 3 op/s 2024-04-21T08:57:03.597 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-04-21T08:57:03.597 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-04-21T08:57:03.598 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-04-21T08:57:03.598 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2024-04-21T08:57:03.598 DEBUG:tasks.check_counter:Getting stats from h 2024-04-21T08:57:03.598 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-21T08:57:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:57:05.019 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:04 smithi016 ceph-mon[31259]: pgmap v392: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 19 op/s 2024-04-21T08:57:05.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:04 smithi077 ceph-mon[31615]: pgmap v392: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 19 op/s 2024-04-21T08:57:05.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:04 smithi007 ceph-mon[28477]: pgmap v392: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 19 op/s 2024-04-21T08:57:05.810 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-21T08:57:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:57:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-04-21T08:57:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-04-21T08:57:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:57:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-04-21T08:57:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-04-21T08:57:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-04-21T08:57:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001625871, 2024-04-21T08:57:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:57:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000446535, 2024-04-21T08:57:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:57:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:57:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:57:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 254, 2024-04-21T08:57:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 243, 2024-04-21T08:57:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1599040, 2024-04-21T08:57:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 149520, 2024-04-21T08:57:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-04-21T08:57:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-04-21T08:57:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.043506000, 2024-04-21T08:57:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.012671297, 2024-04-21T08:57:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.033910108, 2024-04-21T08:57:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.005610182, 2024-04-21T08:57:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 242, 2024-04-21T08:57:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.010879876, 2024-04-21T08:57:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000044958 2024-04-21T08:57:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1599040, 2024-04-21T08:57:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 149520 2024-04-21T08:57:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:57:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 6, 2024-04-21T08:57:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 187, 2024-04-21T08:57:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3264, 2024-04-21T08:57:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 188992, 2024-04-21T08:57:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-04-21T08:57:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-04-21T08:57:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.010130595, 2024-04-21T08:57:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.006499165, 2024-04-21T08:57:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001838221, 2024-04-21T08:57:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 185, 2024-04-21T08:57:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.019878791, 2024-04-21T08:57:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000107452 2024-04-21T08:57:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-04-21T08:57:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 188992 2024-04-21T08:57:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-21T08:57:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-04-21T08:57:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-21T08:57:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-21T08:57:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:57:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-21T08:57:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:57:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:57:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:57:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:57:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:57:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:57:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:57:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:57:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:57:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:57:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:57:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:57:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:57:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-21T08:57:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:57:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-21T08:57:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:57:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:57:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:57:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:57:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:57:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:57:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:57:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-21T08:57:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:57:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:57:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:57:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-21T08:57:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-21T08:57:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-21T08:57:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:57:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:57:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:57:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:57:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148994, 2024-04-21T08:57:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 31, 2024-04-21T08:57:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:57:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-21T08:57:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-21T08:57:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-21T08:57:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:57:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-21T08:57:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:57:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-21T08:57:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-04-21T08:57:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-04-21T08:57:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:57:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:57:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-21T08:57:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-21T08:57:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-04-21T08:57:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-04-21T08:57:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-21T08:57:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-21T08:57:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-21T08:57:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-21T08:57:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:57:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-21T08:57:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-21T08:57:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-21T08:57:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 256, 2024-04-21T08:57:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1563212, 2024-04-21T08:57:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 256, 2024-04-21T08:57:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-21T08:57:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-21T08:57:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 256, 2024-04-21T08:57:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1563212, 2024-04-21T08:57:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-21T08:57:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:57:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:57:05.827 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-21T08:57:06.566 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2024-04-21T08:57:06.566 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2024-04-21T08:57:06.566 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2024-04-21T08:57:06.567 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.h 2024-04-21T08:57:06.567 DEBUG:tasks.check_counter:Getting stats from k 2024-04-21T08:57:06.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-21T08:57:06.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:06 smithi016 ceph-mon[31259]: pgmap v393: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 18 op/s 2024-04-21T08:57:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:57:07.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:06 smithi077 ceph-mon[31615]: pgmap v393: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 18 op/s 2024-04-21T08:57:07.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:06 smithi007 ceph-mon[28477]: pgmap v393: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 18 op/s 2024-04-21T08:57:08.800 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:08 smithi016 ceph-mon[31259]: pgmap v394: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 413 KiB/s wr, 20 op/s 2024-04-21T08:57:08.800 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:08 smithi016 ceph-mon[31259]: osdmap e114: 12 total, 12 up, 12 in 2024-04-21T08:57:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:08 smithi077 ceph-mon[31615]: pgmap v394: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 413 KiB/s wr, 20 op/s 2024-04-21T08:57:09.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:08 smithi077 ceph-mon[31615]: osdmap e114: 12 total, 12 up, 12 in 2024-04-21T08:57:09.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:08 smithi007 ceph-mon[28477]: pgmap v394: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 413 KiB/s wr, 20 op/s 2024-04-21T08:57:09.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:08 smithi007 ceph-mon[28477]: osdmap e114: 12 total, 12 up, 12 in 2024-04-21T08:57:09.620 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-21T08:57:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:57:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 4, 2024-04-21T08:57:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 4, 2024-04-21T08:57:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:57:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 448, 2024-04-21T08:57:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-04-21T08:57:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-04-21T08:57:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001877451, 2024-04-21T08:57:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000042306, 2024-04-21T08:57:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000819184, 2024-04-21T08:57:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3, 2024-04-21T08:57:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000755608, 2024-04-21T08:57:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000251869 2024-04-21T08:57:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:57:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:57:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:57:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2, 2024-04-21T08:57:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 189, 2024-04-21T08:57:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:57:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 189248, 2024-04-21T08:57:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-04-21T08:57:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-04-21T08:57:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.010666059, 2024-04-21T08:57:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.006562353, 2024-04-21T08:57:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001929705, 2024-04-21T08:57:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 186, 2024-04-21T08:57:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.012143174, 2024-04-21T08:57:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000065285 2024-04-21T08:57:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:57:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 189248 2024-04-21T08:57:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:57:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 259, 2024-04-21T08:57:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 248, 2024-04-21T08:57:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1600400, 2024-04-21T08:57:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 152720, 2024-04-21T08:57:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-04-21T08:57:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-04-21T08:57:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.044572590, 2024-04-21T08:57:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.012464467, 2024-04-21T08:57:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.035817363, 2024-04-21T08:57:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.005770929, 2024-04-21T08:57:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 247, 2024-04-21T08:57:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.011391889, 2024-04-21T08:57:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000046121 2024-04-21T08:57:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1600400, 2024-04-21T08:57:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 152720 2024-04-21T08:57:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-21T08:57:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-04-21T08:57:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-21T08:57:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-21T08:57:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:57:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-21T08:57:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:57:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:57:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:57:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:57:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:57:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:57:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:57:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:57:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:57:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:57:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:57:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:57:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:57:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-21T08:57:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:57:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-21T08:57:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:57:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:57:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:57:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:57:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:57:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:57:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:57:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-21T08:57:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:57:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:57:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:57:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-21T08:57:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-21T08:57:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-21T08:57:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:57:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:57:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:57:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:57:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 150008, 2024-04-21T08:57:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 31, 2024-04-21T08:57:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:57:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-21T08:57:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-21T08:57:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-21T08:57:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:57:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-21T08:57:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:57:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-21T08:57:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-04-21T08:57:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-04-21T08:57:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:57:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:57:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-21T08:57:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-21T08:57:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-04-21T08:57:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-04-21T08:57:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-21T08:57:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-21T08:57:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-21T08:57:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-21T08:57:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-21T08:57:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:57:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-21T08:57:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-21T08:57:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-21T08:57:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 261, 2024-04-21T08:57:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1563862, 2024-04-21T08:57:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 261, 2024-04-21T08:57:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-21T08:57:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-21T08:57:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 261, 2024-04-21T08:57:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1563862, 2024-04-21T08:57:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-21T08:57:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-21T08:57:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-21T08:57:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-21T08:57:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:57:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-21T08:57:09.639 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-21T08:57:10.412 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-21T08:57:10.412 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-21T08:57:10.412 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-21T08:57:10.412 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-04-21T08:57:10.413 DEBUG:tasks.check_counter:Getting stats from c 2024-04-21T08:57:10.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-21T08:57:10.763 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:57:10.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:10 smithi077 ceph-mon[31615]: pgmap v396: 129 pgs: 9 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 95 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 25 op/s 2024-04-21T08:57:11.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:10 smithi016 ceph-mon[31259]: pgmap v396: 129 pgs: 9 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 95 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 25 op/s 2024-04-21T08:57:11.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:10 smithi007 ceph-mon[28477]: pgmap v396: 129 pgs: 9 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 95 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 25 op/s 2024-04-21T08:57:12.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:11 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:57:12.086 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:11 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:57:12.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:11 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:57:12.768 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-21T08:57:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:57:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 0, 2024-04-21T08:57:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 2, 2024-04-21T08:57:12.768 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:57:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 0, 2024-04-21T08:57:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2024-04-21T08:57:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2024-04-21T08:57:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001303538, 2024-04-21T08:57:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:57:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000427277, 2024-04-21T08:57:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:12.769 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:12.770 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:57:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:57:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:57:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 2, 2024-04-21T08:57:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 188, 2024-04-21T08:57:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:57:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 189072, 2024-04-21T08:57:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 5, 2024-04-21T08:57:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 3, 2024-04-21T08:57:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.010907352, 2024-04-21T08:57:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.006979070, 2024-04-21T08:57:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001786622, 2024-04-21T08:57:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 185, 2024-04-21T08:57:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.019562357, 2024-04-21T08:57:12.772 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000105742 2024-04-21T08:57:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:57:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 189072 2024-04-21T08:57:12.773 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:57:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 260, 2024-04-21T08:57:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 247, 2024-04-21T08:57:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 1602528, 2024-04-21T08:57:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 151248, 2024-04-21T08:57:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2024-04-21T08:57:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2024-04-21T08:57:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.045008733, 2024-04-21T08:57:12.774 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.011341281, 2024-04-21T08:57:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.033949543, 2024-04-21T08:57:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.006470172, 2024-04-21T08:57:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 247, 2024-04-21T08:57:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.014536834, 2024-04-21T08:57:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000058853 2024-04-21T08:57:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:12.775 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 1602528, 2024-04-21T08:57:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 151248 2024-04-21T08:57:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-04-21T08:57:12.776 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2024-04-21T08:57:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-04-21T08:57:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-04-21T08:57:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:57:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-04-21T08:57:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:57:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:57:12.777 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:57:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:57:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:57:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:57:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:57:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:57:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:57:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:57:12.778 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:57:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:57:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:57:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-04-21T08:57:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:57:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-04-21T08:57:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:57:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:57:12.779 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:57:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:57:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:57:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:57:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:57:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-04-21T08:57:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:57:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:57:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:57:12.780 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-04-21T08:57:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-04-21T08:57:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-04-21T08:57:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:57:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:57:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:57:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:57:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 150008, 2024-04-21T08:57:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 31, 2024-04-21T08:57:12.781 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:57:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-04-21T08:57:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-04-21T08:57:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-04-21T08:57:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:57:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-04-21T08:57:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:57:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-04-21T08:57:12.782 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2024-04-21T08:57:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2024-04-21T08:57:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:57:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:57:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-04-21T08:57:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-04-21T08:57:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2024-04-21T08:57:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2024-04-21T08:57:12.783 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-04-21T08:57:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-04-21T08:57:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-04-21T08:57:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-04-21T08:57:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:57:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-04-21T08:57:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-04-21T08:57:12.784 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-04-21T08:57:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 258, 2024-04-21T08:57:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 1563472, 2024-04-21T08:57:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 258, 2024-04-21T08:57:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-04-21T08:57:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-04-21T08:57:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 258, 2024-04-21T08:57:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 1563472, 2024-04-21T08:57:12.785 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-04-21T08:57:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:12.786 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:12.786 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-21T08:57:12.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:12 smithi016 ceph-mon[31259]: pgmap v397: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 23 op/s 2024-04-21T08:57:12.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:12 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:57:12.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:12 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:57:12.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:12 smithi016 ceph-mon[31259]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:57:13.097 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:12 smithi077 ceph-mon[31615]: pgmap v397: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 23 op/s 2024-04-21T08:57:13.097 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:12 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:57:13.097 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:12 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:57:13.098 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:12 smithi077 ceph-mon[31615]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:57:13.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:12 smithi007 ceph-mon[28477]: pgmap v397: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 23 op/s 2024-04-21T08:57:13.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:12 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:57:13.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:12 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:57:13.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:12 smithi007 ceph-mon[28477]: from='mgr.15408 ' entity='mgr.x' 2024-04-21T08:57:13.531 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.c 2024-04-21T08:57:13.531 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.c 2024-04-21T08:57:13.531 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.c 2024-04-21T08:57:13.531 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.c 2024-04-21T08:57:13.531 DEBUG:tasks.check_counter:Getting stats from f 2024-04-21T08:57:13.532 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-21T08:57:14.023 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:57:14.788 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:14 smithi077 ceph-mon[31615]: pgmap v398: 129 pgs: 18 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 66 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 8 op/s 2024-04-21T08:57:15.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:14 smithi016 ceph-mon[31259]: pgmap v398: 129 pgs: 18 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 66 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 8 op/s 2024-04-21T08:57:15.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:14 smithi007 ceph-mon[28477]: pgmap v398: 129 pgs: 18 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 66 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 8 op/s 2024-04-21T08:57:16.458 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-21T08:57:16.458 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:57:16.459 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 258, 2024-04-21T08:57:16.459 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 247, 2024-04-21T08:57:16.459 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 1600128, 2024-04-21T08:57:16.459 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 152080, 2024-04-21T08:57:16.459 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2024-04-21T08:57:16.459 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2024-04-21T08:57:16.459 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.047985272, 2024-04-21T08:57:16.460 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.014076158, 2024-04-21T08:57:16.460 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.036404154, 2024-04-21T08:57:16.460 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.006100237, 2024-04-21T08:57:16.460 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:16.460 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 246, 2024-04-21T08:57:16.460 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.011886537, 2024-04-21T08:57:16.460 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000048319 2024-04-21T08:57:16.460 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:16.461 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:16.461 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:16.461 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:16.461 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:16.461 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:16.461 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 1600128, 2024-04-21T08:57:16.461 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 152080 2024-04-21T08:57:16.461 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:16.461 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:57:16.462 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 6, 2024-04-21T08:57:16.462 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 190, 2024-04-21T08:57:16.462 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 3264, 2024-04-21T08:57:16.462 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 189520, 2024-04-21T08:57:16.462 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 5, 2024-04-21T08:57:16.462 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 3, 2024-04-21T08:57:16.462 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.014551829, 2024-04-21T08:57:16.462 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.007930188, 2024-04-21T08:57:16.463 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001898346, 2024-04-21T08:57:16.463 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:16.463 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:16.463 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 188, 2024-04-21T08:57:16.463 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.015689873, 2024-04-21T08:57:16.463 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000083456 2024-04-21T08:57:16.463 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:16.463 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:16.463 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:16.464 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:16.464 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:16.464 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:16.464 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-04-21T08:57:16.464 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 189520 2024-04-21T08:57:16.464 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:16.464 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:57:16.464 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 0, 2024-04-21T08:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 2, 2024-04-21T08:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 0, 2024-04-21T08:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2024-04-21T08:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2024-04-21T08:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.000359849, 2024-04-21T08:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000290387, 2024-04-21T08:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:16.466 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:16.466 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:16.466 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:16.466 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:16.466 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:16.466 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:16.466 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:16.466 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:16.467 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:16.467 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:57:16.467 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:57:16.467 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:16.467 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-04-21T08:57:16.467 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2024-04-21T08:57:16.467 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-04-21T08:57:16.467 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:16.467 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-04-21T08:57:16.468 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:57:16.468 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-04-21T08:57:16.468 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:57:16.468 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:57:16.468 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:57:16.468 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:57:16.468 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:57:16.468 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:57:16.468 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:57:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:57:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:57:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:57:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:57:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:57:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:57:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-04-21T08:57:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:57:16.469 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-04-21T08:57:16.470 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:57:16.470 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:57:16.470 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:57:16.470 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:57:16.470 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:57:16.470 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:57:16.470 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:57:16.470 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-04-21T08:57:16.471 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:57:16.471 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:57:16.471 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:57:16.471 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-04-21T08:57:16.471 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-04-21T08:57:16.471 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-04-21T08:57:16.471 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:57:16.471 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:57:16.471 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:57:16.471 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:57:16.472 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 149024, 2024-04-21T08:57:16.472 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 31, 2024-04-21T08:57:16.472 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:57:16.472 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-04-21T08:57:16.472 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-04-21T08:57:16.472 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-04-21T08:57:16.472 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:57:16.472 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-04-21T08:57:16.472 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:57:16.473 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-04-21T08:57:16.473 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2024-04-21T08:57:16.473 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2024-04-21T08:57:16.473 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:57:16.473 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:57:16.473 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-04-21T08:57:16.473 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-04-21T08:57:16.473 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2024-04-21T08:57:16.473 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2024-04-21T08:57:16.473 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-04-21T08:57:16.474 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-04-21T08:57:16.474 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-04-21T08:57:16.474 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-04-21T08:57:16.474 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:16.474 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:57:16.474 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-04-21T08:57:16.474 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-04-21T08:57:16.474 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-04-21T08:57:16.475 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 260, 2024-04-21T08:57:16.475 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 1563732, 2024-04-21T08:57:16.475 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:16.475 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 260, 2024-04-21T08:57:16.475 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-04-21T08:57:16.475 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-04-21T08:57:16.475 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 260, 2024-04-21T08:57:16.475 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 1563732, 2024-04-21T08:57:16.475 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-04-21T08:57:16.475 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:16.476 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:16.476 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:16.476 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:16.476 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:16.476 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-21T08:57:16.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:16 smithi077 ceph-mon[31615]: pgmap v399: 129 pgs: 11 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 99 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 16 op/s 2024-04-21T08:57:17.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:16 smithi016 ceph-mon[31259]: pgmap v399: 129 pgs: 11 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 99 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 16 op/s 2024-04-21T08:57:17.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:16 smithi007 ceph-mon[28477]: pgmap v399: 129 pgs: 11 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 99 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 16 op/s 2024-04-21T08:57:17.262 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-04-21T08:57:17.263 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-04-21T08:57:17.263 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-04-21T08:57:17.263 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2024-04-21T08:57:17.263 DEBUG:tasks.check_counter:Getting stats from i 2024-04-21T08:57:17.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-21T08:57:17.518 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:57:18.938 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:18 smithi077 ceph-mon[31615]: pgmap v400: 129 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 118 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 18 op/s 2024-04-21T08:57:18.954 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:18 smithi007 ceph-mon[28477]: pgmap v400: 129 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 118 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 18 op/s 2024-04-21T08:57:19.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:18 smithi016 ceph-mon[31259]: pgmap v400: 129 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 118 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 18 op/s 2024-04-21T08:57:19.202 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-21T08:57:19.203 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:57:19.203 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 1495, 2024-04-21T08:57:19.203 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 1495, 2024-04-21T08:57:19.203 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 239551209, 2024-04-21T08:57:19.203 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 474596, 2024-04-21T08:57:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 8, 2024-04-21T08:57:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 4, 2024-04-21T08:57:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.253751895, 2024-04-21T08:57:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.038048580, 2024-04-21T08:57:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.158844695, 2024-04-21T08:57:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.044677514, 2024-04-21T08:57:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1494, 2024-04-21T08:57:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.075041179, 2024-04-21T08:57:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000050228 2024-04-21T08:57:19.205 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.205 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:19.205 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.205 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.205 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.205 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.205 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:57:19.205 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:57:19.205 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.206 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:57:19.206 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 1121, 2024-04-21T08:57:19.206 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 1119, 2024-04-21T08:57:19.206 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 267542914, 2024-04-21T08:57:19.206 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 405964, 2024-04-21T08:57:19.206 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 6, 2024-04-21T08:57:19.206 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 5, 2024-04-21T08:57:19.206 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.241091985, 2024-04-21T08:57:19.207 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.030823251, 2024-04-21T08:57:19.207 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.173473435, 2024-04-21T08:57:19.207 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.030688012, 2024-04-21T08:57:19.207 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:19.207 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 1118, 2024-04-21T08:57:19.207 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.069401673, 2024-04-21T08:57:19.207 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000062076 2024-04-21T08:57:19.207 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.207 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:19.208 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.208 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.208 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.208 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.208 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 358384, 2024-04-21T08:57:19.208 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 114928 2024-04-21T08:57:19.208 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.208 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:57:19.208 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 539, 2024-04-21T08:57:19.208 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 675, 2024-04-21T08:57:19.209 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 167942179, 2024-04-21T08:57:19.209 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 497309, 2024-04-21T08:57:19.209 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 6, 2024-04-21T08:57:19.209 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 5, 2024-04-21T08:57:19.209 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.136949615, 2024-04-21T08:57:19.209 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.018283598, 2024-04-21T08:57:19.209 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.099206627, 2024-04-21T08:57:19.209 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.014840119, 2024-04-21T08:57:19.209 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:19.210 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 673, 2024-04-21T08:57:19.210 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.036109326, 2024-04-21T08:57:19.210 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000053654 2024-04-21T08:57:19.210 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.210 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:19.210 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.210 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.210 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.210 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.211 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-21T08:57:19.211 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 326224 2024-04-21T08:57:19.211 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.211 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-04-21T08:57:19.211 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 1, 2024-04-21T08:57:19.211 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-04-21T08:57:19.211 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.211 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-MDSRank": { 2024-04-21T08:57:19.211 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-04-21T08:57:19.212 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-04-21T08:57:19.212 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 4499, 2024-04-21T08:57:19.212 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 1.418939584, 2024-04-21T08:57:19.212 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000315389 2024-04-21T08:57:19.212 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:19.212 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.212 INFO:teuthology.orchestra.run.smithi077.stdout: "finisher-PurgeQueue": { 2024-04-21T08:57:19.213 INFO:teuthology.orchestra.run.smithi077.stdout: "queue_len": 0, 2024-04-21T08:57:19.213 INFO:teuthology.orchestra.run.smithi077.stdout: "complete_latency": { 2024-04-21T08:57:19.213 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.213 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.213 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.213 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:19.213 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.213 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-04-21T08:57:19.213 INFO:teuthology.orchestra.run.smithi077.stdout: "request": 0, 2024-04-21T08:57:19.214 INFO:teuthology.orchestra.run.smithi077.stdout: "reply": 0, 2024-04-21T08:57:19.214 INFO:teuthology.orchestra.run.smithi077.stdout: "reply_latency": { 2024-04-21T08:57:19.214 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.214 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.214 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.214 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.214 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_reply": 0, 2024-04-21T08:57:19.214 INFO:teuthology.orchestra.run.smithi077.stdout: "forward": 0, 2024-04-21T08:57:19.215 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_complete": 0, 2024-04-21T08:57:19.215 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_fetch_keys": 0, 2024-04-21T08:57:19.215 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_commit": 0, 2024-04-21T08:57:19.215 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_split": 0, 2024-04-21T08:57:19.215 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_merge": 0, 2024-04-21T08:57:19.215 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes": 108346, 2024-04-21T08:57:19.216 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_top": 71406, 2024-04-21T08:57:19.216 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_bottom": 36940, 2024-04-21T08:57:19.216 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pin_tail": 0, 2024-04-21T08:57:19.216 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_pinned": 16425, 2024-04-21T08:57:19.216 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_expired": 0, 2024-04-21T08:57:19.216 INFO:teuthology.orchestra.run.smithi077.stdout: "inodes_with_caps": 0, 2024-04-21T08:57:19.216 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": 0, 2024-04-21T08:57:19.216 INFO:teuthology.orchestra.run.smithi077.stdout: "subtrees": 2, 2024-04-21T08:57:19.216 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse": 0, 2024-04-21T08:57:19.217 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_hit": 0, 2024-04-21T08:57:19.217 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_forward": 0, 2024-04-21T08:57:19.217 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_discover": 0, 2024-04-21T08:57:19.217 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_dir_fetch": 0, 2024-04-21T08:57:19.217 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_remote_ino": 0, 2024-04-21T08:57:19.217 INFO:teuthology.orchestra.run.smithi077.stdout: "traverse_lock": 0, 2024-04-21T08:57:19.217 INFO:teuthology.orchestra.run.smithi077.stdout: "load_cent": 0, 2024-04-21T08:57:19.217 INFO:teuthology.orchestra.run.smithi077.stdout: "q": 0, 2024-04-21T08:57:19.218 INFO:teuthology.orchestra.run.smithi077.stdout: "exported": 0, 2024-04-21T08:57:19.218 INFO:teuthology.orchestra.run.smithi077.stdout: "exported_inodes": 0, 2024-04-21T08:57:19.218 INFO:teuthology.orchestra.run.smithi077.stdout: "imported": 0, 2024-04-21T08:57:19.218 INFO:teuthology.orchestra.run.smithi077.stdout: "imported_inodes": 0, 2024-04-21T08:57:19.218 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_dir_fetch": 0, 2024-04-21T08:57:19.218 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_backtrace_fetch": 0, 2024-04-21T08:57:19.219 INFO:teuthology.orchestra.run.smithi077.stdout: "openino_peer_discover": 0, 2024-04-21T08:57:19.219 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rfiles": 84661, 2024-04-21T08:57:19.219 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rbytes": 1548381344, 2024-04-21T08:57:19.219 INFO:teuthology.orchestra.run.smithi077.stdout: "root_rsnaps": 6, 2024-04-21T08:57:19.219 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T08:57:19.219 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_set_tag": 0, 2024-04-21T08:57:19.219 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T08:57:19.219 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_inotable_repaired": 0, 2024-04-21T08:57:19.219 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_inodes": 0, 2024-04-21T08:57:19.220 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T08:57:19.220 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T08:57:19.220 INFO:teuthology.orchestra.run.smithi077.stdout: "scrub_file_inodes": 0, 2024-04-21T08:57:19.220 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_inode_file_caps": 0, 2024-04-21T08:57:19.220 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T08:57:19.220 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_grant": 0, 2024-04-21T08:57:19.220 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T08:57:19.220 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T08:57:19.220 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T08:57:19.221 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps": 0, 2024-04-21T08:57:19.221 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_caps_dirty": 0, 2024-04-21T08:57:19.221 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_cap_release": 0, 2024-04-21T08:57:19.221 INFO:teuthology.orchestra.run.smithi077.stdout: "process_request_cap_release": 0 2024-04-21T08:57:19.221 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.221 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_cache": { 2024-04-21T08:57:19.221 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update": 0, 2024-04-21T08:57:19.221 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_update_receipt": 0, 2024-04-21T08:57:19.221 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_try_discover": 0, 2024-04-21T08:57:19.222 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_send_discover": 0, 2024-04-21T08:57:19.222 INFO:teuthology.orchestra.run.smithi077.stdout: "dir_handle_discover": 0, 2024-04-21T08:57:19.222 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays": 0, 2024-04-21T08:57:19.222 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_delayed": 0, 2024-04-21T08:57:19.222 INFO:teuthology.orchestra.run.smithi077.stdout: "num_strays_enqueuing": 0, 2024-04-21T08:57:19.222 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_created": 0, 2024-04-21T08:57:19.222 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_enqueued": 0, 2024-04-21T08:57:19.222 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_reintegrated": 0, 2024-04-21T08:57:19.223 INFO:teuthology.orchestra.run.smithi077.stdout: "strays_migrated": 0, 2024-04-21T08:57:19.223 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_processing": 0, 2024-04-21T08:57:19.223 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_enqueued": 0, 2024-04-21T08:57:19.223 INFO:teuthology.orchestra.run.smithi077.stdout: "num_recovering_prioritized": 0, 2024-04-21T08:57:19.223 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_started": 0, 2024-04-21T08:57:19.223 INFO:teuthology.orchestra.run.smithi077.stdout: "recovery_completed": 0, 2024-04-21T08:57:19.223 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_path": 0, 2024-04-21T08:57:19.223 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_quiesce_inode": 0, 2024-04-21T08:57:19.224 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T08:57:19.224 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_exportdir": 0, 2024-04-21T08:57:19.224 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_flush": 0, 2024-04-21T08:57:19.224 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragmentdir": 0, 2024-04-21T08:57:19.224 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_fragstats": 0, 2024-04-21T08:57:19.224 INFO:teuthology.orchestra.run.smithi077.stdout: "ireq_inodestats": 0 2024-04-21T08:57:19.224 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.224 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_log": { 2024-04-21T08:57:19.224 INFO:teuthology.orchestra.run.smithi077.stdout: "evlrg": 0, 2024-04-21T08:57:19.224 INFO:teuthology.orchestra.run.smithi077.stdout: "evadd": 0, 2024-04-21T08:57:19.225 INFO:teuthology.orchestra.run.smithi077.stdout: "evex": 0, 2024-04-21T08:57:19.225 INFO:teuthology.orchestra.run.smithi077.stdout: "evtrm": 0, 2024-04-21T08:57:19.225 INFO:teuthology.orchestra.run.smithi077.stdout: "ev": 243405, 2024-04-21T08:57:19.225 INFO:teuthology.orchestra.run.smithi077.stdout: "evexg": 0, 2024-04-21T08:57:19.225 INFO:teuthology.orchestra.run.smithi077.stdout: "evexd": 0, 2024-04-21T08:57:19.225 INFO:teuthology.orchestra.run.smithi077.stdout: "segadd": 0, 2024-04-21T08:57:19.225 INFO:teuthology.orchestra.run.smithi077.stdout: "segex": 0, 2024-04-21T08:57:19.225 INFO:teuthology.orchestra.run.smithi077.stdout: "segtrm": 0, 2024-04-21T08:57:19.225 INFO:teuthology.orchestra.run.smithi077.stdout: "seg": 26, 2024-04-21T08:57:19.226 INFO:teuthology.orchestra.run.smithi077.stdout: "segmjr": 41, 2024-04-21T08:57:19.226 INFO:teuthology.orchestra.run.smithi077.stdout: "segexg": 0, 2024-04-21T08:57:19.226 INFO:teuthology.orchestra.run.smithi077.stdout: "segexd": 0, 2024-04-21T08:57:19.226 INFO:teuthology.orchestra.run.smithi077.stdout: "expos": 614683639, 2024-04-21T08:57:19.226 INFO:teuthology.orchestra.run.smithi077.stdout: "wrpos": 692060160, 2024-04-21T08:57:19.226 INFO:teuthology.orchestra.run.smithi077.stdout: "rdpos": 689520859, 2024-04-21T08:57:19.227 INFO:teuthology.orchestra.run.smithi077.stdout: "jlat": { 2024-04-21T08:57:19.227 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.227 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.227 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.227 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.227 INFO:teuthology.orchestra.run.smithi077.stdout: "replayed": 243405 2024-04-21T08:57:19.227 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.228 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_mem": { 2024-04-21T08:57:19.228 INFO:teuthology.orchestra.run.smithi077.stdout: "ino": 106336, 2024-04-21T08:57:19.228 INFO:teuthology.orchestra.run.smithi077.stdout: "ino+": 107392, 2024-04-21T08:57:19.228 INFO:teuthology.orchestra.run.smithi077.stdout: "ino-": 1056, 2024-04-21T08:57:19.228 INFO:teuthology.orchestra.run.smithi077.stdout: "dir": 6751, 2024-04-21T08:57:19.228 INFO:teuthology.orchestra.run.smithi077.stdout: "dir+": 7051, 2024-04-21T08:57:19.228 INFO:teuthology.orchestra.run.smithi077.stdout: "dir-": 300, 2024-04-21T08:57:19.228 INFO:teuthology.orchestra.run.smithi077.stdout: "dn": 108346, 2024-04-21T08:57:19.228 INFO:teuthology.orchestra.run.smithi077.stdout: "dn+": 108421, 2024-04-21T08:57:19.229 INFO:teuthology.orchestra.run.smithi077.stdout: "dn-": 75, 2024-04-21T08:57:19.229 INFO:teuthology.orchestra.run.smithi077.stdout: "cap": 0, 2024-04-21T08:57:19.229 INFO:teuthology.orchestra.run.smithi077.stdout: "cap+": 0, 2024-04-21T08:57:19.229 INFO:teuthology.orchestra.run.smithi077.stdout: "cap-": 0, 2024-04-21T08:57:19.229 INFO:teuthology.orchestra.run.smithi077.stdout: "rss": 463420, 2024-04-21T08:57:19.229 INFO:teuthology.orchestra.run.smithi077.stdout: "heap": 215416 2024-04-21T08:57:19.229 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.229 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_server": { 2024-04-21T08:57:19.229 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_client_request": 0, 2024-04-21T08:57:19.229 INFO:teuthology.orchestra.run.smithi077.stdout: "dispatch_server_request": 0, 2024-04-21T08:57:19.230 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_request": 0, 2024-04-21T08:57:19.230 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_client_session": 0, 2024-04-21T08:57:19.230 INFO:teuthology.orchestra.run.smithi077.stdout: "handle_peer_request": 0, 2024-04-21T08:57:19.230 INFO:teuthology.orchestra.run.smithi077.stdout: "req_create_latency": { 2024-04-21T08:57:19.230 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.230 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.230 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.230 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.231 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getattr_latency": { 2024-04-21T08:57:19.231 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.231 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.231 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.231 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.231 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getfilelock_latency": { 2024-04-21T08:57:19.231 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.231 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.231 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.232 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.232 INFO:teuthology.orchestra.run.smithi077.stdout: "req_link_latency": { 2024-04-21T08:57:19.232 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.232 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.232 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.232 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.233 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookup_latency": { 2024-04-21T08:57:19.233 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.233 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.233 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.233 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.233 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookuphash_latency": { 2024-04-21T08:57:19.233 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.233 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.233 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.234 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.234 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupino_latency": { 2024-04-21T08:57:19.234 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.234 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.234 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.234 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.234 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupname_latency": { 2024-04-21T08:57:19.234 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.235 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.235 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.235 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.235 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupparent_latency": { 2024-04-21T08:57:19.235 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.235 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.235 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.235 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.236 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lookupsnap_latency": { 2024-04-21T08:57:19.236 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.236 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.236 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.236 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.236 INFO:teuthology.orchestra.run.smithi077.stdout: "req_lssnap_latency": { 2024-04-21T08:57:19.236 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.236 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.237 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.237 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.237 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mkdir_latency": { 2024-04-21T08:57:19.237 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.237 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.237 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.238 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.238 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mknod_latency": { 2024-04-21T08:57:19.238 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.238 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.238 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.238 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.238 INFO:teuthology.orchestra.run.smithi077.stdout: "req_mksnap_latency": { 2024-04-21T08:57:19.238 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.239 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.239 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.239 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.239 INFO:teuthology.orchestra.run.smithi077.stdout: "req_open_latency": { 2024-04-21T08:57:19.239 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.239 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.239 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.239 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.240 INFO:teuthology.orchestra.run.smithi077.stdout: "req_readdir_latency": { 2024-04-21T08:57:19.240 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.240 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.240 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.240 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.240 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rename_latency": { 2024-04-21T08:57:19.240 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.240 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.240 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.240 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.241 INFO:teuthology.orchestra.run.smithi077.stdout: "req_renamesnap_latency": { 2024-04-21T08:57:19.241 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.241 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.241 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.241 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.241 INFO:teuthology.orchestra.run.smithi077.stdout: "req_snapdiff_latency": { 2024-04-21T08:57:19.241 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.242 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.242 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.242 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.242 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmdir_latency": { 2024-04-21T08:57:19.242 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.242 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.242 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.242 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.242 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmsnap_latency": { 2024-04-21T08:57:19.243 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.243 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.243 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.243 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.243 INFO:teuthology.orchestra.run.smithi077.stdout: "req_rmxattr_latency": { 2024-04-21T08:57:19.243 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.243 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.243 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.244 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.244 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setattr_latency": { 2024-04-21T08:57:19.244 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.244 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.244 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.244 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.244 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setdirlayout_latency": { 2024-04-21T08:57:19.244 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.244 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.244 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.245 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.245 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setfilelock_latency": { 2024-04-21T08:57:19.245 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.245 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.245 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.245 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.245 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setlayout_latency": { 2024-04-21T08:57:19.245 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.245 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.246 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.246 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.246 INFO:teuthology.orchestra.run.smithi077.stdout: "req_setxattr_latency": { 2024-04-21T08:57:19.246 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.246 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.246 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.247 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.247 INFO:teuthology.orchestra.run.smithi077.stdout: "req_symlink_latency": { 2024-04-21T08:57:19.247 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.247 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.247 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.247 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.247 INFO:teuthology.orchestra.run.smithi077.stdout: "req_unlink_latency": { 2024-04-21T08:57:19.247 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.247 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.248 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.248 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.248 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_revoke_eviction": 0, 2024-04-21T08:57:19.248 INFO:teuthology.orchestra.run.smithi077.stdout: "cap_acquisition_throttle": 0, 2024-04-21T08:57:19.248 INFO:teuthology.orchestra.run.smithi077.stdout: "req_getvxattr_latency": { 2024-04-21T08:57:19.248 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.248 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.248 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.249 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:19.249 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.249 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_sessions": { 2024-04-21T08:57:19.249 INFO:teuthology.orchestra.run.smithi077.stdout: "session_count": 2, 2024-04-21T08:57:19.249 INFO:teuthology.orchestra.run.smithi077.stdout: "session_add": 4, 2024-04-21T08:57:19.249 INFO:teuthology.orchestra.run.smithi077.stdout: "session_remove": 2, 2024-04-21T08:57:19.249 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_open": 0, 2024-04-21T08:57:19.249 INFO:teuthology.orchestra.run.smithi077.stdout: "sessions_stale": 0, 2024-04-21T08:57:19.249 INFO:teuthology.orchestra.run.smithi077.stdout: "total_load": 0, 2024-04-21T08:57:19.250 INFO:teuthology.orchestra.run.smithi077.stdout: "average_load": 0, 2024-04-21T08:57:19.250 INFO:teuthology.orchestra.run.smithi077.stdout: "avg_session_uptime": 3183, 2024-04-21T08:57:19.250 INFO:teuthology.orchestra.run.smithi077.stdout: "mdthresh_evicted": 0 2024-04-21T08:57:19.250 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.250 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-04-21T08:57:19.250 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:57:19.250 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-04-21T08:57:19.250 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:57:19.251 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:57:19.251 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:57:19.251 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:57:19.251 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:57:19.251 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:57:19.251 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:57:19.251 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:57:19.251 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:57:19.251 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:57:19.252 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:57:19.252 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:57:19.252 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:57:19.252 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-04-21T08:57:19.252 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:57:19.252 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-04-21T08:57:19.252 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:57:19.252 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:57:19.252 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:57:19.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:57:19.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:57:19.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:57:19.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:57:19.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-04-21T08:57:19.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:57:19.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:57:19.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:57:19.253 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-04-21T08:57:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-04-21T08:57:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-04-21T08:57:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:57:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:57:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:57:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:57:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 11763601, 2024-04-21T08:57:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 121959, 2024-04-21T08:57:19.255 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 176, 2024-04-21T08:57:19.255 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 2, 2024-04-21T08:57:19.255 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-04-21T08:57:19.255 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-04-21T08:57:19.255 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:57:19.255 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-04-21T08:57:19.255 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:57:19.256 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-04-21T08:57:19.256 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 10136, 2024-04-21T08:57:19.256 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 196, 2024-04-21T08:57:19.256 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:57:19.256 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:57:19.256 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-04-21T08:57:19.256 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-04-21T08:57:19.256 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 347350675, 2024-04-21T08:57:19.257 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 3974295, 2024-04-21T08:57:19.257 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-04-21T08:57:19.257 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-04-21T08:57:19.257 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-04-21T08:57:19.257 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-04-21T08:57:19.257 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.257 INFO:teuthology.orchestra.run.smithi077.stdout: "objecter": { 2024-04-21T08:57:19.257 INFO:teuthology.orchestra.run.smithi077.stdout: "op_active": 0, 2024-04-21T08:57:19.258 INFO:teuthology.orchestra.run.smithi077.stdout: "op_laggy": 0, 2024-04-21T08:57:19.258 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send": 2944, 2024-04-21T08:57:19.258 INFO:teuthology.orchestra.run.smithi077.stdout: "op_send_bytes": 26, 2024-04-21T08:57:19.258 INFO:teuthology.orchestra.run.smithi077.stdout: "op_resend": 3, 2024-04-21T08:57:19.258 INFO:teuthology.orchestra.run.smithi077.stdout: "op_reply": 2944, 2024-04-21T08:57:19.258 INFO:teuthology.orchestra.run.smithi077.stdout: "op_latency": { 2024-04-21T08:57:19.258 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 2944, 2024-04-21T08:57:19.258 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 69.090584866, 2024-04-21T08:57:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.023468269 2024-04-21T08:57:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "op_inflight": 0, 2024-04-21T08:57:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "oplen_avg": { 2024-04-21T08:57:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 2944, 2024-04-21T08:57:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 2945 2024-04-21T08:57:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "op": 2944, 2024-04-21T08:57:19.259 INFO:teuthology.orchestra.run.smithi077.stdout: "op_r": 2944, 2024-04-21T08:57:19.260 INFO:teuthology.orchestra.run.smithi077.stdout: "op_w": 0, 2024-04-21T08:57:19.260 INFO:teuthology.orchestra.run.smithi077.stdout: "op_rmw": 0, 2024-04-21T08:57:19.260 INFO:teuthology.orchestra.run.smithi077.stdout: "op_pg": 0, 2024-04-21T08:57:19.260 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_stat": 1761, 2024-04-21T08:57:19.260 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_create": 0, 2024-04-21T08:57:19.260 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_read": 1180, 2024-04-21T08:57:19.260 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_write": 0, 2024-04-21T08:57:19.260 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writefull": 0, 2024-04-21T08:57:19.260 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_writesame": 0, 2024-04-21T08:57:19.261 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_append": 0, 2024-04-21T08:57:19.261 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_zero": 0, 2024-04-21T08:57:19.261 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_truncate": 0, 2024-04-21T08:57:19.261 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_delete": 0, 2024-04-21T08:57:19.261 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_mapext": 0, 2024-04-21T08:57:19.261 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_sparse_read": 0, 2024-04-21T08:57:19.261 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_clonerange": 0, 2024-04-21T08:57:19.261 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_getxattr": 2, 2024-04-21T08:57:19.261 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_setxattr": 0, 2024-04-21T08:57:19.262 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_cmpxattr": 0, 2024-04-21T08:57:19.262 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_rmxattr": 0, 2024-04-21T08:57:19.262 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_resetxattrs": 0, 2024-04-21T08:57:19.262 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_call": 0, 2024-04-21T08:57:19.262 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_watch": 0, 2024-04-21T08:57:19.262 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_notify": 0, 2024-04-21T08:57:19.262 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T08:57:19.262 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls": 0, 2024-04-21T08:57:19.262 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_pgls_filter": 0, 2024-04-21T08:57:19.263 INFO:teuthology.orchestra.run.smithi077.stdout: "osdop_other": 0, 2024-04-21T08:57:19.263 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_active": 0, 2024-04-21T08:57:19.263 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_send": 0, 2024-04-21T08:57:19.263 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_resend": 0, 2024-04-21T08:57:19.263 INFO:teuthology.orchestra.run.smithi077.stdout: "linger_ping": 0, 2024-04-21T08:57:19.263 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_active": 0, 2024-04-21T08:57:19.263 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_send": 0, 2024-04-21T08:57:19.263 INFO:teuthology.orchestra.run.smithi077.stdout: "poolop_resend": 0, 2024-04-21T08:57:19.263 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_active": 0, 2024-04-21T08:57:19.264 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_send": 0, 2024-04-21T08:57:19.264 INFO:teuthology.orchestra.run.smithi077.stdout: "poolstat_resend": 0, 2024-04-21T08:57:19.264 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_active": 0, 2024-04-21T08:57:19.264 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_send": 0, 2024-04-21T08:57:19.264 INFO:teuthology.orchestra.run.smithi077.stdout: "statfs_resend": 0, 2024-04-21T08:57:19.264 INFO:teuthology.orchestra.run.smithi077.stdout: "command_active": 0, 2024-04-21T08:57:19.264 INFO:teuthology.orchestra.run.smithi077.stdout: "command_send": 0, 2024-04-21T08:57:19.264 INFO:teuthology.orchestra.run.smithi077.stdout: "command_resend": 0, 2024-04-21T08:57:19.264 INFO:teuthology.orchestra.run.smithi077.stdout: "map_epoch": 114, 2024-04-21T08:57:19.265 INFO:teuthology.orchestra.run.smithi077.stdout: "map_full": 0, 2024-04-21T08:57:19.265 INFO:teuthology.orchestra.run.smithi077.stdout: "map_inc": 18, 2024-04-21T08:57:19.265 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_sessions": 11, 2024-04-21T08:57:19.265 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_open": 11, 2024-04-21T08:57:19.265 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_session_close": 0, 2024-04-21T08:57:19.265 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_laggy": 0, 2024-04-21T08:57:19.265 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_wr": 0, 2024-04-21T08:57:19.265 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_rd": 2, 2024-04-21T08:57:19.265 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_del": 0 2024-04-21T08:57:19.266 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.266 INFO:teuthology.orchestra.run.smithi077.stdout: "oft": { 2024-04-21T08:57:19.266 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_objs": 0, 2024-04-21T08:57:19.266 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_kv_pairs": 0, 2024-04-21T08:57:19.266 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_updates": 0, 2024-04-21T08:57:19.266 INFO:teuthology.orchestra.run.smithi077.stdout: "omap_total_removes": 0 2024-04-21T08:57:19.266 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.266 INFO:teuthology.orchestra.run.smithi077.stdout: "osd-slow-ops": { 2024-04-21T08:57:19.267 INFO:teuthology.orchestra.run.smithi077.stdout: "slow_ops_count": 0 2024-04-21T08:57:19.267 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.267 INFO:teuthology.orchestra.run.smithi077.stdout: "purge_queue": { 2024-04-21T08:57:19.267 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops": 0, 2024-04-21T08:57:19.267 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T08:57:19.267 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing": 0, 2024-04-21T08:57:19.267 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executing_high_water": 0, 2024-04-21T08:57:19.267 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed_ops": 0, 2024-04-21T08:57:19.267 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_executed": 0, 2024-04-21T08:57:19.268 INFO:teuthology.orchestra.run.smithi077.stdout: "pq_item_in_journal": 0 2024-04-21T08:57:19.268 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.268 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:57:19.268 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-04-21T08:57:19.268 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-04-21T08:57:19.268 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-04-21T08:57:19.268 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 3151, 2024-04-21T08:57:19.268 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 674739261, 2024-04-21T08:57:19.268 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:19.268 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 3151, 2024-04-21T08:57:19.269 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-04-21T08:57:19.269 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-04-21T08:57:19.269 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 3151, 2024-04-21T08:57:19.269 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 674739261, 2024-04-21T08:57:19.269 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-04-21T08:57:19.269 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.269 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.269 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.269 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:19.270 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.270 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_bytes": { 2024-04-21T08:57:19.270 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-04-21T08:57:19.270 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-04-21T08:57:19.270 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-04-21T08:57:19.270 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-04-21T08:57:19.270 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-04-21T08:57:19.270 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:19.270 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-04-21T08:57:19.271 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 2944, 2024-04-21T08:57:19.271 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 673856736, 2024-04-21T08:57:19.271 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 2944, 2024-04-21T08:57:19.271 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 673856736, 2024-04-21T08:57:19.271 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-04-21T08:57:19.271 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.271 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.271 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.271 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:19.272 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.272 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-objecter_ops": { 2024-04-21T08:57:19.272 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-04-21T08:57:19.272 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 1024, 2024-04-21T08:57:19.272 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-04-21T08:57:19.272 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-04-21T08:57:19.272 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-04-21T08:57:19.272 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:19.272 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-04-21T08:57:19.273 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 2944, 2024-04-21T08:57:19.273 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 2944, 2024-04-21T08:57:19.273 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 2944, 2024-04-21T08:57:19.273 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 2944, 2024-04-21T08:57:19.273 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-04-21T08:57:19.273 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.273 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.273 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.273 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:19.274 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.274 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle": { 2024-04-21T08:57:19.274 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-04-21T08:57:19.274 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-04-21T08:57:19.274 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-04-21T08:57:19.274 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-04-21T08:57:19.274 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-04-21T08:57:19.274 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:19.274 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-04-21T08:57:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-04-21T08:57:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-04-21T08:57:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 0, 2024-04-21T08:57:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 0, 2024-04-21T08:57:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-04-21T08:57:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.275 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-write_buf_throttle-0x55f87544c6e0": { 2024-04-21T08:57:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-04-21T08:57:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 3758096384, 2024-04-21T08:57:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-04-21T08:57:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 0, 2024-04-21T08:57:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 0, 2024-04-21T08:57:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 0, 2024-04-21T08:57:19.276 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-04-21T08:57:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-04-21T08:57:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 0, 2024-04-21T08:57:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 0, 2024-04-21T08:57:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-04-21T08:57:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:19.277 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:19.278 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-21T08:57:20.017 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-21T08:57:20.017 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-21T08:57:20.017 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T08:57:20.018 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-04-21T08:57:20.018 DEBUG:tasks.check_counter:Getting stats from l 2024-04-21T08:57:20.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-21T08:57:20.254 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:57:21.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:20 smithi016 ceph-mon[31259]: pgmap v401: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 25 op/s 2024-04-21T08:57:21.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:20 smithi077 ceph-mon[31615]: pgmap v401: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 25 op/s 2024-04-21T08:57:21.151 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:20 smithi007 ceph-mon[28477]: pgmap v401: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 25 op/s 2024-04-21T08:57:22.486 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:22 smithi077 ceph-mon[31615]: osdmap e115: 12 total, 12 up, 12 in 2024-04-21T08:57:22.496 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-21T08:57:22.496 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:57:22.496 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 4, 2024-04-21T08:57:22.496 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 4, 2024-04-21T08:57:22.496 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:57:22.496 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 448, 2024-04-21T08:57:22.497 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 4, 2024-04-21T08:57:22.497 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 0, 2024-04-21T08:57:22.497 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.001202380, 2024-04-21T08:57:22.497 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.000030101, 2024-04-21T08:57:22.497 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.000437947, 2024-04-21T08:57:22.497 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:22.497 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:22.497 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 3, 2024-04-21T08:57:22.498 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000649552, 2024-04-21T08:57:22.498 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000216517 2024-04-21T08:57:22.498 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:22.498 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:22.498 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:22.498 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:22.498 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:22.498 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:22.499 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:57:22.499 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:57:22.499 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:22.499 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:57:22.499 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 2, 2024-04-21T08:57:22.499 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 193, 2024-04-21T08:57:22.499 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:57:22.499 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 189952, 2024-04-21T08:57:22.500 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 5, 2024-04-21T08:57:22.500 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 3, 2024-04-21T08:57:22.500 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.010990245, 2024-04-21T08:57:22.500 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.007299946, 2024-04-21T08:57:22.500 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.001418839, 2024-04-21T08:57:22.500 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:22.500 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:22.500 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 190, 2024-04-21T08:57:22.501 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.019908911, 2024-04-21T08:57:22.501 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000104783 2024-04-21T08:57:22.501 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:22.501 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:22.501 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:22.501 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:22.502 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:22.502 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:22.502 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:57:22.502 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 189952 2024-04-21T08:57:22.502 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:22.502 INFO:teuthology.orchestra.run.smithi077.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:57:22.502 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_messages": 263, 2024-04-21T08:57:22.502 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages": 252, 2024-04-21T08:57:22.503 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_bytes": 1601488, 2024-04-21T08:57:22.503 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_bytes": 155280, 2024-04-21T08:57:22.503 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_created_connections": 2, 2024-04-21T08:57:22.503 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_active_connections": 1, 2024-04-21T08:57:22.503 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_total_time": 0.047146396, 2024-04-21T08:57:22.503 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_send_time": 0.013933839, 2024-04-21T08:57:22.503 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_recv_time": 0.036759821, 2024-04-21T08:57:22.503 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_running_fast_dispatch_time": 0.005630645, 2024-04-21T08:57:22.503 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:22.504 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 251, 2024-04-21T08:57:22.504 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.012492950, 2024-04-21T08:57:22.504 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000049772 2024-04-21T08:57:22.504 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:22.504 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:22.504 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:22.504 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:22.504 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:22.504 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:22.505 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_recv_encrypted_bytes": 1601488, 2024-04-21T08:57:22.505 INFO:teuthology.orchestra.run.smithi077.stdout: "msgr_send_encrypted_bytes": 155280 2024-04-21T08:57:22.505 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:22.505 INFO:teuthology.orchestra.run.smithi077.stdout: "cct": { 2024-04-21T08:57:22.505 INFO:teuthology.orchestra.run.smithi077.stdout: "total_workers": 0, 2024-04-21T08:57:22.505 INFO:teuthology.orchestra.run.smithi077.stdout: "unhealthy_workers": 0 2024-04-21T08:57:22.505 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:22.505 INFO:teuthology.orchestra.run.smithi077.stdout: "mempool": { 2024-04-21T08:57:22.506 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:57:22.506 INFO:teuthology.orchestra.run.smithi077.stdout: "bloom_filter_items": 0, 2024-04-21T08:57:22.506 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:57:22.506 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:57:22.506 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:57:22.506 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:57:22.506 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:57:22.506 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:57:22.507 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:57:22.507 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:57:22.507 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:57:22.507 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:57:22.507 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:57:22.507 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:57:22.507 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:57:22.507 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_extent_items": 0, 2024-04-21T08:57:22.507 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:57:22.508 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_blob_items": 0, 2024-04-21T08:57:22.508 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:57:22.508 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:57:22.508 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:57:22.508 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:57:22.508 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:57:22.508 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:57:22.508 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:57:22.509 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_txc_items": 0, 2024-04-21T08:57:22.509 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:57:22.509 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:57:22.509 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:57:22.509 INFO:teuthology.orchestra.run.smithi077.stdout: "bluestore_writing_items": 0, 2024-04-21T08:57:22.509 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_bytes": 0, 2024-04-21T08:57:22.509 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_items": 0, 2024-04-21T08:57:22.509 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:57:22.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:57:22.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:57:22.510 INFO:teuthology.orchestra.run.smithi077.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:57:22.510 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_bytes": 148732, 2024-04-21T08:57:22.510 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_anon_items": 31, 2024-04-21T08:57:22.510 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:57:22.510 INFO:teuthology.orchestra.run.smithi077.stdout: "buffer_meta_items": 0, 2024-04-21T08:57:22.510 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_bytes": 0, 2024-04-21T08:57:22.511 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_items": 0, 2024-04-21T08:57:22.511 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:57:22.511 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_mapbl_items": 0, 2024-04-21T08:57:22.511 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:57:22.511 INFO:teuthology.orchestra.run.smithi077.stdout: "osd_pglog_items": 0, 2024-04-21T08:57:22.511 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_bytes": 0, 2024-04-21T08:57:22.511 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_items": 0, 2024-04-21T08:57:22.511 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:57:22.511 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:57:22.512 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_bytes": 0, 2024-04-21T08:57:22.512 INFO:teuthology.orchestra.run.smithi077.stdout: "pgmap_items": 0, 2024-04-21T08:57:22.512 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_bytes": 928, 2024-04-21T08:57:22.512 INFO:teuthology.orchestra.run.smithi077.stdout: "mds_co_items": 2, 2024-04-21T08:57:22.512 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_bytes": 0, 2024-04-21T08:57:22.512 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_1_items": 0, 2024-04-21T08:57:22.512 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_bytes": 0, 2024-04-21T08:57:22.512 INFO:teuthology.orchestra.run.smithi077.stdout: "unittest_2_items": 0 2024-04-21T08:57:22.512 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-04-21T08:57:22.513 INFO:teuthology.orchestra.run.smithi077.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:57:22.513 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 0, 2024-04-21T08:57:22.513 INFO:teuthology.orchestra.run.smithi077.stdout: "max": 104857600, 2024-04-21T08:57:22.513 INFO:teuthology.orchestra.run.smithi077.stdout: "get_started": 0, 2024-04-21T08:57:22.513 INFO:teuthology.orchestra.run.smithi077.stdout: "get": 265, 2024-04-21T08:57:22.513 INFO:teuthology.orchestra.run.smithi077.stdout: "get_sum": 1564382, 2024-04-21T08:57:22.513 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:22.513 INFO:teuthology.orchestra.run.smithi077.stdout: "get_or_fail_success": 265, 2024-04-21T08:57:22.513 INFO:teuthology.orchestra.run.smithi077.stdout: "take": 0, 2024-04-21T08:57:22.514 INFO:teuthology.orchestra.run.smithi077.stdout: "take_sum": 0, 2024-04-21T08:57:22.514 INFO:teuthology.orchestra.run.smithi077.stdout: "put": 265, 2024-04-21T08:57:22.514 INFO:teuthology.orchestra.run.smithi077.stdout: "put_sum": 1564382, 2024-04-21T08:57:22.514 INFO:teuthology.orchestra.run.smithi077.stdout: "wait": { 2024-04-21T08:57:22.514 INFO:teuthology.orchestra.run.smithi077.stdout: "avgcount": 0, 2024-04-21T08:57:22.514 INFO:teuthology.orchestra.run.smithi077.stdout: "sum": 0.000000000, 2024-04-21T08:57:22.514 INFO:teuthology.orchestra.run.smithi077.stdout: "avgtime": 0.000000000 2024-04-21T08:57:22.514 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:22.514 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-04-21T08:57:22.515 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-21T08:57:22.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:22 smithi016 ceph-mon[31259]: osdmap e115: 12 total, 12 up, 12 in 2024-04-21T08:57:22.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:22 smithi007 ceph-mon[28477]: osdmap e115: 12 total, 12 up, 12 in 2024-04-21T08:57:23.288 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-21T08:57:23.288 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-21T08:57:23.288 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-21T08:57:23.288 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-04-21T08:57:23.289 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-21T08:57:23.300 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-21T08:57:23.300 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:57:23.300 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:57:23.333 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:57:23.333 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:57:23.354 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:23 smithi077 ceph-mon[31615]: pgmap v403: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 29 op/s 2024-04-21T08:57:23.392 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:57:23.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.549+0000 7f7d026eb640 1 -- 172.21.15.7:0/3919288668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cfc069f10 msgr2=0x7f7cfc106be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:23.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.549+0000 7f7d026eb640 1 --2- 172.21.15.7:0/3919288668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cfc069f10 0x7f7cfc106be0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009900 tx=0x7f7cf002bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:23.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.550+0000 7f7d026eb640 1 -- 172.21.15.7:0/3919288668 shutdown_connections 2024-04-21T08:57:23.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.550+0000 7f7d026eb640 1 --2- 172.21.15.7:0/3919288668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cfc1072c0 0x7f7cfc10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.550+0000 7f7d026eb640 1 --2- 172.21.15.7:0/3919288668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cfc069f10 0x7f7cfc106be0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.550+0000 7f7d026eb640 1 --2- 172.21.15.7:0/3919288668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc069560 0x7f7cfc069940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.550+0000 7f7d026eb640 1 -- 172.21.15.7:0/3919288668 >> 172.21.15.7:0/3919288668 conn(0x7f7cfc0771d0 msgr2=0x7f7cfc0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:23.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.550+0000 7f7d026eb640 1 -- 172.21.15.7:0/3919288668 shutdown_connections 2024-04-21T08:57:23.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.550+0000 7f7d026eb640 1 -- 172.21.15.7:0/3919288668 wait complete. 2024-04-21T08:57:23.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.550+0000 7f7d026eb640 1 Processor -- start 2024-04-21T08:57:23.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.550+0000 7f7d026eb640 1 -- start start 2024-04-21T08:57:23.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7d026eb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc069560 0x7f7cfc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:23.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7d026eb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cfc1072c0 0x7f7cfc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:23.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7d026eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cfc19cfd0 0x7f7cfc19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:23.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7d026eb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc1121d0 con 0x7f7cfc1072c0 2024-04-21T08:57:23.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7d026eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc112050 con 0x7f7cfc19cfd0 2024-04-21T08:57:23.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7d026eb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7cfc112350 con 0x7f7cfc069560 2024-04-21T08:57:23.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cfb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cfc1072c0 0x7f7cfc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:23.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cfb7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cfc1072c0 0x7f7cfc198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52604/0 (socket says 172.21.15.7:52604) 2024-04-21T08:57:23.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cfb7fe640 1 -- 172.21.15.7:0/436861305 learned_addr learned my addr 172.21.15.7:0/436861305 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:23.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cfbfff640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc069560 0x7f7cfc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:23.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7d00c88640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cfc19cfd0 0x7f7cfc19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:23.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cfb7fe640 1 -- 172.21.15.7:0/436861305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc069560 msgr2=0x7f7cfc198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:23.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cfb7fe640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc069560 0x7f7cfc198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cfb7fe640 1 -- 172.21.15.7:0/436861305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cfc19cfd0 msgr2=0x7f7cfc19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:23.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cfb7fe640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cfc19cfd0 0x7f7cfc19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cfb7fe640 1 -- 172.21.15.7:0/436861305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cfc19dac0 con 0x7f7cfc1072c0 2024-04-21T08:57:23.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cfb7fe640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cfc1072c0 0x7f7cfc198c50 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0005ab0 tx=0x7f7cf00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:23.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cf97fa640 1 -- 172.21.15.7:0/436861305 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0043020 con 0x7f7cfc1072c0 2024-04-21T08:57:23.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.551+0000 7f7cf97fa640 1 -- 172.21.15.7:0/436861305 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cf002cb70 con 0x7f7cfc1072c0 2024-04-21T08:57:23.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.552+0000 7f7d026eb640 1 -- 172.21.15.7:0/436861305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cfc19dd50 con 0x7f7cfc1072c0 2024-04-21T08:57:23.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.552+0000 7f7cf97fa640 1 -- 172.21.15.7:0/436861305 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0003af0 con 0x7f7cfc1072c0 2024-04-21T08:57:23.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.552+0000 7f7d026eb640 1 -- 172.21.15.7:0/436861305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cfc1a5570 con 0x7f7cfc1072c0 2024-04-21T08:57:23.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.553+0000 7f7d026eb640 1 -- 172.21.15.7:0/436861305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cfc068e00 con 0x7f7cfc1072c0 2024-04-21T08:57:23.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.554+0000 7f7cf97fa640 1 -- 172.21.15.7:0/436861305 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f7cf0046020 con 0x7f7cfc1072c0 2024-04-21T08:57:23.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.554+0000 7f7cf97fa640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ce40a3da0 0x7f7ce40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:23.557 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:23 smithi016 ceph-mon[31259]: pgmap v403: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 29 op/s 2024-04-21T08:57:23.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.554+0000 7f7cfbfff640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ce40a3da0 0x7f7ce40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:23.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.554+0000 7f7cf97fa640 1 -- 172.21.15.7:0/436861305 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9612+0+0 (secure 0 0 0) 0x7f7cf00f2bd0 con 0x7f7cfc1072c0 2024-04-21T08:57:23.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.555+0000 7f7cfbfff640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ce40a3da0 0x7f7ce40a6260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009bd0 tx=0x7f7cec005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:23.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.559+0000 7f7cf97fa640 1 -- 172.21.15.7:0/436861305 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7cf004d020 con 0x7f7cfc1072c0 2024-04-21T08:57:23.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:23 smithi007 ceph-mon[28477]: pgmap v403: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 29 op/s 2024-04-21T08:57:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.748+0000 7f7d026eb640 1 -- 172.21.15.7:0/436861305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7cfc069f10 con 0x7f7cfc1072c0 2024-04-21T08:57:23.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.750+0000 7f7cf97fa640 1 -- 172.21.15.7:0/436861305 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+25270 (secure 0 0 0) 0x7f7cf00bef40 con 0x7f7cfc1072c0 2024-04-21T08:57:23.750 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:57:23.750 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":115,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:57:21.258201+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3451243741":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/117789107":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/4167672076":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/3630897301":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:57:23.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 -- 172.21.15.7:0/436861305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ce40a3da0 msgr2=0x7f7ce40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:23.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ce40a3da0 0x7f7ce40a6260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009bd0 tx=0x7f7cec005f90 comp rx=0 tx=0).stop 2024-04-21T08:57:23.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 -- 172.21.15.7:0/436861305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cfc1072c0 msgr2=0x7f7cfc198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:23.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cfc1072c0 0x7f7cfc198c50 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0005ab0 tx=0x7f7cf00045e0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 -- 172.21.15.7:0/436861305 shutdown_connections 2024-04-21T08:57:23.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cfc19cfd0 0x7f7cfc19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cfc1072c0 0x7f7cfc198c50 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f7ce40a3da0 0x7f7ce40a6260 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 --2- 172.21.15.7:0/436861305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cfc069560 0x7f7cfc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 -- 172.21.15.7:0/436861305 >> 172.21.15.7:0/436861305 conn(0x7f7cfc0771d0 msgr2=0x7f7cfc0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:23.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 -- 172.21.15.7:0/436861305 shutdown_connections 2024-04-21T08:57:23.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.752+0000 7f7d026eb640 1 -- 172.21.15.7:0/436861305 wait complete. 2024-04-21T08:57:23.763 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:37:33.631932+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:15.165234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:15.944352+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 114, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:19.645372+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 114, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:57:23.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:57:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.869+0000 7f8d56b96640 1 -- 172.21.15.7:0/4143991515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d50107260 msgr2=0x7f8d5010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.869+0000 7f8d56b96640 1 --2- 172.21.15.7:0/4143991515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d50107260 0x7f8d5010ee30 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009900 tx=0x7f8d4002bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.869+0000 7f8d56b96640 1 -- 172.21.15.7:0/4143991515 shutdown_connections 2024-04-21T08:57:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.869+0000 7f8d56b96640 1 --2- 172.21.15.7:0/4143991515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d50107260 0x7f8d5010ee30 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.869+0000 7f8d56b96640 1 --2- 172.21.15.7:0/4143991515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d50102b40 0x7f8d50106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.869+0000 7f8d56b96640 1 --2- 172.21.15.7:0/4143991515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d50102190 0x7f8d50102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.869+0000 7f8d56b96640 1 -- 172.21.15.7:0/4143991515 >> 172.21.15.7:0/4143991515 conn(0x7f8d500fda10 msgr2=0x7f8d500ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.869+0000 7f8d56b96640 1 -- 172.21.15.7:0/4143991515 shutdown_connections 2024-04-21T08:57:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.869+0000 7f8d56b96640 1 -- 172.21.15.7:0/4143991515 wait complete. 2024-04-21T08:57:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.869+0000 7f8d56b96640 1 Processor -- start 2024-04-21T08:57:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.869+0000 7f8d56b96640 1 -- start start 2024-04-21T08:57:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d56b96640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d50102190 0x7f8d50198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d56b96640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d50107260 0x7f8d50198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d56b96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d5019cfe0 0x7f8d5019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d56b96640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8d501121d0 con 0x7f8d50102190 2024-04-21T08:57:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d56b96640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8d50112050 con 0x7f8d50107260 2024-04-21T08:57:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d56b96640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d50112350 con 0x7f8d5019cfe0 2024-04-21T08:57:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d54932640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d50102190 0x7f8d50198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d54932640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d50102190 0x7f8d50198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52620/0 (socket says 172.21.15.7:52620) 2024-04-21T08:57:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d54932640 1 -- 172.21.15.7:0/3908884731 learned_addr learned my addr 172.21.15.7:0/3908884731 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d4ffff640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d50107260 0x7f8d50198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d55133640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d5019cfe0 0x7f8d5019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d54932640 1 -- 172.21.15.7:0/3908884731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d5019cfe0 msgr2=0x7f8d5019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d54932640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d5019cfe0 0x7f8d5019d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d54932640 1 -- 172.21.15.7:0/3908884731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d50107260 msgr2=0x7f8d50198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d54932640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d50107260 0x7f8d50198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d54932640 1 -- 172.21.15.7:0/3908884731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d5019dad0 con 0x7f8d50102190 2024-04-21T08:57:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d4ffff640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d50107260 0x7f8d50198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d55133640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d5019cfe0 0x7f8d5019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d54932640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d50102190 0x7f8d50198770 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800eb90 tx=0x7f8d4800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d4dffb640 1 -- 172.21.15.7:0/3908884731 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d48004d20 con 0x7f8d50102190 2024-04-21T08:57:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d4dffb640 1 -- 172.21.15.7:0/3908884731 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d4800ee90 con 0x7f8d50102190 2024-04-21T08:57:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d56b96640 1 -- 172.21.15.7:0/3908884731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d5019ddc0 con 0x7f8d50102190 2024-04-21T08:57:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.870+0000 7f8d4dffb640 1 -- 172.21.15.7:0/3908884731 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d48010670 con 0x7f8d50102190 2024-04-21T08:57:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.871+0000 7f8d56b96640 1 -- 172.21.15.7:0/3908884731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d501a5560 con 0x7f8d50102190 2024-04-21T08:57:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.872+0000 7f8d56b96640 1 -- 172.21.15.7:0/3908884731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d50068e00 con 0x7f8d50102190 2024-04-21T08:57:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.872+0000 7f8d4dffb640 1 -- 172.21.15.7:0/3908884731 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f8d48005020 con 0x7f8d50102190 2024-04-21T08:57:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.873+0000 7f8d4dffb640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8d380a3d20 0x7f8d380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.873+0000 7f8d4ffff640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8d380a3d20 0x7f8d380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:23.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.873+0000 7f8d4dffb640 1 -- 172.21.15.7:0/3908884731 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9612+0+0 (secure 0 0 0) 0x7f8d480d13d0 con 0x7f8d50102190 2024-04-21T08:57:23.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.874+0000 7f8d4ffff640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8d380a3d20 0x7f8d380a61e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8d500fdec0 tx=0x7f8d44005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:23.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:23.878+0000 7f8d4dffb640 1 -- 172.21.15.7:0/3908884731 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d4809d630 con 0x7f8d50102190 2024-04-21T08:57:24.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.067+0000 7f8d56b96640 1 -- 172.21.15.7:0/3908884731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d5010a4f0 con 0x7f8d50102190 2024-04-21T08:57:24.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.067+0000 7f8d4dffb640 1 -- 172.21.15.7:0/3908884731 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) ==== 93+0+10 (secure 0 0 0) 0x7f8d4809cf30 con 0x7f8d50102190 2024-04-21T08:57:24.067 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-21T08:57:24.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 -- 172.21.15.7:0/3908884731 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8d380a3d20 msgr2=0x7f8d380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8d380a3d20 0x7f8d380a61e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8d500fdec0 tx=0x7f8d44005f90 comp rx=0 tx=0).stop 2024-04-21T08:57:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 -- 172.21.15.7:0/3908884731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d50102190 msgr2=0x7f8d50198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d50102190 0x7f8d50198770 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800eb90 tx=0x7f8d4800c6a0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 -- 172.21.15.7:0/3908884731 shutdown_connections 2024-04-21T08:57:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f8d380a3d20 0x7f8d380a61e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d5019cfe0 0x7f8d5019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d50107260 0x7f8d50198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 --2- 172.21.15.7:0/3908884731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d50102190 0x7f8d50198770 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 -- 172.21.15.7:0/3908884731 >> 172.21.15.7:0/3908884731 conn(0x7f8d500fda10 msgr2=0x7f8d500ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 -- 172.21.15.7:0/3908884731 shutdown_connections 2024-04-21T08:57:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.069+0000 7f8d56b96640 1 -- 172.21.15.7:0/3908884731 wait complete. 2024-04-21T08:57:24.080 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:57:24.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.185+0000 7f7929a5c640 1 -- 172.21.15.7:0/2589961989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7924107250 msgr2=0x7f792410ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.185+0000 7f7929a5c640 1 --2- 172.21.15.7:0/2589961989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7924107250 0x7f792410ee20 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f791c009ca0 tx=0x7f791c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.185+0000 7f7929a5c640 1 -- 172.21.15.7:0/2589961989 shutdown_connections 2024-04-21T08:57:24.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.185+0000 7f7929a5c640 1 --2- 172.21.15.7:0/2589961989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7924107250 0x7f792410ee20 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.185+0000 7f7929a5c640 1 --2- 172.21.15.7:0/2589961989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7924102b30 0x7f7924106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.185+0000 7f7929a5c640 1 --2- 172.21.15.7:0/2589961989 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7924102180 0x7f7924102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.185+0000 7f7929a5c640 1 -- 172.21.15.7:0/2589961989 >> 172.21.15.7:0/2589961989 conn(0x7f79240fda00 msgr2=0x7f79240ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:24.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f7929a5c640 1 -- 172.21.15.7:0/2589961989 shutdown_connections 2024-04-21T08:57:24.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f7929a5c640 1 -- 172.21.15.7:0/2589961989 wait complete. 2024-04-21T08:57:24.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f7929a5c640 1 Processor -- start 2024-04-21T08:57:24.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f7929a5c640 1 -- start start 2024-04-21T08:57:24.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f7929a5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7924102180 0x7f7924198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f7929a5c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7924107250 0x7f7924198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f7929a5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f792419d020 0x7f792419d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f7929a5c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f79241121c0 con 0x7f7924107250 2024-04-21T08:57:24.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f7929a5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7924112040 con 0x7f7924102180 2024-04-21T08:57:24.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f7929a5c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7924112340 con 0x7f792419d020 2024-04-21T08:57:24.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f791bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7924107250 0x7f7924198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f791bfff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7924107250 0x7f7924198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52636/0 (socket says 172.21.15.7:52636) 2024-04-21T08:57:24.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.186+0000 7f791bfff640 1 -- 172.21.15.7:0/1797775112 learned_addr learned my addr 172.21.15.7:0/1797775112 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:24.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f7922ffd640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7924102180 0x7f7924198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f79237fe640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f792419d020 0x7f792419d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f791bfff640 1 -- 172.21.15.7:0/1797775112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f792419d020 msgr2=0x7f792419d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f791bfff640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f792419d020 0x7f792419d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f791bfff640 1 -- 172.21.15.7:0/1797775112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7924102180 msgr2=0x7f7924198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f791bfff640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7924102180 0x7f7924198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f791bfff640 1 -- 172.21.15.7:0/1797775112 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792419db10 con 0x7f7924107250 2024-04-21T08:57:24.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f791bfff640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7924107250 0x7f7924198ca0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f790c00d740 tx=0x7f790c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:24.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f7928a5a640 1 -- 172.21.15.7:0/1797775112 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c017070 con 0x7f7924107250 2024-04-21T08:57:24.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f7928a5a640 1 -- 172.21.15.7:0/1797775112 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f790c00de50 con 0x7f7924107250 2024-04-21T08:57:24.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f7928a5a640 1 -- 172.21.15.7:0/1797775112 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c004ce0 con 0x7f7924107250 2024-04-21T08:57:24.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f7929a5c640 1 -- 172.21.15.7:0/1797775112 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792419de00 con 0x7f7924107250 2024-04-21T08:57:24.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.187+0000 7f7929a5c640 1 -- 172.21.15.7:0/1797775112 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79241a55f0 con 0x7f7924107250 2024-04-21T08:57:24.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.188+0000 7f7929a5c640 1 -- 172.21.15.7:0/1797775112 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7924069250 con 0x7f7924107250 2024-04-21T08:57:24.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.189+0000 7f7928a5a640 1 -- 172.21.15.7:0/1797775112 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f790c004e80 con 0x7f7924107250 2024-04-21T08:57:24.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.190+0000 7f7928a5a640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f79100a3da0 0x7f79100a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.190+0000 7f7922ffd640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f79100a3da0 0x7f79100a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.190+0000 7f7928a5a640 1 -- 172.21.15.7:0/1797775112 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9612+0+0 (secure 0 0 0) 0x7f790c0d0930 con 0x7f7924107250 2024-04-21T08:57:24.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.191+0000 7f7922ffd640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f79100a3da0 0x7f79100a6260 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f791400bbe0 tx=0x7f791400b390 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:24.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.194+0000 7f7928a5a640 1 -- 172.21.15.7:0/1797775112 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f790c09cc10 con 0x7f7924107250 2024-04-21T08:57:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.380+0000 7f7929a5c640 1 -- 172.21.15.7:0/1797775112 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7924102b30 con 0x7f7924107250 2024-04-21T08:57:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.380+0000 7f7928a5a640 1 -- 172.21.15.7:0/1797775112 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) ==== 104+0+11 (secure 0 0 0) 0x7f790c09c510 con 0x7f7924107250 2024-04-21T08:57:24.381 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:57:24.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 -- 172.21.15.7:0/1797775112 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f79100a3da0 msgr2=0x7f79100a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f79100a3da0 0x7f79100a6260 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f791400bbe0 tx=0x7f791400b390 comp rx=0 tx=0).stop 2024-04-21T08:57:24.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 -- 172.21.15.7:0/1797775112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7924107250 msgr2=0x7f7924198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7924107250 0x7f7924198ca0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f790c00d740 tx=0x7f790c00dc00 comp rx=0 tx=0).stop 2024-04-21T08:57:24.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 -- 172.21.15.7:0/1797775112 shutdown_connections 2024-04-21T08:57:24.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f79100a3da0 0x7f79100a6260 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f792419d020 0x7f792419d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7924107250 0x7f7924198ca0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 --2- 172.21.15.7:0/1797775112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7924102180 0x7f7924198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 -- 172.21.15.7:0/1797775112 >> 172.21.15.7:0/1797775112 conn(0x7f79240fda00 msgr2=0x7f79240fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:24.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 -- 172.21.15.7:0/1797775112 shutdown_connections 2024-04-21T08:57:24.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.382+0000 7f7929a5c640 1 -- 172.21.15.7:0/1797775112 wait complete. 2024-04-21T08:57:24.392 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:24 smithi007 ceph-mon[28477]: pgmap v404: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 28 op/s 2024-04-21T08:57:24.392 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:24 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/436861305' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:24.392 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:24 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3908884731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:57:24.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:57:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.498+0000 7fe0e9b04640 1 -- 172.21.15.7:0/1129652340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0e41031c0 msgr2=0x7fe0e41035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.498+0000 7fe0e9b04640 1 --2- 172.21.15.7:0/1129652340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0e41031c0 0x7fe0e41035a0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4009900 tx=0x7fe0d402bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.498+0000 7fe0e9b04640 1 -- 172.21.15.7:0/1129652340 shutdown_connections 2024-04-21T08:57:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.498+0000 7fe0e9b04640 1 --2- 172.21.15.7:0/1129652340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0e410c6d0 0x7fe0e410eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.498+0000 7fe0e9b04640 1 --2- 172.21.15.7:0/1129652340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0e4103b70 0x7fe0e410bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.498+0000 7fe0e9b04640 1 --2- 172.21.15.7:0/1129652340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0e41031c0 0x7fe0e41035a0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.498+0000 7fe0e9b04640 1 -- 172.21.15.7:0/1129652340 >> 172.21.15.7:0/1129652340 conn(0x7fe0e4077f70 msgr2=0x7fe0e4078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.498+0000 7fe0e9b04640 1 -- 172.21.15.7:0/1129652340 shutdown_connections 2024-04-21T08:57:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.498+0000 7fe0e9b04640 1 -- 172.21.15.7:0/1129652340 wait complete. 2024-04-21T08:57:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e9b04640 1 Processor -- start 2024-04-21T08:57:24.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e9b04640 1 -- start start 2024-04-21T08:57:24.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e9b04640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0e41031c0 0x7fe0e419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e9b04640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0e410c6d0 0x7fe0e419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e9b04640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0e4198760 0x7fe0e4198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e9b04640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e41123a0 con 0x7fe0e41031c0 2024-04-21T08:57:24.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e9b04640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e4112220 con 0x7fe0e410c6d0 2024-04-21T08:57:24.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e9b04640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e4112520 con 0x7fe0e4198760 2024-04-21T08:57:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0e41031c0 0x7fe0e419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e37fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0e41031c0 0x7fe0e419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52662/0 (socket says 172.21.15.7:52662) 2024-04-21T08:57:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e37fe640 1 -- 172.21.15.7:0/2195618803 learned_addr learned my addr 172.21.15.7:0/2195618803 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e2ffd640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0e410c6d0 0x7fe0e419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e3fff640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0e4198760 0x7fe0e4198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e37fe640 1 -- 172.21.15.7:0/2195618803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0e4198760 msgr2=0x7fe0e4198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e37fe640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0e4198760 0x7fe0e4198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e37fe640 1 -- 172.21.15.7:0/2195618803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0e410c6d0 msgr2=0x7fe0e419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e37fe640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0e410c6d0 0x7fe0e419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e37fe640 1 -- 172.21.15.7:0/2195618803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e4199360 con 0x7fe0e41031c0 2024-04-21T08:57:24.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.499+0000 7fe0e2ffd640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0e410c6d0 0x7fe0e419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:24.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.500+0000 7fe0e37fe640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0e41031c0 0x7fe0e419e6a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4005940 tx=0x7fe0d40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:24.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.500+0000 7fe0e0ff9640 1 -- 172.21.15.7:0/2195618803 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4043020 con 0x7fe0e41031c0 2024-04-21T08:57:24.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.500+0000 7fe0e0ff9640 1 -- 172.21.15.7:0/2195618803 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0d402caf0 con 0x7fe0e41031c0 2024-04-21T08:57:24.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.500+0000 7fe0e9b04640 1 -- 172.21.15.7:0/2195618803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e41995f0 con 0x7fe0e41031c0 2024-04-21T08:57:24.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.500+0000 7fe0e0ff9640 1 -- 172.21.15.7:0/2195618803 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4003af0 con 0x7fe0e41031c0 2024-04-21T08:57:24.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.500+0000 7fe0e9b04640 1 -- 172.21.15.7:0/2195618803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e41a57e0 con 0x7fe0e41031c0 2024-04-21T08:57:24.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.501+0000 7fe0e9b04640 1 -- 172.21.15.7:0/2195618803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e406ae40 con 0x7fe0e41031c0 2024-04-21T08:57:24.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.502+0000 7fe0e0ff9640 1 -- 172.21.15.7:0/2195618803 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fe0d4046020 con 0x7fe0e41031c0 2024-04-21T08:57:24.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.502+0000 7fe0e0ff9640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0cc0a3da0 0x7fe0cc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.503+0000 7fe0e2ffd640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0cc0a3da0 0x7fe0cc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.503+0000 7fe0e0ff9640 1 -- 172.21.15.7:0/2195618803 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9612+0+0 (secure 0 0 0) 0x7fe0d40f2d50 con 0x7fe0e41031c0 2024-04-21T08:57:24.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.503+0000 7fe0e2ffd640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0cc0a3da0 0x7fe0cc0a6260 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4199fb0 tx=0x7fe0d8005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:24.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.507+0000 7fe0e0ff9640 1 -- 172.21.15.7:0/2195618803 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe0d404d020 con 0x7fe0e41031c0 2024-04-21T08:57:24.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:24 smithi016 ceph-mon[31259]: pgmap v404: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 28 op/s 2024-04-21T08:57:24.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:24 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/436861305' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:24.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:24 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3908884731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:57:24.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:24 smithi077 ceph-mon[31615]: pgmap v404: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 28 op/s 2024-04-21T08:57:24.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:24 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/436861305' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:24.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:24 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3908884731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:57:24.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.696+0000 7fe0e9b04640 1 -- 172.21.15.7:0/2195618803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe0e4107680 con 0x7fe0e41031c0 2024-04-21T08:57:24.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.697+0000 7fe0e0ff9640 1 -- 172.21.15.7:0/2195618803 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) ==== 100+0+11 (secure 0 0 0) 0x7fe0d403c350 con 0x7fe0e41031c0 2024-04-21T08:57:24.697 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-04-21T08:57:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 -- 172.21.15.7:0/2195618803 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0cc0a3da0 msgr2=0x7fe0cc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0cc0a3da0 0x7fe0cc0a6260 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4199fb0 tx=0x7fe0d8005f90 comp rx=0 tx=0).stop 2024-04-21T08:57:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 -- 172.21.15.7:0/2195618803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0e41031c0 msgr2=0x7fe0e419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0e41031c0 0x7fe0e419e6a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4005940 tx=0x7fe0d40045e0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 -- 172.21.15.7:0/2195618803 shutdown_connections 2024-04-21T08:57:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fe0cc0a3da0 0x7fe0cc0a6260 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0e4198760 0x7fe0e4198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0e410c6d0 0x7fe0e419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 --2- 172.21.15.7:0/2195618803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0e41031c0 0x7fe0e419e6a0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 -- 172.21.15.7:0/2195618803 >> 172.21.15.7:0/2195618803 conn(0x7fe0e4077f70 msgr2=0x7fe0e40fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 -- 172.21.15.7:0/2195618803 shutdown_connections 2024-04-21T08:57:24.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.699+0000 7fe0e9b04640 1 -- 172.21.15.7:0/2195618803 wait complete. 2024-04-21T08:57:24.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:57:24.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.815+0000 7f753f1f4640 1 -- 172.21.15.7:0/3689549920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7538101180 msgr2=0x7f7538101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.815+0000 7f753f1f4640 1 --2- 172.21.15.7:0/3689549920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7538101180 0x7f7538101560 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f752c009930 tx=0x7f752c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:24.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.815+0000 7f753f1f4640 1 -- 172.21.15.7:0/3689549920 shutdown_connections 2024-04-21T08:57:24.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.815+0000 7f753f1f4640 1 --2- 172.21.15.7:0/3689549920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7538103400 0x7f753810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.815+0000 7f753f1f4640 1 --2- 172.21.15.7:0/3689549920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7538102a60 0x7f7538102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.815+0000 7f753f1f4640 1 --2- 172.21.15.7:0/3689549920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7538101180 0x7f7538101560 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.815+0000 7f753f1f4640 1 -- 172.21.15.7:0/3689549920 >> 172.21.15.7:0/3689549920 conn(0x7f7538077f70 msgr2=0x7f7538078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.815+0000 7f753f1f4640 1 -- 172.21.15.7:0/3689549920 shutdown_connections 2024-04-21T08:57:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.815+0000 7f753f1f4640 1 -- 172.21.15.7:0/3689549920 wait complete. 2024-04-21T08:57:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.815+0000 7f753f1f4640 1 Processor -- start 2024-04-21T08:57:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.815+0000 7f753f1f4640 1 -- start start 2024-04-21T08:57:24.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753f1f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7538101180 0x7f7538199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753f1f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7538102a60 0x7f7538199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753f1f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7538103400 0x7f75381a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753f1f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f75381111a0 con 0x7f7538103400 2024-04-21T08:57:24.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753f1f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7538111020 con 0x7f7538102a60 2024-04-21T08:57:24.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753f1f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7538111320 con 0x7f7538101180 2024-04-21T08:57:24.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753d791640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7538103400 0x7f75381a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753d791640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7538103400 0x7f75381a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43224/0 (socket says 172.21.15.7:43224) 2024-04-21T08:57:24.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753d791640 1 -- 172.21.15.7:0/3842799805 learned_addr learned my addr 172.21.15.7:0/3842799805 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:24.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753cf90640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7538101180 0x7f7538199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f7537fff640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7538102a60 0x7f7538199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753d791640 1 -- 172.21.15.7:0/3842799805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7538101180 msgr2=0x7f7538199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753d791640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7538101180 0x7f7538199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753d791640 1 -- 172.21.15.7:0/3842799805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7538102a60 msgr2=0x7f7538199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:24.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753d791640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7538102a60 0x7f7538199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:24.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753d791640 1 -- 172.21.15.7:0/3842799805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75381a4c00 con 0x7f7538103400 2024-04-21T08:57:24.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f7537fff640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7538102a60 0x7f7538199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:24.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753d791640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7538103400 0x7f75381a46c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f753000efd0 tx=0x7f753000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:24.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f7535ffb640 1 -- 172.21.15.7:0/3842799805 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7530003b70 con 0x7f7538103400 2024-04-21T08:57:24.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f7535ffb640 1 -- 172.21.15.7:0/3842799805 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7530003d10 con 0x7f7538103400 2024-04-21T08:57:24.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f753f1f4640 1 -- 172.21.15.7:0/3842799805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75381a4dd0 con 0x7f7538103400 2024-04-21T08:57:24.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.816+0000 7f7535ffb640 1 -- 172.21.15.7:0/3842799805 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75300108a0 con 0x7f7538103400 2024-04-21T08:57:24.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.817+0000 7f753f1f4640 1 -- 172.21.15.7:0/3842799805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75381a52f0 con 0x7f7538103400 2024-04-21T08:57:24.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.818+0000 7f753f1f4640 1 -- 172.21.15.7:0/3842799805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7538069250 con 0x7f7538103400 2024-04-21T08:57:24.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.819+0000 7f7535ffb640 1 -- 172.21.15.7:0/3842799805 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f7530004020 con 0x7f7538103400 2024-04-21T08:57:24.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.819+0000 7f7535ffb640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75280a3da0 0x7f75280a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:24.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.820+0000 7f753cf90640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75280a3da0 0x7f75280a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:24.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.820+0000 7f7535ffb640 1 -- 172.21.15.7:0/3842799805 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9612+0+0 (secure 0 0 0) 0x7f75300d0fe0 con 0x7f7538103400 2024-04-21T08:57:24.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.820+0000 7f753cf90640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75280a3da0 0x7f75280a6260 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f752c005ab0 tx=0x7f752c005fd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:24.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:24.824+0000 7f7535ffb640 1 -- 172.21.15.7:0/3842799805 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f753009d240 con 0x7f7538103400 2024-04-21T08:57:25.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.009+0000 7f753f1f4640 1 -- 172.21.15.7:0/3842799805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7538109480 con 0x7f7538103400 2024-04-21T08:57:25.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.009+0000 7f7535ffb640 1 -- 172.21.15.7:0/3842799805 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v115) ==== 103+0+11 (secure 0 0 0) 0x7f753009cb40 con 0x7f7538103400 2024-04-21T08:57:25.009 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-04-21T08:57:25.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 -- 172.21.15.7:0/3842799805 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75280a3da0 msgr2=0x7f75280a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75280a3da0 0x7f75280a6260 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f752c005ab0 tx=0x7f752c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 -- 172.21.15.7:0/3842799805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7538103400 msgr2=0x7f75381a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7538103400 0x7f75381a46c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f753000efd0 tx=0x7f753000c6a0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 -- 172.21.15.7:0/3842799805 shutdown_connections 2024-04-21T08:57:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7538103400 0x7f75381a46c0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7538102a60 0x7f7538199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f75280a3da0 0x7f75280a6260 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 --2- 172.21.15.7:0/3842799805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7538101180 0x7f7538199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 -- 172.21.15.7:0/3842799805 >> 172.21.15.7:0/3842799805 conn(0x7f7538077f70 msgr2=0x7f7538104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 -- 172.21.15.7:0/3842799805 shutdown_connections 2024-04-21T08:57:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.011+0000 7f753f1f4640 1 -- 172.21.15.7:0/3842799805 wait complete. 2024-04-21T08:57:25.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T08:57:25.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.125+0000 7fbac76fd640 1 -- 172.21.15.7:0/239729856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac0103b70 msgr2=0x7fbac010bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:25.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.125+0000 7fbac76fd640 1 --2- 172.21.15.7:0/239729856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac0103b70 0x7fbac010bfc0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009900 tx=0x7fbab002bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:25.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.125+0000 7fbac76fd640 1 -- 172.21.15.7:0/239729856 shutdown_connections 2024-04-21T08:57:25.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.125+0000 7fbac76fd640 1 --2- 172.21.15.7:0/239729856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbac010c6d0 0x7fbac010eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.125+0000 7fbac76fd640 1 --2- 172.21.15.7:0/239729856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac0103b70 0x7fbac010bfc0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.125+0000 7fbac76fd640 1 --2- 172.21.15.7:0/239729856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac01031c0 0x7fbac01035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.125+0000 7fbac76fd640 1 -- 172.21.15.7:0/239729856 >> 172.21.15.7:0/239729856 conn(0x7fbac0077f70 msgr2=0x7fbac0078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:25.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.125+0000 7fbac76fd640 1 -- 172.21.15.7:0/239729856 shutdown_connections 2024-04-21T08:57:25.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.125+0000 7fbac76fd640 1 -- 172.21.15.7:0/239729856 wait complete. 2024-04-21T08:57:25.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac76fd640 1 Processor -- start 2024-04-21T08:57:25.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac76fd640 1 -- start start 2024-04-21T08:57:25.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac76fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbac01031c0 0x7fbac0100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac76fd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac010c6d0 0x7fbac0101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac76fd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac0101890 0x7fbac01a6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac76fd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbac0112330 con 0x7fbac010c6d0 2024-04-21T08:57:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac76fd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbac01121b0 con 0x7fbac0101890 2024-04-21T08:57:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac76fd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbac01124b0 con 0x7fbac01031c0 2024-04-21T08:57:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac4c98640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac010c6d0 0x7fbac0101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac4c98640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac010c6d0 0x7fbac0101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43240/0 (socket says 172.21.15.7:43240) 2024-04-21T08:57:25.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac4c98640 1 -- 172.21.15.7:0/2534810367 learned_addr learned my addr 172.21.15.7:0/2534810367 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac5499640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbac01031c0 0x7fbac0100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.126+0000 7fbac5c9a640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac0101890 0x7fbac01a6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.127+0000 7fbac4c98640 1 -- 172.21.15.7:0/2534810367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbac01031c0 msgr2=0x7fbac0100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.127+0000 7fbac4c98640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbac01031c0 0x7fbac0100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.127+0000 7fbac4c98640 1 -- 172.21.15.7:0/2534810367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac0101890 msgr2=0x7fbac01a6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.127+0000 7fbac4c98640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac0101890 0x7fbac01a6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.127+0000 7fbac4c98640 1 -- 172.21.15.7:0/2534810367 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac01a71a0 con 0x7fbac010c6d0 2024-04-21T08:57:25.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.127+0000 7fbac4c98640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac010c6d0 0x7fbac0101350 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fbab0036040 tx=0x7fbab00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:25.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.127+0000 7fbab67fc640 1 -- 172.21.15.7:0/2534810367 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab0043020 con 0x7fbac010c6d0 2024-04-21T08:57:25.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.127+0000 7fbab67fc640 1 -- 172.21.15.7:0/2534810367 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbab002ca70 con 0x7fbac010c6d0 2024-04-21T08:57:25.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.127+0000 7fbac76fd640 1 -- 172.21.15.7:0/2534810367 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac01a7430 con 0x7fbac010c6d0 2024-04-21T08:57:25.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.127+0000 7fbab67fc640 1 -- 172.21.15.7:0/2534810367 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab0003ac0 con 0x7fbac010c6d0 2024-04-21T08:57:25.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.127+0000 7fbac76fd640 1 -- 172.21.15.7:0/2534810367 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac01a7980 con 0x7fbac010c6d0 2024-04-21T08:57:25.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.128+0000 7fbac76fd640 1 -- 172.21.15.7:0/2534810367 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac0069250 con 0x7fbac010c6d0 2024-04-21T08:57:25.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.129+0000 7fbab67fc640 1 -- 172.21.15.7:0/2534810367 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fbab0046020 con 0x7fbac010c6d0 2024-04-21T08:57:25.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.130+0000 7fbab67fc640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbaac0a3d50 0x7fbaac0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:25.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.130+0000 7fbac5499640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbaac0a3d50 0x7fbaac0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:25.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.130+0000 7fbab67fc640 1 -- 172.21.15.7:0/2534810367 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9612+0+0 (secure 0 0 0) 0x7fbab00f3040 con 0x7fbac010c6d0 2024-04-21T08:57:25.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.130+0000 7fbac5499640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbaac0a3d50 0x7fbaac0a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009bd0 tx=0x7fbabc005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:25.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.133+0000 7fbab67fc640 1 -- 172.21.15.7:0/2534810367 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbab004d020 con 0x7fbac010c6d0 2024-04-21T08:57:25.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.332+0000 7fbac76fd640 1 -- 172.21.15.7:0/2534810367 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbac0107680 con 0x7fbac010c6d0 2024-04-21T08:57:25.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.333+0000 7fbab67fc640 1 -- 172.21.15.7:0/2534810367 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v115) ==== 81+0+926 (secure 0 0 0) 0x7fbab00bf350 con 0x7fbac010c6d0 2024-04-21T08:57:25.333 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3451243741 2024-04-22T08:44:41.244700+0000 2024-04-21T08:57:25.333 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/117789107 2024-04-22T08:44:41.244700+0000 2024-04-21T08:57:25.333 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/4167672076 2024-04-22T08:44:41.244700+0000 2024-04-21T08:57:25.333 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/1764088272 2024-04-22T08:35:09.541897+0000 2024-04-21T08:57:25.334 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3825837962 2024-04-22T08:35:09.541897+0000 2024-04-21T08:57:25.334 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/2555078032 2024-04-22T08:44:41.244700+0000 2024-04-21T08:57:25.334 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2463266160 2024-04-22T08:35:09.541897+0000 2024-04-21T08:57:25.334 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3631314876 2024-04-22T08:34:42.863861+0000 2024-04-21T08:57:25.334 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/102274072 2024-04-22T08:35:09.541897+0000 2024-04-21T08:57:25.334 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/2031045761 2024-04-22T08:34:42.863861+0000 2024-04-21T08:57:25.334 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/2031045761 2024-04-22T08:34:42.863861+0000 2024-04-21T08:57:25.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/2555078032 2024-04-22T08:44:41.244700+0000 2024-04-21T08:57:25.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/865111185 2024-04-22T08:34:42.863861+0000 2024-04-21T08:57:25.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3630897301 2024-04-22T08:44:41.244700+0000 2024-04-21T08:57:25.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2778026231 2024-04-22T08:34:42.863861+0000 2024-04-21T08:57:25.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/102274072 2024-04-22T08:35:09.541897+0000 2024-04-21T08:57:25.336 INFO:teuthology.orchestra.run.smithi007.stderr:listed 16 entries 2024-04-21T08:57:25.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 -- 172.21.15.7:0/2534810367 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbaac0a3d50 msgr2=0x7fbaac0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:25.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbaac0a3d50 0x7fbaac0a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009bd0 tx=0x7fbabc005f90 comp rx=0 tx=0).stop 2024-04-21T08:57:25.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 -- 172.21.15.7:0/2534810367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac010c6d0 msgr2=0x7fbac0101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac010c6d0 0x7fbac0101350 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fbab0036040 tx=0x7fbab00045e0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 -- 172.21.15.7:0/2534810367 shutdown_connections 2024-04-21T08:57:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbac0101890 0x7fbac01a6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac010c6d0 0x7fbac0101350 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fbaac0a3d50 0x7fbaac0a6210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 --2- 172.21.15.7:0/2534810367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbac01031c0 0x7fbac0100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 -- 172.21.15.7:0/2534810367 >> 172.21.15.7:0/2534810367 conn(0x7fbac0077f70 msgr2=0x7fbac00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 -- 172.21.15.7:0/2534810367 shutdown_connections 2024-04-21T08:57:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:25.335+0000 7fbac76fd640 1 -- 172.21.15.7:0/2534810367 wait complete. 2024-04-21T08:57:25.346 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-21T08:57:25.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:25.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-21T08:57:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:25 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1797775112' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:57:25.554 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:25 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2195618803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:57:25.555 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:25 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3842799805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:57:25.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:25 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1797775112' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:57:25.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:25 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2195618803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:57:25.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:25 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3842799805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:57:25.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:25 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1797775112' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:57:25.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:25 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2195618803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:57:25.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:25 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3842799805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:57:26.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:26 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2534810367' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:57:26.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:26 smithi007 ceph-mon[28477]: pgmap v405: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 31 op/s 2024-04-21T08:57:26.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:26 smithi007 ceph-mon[28477]: osdmap e116: 12 total, 12 up, 12 in 2024-04-21T08:57:26.575 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi007.front.sepia.ceph.com 2024-04-21T08:57:26.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:26.575 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-21T08:57:26.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:26.601 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-04-21T08:57:26.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:26 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2534810367' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:57:26.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:26 smithi077 ceph-mon[31615]: pgmap v405: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 31 op/s 2024-04-21T08:57:26.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:26 smithi077 ceph-mon[31615]: osdmap e116: 12 total, 12 up, 12 in 2024-04-21T08:57:26.658 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-21T08:57:26.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:26.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T08:57:26.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:26.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link delete ceph-brx 2024-04-21T08:57:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:26 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2534810367' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:57:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:26 smithi016 ceph-mon[31259]: pgmap v405: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 31 op/s 2024-04-21T08:57:26.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:26 smithi016 ceph-mon[31259]: osdmap e116: 12 total, 12 up, 12 in 2024-04-21T08:57:26.817 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-21T08:57:26.817 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-21T08:57:26.818 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:57:26.834 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056881,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19958 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/a0f5e2f638aa21b276b9cbfec5aa931cc72863d67cd8bb3b969f2a66243d3a2d/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZIIGELFEOYBC55XD6EOJ3W55QZ:/var/lib/containers/storage/overlay/l/RW56ZXTELIAJNPPDOK2234TOKT,upperdir=/var/lib/containers/storage/overlay/a0f5e2f638aa21b276b9cbfec5aa931cc72863d67cd8bb3b969f2a66243d3a2d/diff,workdir=/var/lib/containers/storage/overlay/a0f5e2f638aa21b276b9cbfec5aa931cc72863d67cd8bb3b969f2a66243d3a2d/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d5489e31291d1b3f72e030f0bfbe67f721f48cccc09edc2ef9029db75edb1406/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZIIGELFEOYBC55XD6EOJ3W55QZ:/var/lib/containers/storage/overlay/l/RW56ZXTELIAJNPPDOK2234TOKT,upperdir=/var/lib/containers/storage/overlay/d5489e31291d1b3f72e030f0bfbe67f721f48cccc09edc2ef9029db75edb1406/diff,workdir=/var/lib/containers/storage/overlay/d5489e31291d1b3f72e030f0bfbe67f721f48cccc09edc2ef9029db75edb1406/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/bcfcc69acbe60c207a63adbce7ae90624b8d853921022c2df3b7215027b39f7e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZIIGELFEOYBC55XD6EOJ3W55QZ:/var/lib/containers/storage/overlay/l/RW56ZXTELIAJNPPDOK2234TOKT,upperdir=/var/lib/containers/storage/overlay/bcfcc69acbe60c207a63adbce7ae90624b8d853921022c2df3b7215027b39f7e/diff,workdir=/var/lib/containers/storage/overlay/bcfcc69acbe60c207a63adbce7ae90624b8d853921022c2df3b7215027b39f7e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/80a775629437d59ec89bbca28e609da213d430dca9579db258fe567ee425cab3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZIIGELFEOYBC55XD6EOJ3W55QZ:/var/lib/containers/storage/overlay/l/RW56ZXTELIAJNPPDOK2234TOKT,upperdir=/var/lib/containers/storage/overlay/80a775629437d59ec89bbca28e609da213d430dca9579db258fe567ee425cab3/diff,workdir=/var/lib/containers/storage/overlay/80a775629437d59ec89bbca28e609da213d430dca9579db258fe567ee425cab3/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e7de7116ed0211039967e54be58c48e52a9dd294318d8f85157e08d6b928366b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZIIGELFEOYBC55XD6EOJ3W55QZ:/var/lib/containers/storage/overlay/l/RW56ZXTELIAJNPPDOK2234TOKT,upperdir=/var/lib/containers/storage/overlay/e7de7116ed0211039967e54be58c48e52a9dd294318d8f85157e08d6b928366b/diff,workdir=/var/lib/containers/storage/overlay/e7de7116ed0211039967e54be58c48e52a9dd294318d8f85157e08d6b928366b/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/08d8e987a9d769654a6db6a33b3946386d26c82e54f726af547d5b340b4bb1b1/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZIIGELFEOYBC55XD6EOJ3W55QZ:/var/lib/containers/storage/overlay/l/RW56ZXTELIAJNPPDOK2234TOKT,upperdir=/var/lib/containers/storage/overlay/08d8e987a9d769654a6db6a33b3946386d26c82e54f726af547d5b340b4bb1b1/diff,workdir=/var/lib/containers/storage/overlay/08d8e987a9d769654a6db6a33b3946386d26c82e54f726af547d5b340b4bb1b1/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d6c148ecb737b74c6046760a001fd7c2deacba602caf8bf463776baa11b028c5/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZIIGELFEOYBC55XD6EOJ3W55QZ:/var/lib/containers/storage/overlay/l/RW56ZXTELIAJNPPDOK2234TOKT,upperdir=/var/lib/containers/storage/overlay/d6c148ecb737b74c6046760a001fd7c2deacba602caf8bf463776baa11b028c5/diff,workdir=/var/lib/containers/storage/overlay/d6c148ecb737b74c6046760a001fd7c2deacba602caf8bf463776baa11b028c5/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/869ba15e1bec9a8fd27be7e33609e021d0dfac9873d420e6718e83cee64e511c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZIIGELFEOYBC55XD6EOJ3W55QZ:/var/lib/containers/storage/overlay/l/RW56ZXTELIAJNPPDOK2234TOKT,upperdir=/var/lib/containers/storage/overlay/869ba15e1bec9a8fd27be7e33609e021d0dfac9873d420e6718e83cee64e511c/diff,workdir=/var/lib/containers/storage/overlay/869ba15e1bec9a8fd27be7e33609e021d0dfac9873d420e6718e83cee64e511c/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9125adcde3b2ceffc426d04bd4b6c9ad6b3b53f46f8a8727f97b144ba59da9e0/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZIIGELFEOYBC55XD6EOJ3W55QZ:/var/lib/containers/storage/overlay/l/RW56ZXTELIAJNPPDOK2234TOKT,upperdir=/var/lib/containers/storage/overlay/9125adcde3b2ceffc426d04bd4b6c9ad6b3b53f46f8a8727f97b144ba59da9e0/diff,workdir=/var/lib/containers/storage/overlay/9125adcde3b2ceffc426d04bd4b6c9ad6b3b53f46f8a8727f97b144ba59da9e0/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/70c0bd4f1e8503b924a5d7ec922d355fc787d4d7df152f4d888b4d16454f5e21/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZIIGELFEOYBC55XD6EOJ3W55QZ:/var/lib/containers/storage/overlay/l/RW56ZXTELIAJNPPDOK2234TOKT,upperdir=/var/lib/containers/storage/overlay/70c0bd4f1e8503b924a5d7ec922d355fc787d4d7df152f4d888b4d16454f5e21/diff,workdir=/var/lib/containers/storage/overlay/70c0bd4f1e8503b924a5d7ec922d355fc787d4d7df152f4d888b4d16454f5e21/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-21T08:57:26.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:26.834 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-04-21T08:57:26.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:26.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link delete ceph-brx 2024-04-21T08:57:26.958 INFO:teuthology.orchestra.run.smithi007.stderr:Cannot find device "ceph-brx" 2024-04-21T08:57:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:57:26.960 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-21T08:57:26.971 INFO:tasks.cephadm:Teardown begin 2024-04-21T08:57:26.971 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-21T08:57:26.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:57:27.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.103+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2506103095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce64069f10 msgr2=0x7fce64106be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.103+0000 7fce6a1b1640 1 --2- 172.21.15.7:0/2506103095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce64069f10 0x7fce64106be0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fce58009900 tx=0x7fce5802bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:27.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.103+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2506103095 shutdown_connections 2024-04-21T08:57:27.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.103+0000 7fce6a1b1640 1 --2- 172.21.15.7:0/2506103095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce641072c0 0x7fce6410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.103+0000 7fce6a1b1640 1 --2- 172.21.15.7:0/2506103095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce64069f10 0x7fce64106be0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.103+0000 7fce6a1b1640 1 --2- 172.21.15.7:0/2506103095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce64069560 0x7fce64069940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.103+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2506103095 >> 172.21.15.7:0/2506103095 conn(0x7fce640771d0 msgr2=0x7fce640775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.103+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2506103095 shutdown_connections 2024-04-21T08:57:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.103+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2506103095 wait complete. 2024-04-21T08:57:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce6a1b1640 1 Processor -- start 2024-04-21T08:57:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce6a1b1640 1 -- start start 2024-04-21T08:57:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce6a1b1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce64069560 0x7fce641986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce6a1b1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce641072c0 0x7fce64198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce6a1b1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6419cfa0 0x7fce6419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce6a1b1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fce64112160 con 0x7fce64069560 2024-04-21T08:57:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce6a1b1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fce64111fe0 con 0x7fce6419cfa0 2024-04-21T08:57:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce6a1b1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fce641122e0 con 0x7fce641072c0 2024-04-21T08:57:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce637fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce64069560 0x7fce641986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce637fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce64069560 0x7fce641986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43264/0 (socket says 172.21.15.7:43264) 2024-04-21T08:57:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce637fe640 1 -- 172.21.15.7:0/2662421077 learned_addr learned my addr 172.21.15.7:0/2662421077 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce62ffd640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce641072c0 0x7fce64198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.104+0000 7fce63fff640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6419cfa0 0x7fce6419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.105+0000 7fce637fe640 1 -- 172.21.15.7:0/2662421077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce641072c0 msgr2=0x7fce64198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.105+0000 7fce637fe640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce641072c0 0x7fce64198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.105+0000 7fce637fe640 1 -- 172.21.15.7:0/2662421077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6419cfa0 msgr2=0x7fce6419d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.105+0000 7fce637fe640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6419cfa0 0x7fce6419d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.105+0000 7fce637fe640 1 -- 172.21.15.7:0/2662421077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6419da90 con 0x7fce64069560 2024-04-21T08:57:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.105+0000 7fce637fe640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce64069560 0x7fce641986e0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fce5400c890 tx=0x7fce5400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:27.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.105+0000 7fce60ff9640 1 -- 172.21.15.7:0/2662421077 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54004220 con 0x7fce64069560 2024-04-21T08:57:27.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.105+0000 7fce60ff9640 1 -- 172.21.15.7:0/2662421077 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce540043c0 con 0x7fce64069560 2024-04-21T08:57:27.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.105+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2662421077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce6419dd80 con 0x7fce64069560 2024-04-21T08:57:27.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.105+0000 7fce60ff9640 1 -- 172.21.15.7:0/2662421077 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54004a00 con 0x7fce64069560 2024-04-21T08:57:27.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.105+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2662421077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce641a5540 con 0x7fce64069560 2024-04-21T08:57:27.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.106+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2662421077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce64068e00 con 0x7fce64069560 2024-04-21T08:57:27.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.107+0000 7fce60ff9640 1 -- 172.21.15.7:0/2662421077 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fce540079b0 con 0x7fce64069560 2024-04-21T08:57:27.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.108+0000 7fce60ff9640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fce4c0a3d20 0x7fce4c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.108+0000 7fce62ffd640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fce4c0a3d20 0x7fce4c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.108+0000 7fce60ff9640 1 -- 172.21.15.7:0/2662421077 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9568+0+0 (secure 0 0 0) 0x7fce540d08b0 con 0x7fce64069560 2024-04-21T08:57:27.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.108+0000 7fce62ffd640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fce4c0a3d20 0x7fce4c0a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fce58000c00 tx=0x7fce58005fd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:27.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.112+0000 7fce60ff9640 1 -- 172.21.15.7:0/2662421077 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce54014030 con 0x7fce64069560 2024-04-21T08:57:27.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.359+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2662421077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce64199a30 con 0x7fce64069560 2024-04-21T08:57:27.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.360+0000 7fce60ff9640 1 -- 172.21.15.7:0/2662421077 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7fce5409c4f0 con 0x7fce64069560 2024-04-21T08:57:27.360 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:57:27.360 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:57:27.360 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:57:27.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.362+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2662421077 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fce4c0a3d20 msgr2=0x7fce4c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.362+0000 7fce6a1b1640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fce4c0a3d20 0x7fce4c0a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fce58000c00 tx=0x7fce58005fd0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.362+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2662421077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce64069560 msgr2=0x7fce641986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.362+0000 7fce6a1b1640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce64069560 0x7fce641986e0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fce5400c890 tx=0x7fce5400cd50 comp rx=0 tx=0).stop 2024-04-21T08:57:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.362+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2662421077 shutdown_connections 2024-04-21T08:57:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.362+0000 7fce6a1b1640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce6419cfa0 0x7fce6419d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.362+0000 7fce6a1b1640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fce4c0a3d20 0x7fce4c0a61e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.362+0000 7fce6a1b1640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fce641072c0 0x7fce64198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.362+0000 7fce6a1b1640 1 --2- 172.21.15.7:0/2662421077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce64069560 0x7fce641986e0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.362+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2662421077 >> 172.21.15.7:0/2662421077 conn(0x7fce640771d0 msgr2=0x7fce640fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:27.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.363+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2662421077 shutdown_connections 2024-04-21T08:57:27.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.363+0000 7fce6a1b1640 1 -- 172.21.15.7:0/2662421077 wait complete. 2024-04-21T08:57:27.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:57:27.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.501+0000 7fa194915640 1 -- 172.21.15.7:0/2502467386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa190108060 msgr2=0x7fa190108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.501+0000 7fa194915640 1 --2- 172.21.15.7:0/2502467386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa190108060 0x7fa190108440 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fa180009930 tx=0x7fa18002bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:27.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.501+0000 7fa194915640 1 -- 172.21.15.7:0/2502467386 shutdown_connections 2024-04-21T08:57:27.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.501+0000 7fa194915640 1 --2- 172.21.15.7:0/2502467386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa19006b000 0x7fa19010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.501+0000 7fa194915640 1 --2- 172.21.15.7:0/2502467386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa19006a640 0x7fa19006aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.501+0000 7fa194915640 1 --2- 172.21.15.7:0/2502467386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa190108060 0x7fa190108440 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.501+0000 7fa194915640 1 -- 172.21.15.7:0/2502467386 >> 172.21.15.7:0/2502467386 conn(0x7fa190077c40 msgr2=0x7fa190078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.501+0000 7fa194915640 1 -- 172.21.15.7:0/2502467386 shutdown_connections 2024-04-21T08:57:27.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.501+0000 7fa194915640 1 -- 172.21.15.7:0/2502467386 wait complete. 2024-04-21T08:57:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa194915640 1 Processor -- start 2024-04-21T08:57:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa194915640 1 -- start start 2024-04-21T08:57:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa194915640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa19006a640 0x7fa1901986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa194915640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19006b000 0x7fa190198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa194915640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa190108060 0x7fa19019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa194915640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa190112180 con 0x7fa19006b000 2024-04-21T08:57:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa194915640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa190112000 con 0x7fa190108060 2024-04-21T08:57:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa194915640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa190112300 con 0x7fa19006a640 2024-04-21T08:57:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa18dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19006b000 0x7fa190198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa18dd9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19006b000 0x7fa190198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43270/0 (socket says 172.21.15.7:43270) 2024-04-21T08:57:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa18dd9b640 1 -- 172.21.15.7:0/2034656252 learned_addr learned my addr 172.21.15.7:0/2034656252 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.502+0000 7fa18ed9d640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa190108060 0x7fa19019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa18e59c640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa19006a640 0x7fa1901986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa18dd9b640 1 -- 172.21.15.7:0/2034656252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa19006a640 msgr2=0x7fa1901986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa18dd9b640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa19006a640 0x7fa1901986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa18dd9b640 1 -- 172.21.15.7:0/2034656252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa190108060 msgr2=0x7fa19019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa18dd9b640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa190108060 0x7fa19019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa18dd9b640 1 -- 172.21.15.7:0/2034656252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19019d6b0 con 0x7fa19006b000 2024-04-21T08:57:27.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa18dd9b640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19006b000 0x7fa190198c00 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa18400d950 tx=0x7fa18400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:27.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa17f7fe640 1 -- 172.21.15.7:0/2034656252 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184017070 con 0x7fa19006b000 2024-04-21T08:57:27.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa17f7fe640 1 -- 172.21.15.7:0/2034656252 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa184009c30 con 0x7fa19006b000 2024-04-21T08:57:27.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa194915640 1 -- 172.21.15.7:0/2034656252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa19019d9a0 con 0x7fa19006b000 2024-04-21T08:57:27.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa17f7fe640 1 -- 172.21.15.7:0/2034656252 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18400f620 con 0x7fa19006b000 2024-04-21T08:57:27.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.503+0000 7fa194915640 1 -- 172.21.15.7:0/2034656252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1901a5230 con 0x7fa19006b000 2024-04-21T08:57:27.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.504+0000 7fa194915640 1 -- 172.21.15.7:0/2034656252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa190103650 con 0x7fa19006b000 2024-04-21T08:57:27.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.505+0000 7fa17f7fe640 1 -- 172.21.15.7:0/2034656252 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa184005030 con 0x7fa19006b000 2024-04-21T08:57:27.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.506+0000 7fa17f7fe640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1780a3d20 0x7fa1780a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.506+0000 7fa18e59c640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1780a3d20 0x7fa1780a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.506+0000 7fa17f7fe640 1 -- 172.21.15.7:0/2034656252 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9568+0+0 (secure 0 0 0) 0x7fa1840d0870 con 0x7fa19006b000 2024-04-21T08:57:27.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.506+0000 7fa18e59c640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1780a3d20 0x7fa1780a61e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa180006010 tx=0x7fa18002c670 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:27.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.509+0000 7fa17f7fe640 1 -- 172.21.15.7:0/2034656252 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa18409cac0 con 0x7fa19006b000 2024-04-21T08:57:27.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:27 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2662421077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:27.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.737+0000 7fa194915640 1 -- 172.21.15.7:0/2034656252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa190069250 con 0x7fa19006b000 2024-04-21T08:57:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.738+0000 7fa17f7fe640 1 -- 172.21.15.7:0/2034656252 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11785 (secure 0 0 0) 0x7fa184009de0 con 0x7fa19006b000 2024-04-21T08:57:27.738 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:57:27.738 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2001227851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":2001227851},{"type":"v1","addr":"172.21.15.7:6835","nonce":2001227851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/1152814354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":1152814354},{"type":"v1","addr":"172.21.15.7:6837","nonce":1152814354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14607,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/674822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":674822179},{"type":"v1","addr":"172.21.15.77:6837","nonce":674822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/2635298189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":2635298189},{"type":"v1","addr":"172.21.15.7:6839","nonce":2635298189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/3576958984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":3576958984},{"type":"v1","addr":"172.21.15.7:6841","nonce":3576958984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2178684530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2178684530},{"type":"v1","addr":"172.21.15.16:6833","nonce":2178684530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24436,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1817896021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1817896021},{"type":"v1","addr":"172.21.15.77:6833","nonce":1817896021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/4020622072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":4020622072},{"type":"v1","addr":"172.21.15.16:6835","nonce":4020622072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/3699203122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":3699203122},{"type":"v1","addr":"172.21.15.16:6837","nonce":3699203122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/2205305943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":2205305943},{"type":"v1","addr":"172.21.15.77:6835","nonce":2205305943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:16.996517+0000","modified":"2024-04-21T08:46:09.889915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6839/1703654929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":1703654929},{"type":"v1","addr":"172.21.15.16:6839","nonce":1703654929}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25663":{"gid":25663,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.77:6839/3755986498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":3755986498},{"type":"v1","addr":"172.21.15.77:6839","nonce":3755986498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533]},"id":1}]} 2024-04-21T08:57:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2024-04-21T08:57:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 -- 172.21.15.7:0/2034656252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1780a3d20 msgr2=0x7fa1780a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1780a3d20 0x7fa1780a61e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa180006010 tx=0x7fa18002c670 comp rx=0 tx=0).stop 2024-04-21T08:57:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 -- 172.21.15.7:0/2034656252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19006b000 msgr2=0x7fa190198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19006b000 0x7fa190198c00 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa18400d950 tx=0x7fa18400de10 comp rx=0 tx=0).stop 2024-04-21T08:57:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 -- 172.21.15.7:0/2034656252 shutdown_connections 2024-04-21T08:57:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa190108060 0x7fa19019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa19006b000 0x7fa190198c00 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa1780a3d20 0x7fa1780a61e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 --2- 172.21.15.7:0/2034656252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa19006a640 0x7fa1901986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 -- 172.21.15.7:0/2034656252 >> 172.21.15.7:0/2034656252 conn(0x7fa190077c40 msgr2=0x7fa1900ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 -- 172.21.15.7:0/2034656252 shutdown_connections 2024-04-21T08:57:27.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.740+0000 7fa194915640 1 -- 172.21.15.7:0/2034656252 wait complete. 2024-04-21T08:57:27.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:57:27.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:27 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2662421077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:27.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.853+0000 7f6750515640 1 -- 172.21.15.7:0/4127070697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f674806b000 msgr2=0x7f674810ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.853+0000 7f6750515640 1 --2- 172.21.15.7:0/4127070697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f674806b000 0x7f674810ede0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f6740009cd0 tx=0x7f674002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.853+0000 7f6750515640 1 -- 172.21.15.7:0/4127070697 shutdown_connections 2024-04-21T08:57:27.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.853+0000 7f6750515640 1 --2- 172.21.15.7:0/4127070697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f674806b000 0x7f674810ede0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.853+0000 7f6750515640 1 --2- 172.21.15.7:0/4127070697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674806a640 0x7f674806aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.853+0000 7f6750515640 1 --2- 172.21.15.7:0/4127070697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6748108060 0x7f6748108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.853+0000 7f6750515640 1 -- 172.21.15.7:0/4127070697 >> 172.21.15.7:0/4127070697 conn(0x7f6748077c40 msgr2=0x7f6748078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:27.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.853+0000 7f6750515640 1 -- 172.21.15.7:0/4127070697 shutdown_connections 2024-04-21T08:57:27.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.853+0000 7f6750515640 1 -- 172.21.15.7:0/4127070697 wait complete. 2024-04-21T08:57:27.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.853+0000 7f6750515640 1 Processor -- start 2024-04-21T08:57:27.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f6750515640 1 -- start start 2024-04-21T08:57:27.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f6750515640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f674806a640 0x7f6748198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f6750515640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674806b000 0x7f6748198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f6750515640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6748108060 0x7f674819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f6750515640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f67481121a0 con 0x7f674806a640 2024-04-21T08:57:27.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f6750515640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6748112020 con 0x7f674806b000 2024-04-21T08:57:27.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f6750515640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6748112320 con 0x7f6748108060 2024-04-21T08:57:27.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674e2b1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f674806a640 0x7f6748198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674e2b1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f674806a640 0x7f6748198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43286/0 (socket says 172.21.15.7:43286) 2024-04-21T08:57:27.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674e2b1640 1 -- 172.21.15.7:0/3567188278 learned_addr learned my addr 172.21.15.7:0/3567188278 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:27.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674dab0640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674806b000 0x7f6748198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674eab2640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6748108060 0x7f674819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674e2b1640 1 -- 172.21.15.7:0/3567188278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6748108060 msgr2=0x7f674819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674e2b1640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6748108060 0x7f674819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674e2b1640 1 -- 172.21.15.7:0/3567188278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674806b000 msgr2=0x7f6748198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:27.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674e2b1640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674806b000 0x7f6748198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:27.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674e2b1640 1 -- 172.21.15.7:0/3567188278 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674819d5f0 con 0x7f674806a640 2024-04-21T08:57:27.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674eab2640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6748108060 0x7f674819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:27.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f674e2b1640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f674806a640 0x7f6748198600 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f674400d770 tx=0x7f674400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:27.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.854+0000 7f673f7fe640 1 -- 172.21.15.7:0/3567188278 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744017070 con 0x7f674806a640 2024-04-21T08:57:27.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.855+0000 7f673f7fe640 1 -- 172.21.15.7:0/3567188278 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67440054d0 con 0x7f674806a640 2024-04-21T08:57:27.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.855+0000 7f6750515640 1 -- 172.21.15.7:0/3567188278 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674819d8e0 con 0x7f674806a640 2024-04-21T08:57:27.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.855+0000 7f673f7fe640 1 -- 172.21.15.7:0/3567188278 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744004c30 con 0x7f674806a640 2024-04-21T08:57:27.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.855+0000 7f6750515640 1 -- 172.21.15.7:0/3567188278 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67481a5170 con 0x7f674806a640 2024-04-21T08:57:27.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:27 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2662421077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:27.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.856+0000 7f6750515640 1 -- 172.21.15.7:0/3567188278 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6748103650 con 0x7f674806a640 2024-04-21T08:57:27.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.857+0000 7f673f7fe640 1 -- 172.21.15.7:0/3567188278 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f6744004e10 con 0x7f674806a640 2024-04-21T08:57:27.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.857+0000 7f673f7fe640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67340a3d20 0x7f67340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:27.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.858+0000 7f673f7fe640 1 -- 172.21.15.7:0/3567188278 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9568+0+0 (secure 0 0 0) 0x7f67440d06e0 con 0x7f674806a640 2024-04-21T08:57:27.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.858+0000 7f674dab0640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67340a3d20 0x7f67340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:27.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.858+0000 7f674dab0640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67340a3d20 0x7f67340a61e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6748107620 tx=0x7f6738005f90 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:27.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:27.862+0000 7f673f7fe640 1 -- 172.21.15.7:0/3567188278 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f674409c970 con 0x7f674806a640 2024-04-21T08:57:28.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.050+0000 7f6750515640 1 -- 172.21.15.7:0/3567188278 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6748069250 con 0x7f674806a640 2024-04-21T08:57:28.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.052+0000 7f673f7fe640 1 -- 172.21.15.7:0/3567188278 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+25203 (secure 0 0 0) 0x7f674409c270 con 0x7f674806a640 2024-04-21T08:57:28.052 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-21T08:57:28.052 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":116,"fsid":"c9c60282-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:14.986230+0000","modified":"2024-04-21T08:57:26.259735+0000","last_up_change":"2024-04-21T08:41:24.405115+0000","last_in_change":"2024-04-21T08:41:03.547718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:37:33.631932+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:15.165234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:15.944352+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:19.645372+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c55caf13-53cc-4a4d-9324-fc3ae9c6f182","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6803","nonce":3999850241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6805","nonce":3999850241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6809","nonce":3999850241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3999850241},{"type":"v1","addr":"172.21.15.7:6807","nonce":3999850241}]},"public_addr":"172.21.15.7:6803/3999850241","cluster_addr":"172.21.15.7:6805/3999850241","heartbeat_back_addr":"172.21.15.7:6809/3999850241","heartbeat_front_addr":"172.21.15.7:6807/3999850241","state":["exists","up"]},{"osd":1,"uuid":"3d953784-5cd8-4152-b25e-c1b36ccc8635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6801","nonce":1911240202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6803","nonce":1911240202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6807","nonce":1911240202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":1911240202},{"type":"v1","addr":"172.21.15.16:6805","nonce":1911240202}]},"public_addr":"172.21.15.16:6801/1911240202","cluster_addr":"172.21.15.16:6803/1911240202","heartbeat_back_addr":"172.21.15.16:6807/1911240202","heartbeat_front_addr":"172.21.15.16:6805/1911240202","state":["exists","up"]},{"osd":2,"uuid":"fb1a5dad-fdc9-4a5b-a41d-de604769e356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6801","nonce":1494490623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6803","nonce":1494490623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6807","nonce":1494490623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1494490623},{"type":"v1","addr":"172.21.15.77:6805","nonce":1494490623}]},"public_addr":"172.21.15.77:6801/1494490623","cluster_addr":"172.21.15.77:6803/1494490623","heartbeat_back_addr":"172.21.15.77:6807/1494490623","heartbeat_front_addr":"172.21.15.77:6805/1494490623","state":["exists","up"]},{"osd":3,"uuid":"4b911243-4599-4632-9186-43b2714c8d39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6811","nonce":3942680881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6813","nonce":3942680881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6817","nonce":3942680881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":3942680881},{"type":"v1","addr":"172.21.15.7:6815","nonce":3942680881}]},"public_addr":"172.21.15.7:6811/3942680881","cluster_addr":"172.21.15.7:6813/3942680881","heartbeat_back_addr":"172.21.15.7:6817/3942680881","heartbeat_front_addr":"172.21.15.7:6815/3942680881","state":["exists","up"]},{"osd":4,"uuid":"a3dffc2c-f822-4b6e-9e2a-ffb53099e6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6809","nonce":849126848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6811","nonce":849126848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6815","nonce":849126848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":849126848},{"type":"v1","addr":"172.21.15.16:6813","nonce":849126848}]},"public_addr":"172.21.15.16:6809/849126848","cluster_addr":"172.21.15.16:6811/849126848","heartbeat_back_addr":"172.21.15.16:6815/849126848","heartbeat_front_addr":"172.21.15.16:6813/849126848","state":["exists","up"]},{"osd":5,"uuid":"93e0c8bb-dee9-4c3b-83f7-fd536ebdee21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6809","nonce":286186994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6811","nonce":286186994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6815","nonce":286186994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":286186994},{"type":"v1","addr":"172.21.15.77:6813","nonce":286186994}]},"public_addr":"172.21.15.77:6809/286186994","cluster_addr":"172.21.15.77:6811/286186994","heartbeat_back_addr":"172.21.15.77:6815/286186994","heartbeat_front_addr":"172.21.15.77:6813/286186994","state":["exists","up"]},{"osd":6,"uuid":"e1f722c8-d99f-47c0-8d9c-35d391a82058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6819","nonce":3776602687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6821","nonce":3776602687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6825","nonce":3776602687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3776602687},{"type":"v1","addr":"172.21.15.7:6823","nonce":3776602687}]},"public_addr":"172.21.15.7:6819/3776602687","cluster_addr":"172.21.15.7:6821/3776602687","heartbeat_back_addr":"172.21.15.7:6825/3776602687","heartbeat_front_addr":"172.21.15.7:6823/3776602687","state":["exists","up"]},{"osd":7,"uuid":"ce43f302-448d-4896-8a13-3f52413a2252","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6817","nonce":2331386881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6819","nonce":2331386881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6823","nonce":2331386881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2331386881},{"type":"v1","addr":"172.21.15.16:6821","nonce":2331386881}]},"public_addr":"172.21.15.16:6817/2331386881","cluster_addr":"172.21.15.16:6819/2331386881","heartbeat_back_addr":"172.21.15.16:6823/2331386881","heartbeat_front_addr":"172.21.15.16:6821/2331386881","state":["exists","up"]},{"osd":8,"uuid":"84019e97-520b-4134-93dc-67d0654f29fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6817","nonce":3684606717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6819","nonce":3684606717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6823","nonce":3684606717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3684606717},{"type":"v1","addr":"172.21.15.77:6821","nonce":3684606717}]},"public_addr":"172.21.15.77:6817/3684606717","cluster_addr":"172.21.15.77:6819/3684606717","heartbeat_back_addr":"172.21.15.77:6823/3684606717","heartbeat_front_addr":"172.21.15.77:6821/3684606717","state":["exists","up"]},{"osd":9,"uuid":"c7d8f7ae-dbba-42eb-9bc2-7a26d10f73e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6827","nonce":485389831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6829","nonce":485389831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6833","nonce":485389831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":485389831},{"type":"v1","addr":"172.21.15.7:6831","nonce":485389831}]},"public_addr":"172.21.15.7:6827/485389831","cluster_addr":"172.21.15.7:6829/485389831","heartbeat_back_addr":"172.21.15.7:6833/485389831","heartbeat_front_addr":"172.21.15.7:6831/485389831","state":["exists","up"]},{"osd":10,"uuid":"d4b74336-65ed-4ff2-a634-217a0e7ce75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6825","nonce":368172333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6827","nonce":368172333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6831","nonce":368172333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":368172333},{"type":"v1","addr":"172.21.15.16:6829","nonce":368172333}]},"public_addr":"172.21.15.16:6825/368172333","cluster_addr":"172.21.15.16:6827/368172333","heartbeat_back_addr":"172.21.15.16:6831/368172333","heartbeat_front_addr":"172.21.15.16:6829/368172333","state":["exists","up"]},{"osd":11,"uuid":"3ee0a8f3-b1fc-4fd6-a5e5-9ad38e14000c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6825","nonce":3994048303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6827","nonce":3994048303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6831","nonce":3994048303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3994048303},{"type":"v1","addr":"172.21.15.77:6829","nonce":3994048303}]},"public_addr":"172.21.15.77:6825/3994048303","cluster_addr":"172.21.15.77:6827/3994048303","heartbeat_back_addr":"172.21.15.77:6831/3994048303","heartbeat_front_addr":"172.21.15.77:6829/3994048303","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:36:42.517109+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:06.139705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:30.321407+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:54.843338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:20.234994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:45.483029+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:10.996981+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:36.918740+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:02.835117+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:29.398630+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:55.275298+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:22.191827+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":5,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3451243741":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/117789107":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/4167672076":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/1764088272":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3825837962":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2463266160":"2024-04-22T08:35:09.541897+0000","172.21.15.7:0/3631314876":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6801/102274072":"2024-04-22T08:35:09.541897+0000","172.21.15.7:6801/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2031045761":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/2555078032":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/865111185":"2024-04-22T08:34:42.863861+0000","172.21.15.7:0/3630897301":"2024-04-22T08:44:41.244700+0000","172.21.15.7:0/2778026231":"2024-04-22T08:34:42.863861+0000","172.21.15.7:6800/102274072":"2024-04-22T08:35:09.541897+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:57:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 -- 172.21.15.7:0/3567188278 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67340a3d20 msgr2=0x7f67340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67340a3d20 0x7f67340a61e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6748107620 tx=0x7f6738005f90 comp rx=0 tx=0).stop 2024-04-21T08:57:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 -- 172.21.15.7:0/3567188278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f674806a640 msgr2=0x7f6748198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f674806a640 0x7f6748198600 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f674400d770 tx=0x7f674400dc30 comp rx=0 tx=0).stop 2024-04-21T08:57:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 -- 172.21.15.7:0/3567188278 shutdown_connections 2024-04-21T08:57:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f67340a3d20 0x7f67340a61e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6748108060 0x7f674819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674806b000 0x7f6748198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 --2- 172.21.15.7:0/3567188278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f674806a640 0x7f6748198600 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 -- 172.21.15.7:0/3567188278 >> 172.21.15.7:0/3567188278 conn(0x7f6748077c40 msgr2=0x7f67480ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 -- 172.21.15.7:0/3567188278 shutdown_connections 2024-04-21T08:57:28.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.054+0000 7f6750515640 1 -- 172.21.15.7:0/3567188278 wait complete. 2024-04-21T08:57:28.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-21T08:57:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.168+0000 7f04449f4640 1 -- 172.21.15.7:0/1516506753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0440101a40 msgr2=0x7f0440105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.168+0000 7f04449f4640 1 --2- 172.21.15.7:0/1516506753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0440101a40 0x7f0440105a80 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f0428009900 tx=0x7f042802bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.168+0000 7f04449f4640 1 -- 172.21.15.7:0/1516506753 shutdown_connections 2024-04-21T08:57:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.168+0000 7f04449f4640 1 --2- 172.21.15.7:0/1516506753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0440106160 0x7f044010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.168+0000 7f04449f4640 1 --2- 172.21.15.7:0/1516506753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0440101a40 0x7f0440105a80 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.168+0000 7f04449f4640 1 --2- 172.21.15.7:0/1516506753 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0440101090 0x7f0440101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.168+0000 7f04449f4640 1 -- 172.21.15.7:0/1516506753 >> 172.21.15.7:0/1516506753 conn(0x7f0440078000 msgr2=0x7f0440078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.168+0000 7f04449f4640 1 -- 172.21.15.7:0/1516506753 shutdown_connections 2024-04-21T08:57:28.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.168+0000 7f04449f4640 1 -- 172.21.15.7:0/1516506753 wait complete. 2024-04-21T08:57:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.169+0000 7f04449f4640 1 Processor -- start 2024-04-21T08:57:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.169+0000 7f04449f4640 1 -- start start 2024-04-21T08:57:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.169+0000 7f04449f4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0440101090 0x7f0440102b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.169+0000 7f04449f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0440106160 0x7f0440103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.169+0000 7f04449f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04401035a0 0x7f04401a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.169+0000 7f04449f4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0440111080 con 0x7f0440101090 2024-04-21T08:57:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.169+0000 7f04449f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0440110f00 con 0x7f0440106160 2024-04-21T08:57:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.169+0000 7f04449f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0440111200 con 0x7f04401035a0 2024-04-21T08:57:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f043e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0440101090 0x7f0440102b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f043e59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0440101090 0x7f0440102b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43310/0 (socket says 172.21.15.7:43310) 2024-04-21T08:57:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f043e59c640 1 -- 172.21.15.7:0/4055822417 learned_addr learned my addr 172.21.15.7:0/4055822417 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f0435d9b640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0440106160 0x7f0440103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f043ed9d640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04401035a0 0x7f04401a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f043e59c640 1 -- 172.21.15.7:0/4055822417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04401035a0 msgr2=0x7f04401a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f043e59c640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04401035a0 0x7f04401a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f043e59c640 1 -- 172.21.15.7:0/4055822417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0440106160 msgr2=0x7f0440103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f043e59c640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0440106160 0x7f0440103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f043e59c640 1 -- 172.21.15.7:0/4055822417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04401a6f90 con 0x7f0440101090 2024-04-21T08:57:28.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f043ed9d640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04401035a0 0x7f04401a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f043e59c640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0440101090 0x7f0440102b20 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f043000c920 tx=0x7f043000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f0437fff640 1 -- 172.21.15.7:0/4055822417 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0430007c70 con 0x7f0440101090 2024-04-21T08:57:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f0437fff640 1 -- 172.21.15.7:0/4055822417 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0430007e10 con 0x7f0440101090 2024-04-21T08:57:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f04449f4640 1 -- 172.21.15.7:0/4055822417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04401a7280 con 0x7f0440101090 2024-04-21T08:57:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f0437fff640 1 -- 172.21.15.7:0/4055822417 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0430018820 con 0x7f0440101090 2024-04-21T08:57:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.170+0000 7f04449f4640 1 -- 172.21.15.7:0/4055822417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04401a77a0 con 0x7f0440101090 2024-04-21T08:57:28.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.171+0000 7f04449f4640 1 -- 172.21.15.7:0/4055822417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0440068e00 con 0x7f0440101090 2024-04-21T08:57:28.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.172+0000 7f0437fff640 1 -- 172.21.15.7:0/4055822417 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f043000f450 con 0x7f0440101090 2024-04-21T08:57:28.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.173+0000 7f0437fff640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f042c0a3d50 0x7f042c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:28.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.173+0000 7f0435d9b640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f042c0a3d50 0x7f042c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:28.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.173+0000 7f0437fff640 1 -- 172.21.15.7:0/4055822417 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9568+0+0 (secure 0 0 0) 0x7f04300d0d90 con 0x7f0440101090 2024-04-21T08:57:28.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.174+0000 7f0435d9b640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f042c0a3d50 0x7f042c0a6210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0428009230 tx=0x7f0428005fd0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:28.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.178+0000 7f0437fff640 1 -- 172.21.15.7:0/4055822417 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0430014030 con 0x7f0440101090 2024-04-21T08:57:28.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.407+0000 7f04449f4640 1 -- 172.21.15.7:0/4055822417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f044010a3a0 con 0x7f0440101090 2024-04-21T08:57:28.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.409+0000 7f0437fff640 1 -- 172.21.15.7:0/4055822417 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) ==== 161+0+0 (secure 0 0 0) 0x7f043009cfe0 con 0x7f0440101090 2024-04-21T08:57:28.409 INFO:teuthology.orchestra.run.smithi007.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-21T08:57:28.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 -- 172.21.15.7:0/4055822417 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f042c0a3d50 msgr2=0x7f042c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:28.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f042c0a3d50 0x7f042c0a6210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0428009230 tx=0x7f0428005fd0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 -- 172.21.15.7:0/4055822417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0440101090 msgr2=0x7f0440102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:28.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0440101090 0x7f0440102b20 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f043000c920 tx=0x7f043000cde0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 -- 172.21.15.7:0/4055822417 shutdown_connections 2024-04-21T08:57:28.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f042c0a3d50 0x7f042c0a6210 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04401035a0 0x7f04401a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0440106160 0x7f0440103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 --2- 172.21.15.7:0/4055822417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0440101090 0x7f0440102b20 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 -- 172.21.15.7:0/4055822417 >> 172.21.15.7:0/4055822417 conn(0x7f0440078000 msgr2=0x7f04400695d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:28.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 -- 172.21.15.7:0/4055822417 shutdown_connections 2024-04-21T08:57:28.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.413+0000 7f04449f4640 1 -- 172.21.15.7:0/4055822417 wait complete. 2024-04-21T08:57:28.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-21T08:57:28.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.532+0000 7fa312b6d640 1 -- 172.21.15.7:0/2690755625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa30c106170 msgr2=0x7fa30c10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:28.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.532+0000 7fa312b6d640 1 --2- 172.21.15.7:0/2690755625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa30c106170 0x7fa30c10dd40 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fa304009ca0 tx=0x7fa30402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.532+0000 7fa312b6d640 1 -- 172.21.15.7:0/2690755625 shutdown_connections 2024-04-21T08:57:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.532+0000 7fa312b6d640 1 --2- 172.21.15.7:0/2690755625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa30c106170 0x7fa30c10dd40 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.532+0000 7fa312b6d640 1 --2- 172.21.15.7:0/2690755625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa30c101a50 0x7fa30c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.532+0000 7fa312b6d640 1 --2- 172.21.15.7:0/2690755625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa30c1010a0 0x7fa30c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.532+0000 7fa312b6d640 1 -- 172.21.15.7:0/2690755625 >> 172.21.15.7:0/2690755625 conn(0x7fa30c077f40 msgr2=0x7fa30c078350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.532+0000 7fa312b6d640 1 -- 172.21.15.7:0/2690755625 shutdown_connections 2024-04-21T08:57:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.532+0000 7fa312b6d640 1 -- 172.21.15.7:0/2690755625 wait complete. 2024-04-21T08:57:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.533+0000 7fa312b6d640 1 Processor -- start 2024-04-21T08:57:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.533+0000 7fa312b6d640 1 -- start start 2024-04-21T08:57:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.533+0000 7fa312b6d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa30c1010a0 0x7fa30c102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.533+0000 7fa312b6d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa30c106170 0x7fa30c103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.533+0000 7fa312b6d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa30c1035b0 0x7fa30c1a69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.533+0000 7fa312b6d640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c111130 con 0x7fa30c1010a0 2024-04-21T08:57:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.533+0000 7fa312b6d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c110fb0 con 0x7fa30c106170 2024-04-21T08:57:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.533+0000 7fa312b6d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c1112b0 con 0x7fa30c1035b0 2024-04-21T08:57:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.533+0000 7fa310909640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa30c1010a0 0x7fa30c102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.533+0000 7fa310909640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa30c1010a0 0x7fa30c102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43318/0 (socket says 172.21.15.7:43318) 2024-04-21T08:57:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.533+0000 7fa310909640 1 -- 172.21.15.7:0/1523440021 learned_addr learned my addr 172.21.15.7:0/1523440021 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa30bfff640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa30c106170 0x7fa30c103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:28.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa31110a640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa30c1035b0 0x7fa30c1a69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:28.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa310909640 1 -- 172.21.15.7:0/1523440021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa30c1035b0 msgr2=0x7fa30c1a69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:28.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa310909640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa30c1035b0 0x7fa30c1a69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa310909640 1 -- 172.21.15.7:0/1523440021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa30c106170 msgr2=0x7fa30c103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:28.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa310909640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa30c106170 0x7fa30c103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:28.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa310909640 1 -- 172.21.15.7:0/1523440021 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30c1a70f0 con 0x7fa30c1010a0 2024-04-21T08:57:28.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa310909640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa30c1010a0 0x7fa30c102b30 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00e9f0 tx=0x7fa2fc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:28.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa309ffb640 1 -- 172.21.15.7:0/1523440021 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc00ccc0 con 0x7fa30c1010a0 2024-04-21T08:57:28.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa312b6d640 1 -- 172.21.15.7:0/1523440021 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30c1a73e0 con 0x7fa30c1010a0 2024-04-21T08:57:28.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa309ffb640 1 -- 172.21.15.7:0/1523440021 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2fc0108e0 con 0x7fa30c1010a0 2024-04-21T08:57:28.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa309ffb640 1 -- 172.21.15.7:0/1523440021 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc018880 con 0x7fa30c1010a0 2024-04-21T08:57:28.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.534+0000 7fa312b6d640 1 -- 172.21.15.7:0/1523440021 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa30c1a7980 con 0x7fa30c1010a0 2024-04-21T08:57:28.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.535+0000 7fa312b6d640 1 -- 172.21.15.7:0/1523440021 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa30c068570 con 0x7fa30c1010a0 2024-04-21T08:57:28.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.537+0000 7fa309ffb640 1 -- 172.21.15.7:0/1523440021 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa2fc010430 con 0x7fa30c1010a0 2024-04-21T08:57:28.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.538+0000 7fa309ffb640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa2f40a3d50 0x7fa2f40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:28.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.538+0000 7fa30bfff640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa2f40a3d50 0x7fa2f40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:28.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.538+0000 7fa309ffb640 1 -- 172.21.15.7:0/1523440021 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9610+0+0 (secure 0 0 0) 0x7fa2fc0d14b0 con 0x7fa30c1010a0 2024-04-21T08:57:28.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.538+0000 7fa30bfff640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa2f40a3d50 0x7fa2f40a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa30c1045c0 tx=0x7fa300009110 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:28.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.542+0000 7fa309ffb640 1 -- 172.21.15.7:0/1523440021 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa2fc09d6b0 con 0x7fa30c1010a0 2024-04-21T08:57:28.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:28.770+0000 7fa312b6d640 1 -- 172.21.15.7:0/1523440021 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa30c10a3b0 con 0x7fa30c1010a0 2024-04-21T08:57:28.804 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:57:28 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-b[61177]: ignoring --setuser ceph since I am not root 2024-04-21T08:57:28.804 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:57:28 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-b[61177]: ignoring --setgroup ceph since I am not root 2024-04-21T08:57:28.804 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:57:28 smithi016 ceph-mds[61181]: -- 172.21.15.16:0/819681415 <== mon.2 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561434361a40 con 0x561434334800 2024-04-21T08:57:28.805 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:57:28 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-b[61177]: starting mds.b at 2024-04-21T08:57:28.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:28 smithi016 ceph-mon[31259]: pgmap v407: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-04-21T08:57:28.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:28 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2034656252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:28.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:28 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/3567188278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:28.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:28 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4055822417' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:57:28.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:28 smithi016 ceph-mon[31259]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T08:57:28.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:28 smithi016 ceph-mon[31259]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:57:28.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:28 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4055822417' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T08:57:28.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:28 smithi077 ceph-mon[31615]: pgmap v407: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-04-21T08:57:28.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:28 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2034656252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:28.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:28 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/3567188278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:28.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:28 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4055822417' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:57:28.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:28 smithi077 ceph-mon[31615]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T08:57:28.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:28 smithi077 ceph-mon[31615]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:57:28.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:28 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4055822417' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T08:57:28.855 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:57:28 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i[65876]: ignoring --setuser ceph since I am not root 2024-04-21T08:57:28.855 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:57:28 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i[65876]: ignoring --setgroup ceph since I am not root 2024-04-21T08:57:28.855 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:57:28 smithi077 ceph-mds[65880]: -- 172.21.15.77:0/2544351825 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c4d3717a40 con 0x55c4d36eac00 2024-04-21T08:57:28.855 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:57:28 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i[65876]: starting mds.i at 2024-04-21T08:57:28.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:28 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a[28473]: 2024-04-21T08:57:28.407+0000 7fd79fe3d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:57:28.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:28 smithi007 ceph-mon[28477]: pgmap v407: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2024-04-21T08:57:28.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:28 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2034656252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:28.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:28 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/3567188278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:28.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:28 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4055822417' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:57:28.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:28 smithi007 ceph-mon[28477]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T08:57:28.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:28 smithi007 ceph-mon[28477]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:57:28.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:28 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4055822417' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T08:57:29.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.425+0000 7fa309ffb640 1 -- 172.21.15.7:0/1523440021 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7fa2fc09cfb0 con 0x7fa30c1010a0 2024-04-21T08:57:29.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 -- 172.21.15.7:0/1523440021 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa2f40a3d50 msgr2=0x7fa2f40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:29.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa2f40a3d50 0x7fa2f40a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa30c1045c0 tx=0x7fa300009110 comp rx=0 tx=0).stop 2024-04-21T08:57:29.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 -- 172.21.15.7:0/1523440021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa30c1010a0 msgr2=0x7fa30c102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:29.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa30c1010a0 0x7fa30c102b30 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00e9f0 tx=0x7fa2fc00edf0 comp rx=0 tx=0).stop 2024-04-21T08:57:29.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 -- 172.21.15.7:0/1523440021 shutdown_connections 2024-04-21T08:57:29.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fa2f40a3d50 0x7fa2f40a6210 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:29.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa30c1035b0 0x7fa30c1a69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:29.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa30c106170 0x7fa30c103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:29.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 --2- 172.21.15.7:0/1523440021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa30c1010a0 0x7fa30c102b30 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:29.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 -- 172.21.15.7:0/1523440021 >> 172.21.15.7:0/1523440021 conn(0x7fa30c077f40 msgr2=0x7fa30c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:29.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 -- 172.21.15.7:0/1523440021 shutdown_connections 2024-04-21T08:57:29.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.429+0000 7fa312b6d640 1 -- 172.21.15.7:0/1523440021 wait complete. 2024-04-21T08:57:29.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-21T08:57:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.555+0000 7f38bd332640 1 -- 172.21.15.7:0/1044873252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8101a50 msgr2=0x7f38b8105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.555+0000 7f38bd332640 1 --2- 172.21.15.7:0/1044873252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8101a50 0x7f38b8105a90 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f38a0009900 tx=0x7f38a002bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.555+0000 7f38bd332640 1 -- 172.21.15.7:0/1044873252 shutdown_connections 2024-04-21T08:57:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.555+0000 7f38bd332640 1 --2- 172.21.15.7:0/1044873252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b8106170 0x7f38b810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.555+0000 7f38bd332640 1 --2- 172.21.15.7:0/1044873252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8101a50 0x7f38b8105a90 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.555+0000 7f38bd332640 1 --2- 172.21.15.7:0/1044873252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38b81010a0 0x7f38b8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.555+0000 7f38bd332640 1 -- 172.21.15.7:0/1044873252 >> 172.21.15.7:0/1044873252 conn(0x7f38b8077f40 msgr2=0x7f38b8078350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.555+0000 7f38bd332640 1 -- 172.21.15.7:0/1044873252 shutdown_connections 2024-04-21T08:57:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.555+0000 7f38bd332640 1 -- 172.21.15.7:0/1044873252 wait complete. 2024-04-21T08:57:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38bd332640 1 Processor -- start 2024-04-21T08:57:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38bd332640 1 -- start start 2024-04-21T08:57:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38bd332640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38b81010a0 0x7f38b81986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38bd332640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b8106170 0x7f38b8198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38bd332640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b819cf80 0x7f38b819d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:29.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38bd332640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f38b8111090 con 0x7f38b819cf80 2024-04-21T08:57:29.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38bd332640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f38b8110f10 con 0x7f38b8106170 2024-04-21T08:57:29.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38bd332640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f38b8111210 con 0x7f38b81010a0 2024-04-21T08:57:29.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38b77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b819cf80 0x7f38b819d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:29.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38b77fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b819cf80 0x7f38b819d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43338/0 (socket says 172.21.15.7:43338) 2024-04-21T08:57:29.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38b77fe640 1 -- 172.21.15.7:0/2021200648 learned_addr learned my addr 172.21.15.7:0/2021200648 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:29.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.556+0000 7f38affff640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b8106170 0x7f38b8198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:29.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38b6ffd640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38b81010a0 0x7f38b81986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:29.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38b77fe640 1 -- 172.21.15.7:0/2021200648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38b81010a0 msgr2=0x7f38b81986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:29.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38b77fe640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38b81010a0 0x7f38b81986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:29.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38b77fe640 1 -- 172.21.15.7:0/2021200648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b8106170 msgr2=0x7f38b8198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:29.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38b77fe640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b8106170 0x7f38b8198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:29.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38b77fe640 1 -- 172.21.15.7:0/2021200648 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b819da70 con 0x7f38b819cf80 2024-04-21T08:57:29.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38b77fe640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b819cf80 0x7f38b819d430 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f38b000eaa0 tx=0x7f38b000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:29.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38b4ff9640 1 -- 172.21.15.7:0/2021200648 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b000cdb0 con 0x7f38b819cf80 2024-04-21T08:57:29.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38b4ff9640 1 -- 172.21.15.7:0/2021200648 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38b0005020 con 0x7f38b819cf80 2024-04-21T08:57:29.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38bd332640 1 -- 172.21.15.7:0/2021200648 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b819dd60 con 0x7f38b819cf80 2024-04-21T08:57:29.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38b4ff9640 1 -- 172.21.15.7:0/2021200648 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b00116d0 con 0x7f38b819cf80 2024-04-21T08:57:29.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.557+0000 7f38bd332640 1 -- 172.21.15.7:0/2021200648 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b81a54d0 con 0x7f38b819cf80 2024-04-21T08:57:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.558+0000 7f38bd332640 1 -- 172.21.15.7:0/2021200648 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b8102570 con 0x7f38b819cf80 2024-04-21T08:57:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.559+0000 7f38b4ff9640 1 -- 172.21.15.7:0/2021200648 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f38b00051d0 con 0x7f38b819cf80 2024-04-21T08:57:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.560+0000 7f38b4ff9640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38a40a3d20 0x7f38a40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.560+0000 7f38b4ff9640 1 -- 172.21.15.7:0/2021200648 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9610+0+0 (secure 0 0 0) 0x7f38b00d0de0 con 0x7f38b819cf80 2024-04-21T08:57:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.560+0000 7f38b6ffd640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38a40a3d20 0x7f38a40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.560+0000 7f38b6ffd640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38a40a3d20 0x7f38a40a61e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f38a8005e10 tx=0x7f38a8005da0 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:29.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.565+0000 7f38b4ff9640 1 -- 172.21.15.7:0/2021200648 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38b0016030 con 0x7f38b819cf80 2024-04-21T08:57:29.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:29.754+0000 7f38bd332640 1 -- 172.21.15.7:0/2021200648 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f38b8101a50 con 0x7f38b819cf80 2024-04-21T08:57:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:29 smithi016 ceph-mon[31259]: osdmap e117: 12 total, 12 up, 12 in 2024-04-21T08:57:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:29 smithi016 ceph-mon[31259]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-21T08:57:29.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:29 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1523440021' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:57:29.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:29 smithi077 ceph-mon[31615]: osdmap e117: 12 total, 12 up, 12 in 2024-04-21T08:57:29.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:29 smithi077 ceph-mon[31615]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-21T08:57:29.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:29 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1523440021' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:57:29.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:29 smithi007 ceph-mon[28477]: osdmap e117: 12 total, 12 up, 12 in 2024-04-21T08:57:29.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:29 smithi007 ceph-mon[28477]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-21T08:57:29.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:29 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1523440021' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:57:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.431+0000 7f38b4ff9640 1 -- 172.21.15.7:0/2021200648 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) ==== 181+0+0 (secure 0 0 0) 0x7f38b009d040 con 0x7f38b819cf80 2024-04-21T08:57:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_metadata' removed 2024-04-21T08:57:30.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.435+0000 7f38bd332640 1 -- 172.21.15.7:0/2021200648 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38a40a3d20 msgr2=0x7f38a40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:30.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.435+0000 7f38bd332640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38a40a3d20 0x7f38a40a61e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f38a8005e10 tx=0x7f38a8005da0 comp rx=0 tx=0).stop 2024-04-21T08:57:30.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.435+0000 7f38bd332640 1 -- 172.21.15.7:0/2021200648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b819cf80 msgr2=0x7f38b819d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:30.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.435+0000 7f38bd332640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b819cf80 0x7f38b819d430 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f38b000eaa0 tx=0x7f38b000ef60 comp rx=0 tx=0).stop 2024-04-21T08:57:30.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.435+0000 7f38bd332640 1 -- 172.21.15.7:0/2021200648 shutdown_connections 2024-04-21T08:57:30.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.435+0000 7f38bd332640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b819cf80 0x7f38b819d430 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:30.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.435+0000 7f38bd332640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b8106170 0x7f38b8198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:30.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.435+0000 7f38bd332640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f38a40a3d20 0x7f38a40a61e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.435+0000 7f38bd332640 1 --2- 172.21.15.7:0/2021200648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38b81010a0 0x7f38b81986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.435+0000 7f38bd332640 1 -- 172.21.15.7:0/2021200648 >> 172.21.15.7:0/2021200648 conn(0x7f38b8077f40 msgr2=0x7f38b806a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.436+0000 7f38bd332640 1 -- 172.21.15.7:0/2021200648 shutdown_connections 2024-04-21T08:57:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.436+0000 7f38bd332640 1 -- 172.21.15.7:0/2021200648 wait complete. 2024-04-21T08:57:30.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-21T08:57:30.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.565+0000 7fd15092b640 1 -- 172.21.15.7:0/159913117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd14c103b50 msgr2=0x7fd14c103fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:30.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.565+0000 7fd15092b640 1 --2- 172.21.15.7:0/159913117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd14c103b50 0x7fd14c103fb0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd140009930 tx=0x7fd14002bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:30.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.565+0000 7fd15092b640 1 -- 172.21.15.7:0/159913117 shutdown_connections 2024-04-21T08:57:30.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.565+0000 7fd15092b640 1 --2- 172.21.15.7:0/159913117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd14c1044f0 0x7fd14c10eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:30.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.565+0000 7fd15092b640 1 --2- 172.21.15.7:0/159913117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd14c103b50 0x7fd14c103fb0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:30.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.565+0000 7fd15092b640 1 --2- 172.21.15.7:0/159913117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd14c102270 0x7fd14c102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:30.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.565+0000 7fd15092b640 1 -- 172.21.15.7:0/159913117 >> 172.21.15.7:0/159913117 conn(0x7fd14c0fda10 msgr2=0x7fd14c0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:30.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.565+0000 7fd15092b640 1 -- 172.21.15.7:0/159913117 shutdown_connections 2024-04-21T08:57:30.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.565+0000 7fd15092b640 1 -- 172.21.15.7:0/159913117 wait complete. 2024-04-21T08:57:30.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd15092b640 1 Processor -- start 2024-04-21T08:57:30.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd15092b640 1 -- start start 2024-04-21T08:57:30.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd15092b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd14c102270 0x7fd14c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:30.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd15092b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd14c103b50 0x7fd14c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:30.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd15092b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd14c1044f0 0x7fd14c074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:30.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd15092b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c112270 con 0x7fd14c103b50 2024-04-21T08:57:30.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd15092b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c1120f0 con 0x7fd14c102270 2024-04-21T08:57:30.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd15092b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c1123f0 con 0x7fd14c1044f0 2024-04-21T08:57:30.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd149d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd14c103b50 0x7fd14c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:30.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd149d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd14c103b50 0x7fd14c074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43360/0 (socket says 172.21.15.7:43360) 2024-04-21T08:57:30.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd149d9b640 1 -- 172.21.15.7:0/1156187434 learned_addr learned my addr 172.21.15.7:0/1156187434 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:30.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd14ad9d640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd14c1044f0 0x7fd14c074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:30.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd149d9b640 1 -- 172.21.15.7:0/1156187434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd14c1044f0 msgr2=0x7fd14c074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:30.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd14a59c640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd14c102270 0x7fd14c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:30.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd149d9b640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd14c1044f0 0x7fd14c074cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:30.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd149d9b640 1 -- 172.21.15.7:0/1156187434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd14c102270 msgr2=0x7fd14c074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:30.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd149d9b640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd14c102270 0x7fd14c074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:30.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.566+0000 7fd149d9b640 1 -- 172.21.15.7:0/1156187434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd14c1a6cd0 con 0x7fd14c103b50 2024-04-21T08:57:30.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.567+0000 7fd149d9b640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd14c103b50 0x7fd14c074790 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd140009230 tx=0x7fd1400045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:30.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.567+0000 7fd13b7fe640 1 -- 172.21.15.7:0/1156187434 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140043020 con 0x7fd14c103b50 2024-04-21T08:57:30.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.567+0000 7fd13b7fe640 1 -- 172.21.15.7:0/1156187434 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd14002cb70 con 0x7fd14c103b50 2024-04-21T08:57:30.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.567+0000 7fd15092b640 1 -- 172.21.15.7:0/1156187434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd14c1a6f60 con 0x7fd14c103b50 2024-04-21T08:57:30.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.567+0000 7fd13b7fe640 1 -- 172.21.15.7:0/1156187434 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140003b50 con 0x7fd14c103b50 2024-04-21T08:57:30.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.567+0000 7fd15092b640 1 -- 172.21.15.7:0/1156187434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd14c1a7420 con 0x7fd14c103b50 2024-04-21T08:57:30.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.568+0000 7fd15092b640 1 -- 172.21.15.7:0/1156187434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd14c069250 con 0x7fd14c103b50 2024-04-21T08:57:30.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.569+0000 7fd13b7fe640 1 -- 172.21.15.7:0/1156187434 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd140046020 con 0x7fd14c103b50 2024-04-21T08:57:30.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.569+0000 7fd13b7fe640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1340a3d20 0x7fd1340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:30.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.570+0000 7fd14a59c640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1340a3d20 0x7fd1340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:30.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.570+0000 7fd13b7fe640 1 -- 172.21.15.7:0/1156187434 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9027+0+0 (secure 0 0 0) 0x7fd1400f2bf0 con 0x7fd14c103b50 2024-04-21T08:57:30.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.570+0000 7fd14a59c640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1340a3d20 0x7fd1340a61e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0058c0 tx=0x7fd13c005c60 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:30.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.574+0000 7fd13b7fe640 1 -- 172.21.15.7:0/1156187434 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd14004d020 con 0x7fd14c103b50 2024-04-21T08:57:30.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:30.761+0000 7fd15092b640 1 -- 172.21.15.7:0/1156187434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd14c10a540 con 0x7fd14c103b50 2024-04-21T08:57:30.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:30 smithi016 ceph-mon[31259]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T08:57:30.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:30 smithi016 ceph-mon[31259]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T08:57:30.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:30 smithi016 ceph-mon[31259]: Cluster is now healthy 2024-04-21T08:57:30.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:30 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1523440021' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:57:30.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:30 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:57:30.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:30 smithi016 ceph-mon[31259]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:57:30.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:30 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.77:6838/4140716318,v1:172.21.15.77:6839/4140716318] up:boot 2024-04-21T08:57:30.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:30 smithi016 ceph-mon[31259]: mds.? [v2:172.21.15.16:6838/262642674,v1:172.21.15.16:6839/262642674] up:boot 2024-04-21T08:57:30.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:30 smithi016 ceph-mon[31259]: fsmap 12 up:standby 2024-04-21T08:57:30.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:30 smithi016 ceph-mon[31259]: pgmap v409: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 738 KiB/s wr, 20 op/s 2024-04-21T08:57:30.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:30 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2021200648' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:30 smithi077 ceph-mon[31615]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T08:57:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:30 smithi077 ceph-mon[31615]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T08:57:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:30 smithi077 ceph-mon[31615]: Cluster is now healthy 2024-04-21T08:57:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:30 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1523440021' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:57:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:30 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:57:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:30 smithi077 ceph-mon[31615]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:57:30.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:30 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.77:6838/4140716318,v1:172.21.15.77:6839/4140716318] up:boot 2024-04-21T08:57:30.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:30 smithi077 ceph-mon[31615]: mds.? [v2:172.21.15.16:6838/262642674,v1:172.21.15.16:6839/262642674] up:boot 2024-04-21T08:57:30.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:30 smithi077 ceph-mon[31615]: fsmap 12 up:standby 2024-04-21T08:57:30.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:30 smithi077 ceph-mon[31615]: pgmap v409: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 738 KiB/s wr, 20 op/s 2024-04-21T08:57:30.855 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:30 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2021200648' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:30 smithi007 ceph-mon[28477]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T08:57:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:30 smithi007 ceph-mon[28477]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T08:57:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:30 smithi007 ceph-mon[28477]: Cluster is now healthy 2024-04-21T08:57:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:30 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1523440021' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:57:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:30 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:57:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:30 smithi007 ceph-mon[28477]: from='mgr.15408 172.21.15.7:0/3156474090' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:57:30.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:30 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.77:6838/4140716318,v1:172.21.15.77:6839/4140716318] up:boot 2024-04-21T08:57:30.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:30 smithi007 ceph-mon[28477]: mds.? [v2:172.21.15.16:6838/262642674,v1:172.21.15.16:6839/262642674] up:boot 2024-04-21T08:57:30.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:30 smithi007 ceph-mon[28477]: fsmap 12 up:standby 2024-04-21T08:57:30.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:30 smithi007 ceph-mon[28477]: pgmap v409: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 738 KiB/s wr, 20 op/s 2024-04-21T08:57:30.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:30 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2021200648' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:31.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.438+0000 7fd13b7fe640 1 -- 172.21.15.7:0/1156187434 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v119) ==== 169+0+0 (secure 0 0 0) 0x7fd14003c350 con 0x7fd14c103b50 2024-04-21T08:57:31.438 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data' removed 2024-04-21T08:57:31.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.441+0000 7fd15092b640 1 -- 172.21.15.7:0/1156187434 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1340a3d20 msgr2=0x7fd1340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:31.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.441+0000 7fd15092b640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1340a3d20 0x7fd1340a61e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0058c0 tx=0x7fd13c005c60 comp rx=0 tx=0).stop 2024-04-21T08:57:31.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.441+0000 7fd15092b640 1 -- 172.21.15.7:0/1156187434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd14c103b50 msgr2=0x7fd14c074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:31.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.441+0000 7fd15092b640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd14c103b50 0x7fd14c074790 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd140009230 tx=0x7fd1400045e0 comp rx=0 tx=0).stop 2024-04-21T08:57:31.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.441+0000 7fd15092b640 1 -- 172.21.15.7:0/1156187434 shutdown_connections 2024-04-21T08:57:31.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.441+0000 7fd15092b640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7fd1340a3d20 0x7fd1340a61e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:31.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.441+0000 7fd15092b640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd14c1044f0 0x7fd14c074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:31.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.441+0000 7fd15092b640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd14c103b50 0x7fd14c074790 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:31.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.441+0000 7fd15092b640 1 --2- 172.21.15.7:0/1156187434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd14c102270 0x7fd14c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:31.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.441+0000 7fd15092b640 1 -- 172.21.15.7:0/1156187434 >> 172.21.15.7:0/1156187434 conn(0x7fd14c0fda10 msgr2=0x7fd14c0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:31.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.442+0000 7fd15092b640 1 -- 172.21.15.7:0/1156187434 shutdown_connections 2024-04-21T08:57:31.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.442+0000 7fd15092b640 1 -- 172.21.15.7:0/1156187434 wait complete. 2024-04-21T08:57:31.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-21T08:57:31.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.607+0000 7f34e03c3640 1 -- 172.21.15.7:0/2465687928 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34d8107240 msgr2=0x7f34d810ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:31.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.607+0000 7f34e03c3640 1 --2- 172.21.15.7:0/2465687928 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34d8107240 0x7f34d810ee10 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009900 tx=0x7f34d002bc40 comp rx=0 tx=0).stop 2024-04-21T08:57:31.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.607+0000 7f34e03c3640 1 -- 172.21.15.7:0/2465687928 shutdown_connections 2024-04-21T08:57:31.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.607+0000 7f34e03c3640 1 --2- 172.21.15.7:0/2465687928 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34d8107240 0x7f34d810ee10 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:31.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.607+0000 7f34e03c3640 1 --2- 172.21.15.7:0/2465687928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d8102b20 0x7f34d8106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:31.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.607+0000 7f34e03c3640 1 --2- 172.21.15.7:0/2465687928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d8102170 0x7f34d8102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:31.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.607+0000 7f34e03c3640 1 -- 172.21.15.7:0/2465687928 >> 172.21.15.7:0/2465687928 conn(0x7f34d80fd9b0 msgr2=0x7f34d80ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:31.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.607+0000 7f34e03c3640 1 -- 172.21.15.7:0/2465687928 shutdown_connections 2024-04-21T08:57:31.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.607+0000 7f34e03c3640 1 -- 172.21.15.7:0/2465687928 wait complete. 2024-04-21T08:57:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34e03c3640 1 Processor -- start 2024-04-21T08:57:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34e03c3640 1 -- start start 2024-04-21T08:57:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34e03c3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34d8102170 0x7f34d810f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34e03c3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d8107240 0x7f34d810fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34e03c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d8110020 0x7f34d81aae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34e03c3640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f34d8114410 con 0x7f34d8102170 2024-04-21T08:57:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34e03c3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34d8114290 con 0x7f34d8110020 2024-04-21T08:57:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34e03c3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f34d8114590 con 0x7f34d8107240 2024-04-21T08:57:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34de960640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d8110020 0x7f34d81aae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34dd95e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d8107240 0x7f34d810fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34de960640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d8110020 0x7f34d81aae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.7:36808/0 (socket says 172.21.15.7:36808) 2024-04-21T08:57:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34de960640 1 -- 172.21.15.7:0/4203023539 learned_addr learned my addr 172.21.15.7:0/4203023539 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-04-21T08:57:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.608+0000 7f34de15f640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34d8102170 0x7f34d810f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34de15f640 1 -- 172.21.15.7:0/4203023539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d8107240 msgr2=0x7f34d810fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34de15f640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d8107240 0x7f34d810fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34de15f640 1 -- 172.21.15.7:0/4203023539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d8110020 msgr2=0x7f34d81aae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34de15f640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d8110020 0x7f34d81aae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34de15f640 1 -- 172.21.15.7:0/4203023539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d81ab3c0 con 0x7f34d8102170 2024-04-21T08:57:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34de960640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d8110020 0x7f34d81aae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:57:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34dd95e640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d8107240 0x7f34d810fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:57:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34de15f640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34d8102170 0x7f34d810f5a0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00e990 tx=0x7f34cc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34cb7fe640 1 -- 172.21.15.7:0/4203023539 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00cbe0 con 0x7f34d8102170 2024-04-21T08:57:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34e03c3640 1 -- 172.21.15.7:0/4203023539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d81ab6b0 con 0x7f34d8102170 2024-04-21T08:57:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34cb7fe640 1 -- 172.21.15.7:0/4203023539 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34cc00cd80 con 0x7f34d8102170 2024-04-21T08:57:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34cb7fe640 1 -- 172.21.15.7:0/4203023539 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00cbe0 con 0x7f34d8102170 2024-04-21T08:57:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.609+0000 7f34e03c3640 1 -- 172.21.15.7:0/4203023539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d81abc20 con 0x7f34d8102170 2024-04-21T08:57:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.611+0000 7f34e03c3640 1 -- 172.21.15.7:0/4203023539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d8103640 con 0x7f34d8102170 2024-04-21T08:57:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.612+0000 7f34cb7fe640 1 -- 172.21.15.7:0/4203023539 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f34cc0119b0 con 0x7f34d8102170 2024-04-21T08:57:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.613+0000 7f34cb7fe640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34c40a3f20 0x7f34c40a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.613+0000 7f34dd95e640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34c40a3f20 0x7f34c40a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.614+0000 7f34cb7fe640 1 -- 172.21.15.7:0/4203023539 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8410+0+0 (secure 0 0 0) 0x7f34cc0d0530 con 0x7f34d8102170 2024-04-21T08:57:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.614+0000 7f34dd95e640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34c40a3f20 0x7f34c40a63e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009fd0 tx=0x7f34d4009680 comp rx=0 tx=0).ready entity=mgr.15408 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.620+0000 7f34cb7fe640 1 -- 172.21.15.7:0/4203023539 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34cc01f060 con 0x7f34d8102170 2024-04-21T08:57:31.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:31 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/2021200648' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:31.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:31 smithi016 ceph-mon[31259]: osdmap e118: 12 total, 12 up, 12 in 2024-04-21T08:57:31.805 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:31 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1156187434' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:31.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:31 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/2021200648' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:31.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:31 smithi077 ceph-mon[31615]: osdmap e118: 12 total, 12 up, 12 in 2024-04-21T08:57:31.854 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:31 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1156187434' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:31.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:31 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/2021200648' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:31.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:31 smithi007 ceph-mon[28477]: osdmap e118: 12 total, 12 up, 12 in 2024-04-21T08:57:31.902 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:31 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1156187434' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:31.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:31.924+0000 7f34e03c3640 1 -- 172.21.15.7:0/4203023539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f34d810a470 con 0x7f34d8102170 2024-04-21T08:57:32.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.439+0000 7f34cb7fe640 1 -- 172.21.15.7:0/4203023539 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v120) ==== 178+0+0 (secure 0 0 0) 0x7f34cc01f060 con 0x7f34d8102170 2024-04-21T08:57:32.439 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data_ec' removed 2024-04-21T08:57:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.442+0000 7f34e03c3640 1 -- 172.21.15.7:0/4203023539 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34c40a3f20 msgr2=0x7f34c40a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.442+0000 7f34e03c3640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34c40a3f20 0x7f34c40a63e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009fd0 tx=0x7f34d4009680 comp rx=0 tx=0).stop 2024-04-21T08:57:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.443+0000 7f34e03c3640 1 -- 172.21.15.7:0/4203023539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34d8102170 msgr2=0x7f34d810f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.443+0000 7f34e03c3640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34d8102170 0x7f34d810f5a0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00e990 tx=0x7f34cc00ee50 comp rx=0 tx=0).stop 2024-04-21T08:57:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.443+0000 7f34e03c3640 1 -- 172.21.15.7:0/4203023539 shutdown_connections 2024-04-21T08:57:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.443+0000 7f34e03c3640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34d8110020 0x7f34d81aae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.443+0000 7f34e03c3640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.7:6800/3443135673,v1:172.21.15.7:6801/3443135673] conn(0x7f34c40a3f20 0x7f34c40a63e0 secure :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009fd0 tx=0x7f34d4009680 comp rx=0 tx=0).stop 2024-04-21T08:57:32.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.443+0000 7f34e03c3640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34d8107240 0x7f34d810fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:32.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.443+0000 7f34e03c3640 1 --2- 172.21.15.7:0/4203023539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34d8102170 0x7f34d810f5a0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:32.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.443+0000 7f34e03c3640 1 -- 172.21.15.7:0/4203023539 >> 172.21.15.7:0/4203023539 conn(0x7f34d80fd9b0 msgr2=0x7f34d80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:32.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.445+0000 7f34e03c3640 1 -- 172.21.15.7:0/4203023539 shutdown_connections 2024-04-21T08:57:32.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-04-21T08:57:32.445+0000 7f34e03c3640 1 -- 172.21.15.7:0/4203023539 wait complete. 2024-04-21T08:57:32.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:57:32.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:57:32.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:57:32.570 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-21T08:57:32.570 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:57:32.600 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:57:32.624 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:57:32.640 INFO:tasks.cephadm:Stopping all daemons... 2024-04-21T08:57:32.640 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-21T08:57:32.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.a 2024-04-21T08:57:32.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:32 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/1156187434' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:32.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:32 smithi007 ceph-mon[28477]: osdmap e119: 12 total, 12 up, 12 in 2024-04-21T08:57:32.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:32 smithi007 ceph-mon[28477]: pgmap v412: 33 pgs: 33 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:57:32.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:32 smithi007 ceph-mon[28477]: from='client.? 172.21.15.7:0/4203023539' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:32.698 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:32 smithi007 systemd[1]: Stopping Ceph mon.a for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:32.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:32 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/1156187434' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:32.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:32 smithi077 ceph-mon[31615]: osdmap e119: 12 total, 12 up, 12 in 2024-04-21T08:57:32.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:32 smithi077 ceph-mon[31615]: pgmap v412: 33 pgs: 33 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:57:32.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:32 smithi077 ceph-mon[31615]: from='client.? 172.21.15.7:0/4203023539' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:32.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:32 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/1156187434' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:32.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:32 smithi016 ceph-mon[31259]: osdmap e119: 12 total, 12 up, 12 in 2024-04-21T08:57:32.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:32 smithi016 ceph-mon[31259]: pgmap v412: 33 pgs: 33 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:57:32.804 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:32 smithi016 ceph-mon[31259]: from='client.? 172.21.15.7:0/4203023539' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:33.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a[28473]: 2024-04-21T08:57:33.132+0000 7fd7a5648700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:33.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a[28473]: 2024-04-21T08:57:33.132+0000 7fd7a5648700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-21T08:57:33.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:33 smithi007 podman[103284]: 2024-04-21 08:57:33.349441323 +0000 UTC m=+0.366678975 container died 1e1e16479767a4409873e9eeadf797279f7340849ee2935be063ebb7e9f567e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-21T08:57:34.358 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:34 smithi007 podman[103284]: 2024-04-21 08:57:34.056624797 +0000 UTC m=+1.073862437 container cleanup 1e1e16479767a4409873e9eeadf797279f7340849ee2935be063ebb7e9f567e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-21T08:57:34.358 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:34 smithi007 bash[103284]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a 2024-04-21T08:57:34.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.a.service' 2024-04-21T08:57:34.609 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:34 smithi007 podman[103295]: 2024-04-21 08:57:34.358898113 +0000 UTC m=+0.999886922 container remove 1e1e16479767a4409873e9eeadf797279f7340849ee2935be063ebb7e9f567e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True) 2024-04-21T08:57:34.609 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:34 smithi007 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-21T08:57:34.609 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:34 smithi007 systemd[1]: Stopped Ceph mon.a for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:57:34.609 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 21 08:57:34 smithi007 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.a.service: Consumed 22.543s CPU time. 2024-04-21T08:57:34.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:34.638 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-21T08:57:34.638 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-21T08:57:34.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.b 2024-04-21T08:57:35.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:34 smithi016 systemd[1]: Stopping Ceph mon.b for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:35.766 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:35 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-b[31255]: 2024-04-21T08:57:35.332+0000 7f259752d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:35.766 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:35 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-b[31255]: 2024-04-21T08:57:35.332+0000 7f259752d700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-21T08:57:35.766 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:35 smithi016 podman[66723]: 2024-04-21 08:57:35.465692284 +0000 UTC m=+0.256061065 container died b174a6eca6abb23896602b5db296c97e45ae9f31083876c0391fbccb47ebd97d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-b, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-21T08:57:36.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:35 smithi016 podman[66723]: 2024-04-21 08:57:35.891509154 +0000 UTC m=+0.681877925 container cleanup b174a6eca6abb23896602b5db296c97e45ae9f31083876c0391fbccb47ebd97d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:57:36.054 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:35 smithi016 bash[66723]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-b 2024-04-21T08:57:36.364 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.b.service' 2024-04-21T08:57:36.386 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:36 smithi016 podman[66738]: 2024-04-21 08:57:36.13501292 +0000 UTC m=+0.663557060 container remove b174a6eca6abb23896602b5db296c97e45ae9f31083876c0391fbccb47ebd97d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:57:36.386 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:36 smithi016 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-21T08:57:36.386 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:36 smithi016 systemd[1]: Stopped Ceph mon.b for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:57:36.387 INFO:journalctl@ceph.mon.b.smithi016.stdout:Apr 21 08:57:36 smithi016 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.b.service: Consumed 14.682s CPU time. 2024-04-21T08:57:36.490 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:36.490 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-21T08:57:36.490 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-21T08:57:36.490 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.c 2024-04-21T08:57:36.821 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:36 smithi077 systemd[1]: Stopping Ceph mon.c for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:37.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-c[31608]: 2024-04-21T08:57:36.820+0000 7f8bf0b82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:37.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-c[31608]: 2024-04-21T08:57:36.820+0000 7f8bf0b82700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-21T08:57:37.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:36 smithi077 podman[68868]: 2024-04-21 08:57:36.954026677 +0000 UTC m=+0.271329835 container died b616f89a2b7fca97f9a0ed63592588b789428ec56f0d5acbe040f4696509bbe2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-21T08:57:38.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:37 smithi077 podman[68868]: 2024-04-21 08:57:37.797358616 +0000 UTC m=+1.114661774 container cleanup b616f89a2b7fca97f9a0ed63592588b789428ec56f0d5acbe040f4696509bbe2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-21T08:57:38.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:Apr 21 08:57:37 smithi077 bash[68868]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mon-c 2024-04-21T08:57:38.278 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mon.c.service' 2024-04-21T08:57:38.318 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:38.319 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-21T08:57:38.319 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-21T08:57:38.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.x 2024-04-21T08:57:38.652 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:57:38 smithi007 systemd[1]: Stopping Ceph mgr.x for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:39.902 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:57:39 smithi007 podman[103401]: 2024-04-21 08:57:39.631056091 +0000 UTC m=+1.111186055 container died d8b0a57a71a30ac1cf5a83a9b4d7677cc81b235cb0544d029b9121988eb46591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:57:40.889 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:57:40 smithi007 podman[103401]: 2024-04-21 08:57:40.654930025 +0000 UTC m=+2.135059979 container cleanup d8b0a57a71a30ac1cf5a83a9b4d7677cc81b235cb0544d029b9121988eb46591 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-21T08:57:40.889 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Apr 21 08:57:40 smithi007 bash[103401]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-x 2024-04-21T08:57:41.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.x.service' 2024-04-21T08:57:41.139 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:41.139 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-21T08:57:41.139 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-21T08:57:41.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.y 2024-04-21T08:57:41.475 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:57:41 smithi016 systemd[1]: Stopping Ceph mgr.y for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:41.772 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:57:41 smithi016 podman[66845]: 2024-04-21 08:57:41.475929187 +0000 UTC m=+0.140007357 container died 615c421659eb15eb8198f9c591e9510cc524bdb6867eb197be6ba3e0f9290764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True) 2024-04-21T08:57:42.054 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:57:41 smithi016 podman[66845]: 2024-04-21 08:57:41.863471027 +0000 UTC m=+0.527549190 container cleanup 615c421659eb15eb8198f9c591e9510cc524bdb6867eb197be6ba3e0f9290764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:57:42.054 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 21 08:57:41 smithi016 bash[66845]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-y 2024-04-21T08:57:42.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.y.service' 2024-04-21T08:57:42.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:42.979 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-21T08:57:42.979 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-21T08:57:42.979 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.z 2024-04-21T08:57:43.322 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:57:43 smithi077 systemd[1]: Stopping Ceph mgr.z for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:43.604 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:57:43 smithi077 podman[68990]: 2024-04-21 08:57:43.322805392 +0000 UTC m=+0.144724535 container died 5e0b032c4a9e8bf3730acfca31fcc169094f8d6d1564368714cbfef9fe1d2bce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:57:44.102 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:57:43 smithi077 podman[68990]: 2024-04-21 08:57:43.65260936 +0000 UTC m=+0.474528503 container cleanup 5e0b032c4a9e8bf3730acfca31fcc169094f8d6d1564368714cbfef9fe1d2bce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-21T08:57:44.102 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:57:43 smithi077 bash[68990]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z 2024-04-21T08:57:44.102 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:57:43 smithi077 podman[69004]: 2024-04-21 08:57:43.871220544 +0000 UTC m=+0.541194979 container remove 5e0b032c4a9e8bf3730acfca31fcc169094f8d6d1564368714cbfef9fe1d2bce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-21T08:57:44.102 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Apr 21 08:57:43 smithi077 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-21T08:57:44.121 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mgr.z.service' 2024-04-21T08:57:44.154 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:44.155 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-21T08:57:44.155 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-21T08:57:44.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.0 2024-04-21T08:57:44.468 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:44 smithi007 systemd[1]: Stopping Ceph osd.0 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:44.902 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:44 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0[41045]: 2024-04-21T08:57:44.467+0000 7f81e4a74700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:44.902 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:44 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0[41045]: 2024-04-21T08:57:44.467+0000 7f81e4a74700 -1 osd.0 120 *** Got signal Terminated *** 2024-04-21T08:57:44.902 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:44 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0[41045]: 2024-04-21T08:57:44.467+0000 7f81e4a74700 -1 osd.0 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:57:50.260 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:49 smithi007 podman[103523]: 2024-04-21 08:57:49.95500472 +0000 UTC m=+5.608585366 container died 8bf7f93ee52e57f30c5f4e939943392b45d739f3c843a6f4ce84ca8ebe83a245 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:57:50.602 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:50 smithi007 podman[103523]: 2024-04-21 08:57:50.351665351 +0000 UTC m=+6.005245997 container cleanup 8bf7f93ee52e57f30c5f4e939943392b45d739f3c843a6f4ce84ca8ebe83a245 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-21T08:57:50.603 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:50 smithi007 bash[103523]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0 2024-04-21T08:57:50.902 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:50 smithi007 podman[103542]: 2024-04-21 08:57:50.603533518 +0000 UTC m=+0.649122458 container remove 8bf7f93ee52e57f30c5f4e939943392b45d739f3c843a6f4ce84ca8ebe83a245 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD) 2024-04-21T08:57:51.185 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:50 smithi007 podman[103641]: 2024-04-21 08:57:50.837961949 +0000 UTC m=+0.019304104 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:57:51.185 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:50 smithi007 podman[103641]: 2024-04-21 08:57:50.946271427 +0000 UTC m=+0.127613565 container create 8b3e0f86e784fc1e956e559a408ee4ef70df12a4059fcec6c7df38e5098ba426 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2) 2024-04-21T08:57:51.185 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:51 smithi007 podman[103641]: 2024-04-21 08:57:51.182522737 +0000 UTC m=+0.363864878 container init 8b3e0f86e784fc1e956e559a408ee4ef70df12a4059fcec6c7df38e5098ba426 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-21T08:57:51.484 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:51 smithi007 podman[103641]: 2024-04-21 08:57:51.185828859 +0000 UTC m=+0.367171005 container start 8b3e0f86e784fc1e956e559a408ee4ef70df12a4059fcec6c7df38e5098ba426 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:57:51.484 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:51 smithi007 podman[103641]: 2024-04-21 08:57:51.28286096 +0000 UTC m=+0.464203108 container attach 8b3e0f86e784fc1e956e559a408ee4ef70df12a4059fcec6c7df38e5098ba426 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-21T08:57:51.902 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 21 08:57:51 smithi007 podman[103641]: 2024-04-21 08:57:51.512487392 +0000 UTC m=+0.693829540 container died 8b3e0f86e784fc1e956e559a408ee4ef70df12a4059fcec6c7df38e5098ba426 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-21T08:57:52.119 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.0.service' 2024-04-21T08:57:52.162 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:52.163 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-21T08:57:52.163 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-21T08:57:52.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.1 2024-04-21T08:57:52.494 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:52 smithi016 systemd[1]: Stopping Ceph osd.1 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:52.804 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:52 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1[35871]: 2024-04-21T08:57:52.494+0000 7fdaf1ae7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:52.804 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:52 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1[35871]: 2024-04-21T08:57:52.494+0000 7fdaf1ae7700 -1 osd.1 120 *** Got signal Terminated *** 2024-04-21T08:57:52.805 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:52 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1[35871]: 2024-04-21T08:57:52.494+0000 7fdaf1ae7700 -1 osd.1 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:57:58.191 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:57 smithi016 podman[66970]: 2024-04-21 08:57:57.91604947 +0000 UTC m=+5.552880248 container died 23610ec72877a5ede66dbb94687222f3f5b1951ae3c8c7c62192f3fbc779505c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:57:58.554 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:58 smithi016 podman[66970]: 2024-04-21 08:57:58.285885613 +0000 UTC m=+5.922716383 container cleanup 23610ec72877a5ede66dbb94687222f3f5b1951ae3c8c7c62192f3fbc779505c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-21T08:57:58.554 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:58 smithi016 bash[66970]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1 2024-04-21T08:57:58.923 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:58 smithi016 podman[66988]: 2024-04-21 08:57:58.579105497 +0000 UTC m=+0.656595403 container remove 23610ec72877a5ede66dbb94687222f3f5b1951ae3c8c7c62192f3fbc779505c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-21T08:57:59.215 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:58 smithi016 podman[67079]: 2024-04-21 08:57:58.827740364 +0000 UTC m=+0.018807962 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:57:59.216 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:58 smithi016 podman[67079]: 2024-04-21 08:57:58.955259757 +0000 UTC m=+0.146327329 container create 142711e46ce624df2dfc4c101bf12e3ecb86b925163f0e245acfedb16041ff5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-21T08:57:59.216 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:59 smithi016 podman[67079]: 2024-04-21 08:57:59.216440822 +0000 UTC m=+0.407508393 container init 142711e46ce624df2dfc4c101bf12e3ecb86b925163f0e245acfedb16041ff5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-21T08:57:59.554 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:59 smithi016 podman[67079]: 2024-04-21 08:57:59.219702375 +0000 UTC m=+0.410769947 container start 142711e46ce624df2dfc4c101bf12e3ecb86b925163f0e245acfedb16041ff5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-21T08:57:59.554 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:59 smithi016 podman[67079]: 2024-04-21 08:57:59.349820356 +0000 UTC m=+0.540887936 container attach 142711e46ce624df2dfc4c101bf12e3ecb86b925163f0e245acfedb16041ff5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=) 2024-04-21T08:58:00.054 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 21 08:57:59 smithi016 podman[67079]: 2024-04-21 08:57:59.770174211 +0000 UTC m=+0.961241783 container died 142711e46ce624df2dfc4c101bf12e3ecb86b925163f0e245acfedb16041ff5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-21T08:58:00.355 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.1.service' 2024-04-21T08:58:00.395 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:00.396 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-21T08:58:00.396 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-21T08:58:00.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.2 2024-04-21T08:58:00.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:00 smithi077 systemd[1]: Stopping Ceph osd.2 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:01.104 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:00 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2[36765]: 2024-04-21T08:58:00.706+0000 7f25aabb5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:01.104 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:00 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2[36765]: 2024-04-21T08:58:00.706+0000 7f25aabb5700 -1 osd.2 120 *** Got signal Terminated *** 2024-04-21T08:58:01.104 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:00 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2[36765]: 2024-04-21T08:58:00.706+0000 7f25aabb5700 -1 osd.2 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:06.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:06 smithi077 podman[69112]: 2024-04-21 08:58:06.130838784 +0000 UTC m=+5.544112152 container died 60ea20e31cd85d1c53f79fa38aa0d2af770d581ba221a5feabfedbfc27effeba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:06.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:06 smithi077 podman[69112]: 2024-04-21 08:58:06.541216844 +0000 UTC m=+5.954490285 container cleanup 60ea20e31cd85d1c53f79fa38aa0d2af770d581ba221a5feabfedbfc27effeba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-21T08:58:06.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:06 smithi077 bash[69112]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2 2024-04-21T08:58:07.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:06 smithi077 podman[69132]: 2024-04-21 08:58:06.78419065 +0000 UTC m=+0.655460384 container remove 60ea20e31cd85d1c53f79fa38aa0d2af770d581ba221a5feabfedbfc27effeba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-21T08:58:07.354 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:07 smithi077 podman[69223]: 2024-04-21 08:58:06.999605044 +0000 UTC m=+0.017840549 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:07.354 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:07 smithi077 podman[69223]: 2024-04-21 08:58:07.11062532 +0000 UTC m=+0.128860817 container create 98b1789ab24bc3ceb1dac62a59360afdc1006392193b3153495ad61d4ac58594 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T08:58:07.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:07 smithi077 podman[69223]: 2024-04-21 08:58:07.379862805 +0000 UTC m=+0.398098304 container init 98b1789ab24bc3ceb1dac62a59360afdc1006392193b3153495ad61d4ac58594 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:07.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:07 smithi077 podman[69223]: 2024-04-21 08:58:07.383344051 +0000 UTC m=+0.401579546 container start 98b1789ab24bc3ceb1dac62a59360afdc1006392193b3153495ad61d4ac58594 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:58:07.674 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:07 smithi077 podman[69223]: 2024-04-21 08:58:07.463377161 +0000 UTC m=+0.481612657 container attach 98b1789ab24bc3ceb1dac62a59360afdc1006392193b3153495ad61d4ac58594 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:58:08.104 INFO:journalctl@ceph.osd.2.smithi077.stdout:Apr 21 08:58:07 smithi077 podman[69223]: 2024-04-21 08:58:07.674364087 +0000 UTC m=+0.692599587 container died 98b1789ab24bc3ceb1dac62a59360afdc1006392193b3153495ad61d4ac58594 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:08.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.2.service' 2024-04-21T08:58:08.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:08.518 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-21T08:58:08.518 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-21T08:58:08.518 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.3 2024-04-21T08:58:08.838 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:08 smithi007 systemd[1]: Stopping Ceph osd.3 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:09.152 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:08 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3[47619]: 2024-04-21T08:58:08.838+0000 7f18ae384700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:09.152 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:08 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3[47619]: 2024-04-21T08:58:08.838+0000 7f18ae384700 -1 osd.3 120 *** Got signal Terminated *** 2024-04-21T08:58:09.152 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:08 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3[47619]: 2024-04-21T08:58:08.838+0000 7f18ae384700 -1 osd.3 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:11.315 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:11 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:11.021+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:11.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:11 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:11.287+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:11.652 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:11 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3[47619]: 2024-04-21T08:58:11.314+0000 7f18aa978700 -1 osd.3 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:45.474668+0000 front 2024-04-21T08:57:45.474636+0000 (oldest deadline 2024-04-21T08:58:11.275419+0000) 2024-04-21T08:58:12.104 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:11 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:11.694+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:12.342 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:12 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:12.036+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:12.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:12 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:12.252+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:12.652 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:12 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3[47619]: 2024-04-21T08:58:12.342+0000 7f18aa978700 -1 osd.3 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:45.474668+0000 front 2024-04-21T08:57:45.474636+0000 (oldest deadline 2024-04-21T08:58:11.275419+0000) 2024-04-21T08:58:13.104 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:12 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:12.741+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:13.328 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:12 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:12.989+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:13.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:13 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:13.224+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:13.652 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:13 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3[47619]: 2024-04-21T08:58:13.328+0000 7f18aa978700 -1 osd.3 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:45.474668+0000 front 2024-04-21T08:57:45.474636+0000 (oldest deadline 2024-04-21T08:58:11.275419+0000) 2024-04-21T08:58:14.104 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:13 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:13.742+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:14.296 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:14 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:14.002+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:14.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:14 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:14.218+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:14.566 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:14 smithi007 podman[103749]: 2024-04-21 08:58:14.297122438 +0000 UTC m=+5.571422394 container died 72d3379f44a0de86a7210d8e0f70477a4062e9a9bcdef1d63e92b47b992b3543 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True) 2024-04-21T08:58:14.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:14.151+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:14.902 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:14 smithi007 podman[103749]: 2024-04-21 08:58:14.65720921 +0000 UTC m=+5.931509174 container cleanup 72d3379f44a0de86a7210d8e0f70477a4062e9a9bcdef1d63e92b47b992b3543 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, ceph=True) 2024-04-21T08:58:14.902 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:14 smithi007 bash[103749]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3 2024-04-21T08:58:15.104 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:14.779+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:15.325 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:14 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:14.956+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:15.325 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:14 smithi007 podman[103768]: 2024-04-21 08:58:14.975083792 +0000 UTC m=+0.669717271 container remove 72d3379f44a0de86a7210d8e0f70477a4062e9a9bcdef1d63e92b47b992b3543 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-21T08:58:15.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:15 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:15.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:15.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:15 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:15.141+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:15.652 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:15 smithi007 podman[103853]: 2024-04-21 08:58:15.228936949 +0000 UTC m=+0.018278242 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:15.652 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:15 smithi007 podman[103853]: 2024-04-21 08:58:15.342699829 +0000 UTC m=+0.132041136 container create 8fad583cc0ab8898a2e40f74b63dcb079292eda5331209e857b3f2aed9c6514d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-21T08:58:15.911 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:15 smithi007 podman[103853]: 2024-04-21 08:58:15.689538158 +0000 UTC m=+0.478879426 container init 8fad583cc0ab8898a2e40f74b63dcb079292eda5331209e857b3f2aed9c6514d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-21T08:58:15.911 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:15 smithi007 podman[103853]: 2024-04-21 08:58:15.693760081 +0000 UTC m=+0.483101347 container start 8fad583cc0ab8898a2e40f74b63dcb079292eda5331209e857b3f2aed9c6514d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-21T08:58:15.911 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:15 smithi007 podman[103853]: 2024-04-21 08:58:15.795372042 +0000 UTC m=+0.584713317 container attach 8fad583cc0ab8898a2e40f74b63dcb079292eda5331209e857b3f2aed9c6514d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-21T08:58:16.104 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:15 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:15.736+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:16.276 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:15 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:15.910+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:16.277 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 21 08:58:15 smithi007 podman[103853]: 2024-04-21 08:58:15.994964457 +0000 UTC m=+0.784305741 container died 8fad583cc0ab8898a2e40f74b63dcb079292eda5331209e857b3f2aed9c6514d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:58:16.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:16 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:16.145+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:16.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:16 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:16.121+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:16.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.3.service' 2024-04-21T08:58:16.948 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:16.949 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-21T08:58:16.949 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-21T08:58:16.949 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.4 2024-04-21T08:58:17.104 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:16 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:16.723+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:17.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:16 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:16.895+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:17.274 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:17 smithi016 systemd[1]: Stopping Ceph osd.4 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:17.274 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:17 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:17.121+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:17.554 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:17 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4[42366]: 2024-04-21T08:58:17.272+0000 7f0f9bfc1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:17.555 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:17 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4[42366]: 2024-04-21T08:58:17.272+0000 7f0f9bfc1700 -1 osd.4 120 *** Got signal Terminated *** 2024-04-21T08:58:17.555 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:17 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4[42366]: 2024-04-21T08:58:17.272+0000 7f0f9bfc1700 -1 osd.4 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:17.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:17 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:17.113+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:18.087 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:17 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:17.695+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:18.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:18 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:18.086+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:18.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:17 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:17.905+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:18.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:18 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:18.199+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:18.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:18 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:18.167+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:19.104 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:18 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:18.656+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:19.303 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:18 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:18.934+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:19.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:19 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:19.164+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:19.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:19 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:19.171+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:19.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:19.130+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:19.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:19 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:19.451+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:19.971 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:19.643+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:19.971 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:19.643+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:20.054 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:19 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4[42366]: 2024-04-21T08:58:19.798+0000 7f0f945ad700 -1 osd.4 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.453228+0000 front 2024-04-21T08:57:54.453200+0000 (oldest deadline 2024-04-21T08:58:19.153457+0000) 2024-04-21T08:58:20.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:20 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:20.138+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:20.355 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:19.971+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:20.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:19 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:19.925+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:20.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:19 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:19.925+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:20.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:20 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:20.126+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:20.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:20 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:20.181+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:20.902 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:20 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:20.479+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:20.994 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:20 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:20.630+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:20.994 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:20 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:20.630+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:21.054 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:20 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4[42366]: 2024-04-21T08:58:20.762+0000 7f0f945ad700 -1 osd.4 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.453228+0000 front 2024-04-21T08:57:54.453200+0000 (oldest deadline 2024-04-21T08:58:19.153457+0000) 2024-04-21T08:58:21.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:20 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:20.994+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:21.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:21 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:21.188+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:21.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:20 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:20.939+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:21.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:20 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:20.939+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:21.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:21 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:21.218+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:21.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:21 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:21.132+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:21.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:21 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:21.132+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:21.854 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:21 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:21.581+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:21.854 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:21 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:21.581+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:21.902 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:21 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:21.460+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:22.172 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:21 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4[42366]: 2024-04-21T08:58:21.810+0000 7f0f945ad700 -1 osd.4 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.453228+0000 front 2024-04-21T08:57:54.453200+0000 (oldest deadline 2024-04-21T08:58:19.153457+0000) 2024-04-21T08:58:22.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:21 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:21.952+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:22.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:22 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:22.200+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:22.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:21 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:21.938+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:22.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:21 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:21.938+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:22.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:22 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:22.201+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:22.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:22 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:22.173+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:22.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:22 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:22.173+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:22.902 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:22 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:22.451+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:22.961 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:22 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:22.630+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:22.961 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:22 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:22.630+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:23.133 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:22 smithi016 podman[67198]: 2024-04-21 08:58:22.820871114 +0000 UTC m=+5.668575031 container died a0452c7b5e5bbf57594bc660281cbb0aaa2e004ace277be8af8739ba90c09241 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-21T08:58:23.222 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:22 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:22.961+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:23.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:22 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:22.967+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:23.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:22 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:22.967+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:23.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:23 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:23.163+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:23.555 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:23 smithi016 podman[67198]: 2024-04-21 08:58:23.224018508 +0000 UTC m=+6.071722424 container cleanup a0452c7b5e5bbf57594bc660281cbb0aaa2e004ace277be8af8739ba90c09241 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:58:23.555 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:23 smithi016 bash[67198]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4 2024-04-21T08:58:23.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:23 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:23.141+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:23.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:23 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:23.141+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:23.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:23 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:23.222+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:23.902 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:23 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:23.464+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:23.911 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:23 smithi016 podman[67215]: 2024-04-21 08:58:23.609032882 +0000 UTC m=+0.781845933 container remove a0452c7b5e5bbf57594bc660281cbb0aaa2e004ace277be8af8739ba90c09241 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-21T08:58:23.912 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:23 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:23.613+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:23.912 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:23 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:23.613+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:24.173 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:23 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:23.912+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:24.194 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:24 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:24.141+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:24.195 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:24 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:24.102+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:24.195 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:24 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:24.102+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:24.195 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:23 smithi016 podman[67305]: 2024-04-21 08:58:23.815528369 +0000 UTC m=+0.018171276 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:24.195 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:23 smithi016 podman[67305]: 2024-04-21 08:58:23.926218836 +0000 UTC m=+0.128861754 container create 894fd8dcc3a0226392a4e3fd1fb7600d39cc56e9dd1437631252acd50310301f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-04-21T08:58:24.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:23 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:23.920+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:24.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:23 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:23.920+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:24.504 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:24 smithi016 podman[67305]: 2024-04-21 08:58:24.195716108 +0000 UTC m=+0.398359027 container init 894fd8dcc3a0226392a4e3fd1fb7600d39cc56e9dd1437631252acd50310301f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:24.504 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:24 smithi016 podman[67305]: 2024-04-21 08:58:24.199051337 +0000 UTC m=+0.401694259 container start 894fd8dcc3a0226392a4e3fd1fb7600d39cc56e9dd1437631252acd50310301f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-21T08:58:24.504 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:24 smithi016 podman[67305]: 2024-04-21 08:58:24.271322085 +0000 UTC m=+0.473965016 container attach 894fd8dcc3a0226392a4e3fd1fb7600d39cc56e9dd1437631252acd50310301f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T08:58:24.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:24 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:24.173+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:24.804 INFO:journalctl@ceph.osd.4.smithi016.stdout:Apr 21 08:58:24 smithi016 podman[67305]: 2024-04-21 08:58:24.505523366 +0000 UTC m=+0.708166287 container died 894fd8dcc3a0226392a4e3fd1fb7600d39cc56e9dd1437631252acd50310301f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:24.902 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:24 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:24.453+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:24.958 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:24 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:24.606+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:24.958 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:24 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:24.606+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:25.300 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:25 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:25.093+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:25.301 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:25 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:25.131+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:25.301 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:25 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:25.131+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:25.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.4.service' 2024-04-21T08:58:25.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:25 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:25.149+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:25.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:24 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:24.958+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:25.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:25.363 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-21T08:58:25.363 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-21T08:58:25.363 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.5 2024-04-21T08:58:25.401 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:24 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:24.965+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:25.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:24 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:24.965+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:25.652 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:25 smithi077 systemd[1]: Stopping Ceph osd.5 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:25.902 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:25 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:25.452+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:25.904 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:25 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:25.652+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:25.904 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:25 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:25.652+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:26.304 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:26 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:26.046+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:26.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:26 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:26.169+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:26.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:26 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:26.169+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:26.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:26 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:26.138+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:26.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:25 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:25.941+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:26.355 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:25 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:25.904+0000 7ff32bdf7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:26.355 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:25 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:25.904+0000 7ff32bdf7700 -1 osd.5 120 *** Got signal Terminated *** 2024-04-21T08:58:26.355 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:25 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:25.904+0000 7ff32bdf7700 -1 osd.5 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:26.401 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:26 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:26.003+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:26.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:26 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:26.003+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:26.733 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:26 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:26.403+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:26.974 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:26 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:26.647+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:26.974 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:26 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:26.647+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:27.304 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:27 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:27.154+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:27.304 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:27 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:27.154+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:27.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:27 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:27.001+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:27.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:27 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:27.001+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:27.328 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:26 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:26.992+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:27.328 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:26 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:26.992+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:27.328 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:26 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:26.992+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:27.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:26 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:26.974+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:27.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:27 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:27.169+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:27.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:27 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:27.409+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:27.995 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:27 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:27.666+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:27.996 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:27 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:27.666+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:28.304 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:28 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:28.025+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:28.304 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:28 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:28.025+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:28.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:28 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:28.194+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:28.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:28 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:28.194+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:28.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:27 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:27.995+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:28.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:28 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:28.212+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:28.372 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:27 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:27.960+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:28.372 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:27 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:27.960+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:28.372 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:27 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:27.960+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:28.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:28 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:28.372+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:28.960 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:28 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:28.624+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:28.960 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:28 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:28.624+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:29.304 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:29 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:29.153+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:29.304 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:29 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:29.153+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:29.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:29 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:29.041+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:29.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:29 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:29.041+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:29.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:28 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:28.960+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:29.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:29 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:29.190+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:29.365 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:28 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:28.958+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:29.365 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:28 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:28.958+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:29.365 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:28 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:28.958+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:29.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:29 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:29.365+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:29.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:29 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:29.365+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.518537+0000 front 2024-04-21T08:58:05.518520+0000 (oldest deadline 2024-04-21T08:58:28.418531+0000) 2024-04-21T08:58:29.981 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:29 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:29.656+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:29.981 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:29 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:29.656+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:29.982 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:29 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:29.656+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:04.360310+0000 front 2024-04-21T08:58:04.360276+0000 (oldest deadline 2024-04-21T08:58:29.560843+0000) 2024-04-21T08:58:30.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:29 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:29.981+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:30.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:30 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:30.190+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:30.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:30 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:30.190+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:30.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:29 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:29.952+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:30.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:29 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:29.952+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:30.402 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:29 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:29.952+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:30.464 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:30 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:30.076+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:30.464 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:30 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:30.076+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:30.464 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:30 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:30.109+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:30.464 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:30 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:30.109+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:30.465 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:30 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:30.109+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:30.902 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:30 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:30.412+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:30.902 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:30 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:30.412+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.518537+0000 front 2024-04-21T08:58:05.518520+0000 (oldest deadline 2024-04-21T08:58:28.418531+0000) 2024-04-21T08:58:30.968 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:30 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:30.619+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.758411+0000 front 2024-04-21T08:57:46.758470+0000 (oldest deadline 2024-04-21T08:58:10.858357+0000) 2024-04-21T08:58:30.968 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:30 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:30.619+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:53.759399+0000 front 2024-04-21T08:57:53.759430+0000 (oldest deadline 2024-04-21T08:58:19.059301+0000) 2024-04-21T08:58:30.968 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:30 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5[43301]: 2024-04-21T08:58:30.619+0000 7ff3283eb700 -1 osd.5 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:04.360310+0000 front 2024-04-21T08:58:04.360276+0000 (oldest deadline 2024-04-21T08:58:29.560843+0000) 2024-04-21T08:58:31.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:30 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:30.968+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:31.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:30 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:30.968+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:31.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:31 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:31.195+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:31.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:31 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:31.195+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:31.372 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:30 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:30.937+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:31.373 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:30 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:30.937+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:31.373 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:30 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:30.937+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:31.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:31 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:31.111+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:31.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:31 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:31.111+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:31.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:31 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:31.111+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:31.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:31 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:31.089+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:31.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:31 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:31.089+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:31.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:31 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:31.372+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:31.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:31 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:31.372+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.518537+0000 front 2024-04-21T08:58:05.518520+0000 (oldest deadline 2024-04-21T08:58:28.418531+0000) 2024-04-21T08:58:31.697 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:31 smithi077 podman[69341]: 2024-04-21 08:58:31.390472816 +0000 UTC m=+5.825059240 container died b3032ec765f888c822b30cb639f32683c221caeae929619f1222f34e82f2c3eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, ceph=True) 2024-04-21T08:58:31.998 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:31 smithi077 podman[69341]: 2024-04-21 08:58:31.796778413 +0000 UTC m=+6.231364837 container cleanup b3032ec765f888c822b30cb639f32683c221caeae929619f1222f34e82f2c3eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:58:31.998 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:31 smithi077 bash[69341]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5 2024-04-21T08:58:32.321 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:32 smithi077 podman[69358]: 2024-04-21 08:58:32.031448919 +0000 UTC m=+0.643901389 container remove b3032ec765f888c822b30cb639f32683c221caeae929619f1222f34e82f2c3eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-21T08:58:32.322 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:31 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:31.998+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:32.322 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:31 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:31.998+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:32.322 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:32 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:32.174+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:32.322 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:32 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:32.174+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:32.339 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:31 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:31.944+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:32.339 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:31 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:31.944+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:32.339 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:31 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:31.944+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:32.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:32 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:32.079+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:32.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:32 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:32.079+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:32.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:32 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:32.142+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:32.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:32 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:32.142+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:32.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:32 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:32.142+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:32.604 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:32 smithi077 podman[69446]: 2024-04-21 08:58:32.225389325 +0000 UTC m=+0.018085529 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:32.604 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:32 smithi077 podman[69446]: 2024-04-21 08:58:32.366057626 +0000 UTC m=+0.158753821 container create d35d56a3bd55b4f3ef19adadb50739b0f2272e9655c4c3cfe343d1f12be50175 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:58:32.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:32 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:32.339+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:32.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:32 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:32.339+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.518537+0000 front 2024-04-21T08:58:05.518520+0000 (oldest deadline 2024-04-21T08:58:28.418531+0000) 2024-04-21T08:58:32.934 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:32 smithi077 podman[69446]: 2024-04-21 08:58:32.63512283 +0000 UTC m=+0.427819026 container init d35d56a3bd55b4f3ef19adadb50739b0f2272e9655c4c3cfe343d1f12be50175 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-21T08:58:32.935 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:32 smithi077 podman[69446]: 2024-04-21 08:58:32.638286899 +0000 UTC m=+0.430983097 container start d35d56a3bd55b4f3ef19adadb50739b0f2272e9655c4c3cfe343d1f12be50175 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:58:32.935 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:32 smithi077 podman[69446]: 2024-04-21 08:58:32.735942932 +0000 UTC m=+0.528639142 container attach d35d56a3bd55b4f3ef19adadb50739b0f2272e9655c4c3cfe343d1f12be50175 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-21T08:58:33.217 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 21 08:58:32 smithi077 podman[69446]: 2024-04-21 08:58:32.935963474 +0000 UTC m=+0.728659680 container died d35d56a3bd55b4f3ef19adadb50739b0f2272e9655c4c3cfe343d1f12be50175 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-21T08:58:33.218 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:32 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:32.961+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:33.218 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:32 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:32.961+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:33.379 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:32 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:32.918+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:33.380 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:32 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:32.918+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:33.380 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:32 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:32.918+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:33.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:33 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:33.110+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:33.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:33 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:33.110+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:33.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:33 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:33.174+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:33.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:33 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:33.174+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:33.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:33 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:33.174+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:33.589 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:33 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:33.217+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:33.589 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:33 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:33.217+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:33.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.5.service' 2024-04-21T08:58:33.648 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:33.649 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-21T08:58:33.649 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-21T08:58:33.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.6 2024-04-21T08:58:33.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:33.379+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:33.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:33.379+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.518537+0000 front 2024-04-21T08:58:05.518520+0000 (oldest deadline 2024-04-21T08:58:28.418531+0000) 2024-04-21T08:58:33.951 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:33 smithi007 systemd[1]: Stopping Ceph osd.6 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:33.951 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:33.899+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:33.951 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:33.899+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:33.951 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:33.899+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:34.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:33 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:33.965+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:34.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:33 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:33.965+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:34.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:34 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:34.168+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:34.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:34 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:34.168+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:34.402 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:33.951+0000 7fa34a5ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:34.402 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:33.951+0000 7fa34a5ea700 -1 osd.6 120 *** Got signal Terminated *** 2024-04-21T08:58:34.402 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:33 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:33.951+0000 7fa34a5ea700 -1 osd.6 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:34.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:34 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:34.079+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:34.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:34 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:34.079+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:34.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:34 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:34.138+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:34.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:34 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:34.138+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:34.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:34 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:34.138+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:34.855 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:34.416+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:34.855 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:34.416+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.518537+0000 front 2024-04-21T08:58:05.518520+0000 (oldest deadline 2024-04-21T08:58:28.418531+0000) 2024-04-21T08:58:35.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:34.855+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:35.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:34.855+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:35.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:34.855+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:35.304 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:35 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:35.097+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:35.304 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:35 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:35.097+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:35.304 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:35 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:35.097+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:35.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:35 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:35.042+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:35.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:35 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:35.042+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:35.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:35 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:35.042+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:35.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:34 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:34.989+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:35.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:34 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:34.990+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:35.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:35 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:35.122+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:35.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:35 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:35.122+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:35.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:35 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:35.122+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:35.810 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:35 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:35.413+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:35.810 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:35 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:35.413+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.518537+0000 front 2024-04-21T08:58:05.518520+0000 (oldest deadline 2024-04-21T08:58:28.418531+0000) 2024-04-21T08:58:35.810 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:35 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:35.413+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:09.519587+0000 front 2024-04-21T08:58:09.519822+0000 (oldest deadline 2024-04-21T08:58:34.819777+0000) 2024-04-21T08:58:36.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:35 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:35.810+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:36.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:35 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:35.810+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:36.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:35 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:35.810+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:36.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:35 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:35.810+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:36.255 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:35 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:35.986+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:36.256 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:35 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:35.986+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:36.256 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:35 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:35.986+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:36.256 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:36.077+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:36.256 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:36.077+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:36.256 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:36.077+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:36.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:36 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:36.111+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:36.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:36 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:36.111+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:36.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:36 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:36.111+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:36.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:36 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:36.064+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:36.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:36 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:36.064+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:36.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:36 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:36.064+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:36.814 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:36.449+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:36.814 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:36.449+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.518537+0000 front 2024-04-21T08:58:05.518520+0000 (oldest deadline 2024-04-21T08:58:28.418531+0000) 2024-04-21T08:58:36.814 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:36.449+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:09.519587+0000 front 2024-04-21T08:58:09.519822+0000 (oldest deadline 2024-04-21T08:58:34.819777+0000) 2024-04-21T08:58:37.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:36.814+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:37.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:36.814+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:37.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:36.814+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:37.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:36 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:36.814+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:37.272 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:37 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:37.124+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:37.272 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:37 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:37.124+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:37.272 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:37 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:37.124+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:37.273 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:36.975+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:37.273 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:36.975+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:37.273 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:36 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:36.975+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:37.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:37 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:37.093+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:37.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:37 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:37.093+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:37.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:37 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:37.093+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:37.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:37 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:37.074+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:37.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:37 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:37.074+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:37.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:37 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:37.074+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:37.778 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:37 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:37.418+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:37.778 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:37 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:37.418+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.518537+0000 front 2024-04-21T08:58:05.518520+0000 (oldest deadline 2024-04-21T08:58:28.418531+0000) 2024-04-21T08:58:37.779 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:37 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:37.418+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:09.519587+0000 front 2024-04-21T08:58:09.519822+0000 (oldest deadline 2024-04-21T08:58:34.819777+0000) 2024-04-21T08:58:38.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:37 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:37.778+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:38.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:37 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:37.778+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:38.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:37 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:37.778+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:38.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:37 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:37.778+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:38.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:38 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:38.098+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:38.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:38 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:38.098+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:38.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:38 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:38.098+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:38.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:37 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:37.985+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:38.355 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:37 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:37.985+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:38.355 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:37 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:37.985+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:38.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:38.078+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:38.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:38.078+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:38.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:38.078+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:38.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:38.102+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:38.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:38.102+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:38.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:38.102+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:38.739 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:38.463+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.116294+0000 front 2024-04-21T08:57:55.116081+0000 (oldest deadline 2024-04-21T08:58:18.616154+0000) 2024-04-21T08:58:38.739 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:38.463+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.518537+0000 front 2024-04-21T08:58:05.518520+0000 (oldest deadline 2024-04-21T08:58:28.418531+0000) 2024-04-21T08:58:38.739 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6[54581]: 2024-04-21T08:58:38.463+0000 7fa342bd6700 -1 osd.6 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:09.519587+0000 front 2024-04-21T08:58:09.519822+0000 (oldest deadline 2024-04-21T08:58:34.819777+0000) 2024-04-21T08:58:39.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:38.739+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:39.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:38.739+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:39.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:38.739+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:39.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:38 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:38.739+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:39.304 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:39.041+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:39.304 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:39.041+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:39.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:39.041+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:39.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:39.086+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:39.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:39.086+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:39.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:39 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:39.086+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:39.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:38 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:38.993+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:39.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:38 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:38.993+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:39.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:38 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:38.993+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:39.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:39 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:39.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:39.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:39 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:39.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:39.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:39 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:39.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:39.652 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:39 smithi007 podman[103971]: 2024-04-21 08:58:39.398647692 +0000 UTC m=+5.549151467 container died a44a62def04a9eb132cc8fadf5fa5033a6328e8c69b45b8c57acfb82240b15b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-21T08:58:40.085 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:39 smithi007 podman[103971]: 2024-04-21 08:58:39.909504832 +0000 UTC m=+6.060008607 container cleanup a44a62def04a9eb132cc8fadf5fa5033a6328e8c69b45b8c57acfb82240b15b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T08:58:40.085 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:39 smithi007 bash[103971]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6 2024-04-21T08:58:40.085 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:39.734+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:40.086 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:39.734+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:40.086 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:39.734+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:40.086 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:39 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:39.734+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:40.304 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:40 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:40.118+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:40.304 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:40 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:40.118+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:40.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:40 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:40.118+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:40.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:40 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:40.069+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:40.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:40 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:40.069+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:40.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:40 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:40.069+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:40.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:40 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:40.069+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:40.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:40 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:40.035+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:40.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:40 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:40.035+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:40.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:40 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:40.035+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:40.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:40 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:40.127+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:40.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:40 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:40.127+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:40.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:40 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:40.127+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:40.401 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:40 smithi007 podman[103990]: 2024-04-21 08:58:40.085780497 +0000 UTC m=+0.687109923 container remove a44a62def04a9eb132cc8fadf5fa5033a6328e8c69b45b8c57acfb82240b15b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-21T08:58:40.837 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:40 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:40.717+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:40.838 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:40 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:40.717+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:40.838 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:40 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:40.717+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:40.838 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:40 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:40.717+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:40.838 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:40 smithi007 podman[104084]: 2024-04-21 08:58:40.528146748 +0000 UTC m=+0.111113532 container create d118fc6649d50193f94cc95c50dbafa0e20aa441e5d07ff58577959f0e5f1d4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-21T08:58:40.839 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:40 smithi007 podman[104084]: 2024-04-21 08:58:40.435158045 +0000 UTC m=+0.018124857 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:40.839 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:40 smithi007 podman[104084]: 2024-04-21 08:58:40.738887159 +0000 UTC m=+0.321853954 container init d118fc6649d50193f94cc95c50dbafa0e20aa441e5d07ff58577959f0e5f1d4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-21T08:58:40.839 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:40 smithi007 podman[104084]: 2024-04-21 08:58:40.74208796 +0000 UTC m=+0.325054748 container start d118fc6649d50193f94cc95c50dbafa0e20aa441e5d07ff58577959f0e5f1d4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-21T08:58:40.839 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:40 smithi007 podman[104084]: 2024-04-21 08:58:40.838785674 +0000 UTC m=+0.421752459 container attach d118fc6649d50193f94cc95c50dbafa0e20aa441e5d07ff58577959f0e5f1d4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416) 2024-04-21T08:58:41.152 INFO:journalctl@ceph.osd.6.smithi007.stdout:Apr 21 08:58:41 smithi007 podman[104084]: 2024-04-21 08:58:41.042211682 +0000 UTC m=+0.625178485 container died d118fc6649d50193f94cc95c50dbafa0e20aa441e5d07ff58577959f0e5f1d4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:58:41.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:41 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:41.019+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:41.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:41 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:41.019+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:41.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:41 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:41.019+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:41.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:41 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:41.089+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:41.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:41 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:41.089+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:41.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:41 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:41.089+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:41.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:41 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:41.148+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:41.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:41 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:41.148+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:41.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:41 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:41.148+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:41.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:41 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:41.068+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:41.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:41 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:41.068+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:41.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:41 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:41.068+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:41.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:41 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:41.068+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:41.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.6.service' 2024-04-21T08:58:41.678 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:41.678 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-21T08:58:41.678 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-21T08:58:41.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.7 2024-04-21T08:58:41.902 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:41.747+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:41.902 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:41.747+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:41.902 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:41.747+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:41.902 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:41 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:41.747+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:42.029 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:41 smithi016 systemd[1]: Stopping Ceph osd.7 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:42.304 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:42 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:42.051+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:42.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:42 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:42.051+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:42.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:42 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:42.051+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:42.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:42 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:42.051+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:42.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:42 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:42.029+0000 7f0c2578a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:42.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:42 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:42.029+0000 7f0c2578a700 -1 osd.7 120 *** Got signal Terminated *** 2024-04-21T08:58:42.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:42 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:42.029+0000 7f0c2578a700 -1 osd.7 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:42.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:42 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:42.127+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:42.306 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:42 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:42.127+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:42.306 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:42 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:42.127+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:42.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:42 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:42.059+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:42.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:42 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:42.059+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:42.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:42 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:42.059+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:42.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:42 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:42.137+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:42.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:42 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:42.137+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:42.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:42 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:42.137+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:43.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:42 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:42.749+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:43.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:42 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:42.749+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:43.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:42 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:42.749+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:43.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:42 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:42.749+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:43.304 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:43 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:43.122+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:43.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:43 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:43.122+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:43.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:43 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:43.122+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:43.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:43 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:43.050+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:43.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:43 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:43.050+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:43.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:43 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:43.050+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:43.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:43 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:43.050+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:43.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:43 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:43.026+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:43.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:43 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:43.026+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:43.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:43 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:43.026+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:43.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:43 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:43.026+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:43.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:43 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:43.162+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:43.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:43 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:43.162+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:43.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:43 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:43.162+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:44.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:43 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:43.796+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:44.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:43 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:43.796+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:44.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:43 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:43.796+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:44.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:43 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:43.796+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:44.304 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:44 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:44.164+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:44.304 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:44 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:44.164+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:44.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:44 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:44.164+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:44.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:44 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:44.164+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.895013+0000 front 2024-04-21T08:58:19.894995+0000 (oldest deadline 2024-04-21T08:58:43.995182+0000) 2024-04-21T08:58:44.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:44 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:44.040+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:44.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:44 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:44.040+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:44.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:44 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:44.040+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:44.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:44 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:44.040+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:44.306 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:44 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:44.040+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:44.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:44 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:44.072+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:44.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:44 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:44.072+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:44.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:44 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:44.072+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:44.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:44 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:44.072+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:44.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:44 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:44.202+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:44.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:44 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:44.202+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:44.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:44 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:44.202+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:45.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:44 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:44.797+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:45.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:44 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:44.797+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:45.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:44 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:44.797+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:45.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:44 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:44.797+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:45.304 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:45 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:45.120+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:45.304 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:45 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:45.120+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:45.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:45 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:45.120+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:45.305 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:45 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:45.120+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.895013+0000 front 2024-04-21T08:58:19.894995+0000 (oldest deadline 2024-04-21T08:58:43.995182+0000) 2024-04-21T08:58:45.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:45 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:45.041+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:45.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:45 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:45.041+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:45.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:45 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:45.041+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:45.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:45 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:45.041+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:45.305 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:45 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:45.041+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:45.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:45 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:45.089+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:45.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:45 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:45.089+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:45.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:45 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:45.089+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:45.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:45 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:45.089+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:45.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:45 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:45.219+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:45.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:45 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:45.219+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:45.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:45 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:45.219+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:46.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:45 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:45.843+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:46.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:45 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:45.843+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:46.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:45 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:45.843+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:46.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:45 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:45.843+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:46.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:45 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:45.843+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:46.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:46 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:46.098+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:46.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:46 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:46.098+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:46.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:46 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:46.098+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:46.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:46 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:46.098+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:46.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:46 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:46.248+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:46.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:46 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:46.248+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:46.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:46 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:46.248+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:46.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:46 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:46.166+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:52.888550+0000 front 2024-04-21T08:57:52.888593+0000 (oldest deadline 2024-04-21T08:58:18.188735+0000) 2024-04-21T08:58:46.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:46 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:46.166+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.289713+0000 front 2024-04-21T08:58:02.289689+0000 (oldest deadline 2024-04-21T08:58:26.890117+0000) 2024-04-21T08:58:46.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:46 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:46.166+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.893456+0000 front 2024-04-21T08:58:11.893272+0000 (oldest deadline 2024-04-21T08:58:34.793122+0000) 2024-04-21T08:58:46.555 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:46 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7[49347]: 2024-04-21T08:58:46.166+0000 7f0c1dd76700 -1 osd.7 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.895013+0000 front 2024-04-21T08:58:19.894995+0000 (oldest deadline 2024-04-21T08:58:43.995182+0000) 2024-04-21T08:58:46.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:46 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:46.088+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:46.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:46 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:46.088+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:46.556 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:46 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:46.088+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:46.556 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:46 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:46.088+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:46.556 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:46 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:46.088+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:47.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:46 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:46.836+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:47.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:46 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:46.836+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:47.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:46 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:46.836+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:47.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:46 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:46.836+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:47.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:46 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:46.836+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:47.441 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:47 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:47.131+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:47.442 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:47 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:47.131+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:47.442 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:47 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:47.131+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:47.442 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:47 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:47.131+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:47.442 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:47 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:47.131+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:47.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:47 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:47.141+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:47.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:47 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:47.141+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:47.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:47 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:47.141+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:47.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:47 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:47.141+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:47.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:47 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:47.253+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:47.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:47 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:47.253+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:47.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:47 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:47.253+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:47.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:47 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:47.253+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:47.723 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:47 smithi016 podman[67425]: 2024-04-21 08:58:47.442330581 +0000 UTC m=+5.539924804 container died c1bbed5ce7fd9b1389e94969f97e8b70991e0290d8640201f1d038d0fe5da967 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:48.054 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:47 smithi016 podman[67425]: 2024-04-21 08:58:47.796994333 +0000 UTC m=+5.894588554 container cleanup c1bbed5ce7fd9b1389e94969f97e8b70991e0290d8640201f1d038d0fe5da967 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=) 2024-04-21T08:58:48.054 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:47 smithi016 bash[67425]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7 2024-04-21T08:58:48.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:47.845+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:48.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:47.845+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:48.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:47.845+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:48.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:47.845+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:48.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:47 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:47.845+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:48.221 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:48 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:48.143+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:48.221 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:48 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:48.143+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:48.221 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:48 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:48.143+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:48.222 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:48 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:48.143+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:48.525 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:48 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:48.220+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:48.526 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:48 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:48.220+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:48.526 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:48 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:48.220+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:48.526 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:48 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:48.220+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:48.554 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:48 smithi016 podman[67442]: 2024-04-21 08:58:48.248351709 +0000 UTC m=+0.802851338 container remove c1bbed5ce7fd9b1389e94969f97e8b70991e0290d8640201f1d038d0fe5da967 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:58:48.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:48 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:48.146+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:48.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:48 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:48.146+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:48.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:48 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:48.146+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:48.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:48 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:48.146+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:48.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:48 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:48.146+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:48.892 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:48 smithi016 podman[67532]: 2024-04-21 08:58:48.47650087 +0000 UTC m=+0.018485616 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:48.892 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:48 smithi016 podman[67532]: 2024-04-21 08:58:48.607480604 +0000 UTC m=+0.149465324 container create 64c7cb2ec0c873333ce3215a0204f9ffb472e78ae35258a3a11e52d5d6b6bd32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:58:48.893 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:48 smithi016 podman[67532]: 2024-04-21 08:58:48.810579092 +0000 UTC m=+0.352563825 container init 64c7cb2ec0c873333ce3215a0204f9ffb472e78ae35258a3a11e52d5d6b6bd32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:48.893 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:48 smithi016 podman[67532]: 2024-04-21 08:58:48.814140531 +0000 UTC m=+0.356125253 container start 64c7cb2ec0c873333ce3215a0204f9ffb472e78ae35258a3a11e52d5d6b6bd32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-21T08:58:48.893 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:48 smithi016 podman[67532]: 2024-04-21 08:58:48.893375699 +0000 UTC m=+0.435360420 container attach 64c7cb2ec0c873333ce3215a0204f9ffb472e78ae35258a3a11e52d5d6b6bd32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:58:49.053 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:48 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:48.800+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:49.054 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:48 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:48.800+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:49.054 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:48 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:48.800+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:49.054 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:48 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:48.800+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:49.054 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:48 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:48.800+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:49.188 INFO:journalctl@ceph.osd.7.smithi016.stdout:Apr 21 08:58:49 smithi016 podman[67532]: 2024-04-21 08:58:49.115723419 +0000 UTC m=+0.657708151 container died 64c7cb2ec0c873333ce3215a0204f9ffb472e78ae35258a3a11e52d5d6b6bd32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-21T08:58:49.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:49 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:49.187+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:49.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:49 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:49.187+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:49.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:49 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:49.187+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:49.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:49 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:49.187+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:49.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:49 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:49.187+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:49.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:49 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:49.153+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:49.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:49 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:49.153+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:49.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:49 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:49.153+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:49.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:49 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:49.153+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:49.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:49 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:49.178+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:49.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:49 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:49.178+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:49.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:49 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:49.178+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:49.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:49 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:49.178+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:49.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.7.service' 2024-04-21T08:58:49.920 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:49.920 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-21T08:58:49.921 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-21T08:58:49.921 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.8 2024-04-21T08:58:50.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:49 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:49.801+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:50.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:49 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:49.801+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:50.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:49 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:49.801+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:50.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:49 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:49.801+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:50.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:49 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:49.801+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:50.215 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:49 smithi077 systemd[1]: Stopping Ceph osd.8 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:50.215 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:50 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:50.162+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:50.215 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:50 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:50.162+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:50.215 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:50 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:50.162+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:50.216 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:50 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:50.162+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:50.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:50 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:50.225+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:50.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:50 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:50.225+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:50.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:50 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:50.225+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:50.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:50 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:50.225+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:50.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:50 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:50.225+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:50.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:50 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:50.261+0000 7f42b2b12700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:50.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:50 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:50.261+0000 7f42b2b12700 -1 osd.8 120 *** Got signal Terminated *** 2024-04-21T08:58:50.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:50 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:50.261+0000 7f42b2b12700 -1 osd.8 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:50.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:50 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:50.215+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:50.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:50 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:50.215+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:50.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:50 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:50.215+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:50.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:50 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:50.215+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:51.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:50 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:50.767+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:51.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:50 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:50.767+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:51.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:50 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:50.767+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:51.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:50 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:50.767+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:51.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:50 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:50.767+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:51.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:51 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:51.248+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:51.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:51 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:51.248+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:51.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:51 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:51.248+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:51.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:51 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:51.248+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:51.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:51 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:51.248+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:51.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:51 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:51.140+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:51.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:51 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:51.140+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:51.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:51 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:51.140+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:51.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:51 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:51.140+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:51.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:51 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:51.222+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:51.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:51 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:51.222+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:51.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:51 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:51.222+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:51.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:51 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:51.222+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:52.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:51 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:51.808+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:52.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:51 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:51.808+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:52.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:51 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:51.808+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:52.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:51 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:51.808+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:52.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:51 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:51.808+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:52.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:52 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:52.227+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:52.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:52 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:52.227+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:52.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:52 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:52.227+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:52.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:52 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:52.227+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:52.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:52 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:52.227+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:52.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:52.183+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:52.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:52.183+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:52.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:52.183+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:52.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:52.183+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:52.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:52.183+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:58:52.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:52.122+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:52.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:52.122+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:52.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:52.122+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:52.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:52.122+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:52.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:52.122+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:26.322118+0000 front 2024-04-21T08:58:26.322080+0000 (oldest deadline 2024-04-21T08:58:52.122880+0000) 2024-04-21T08:58:53.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:52 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:52.807+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:53.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:52 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:52.807+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:53.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:52 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:52.807+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:53.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:52 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:52.807+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:53.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:52 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:52.807+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:53.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:53 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:53.191+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:53.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:53 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:53.191+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:53.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:53 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:53.191+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:53.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:53 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:53.191+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:53.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:53 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:53.191+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:53.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:53 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:53.191+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:58:53.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:53 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:53.135+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:53.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:53 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:53.135+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:53.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:53 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:53.135+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:53.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:53 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:53.135+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:53.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:53 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:53.135+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:26.322118+0000 front 2024-04-21T08:58:26.322080+0000 (oldest deadline 2024-04-21T08:58:52.122880+0000) 2024-04-21T08:58:53.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:53 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:53.156+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:53.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:53 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:53.156+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:53.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:53 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:53.156+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:53.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:53 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:53.156+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:53.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:53 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:53.156+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:58:54.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:53 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:53.785+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:54.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:53 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:53.785+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:54.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:53 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:53.785+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:54.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:53 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:53.785+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:54.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:53 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:53.785+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:54.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:54 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:54.230+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:54.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:54 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:54.230+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:54.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:54 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:54.230+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:54.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:54 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:54.230+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:54.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:54 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:54.230+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:54.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:54 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:54.230+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:58:54.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:54.161+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:54.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:54.161+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:54.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:54.161+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:54.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:54.161+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:54.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:54.161+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:58:54.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:54.169+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:54.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:54.169+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:54.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:54.169+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:54.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:54.169+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:54.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:54.169+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:26.322118+0000 front 2024-04-21T08:58:26.322080+0000 (oldest deadline 2024-04-21T08:58:52.122880+0000) 2024-04-21T08:58:55.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:54 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:54.827+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:55.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:54 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:54.827+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:55.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:54 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:54.827+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:55.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:54 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:54.827+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:55.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:54 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:54.827+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:55.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:54 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:54.827+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.894724+0000 front 2024-04-21T08:58:27.894626+0000 (oldest deadline 2024-04-21T08:58:53.794423+0000) 2024-04-21T08:58:55.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:55 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:55.262+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:55.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:55 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:55.262+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:55.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:55 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:55.262+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:55.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:55 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:55.262+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:55.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:55 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:55.262+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:55.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:55 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:55.262+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:58:55.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:55 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:55.167+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:56.716215+0000 front 2024-04-21T08:57:56.716304+0000 (oldest deadline 2024-04-21T08:58:19.616201+0000) 2024-04-21T08:58:55.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:55 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:55.167+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.318398+0000 front 2024-04-21T08:58:05.318425+0000 (oldest deadline 2024-04-21T08:58:30.018524+0000) 2024-04-21T08:58:55.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:55 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:55.167+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:11.719765+0000 front 2024-04-21T08:58:11.719842+0000 (oldest deadline 2024-04-21T08:58:35.819786+0000) 2024-04-21T08:58:55.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:55 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:55.167+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:20.521231+0000 front 2024-04-21T08:58:20.521260+0000 (oldest deadline 2024-04-21T08:58:42.821109+0000) 2024-04-21T08:58:55.605 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:55 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8[49884]: 2024-04-21T08:58:55.167+0000 7f42ab0fe700 -1 osd.8 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:26.322118+0000 front 2024-04-21T08:58:26.322080+0000 (oldest deadline 2024-04-21T08:58:52.122880+0000) 2024-04-21T08:58:55.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:55 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:55.134+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:55.606 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:55 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:55.134+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:55.606 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:55 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:55.134+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:55.606 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:55 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:55.134+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:55.606 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:55 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:55.134+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:58:56.099 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:55 smithi077 podman[69562]: 2024-04-21 08:58:55.695523536 +0000 UTC m=+5.572272640 container died 078c4b0dcb591b461c126280bb28bc58aeeacd6119fbab921974d2aa2b8e8634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:58:56.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:55 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:55.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:56.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:55 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:55.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:56.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:55 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:55.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:56.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:55 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:55.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:56.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:55 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:55.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:56.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:55 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:55.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.894724+0000 front 2024-04-21T08:58:27.894626+0000 (oldest deadline 2024-04-21T08:58:53.794423+0000) 2024-04-21T08:58:56.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:56 smithi077 podman[69562]: 2024-04-21 08:58:56.188271362 +0000 UTC m=+6.065020466 container cleanup 078c4b0dcb591b461c126280bb28bc58aeeacd6119fbab921974d2aa2b8e8634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-21T08:58:56.354 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:56 smithi077 bash[69562]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8 2024-04-21T08:58:56.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:56 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:56.099+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:56.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:56 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:56.099+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:56.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:56 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:56.099+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:56.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:56 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:56.099+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:56.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:56 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:56.099+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:58:56.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:56 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:56.296+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:56.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:56 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:56.296+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:56.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:56 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:56.296+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:56.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:56 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:56.296+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:56.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:56 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:56.296+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:56.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:56 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:56.296+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:58:56.767 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:56 smithi077 podman[69579]: 2024-04-21 08:58:56.475587777 +0000 UTC m=+0.773243537 container remove 078c4b0dcb591b461c126280bb28bc58aeeacd6119fbab921974d2aa2b8e8634 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:58:57.020 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:56 smithi077 podman[69671]: 2024-04-21 08:58:56.670565291 +0000 UTC m=+0.018482278 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:57.020 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:56 smithi077 podman[69671]: 2024-04-21 08:58:56.77626096 +0000 UTC m=+0.124177978 container create 3fa4cd531333872e98c3718cbfe259478fdb0aed01ddaedd7d471bda325eb536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:58:57.020 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:57 smithi077 podman[69671]: 2024-04-21 08:58:57.02092971 +0000 UTC m=+0.368846695 container init 3fa4cd531333872e98c3718cbfe259478fdb0aed01ddaedd7d471bda325eb536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:58:57.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:56 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:56.849+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:57.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:56 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:56.849+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:57.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:56 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:56.849+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:57.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:56 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:56.849+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:57.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:56 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:56.849+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:57.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:56 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:56.849+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.894724+0000 front 2024-04-21T08:58:27.894626+0000 (oldest deadline 2024-04-21T08:58:53.794423+0000) 2024-04-21T08:58:57.311 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:57 smithi077 podman[69671]: 2024-04-21 08:58:57.024343737 +0000 UTC m=+0.372260714 container start 3fa4cd531333872e98c3718cbfe259478fdb0aed01ddaedd7d471bda325eb536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-21T08:58:57.311 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:57 smithi077 podman[69671]: 2024-04-21 08:58:57.128965604 +0000 UTC m=+0.476882580 container attach 3fa4cd531333872e98c3718cbfe259478fdb0aed01ddaedd7d471bda325eb536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-21T08:58:57.311 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:57 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:57.090+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:57.311 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:57 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:57.090+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:57.312 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:57 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:57.090+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:57.312 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:57 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:57.090+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:57.312 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:57 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:57.090+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:58:57.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:57 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:57.261+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:57.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:57 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:57.261+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:57.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:57 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:57.261+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:57.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:57 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:57.261+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:57.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:57 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:57.261+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:57.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:57 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:57.261+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:58:57.604 INFO:journalctl@ceph.osd.8.smithi077.stdout:Apr 21 08:58:57 smithi077 podman[69671]: 2024-04-21 08:58:57.311885078 +0000 UTC m=+0.659802063 container died 3fa4cd531333872e98c3718cbfe259478fdb0aed01ddaedd7d471bda325eb536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-21T08:58:57.975 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.8.service' 2024-04-21T08:58:58.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:58.015 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-21T08:58:58.015 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-21T08:58:58.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.9 2024-04-21T08:58:58.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:57 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:57.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:58.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:57 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:57.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:58.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:57 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:57.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:58.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:57 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:57.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:58.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:57 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:57.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:58.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:57 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:57.844+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.894724+0000 front 2024-04-21T08:58:27.894626+0000 (oldest deadline 2024-04-21T08:58:53.794423+0000) 2024-04-21T08:58:58.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:58 smithi007 systemd[1]: Stopping Ceph osd.9 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:58.268 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:58 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:58.078+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:58.268 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:58 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:58.078+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:58.268 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:58 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:58.078+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:58.268 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:58 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:58.078+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:58.268 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:58 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:58.078+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:58:58.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:58 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:58.236+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:58.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:58 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:58.236+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:58.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:58 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:58.236+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:58.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:58 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:58.236+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:58.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:58 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:58.236+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:58.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:58 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:58.236+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:58:58.652 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:58 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:58.327+0000 7f96fcfd5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:58.652 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:58 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:58.327+0000 7f96fcfd5700 -1 osd.9 120 *** Got signal Terminated *** 2024-04-21T08:58:58.652 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:58 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:58.327+0000 7f96fcfd5700 -1 osd.9 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:59.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:58 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:58.800+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:58:59.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:58 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:58.800+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:58:59.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:58 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:58.800+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:58:59.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:58 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:58.800+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:58:59.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:58 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:58.800+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:58:59.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:58 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:58.800+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.894724+0000 front 2024-04-21T08:58:27.894626+0000 (oldest deadline 2024-04-21T08:58:53.794423+0000) 2024-04-21T08:58:59.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:59 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:59.258+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:58:59.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:59 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:59.258+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:58:59.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:59 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:59.258+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:58:59.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:59 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:59.258+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:58:59.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:59 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:59.258+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:58:59.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:58:59 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:58:59.258+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:58:59.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:59 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:59.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:58:59.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:59 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:59.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:58:59.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:59 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:59.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:58:59.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:59 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:59.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:58:59.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:58:59 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:58:59.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:00.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:59 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:59.818+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:59:00.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:59 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:59.818+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:59:00.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:59 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:59.818+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:59:00.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:59 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:59.818+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:59:00.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:59 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:59.818+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:59:00.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:58:59 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:58:59.818+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.894724+0000 front 2024-04-21T08:58:27.894626+0000 (oldest deadline 2024-04-21T08:58:53.794423+0000) 2024-04-21T08:59:00.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:00 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:00.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:00.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:00 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:00.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:00.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:00 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:00.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:00.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:00 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:00.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:00.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:00 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:00.103+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:00.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:00 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:00.229+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:00.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:00 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:00.229+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:00.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:00 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:00.229+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:00.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:00 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:00.229+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:00.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:00 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:00.229+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:00.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:00 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:00.229+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:01.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:00 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:00.808+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:59:01.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:00 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:00.808+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:59:01.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:00 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:00.808+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:59:01.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:00 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:00.808+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:59:01.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:00 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:00.808+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:59:01.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:00 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:00.808+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.894724+0000 front 2024-04-21T08:58:27.894626+0000 (oldest deadline 2024-04-21T08:58:53.794423+0000) 2024-04-21T08:59:01.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:01 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:01.192+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:01.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:01 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:01.192+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:01.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:01 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:01.192+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:01.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:01 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:01.192+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:01.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:01 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:01.192+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:01.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:01 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:01.192+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:01.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:01 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:01.124+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:01.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:01 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:01.124+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:01.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:01 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:01.124+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:01.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:01 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:01.124+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:01.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:01 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:01.124+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:01.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:01 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:01.124+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:02.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:01 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:01.835+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:59:02.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:01 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:01.835+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:59:02.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:01 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:01.835+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:59:02.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:01 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:01.835+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:59:02.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:01 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:01.835+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:59:02.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:01 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:01.835+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.894724+0000 front 2024-04-21T08:58:27.894626+0000 (oldest deadline 2024-04-21T08:58:53.794423+0000) 2024-04-21T08:59:02.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:02 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:02.165+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:02.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:02 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:02.165+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:02.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:02 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:02.165+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:02.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:02 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:02.165+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:02.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:02 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:02.165+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:02.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:02 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:02.165+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:02.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:02 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:02.165+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:37.836547+0000 front 2024-04-21T08:58:37.836572+0000 (oldest deadline 2024-04-21T08:59:01.336410+0000) 2024-04-21T08:59:02.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:02 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:02.120+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:02.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:02 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:02.120+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:02.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:02 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:02.120+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:02.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:02 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:02.120+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:02.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:02 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:02.120+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:02.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:02 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:02.120+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:03.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:02 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:02.884+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:46.486628+0000 front 2024-04-21T08:57:46.486442+0000 (oldest deadline 2024-04-21T08:58:10.586676+0000) 2024-04-21T08:59:03.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:02 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:02.885+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:55.288031+0000 front 2024-04-21T08:57:55.288115+0000 (oldest deadline 2024-04-21T08:58:19.888659+0000) 2024-04-21T08:59:03.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:02 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:02.885+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:02.789924+0000 front 2024-04-21T08:58:02.789774+0000 (oldest deadline 2024-04-21T08:58:26.889727+0000) 2024-04-21T08:59:03.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:02 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:02.885+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:10.991074+0000 front 2024-04-21T08:58:10.991046+0000 (oldest deadline 2024-04-21T08:58:35.691168+0000) 2024-04-21T08:59:03.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:02 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:02.885+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:22.093134+0000 front 2024-04-21T08:58:22.093169+0000 (oldest deadline 2024-04-21T08:58:44.993283+0000) 2024-04-21T08:59:03.153 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:02 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9[61641]: 2024-04-21T08:59:02.885+0000 7f96f55c1700 -1 osd.9 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.894724+0000 front 2024-04-21T08:58:27.894626+0000 (oldest deadline 2024-04-21T08:58:53.794423+0000) 2024-04-21T08:59:03.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:03 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:03.086+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:03.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:03 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:03.086+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:03.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:03 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:03.086+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:03.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:03 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:03.086+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:03.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:03 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:03.086+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:03.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:03 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:03.086+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:03.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:03 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:03.117+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:03.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:03 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:03.117+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:03.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:03 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:03.117+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:03.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:03 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:03.117+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:03.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:03 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:03.117+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:03.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:03 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:03.117+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:03.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:03 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:03.117+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:37.836547+0000 front 2024-04-21T08:58:37.836572+0000 (oldest deadline 2024-04-21T08:59:01.336410+0000) 2024-04-21T08:59:04.240 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:03 smithi007 podman[104202]: 2024-04-21 08:59:03.972063125 +0000 UTC m=+5.759968447 container died fd7fd400012683bece649e487b805b2f499b37390fba627e0bf9da2db61946ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-21T08:59:04.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:04 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:04.134+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:04.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:04 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:04.134+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:04.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:04 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:04.134+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:04.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:04 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:04.134+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:04.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:04 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:04.134+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:04.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:04 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:04.134+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:04.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:04 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:04.134+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:37.836547+0000 front 2024-04-21T08:58:37.836572+0000 (oldest deadline 2024-04-21T08:59:01.336410+0000) 2024-04-21T08:59:04.560 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:04 smithi007 podman[104202]: 2024-04-21 08:59:04.318438778 +0000 UTC m=+6.106344099 container cleanup fd7fd400012683bece649e487b805b2f499b37390fba627e0bf9da2db61946ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-21T08:59:04.560 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:04 smithi007 bash[104202]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9 2024-04-21T08:59:04.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:04 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:04.116+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:04.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:04 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:04.116+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:04.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:04 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:04.116+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:04.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:04 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:04.116+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:04.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:04 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:04.116+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:04.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:04 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:04.116+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:04.867 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:04 smithi007 podman[104222]: 2024-04-21 08:59:04.56122597 +0000 UTC m=+0.589180655 container remove fd7fd400012683bece649e487b805b2f499b37390fba627e0bf9da2db61946ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-21T08:59:05.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:04 smithi007 podman[104315]: 2024-04-21 08:59:04.770740817 +0000 UTC m=+0.018111030 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:59:05.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:04 smithi007 podman[104315]: 2024-04-21 08:59:04.895305588 +0000 UTC m=+0.142675812 container create 90b25316a624760cd13a1ed5f1a0bf0af6e2e3b0c82b785f8df9deb06e7d681e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-21T08:59:05.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:05 smithi007 podman[104315]: 2024-04-21 08:59:05.097752179 +0000 UTC m=+0.345122358 container init 90b25316a624760cd13a1ed5f1a0bf0af6e2e3b0c82b785f8df9deb06e7d681e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-21T08:59:05.152 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:05 smithi007 podman[104315]: 2024-04-21 08:59:05.101019432 +0000 UTC m=+0.348389616 container start 90b25316a624760cd13a1ed5f1a0bf0af6e2e3b0c82b785f8df9deb06e7d681e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-21T08:59:05.405 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:05 smithi007 podman[104315]: 2024-04-21 08:59:05.197942096 +0000 UTC m=+0.445312290 container attach 90b25316a624760cd13a1ed5f1a0bf0af6e2e3b0c82b785f8df9deb06e7d681e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-21T08:59:05.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:05 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:05.139+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:05.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:05 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:05.139+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:05.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:05 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:05.139+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:05.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:05 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:05.139+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:05.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:05 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:05.139+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:05.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:05 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:05.139+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:05.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:05 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:05.139+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:37.836547+0000 front 2024-04-21T08:58:37.836572+0000 (oldest deadline 2024-04-21T08:59:01.336410+0000) 2024-04-21T08:59:05.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:05 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:05.146+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:05.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:05 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:05.146+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:05.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:05 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:05.146+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:05.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:05 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:05.146+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:05.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:05 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:05.146+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:05.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:05 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:05.146+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:05.691 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:05 smithi007 podman[104315]: 2024-04-21 08:59:05.406340626 +0000 UTC m=+0.653710825 container died 90b25316a624760cd13a1ed5f1a0bf0af6e2e3b0c82b785f8df9deb06e7d681e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416) 2024-04-21T08:59:05.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.9.service' 2024-04-21T08:59:05.964 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:05 smithi007 podman[104340]: 2024-04-21 08:59:05.926009046 +0000 UTC m=+0.510679091 container remove 90b25316a624760cd13a1ed5f1a0bf0af6e2e3b0c82b785f8df9deb06e7d681e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:59:05.964 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:05 smithi007 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.9.service: Deactivated successfully. 2024-04-21T08:59:05.964 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:05 smithi007 systemd[1]: Stopped Ceph osd.9 for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:59:05.964 INFO:journalctl@ceph.osd.9.smithi007.stdout:Apr 21 08:59:05 smithi007 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.9.service: Consumed 11min 31.959s CPU time. 2024-04-21T08:59:05.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:05.980 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-21T08:59:05.980 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-21T08:59:05.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.10 2024-04-21T08:59:06.301 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:06 smithi016 systemd[1]: Stopping Ceph osd.10 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:06.301 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:06 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:06.161+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:06.301 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:06 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:06.161+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:06.302 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:06 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:06.161+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:06.302 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:06 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:06.161+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:06.302 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:06 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:06.161+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:06.302 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:06 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:06.161+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:06.302 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:06 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:06.161+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:37.836547+0000 front 2024-04-21T08:58:37.836572+0000 (oldest deadline 2024-04-21T08:59:01.336410+0000) 2024-04-21T08:59:06.302 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:06 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:06.301+0000 7fe2722be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:06.302 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:06 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:06.301+0000 7fe2722be700 -1 osd.10 120 *** Got signal Terminated *** 2024-04-21T08:59:06.302 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:06 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:06.301+0000 7fe2722be700 -1 osd.10 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:59:06.505 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:06 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:06.126+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:06.506 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:06 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:06.126+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:06.506 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:06 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:06.126+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:06.506 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:06 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:06.126+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:06.506 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:06 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:06.126+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:06.506 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:06 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:06.126+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:07.517 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:07 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:07.130+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:07.517 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:07 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:07.130+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:07.517 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:07 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:07.130+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:07.517 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:07 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:07.130+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:07.518 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:07 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:07.130+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:07.518 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:07 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:07.130+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:07.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:07 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:07.175+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:07.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:07 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:07.175+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:07.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:07 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:07.175+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:07.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:07 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:07.175+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:07.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:07 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:07.175+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:07.556 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:07 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:07.175+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:07.556 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:07 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:07.175+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:37.836547+0000 front 2024-04-21T08:58:37.836572+0000 (oldest deadline 2024-04-21T08:59:01.336410+0000) 2024-04-21T08:59:08.522 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:08 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:08.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:08.522 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:08 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:08.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:08.522 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:08 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:08.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:08.522 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:08 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:08.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:08.522 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:08 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:08.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:08.523 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:08 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:08.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:08.523 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:08 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:08.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:37.836547+0000 front 2024-04-21T08:58:37.836572+0000 (oldest deadline 2024-04-21T08:59:01.336410+0000) 2024-04-21T08:59:08.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:08 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:08.123+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:08.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:08 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:08.123+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:08.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:08 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:08.123+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:08.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:08 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:08.123+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:08.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:08 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:08.123+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:08.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:08 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:08.123+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:09.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:09 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:09.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:09.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:09 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:09.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:09.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:09 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:09.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:09.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:09 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:09.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:09.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:09 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:09.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:09.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:09 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:09.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:09.556 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:09 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:09.181+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:37.836547+0000 front 2024-04-21T08:58:37.836572+0000 (oldest deadline 2024-04-21T08:59:01.336410+0000) 2024-04-21T08:59:09.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:09 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:09.131+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:09.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:09 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:09.131+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:09.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:09 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:09.131+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:09.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:09 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:09.131+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:09.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:09 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:09.132+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:09.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:09 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:09.132+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:10.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:10 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:10.188+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:10.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:10 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:10.188+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:10.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:10 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:10.188+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:10.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:10 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:10.188+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:10.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:10 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:10.188+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:10.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:10 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:10.188+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:10.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:10 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:10.188+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:37.836547+0000 front 2024-04-21T08:58:37.836572+0000 (oldest deadline 2024-04-21T08:59:01.336410+0000) 2024-04-21T08:59:10.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:10 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:10.109+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:10.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:10 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:10.109+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:10.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:10 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:10.109+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:10.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:10 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:10.109+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:10.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:10 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:10.109+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:10.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:10 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:10.109+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:10.605 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:10 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:10.109+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6820 osd.7 since back 2024-04-21T08:58:43.755332+0000 front 2024-04-21T08:58:43.755398+0000 (oldest deadline 2024-04-21T08:59:09.655124+0000) 2024-04-21T08:59:11.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:11 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:11.069+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:11.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:11 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:11.069+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:11.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:11 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:11.069+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:11.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:11 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:11.069+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:11.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:11 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:11.069+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:11.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:11 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:11.069+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:11.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:11 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:11.069+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6820 osd.7 since back 2024-04-21T08:58:43.755332+0000 front 2024-04-21T08:58:43.755398+0000 (oldest deadline 2024-04-21T08:59:09.655124+0000) 2024-04-21T08:59:11.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:11 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:11.218+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.227988+0000 front 2024-04-21T08:57:48.228084+0000 (oldest deadline 2024-04-21T08:58:10.527899+0000) 2024-04-21T08:59:11.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:11 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:11.218+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6804 osd.1 since back 2024-04-21T08:57:54.529719+0000 front 2024-04-21T08:57:54.529812+0000 (oldest deadline 2024-04-21T08:58:20.429788+0000) 2024-04-21T08:59:11.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:11 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:11.218+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:03.330999+0000 front 2024-04-21T08:58:03.331035+0000 (oldest deadline 2024-04-21T08:58:29.230952+0000) 2024-04-21T08:59:11.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:11 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:11.218+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.232705+0000 front 2024-04-21T08:58:13.232795+0000 (oldest deadline 2024-04-21T08:58:39.132704+0000) 2024-04-21T08:59:11.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:11 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:11.218+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:19.133150+0000 front 2024-04-21T08:58:19.133190+0000 (oldest deadline 2024-04-21T08:58:43.233288+0000) 2024-04-21T08:59:11.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:11 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:11.218+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:30.235439+0000 front 2024-04-21T08:58:30.235520+0000 (oldest deadline 2024-04-21T08:58:52.535305+0000) 2024-04-21T08:59:11.555 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:11 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10[55943]: 2024-04-21T08:59:11.218+0000 7fe26a8aa700 -1 osd.10 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:37.836547+0000 front 2024-04-21T08:58:37.836572+0000 (oldest deadline 2024-04-21T08:59:01.336410+0000) 2024-04-21T08:59:12.054 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:11 smithi016 podman[67649]: 2024-04-21 08:59:11.802621624 +0000 UTC m=+5.624100692 container died fe0c2279ec16d2736554593f485a19c21d4a394cb36258e04356f229eb809d29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:59:12.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:12 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:12.061+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:12.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:12 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:12.061+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:12.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:12 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:12.061+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:12.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:12 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:12.061+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:12.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:12 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:12.061+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:12.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:12 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:12.061+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:12.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:12 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:12.061+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6820 osd.7 since back 2024-04-21T08:58:43.755332+0000 front 2024-04-21T08:58:43.755398+0000 (oldest deadline 2024-04-21T08:59:09.655124+0000) 2024-04-21T08:59:12.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:12 smithi016 podman[67649]: 2024-04-21 08:59:12.143370291 +0000 UTC m=+5.964849359 container cleanup fe0c2279ec16d2736554593f485a19c21d4a394cb36258e04356f229eb809d29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:59:12.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:12 smithi016 bash[67649]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10 2024-04-21T08:59:12.554 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:12 smithi016 podman[67666]: 2024-04-21 08:59:12.319505257 +0000 UTC m=+0.509250007 container remove fe0c2279ec16d2736554593f485a19c21d4a394cb36258e04356f229eb809d29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-21T08:59:13.017 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:12 smithi016 podman[67759]: 2024-04-21 08:59:12.670164033 +0000 UTC m=+0.018150349 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:59:13.017 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:12 smithi016 podman[67759]: 2024-04-21 08:59:12.778686286 +0000 UTC m=+0.126672636 container create 0c73ac578e081ea90346c5f1f90c645df256c734f07debca6f59130a79766fd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5) 2024-04-21T08:59:13.017 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:13 smithi016 podman[67759]: 2024-04-21 08:59:13.014461951 +0000 UTC m=+0.362448258 container init 0c73ac578e081ea90346c5f1f90c645df256c734f07debca6f59130a79766fd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-21T08:59:13.017 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:13 smithi016 podman[67759]: 2024-04-21 08:59:13.017930596 +0000 UTC m=+0.365916901 container start 0c73ac578e081ea90346c5f1f90c645df256c734f07debca6f59130a79766fd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-21T08:59:13.304 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:13 smithi016 podman[67759]: 2024-04-21 08:59:13.122845088 +0000 UTC m=+0.470831413 container attach 0c73ac578e081ea90346c5f1f90c645df256c734f07debca6f59130a79766fd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:59:13.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:13 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:13.034+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:13.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:13 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:13.034+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:13.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:13 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:13.034+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:13.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:13 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:13.034+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:13.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:13 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:13.034+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:13.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:13 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:13.034+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:13.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:13 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:13.034+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6820 osd.7 since back 2024-04-21T08:58:43.755332+0000 front 2024-04-21T08:58:43.755398+0000 (oldest deadline 2024-04-21T08:59:09.655124+0000) 2024-04-21T08:59:13.617 INFO:journalctl@ceph.osd.10.smithi016.stdout:Apr 21 08:59:13 smithi016 podman[67759]: 2024-04-21 08:59:13.326267576 +0000 UTC m=+0.674253883 container died 0c73ac578e081ea90346c5f1f90c645df256c734f07debca6f59130a79766fd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:59:14.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.10.service' 2024-04-21T08:59:14.061 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:14.062 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-21T08:59:14.062 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-21T08:59:14.062 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.11 2024-04-21T08:59:14.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:14.081+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:14.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:14.081+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:14.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:14.081+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:14.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:14.081+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:14.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:14.081+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:14.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:14.081+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:14.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:14.081+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6820 osd.7 since back 2024-04-21T08:58:43.755332+0000 front 2024-04-21T08:58:43.755398+0000 (oldest deadline 2024-04-21T08:59:09.655124+0000) 2024-04-21T08:59:14.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:14 smithi077 systemd[1]: Stopping Ceph osd.11 for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:14.854 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:14.376+0000 7f9b8109c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:14.854 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:14.376+0000 7f9b8109c700 -1 osd.11 120 *** Got signal Terminated *** 2024-04-21T08:59:14.854 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:14 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:14.376+0000 7f9b8109c700 -1 osd.11 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:59:15.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:15 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:15.077+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:15.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:15 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:15.077+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:15.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:15 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:15.077+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:15.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:15 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:15.077+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:15.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:15 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:15.077+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:15.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:15 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:15.077+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:15.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:15 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:15.077+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6820 osd.7 since back 2024-04-21T08:58:43.755332+0000 front 2024-04-21T08:58:43.755398+0000 (oldest deadline 2024-04-21T08:59:09.655124+0000) 2024-04-21T08:59:16.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:16 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:16.098+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:16.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:16 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:16.098+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:16.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:16 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:16.098+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:16.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:16 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:16.098+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:16.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:16 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:16.098+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:16.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:16 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:16.098+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:16.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:16 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:16.098+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6820 osd.7 since back 2024-04-21T08:58:43.755332+0000 front 2024-04-21T08:58:43.755398+0000 (oldest deadline 2024-04-21T08:59:09.655124+0000) 2024-04-21T08:59:17.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:17 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:17.092+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:17.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:17 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:17.092+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:17.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:17 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:17.092+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:17.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:17 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:17.092+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:17.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:17 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:17.092+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:17.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:17 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:17.092+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:17.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:17 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:17.092+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6820 osd.7 since back 2024-04-21T08:58:43.755332+0000 front 2024-04-21T08:58:43.755398+0000 (oldest deadline 2024-04-21T08:59:09.655124+0000) 2024-04-21T08:59:18.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:18 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:18.051+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:18.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:18 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:18.051+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:18.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:18 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:18.051+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:18.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:18 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:18.051+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:18.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:18 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:18.051+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:18.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:18 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:18.051+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:18.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:18 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:18.051+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6820 osd.7 since back 2024-04-21T08:58:43.755332+0000 front 2024-04-21T08:58:43.755398+0000 (oldest deadline 2024-04-21T08:59:09.655124+0000) 2024-04-21T08:59:19.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:19.026+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-21T08:57:48.843745+0000 front 2024-04-21T08:57:48.843806+0000 (oldest deadline 2024-04-21T08:58:14.044339+0000) 2024-04-21T08:59:19.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:19.026+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6804 osd.2 since back 2024-04-21T08:58:05.047565+0000 front 2024-04-21T08:58:05.047539+0000 (oldest deadline 2024-04-21T08:58:29.747680+0000) 2024-04-21T08:59:19.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:19.026+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-04-21T08:58:13.248758+0000 front 2024-04-21T08:58:13.248788+0000 (oldest deadline 2024-04-21T08:58:34.348705+0000) 2024-04-21T08:59:19.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:19.026+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6812 osd.4 since back 2024-04-21T08:58:21.851078+0000 front 2024-04-21T08:58:21.851137+0000 (oldest deadline 2024-04-21T08:58:46.550941+0000) 2024-04-21T08:59:19.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:19.026+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.77:6812 osd.5 since back 2024-04-21T08:58:27.651822+0000 front 2024-04-21T08:58:27.651933+0000 (oldest deadline 2024-04-21T08:58:51.652532+0000) 2024-04-21T08:59:19.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:19.026+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.7:6822 osd.6 since back 2024-04-21T08:58:38.053848+0000 front 2024-04-21T08:58:38.053862+0000 (oldest deadline 2024-04-21T08:59:00.353709+0000) 2024-04-21T08:59:19.355 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:19 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11[56913]: 2024-04-21T08:59:19.026+0000 7f9b79688700 -1 osd.11 120 heartbeat_check: no reply from 172.21.15.16:6820 osd.7 since back 2024-04-21T08:58:43.755332+0000 front 2024-04-21T08:58:43.755398+0000 (oldest deadline 2024-04-21T08:59:09.655124+0000) 2024-04-21T08:59:20.226 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:19 smithi077 podman[69789]: 2024-04-21 08:59:19.924721699 +0000 UTC m=+5.669147063 container died f3266d0060c043392e8d13288cc4bf2d1115212e9b44efc87665e9e689949b3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-21T08:59:20.226 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:20 smithi077 podman[69789]: 2024-04-21 08:59:20.226762072 +0000 UTC m=+5.971187443 container cleanup f3266d0060c043392e8d13288cc4bf2d1115212e9b44efc87665e9e689949b3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:59:20.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:20 smithi077 bash[69789]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11 2024-04-21T08:59:20.604 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:20 smithi077 podman[69808]: 2024-04-21 08:59:20.469841876 +0000 UTC m=+0.543276905 container remove f3266d0060c043392e8d13288cc4bf2d1115212e9b44efc87665e9e689949b3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-21T08:59:21.089 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:20 smithi077 podman[69902]: 2024-04-21 08:59:20.6963801 +0000 UTC m=+0.017823865 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:59:21.089 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:20 smithi077 podman[69902]: 2024-04-21 08:59:20.812247851 +0000 UTC m=+0.133691615 container create eb117ecb64736cd19e351614c5a054c95ead2df4eca196996cc7337ed4f45973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-21T08:59:21.089 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:21 smithi077 podman[69902]: 2024-04-21 08:59:21.007525633 +0000 UTC m=+0.328969400 container init eb117ecb64736cd19e351614c5a054c95ead2df4eca196996cc7337ed4f45973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5) 2024-04-21T08:59:21.089 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:21 smithi077 podman[69902]: 2024-04-21 08:59:21.010906838 +0000 UTC m=+0.332350606 container start eb117ecb64736cd19e351614c5a054c95ead2df4eca196996cc7337ed4f45973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-21T08:59:21.089 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:21 smithi077 podman[69902]: 2024-04-21 08:59:21.089671619 +0000 UTC m=+0.411115410 container attach eb117ecb64736cd19e351614c5a054c95ead2df4eca196996cc7337ed4f45973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-21T08:59:21.354 INFO:journalctl@ceph.osd.11.smithi077.stdout:Apr 21 08:59:21 smithi077 podman[69902]: 2024-04-21 08:59:21.300672479 +0000 UTC m=+0.622116247 container died eb117ecb64736cd19e351614c5a054c95ead2df4eca196996cc7337ed4f45973 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-21T08:59:21.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@osd.11.service' 2024-04-21T08:59:21.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:21.980 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-21T08:59:21.980 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-21T08:59:21.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.a 2024-04-21T08:59:22.260 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:59:22 smithi007 systemd[1]: Stopping Ceph mds.a for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:22.652 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:59:22 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-a[68646]: 2024-04-21T08:59:22.259+0000 7fce939a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:22.652 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:59:22 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-a[68646]: 2024-04-21T08:59:22.259+0000 7fce939a3700 -1 mds.a *** got signal Terminated *** 2024-04-21T08:59:25.018 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:59:24 smithi007 podman[104432]: 2024-04-21 08:59:24.766522559 +0000 UTC m=+2.585731801 container died 7efc3f16e9ed2eb155f19edd0c090e921ef04e358f32eb6115b54be336a8524c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:59:25.310 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:59:25 smithi007 podman[104432]: 2024-04-21 08:59:25.110125527 +0000 UTC m=+2.929334768 container cleanup 7efc3f16e9ed2eb155f19edd0c090e921ef04e358f32eb6115b54be336a8524c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-21T08:59:25.311 INFO:journalctl@ceph.mds.a.smithi007.stdout:Apr 21 08:59:25 smithi007 bash[104432]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-a 2024-04-21T08:59:25.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.a.service' 2024-04-21T08:59:25.562 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:25.562 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-21T08:59:25.562 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-21T08:59:25.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.d 2024-04-21T08:59:25.888 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:59:25 smithi007 systemd[1]: Stopping Ceph mds.d for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:26.152 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:59:25 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-d[67691]: 2024-04-21T08:59:25.887+0000 7f09ad8bd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:26.152 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:59:25 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-d[67691]: 2024-04-21T08:59:25.887+0000 7f09ad8bd700 -1 mds.d *** got signal Terminated *** 2024-04-21T08:59:28.902 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:59:28 smithi007 podman[104554]: 2024-04-21 08:59:28.652602881 +0000 UTC m=+2.862617235 container died 6812fd59cc5d014656f46b92a96b39317c26b04b5ccabea195a913dfabf62c2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-21T08:59:29.402 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:59:28 smithi007 podman[104554]: 2024-04-21 08:59:28.980260651 +0000 UTC m=+3.190274959 container cleanup 6812fd59cc5d014656f46b92a96b39317c26b04b5ccabea195a913dfabf62c2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-d, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:59:29.402 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:59:28 smithi007 bash[104554]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-d 2024-04-21T08:59:29.402 INFO:journalctl@ceph.mds.d.smithi007.stdout:Apr 21 08:59:29 smithi007 podman[104568]: 2024-04-21 08:59:29.173137201 +0000 UTC m=+0.513258383 container remove 6812fd59cc5d014656f46b92a96b39317c26b04b5ccabea195a913dfabf62c2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-21T08:59:29.419 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.d.service' 2024-04-21T08:59:29.451 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:29.451 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-21T08:59:29.451 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-21T08:59:29.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.g 2024-04-21T08:59:29.774 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:59:29 smithi007 systemd[1]: Stopping Ceph mds.g for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:30.152 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:59:29 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-g[68289]: 2024-04-21T08:59:29.774+0000 7f0906d60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:30.152 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:59:29 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-g[68289]: 2024-04-21T08:59:29.774+0000 7f0906d60700 -1 mds.g *** got signal Terminated *** 2024-04-21T08:59:34.041 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:59:33 smithi007 podman[104675]: 2024-04-21 08:59:33.788865917 +0000 UTC m=+4.114308318 container died 0cca774d80e46bb950acc667f8d3390e682343ca291366cb88f8374857098785 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:59:34.317 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:59:34 smithi007 podman[104675]: 2024-04-21 08:59:34.133518353 +0000 UTC m=+4.458960736 container cleanup 0cca774d80e46bb950acc667f8d3390e682343ca291366cb88f8374857098785 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-g, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416) 2024-04-21T08:59:34.317 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:59:34 smithi007 bash[104675]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-g 2024-04-21T08:59:34.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.g.service' 2024-04-21T08:59:34.587 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:59:34 smithi007 podman[104690]: 2024-04-21 08:59:34.318166186 +0000 UTC m=+0.526405506 container remove 0cca774d80e46bb950acc667f8d3390e682343ca291366cb88f8374857098785 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:59:34.587 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:59:34 smithi007 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.g.service: Deactivated successfully. 2024-04-21T08:59:34.587 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:59:34 smithi007 systemd[1]: Stopped Ceph mds.g for c9c60282-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:59:34.587 INFO:journalctl@ceph.mds.g.smithi007.stdout:Apr 21 08:59:34 smithi007 systemd[1]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.g.service: Consumed 1.099s CPU time. 2024-04-21T08:59:34.597 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:34.597 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-21T08:59:34.598 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-21T08:59:34.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.j 2024-04-21T08:59:34.902 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:59:34 smithi007 systemd[1]: Stopping Ceph mds.j for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:35.402 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:59:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-j[67336]: 2024-04-21T08:59:34.928+0000 7f73da298700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:35.402 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:59:34 smithi007 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-j[67336]: 2024-04-21T08:59:34.928+0000 7f73da298700 -1 mds.j *** got signal Terminated *** 2024-04-21T08:59:38.227 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:59:37 smithi007 podman[104795]: 2024-04-21 08:59:37.927398926 +0000 UTC m=+3.103824656 container died b54239fdaa877e0d8e1435ad5c16cd04f9db6ee0fd41413a56294ebf14564809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-21T08:59:38.228 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:59:38 smithi007 podman[104795]: 2024-04-21 08:59:38.228539748 +0000 UTC m=+3.404965486 container cleanup b54239fdaa877e0d8e1435ad5c16cd04f9db6ee0fd41413a56294ebf14564809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-21T08:59:38.557 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:59:38 smithi007 bash[104795]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-j 2024-04-21T08:59:38.557 INFO:journalctl@ceph.mds.j.smithi007.stdout:Apr 21 08:59:38 smithi007 podman[104809]: 2024-04-21 08:59:38.429459843 +0000 UTC m=+0.496222172 container remove b54239fdaa877e0d8e1435ad5c16cd04f9db6ee0fd41413a56294ebf14564809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:59:38.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.j.service' 2024-04-21T08:59:38.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:38.604 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-21T08:59:38.604 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-21T08:59:38.604 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.b 2024-04-21T08:59:38.919 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:59:38 smithi016 systemd[1]: Stopping Ceph mds.b for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:39.304 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:59:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-b[61177]: 2024-04-21T08:59:38.919+0000 7f651b5ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:39.304 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:59:38 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-b[61177]: 2024-04-21T08:59:38.919+0000 7f651b5ff700 -1 mds.b *** got signal Terminated *** 2024-04-21T08:59:40.804 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:59:40 smithi016 podman[67865]: 2024-04-21 08:59:40.552168826 +0000 UTC m=+1.739191037 container died e079e6b260924e434acad7ed2ecff37d91351259b950920ad10831287e08e49d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-21T08:59:41.304 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:59:40 smithi016 podman[67865]: 2024-04-21 08:59:40.894981047 +0000 UTC m=+2.082003253 container cleanup e079e6b260924e434acad7ed2ecff37d91351259b950920ad10831287e08e49d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-b, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:59:41.304 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:59:40 smithi016 bash[67865]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-b 2024-04-21T08:59:41.304 INFO:journalctl@ceph.mds.b.smithi016.stdout:Apr 21 08:59:41 smithi016 podman[67888]: 2024-04-21 08:59:41.087538456 +0000 UTC m=+0.528370084 container remove e079e6b260924e434acad7ed2ecff37d91351259b950920ad10831287e08e49d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-b, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:59:41.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.b.service' 2024-04-21T08:59:41.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:41.363 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-21T08:59:41.363 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-21T08:59:41.363 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.e 2024-04-21T08:59:41.698 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:59:41 smithi016 systemd[1]: Stopping Ceph mds.e for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:42.054 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:59:41 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-e[60555]: 2024-04-21T08:59:41.698+0000 7f6fa42c4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:42.054 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:59:41 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-e[60555]: 2024-04-21T08:59:41.698+0000 7f6fa42c4700 -1 mds.e *** got signal Terminated *** 2024-04-21T08:59:43.966 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:59:43 smithi016 podman[67997]: 2024-04-21 08:59:43.689029594 +0000 UTC m=+2.099362301 container died 01e19dd24fe98e83914f151bd50eb300aca92f11ebc0de41cd167540502f34a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-21T08:59:44.304 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:59:44 smithi016 podman[67997]: 2024-04-21 08:59:44.04030292 +0000 UTC m=+2.450635630 container cleanup 01e19dd24fe98e83914f151bd50eb300aca92f11ebc0de41cd167540502f34a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:59:44.304 INFO:journalctl@ceph.mds.e.smithi016.stdout:Apr 21 08:59:44 smithi016 bash[67997]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-e 2024-04-21T08:59:44.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.e.service' 2024-04-21T08:59:44.499 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:44.499 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-21T08:59:44.499 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-21T08:59:44.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.h 2024-04-21T08:59:44.960 INFO:journalctl@ceph.mds.h.smithi016.stdout:Apr 21 08:59:44 smithi016 systemd[1]: Stopping Ceph mds.h for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:45.304 INFO:journalctl@ceph.mds.h.smithi016.stdout:Apr 21 08:59:44 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-h[60865]: 2024-04-21T08:59:44.961+0000 7f9a19cc3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:45.304 INFO:journalctl@ceph.mds.h.smithi016.stdout:Apr 21 08:59:44 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-h[60865]: 2024-04-21T08:59:44.961+0000 7f9a19cc3700 -1 mds.h *** got signal Terminated *** 2024-04-21T08:59:48.611 INFO:journalctl@ceph.mds.h.smithi016.stdout:Apr 21 08:59:48 smithi016 podman[68118]: 2024-04-21 08:59:48.325171665 +0000 UTC m=+3.598165876 container died 937012de44324d5c1412350062a1858d153dffcfd167f9a273e46ee374a44df1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-21T08:59:48.986 INFO:journalctl@ceph.mds.h.smithi016.stdout:Apr 21 08:59:48 smithi016 podman[68118]: 2024-04-21 08:59:48.711183894 +0000 UTC m=+3.984178104 container cleanup 937012de44324d5c1412350062a1858d153dffcfd167f9a273e46ee374a44df1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-21T08:59:48.986 INFO:journalctl@ceph.mds.h.smithi016.stdout:Apr 21 08:59:48 smithi016 bash[68118]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-h 2024-04-21T08:59:49.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.h.service' 2024-04-21T08:59:49.139 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:49.140 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-21T08:59:49.140 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-21T08:59:49.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.k 2024-04-21T08:59:49.304 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:59:49 smithi016 systemd[1]: Stopping Ceph mds.k for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:49.804 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:59:49 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-k[60238]: 2024-04-21T08:59:49.481+0000 7fded1199700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:49.805 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:59:49 smithi016 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-k[60238]: 2024-04-21T08:59:49.481+0000 7fded1199700 -1 mds.k *** got signal Terminated *** 2024-04-21T08:59:51.304 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:59:50 smithi016 podman[68237]: 2024-04-21 08:59:50.969364371 +0000 UTC m=+1.607693961 container died 06fcf0ad3eaa8889d63aba9728cede48dadcddce2cf7740b71313c2a92138243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-21T08:59:51.706 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:59:51 smithi016 podman[68237]: 2024-04-21 08:59:51.439855325 +0000 UTC m=+2.078184904 container cleanup 06fcf0ad3eaa8889d63aba9728cede48dadcddce2cf7740b71313c2a92138243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, ceph=True) 2024-04-21T08:59:51.706 INFO:journalctl@ceph.mds.k.smithi016.stdout:Apr 21 08:59:51 smithi016 bash[68237]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-k 2024-04-21T08:59:51.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.k.service' 2024-04-21T08:59:51.909 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:51.910 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-21T08:59:51.910 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-21T08:59:51.910 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.c 2024-04-21T08:59:52.301 INFO:journalctl@ceph.mds.c.smithi077.stdout:Apr 21 08:59:51 smithi077 systemd[1]: Stopping Ceph mds.c for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:52.604 INFO:journalctl@ceph.mds.c.smithi077.stdout:Apr 21 08:59:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-c[62366]: 2024-04-21T08:59:52.300+0000 7f802b7b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:52.605 INFO:journalctl@ceph.mds.c.smithi077.stdout:Apr 21 08:59:52 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-c[62366]: 2024-04-21T08:59:52.300+0000 7f802b7b6700 -1 mds.c *** got signal Terminated *** 2024-04-21T08:59:53.604 INFO:journalctl@ceph.mds.c.smithi077.stdout:Apr 21 08:59:53 smithi077 podman[70008]: 2024-04-21 08:59:53.431789095 +0000 UTC m=+1.323356451 container died 73c3870a11e1a7b1148624948f9a7208a23daabe8ac19fc071d7155e24d139d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-21T08:59:54.102 INFO:journalctl@ceph.mds.c.smithi077.stdout:Apr 21 08:59:53 smithi077 podman[70008]: 2024-04-21 08:59:53.793644214 +0000 UTC m=+1.685211586 container cleanup 73c3870a11e1a7b1148624948f9a7208a23daabe8ac19fc071d7155e24d139d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-21T08:59:54.102 INFO:journalctl@ceph.mds.c.smithi077.stdout:Apr 21 08:59:53 smithi077 bash[70008]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-c 2024-04-21T08:59:54.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.c.service' 2024-04-21T08:59:54.254 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:54.254 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-21T08:59:54.254 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-21T08:59:54.255 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.f 2024-04-21T08:59:54.354 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:59:54 smithi077 systemd[1]: Stopping Ceph mds.f for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:55.104 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:59:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-f[62058]: 2024-04-21T08:59:54.607+0000 7f12123ac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:55.104 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:59:54 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-f[62058]: 2024-04-21T08:59:54.607+0000 7f12123ac700 -1 mds.f *** got signal Terminated *** 2024-04-21T08:59:58.780 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:59:58 smithi077 podman[70138]: 2024-04-21 08:59:58.521291509 +0000 UTC m=+4.047262105 container died decaa5941cd00e0ab21bbd3541ac93764c36b567e5bd39d8d1e2ad24983042f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-f, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-21T08:59:59.031 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:59:58 smithi077 podman[70138]: 2024-04-21 08:59:58.8641027 +0000 UTC m=+4.390073303 container cleanup decaa5941cd00e0ab21bbd3541ac93764c36b567e5bd39d8d1e2ad24983042f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-21T08:59:59.031 INFO:journalctl@ceph.mds.f.smithi077.stdout:Apr 21 08:59:58 smithi077 bash[70138]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-f 2024-04-21T08:59:59.235 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.f.service' 2024-04-21T08:59:59.267 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:59.267 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-21T08:59:59.267 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-21T08:59:59.267 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.i 2024-04-21T08:59:59.592 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:59:59 smithi077 systemd[1]: Stopping Ceph mds.i for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:59.854 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:59:59 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i[65876]: 2024-04-21T08:59:59.592+0000 7f5a4879a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:59.854 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 08:59:59 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i[65876]: 2024-04-21T08:59:59.592+0000 7f5a4879a700 -1 mds.i *** got signal Terminated *** 2024-04-21T09:00:03.855 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 09:00:03 smithi077 podman[70258]: 2024-04-21 09:00:03.599364534 +0000 UTC m=+4.111314375 container died ea868477292cecddd08fc1fc49a28a8531c0129f41958ba23e60fd9c0d5c430b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T09:00:04.226 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 09:00:03 smithi077 podman[70258]: 2024-04-21 09:00:03.976351483 +0000 UTC m=+4.488301362 container cleanup ea868477292cecddd08fc1fc49a28a8531c0129f41958ba23e60fd9c0d5c430b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416) 2024-04-21T09:00:04.226 INFO:journalctl@ceph.mds.i.smithi077.stdout:Apr 21 09:00:03 smithi077 bash[70258]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-i 2024-04-21T09:00:04.390 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.i.service' 2024-04-21T09:00:04.422 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:00:04.422 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-21T09:00:04.422 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-21T09:00:04.423 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.l 2024-04-21T09:00:04.746 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 09:00:04 smithi077 systemd[1]: Stopping Ceph mds.l for c9c60282-ffb9-11ee-bc93-c7b262605968... 2024-04-21T09:00:05.104 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 09:00:04 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-l[61738]: 2024-04-21T09:00:04.746+0000 7f62824b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:00:05.104 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 09:00:04 smithi077 ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-l[61738]: 2024-04-21T09:00:04.746+0000 7f62824b8700 -1 mds.l *** got signal Terminated *** 2024-04-21T09:00:08.262 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 09:00:07 smithi077 podman[70379]: 2024-04-21 09:00:07.954437176 +0000 UTC m=+3.315585315 container died ae73de27efd79a63a1aeca8140c8f90f73804939126cc08f9250b0fafe28d3b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-l, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-21T09:00:08.262 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 09:00:08 smithi077 podman[70379]: 2024-04-21 09:00:08.263087089 +0000 UTC m=+3.624235229 container cleanup ae73de27efd79a63a1aeca8140c8f90f73804939126cc08f9250b0fafe28d3b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-21T09:00:08.597 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 09:00:08 smithi077 bash[70379]: ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-l 2024-04-21T09:00:08.597 INFO:journalctl@ceph.mds.l.smithi077.stdout:Apr 21 09:00:08 smithi077 podman[70393]: 2024-04-21 09:00:08.480652913 +0000 UTC m=+0.522522596 container remove ae73de27efd79a63a1aeca8140c8f90f73804939126cc08f9250b0fafe28d3b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-c9c60282-ffb9-11ee-bc93-c7b262605968-mds-l, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T09:00:08.621 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c9c60282-ffb9-11ee-bc93-c7b262605968@mds.l.service' 2024-04-21T09:00:08.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:00:08.652 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-21T09:00:08.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T09:00:08.852 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T09:00:11.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T09:00:11.685 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T09:00:14.297 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T09:00:14.488 INFO:teuthology.orchestra.run.smithi077.stdout:Deleting cluster with fsid: c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T09:00:16.815 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:00:16.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:00:16.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:00:16.905 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-21T09:00:16.907 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518/remote/smithi007/crash 2024-04-21T09:00:16.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/crash -- . 2024-04-21T09:00:16.936 INFO:teuthology.orchestra.run.smithi007.stderr:tar: /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T09:00:16.937 INFO:teuthology.orchestra.run.smithi007.stderr:tar: Error is not recoverable: exiting now 2024-04-21T09:00:16.938 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518/remote/smithi016/crash 2024-04-21T09:00:16.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/crash -- . 2024-04-21T09:00:16.985 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T09:00:16.986 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-04-21T09:00:16.987 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518/remote/smithi077/crash 2024-04-21T09:00:16.988 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/crash -- . 2024-04-21T09:00:17.026 INFO:teuthology.orchestra.run.smithi077.stderr:tar: /var/lib/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T09:00:17.027 INFO:teuthology.orchestra.run.smithi077.stderr:tar: Error is not recoverable: exiting now 2024-04-21T09:00:17.027 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-21T09:00:17.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-21T09:00:17.089 INFO:tasks.cephadm:Compressing logs... 2024-04-21T09:00:17.089 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T09:00:17.092 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T09:00:17.094 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T09:00:17.115 INFO:teuthology.orchestra.run.smithi007.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-04-21T09:00:17.115 INFO:teuthology.orchestra.run.smithi007.stderr: /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.log 2024-04-21T09:00:17.115 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T09:00:17.117 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.g.log 2024-04-21T09:00:17.117 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.log: /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.j.log 2024-04-21T09:00:17.117 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.3.log 2024-04-21T09:00:17.117 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.0.log 2024-04-21T09:00:17.117 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mgr.x.log 2024-04-21T09:00:17.118 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mon.a.log 2024-04-21T09:00:17.118 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.6.log 2024-04-21T09:00:17.122 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T09:00:17.123 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.6.log: 87.1% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T09:00:17.123 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.a.log 2024-04-21T09:00:17.123 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T09:00:17.123 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.d.log 2024-04-21T09:00:17.123 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.9.log 2024-04-21T09:00:17.123 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T09:00:17.123 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.d.log: /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-21T09:00:17.123 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.admin.log: 94.5% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-21T09:00:17.124 INFO:teuthology.orchestra.run.smithi007.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-21T09:00:17.127 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.log 2024-04-21T09:00:17.127 INFO:teuthology.orchestra.run.smithi077.stderr:: No such file or directory 2024-04-21T09:00:17.127 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.2.log 2024-04-21T09:00:17.127 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.f.log 2024-04-21T09:00:17.127 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T09:00:17.127 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.5.log 2024-04-21T09:00:17.127 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.c.log 2024-04-21T09:00:17.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mon.c.log 2024-04-21T09:00:17.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mgr.z.log 2024-04-21T09:00:17.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.i.log 2024-04-21T09:00:17.128 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mgr.z.log: --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T09:00:17.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.11.log 2024-04-21T09:00:17.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.8.log 2024-04-21T09:00:17.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T09:00:17.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.l.log 2024-04-21T09:00:17.128 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T09:00:17.129 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 94.6% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-21T09:00:17.129 INFO:teuthology.orchestra.run.smithi077.stderr: 94.5% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-21T09:00:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.log 2024-04-21T09:00:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:: No such file or directory 2024-04-21T09:00:17.131 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T09:00:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.7.log 2024-04-21T09:00:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.b.log 2024-04-21T09:00:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.4.log 2024-04-21T09:00:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.1.log 2024-04-21T09:00:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mgr.y.log 2024-04-21T09:00:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.h.log 2024-04-21T09:00:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.10.log 2024-04-21T09:00:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T09:00:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.e.log 2024-04-21T09:00:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T09:00:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mon.b.log 2024-04-21T09:00:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.k.log 2024-04-21T09:00:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T09:00:17.143 INFO:teuthology.orchestra.run.smithi077.stderr: 87.1% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T09:00:17.169 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 83.8% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T09:00:17.186 INFO:teuthology.orchestra.run.smithi077.stderr: 83.8% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T09:00:17.188 INFO:teuthology.orchestra.run.smithi077.stderr: 90.3% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T09:00:17.210 INFO:teuthology.orchestra.run.smithi007.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T09:00:17.218 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T09:00:17.225 INFO:teuthology.orchestra.run.smithi007.stderr: 84.1% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T09:00:17.246 INFO:teuthology.orchestra.run.smithi077.stderr: 92.1% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-21T09:00:17.249 INFO:teuthology.orchestra.run.smithi007.stderr: 94.5% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-21T09:00:17.256 INFO:teuthology.orchestra.run.smithi077.stderr: 94.5% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-21T09:00:17.259 INFO:teuthology.orchestra.run.smithi016.stderr: 92.0% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-21T09:00:17.311 INFO:teuthology.orchestra.run.smithi077.stderr: 91.0% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T09:00:17.323 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T09:00:17.330 INFO:teuthology.orchestra.run.smithi016.stderr: 87.1% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T09:00:17.341 INFO:teuthology.orchestra.run.smithi007.stderr: 90.2% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T09:00:17.347 INFO:teuthology.orchestra.run.smithi007.stderr: 94.4% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-21T09:00:17.351 INFO:teuthology.orchestra.run.smithi016.stderr: 90.3% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T09:00:17.354 INFO:teuthology.orchestra.run.smithi007.stderr: 94.4% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-21T09:00:17.367 INFO:teuthology.orchestra.run.smithi007.stderr: 91.0% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T09:00:17.404 INFO:teuthology.orchestra.run.smithi016.stderr: 94.5% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-21T09:00:17.424 INFO:teuthology.orchestra.run.smithi016.stderr: 94.5% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-21T09:00:17.593 INFO:teuthology.orchestra.run.smithi016.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T09:00:17.691 INFO:teuthology.orchestra.run.smithi007.stderr: 90.2% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-21T09:00:17.697 INFO:teuthology.orchestra.run.smithi016.stderr: 94.4% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-21T09:00:17.737 INFO:teuthology.orchestra.run.smithi077.stderr: 91.7% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-21T09:00:19.017 INFO:teuthology.orchestra.run.smithi007.stderr: 90.3% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-21T09:00:22.999 INFO:teuthology.orchestra.run.smithi016.stderr: 91.7% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-21T09:00:39.390 INFO:teuthology.orchestra.run.smithi077.stderr: 95.2% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-21T09:01:00.991 INFO:teuthology.orchestra.run.smithi077.stderr: 91.9% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-21T09:01:05.540 INFO:teuthology.orchestra.run.smithi077.stderr: 91.9% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-21T09:01:05.776 INFO:teuthology.orchestra.run.smithi007.stderr: 91.7% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-21T09:01:07.244 INFO:teuthology.orchestra.run.smithi077.stderr: 92.0% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-21T09:01:09.169 INFO:teuthology.orchestra.run.smithi007.stderr: 91.7% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-21T09:01:11.697 INFO:teuthology.orchestra.run.smithi007.stderr: 91.5% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-21T09:01:14.767 INFO:teuthology.orchestra.run.smithi007.stderr: 91.6% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-21T09:01:14.771 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-21T09:01:14.771 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m57.665s 2024-04-21T09:01:14.771 INFO:teuthology.orchestra.run.smithi007.stderr:user 3m12.928s 2024-04-21T09:01:14.771 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m5.799s 2024-04-21T09:01:19.597 INFO:teuthology.orchestra.run.smithi077.stderr: 91.6% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-21T09:01:19.599 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-21T09:01:19.599 INFO:teuthology.orchestra.run.smithi077.stderr:real 1m2.491s 2024-04-21T09:01:19.599 INFO:teuthology.orchestra.run.smithi077.stderr:user 3m27.766s 2024-04-21T09:01:19.600 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m6.547s 2024-04-21T09:03:21.986 INFO:teuthology.orchestra.run.smithi016.stderr: 91.9% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-21T09:06:32.472 INFO:teuthology.orchestra.run.smithi016.stderr: 91.8% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-21T09:08:37.064 INFO:teuthology.orchestra.run.smithi016.stderr: 91.6% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-21T09:11:44.182 INFO:teuthology.orchestra.run.smithi016.stderr: 92.1% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-21T09:14:13.193 INFO:teuthology.orchestra.run.smithi016.stderr: 93.1% -- replaced with /var/log/ceph/c9c60282-ffb9-11ee-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-21T09:14:13.211 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-21T09:14:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:real 13m56.084s 2024-04-21T09:14:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:user 6m24.869s 2024-04-21T09:14:13.212 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m15.087s 2024-04-21T09:14:13.213 INFO:tasks.cephadm:Archiving logs... 2024-04-21T09:14:13.213 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518/remote/smithi007/log 2024-04-21T09:14:13.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T09:14:46.228 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518/remote/smithi016/log 2024-04-21T09:14:46.229 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T09:15:45.388 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518/remote/smithi077/log 2024-04-21T09:15:45.401 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T09:16:19.118 INFO:tasks.cephadm:Removing cluster... 2024-04-21T09:16:19.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 --force 2024-04-21T09:16:19.320 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T09:16:19.870 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 --force 2024-04-21T09:16:20.338 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T09:16:21.036 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c9c60282-ffb9-11ee-bc93-c7b262605968 --force 2024-04-21T09:16:21.226 INFO:teuthology.orchestra.run.smithi077.stdout:Deleting cluster with fsid: c9c60282-ffb9-11ee-bc93-c7b262605968 2024-04-21T09:16:21.762 INFO:tasks.cephadm:Removing cephadm ... 2024-04-21T09:16:21.763 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T09:16:21.782 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T09:16:21.801 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T09:16:21.817 INFO:tasks.cephadm:Teardown complete 2024-04-21T09:16:21.817 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-21T09:16:21.828 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-21T09:16:21.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T09:16:21.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T09:16:21.844 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T09:16:21.982 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-21T09:16:21.993 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-21T09:16:21.993 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T09:16:21.996 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T09:16:21.999 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T09:16:22.011 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-04-21T09:16:22.012 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-04-21T09:16:22.014 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-21T09:16:22.016 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T09:16:22.016 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-04-21T09:16:22.016 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 -221us[ -221us] +/- 97ms 2024-04-21T09:16:22.016 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -467us[ -478us] +/- 61ms 2024-04-21T09:16:22.016 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +727us[ +727us] +/- 70ms 2024-04-21T09:16:22.016 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T09:16:22.018 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T09:16:22.018 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-04-21T09:16:22.018 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 -213us[ -203us] +/- 97ms 2024-04-21T09:16:22.018 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -466us[ -456us] +/- 61ms 2024-04-21T09:16:22.019 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 +724us[ +734us] +/- 70ms 2024-04-21T09:16:22.019 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T09:16:22.052 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T09:16:22.052 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-21T09:16:22.052 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 -208us[ -208us] +/- 97ms 2024-04-21T09:16:22.052 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -437us[ -435us] +/- 61ms 2024-04-21T09:16:22.052 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 74 +707us[ +708us] +/- 69ms 2024-04-21T09:16:22.052 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T09:16:22.053 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-21T09:16:22.064 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-21T09:16:22.065 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-21T09:16:22.118 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T09:16:22.253 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T09:16:22.333 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T09:16:22.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T09:16:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:16:22.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T09:16:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:16:22.459 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T09:16:22.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:16:22.485 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-21T09:16:22.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-21T09:16:22.534 INFO:teuthology.task.internal:Duration was 2977.628902 seconds 2024-04-21T09:16:22.534 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-21T09:16:22.568 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-21T09:16:22.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T09:16:22.571 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T09:16:22.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T09:16:22.610 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T09:16:22.633 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T09:16:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T09:16:23.099 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-21T09:16:23.099 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-04-21T09:16:23.099 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T09:16:23.124 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-04-21T09:16:23.124 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T09:16:23.147 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-04-21T09:16:23.148 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T09:16:23.179 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-21T09:16:23.179 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T09:16:23.182 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T09:16:23.190 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T09:16:23.245 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-21T09:16:23.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T09:16:23.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T09:16:23.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T09:16:23.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-21T09:16:23.664 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-21T09:16:23.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T09:16:23.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T09:16:23.714 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T09:16:23.738 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-21T09:16:23.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:16:23.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:16:23.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:16:23.778 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-04-21T09:16:23.790 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-04-21T09:16:23.800 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-04-21T09:16:23.823 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:16:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:16:23.847 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:16:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:16:23.874 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:16:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:16:23.887 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-21T09:16:23.899 INFO:teuthology.task.internal:Transferring archived files... 2024-04-21T09:16:23.900 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518/remote/smithi007 2024-04-21T09:16:23.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T09:16:23.941 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518/remote/smithi016 2024-04-21T09:16:23.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T09:16:23.976 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518/remote/smithi077 2024-04-21T09:16:23.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T09:16:24.040 INFO:teuthology.task.internal:Removing archive directory... 2024-04-21T09:16:24.041 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T09:16:24.044 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T09:16:24.049 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T09:16:24.072 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-21T09:16:24.096 INFO:teuthology.task.internal:Not uploading archives. 2024-04-21T09:16:24.096 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-21T09:16:24.116 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-21T09:16:24.116 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T09:16:24.119 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T09:16:24.122 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T09:16:24.132 INFO:teuthology.orchestra.run.smithi007.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 09:16 /home/ubuntu/cephtest 2024-04-21T09:16:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 09:16 /home/ubuntu/cephtest 2024-04-21T09:16:24.137 INFO:teuthology.orchestra.run.smithi077.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 09:16 /home/ubuntu/cephtest 2024-04-21T09:16:24.146 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-21T09:16:24.169 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-21T09:16:24.244 INFO:teuthology.nuke:Checking targets against current locks 2024-04-21T09:16:24.268 DEBUG:teuthology.nuke:shortname: smithi007 2024-04-21T09:16:24.268 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T09:16:24.289 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:06.310808', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBKjjGJRTuaUxct5Q5Wcl9QsCi7UWUAOc3L0FyKS+058JLFaZuhEyHCRCGbMZByxCoCP/DZLb4KtLwXvi7Y1nt4='} 2024-04-21T09:16:24.293 DEBUG:teuthology.nuke:shortname: smithi016 2024-04-21T09:16:24.293 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T09:16:24.313 DEBUG:teuthology.nuke:shortname: smithi077 2024-04-21T09:16:24.313 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T09:16:24.321 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:06.308956', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCPA5gnhRYpDDSIONHCWDTz5JjIYaaFC7IENuniK6XodNcSPGlnzQ4npkgK5yMsq8C7UxDuUYX2ePEQCIqFyf/o='} 2024-04-21T09:16:24.347 INFO:teuthology.orchestra.console.smithi007:Power off 2024-04-21T09:16:24.348 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T09:16:24.360 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666518', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:06.312132', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAD+Od2KpE3QA4lzWnpiQXsIuqKwIoe2HhQedEiSXETzNJtyvIj8o5FNSDEYOygCckGi3aHGnCyIq4VNj2cxaBc='} 2024-04-21T09:16:24.371 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-04-21T09:16:24.371 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:16:24.387 INFO:teuthology.orchestra.console.smithi016:Power off 2024-04-21T09:16:24.388 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T09:16:24.402 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-04-21T09:16:24.411 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2024-04-21T09:16:24.412 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:16:24.427 INFO:teuthology.orchestra.console.smithi077:Power off 2024-04-21T09:16:24.428 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T09:16:24.437 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-04-21T09:16:24.451 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-04-21T09:16:24.451 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:16:24.479 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-04-21T09:16:28.403 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:16:28.438 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:16:28.480 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:16:28.518 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-04-21T09:16:28.546 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-04-21T09:16:28.590 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-04-21T09:16:32.522 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:16:32.547 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:16:32.591 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:16:32.633 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-04-21T09:16:32.656 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2024-04-21T09:16:32.699 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-04-21T09:16:32.734 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-04-21T09:16:32.757 INFO:teuthology.orchestra.console.smithi016:Power off completed 2024-04-21T09:16:32.800 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-04-21T09:16:32.907 INFO:teuthology.run:Summary data: description: fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} duration: 2977.628901720047 failure_reason: 'Command failed (workunit test kernel_untar_build.sh) on smithi007 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0bf063190c9c4617b0ca798cd5eef841 status: fail success: false 2024-04-21T09:16:32.907 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T09:16:32.991 INFO:teuthology.run:FAIL